Bypass the gate. It's easy to knock down the wooden walls with peasants or some fodder unit. A single ballista has enough ammo to knock down a wooden wall. Once inside, the fort can be attacked too.