I find having a unit or two of archers stationed in settlements often helps, the fire arrows (with a little luck) can take out enemy rams and towers and either send the enemy packing or if they've brought ladders and/or sap points, decrease the number of chokepoints to hold the enemy at.