Upgrade the walls to Large Stone Walls or better, and pack the towns with archers. The enemy will only be able to make rams and towers, which you can burn down with your archers. It's a pretty failsafe defense.
Is this still the case? I've noticed the enemy AI building ladders and sap points in my recent seiges.

(Mind, I've also had rebel stacks attack my settlements while I was playing on M campaign difficulty, so, maybe my rig is just possessed.)