I almost always use ladders. Going head to head against units with the wall defense bonus is going to cost you a lot of lives no matter which type equipment you bring. Ladders are much more flexible, they can draw enemies away from the gate or simply run to an undefended part of the wall and claim it.

All that being said, it comes down to personal preference. If you want to fight on the walls because you prepped your forces with superior infantry bring towers, if you want to take gates and force the enemy to retreat to their town square (Usually because you prepped with archers), bring ladders.

You can use any 4 random militia units to defend any huge walls city against full stacks of AI troops due to bugs. Stop the ram or clog the gate with routers and the AI's pathetic 'trickle in' routine with siege towers cannot capture your walls. It's irrelevant at that point how many men they bring.

If you have less than huge walls and the AI brings more than one ladder crew, you'll be in for a fight, but your max freebie garrison still has a pretty good chance against a typical AI full stack.