Generally, as long as I have a halfway decent force within a city, I'm quite happy to let them try and assault me. Even if they do succeed in capturing the place, I can usually inflict such a mauling that I can waltz in the next turn or so and relieve the AI of their new settlement.

For example, taking Londinium off the WRE in BI, the Romano-British when they pop up go straight for it. Playing as the Saxons recently, 6 basic infantry units, 4 archers and 3 peasants were enough to inflict ~80% casualties on the full stack that popped up, allowing my Eburacum garrison to come in and win the place back without any problems. (Stone walls)

Also, fighting in the streets - any size settlement - effectively negates most of the advantage from numbers - was able to completely destroy a 3 quarter stack Roman army (several comitatenses, limitanei, archers, etc. oh, and a 6 star general) with 2 units of spearmen and a 3 star faction leader. (Stockade)

Basically - cities are deathtraps for assaulting armies, so let them die.