They annoy me too, not because of difficulty or lack thereof, but because of the fact that units in the town square never flee and seem to take forever to kill. Which means I have to sit and watch my 3 or 4 units of shock troops take on a shattered unit of levy spearmen who just refuse to die in any timely fashion.

The one glaring exception to this rule is when I'm playing as Carthage and trying to subjugate Africa. Few things please me more than filling those Numidian skirmishers with arrows as they dash about like headless chickens behind their feeble walls.