Huge is best, because of the EB script only the player gets stung from depleting his cities, not the AI. Which means the player is encouraged to recruit as little as possible and use his army efficiently. I also hire as many mercs as possible, it's worth the money to avoid depleting your town population.
Pathfinding in towns is difficult on Huge unit size, but that is as it should be. There's a reason why most ancient battles were fought on the field and not in city streets - street fighting did make armies very hard for generals to control.
Bookmarks