I tried huge, but the large setting seems to work best for me. Unit pathfinding through city streets isn't great to begin with, and it was more difficult for me to control the huge unit size in cities. With open field battles, I do like the impressive size of the armies at the huge setting. But they look more like ants when I keep the camera at a distance where I can manage all the action. I like seeing things at a little bit closer scale.

But the main thing was the way it affected recruiting. I had to micromanage population flow a lot more (moving peasants from other towns, enslaving cities), to keep enough population in my main military/recruiting cities for building armies. And I was concerned that the AI would have the same problem, leading to less challenging armies to face.