I don't. Remember something, the unit scales are different in ancient battles than they were in medieval battles. Armies of 100,000 men per side were not uncommon in the ancient world. They were unheard of in medieval times. It's a lot easier to raise medieval armies which would in reality consist of a couple of thousand men than it would be to raise 40,000 men from a province in roman times, without depleting the population.

There's already something of a limiting factor in the recruitment pools. Yes, a settlement can recruit up to 3 units per turn, but each unit type also has a maximum number of units available for recruitment at any given time. The better ones don't replenish too quickly. I've just reached the point where I can build musketeers in my Spanish campaign; one unit was available when I completed the barracks (15,000 florins and 8 turns by the way, and that was the last of 8 barracks buildings I had to build to get there) and new ones only come up every other turn afterwards. It's going to be a while before I can even put 4 units of musketeers into one army. Trash troops can be quickly raised all over, elite troops require a lot of investment in infrastructure all over before you can spam hordes of them.