Well your not wrong, every unit u recruit drains population.
I always tend to recruit peasants in recently enslaved towns and transfer them to towns that need population boosts to achieve the next level, especially towns where u focus on military cause u want them to produce the best unit available for your faction to have as soon as possible.The towns u drain will make good money for a litle longer this way.