Don't build all the structures, it's as easy as that. You should only make farms and granaries when you need it. Temples that do trade goods, troop moral and health should also be avoided for towns that are far away from the centre of your empire, the extra 25 or so public order you get from that helps a lot. Bigger the population is, the less effect the garrison has. I try to aim to get 24,000 peeps in each city so I can get the biggest city upgrade. Then I make stuff like colonies to knock the population back down some.