Same for me, I almost never build farm upgrades, because you can't get rid of them. But in my current Julii campaign, I'm using temples of Ceres to get cities up to a useful size, then destroy them and replace them with Jupiter temples. This seems to work very well up to now. Of course, if you take a very small town and enslave/exterminate, it'll be a village of a mere 400 guys. If you play out your campaign at a halfway decent speed, this settlement will never grow big enough to cause you pains.