Quote Originally Posted by Aries
but you can upgrade them when the population grows to a certain ammount
Not really. if you capture a Huge City from the enemy. or as the Barbs capture a minor city or above you will not be able to upgrade... Tech tree limitations.

So sometimes you'll just have to live with it and suck up that 20% odd, penalty.

The lack of an option to demolish the Governor's residence is however rather silly when you think about it.