I think you should wait until you have the next Governor building and then also replace the existing barracks or whatever with your own. So lets say a city has a barracks and the population is under 2000. When it gets to 2000 upgrade to the next level and build the militia barracks, then you can start training units. This way instead of being able to use the existing facilities you force yourself to only train units out of your own buildings. At the same time this will reduce the culture penalty by a lot too.