You need to destroy the barracks because they are in most instances useless to you (unless you share barracks with a faction).

Some temples do not work with your faction, I have found that problem with the Romani too. Some temples cannot be upgraded or give a bonus of 5% while they are at the wonderous level.

~Fluvius