Once I have a world headquarters of something I usually get rid of extras of the same type. The penalty for tearing it down won't hurt you at all then. The things I keep most are Chapterhouses for Knights Orders and Woodsmens Guilds as well as Swordsmith's Guilds...those come in handy.