For roleplaying reasons I would not destroy any temples (except for the Romans, when I had made a proper evocato prior ;-). Any other building, unique or not, can fall victim of plundering if I am not playing someone of the same culture. When I had sacked Rome once with the Sweboz I had behaved in a way that the Romans would have greeted the Vandals as liberators.