It really depends on what temple, i usually keep them due to there bonuses. However in my last campaign i destroyed the temple complex in rome , and about every other destructible building.