I often leave them. Always upgrade them if you can cause then you'll have a temple of your own culture (if different), but if it's the highest level, I can't imagine it being worthwhile to tear down and build all the way back up again for a few culture points. Maybe if you want to do some roleplaying you can desecrate a temple occasionally, but I think the pagan nations of the day had a decent amount of respect for other nation's gods.