If a temple can be upgraded as a setlement grows I'll usually upgrade if not destroy and build a fresh. I'm pretty certain apart from the govt building a temple has the biggest cultural penalty, which can hurt a lot as that plundered cities pop recovers.