Even if you do'nt destrpy there temples the culture difference will go away, where the problem can kick in is when the city grows you ca'nt upgrade it and there may be no way to bring up the happiness other than to destroy it and tech it up for better happiness