Taverns?

In most of the cities that got occupied I demolished the shrines/temples but what I noticed is that the taverns are still there did'nt realize they would cause a cultural difference. Does that cultural difference ever go away because I notice if you leave a temple there the cultural difference will go away, but it seems that these tavern are causing a huge problem I notices the culture differnce has gone down but it seems in those cities it just wo'nt go away. So will it eventually go away or will I have to destroy these to bring full order?