Keeping the pagan/whatever temples may be good for the obvious reasons.
However, I'm wondering if having other cultures' buildings in your city doesn't add to the "culture penalty" (which decreases public order), and which, in my case, is pretty significant in a few towns. So I am really considering destroying those temples and building roman ones, at least I'll see if I get rid of some of that culture penalty thing.