Yep, as I said.. it's a matter of culture penalty (can you get rid of it, without demolishing the building?... since there's a cap at 50% I think that lowers while replacing buildings it is possible to get rid of it, without destroying every building of another culture) and of the benefits (Do you need/want them?)

If you can answer one of the above questions with no, then replace it with your own...