I leave them, because when I upgrade them, they become my faction's type. And no, it doesn't affect at all. Except for churches and mosques, which I raze and build my own ones.