Ok, the temple may not give the cultural penalty but the other buildings will. So if I found it difficult to mainatin order, I would destroy some of them and then rebuild. That will reduce the culture penalty.