25% of the penalty is caused by the residence, the rest by the other buildings. The culture penalty can go 0 even with some buildings of different culture left .
I must say, I'm really happy that this penalty doesn't exist anymore in M2TW...