Culture penalty isn't simply buildings; there's an entire separate mechanic to represent it, and it doesn't change quickly.