Other factions' Temples and Government buildings are the only structures that give culture penalties.
I realised this when I captured Alesia. I destroyed more-or-less all the buildings and just left the Gaul shrine and obviously couldn't destroy the Great Hall. When I restored the garrison, I leveled the Gaul temple and built a Jupiter shrine. This had a double effect giving you 5% public order and 5% happiness AND reduced culture penalty by one of those icons.
Barracks, roads, stables and such don't seem to effect culture.
Bookmarks