Cultural penalties come from buildings in the town that were built by a different culture. Tear them down or upgrade them and the cultural penalties will decrease. Upgrading the city size does great help in decreasing cultural penalties.

Enemy spies are often the reason for weird unrest. But, sometimes having unexplained unrest is just a weird thing that happens to a few cities.