when your population regrows i believe the culture penalty drop quite a bit.

and make sure u destroy their temple