As the Egyptians, I took the town with a half stack, built some peasants, then left. Never had problems with it again.