I imagine that re-taking one of your own starting settlements and enslaving it would have this effect.