If it is the same horde then perhaps it's because you take their last settlement after they have settled down.