Also make sure that the place where you're trying to move them can actually have the settlement or port. Sometimes we may end up putting them on a patch of terrain that doesn't support a settlement, or on top of a resource.