Given that those settlements didn't become factions, due to non-expansionism, let them be not aggressive :P