The settlement with those buildings is stuck all the way down there, because there's no good reason anyone would want to trek down there to take it. Those buildings are markers we need to make the rest of the game work as it should.