Well personally I can't see how the actual logistics of turning a town of 24,000 people into a castle surrounded by rural manors would have worked, so I can understand perfectly why it is impossible in the game.