Its not actually losing that money. It just supports a much larger budget of your army and other expenditure. This is based on the size of the settlement.

Foot