I think it would be great if forts had a sort of 'minimum garrison', IE: just the people required to man the forts guns and keep it operational and in good condition. This would be represented by an upkeep and men manning the cannons in/on the walls (as I believe forts generally had heavier cannons designed specifically for the fortification role, as well as additional smaller guns).
Additional units could be brought in as needed to man the walls and all that.

It would also be great if forts could be abandoned as needed, and, again, as needed, refurbished. Whether by you or your enemy, for a lower cost.