What could be nice is if you built a fort and constantly kept it manned then over time it would get new feature like a wooden keep, then a stone keep and finally some stone walls...

But as soon as you abondoned it it would disappear like any other fort (pesky peasants stealling all the stuff for building farms and low walls..)...

Minimum fuss and a bit of fun...