Population. The tresholds for towns are:

2000 - large town
6000 - city
1200 - large city
2400 - huge city

For castles, it's less than half of that. I think fortresses are 4500, and citadels 9000.