Castles are indeed dependant on population, but not the first three upgrades.
Going from motte and baily to wooden castle, and then to just Castle are not dependant on population. You can do that for any settlement. To upgrade to fortress you need a population of 4500, and to go to citadel you need 9000 population.
Bookmarks