Castles and towns represent you focusing on either the feudal, manorial system or giving more automony (and resources) to building up the towns. Converting a town to a castle simply means you build more castles in the area to control it, install a local nobility, distribute the lands to landowners, establish villages etc. So you aren't actually vaporizing Jerusalem with its 32,000 people and building a castle in its place, you are just concentrating on the castles more. Castles and towns are an abstraction anyway; do you really think that there are only two castles and one town in the whole of Medieval England?

A better explanation would be that they don't want cheesy strategies revolving around constantly and conveniently 'switching' back and forth between castles and towns.