I tend to think in terms of "how many castles do i need, and where do i need them?".

I will try to have one castle near each likely trouble hotspot, to produce frontline troops.

Everything else usually gets converted to a city when i capture it, for the income.