I tend to think in terms of "how many castles do i need, and where do i need them?".
I will try to have one castle near each likely trouble hotspot, to produce frontline troops.
Everything else usually gets converted to a city when i capture it, for the income.
Bookmarks