Personally, I've found castles to be quite useful for border defense. I mean, once you can upgrade an area to a Minor City, sure they aren't that useful, but a castle with Stone Walls is much easier to defend than a Town which can barely kill anything. You can really see this in the Russian campaign. If you turn all of the easter castles into towns, the Mongols will roll right through them.

Also, I like to have some near my border for troop retraining needs. It is a lot easier to fall back a one-turn march than spend ten returning to your troop base.