I like to keep a few island castles as emergency retreats in case of a run of bad luck. Since you can pretty much defend a castle with one unit against any force the AI decides to land on an island, theyre fairly safe as a way to build up your forces again if you happen to suffer an unlucky blow.

Of course, I'm overly cautious in my play, so...