Castles really aren't meant as an offensive weapon. By that I mean you don't capture a territory and start constructing a castle to launch your next attack from. The times I used enormous castles in Total War was at the chokepoints where it was the only way into my territory. That way instead of keeping a garrison in eight-twelve provinces I kept an enormous one in a single highly defencible province that had enormous supplies of many seasons. They never did actually /defeat/ my forces to warrant an attack on the castle but hey.

Another thing I always found useful about building castles were that as soon as I had them up I could build ports. Ports brought in a lot of koku. By the time I finish any game I have in excess of 35,000 troops and 50,000 koku.