You'd really only lack the archery buildings and the stables, you'd have barracks on the maximum or close it however.
Stables aren't that big of an issue as the fortress/citadel upgrades themselves supply you with fairly decent heavy cav. Just the very late units will still need time.
Its really a trade-off which way you do it (town-castle or castle+) but the way to do it remains the same.