This certainly is a delicate matter. Getting the AI to field better armies is easy enough - simply remove every bad unit from the game and lower tech reqs and cost to something merely symbolic. Maintaining gameplay while using this strategy, however, is, at least for me, inconcievable.

How to balance it out? TBH I feel that the XL crew has come pretty darn close. Better infrastructure from the start, bumped farm income and tuned AI preferential values combined make for better armies.

Personally, I feel that most units should be obtainable from castle-level. Only some really elite bad boys should need the citadel (which will be worthwhile still, since there's a valour bonus from master level buildings). Thus all guild-level troop producers should be available at castle-level. The fortress should be given something extra, like a farm 100% bonus, aside from the benefit of the military academy. Perhaps also even more money attached to the special fortress-level buildings.

Removing the armourer line? Remove the effetcs of the line, I'd say. But it can serve it's purposes still. Like Muwahids at Spearmaker2 and Saracens at Spearmaker2 + Armourer1. This way you don't have to go to castle to get Saracens but still need a little more effort to go beyond Muwahids.