Perhaps a restriction on certain units? That for every HA you select you are allowed to pick 1 unit less? And for elites 1 unit less as well?

HA factions were undoubtedly seriously outnumbered when they fought infantry based armies.