It used to be elephants, especially the War and Armored variety... until I modded them to be less than nigh-invincible... I reduced their hit points and more importantly, their armor ratings, to more realistic levels. After modding elephants are still quite fearsome but at least my missile units can make an impression on them now.

Heavy Chariots are also a pain but thankfully they have crappy Defense ratings and are easier to kill once you take the fight to them.

Beyond this I'd have to say that enemy Cataphracts consistently bring sweat to my brow, especially when there's two or three charging across the field and they're not impaling themselves on my spear and phalanx units. There's nothing worse than having enemy Cataphracts trash your cavalry and then take on your general unit or proceed to outflank your infantry only to have you count the moments until it all goes to Hades in an handbasket.