That's because they're basically Balkan Billmen, who get massive bonuses against armour and cavalry, and are specifically designed to kill armoured horse.

Try matching your Voynuk Swordsmen against some Kats and watch them run crying to Mama.