Yeah, playing on Very Hard is much more difficult when you're playing a melee-based faction. The +7 bonus given to enemy defense is only applied to defensive skill, so your arrows still do full damage on Very Hard mode. Without horse archers you'll just get stomped.