They don't charge when you want them to
They take forever to aim those damn pilum
They're not spearmen so not as effective in defense
They can indeed push seige equipment but not all battles are fought on the walls...

I give up already