Riff-raff units too strong?

[Titus Marcellus Scato]

Then I say get rid of elite units (remove them from the game), or buff them.

No need for either. Elites have their uses and have enough of a superiority over the rank and file basic units.

For example, in a phalanx or spearman army, the unit on the extreme right of the line should be an elite unit, because that is the most vulnerable position in the line. The shield is held in the left hand, meaning the unshielded right side of the rightmost men is open to enemy attack. The enemy know that, so it's the most likely place in the line to be attacked. You need a good unit there which won't break when it suffers heavy losses, since if it does rout, your whole line could be rolled up one unit at a time.

The Romans use their elite triarii or pedites extraordinairii to stem an enemy front line breakthrough or hold a collapsing flank when the battle starts going pear-shaped.

Armies of swordsmen need a elite breakthrough unit to hack a hole in the enemy front line, that's what units like the Gaesatae or Solduros are for.

An elite unit of heavy cavalry can either turn an enemy flank by driving off the enemy cavalry, or act as a reserve when enemy cavalry get round your flank.

They are for critical points on the battlefield, and the critical moments, when you absolutely cannot afford them to fail otherwise the whole battle is lost.

Or, you could manage without elite units, and just rely on superior numbers and/or superior generalship to win the day. Depends on your playing style.