I had just assumed I was using it wrong, as I've never had a fleeing unit rally via the General's rally troops ability. Apparently I'm not alone.

I even had my general gallop up behind said fleeing troops (a unit of falxmen who had been cut down to 45 men out of the full 80) and hit the rally button when the lead horse of my general got in amongst the falxmen. No effect, even though the enemy were fleeing the field at the time.