In my experience the Rally command does work, but not every time. It almost never works for low morale units, like peasants or lower value spearmen. It does usually work for higher morale units, and for disciplined units it almost always works.

I think it depends on other factors too - how many other units are routing or rallying, is your general still alive and on the battlefield, are there enemy units on the flanks or in pursuit, how tired the unit is, etc, etc...