Which is one of the most annoying things in the game when it's your unit. No matter how many times you get your general to rally, they just stand there on the wall
This actually ruined a battle in original RTW. I was defending a town, but was overmatched by the enemy. Some of them routed on the walls, but most got through to the town square. By using a cavalry unit to constantly charge into the narrow street I destroyed their entire army. The catch - the enemy soldiers on the walls were still "fighting to the death", and would not come down. As I had no infantry, I could not clear them out. Plus, I had the timer switched off. End result: I had to exit the battle, losing me the town and my hard-won victory.

Those cavalry were the bodyguards of one of my best generals, by the way .