As Subrosa pointed out, it does happen but only, AFAIK, with giant siege towers.

I always build only four tower and get that problem generally with one or two towers.

I have not found any satisfactory way to deal with this.

Clicking as often as possible on the wall will get your men on the wall after a while but it's quite a drag. After several orders , the unit will eventually get on the wall but Once it's there, be carefull if you try to move the unit around because there is quite a good chance that a few soldiers are still running silly at the tower's entrance. Thus the affected unit will (after a while and quite a few click) be able to get on the wall and fight but do not expect it to be available to sprint around the wall to capture towers and gates ...

IMHO, trying to get the unit to abandon the tower without even trying to climb is risky cause you have good chance that some silly men will remain "attached" to the tower despite whatever you tell them to do ... Hence a good chance to get your unit stuck in the no-man's land when it will just get is very fair share of arrows ...

Pretty annoying stuff thus ...

When I have to assault "mega-walls", I usually sap if the city is heavily defended. If the city is lightly defended, getting one or two units on the wall is usually enough to capture all the necessary towers and gates. You'll have one or two useless units down the wall but they will be safe once you get the nearby towers.