Yep, also took me some time to figure out too. But it IS in the help text during the siege. You have to pick units to man the ladders.

The game certainly has it flaws but I am very much enjoying it so far. I am sure they will polish it some more with upcoming patches!