Once they hit the walls you can target an onager at the unit and once you make a piece of the wall fall the ladders will be destroyed.