How to destroy enemy siege units?

[Aedile Rich]

Ladders don't seem to be destructible

I can assure you that they are.
Use an infantry unit and hover the cursor around the bottom of them until it changes to a sword.

[Oaty]

I'd say its bugged in many ways.

First a unit of 5 men can destroy arttillery/siege equipment much faster than a unit of 50 men or more.

Cavalry can't (or it takes too long) destroy equipment. Out of a unit of cavalry, 1 horse will attack the equipment while the rest of the unit disperses and if I had the time, the unit probably would disperse all the way to no mans land(red line). Never have gotten above 0 percent damage due to the time it takes

So far I've only had infantry able to destroy the equipment. And can take ages usually meaning it's better to wait for the timer to run out at triple speed.

This hurts the AI more, as this usually occurs when a good portion of thier infantry have been destroyed and will no longer scale the walls. So it takes ages to destroy thier siege equipment where they will will rush in thier cavalry to get cut into peicemeal while you run your infantry back up the siege equipment.