How to destroy enemy siege units?

[Aedile Rich]

Ladders don't seem to be destructible

I can assure you that they are.
Use an infantry unit and hover the cursor around the bottom of them until it changes to a sword.