Is it possible to kick down ladders from walls or to destroy them as they come in?

I discovered today that troops can knock down ladders if they are at the bottom, but couldn't do anything from the top as far as I could tell. I've never been able to destroy ladders with siege weapons or archers, and the only way I can stop them from getting to the walls is to run out with cavalry and hit the units carrying them. Sometimes they pick the ladders back up, but not too often.