Last time I lost a bit of wall was in a custom game where my defensive artillery shot a hole while trying to see off some routers - nearly a disaster as my attention was still on the melee at the gate!

I find that when I start a siege game all my troops are on the wrong settings - engage for spears, hold formation for skirmishers - and if I don't change them I can end up losing. But with the settings right it's possible to make the enemy pay heavily for every inch of sand - even if you lose the castle you may well have eliminated the enemy army in the process. I once crippled the Golden Horde by bribing a garrison commander and letting the three full stacks in the province batter themselves to pieces against the walls.