I haven't actually played out many sieges but when I have, the greatest effect I've noticed that the walls have is to protect units positioned on them from missile fire (not siege weapons mind!).