I find that the main problem with using only Sap points is that you need to blast more than one hole in the wall to be effective... Otherwise the enemy will just mob you on the other side.

Taking the walls is the best way to minimise casualties, and Siege Towers are best for this because (unlike ladders) your men actually manage to survive long enough to get up there.