I've successfully defended various fortifications several times. It's true that at ANY level, your walls are in danger of being hit by your own defenses. However, rarely is it enough to destroy your wall or make you vulnerable. You should have at least one cav unit in your garrison to take out as many siege engines as possible at the start of the battle. Tha way when your defenses damage your walls, they never destroy them.

Ultimately I would build all the walls I can because they serve as a deterrent and dramatically help in the defense.