Two things :

1) Cavalry can't go up walls. So the uber heavy cav is silent untill the grunts have taken the inner gate.

2) you have huge bonuses for defending walls, not to mention the towers will do a massacre on pinned infantry.

So yeah, the inner walls are less defended than the first, but they still provide a heck of an advantage over just the outer wall .