Were they falling off of the wall? or some were in the tower that accesses the wall?

You will also see this situation in bridge battles where troops either fall off of the bridge, or get too close to the waters edge. Extremely annoying for your cavalry when chasing routers to lose more men to either event than they lost in actual combat......