The towers are the bane of everyone on the walls. They fire what is basically the equivalent of crossbow bolts, with high attack rating and the AP characteristic, so they do a lot of damage to heavily armored units like dismounted knights. Unfortunately, the damage is based on collision detection, so they'll shoot the defenders in the back as well as the attackers. As long as there are a lot of attackers on the walls, they tend to be hurt worse, but once the defenders really outnumber them they'll start getting hit hard by the towers.

At first when defending, I used to put a unit right there where the siege tower or ladders were going to come up against the wall. Now I tend to put a unit on either side near the towers, that way the tower shots hit the attackers more than my troops.