IMO, the defenders should be obliged to have units on the wall close to the tower for it to be able to fire. Once the attacker is in control of the wall, the tower should stay silent until the defender retakes the wall - not until a 5-man unit of exhausted peasants approaches its foot!
Agreed, thats what annoys me.