I'm not convinced that Ballista were incapable of damaging castle walls. They are certainly classed as a seige weapon and had a prominent role in many seiges. There are also depictions of ballista firing missiles with a solid metal ball on the end which would only really have been necessary if one was hoping to smash something solid at the receiving end.