Well it would make *some* sense... I mean, castle walls probably took ages to build, not to mention the manpower required to get the stones, carry them all the way up the cliff/moutain/hill and then set them properly.
It would therefore be understandable that besiegers would want to keep them as intact as possible and opt to wait the defenders out, shortening said wait by lobbing flaming and/or diseased stuff inside and hoping to hit the granaries/wells/living quarters/arrow stocks etc...
Or, in short, target the people and most importantly their will, instead of the castle itself.
I'm not saying that's really what they did, nor that your historian is right in saying so, but it's not a preposterous idea either.
Bookmarks