It seems random. I've had it happen a couple of times myself. Most of the time it's just irritating, but once it forced me to reload a save and fight again. I only had two siege towers (large stone walls), one was set on fire by the defending archers, and the unit on the other one got stuck when they reached the walls. What do you do? No rams, no saps, and couldn't build ladders because the walls were too tall.