I would have agreed with you until yesterday. I pulled up a very high quality siege army with 3 trebuchets to take down a city with huge stone walls. Feeling rather cocky, I setup in the usual style and began flinging stones. Almost as soon as I hit the start button, I began hearing strange booming sound effects. Within moments, one trebuchet fell to pieces and a quick check of the casualty meter showed nearly 10% losses. There was not a single enemy on the walls at all, but it turned out I was attacking a city with cannon towers, something I had not bothered to check beforehand.

I tried to pull back out of range of the towers, but they had a longer range than my trebuchets. Within moment, 4 of my 6 trebuchets were destroyed and I had 20% casualties. This is without the enemy units themselves firing a single shot.

I had to withdraw from the battle and I have postponed the assault on that city until I could build gunsmiths to produce my own cannons. Sure, it took huge stone walls with cannon towers to defeat me, but defeat me they did.

So... some walls may be weak, but not all of them are.