I like the challenge of taking the walls, so I wouldn't say it's a "do" or "don't" thing... just personal style. If you get enough men up there, after the main gate you can send them to capture the adjoining towers, so you don't get shot in the back if you're moving troops around the perimeter of a city. But yeah... you do need overwhelming force, and some attacks with ranged weapons beforehand.

If I'm in a hurry, I'll flood the city with spies so I can come in from multiple directions with open gates. Lots of ways to do it, makes it interesting.