Garrisoning buildings is extremely overrated by the AI. It's a death trap in several ways.
If attacked, the AI always garrisons several buildings, which leads to the following

1. Half it's army is garrisoned in buildings, but since their army is in mortar range, it will respond to coming under artillery fire by assaulting your line. However, it does so without all the garrisoned troops, effectively letting you face half their army and then mopping up the remains.

2. Light infantry, riflemen or indeed anything with a range of more than 70 can comfortably spend all their ammunition killing about half or more of a garrisoned unit, somewhat dependant on the type of building, without taking a single casualty. Those buildings where half of the unit will be on the roof are particularly voulnerable to this.

3. As is commonly mentioned, if you have enough cannon firing at a single building, the concentrated fire can destroy it before the troops can get out.

On the other point, I frequently see potent stacks going around raiding for years when they could easily take the region they're raiding instead. The stupid, it does burn indeed.