It might just be me, but when you're besieging an enemy castle and it is about to fall and you have to keep a large force there for several reasons take out your horses, you'll lose more buildings that way. It also seems to happen when you decide to enter the castle. Just keep horses away.

Can't wait for RTW as destroying buildings won't happen randomly -what ever you destroy when entering the castle will be removed from the province/area and those that u don't won't.