Again, it's been a very long time since I've played, but isn't it just the castle that gets downgraded?
This is another iffy one, when castle garrison's are being sieged out there seems to be a chance that the castle will be downgraded. Which of course gives some incentive to attack the garrison. However the downgrade doesn't seem to be guaranteed. Sometimes when I'm sieging out a tier 1 castle it'll disappear once the garrison falls (therefore is downgraded) and yet other times when I'm sieging out a garrison the tier 1 remains standing. I'm not 100% on this because maybe these were simply tier 2 castles that I didn't notice. I'm recording my campaigns so this would be easy to clarify...

The provinces buildings however seem to be a random army destruction event that the player doesn't have control over. On the turn that a province is conquered even before anything is done about the castle garrison some buildings will be either destroyed or damaged and some will be left standing. Again, this all appears random except in the cases of rebel provinces.

I realize I'm asking a lot from the memory of folks who probably haven't played the game in a while. I'm mostly asking to see if there was a general consensus unknown to me. Also I realize these are minor points in a game where generalized strategy will suffice. My curiosity is mostly driven from the fact that I have started a youtube channel which features this game and I'm simply trying to provide accurate information in my playthroughs. If we can't come to some hard answers it's all good! I'm just happy to be talking about Shogun