I'm guessing you sacked the settlement. My understanding is that buildings aren't normally destroyed if one simply occupies a settlement.