Hmm yes good point. You could get around that easily enough though simply by making it that the infrastructure is only destroyed when the human player takes a settlement. Or you could even make it that the military infrastructure built up in an AI settlement simply isn't accessible to the human player who always has to build his own. But it will be available to the AI again if it retakes the settlement.

But really you raise a good point, which is that the military infrastructure system is pretty silly anyhow. After all, what do you need for training troops? Nothing much but experienced teachers and maybe some spare armour and weapons to train them with.

It would probably make more sense to dump the whole military building paradigm and replace it with something else. Unfortunately, CA seems to be in love with their building paradigm, so I guess we won't be seeing a change any time soon.