Maybe they got lost in the woods or fell into the river.
But really, the garrison shouldn't just die if they never took part in the battle.