This happens during sieges even if relativly little combat happens, or none. Say you're being sieged and you want to try and slip some reinforcements in from the other side of the city (which I guess in a way is unfair, cuz the AI is too stupid too block them even if they hugely outnumber them). So you do a sortie, and maybe your towers kill some enemy or you send a few skirmishers out. You get a successful sortie/close victory by killing maybe 50-100 soldiers and poof, huge enemy stacks will disappear. Sometimes all of them, sometimes maybe just the ones who came in as reinforcements (not the one who initiated the siege). CA really needs to look at this.