Often happens in the Viking campaign (large battles really though), you'll have a battle and the first 16 units will skirmish. Then withdraw, as will all the reinforcements.