I've actually had a unit that I ordered to withdraw (a 1-man unit of Highland Archers - I reckoned almost anything would be better) leave the map by going diagonally across to another edge, instead of turning around and leaving from the edge at which he'd arrived. Crazy.