I've seen this happen with or without troops supporting the skirmishers. It just seems like the skirmishers are much slower to react when charged by cavalry.

I think this is deliberate. Movement speeds being what they are, a unit of skirmishers could otherwise occupy a unit of heavy cavalry for far too long on a flank.

That said, there are some real flaws in skirmish mode. Yesterday, I had a unit of slingers get caught by charging gaul swordsmen in the middle of an open plain. Also, skirmishers shouldn't try to evade when separated from the enemy by an impassable obstacle, like a wall or river, but they do.