I switch to swords with cavalry when facing weak infantry like archers.

The two weapons are also important with infantry. For example, the Sweboz bodyguards have swords and spears. Use the primary weapon (swords) against infantry and the spears against cavalry. Makes a perfect all-around unit.