The reason it works that way is that valor is measured by individual men, not by units. Your 3-valor Swabian swordsmen aren't losing valor, they're just being joined by some 0-valor recruits so the average goes down, and the display only shows you the average for the group. There's still 30 high-valor soldiers in there (and they're not all 3-valor either, that's just their average) who will do just as well in coming battles, but the new guys will still get chewed up. If you want to keep the valor high, just merge multiple veteran units to get up to full strength. If they both have higher valor to begin with the average won't change as much. But keep in mind you're never losing valor by combining units, even though it appears that way.