I recently attacked a fort and a royal knight unit was victimized from arrows and arrow towers. He went from 20 men to 7 and the unit never entered combat so did not kill anyone.

His valor went from 1 to 3!!

I thought you gained valor from killing as many as possible. I always loved fighting peasants due to the possiblity of gaining a lot of valor.