archery tests (are Samurai archers worth it? etc)

[econ21]

Hypothesis: I wanted to see if Samurai archers were worth it compared to Ashigaru ones. The null hypothesis was that there was no difference in their missile kill rates. It has been suggested that Samurai archer fire is "armor piercing", unlike Ashigaru fire, so I wanted to test for differences against targets of different armor.

Method: I tested one unit of archers under my control against one melee unit under AI control. I used the Crossroads map, which is fairly flat - deploying my archers back near some trees and allowing the AI to fire at will. The AI melee infantry started running when it got into range. I paused as the melee infantry hit the archer line to record how many had survived and how many had died to archery. I repeated each trial four times, comparing bow ashigaru and bow samurai against, variously, yari ashigaru, katana samurai and naginata.

Results: Ranged kills in each trial (e.g. 34 Yari Ashigaru died to missile fire in the first time I had them attack Ashigaru archers).


Discussion: There was considerable variation in the results - sometimes this was due to the contest involving one more or less whole volley. This may have been partly due to minor differences in the set up of the tests (the location and alignment of the archers; possibly even the weather). The number of trials is too small for statistical testing but the results are interesting.
The Samurai archers had 22.6% higher missile kill rates than Ashigaru archers against Yari Ashigaru, but 39.3% higher against Katana Samurai and 58.9% higher against Naginata.
Conversely, the Katana Samurai (armour=5) suffered 34.7% fewer casualties than Yari Ashigaru (armor=2) against Ashigaru archers and 25.8% fewer against Samurai. The Naginta (armour=9) suffered 54.1% fewer casualties than Katana Samurai against Ashigaru archers and 47.6% fewer against Samurai.

Inferences:
Despite the smaller unit size (90 vs 120), the Samurai archers kill more than Ashigaru archers. The difference is modest when fighting virtually unarmoured Yari Ashigaru, but substantial against armoured Samurai and even more, the heavily armored Naginata. Subjectively, I would conclude that Samurai archers are "worth it" in your main armies by the mid to late game when most of your enemies are samurai. Added to this, Samurai are much more melee capable (they always beat the Yari ashigaru whereas the Bow Ashigaru always lost).

The comparison of targets by armor suggests that armor upgrades (e.g. from smith provinces) are valuable. The +3 from an armorsmith could cut the ranged casualties of your Yari ashigaru by a third.

Finally, naginata rock - taking about half the casualties of katana samurai to missile fire.

Possible further research
I tried using fire arrows (Ashigaru vs Ashigaru) but observed no discernible difference in kill rates. It may be their main (only?) effect is to lower morale (and burn structures). Usage appeared erratic though - sometimes only one full volley of fire arrows went off, another time, three volleys. It may be that it is better to press fire arrows when the enemy target is in mid-range to maximise time on target (I pressed when they were just out of range).

It would be interesting to see the effect of experience upgrades.

I wondered if manually targeting arrows might give better results than auto-fire. Typically, at maximum range, only one or two archers appeared to fire. The first big volley comes quite a while after the target has passed the red maximum range line. I wonder if this is an unavoidable delay or if autofire has additional lag.