Gendarmes vs. Templars: a statistical analysis

[Rebellious Waffle]

INTRODUCTION

In an earlier thread I was reading, there was a discussion of whether or not to use professional cavalry or stick with feudal cavalry during the Late period. Arguments against professional cavalry (gendarmes) were made on the grounds that gendarmes' stats are nearly identical to that of men-at-arms except for their higher upkeep cost. This is in fact the case; however, wondering if CA would really make a redundant unit that provides no incentive for use -- it takes a time and effort to make nice-looking meshes, which the graphics folks wouldn't want to see wasted -- I decided to run an experiment to see if gendarmes are actually inferior to feudal cavalry. Reasoning that any cavalry inferior to men-at-arms will be inferior to the Knights Templar, I set gendarmes and Templars against each other to see what happened.


THE EXPERIMENT

MAP: Grassy Plain
DIFFICULTY: Medium
FLORINS: 10,000
PERIOD: All
FRANCE: 1 unit of gendarmes
PAPAL STATES: 1 unit of Knights Templar
NUMBER OF TRIALS: 20
PURPOSE: To determine whether the gendarmes of professional armies are comparable in quality to the Knights Templar, representing the pinnacle of pre-professional heavy cavalry.
METHODOLOGY: Blocked experiment. I played trials 1-10 as France and trials 11-20 as the Papal States, to control for human advantage. In each battle, I single-clicked on the opposing unit and hit the fast forward button, not touching the keyboard again until the battle was over. After each battle I recorded how many soldiers each side lost, how many they killed, and whether or not the gendarmes won. Neither the gendarmes nor the Knights Templar were upgraded with weapons, armor, or experience. Statistics were computed by entering the results of the battles as arguments for MATLAB functions. The reason why I blocked the experiment comes from an earlier experiment where I did the same thing with both armies consisting of identical units of Gendarmes, and I beat the computer 16 of 20 trials despite the fact that all I did in each battle was click once and wait. The likelihood of that happening if my controlling the army made no difference is about 1 in 216.

~~~~~~~~~

The following are basic statistical assessments of the outcome of the experiment.

GENDARMES:

Victories: 60%

Casualties
Mean: 29.850
Standard Deviation: 9.0134
Minimum: 14
First Quartile: 23
Median: 29
Third Quartile: 39
Maximum: 41
Interquartile Range: 27


Kills
Mean: 35.80
Standard Deviation: 6.4531
Minimum: 18
First Quartile: 36
Median: 40
Third Quartile: 40
Maximum: 41
Interquartile Range: 23



KNIGHTS TEMPLAR:

Victories: 40%

Casualties
Mean: 33.950
Standard Deviation: 7.7083
Minimum: 15
First Quartile: 31
Median: 37
Third Quartile: 40
Maximum: 41
Interquartile Range: 26

Kills
Mean: 32.350
Standard Deviation: 7.78849
Minimum: 16
First Quartile: 27
Median: 34
Third Quartile: 39
Maximum: 41
Interquartile Range: 25


~~~~~~~~~~~

The following are data from our control group, where French gendarmes fight Spanish gendarmes with the same experimental procedure as when the gendarmes were fighting Knights Templar.

FRANCE

Victories: 60%

Casualties
Mean: 25.90
Standard Deviation: 10.305338
Minimum: 10
First Quartile: 17
Median: 29
Third Quartile: 33
Maximum: 41
Interquartile Range: 31


Kills
Mean: 36.850
Standard Deviation: 4.19618
Minimum: 28
First Quartile: 37
Median: 38
Third Quartile: 40
Maximum: 41
Interquartile Range: 13


SPAIN

Victories: 40%

Casualties
Mean: 36.05
Standard Deviation: 5.45291
Minimum: 24
First Quartile: 34
Median: 39
Third Quartile: 39
Maximum: 41
Interquartile Range: 17


Kills
Mean: 29.0
Standard Deviation: 10.09429
Minimum: 12
First Quartile: 20
Median: 34
Third Quartile: 39
Maximum: 41
Interquartile Range: 29


~~~~~~~~~~~

To test our hypothesis that the gendarmes do not perform differently than Knights Templar, let us average the casualty and kill rates of the gendarmes, assume that the performance against Templars was the same as against gendarmes, and find the z-scores of the observed casualty and kill rates.

(mean control kills) = 32.925
(std. dv. control kills) = 7.1452
(mean control casualties) = 30.975
(std. dv. control casualties) = 7.8791

(z score) = (mean - observed)/(std. dv.)

(z score observed kills) = -0.40237
(z score observed casualties) = .14278


Z scores of absolute value less than 0.5 are nowhere near strong enough for us to reject the null hypothesis. Therefore, to the best of my present ability to determine, there is no reason to claim that gendarmes are more or less effective in battle than Knights Templar. Given that they're cheaper to recruit, don't require Guild buildings, and the high upkeep cost is counterbalanced by high income from cities, gendarmes seem like a very good deal all around.


CONCLUSIONS

Why gendarmes come out about equal with Knights Templar is a matter best left to more experienced players than myself -- my knowledge of the Total War engine is not expansive. I most certainly believe they should not be dismissed as jumped up men-at-arms; similar experiments pitting gendarmes against men-at-arms saw gendarmes win 90% of the time with far fewer casualties and far more losses inflicted. For some reason or another, the unit card simply doesn't tell the whole story.

[Kobal2fr]

Puzzling fact : France vs Spain control group should have come out as 50/50 instead of 60/40. What does that tell us ?

[Rebellious Waffle]

It's awfully close to 50/50; positing that the procedure will even out to 50/50 as the sample size increases, the chances of getting a 60/40 split in 20 trials is

(20 choose 12)*(0.5 to the 20th power) = 0.12013

that is, it should happen about 12% of the time. Unusual, but not outrageously so. We should be more suspicious if the 60/40 split continued as the number of trials increased, but with 20 trials we can afford some leeway.

[John_Longarrow]

Kobal,

My guess is that, since France is supposed to have better cavalry then CA has hard coded a bonus to any cav unit that is French.

A way to test that would be to do the same kind of experiment with English Mailed Knights VS French Mailed Knights. I'm guessing we'd see the same 60/40 split.

[Rebellious Waffle]

That, or the AI does something goofy. There are typically three ways the AI will respond:

1) Rushing its cavalry at you all at once,
2) Rushing a bit of the unit followed by the larger remainder,
3) Reversing the soldiers' position instead of charging (right to left)

When (1) or (2) happens, success is up for grabs. When (3) happens the AI almost always loses because it doesn't get a charge bonus. What we might be seeing in the 60/40 split is the AI's tendency to flip its horsemen instead of charging -- I won't know that for sure until I revise my experiment to control for that variable, but that's what strikes me off the top of my head.

[FactionHeir]

You really ought to write my thesis with all that stat knowledge :grin:

But the findings are interesting. I don't think the 60/40 is significant, considering it numerically means 12-8, but Gendarmes are 10(11)/15 and templars 13(14)/16, meaning the templars should theoretically win.
Gendarmes do have higher armor rating though, so if they get hit at the backside, they tend to live longer.

A major skew in your findings will be the presence of a captain though.

[Doug-Thompson]

Impressive.

FactionHeir's point is not a quibble, though. If one side's captain dies first, that will skew results. I'd argue that the effect of that should have been countered by the sheer number of the tests. Still, it would be good to know whose captain died first in the contests.

[Kobal2fr]

Kobal,

My guess is that, since France is supposed to have better cavalry then CA has hard coded a bonus to any cav unit that is French.

A way to test that would be to do the same kind of experiment with English Mailed Knights VS French Mailed Knights. I'm guessing we'd see the same 60/40 split.

Kinda doubt it. France has better cavalry because they get Lancers, Noble Knights that are probably a bit better than the generic Chivalric ones (though I don't know how - the only difference is their higher upkeep and mount types, but mailed horses and barded horses have the same stats and mass so...:no:) and Gendarmes, plus the Tourney line of buildings.

Oh and @OP yes, I had forgotten about it, but Factionheir is right : the "1 unit vs. 1 unit on grassy plain" test setting is kind of screwy because the occasional early general death can mess with the results. Admittedly, it should be statistically insignificant since it should happen to both sides equally but with 20 tests only it's still noticeable.

[FactionHeir]

Not only the death, but also the positioning of the captain and how many he gets to kill. Captains and generals tend to do one hit kills and are very tough to kill.

The good way to do that test would be to give each side a very slow unit as the general so only the knights get to duke it out properly. Something like halberd militia or so. What you can do is mod your unit file and give each side say a ballista as the general's unit and in the projectile file mod ballistae to have 0 accuracy, so they will always miss.

[Rebellious Waffle]

So, new refinements to the experiment include:

1) Recording whose general died first;
2) Increased sample size;
3) What the AI did -- part charge first, all charge together, flip.


The last might be hard to do since we'd effectively be drawing with replacement, and we'd need maybe 10 of each reaction to gauge how much they matter to the outcome of the whole engagement. We might be able to get around that by bringing up the sample size to, say, 50 or 60; since the AI likely chooses which course it takes at random, we can follow the dictum to "control what you can, randomize the rest" after that fashion.

[FactionHeir]

You'd get more reliable results doing what I said because then it also takes out the captain attack factor.

[Rebellious Waffle]

Aye, I hit the button to make that post before the thread was updated with what you said, so I waited to say anything about it to avoid double-posting.

The zero-accuracy artillery piece sounds like a good idea, but I don't trust myself enough around the config files not to do something terrible and muck up the program beyond fixing. A wise man once said, "no product is foolproof; fools are ingenious" -- a maxim whose truth I prefer not to test until I'm certifiably unfoolish and know what I'm about. Barring that, it may be just as well not to introduce other units onto the field because the game shifts from pure tactics to part strategy, and unexpected confounding variables could well creep in as a result.

[Rhedd]

Where is the file that determines accuracy for siege engines and missile troops, anyway?

[Miracle]

Statistics were computed by entering the results of the battles as arguments for MATLAB functions.
May I see your function m-files?

Anyway one I thing I'd like for you to observe in your future tests is the kill rate at the middle of a battle line vs. the sides.

Cavalry have trouble attacking units directly in front of them, but when they move ahead diagonally and expose their sides to the enemy, they can use a very fast attack animation. Against other cavalry this is lethal whether the attacker has 11 attack or 14.

My hunch is that individual cavalrymen who gain a tactical advantage by "crossing the T" and can therefore "fire their broadsides" against another cavalrymen have a disproportionate effect on a battle. A cavalry vs. cavalry battle then isn't decided totally by stats but mostly by the planar vector of each individual.

Interestingly when a cavalry unit is charging against another cavalry unit, the reverse position (front against side) is more favorable to the attacker.

[Rebellious Waffle]

I didn't so much write a .m file as open the GNU-Octave shell and belt out a few commands -- the statistics package is fairly sophisticated. (Octave is an open-source version of MATLAB, by the by. Great program.)



~~~~~~~

% lostGen is the number of Gendarmes lost in the Gendarmes vs. Templars match, killedGen is the number of Templars the Gendarmes killed, etc.
% Arrays with "Spain" and "France" in the name come from the Gendarmes vs. Gendarmes control group.
% genWon lists in order the outcome of the Gendarmes vs. Templars battles as Booleans, with 1 meaning Gendarmes won and 0 meaning Templars won.
% franceWon follows the same naming convention as genWon, save that it refers to the France vs. Spain control group.
% Entries 1-10 of each array are from when I played as France, 11-20 are from when I played as the Papal States/Spain.


lostGen = [26 40 26 27 40 27 23 14 14 21 29 31 39 37 23 20 39 41 40 40]
killedGen = [40 37 40 36 31 40 40 36 36 38 40 41 22 33 40 40 18 40 28 40]

lostTemp = [40 31 40 36 26 40 40 36 37 38 40 41 19 28 40 40 15 34 24 34]
killedTemp = [30 40 31 32 40 32 27 17 16 25 34 37 39 37 27 23 39 41 40 40]

lostFrance = [41 14 33 29 17 18 20 31 29 34 10 22 13 40 40 31 14 41 17 24]
killedFrance = [33 37 39 39 38 29 38 41 40 41 37 39 38 28 28 40 38 40 34 40]

lostSpain = [28 38 39 39 38 29 39 41 40 41 37 39 38 24 24 40 39 34 34 40]
killedSpain = [40 17 39 34 20 21 23 36 34 40 12 26 15 40 40 37 17 41 20 28]

genWon = [1 0 1 1 0 1 1 1 1 1 1 1 0 0 1 1 0 0 0 0]
franceWon = [0 1 1 1 1 1 1 1 1 1 0 0 0 1 1 0 0 1 0 0]

mean(genWon) % What proportion of battles did the Gendarmes win?
mean(franceWon) % What proportion of the battles did France win?

statistics(lostGen)
statistics(killedGen)
statistics(lostTemp)
statistics(killedTemp)
statistics(lostFrance)
statistics(killedFrance)
statistics(lostSpain)
statistics(killedSpain)

~~~~~~~

The statistics() command gives the minimum, first quartile, median, third quartile, maximum, mean, standard deviation, kurtosis and skewness of the entries in an array. I entered each statistics() command one at a time, copied the results with lables into a .txt file, and used that as the skeletal version of my first post in the thread.

[Philbert]

To get a good picture, you should also reverse the experiment, i.e. control the Templars as the player and have them attack the Gendarmes. I would not be surprised if that would reverse the outcome.

[econ21]

To get a good picture, you should also reverse the experiment, i.e. control the Templars as the player and have them attack the Gendarmes. I would not be surprised if that would reverse the outcome.

I think he did:


I played trials 1-10 as France and trials 11-20 as the Papal States

[Philbert]

Ah, I misunderstood that bit, I thought he meant to check if different factions had built in advantages.

[Rebellious Waffle]

Aye, that's what I meant.

I ran another experiment with 30 trials, and I've come to the conclusion that I need to ramp up the difficulty. Out of 70 battles with this same methodology, I've won 56 of them; the chance of that happening if it doesn't matter which side I play is 1 in 16,369,000 -- patently ridiculous. The symmetry we find between one unit and another might be an artifact of my playing each side an equal number of times on Medium difficulty.

[Moral55]

Some times I find that if the A'I does not move it's knight unit, the knights will keep there lances in the air for a bit even after getting attacked before using there primary weapons.
Perhaps this has something to do with why you get a slightly better win/loss rate.

[alpaca]

Aye, that's what I meant.

I ran another experiment with 30 trials, and I've come to the conclusion that I need to ramp up the difficulty. Out of 70 battles with this same methodology, I've won 56 of them; the chance of that happening if it doesn't matter which side I play is 1 in 16,369,000 -- patently ridiculous. The symmetry we find between one unit and another might be an artifact of my playing each side an equal number of times on Medium difficulty.
According to some dev statements, the AI should play on medium.
Anyways, you do that and we'll see if you did a good job :laugh4:

By the way, the best way to test would be to do multiplayer battles and walking the units into each other instead of charging, then test charging separately. If you have multiple pcs at home you can even do that alone. The reason why you should exclude charging is that because of the bad unit cohesion and other factors, charges are very non-linear. If you do enough tests that should even out since both units have the same charge value, but this could be the reason why you win most battles, because the ai sometimes charges weirdly in 1on1 battles.

[FactionHeir]

Difficulaty levels mainly influence morale.

I.e. if you do a custom battle with 2 peasants, under medium, before they even reach you their morale is down to steady.
At hard, it goes down to steady when they engage and under VH they'll fight as tough as you do, i.e. equal morale.

[alpaca]

Difficulaty levels mainly influence morale.

I.e. if you do a custom battle with 2 peasants, under medium, before they even reach you their morale is down to steady.
At hard, it goes down to steady when they engage and under VH they'll fight as tough as you do, i.e. equal morale.
Are you sure that the AI has a worse morale than you, that is did you test it properly?

[FactionHeir]

Try it in a custom battle.
1 peasant each, only modify difficulty level and watch as they approach each other walking. The AI's units will have much worse morale than yours.

[Rebellious Waffle]

I'll be ready to test out some new stuff tomorrow -- my brother isn't working today, and he's got the only Windows computer in the house, so I won't be able mooch his machine for Total War purposes until about five o'clock tomorrow morning. (I'm one of maybe half a dozen of the fabled "morning people" on the planet -- I've lobbied the EPA for protection under the Endangered Species Act so's I can get out of parking tickets, but for some reason the operator always hangs up when I ask.)

In the meantime, here's a few stem-and-leaf plots for Gendarmes and Knights Templar in the 30-trial experiment:

(Huge unit size)

GENDARMES:

Kills:
0|
1|
2|8
3|07
4|5
5|11133468
6|0056699
7|23456899
8|0011

TEMPLARS:

Kills:
0|
1|0
2|1459
3|23399
4|15
5|2456
6|78
7|44556889
8|0011


I find it intriguing that the Gendarmes' kill rates are skewed so much to the left -- it can't be because the Gendarmes won more often, since in the 30-trial experiment the Gendarmes won only 57% of the battles. It's more likely because the Templars' general died in 33% of the fights while the Gendarmes' only died in 23%. I remember that the Templars' general pretty much only died in the initial charge -- there were three occasions where the Templars and Gendarmes had lost every unit except their generals, and in those circumstances the Templar general always won -- so more battles would have been aborted early by the Templars due to spontaneous morale failure. Gendarmes always fought better in charges, and the derivative of dead Templars with respect to time increased sharply during the charges before leveling off in melee.

[Miracle]

I didn't so much write a .m file as open the GNU-Octave shell and belt out a few commands -- the statistics package is fairly sophisticated. (Octave is an open-source version of MATLAB, by the by. Great program.)
Lol, I happen to have Octave on my comp right now...

Didn't know about the statistic function though, so thanks for the tip.

there were three occasions where the Templars and Gendarmes had lost every unit except their generals, and in those circumstances the Templar general always won -- so more battles would have been aborted early by the Templars due to spontaneous morale failure. \
This is probably why you should go with FactionHeir's tip and make halberd/voulge militia as your generals. To make sure they won't enter the fray for a long time, set your units further back during deployment so the enemy cavalry have time to distance themselves from their general.

Gendarmes always fought better in charges, and the derivative of dead Templars with respect to time increased sharply during the charges before leveling off in melee.
Are you playing 1.02?

When comparable cavalry charge each other head on in Grassy Plain, nothing should really happen. Their charges tend to "cancel" each other out. It's only when their individual sides are exposed to their enemy that they start killing each other.

[Rebellious Waffle]

I'm playing whatever the stock version of the game is -- I haven't downloaded any patches, and I presume they didn't start auto-including the patched version of the game at the warehouse since the patch would be conveniently available on the internet.

[FactionHeir]

In that case your observations aren't worth a lot.

[econ21]

In that case your observations aren't worth a lot.

Maybe to expand on that for a junior member: without the 1.2 patch, the mailed knights don't get the +4 to defence for a shield. In fact, they get a -4. This is the infamous "shield bug" whereby shields reduce, not increase, defence.

[Rebellious Waffle]

Then there's not much I can do about it until tomorrow; I'll have to redo the data collection process with the 1.2 patch. That may go a ways towards explaining why my commanding one army or another mattered so much -- my knights tended to be the ones charging the computer, not the other way around.

Ah, lurking variables! Serves me right for not RTFMing about the 1.2 patch...

:faq: :spider: :faq:

[Ulstan]

Maybe to expand on that for a junior member: without the 1.2 patch, the mailed knights don't get the +4 to defence for a shield. In fact, they get a -4. This is the infamous "shield bug" whereby shields reduce, not increase, defence.


Now that it has been fixed, looking back, I find it amusing how many people insisted it was working as intended and that a shield hurt you melee because it was 'big and bulky'

[Rebellious Waffle]

That, or there's a really clever armorer running a joke shop somewhere. At no extra cost, you can install concealed hinges along the shield boss -- the whole thing collapses like a house of cards when you come into contact with the enemy. Hours of amusement! Guaranteed to break the ice at parties.*


*Some assembly required. Side-effects include dry mouth, nausea, dismemberment and aggressive acupuncture. Smythe, Smythe & Sons are not criminally liable for any ill temper, routing armies, or sharp objects protruding from the abdomen that result from using this product. Consult your doctor before using the product. Do not use the product while riding on horseback. Do not burn, kick or ingest the product. Purchase of the product implies consent to be followed around at all times by the "giggling salesman" anciliary (+3 dread, -3 morale for all cavalry units, -3 Piety, -2% trade income, -1 personal security, decreases chances of having children).

[Doug-Thompson]

Re: Side effects. Now that's funny.

[rvg]

Now that it has been fixed, looking back, I find it amusing how many people insisted it was working as intended and that a shield hurt you melee because it was 'big and bulky'

Haha, yes. The fanboyism in its extreme incarnation.

[Ars Moriendi]

Regarding the speculation on whether the French have some hidden bonus for cavalry, you might want to read this post I made on another forum. I was testing for something else entirely, but I think it could have some relevance to this issue :

Headlong charge english hospitallers vs. french chivalric (normal, medium, grassy plain).
I repeated the test 10 times, alternating between charging and receiving - by receiving I mean starting to move just when enemy charge starts, since that's exactly what the AI does when charged.

Results :


hosp chiv
wins 1 9
men 16 18.1

where "men" means average men remaining when victorious.
Chivalric knights are definitely better in head-to-head combat, isn't it ???

Well, I'm not sure - I ran another battery of 10 test, only change being I used hungarian chiv knights. They have identical stats, I was expecting identical results. Instead I got this :


hosp chiv
wins 5 5
men 21.2 11