Investigation of AI reassessment upon reload

[Pritzl]

hrvojej, your control/test concept is perfect imo. With enough numbers, we can do a simple statistical analysis (t-test?) to confirm that the AI's reassessment routinely results in much fewer provinces changing hands.

Might I suggest an improvement to the test though? Instead of manually noting the province allegiances, maybe a screenshot of the campaign-map mini-window on the bottom left at the beginning of the test and end of the 15 turns would improve accuracy. Furthermore, perhaps proper random selection of who tests with saves and who doesn't is worthwhile to remove any doubts of bias?

Finally, is there thread for participants to sign-up and post their results somewhere on the forum?

[edit]I should read better. 5 and 5 per tester and PMs it is.[/edit]

[A.Saturnus]

- PM me the results, but keep in mind that I can only have 5 PMs at a time, so if I don't write back saying I got it in a day, PM me again; I'll do the analysis of the results, and post them here; I will not soup up the results, or do anything else to them other than analyze them - I do this kind of stuff for a living, and I know how to keep my personal feelings out of it, believe me

You can PM your results to me. My mailbox is bigger.

It would be ideal if FoW is toggled off all the time (including the controls), so every single reassessment can be spotted. However, please add to your results a description whether or not your trials have been exactly as hrvojej has layed out.

[Simetrical]

Probably a lot of you have seen this, but it certainly constitutes evidence that there's a problem: Protectorate: Total Peace.

-Simetrical

[player1]

The problem I see with Brits is that very soon will Gauls or Germans start seiging your mainland province.

[Pode]

Go out and make an alliance with them both using your lone diplomat, that should hold them off long enough to complete the test, and I don't *think* CA will argue that the approach of a diplomat with no money is enough of a threat to make the AI break every siege on the map.

[player1]

Or you could simple give province as a gift to one of those two factions...

[Arphahat]

Probably a lot of you have seen this, but it certainly constitutes evidence that there's a problem: Protectorate: Total Peace.

-Simetrical

I see that Simetrical has already posted the link, but I wanted to elaborate on the exact procedure I followed to gain the protectorates documented in the screenshots.

I started a medium/medium game as the Julii. I split my armies into individual units and built nothing but diplomats, ships and calvary. Each diplomat would travel with one of the army units. When the diplomat encountered a faction that we were not at war with, I would use the army unit to besiege the town and immediately lift the siege (or attack the faction's army) just to get the protectorate option on the negotiation screen. The diplomat would then ask the faction to be a protectorate, offering nothing in exchange. The faction would refuse. I would then use control-s to save and then control-l to load. I would have the same diplomat talk to the same unit or town and make the same offer of a protectorate for nothing. In every case, the other faction accepted.

Occasionally, I would already be on a turn that I had just loaded. When I encountered another faction, I could simply besiege, lift siege and then offer protectorate status and they would accept.

[hrvojej]

It would be ideal if FoW is toggled off all the time (including the controls), so every single reassessment can be spotted. However, please add to your results a description whether or not your trials have been exactly as hrvojej has layed out.

True, but then you would have to scan the map all the time after every turn, instead of just doing it once at the end. Also, you would have to have FoW on all the time, and you wouldn't be able to escape arguments that the game was not intended to run without FoW and that the AI is lifting sieges because it's shy or something... In sum and a priori, I don't think that the information you would get from countinuous sampling outweighs the number of trials you can do by scaning at the end. I know what you're saying, but I predict the effect size to be so large that it won't really matter.

Do people have a lot of problems as the Britons from being besieged? I don't have time right now to do a lot of trials, it ended up being a busy end of the week, but I ran the control twice and was not besieged in 15 turns. Maybe I was just lucky? In any case, I thought that the suggestion to take Britons was a good one, since you're away from the action, and in addition to not being involved yourself you won't stand in a way of expansion of others. Another faction I can think of that also might be good is Parthia, but I have to admit that I have not played the regular campaign with either of the two, so I don't know how likely you are to get attacked.

And to answer therother's point about modding the game: I think that moving a single faction's name from nonplayable to playable factions list is really nothing we should concern ourselves about. I do however think that it is important we pick a faction that is unlockable through regular means, as it would seem that those which are not have some unfinished quirks about them. The only thing that remains to be seen is if with Britons you cannot avoid being attacked all the time, then we should consider other factions like Parthia. I'll test it as soon as I have time (tomorrow night it seems).

Cheers, and thanks to all who intend to participate

[A.Saturnus]

True, but then you would have to scan the map all the time after every turn, instead of just doing it once at the end. Also, you would have to have FoW on all the time, and you wouldn't be able to escape arguments that the game was not intended to run without FoW and that the AI is lifting sieges because it's shy or something... In sum and a priori, I don't think that the information you would get from countinuous sampling outweighs the number of trials you can do by scaning at the end. I know what you're saying, but I predict the effect size to be so large that it won't really matter.

If not, the hypothesis becomes questionable anyway. I suppose we do both. For now, we should do it as you suggested to test our hypothesis. If it is proven beyond the shadow of a doubt that the bug exists, we can explore it to more detail by investigating individual AI actions.

[therother]

And to answer therother's point about modding the game: I think that moving a single faction's name from nonplayable to playable factions list is really nothing we should concern ourselves about.

Okay, just checking to see if you thought it was okay. I think the Brutii are perhaps the best choice, given their sheltered location, the fact that they are unlocked to begin with, and that they are a Roman faction (it is Rome: Total War after all). Of course, that last bit works against choosing them to some extent as you also have the Senate AI hardcode possibly interfering with things. Perhaps we can do both, and see if there are any significant differences?

[hrvojej]

Hi all,
I apologize for not being around in the past few days, life got in the way of testing.

Okay, just checking to see if you thought it was okay. I think the Brutii are perhaps the best choice, given their sheltered location, the fact that they are unlocked to begin with, and that they are a Roman faction (it is Rome: Total War after all). Of course, that last bit works against choosing them to some extent as you also have the Senate AI hardcode possibly interfering with things. Perhaps we can do both, and see if there are any significant differences?

I don't think the Brutii would be great because of the senate, as you have mentioned, and because they are supposed to be an active faction due to the senate as well. Which therefore means that they are more likely to conquer something when controlled by AI and produce results.

As to the player's activity influencing AI moves, that's a separate variable that the test I originally proposed does not test for. However, it's kept constant in it, and hence does not influence its results. The only thing that is not constant, provided that everybody does the test in the exactly same way, is saving/reloading. This very basic test will tell us whether this single variable is doing something to the game, provided everything else is kept constant. Nothing more, nothing less. I won't go in whether the AI responds to player's moves or not right now, as this test is not designed to test for that in the first place. If really nothing happens when the player just stands still, then we'll go from there and redesign the test. But I can say right now that even though I'm highly aggressive in my campaigns, with sufficient reloads everybody else is static no matter what I do. Also, the preliminary result by Bromley are promising (thanks for posting them btw), and hence I think the save/reload test is worth doing in any case.

As far as my testing goes, I'll do my runs by tomorrow. Feel free to PM me your results if you already did your tests.

Cheers,

[roguebolo]

MORE TESTING

I decided to use the same, identical test that was proposed on this thread except starting from my save game at turn 19 rather than at turn 1. The Brutii laid siege to Segestica on the second turn and immediately released it after a save/load and "End Year". However, the following turn it laid siege again and I discovered that the city had suffered from attrition just as if the siege had never been relieved -- it could hold out for 3 turns now instead of 4. I followed this with three save/loads immediately followed by End Year during which time the Brutii maintained the siege. In addition, other factions were still declaring war on each other. I guess this kinda blows away the idea the save/load always forces the AI to relieve sieges, or paralyzes the AI into some kind of inactivity. I think it just needs for some kind of opening moves to be initiated before it can follow up on them a little more decisively. Again, I have the save game if anyone wants to see it. I've done it more than once and the Brutii always besiege Segestica.

OK, I'm coming to the conclusion that CA is right -- as I would expect them to be, since their information comes straight from the developers. "Load game" does indeed seem to cause random events and AI behavior to change, but not in any way that is game-breaking. In the Thapsus/Carthage-Lemonum/Condate Redonum example I'm still forced to ostracize them from my territories -- whether I sally from the siege or attack them in the open terrain. Otherwise I must succumb to repeated attrition and devastation. The change in the behavior of the AI caused by a save/load procedure allows me to select which of those two options I prefer, sallying or open terrain, but it does not otherwise dramatically change the course of the game -- the main problem, as I see it, is that it denies me the privilege of being the defender in a castle assault, which happens to be a lot of fun.

One can hardly come to the conclusion that the AI's behaviour is indicative of losing track of long-term goals in the light of these results. It took a total of five turns for the Brutii to arrive at Segestica by boat, initiate a siege, relieve the siege, then reinstate it, hold it for three turns, and finally occupy the territory.

Furthermore, in the test involving Thapsus/Carthage and Lemonum/Condate Redonum, although the existing sieges were temporarily released, those units that were approaching a settlement to initiate a siege continued toward their objectives. In other words, they did not lose track of those objectives. Apparently in these cases the AI deemed that instigating conflict through a siege or through desecrating the territory were equally acceptable and the load game caused them to choose the opposite one that was in effect at the time the game was loaded. This is something which can be exploited if I feel I'd like to endure the siege for a turn with the likelihood of being able to meet them in the field on the next turn by doing a save/load, but it's definitely not a game-breaker.

[Bromley]

Just a basic test along the lines hrvojej mentions. I freely admit that I haven't counted the provinces that changed, but compare the world maps. The start one looks the same as the save/load one, whereas there's been a lot of movement on the no-save one.

Also, for a smaller close-up view, see Sicily. No loads meant that the Scipii went crazy and kicked everyone off. Save/load each turn and all three factions trade love notes.

1.2 Vanilla. Julii. Both tests over 10 years (20 turns).
Move armies into town. No other actions. Respond "no" to everything.

Starting Position


Save/load every turn


No loads


There's been some talk about a similar issue in GalCiv over in the thread in .strategic. Starting post is the Gerry Quinn response to my OP, currently no. 30.

[Bromley]

Just reproducing Arphahat's and Camp Freddie 1969 protectorate posts from the .com forum, just in case it disappears. I know that Sim already linked to the screenies, but some of this text might be useful.

Please note that I have not personally confirmed any of Arphahat's work, although beezer says that he has.

Arphahat.
There is a simple technique for winning the game. Simply siege a city of a rival faction, immediately lift the siege, save the game, reload the game and then have a diplomat ask the rival to become a protectorate. This works every time, without exception! Look at my glorious conquest in picture form below.
www.stampor.com/Rome/exploit.html
EDIT: here is a link to a list of the images, instead of loading all at once, if it helps www.stampor.com/Rome/exploit_links.html

Question.
One question I have is what happens when 2 of your protectorates go to war with each other. Are you forced to choose 1 and thus you loose the other???
You lose one of the protectorates. But, no problem; just make sure you keep a diplomat next to each of the factions. If you ever lose a protectorate, simply save, reload and ask again.

Camp Freddie 1969.
I found that if I declared war then immediately asked for a protectorate (before hitting end turn), I would always get one due to the much lauded re-evaluation feature.
However, I also tried it on the Thracians, with whom I've been fighting a long war. I outnumber them by a huge margin and regularly spank the armies they send my way with losses that are >10:1 in my favour. I've always been suprised that they don't accept being a protectorate in normal gameplay.
So I asked them after a reload - and they refused! Could it be that the auto-protectorate 'feature' is broken?! Then I looked at the refusal message:
"We cannot accept such generosity, since we have nothing to offer in return".
That's right folks. They refused because the AI re-evaluation decided that my demand of making them a slave nation was 'too generous'. They found themselves not worthy of my benevolent (er, yes, benevolent) rule!
After moving on a turn, I asked again and got the normal response of, "We are proud of our freedom and do not give it up lightly...."
So the re-evaluation feature seems to be totally underestimating the strength of the AI nations.

[MajorFreak]

could we workaround this issue by changing all the faction's AI personalities to aggressive ones?

[rcp]

I can also confirm Araphat's protectorate issue. I also agree that using the Brutii would likely be the best choice for this test if we are to ideally do nothing but press end turn during our testing.

[Epistolary Richard]

could we workaround this issue by changing all the faction's AI personalities to aggressive ones?

The evidence collected so far indicates that this issue applies to all factions, not just those with a less aggressive personality.

"The AI will almost always, upon a reload, abandon a siege and run as far away as possible. Unless you interrupt it again with another load, however, it almost always returns and lays siege again the next turn."

Has any of you had the same experience? It could be a workaround for the bug to minimize the negative effects, allthough it will not eliminate it.

Yes, I noted the same experience in my post above. But even if the AI immediately returns to the siege it would be at least 3 turns before it could assault. This issue particularly effects users who can only play for one or two turns at a time. Users who play for longer stretches are less affected by this issue in any case.

On the FoW issue, it would be interesting if CA tries to use Heisenberg's Uncertainty Principle as an excuse.