Investigation of AI reassessment upon reload

[roguebolo]

Bromley, yours is some of the statistical data that I said was referring to as very good. I guess the point is that although you've been able to create scenarios which duplicate the phenomenon, I've been able to dig up some scenarios which duplicate explicitly contrary behavior. In some cases they illustrate the very kinds of AI behavior that The Shogun mentioned in his post, but that does not necessarily redeem them. Aren't both worth some study if you wish to understand that behavior?

I actually do visually confirm the results of the first 20 moves with an "End Year" without save/loads, and there are typically four territories under dispute that are not rebel. All of the remaining acquisitions are rebel, and if you ask me that's some pretty sensible playing. It's also easier to debate the stance and prove the projection that rebel territories being occupied at a later point in the game will be less developed than if they were occupied earlier. I hope no one disagrees on that point.

It's much more difficult to debate the idea that a vague concept like "the AI is less aggressive to other factions" has any real impact on the gameplay, particularly in midgame -- then you'll need to face the entire slew of possibilities that The Shogun mentioned in his post and show that each and every one was not a reasonable assumption.

I honestly do consider the inactivity on the part of the human player to be a weakness of these tests. I'm not going to debate the reasons why but eliminating that lack of inactivity is what yielded the contrary results that I consider worth studying. In one case, even something as simple as forming three alliances in the space of two turns completely changed the behavior of the AI and made it more aggressive.

I don't really agree that a single large faction makes a better opponent; I would rather see smaller factions allying against me as there is more diversity in the gameplay because of the different unit types they can create. The only case in which I would like to see other factions grow at an equal pace is when I'm playing a Roman faction and eventually have to face the other Roman factions after civil war begins. When I first played I thought this is the way things would turn out, and I was surprised to find my Urban Cohorts facing large bands of Hastati and Velites.

This is a little off-topic, but the thing that reduces the challenge the most for me is that the AI does a poor job of building the more powerful units and a balanced army, and that it does a poor job of upgrading the arms and armor for its units. (There was actually a minor improvement in the latest patch, but not enough to be significant.) If it did a better job at these issues, it would make a considerably superior opponent.

[roguebolo]

Apparently my posts on this forum have become somewhat controversial, but I guess I'm getting used to that. Recently on a different forum regarding a different game some of my posts were originally considered controversial, eventually got me flamed by a few individuals, and ultimately won me an award as their best strategy columnist. Bromley has asked me to post a summary of my position because apparently some people have been referring to my posts as evidence that there is no loadgame bug.

In some respects, they are correct. The problem here is the word "bug". To a programmer, a bug is a clear and undetected error that produces incorrect or unintended results or an interruptive abberation in the behavior of a program, such as divide-by-zero errors or integer overflows or a deadlock due to improper use of synchronization objects. I have encountered, tracked down, and eliminated many such bugs over the last 20+ years, and this does not truly qualify as a "bug." I assure you that if you were to refer to it in those terms to the developer(s) responsible for the AI that they would take at least a small amount of offense to it. In fact, as a fellow programmer, I take a small amount of offense to it myself on their behalf. I do wish everyone would adopt a more accurate description such as the "load game reassessment behavior" or "load game reassessment anomaly".

What we are dealing with here is a peculiar behavior of the reassessment strategy under conditions of polar extrema -- beginning from the opening move of the game, without player interaction except for the refusal of all alliances which serves to reinforce the extrema. I'm not undermining this test, because it quite possibly gives us some insight into the viability of the the algorithms and heurisms used by the AI throughout the course of the game, including midgame and endgame, in the absence of such extrema.

There is a clear tendency for the reassessment that occurs after a load game to often (not always) reverse diplomatic decisions and to relieve sieges. It's also clear that this behavior tends to diminish or manifest itself in a different way as the game moves away from a position of extrema. Unfortunately, I'm one of the few people who has apparently studied the phenomenon with that possibility in mind, although Bromley is moving in the same direction. It helps to have a large database of saved games to draw from.

I think there are some weaknesses with the research that's been undertaken here. It seems that many people have reached "obvious", foregone conclusions from their observations of this fairly limited test, projecting those "absolutes" to the conclusion of the game even when the extrema is removed. There is no support for such conclusions except inasmuch as they can be supported semantically. There is virtually no empirical evidence as to the resulting strength of the AI opposition as a whole due to the AI's reassessment behavior.

There are also a few conclusions that are entirely accurate, although unsupported empirically, such as the fact that the load game behavior will allow the human player more time to acquire rebel territories and more time to develop them than the AI factions, which definitely gives the human player an advantage. I think this could easily be supported empirically, although the mathematical analysis would be quite challenging.

There has been very little attempt to understand the reasoning behind the AI's behavior, which I consider to be a key factor in suggesting how it might be changed. To simply say, "The AI needs to stop relieving sieges" could very well introduce another undesirable behavior in the AI. In order to understand the decision-making process of the AI, and how it might be improved, it is important to analyze those scenarios where the AI is behaving in a different manner than that which was observed under conditions of extrema.

In other words, I think the research here is a good start, but incomplete. It is not conclusive because it does not exhibit empirical evidence of the projected conclusions.

[Pode]

RB, you've put forth a fairly solid case and my opinion of the impact of this issue is being revised accordingly. However, my primary concern is that the anomalous behavior at an extrema is at an *unavoidable* extrema. Every campaign must have a beginning, and if the AI takes 20 turns or so to really get its wits about it (more if you play as I do and never form alliances with people you intend to subjugate one day), that's a crippling handicap. Even if the behavior straightens out and the AIs begin to function better as you suggest, for a lot of players it may very well be too little too late for the AI to pose a challenge. Your evidence is *beginning* to convince me that it is WAD, but even so I stand by my assertion that it's a very bad WAD and needs serious attention for the XP.

[Bhruic]

I'm was curious about roguebolo's claim, so I fired up a game I'd played that was in roughly mid-game state. I was playing the Greeks, and was at war with the Romans (all of them), Macedonians and Pontus.

I had a diplomat in one of my cities, with a diplomat from the Julii near, and a stack of Macedonians. So after loading, I first tried to get the Macedonians to become my protectorate. They refused. I asked for a ceasefire, they demanded 1940 denarii in return.

So no luck with them. I then tried the Julii diplomat. Request they become Protectorate and they agreed.

The only major difference between the two I could see if the advancement of relationships. I've had no real contact with the Julii, they declared war because of the Scipii and Brutii attacking me. In fact, I don't think I've even fought one of their ships.

The Macedonians, on the other hand, I've been fighting since turn 1. I've declared war on them twice.

So it's possible that the "Protectorate bug" only works if you don't have any diplomatic history with a faction?

Bh

[roguebolo]

Pode,

My idea about alliances is not 100% confirmed, but it's true that the human player can force ceasefires between AI factions with his alliances sometimes, which also means it would no longer be a "equally viable" option for the AI to change its mind about certain alliances, bringing about some degree of stabilization. An analysis of alliances and ceasefire agreements in the 20-move test with saveloads seems to support the idea that changing alliances are partially repsonsible for the loadgame behavior, as do the save games from turns 12 and 14 of my campaign in which the AI chose to agressively take territories (even distant rebel territories!) only because I had established three new alliances without any other movement, despite the fact that I was doing save/loads each turn. Furthermore, the exploit that has been reported (I've never tried it myself) which allows you to create alliances with an unwilling faction, or even to establish a protectorate (which I've tried only in midgame, to no avail) also seems to support the idea. I think it merits a little more investigation.

For example, let's take the first 20 moves, with no load games. A good player, who has a good grasp of the game and how to manipulate the other factions, will realize that he can use his starting diplomat to sail to Lilybaeum by the third turn and form trade rights and an alliance with Carthage, and then sail to Thermon in three more turns to form trade rights and an alliance with the Greeks, which causes the Scipii to relieve their siege at Syracuse. (It also establishes some fairly lucrative naval trade.)

Hence, the situation in Sicily is entirely under the control of the human player. If you consider the Scipii takeover of Sicily desirable, you can allow it; or, if you choose, you can prevent it. From that analysis alone I don't see the situation in Sicily as a strong contributing factor to the overall effectiveness of the AI later in the game. The Scipii siege of Syracuse was only as effective as I allowed it to be, so the AI was not any more effective in this case, with save/loads or without them.

However, with save/loads, it is a more difficult proposition to convince the Scipii to attack Syracuse. In this particular example, due to the unpredictability factor caused by save/loads, the AI is somewhat harder to manipulate, just as load games seem to make it easier to manipulate in other ways. Try making these moves yourself if you want -- you don't need to make any changes to the original test except for your diplomat and the boat he came in on.

But you are right about the fact that every single game will begin with a floundering period of uncertain alliances if there are a lot of load games, and that the period will last longer for different players depending on their style of play. One thing is for certain -- the AI won't be picking up a lot of rebel territories, giving the human player more time to invade unclaimed territories (assuming he can spare the resources and manpower) and to develop his own rebel acquisitions. Although I don't consider that to be a massive game imbalance introduced by the load game behavior, it certainly has some effect.

As to whether the AI begins to stabilize, I'm certain that it does as the result of some fairly extensive testing. But nevertheless its behavior is still worth observing because it gives me a clue about how the AI is reassessing the situation on a load game, a topic which continues to fascinate me.

Arphahat:

Chess programs are a great comparison in some respects. If you save between moves, they will reassess their next move each time you load and may not make the same move -- either if it's drawing the information from a database, as in the openings, or if it's in midgame and sees equally viable moves.

It's funny, because those same programmers on the other forum brought up the comparison between the AI in RTW and chess. One of them pointed out that chess is also substantially different because of the fact that it has only 64 squares, and a limited number of pieces with limited movements. So, yes, it's an unfair comparison in that respect.

A chess program would have the same type of bug if after you loaded a saved game the AI would expose itself to checkmate regardless of the difficulty setting.

That is, again, merely an unsupported assessment of the degree of impact of the load game behavior. I have encountered too many midgame scenarios which discredit the viewpoint that it simply gives up the game because of load games.

[Simetrical]

I still don't see the reason that the AI needs to reassess on reload at all. roguebolo suggested that perhaps this was to add an element of unpredictability, but the easiest way to do that would surely be to use exactly the same code with a different pseudorandom seed, and that can't be what's happening—if it were, the AI might lift a siege or not, but it wouldn't do so disproportionately if a new seed were selected. Unless the decision of which action to pick isn't pseudorandom at all, but I don't see why that would be if you're aiming for unpredictability.

Then again, what do I know? I'm not a programmer. Probably none of what I said makes any sense.

-Simetrical

Edit: Removed semantics discussion.

[HarunTaiwan]

Roguebolo,

Is there a way to test your hypothesis?

If I am playing Parthia, engaging in alliances and war, you are saying despite load/save every turn that the AI will figure things out and the Scipii will take all of Sicilly?

This seems like an easy test to run.

Please confim if this test would fit your hypothesis that human interaction is needed to jump start the aggressive behavior.

Or is it only working when the human player's faction is involved directly in the situation?

[roguebolo]

Simetrical:

Right. Exactly. I think the same thing. As I pointed out earlier, software random number generators are not truly random -- they will produce the same sequence with the same seed. (A timer tic can be used as a seed to introduce more randomness, but the sequence will still eventually repeat itself.) Apparently you're familiar with these concepts. I was comparing it to a random number generator because I think that partially explains the phenomenon -- if the list of possible choices were enumerated, it would seem to always start with the same choice after a load.

We know in advance that it does not have an enumerated list of available, equally viable options for different situations. The reassessment process must construct such a list...and whether it is doing so using recursion, or techniques of combinatorial optimization, or whatever...further complicated by a rule-based heirarchy, topological relationships, and so forth...well, as you can see the end result is a heurism. In the long run, all AI boils down to that, especially with a game as complex as RTW.

I've asked once before if anyone was aware of other triggers which might cause a reassessment. One can assume that they would result in another possible decision-change on the part of the AI midturn (to use the random number example, the next number in the sequence without changing the seed), which would explain why user interaction can change the results of the "sanitized" 20-turn test.

I agree with roguebolo that this isn't really a bug. That the AI does reassess its move on reload isn't a bug; it's intended. That it reassesses it poorly is not a bug, it's a flaw or a problem. Bad AI does not constitute a bug. However, this is a matter of semantics that doesn't really need to be discussed further on this thread.

I think it's a matter of semantics also, and I also think it should be dropped. However, I still question the veracity of the statement that it is "poor AI."

HarunTaiwan:

The establishment of alliances will have the most dominant effect on the factions nearest to you and the ones that are bordering them. However, by establishing alliances with distant factions you can still have some control over events on the far end of the map. I generally do both. Is it really going to matter to you who is the strongest faction you have to fight when you expand toward Rome and have to fight Carthage, or the Scipii, or the Brutii, or the Greek Cities? Or some combination thereof?

[HarunTaiwan]

The establishment of alliances will have the most dominant effect on the factions nearest to you and the ones that are bordering them. However, by establishing alliances with distant factions you can still have some control over events on the far end of the map. I generally do both. Is it really going to matter to you who is the strongest faction you have to fight when you expand toward Rome and have to fight Carthage, or the Scipii, or the Brutii, or the Greek Cities? Or some combination thereof?

Yes. Because if the Scipii overrun Sicily early on, they might have more than hastati and velites. If it doesn't matter, then have no AI vs. AI battles and the player can just conquer a static, unchanging world.

And, if alliances are so key, I still can't understand how Scipi overrun Sicily without save/loads and do nothing with save/loads when THEIR FACTION IS LOCKED INTO ALLIANCES WITH ALL THEIR LAND NEIGHBORS EXCEPT IN SICILY.

Not to mention the script or Senate missions that always lead Julli and Brutti to take Segesta (?) and Appolonia like clockwork. I assume Scipii get the same marching orders to take Siciliy, but when it's save/load it somehow goes away?

Scipii are already allies with the other Romans. Without loading they seize all of Sicily. With loading, they do nothing. And somehow the butterfly wings of the player's alliances would alter this result?

Maybe for Greeks and Carthage who definitely can be busy elsewhere your points could be valid, but Scipii? And keep in mind if you are a Roman faction, it will affect your end-game civil war if the Scipii have not expanded enough.

Now, in games where I did not save often, I saw Scipii do better than Brutii...could it be a way to slow down certain factions?

[roguebolo]

And, if alliances are so key, I still can't understand how Scipi overrun Sicily without save/loads and do nothing with save/loads when THEIR FACTION IS LOCKED INTO ALLIANCES WITH ALL THEIR LAND NEIGHBORS EXCEPT IN SICILY.

Umm, heck, I just explained this. I played as the Julii in my experiment, but the Brutii would probably still turn out the same way. All you have to do is make alliances with Carthage and the Greeks, and then your Roman allies will honor that alliance and the Scipii will not overrun Sicily. It depends if you want to weaken your future enemies, the other Romans, later in the game or if you would prefer to weaken Carthage or the Greeks immediately. Carthage is actually considered (historically) as the target of the Scipii and the Greeks are considered (historically) as the target of the Brutii, but both are excellent trade alliances for the Julii, especially if you build a port in Ariminum immediately. However, if you would prefer for the Roman factions to be stronger for more of a challenge later in the game, you can always allow the Scipii to take all of Sicily.

Harun, listen. If you don't feel the game is challenging enough for you, what's to stop you from making your own rules? What's the earliest year that you've hit the victory conditions? Well, OK. Do it ten years earlier.

And if that's not a challenge, then try fifteen. And so on.

Truthfully, Harun, the AI is not even close to a match for me also. Maybe, you know, the very first time you played a TW game, it seemed like a real challenge. You know, maybe, if you think about it, there were numerous possibilites you'd never encountered before in a standard RTS or action game. And maybe you've become so jaded over being a "professional" that you feel content in being critical of an AI.

The first Chess Master that ever got beat by an AI felt the same way. He thought an AI would never be able to beat him, and was so confident that he offered a $10,000 reward (which was nothing to him based on his chess championships). But he did get beat.

[roguebolo]

To a layman, it is not desirable to have the AI lift sieges and wander off when you were about to sally and break them. It is not desirable for an AI faction to make a ceasefire with you and attack you on the next click of the end of turn button.

I usually sally immediately, but the fact of the matter is that in my midgame tests, wherein the AI alternately relieved and reinstated sieges due to save/loads, that was a better strategy for the AI than if they had maintained the sieges (as they do without save/loads). First, it denied me the use of the castle to protect my flanks while approaching them and to protect my onagers, which can fire over the castle walls.

Second, it allowed them to become the defenders on a realtime map of their choice.

So to the devs, it might be "functioning as designed", but that design is fundamentally flawed. I don't think it's any kind of a reach to say poor design logic is a bug.

Again, you're assuming that everyone agrees with you that the design is fundamentally flawed. You're saying that you have a bug and I don't because of our perceptions and opinions of the game design? Because our expectations are different?

[Old Celt]

Again, you're assuming that everyone agrees with you that the design is fundamentally flawed. You're saying that you have a bug and I don't because of our perceptions and opinions of the game design? Because our expectations are different?

No not at all. What I'm saying is that if the AI would have assaulted without a save/load involved, I should be able to get that assault to happen after a load. The save didn't change anything. This special reassessment makes the AI do illogical things because there is no continuity between what it was doing and what it is doing after a load.

You can say it's conditional, and we can test for it, but it isn't logical, and there's no way it makes better sense for the AI to merely harass me with the threat of a siege when the logical, and much better strategic move is for it to capture the settlement outright. The only logical reasons for dropping a siege are to relieve their own threatened outposts, or because they are hopelessly outmatched by the garrison. This is common sense.

I make no assumptions about who agrees with me, as it is immaterial here. We are talking about simple problems in military strategy which can be solved with just a small modicum of logic. There is only one best move for the AI, in each siege situation. That best move is lost quite often by a reload. I say that is a design flaw, and believe from much experience with users and bug reports, that this would show up on my desk as a bug report.

I must also point to the totally haywired Protectorate behavior and say the same thing: it isn't logical for the AI to behave this way; it is a design flaw, and many users would simply call it a bug. I could say it wasn't a bug all day long, but I'd be in there coding a fix for it, just as sure as death and taxes.

[roguebolo]

As far as the Protectorate issue goes, I've had no luck whatsover duplicating it, although I've only tried it in midgame. I just talked with a friend on the phone last night and he had no luck either.

In the example I gave before, the AI actually was outmatched by the garrison. Also, I've mentioned before that one of my complaints with the load game behavior is that it does not allow me to be the defender in a castle assault.

In a castle assault, the defender has the advantage. So it is to the AI's advantage to taunt me until I attack it. That's not a bad strategy. It's actually a far worse strategy for it to maintain a siege with an inferior force, as it attempts to do without the load game reassessment.

In retrospect, in all of my testing, the only time I've ever seen the AI maintain a siege or perform a castle assault despite successive load games is when it seriously outmatches the garrison. However, I was not the besieged party in any of these instances.

I don't think any of the above behavior is a flaw. If I were to point out some serious flaws, they would include things like how easy it was to bribe your way to victory in previous patches/releases. However, that has been fixed in the latest patch.

[Old Celt]

I've seen the Protectorate bug in action myself, and seen the test results in graphic detail posted by others. I think that's the worst one yet as it can totally unbalance the game as the AI creates whacky Protectorate arrangements willy nilly as it reassesses after loads. I don't know why you can't duplicate results RB, but our purpose here is to analyze demonstrated, reproducible results. Until you give me a test sequence to reproduce your results, I can make no sensible comments about your data.

I would like to know if others here support my conclusion that the reassessment process is performing illogically? Specifically, when the AI has the means and time to execute a capture of a city, it will do so when no loading of game interferes, but deliberately choose not to capture the city ASAP when a load game happens.

[roguebolo]

I can send you the save games in question. Since they are midgame saves, I obviously can't describe each and every move from the start of the game in order to duplicate the situation. But I can let you try to establish a protectorate yourself from my save position. It won't work. And that is duplicatable.

I originally encountered the Protectorate exploit because I was bound and determined to have Gaul submit to Protectorate status after laying siege to their last remaining territory in Celtiberia, so that I wouldn't have to worry about having to keep it garrisoned. They would under no conditions yield to my demands. So I went online to see if anyone had any advice about Protectorates. That's when I encountered the exploit; the person described it as working every single time, without failure, so I figured I'd just do it the quick and easy way. After several tries, I decided it just would not work.

One of the developers had also posted in that forum (I wish they'd do that more), and described a process that was somewhat longer and more arduous. I followed his advice, and a few turns later they finally agreed to a Protectorate for about 3-4,000 Florins.

I had the exact same results with Spain in Lusitania a few turns later.

I have a close friend who's also an avid RTW player, and I asked him to try it as well, in his game. He experienced the same results.

The exploit seems to work well near periods of extrema, in the absence of threats or alliances.

However, I HAVE observed an AI faction submitting to Protectorate status to another AI faction, when it did not seem reasonable. Specifically, in the same game Germany is a Protectorate of Brittania -- yet Germany is still strong, with a number of territories. Although this might seem illogical, it certainly doesn't work in my favor. It creates a situation in which they function almost as a single faction -- their alliance and military access and trade agreements give them the combined strength of a very large faction, which seems to be what people are saying they would like to confront later in the game.

As the first respondent, I will state that I agree with you -- but only partially. My agreement is restricted specifically to rebel territories, in which the AI will illogically abandon the territory for no gain. I think this is an oversight in the game design because in general the rebels seem to be treated as "just another faction".

[Pode]

Blasted work and school are keeping me from being able to do this myself, but here's a test I'd like to see. RB has advanced the hypothesis (if I understood him correctly) that most of the AI flakiness we have observed after loads settles out as you approach midgame, especially with the player being active. However, this is tough to test on other machines since being active produces a unique campaign situation and so we don't have a controlled test. I'd like to see someone post a savegame file where we can all download it easily, one from a ways into a campaign. (I'd like to volunteer RB for this since he somehow seems to not have the bu . . fea. . . issue, and I'd like to dig into that a bit. However, anyone's game will do, so long as everyone is working from the same game file). Run the standard player-passive tests from that mid-game point, 20 turns w/o loading, followed by 20 turns loading every turn, and compare AI province conquests. 20 turns loading at different frequencies would be interesting too, and if we coordinated beforehand who would run what frequency test it wouldn't be massively time consuming for any one of us to do. Thoughts?