View Full Version : Load save behavior?
Maybe we should add "Discission of Load save behavior" in Ludus Magna?
Of course, not rants just hard evidence.
Not a bad idea. I know therother would strike hard at any 'warmongers'. ~;)
Since CA supposedly has great respect for the .org perhaps a dedicated thread to evidence could help, though I fear it won't.
But it would go a long way towards presenting our views in a solid and orderly fashion with no bashing and attacks, and that is how I woul like it.
Here is nothing more scary than a coolheaded person in an argument if he present unrefuteable evidence. Hotheadedness will not go a single step towards our goal here.
Productivity
04-05-2005, 11:21
I was thinking of running my next campaign as a test, play for 50 turns to let the AI get a decent empire, then every turn take off FoW, count the number of sieges, save, count how many sieges were lifted, and then reload and repeat and compare the numbers.
Then repeat it over the next 200 or so turns. It may take a bit longer, but I don't play that often that it would bother me.
therother
04-05-2005, 11:27
This is an important and unfortunately heated game mechanics question. The LM staff are discussing this request at the moment. But if opened, it will be for posting hard game mechanics research, not a discussion or commentary on the nature of the issue.
:whip:
Get 'em told therother
Evidence only or you'll be :hanged:
Yes, it needs ruthless moderating, more ruthless then com forums.
:cry: :whip: :cry: :whip: :cry: :whip:
A suggestion for a test:
Start a number of campaigns always with a same nation (one that is regularly unlockable) in vanilla unmodded RTW 1.2. Play a number of turns without reloading and without doing anything. Toggle FoW off, and count the number of provinces that have changed hands. Now do this with saving and reloading every time, and after the same number of turns count provinces that have changed hands. If a few of us do the same thing, take the same nation, play the same number of turns, use the same counting method etc. everything should be controlled for, and we would get a bigger sample size in a less time-consuming fashion. So, we would have to agree on the faction, No. of turns, No. of tests each of us would do, etc., but we would have respectable dataset in a short time, and without any of us spending several nights just clicking end of turn button.
After we gather all the data I volunteer to do the statistical anaysis on it and then we'll have cold hard evidence that something is going on. And no more "probability theory" smokescreen arguments either - as we would have solid basis to counter them (another reason why we need a large sample size). Whether something in terms of an official fix would come out of it is suspect, but at least we could show the CA that the bug is indeed the game, and not us. ~;)
I think this is a fantastic idea, ill happily put a night aside to do this.
Do you want to suggest a faction and the conditions for the test?
therother
04-05-2005, 17:51
Okay, thread open (https://forums.totalwar.org/vb/showthread.php?t=45860). I would appreciate it if someone could give us a good overview of the perceived problem and the methods that have been used in testing for it.
:medievalcheers:
Divinehammer
04-05-2005, 18:17
https://forums.totalwar.org/vb/showthread.php?t=45862
we should try this for testing to make sure that there are no outside threats to the AI
Productivity
04-06-2005, 03:09
A suggestion for a test:
Start a number of campaigns always with a same nation (one that is regularly unlockable) in vanilla unmodded RTW 1.2. Play a number of turns without reloading and without doing anything. Toggle FoW off, and count the number of provinces that have changed hands. Now do this with saving and reloading every time, and after the same number of turns count provinces that have changed hands. If a few of us do the same thing, take the same nation, play the same number of turns, use the same counting method etc. everything should be controlled for, and we would get a bigger sample size in a less time-consuming fashion. So, we would have to agree on the faction, No. of turns, No. of tests each of us would do, etc., but we would have respectable dataset in a short time, and without any of us spending several nights just clicking end of turn button.
After we gather all the data I volunteer to do the statistical anaysis on it and then we'll have cold hard evidence that something is going on. And no more "probability theory" smokescreen arguments either - as we would have solid basis to counter them (another reason why we need a large sample size). Whether something in terms of an official fix would come out of it is suspect, but at least we could show the CA that the bug is indeed the game, and not us. ~;)
You would need a huge sample size to get around the random error though. A campaign will only be linear with a huge aggregate, and I don't think we will have enough people to do it.
I think my matched pairs idea would work better, with one change. For this you would need two computers (or two people). The first plays through without reloading, and at the end of every turn saves and sends the save over to the other computer/person, who loads it and then reports what happened there. This way the randomness is pretty much completely controlled for.
You would need a huge sample size to get around the random error though. A campaign will only be linear with a huge aggregate, and I don't think we will have enough people to do it.
Not necessarily. It would depend on the effect size and how it compares to the variance. If the effect size is large, as I predict that it will be, any randomness should not pose a problem. And I don't understand why the campaign should be linear?? This has no bearing on the test I propose AFAI can see.
Anyway, I'm thinking 50 runs for each control and treatment should be plenty and more. So, e.g. 10 people doing 5 controls (no reloads) and 5 test runs (reloading every turn), and we have the dataset. I would do it myself, but honestly I'm not so hot on RTW any more, even if it's only for testing purposes... Plus I'd rather we do it together if it's going to be done at all.
Productivity
04-06-2005, 04:17
Not necessarily. It would depend on the effect size and how it compares to the variance. If the effect size is large, as I predict that it will be, any randomness should not pose a problem. And I don't understand why the campaign should be linear?? This has no bearing on the test I propose AFAI can see.
Hmm, I mis-read you. You seem to be advocating using the number of times provinces change hands as a surrogate for how effective the AI is at sieging, whereas I am suggesting counting what actually happens to the sieges.
Ie. have pairs of people (or computers for that matter), have one start a campaign, and play through, at the end of each turn, turn FoW off, count the number of active sieges, and then save it, send it to the second person/computer who then loads it.
Both then hit end turn, one having reloaded, the other not, turn off FoW again and have a look at what happened in the sieges. If the bug is present, the loaded game will have no sieges continuing, and the continuous will have plenty. This way you directly measuring the effect of the bug.
This thread has prompted me to register on this forum :) I have already done a great deal of research on this and other related load issue. Count me in. My suggestion would be to use Britian for almost any test where AI behavior needs to be evaluated and left unchecked by reloads, for the test suggested above with the 50 turns deal. This also allows you to get through turns quickly. On a medium campaign setting you wont have to worry about rebel seiges of your settlements, and the island itself is relatively invasion proof (maybe even completely) allowing you to sit back and let the AI do its thing without having to interfere with events as they unfold.
The difficult part of this task is that it seems to affect so many areas of play, and a true study of the issue also includes a very detailed observational analysis of how the AI behaves when left under ideal circumstances(which is almost a seperate study area), with respect to seige behavior, qued troop movements etc. Not to mention the need to isolate behaviors into aggressive, responsive and defensive on the part of the AI, and engineering circumtances so they are such is time consuming to say the least. It is a momumental task but I am game.
SpencerH
04-06-2005, 14:18
I've seen 1 post that suggested that this 'feature' was present in 1.0. I'm not convinced that this is totally correct though. I played 2.5 campaigns with 1.0 (none with 1.1) and saw none of the obvious imbalances that I saw with 1.2. I had an aggressive AI, different cultures expanded, and I fought defensive seiges. With 1.2 I saw none of that. So, either I loaded less times with 1.0 (which is very unlikely) or the 'feature' manifested differently after the patch (maybe especially so with 1.2). Personally I wonder if it may be linked to the changed protectorate code (which didnt work in 1.0 and is a part of this 'feature'.
tai4ji2x
04-06-2005, 14:38
either I loaded less times with 1.0 (which is very unlikely) or the 'feature' manifested differently after the patch (maybe especially so with 1.2).
are you absolutely sure? i found that when i first started playing the game for my first couple campaigns, i played relatively non-stop. it was only after that when i started to testing things out with reloads and also went back to having a life ~;)
as for 1.0 and the bug's existence, every thread i've seen where 1.0 was tested using the same methods as 1.2, has shown the same results. you may be correct that in 1.2 with "fixed" protectorates, etc, things might go even deeper though.
Hmm, I mis-read you. You seem to be advocating using the number of times provinces change hands as a surrogate for how effective the AI is at sieging, whereas I am suggesting counting what actually happens to the sieges.
Yes, I was advocating the number of times provinces change hands. Actually, not even that, just what the state of the affairs is after a given number of turns. Note that this would not include instances where provinces were retaken by the original owners, or those that were taken the second time by a third faction. However, this also means that you wouldn't have to count provinces across the map all the time, and instead count only once per trial. Thus, the possible nuances that you may omit are more than offset by the number of trials you would be able to do in the same amount of time (hence ironing out outliers and oddities).
__________________________________________________________________
Ok, so, for those who are interested, here is the setup I would recommend:
- use umodded RTW 1.2; this means no custom mods either; we want everything to be the same and pristine in every experimenter's case so as to make the data cross-comparable
- take the Britons (thanks rcp ), medium/medium imperial campaign
- for the control, play 15 turns without doing anything; this literally means don't touch a thing other than the end of turn button; 15 turns is enough to bring two consecutive sieges that ended in starving the garrison out, I believe, hence I think this is a good number - not too big, not too small
- once 15 turns have passed (you hit end of turn for 15 times, to be exact), toggle FoW off and count the number of provinces that are not owned by the original owners; this means rebel provinces as well; this requires you to know beyond any doubt who was the original owner of the province - write it down beforehand if you think it's going to be necessary, or something to that effect
- for the "treatment" group, do everything like you did it in the control group - Britons M/M, hit end of turn (EoT) 15 times, don't touch anything - with the exception that now each turn you hit quicksave(QS) and then quickload (QL); hence, start - QS -QL - EoT - QS - QL -EoT -... etc. until you have hit EoT 15 times; at that point, turn FoW off and again count number of provinces not owned by the original owners
- if at any point in either your control or treatment campaigns you get attacked, abort that campaign, and start anew; we don't want to interfere at all, even if it's only autoresolve, and hence discard that campaign and instead do a new one fromt he begginging until you have the 5 controls (without reloads) and 5 treatments (with reloads)
- make 5 separate control campaigns (each running from fresh start to 15th EoT), and 5 separate treatment camapigns (again each running from fresh start to 15th EoT); save the campaigns at that point to a regular save, zip all those saves (you should have 10 of them, and this shuldn't take up much space) and keep them on your HD for the time being inc ase we need to send these to someone or verify something
- PM me the results, but keep in mind that I can only have 5 PMs at a time, so if I don't write back saying I got it in a day, PM me again; I'll do the analysis of the results, and post them here; I will not soup up the results, or do anything else to them other than analyze them - I do this kind of stuff for a living, and I know how to keep my personal feelings out of it, believe me
It is paramount that we all do it the same way, so please, if you think that you won't be able to do it in the way we all agree to do it, do not send me your data. I really do not want to deal with falsifications, souped up data for whatever reason, results that were collected through deviation of the basic rules, and similar things, as this would invalidate everybody else's effort as well. Let's do it right, or not do it at all.
This should tell us whether there is something fishy going on or not with saves.If we can't get 10 people (well, 9 other than myself), maybe each of us could do more runs. Say, 8 or 10 per control/treatment. Added bonus to this is that it cannot be said that it is due to a single PC configuration, or having FoW off all the time (which is a cheat and hence not an out-of-the-box game functionality). Please feel free to comment on the guidelines I posted above. If we agree, and there are no further comments until about 6pm GMT Friday, we can get to work, and by the end of the weekend, we should already be able to discuss the results. ~:cheers:
Cheers,
therother
04-06-2005, 15:22
- PM me the results, but keep in mind that I can only have 5 PMs at a time, so if I don't write back saying I got it in a day, PM me again; Perhaps it would be better to post results in the LM thread (https://forums.totalwar.org/vb/showthread.php?t=45860) on this? Should allow others to see how things are progressing, how to structure data submission, and so on. Also relieves any pressure on your PM box! ~:)
As for the idea itself, I'm onboard. Will do as many as I can when I get home.
Perhaps it would be better to post results in the LM thread (https://forums.totalwar.org/vb/showthread.php?t=45860) on this? Should allow others to see how things are progressing, how to structure data submission, and so on. Also relieves any pressure on your PM box! ~:)
As for the idea itself, I'm onboard. Will do as many as I can when I get home.
I reposted the above message in the LM thread. I hope you don't mind.
Actually, I think it's better to have the results PMed to me. That way, everything is a bit more controlled, since nobody can just copy the data over after readin them here and make up a dataset. I'm not dissing anyone, to be perfectly clear, I'm just taking every precaution I can think of to make it as controlled for and as unbiased as possible.
Also, more trials than 5/5 per person is fine, even if it exceeds the 50 datapoints in the end. It's important it's not less than 50 in the end, but more is always better (though it would be optimal if we all do the same number of trials, but it's ok). I just thought that was a resonable number that would not require too much time from anyone individually.
therother
04-06-2005, 15:37
I reposted the above message in the LM thread. I hope you don't mind.Not even remotely - that's what it's there for! :medievalcheers:
SpencerH
04-06-2005, 19:26
are you absolutely sure? i found that when i first started playing the game for my first couple campaigns, i played relatively non-stop. it was only after that when i started to testing things out with reloads and also went back to having a life ~;)
Unlike other iterations of TW, I never gave up my life ~;) . So while I cant swear that I didnt change my playing habits, its pretty unlikely.
as for 1.0 and the bug's existence, every thread i've seen where 1.0 was tested using the same methods as 1.2, has shown the same results. you may be correct that in 1.2 with "fixed" protectorates, etc, things might go even deeper though.
I just find it very suspicious that protectorates were 'fixed' in 1.2 and are now known to be definitely affected in the load 'feature'. I remember that the CA programmer that worked on the protectorates was very surprised (to put it mildly) when he was told (here) that they were not working as envisioned.
therother
04-06-2005, 22:13
- take the Britons (thanks rcp ), medium/medium imperial campaign
Hmm, I just reinstalled a fresh copy of 1.2 to test this. But there is a slight problem, of course: that I can't play as the Britons without modding the game! Do you think they'd let me off with a slight mod?
Edit: Thinking about it some more, wouldn't it be better to use the Brutii? They are pretty sheltered, and shouldn't get attacked very often. Plus everyone can play as them without modding.
SpencerH
04-07-2005, 00:00
You can play as the Brits without mod. I'm doing so right now and I've never installed a mod.
therother
04-07-2005, 00:08
You can play as the Brits without mod. I'm doing so right now and I've never installed a mod.Yes, if you've either wiped them out or won the game. I have just installed the game afresh, so as to not taint the results with any mods, so I've done neither, and don't really have the time to either. But perhaps I could just use the Protectorate only strategy (http://p223.ezboard.com/fshoguntotalwarfrm7.showMessage?topicID=24408.topic) to quickly win... ~;)
Productivity
04-07-2005, 02:13
Yes, if you've either wiped them out or won the game. I have just installed the game afresh, so as to not taint the results with any mods, so I've done neither, and don't really have the time to either. But perhaps I could just use the Protectorate only strategy (http://p223.ezboard.com/fshoguntotalwarfrm7.showMessage?topicID=24408.topic) to quickly win... ~;)
A quick campaign will unlock them pretty quickly...
Not as fast as using the bug under test to force them to surrender the moment you can get a diplomat to them :) Can I get some props for the loadgame diplomatic training school? Even the graduates from the bottom of their classes can obtain a surrender without costing a single life! ~D
Changes made to the Preferance file can't be considered mods. The file is changed every time we change the options, so that argument is out the door. So naturally opening the Brits is perfectly viable. Hey you could just argue that you played a campaign and killed them. There...
AussieGiant
04-08-2005, 11:31
Not necessarily. It would depend on the effect size and how it compares to the variance. If the effect size is large, as I predict that it will be, any randomness should not pose a problem. And I don't understand why the campaign should be linear?? This has no bearing on the test I propose AFAI can see.
Anyway, I'm thinking 50 runs for each control and treatment should be plenty and more. So, e.g. 10 people doing 5 controls (no reloads) and 5 test runs (reloading every turn), and we have the dataset. I would do it myself, but honestly I'm not so hot on RTW any more, even if it's only for testing purposes... Plus I'd rather we do it together if it's going to be done at all.
That is exactly right.
With some of the reports I have seen it is quite large.
When I mean quite large I mean the difference is regions is like 90%.
You do that 100 times with a strike rate between 87 and 93% then you have a statistical probability "out the wizzo". ~:cool:
AussieGiant
04-08-2005, 11:34
Not as fast as using the bug under test to force them to surrender the moment you can get a diplomat to them :) Can I get some props for the loadgame diplomatic training school? Even the graduates from the bottom of their classes can obtain a surrender without costing a single life! ~D
Oh completely forgot about that doozy. Slap that on the bottom of the territory changing hands test (with screen shots of both) and then ask if the game should be retitled;
ROME: TOTALLY NOT WORKING PROPERLY
or
ROME: TOTAL "FEATURES" THAT DEBILITATE GAME PLAY.
:help:
therother
04-08-2005, 12:14
To play as the Britons, without either defeating them in game or winning the game, requires an edit of descr_strat.txt. My concern was that (and is) that we want this data to be from clean, vanilla games, without the slightest mod, as that will give CA reason to doubt our findings.
To play as the Britons, without either defeating them in game or winning the game, requires an edit of descr_strat.txt. My concern was that (and is) that we want this data to be from clean, vanilla games, without the slightest mod, as that will give CA reason to doubt our findings.
You fail to realize that the Brits are as much vanilla as any other faction such as the Romans. Thus to create a situation where we use the Brits are perfectly viable.
Even if it wasn't wouldn't that say something about the game? "Yeah sure you can play as others than the Romans, but then our game is not stable or operating correctly." The game should behave the same regardless of faction selection. Ok the unplayable factions are perhaps not in that group, but you get the point.
therother
04-08-2005, 12:51
You fail to realize that the Brits are as much vanilla as any other faction such as the Romans. Thus to create a situation where we use the Brits are perfectly viable.The Britons are a vanilla faction, yes, but the way we are supposed to unlock them is clear.
I think your argument is perfectly reasonable, but there is a 3rd party here who may not be as reasonable. Better to play it safe, IMHO.
I'm just arguing that nobody can argue against it. Nobody but us/you have any idea of how the faction is unlocked. I mean what if you in fact had unlocked them by actually killing them or winning the short campaign? They know it and we do too. Should anybody try to use any argument concerning modding while we are using the Brits it becomes obvious that they have no regard to our results whatsoever and and thus using the Romans would generate no improved relations in comparison. There would just be another excuse to disregard our findings. So I think the Brits are the best because they are easier to manage, and are in general far from the action (the Romans).
A.Saturnus
04-09-2005, 22:15
Well, I think we already know that the bug exists. But we want to provide the strongest possible evidence. Devs have already dismissed bug reports on basis that they came from modded versions. Of course, no one impartial would assume that this mod affects AI behavour upon reload, but CA aren't impartial. Neither are we.
vBulletin® v3.7.1, Copyright ©2000-2024, Jelsoft Enterprises Ltd.