Solving the Loadgame AI Bug - A suggested approach

I think we can all agree that the bug is real, it's serious, and CA has no intention of addressing it until the Expansion (if then). All-in-all, a pretty bad business - but there is one piece of good news: Until such time as the game is saved and reloaded, it isn't "broken". So our solution would seem to involve finding a way to reload a game without going through the existing save-load process.

What follows is beyond my technical expertise to implement, but here's why I think it's possible. Back in the day I developed scenarios for Civ2, and had occasion to work with a C++ whiz. One of the laments in the Civ2 mod community involved the limited nature of the event language, which severely restricted the kinds of scripting we could employ. But what this guy came up with was nothing short of brilliant - a way to edit the game in active memory! The process was extremely complicated (and thus little used), but his tools allowed for almost any event to occur in the game.

Now it seems to me that if it's possible to identify where a game resides in working computer memory - and then go in and perform real-time alterations to it - why couldn't you map the location of RTW in working RAM, copy the as-is parameters, and then overwrite a later loaded game with this data - thus avoiding the official save-game process altogether? Hopefully there are some serious "code-monkeys" here who can weigh in on the topic.

This may be easier than I thought. Based on discussions in other threads, windows "Hibernate" mode does exactly what I've suggested - i.e. saves and reloads a copy of the "as-is" active memory. So if we can find a way to easily trigger this........ ~D

...and there is! The link below will take you to a little freeware program called "wizmo", which you can configure to run from the desktop, thereby triggering "hibernate" mode any time you desire:

http://grc.com/wizmo/wizmo.htm

....and I ran a series of tests which pretty well confirm that going in and out of hibernate does not trigger the "loadgame ai" bug. But that still leaves one not-so-little problem, do we REALLY want RTW to be running on our computers, all the time? Uhhh.....no thanks! So, now what we need is to find a way to create the "hibernate" file and access it ONLY when we want (i.e you open up a windows session, start a game of RTW, create the hibernate file when you're ready to stop playing, and then load that file whenever you want to resume the game).

The search continues..... :dizzy2:

This is way over my head but if someone can pull it off I will not only be impressed but dissapointed that some of the great modders here have not thought of it already ~;) Good luck to anyone who trys.

Sounds good, and doable if one knows what he's doing. I don't, that's for sure.

Just how resource-consuming is the operation? [SPECULATION] Could it be used, for example, in the battle mode without (serious) slowdowns? If it could be used then, couldn't someone make a proggy that'll use the "hibernate file" and based on that it'd construct new orders to AI according to the situation in battle. We could have a proper AI with this.[/SPECULATION]

At least the siege bug could be addressed with this, dunno about battle AI.

Quote:

---

some of the great modders here have not thought of it already

---

I (well, I'm not a great modder, but) actually suggested something like that, but for the battle AI and without meddling with RAM.

OK, I've spent pretty much the whole day googling, and am only slightly smarter for the effort. But it does appear the "RAM Save Game" option can be implemented in 4 different ways:

1) Basic Windows Hibernation: As discussed above, this can be implemented easily - and it works - but the downside is you can NEVER shut down RTR. Just doesn't seem to be a realistic solution.

2) Application Hibernation: Imagine being able to hibernate only the RTR Application? That sounded like a great idea, but others have considered the issue and consider it to be extremely difficult (perhaps even impossible) to implement in the Windows environment. See this link for details (the techno-speak is truly impressive, if borderline unintelligible for the average non-geek):
http://blogs.msdn.com/oldnewthing/ar...20/116749.aspx

3) Specialized RTR RAM Copy/Load Application: My original idea, but someone would have to code this from scratch. May be doable, but the magnitude and quantity of the unknowns are.....unknown.

4) Modified Windows Hibernation: Every time you hibernate, Windows creates a file called "hiberfil.sys" in the root directory of the C: drive. When you restart windows, it automatically launches this file. So far so good. But we need additional custom capability:
a) The ability to boot up from a hibernation file at any time (the absolute, bar none, super critical, MUST HAVE!)
b) Create the hibernation file without shutting down (nice to have)
c) Create multiple hibernation copies of the hibernation file and boot up the one you choose (nice to have)

Item 4a is the current focus of my research, and so far it's been "no joy". Here's why it's important. Windows only launches from the hibernation file if that was the method used to close down Windows. I can't seem to find any way to launch from the hibernation file if Windows experienced any other form of shutdown. Which means we can't launch RTR "hibernation saves" at our discretion. Which is the whole point!

So, if anyone can find a way to do this, PLEASE pass it along!

**********************************************

And shortly after making this post, I may have found the answer - it's called "Hibernate Once, Resume Many", or HORM for short. Here's a link which provides more details:

http://blogs.msdn.com/astebner/archi...01/273462.aspx

and the research continues....... ~:)

**********************************************

My enthusiasm may have been premature. HORM is pretty complicated, and doesn't seem to specifically address our need. So the net is cast a little wider......

Very good work Kull but to be honest I can't see hibernation working, did you understand any of the site about application hibernation? Hibernation is basically just a way of closing and restoring a program, this would be near impossible to modify the restore. Also hibernation leads to bugs, serious bugs and it will not work on all systems. I think our best bet would be with option 3 or some other option if there is one. Anyway good luck with it, if you can get this to work it will be a great feat.

In theory, you could use Virtual PC on a WinXP Pro host to create a "guest" operating system (for example, XP Home that is used only to play RTW) that you could hibernate / restore without affecting the host OS. It would, of course, be limited to playing a single campaign, effectively running RTW "all the time", but with the convenience of shutting it down when needed.

You could even create multiple virtual operating systems for playing several campaigns at once, but that'd take a corresponding increase in configuration work and hard drive space for multiple installations of OSes and Rome.

This would be a very hardcore approach, costing a lot of $$, time and HD space... and it would probably have its own problems, such as affecting performance. But... you could eliminate the bug! ~D ...maybe. :dizzy2:

Edit: I'll stress once more that this is just speculation. I have no real idea if it would work or not, and most likely it wouldn't be worth the expenses, time, and grief. ~:)

Edit 2: Dug this up from Micro$oft.

Quote:

---

Users switch between operating systems as easily as they switch between applications. They simply click the window containing the virtual machine. They can pause individual virtual machines so they stop using CPU cycles on the physical computer. They can also save virtual machines to disk and restore them at a later time. The restoration process normally takes a few seconds—much faster than restarting the guest operating system.

---

Sounds like it would work, then. Who has the money? ~D

Edit 3 (last edit, I promise :tongue2: ): Link to the product overview.

http://www.microsoft.com/windows/vir...rview2004.mspx

Edit Comment: I'll keep the post intact, but the Item#1 "Good News" has turned out to be unworkable - at least for now (details in the subsequent post). Likewise with the "my work here is done" comments at the bottom. Sadly no.....the search continues...... :embarassed:

***********************************

Some updates to report....first the good news!

1) A solution that will work! - At this point I am prepared to say that there is one "workable" solution that doesn't necessarily involve added cost. It is complicated, and thus most folks will probably opt out, but it IS the solution I'll be implementing, since my personal frustration with this bug had caused me to abandon RTW. And the answer is?

Dual Boot O/S - I can see the wincing from here! The idea is to establish a dual boot system, the second of which is ultra-clean and dedicated purely to RTW. In essence, this is the "Option #1" hibernate solution, but it's now workable since the second O/S is used only for RTW, and all entry into and out of your game is achieved entirely by hibernation. For everything else you do, stick with your primary O/S, and the existence of the additional O/S costs you nothing more than hard disk space and the time to set it up.

2) Modified Windows Hibernation is Dead (or at least on life support) - Sadly, there seem to be some major impediments to "Option #4". I posted this issue on a Tech board, and received several responses from senior guys who truly seemed to know what they were talking about. Two were just flat out horrified, using phrases like "severe trauma" and "potentially extremely dangerous". In a nutshell, here's the problem: The hibernation file contains all the program data that was loaded into RAM at one point in time - and as we all know, things change. You can install a patch, reset the clock for daylight savings time, move a folder or two, etc. And that's fine, but if you boot up a hibernation file created BEFORE those changes occurred......well, {screetch}, {ca-RASHH}!! ~:eek: One guy did like the concept, and provided links to two commercial programs that will allow you to perform a COMPLETE system restore, so potentially this concept is still workable. But given the relative ease of the dual-boot option, I can't imagine going any further down this road. If you're interested in the two programs (Deep Freeze and Clean Slate), or simply want to check out my "taken to the woodshed" experience, here's the Tech Forum link:
http://forums.techguy.org/showthread...27#post2503527

3) The Virtual Machine: Many thanks to Crandaelon for adding this possibility to the list. I did some footwork, and discovered an additional program that does the same thing (VMware), and - better yet - found an article that independently evaluates them, one against the other:
http://arstechnica.com/reviews/apps/vm.ars/1

For those who may be interested in this option, here are some things to keep in mind:
- Virtual Machines share the same resources with your primary O/S. That means you're probably going to need a LOT of RAM!
- The programs aren't cheap. The article listed Virtual PC at $149 and VMware at $189.
- As Crandaelon noted, there's no guarantee that RTW will perform in this environment. The product testers in the above article tried to run two DOS games, and both wouldn't function. Maybe that's just DOS, but the caution flag is hereby raised.

So....where does that leave us? My personal quest for a solution to the Loadgame AI bug has - at least for now - come to a "successful conclusion" (the proof, of course, is in the pudding!) The dual-boot hibernation concept seems like an effective option, and so I'll give that a go.

That said, I would hope others will continue the search for additional solutions, and certainly I encourage serious coders to take a hard look at whether it's possible to build a RAM Save application for RTW. ~:cheers:

Won't a dual boot O/S mean that everyone has to create a seperate boot for RTW and it will mean that the computer will restart onto the other boot only when out of the program therefore not solving the bug at all? Am I missing something here?

I think Kull means to create a boot menu and install a 2nd OS on the system, and that 2nd OS would be used only for playing Rome. After you've played the game, you'd alt-tab to desktop (leaving the game still running) and put the computer on standby.

It sounds like it could work, but I'm not sure if you can actually use the boot menu while coming out of hibernation. Still, this option is worth checking - it's surely cheaper than using VM software.

Quote:

---

Originally Posted by Sundjata Keita

Am I missing something here?

---

No, it looks like I'm missing something here! Before announcing a "workable solution", you'd think I'd actually test it out first, right? But nooooooo. I mean what could possibly go wrong? :embarassed:

Well, I just finished setting up dual boot on my system, and there's a problem. In a normal dual-boot installation, you eventually get a menu that allows for the selection of the desired O/S. And this works fine during a normal start-up. However, after hibernating and restarting, the boot sequence DOES NOT GIVE YOU THIS OPTION! You can force things using the F8 key, but that simply generates a screen with two choices - continue with restart from hibernation, or delete the hibernation data and go to the O/S selection menu!! Damn!! That is soooo frustrating!! :furious3:

Hopefully there's some way to work around this, but for now I'm back to square one. GRRRRR!!!! (Note: I'll need to make some edits to the previous "good news" post - no sense getting people's hopes up)

And many thanks to Sundjata Keita for calling me out on this!

While burrowing through all the commentary in the Application Hibernation thread (the link was posted earlier), I came upon a few gems that you might enjoy:

1) "I guess (being in games) I always hated games that saved games by writing out all of the allocated memory, and restored them by reading it and fixing up the pointers (often missing a few of the pointers in the process). Serialize everything, and you've solved both saving and reproducing any kind of state you need for your application."

Hmmm, I wonder if this is how we get the "Loadgame AI" bug? ~:cool:

2) "I want this feature. Always have. [snip] I also have Rise Of Nations open and minimised and it's been minimised for 5 days. [snip] I want RON and the IE windows to go away from my taskbar for a while. And to survive a reboot (as 5 days ago I rebooted with another RON minimised)."

And here's the absolutely classic reply:

"I thought most games had a "save" feature anyway?"

I will now die laughing!!! ~D

I don't think the dual boot option is a very good idea either, since it means you have to stop everything else you're doing in order to play.

VMWare might be the best option, if it has improved any since I last used it a long time ago. Unfortunately back then RAM certainly wasn't the only limitation. It rarely worked for 3d apps, and when it worked it was awfully slow.

I will give it a shot though.

Quote:

---

Originally Posted by bouis

I don't think the dual boot option is a very good idea either, since it means you have to stop everything else you're doing in order to play.

---

I'm not particulary thrilled with it either, but the goal is to develop workable solutions to the bug. Since we can't get into the source code and CA is unwilling to address it, we're left with these sledgehammer approaches. But any choice is better than no choice. Speaking of which.....

Quote:

---

VMWare might be the best option, if it has improved any since I last used it a long time ago. Unfortunately back then RAM certainly wasn't the only limitation. It rarely worked for 3d apps, and when it worked it was awfully slow.

I will give it a shot though.

---

Many thanks for taking this on, and I look forward to hearing the results of your tests! Good luck! :builder2:

I've been experimenting with Boot Loader programs, and they do allow for a successful dual-boot hibernate strategy. I'm always leery of recommending anything like this, since these programs usually install to the Master Boot Record, and since every system is different, what works for one person could totally hose someone else.

However. There is a freeware program called Gag45 that can be installed ONLY a floppy, and which successfully enables the Dual-boot Hibernate strategy. It can also be installed on your hard-disk, but that is strictly optional (and as noted above, "user beware"). If anyone is interested, I can provide more details on a safe dual boot-hibernate strategy using Gag45.

http://gag.sourceforge.net/download.html

Quote:

---

Originally Posted by Kull

Well, I just finished setting up dual boot on my system, and there's a problem. In a normal dual-boot installation, you eventually get a menu that allows for the selection of the desired O/S. And this works fine during a normal start-up. However, after hibernating and restarting, the boot sequence DOES NOT GIVE YOU THIS OPTION! You can force things using the F8 key, but that simply generates a screen with two choices - continue with restart from hibernation, or delete the hibernation data and go to the O/S selection menu!! Damn!! That is soooo frustrating!! :furious3:

---

Kull, if your "another" system is Linux, than you get an boot-time option to start either
a) Linux (w/o R:TW obviously)
b) Windows (w. the option of either de-hibernating your R:TW or deleting the hib. data by means of F8).

This is tested and working. What I am fighting for is to overwrite the hibernation data with the previously captured one. I realize it is risky as outlined in the post #9 in this thread. But this seems to be the only way to get a kind of the working savegame file, so to say, given the denying approach of the game developers.

I daresay the Linux bootloader is more forgiving than the M$ one!

sems to me you need to make a emulator,
And save state,

Just a option,...
Programers may be able to look at the way The old snes emulators save state, Then use one of the emulators that linux users use to play windows games,
And edit them to save state in a similar way,

Seems phesable to me,
But dont ask me to try and do it lol:)

I can let you have a Snes emulator,
But if you want a game for it Your going to haft to already Own the game b4 i can send you the rom,
Used to be i was allowed to tell you to delete it after a day or 2 But now you haft to already Own the game.

Just giving you a nother avenue to explore

kull: what's the state of this now? Do you have a working solution with the Dual-boot Hibernate strategy with the Gag45 thing?

Quote:

---

Originally Posted by harlekin65

kull: what's the state of this now? Do you have a working solution with the Dual-boot Hibernate strategy with the Gag45 thing?

---

In short, "Yes". Let me start with the requirements:

1) Either a second hard drive or a partition, each with a functioning Operating System. In my case, it's two hard drives, both loaded with Windows XP (SP2).

2) Download GAG45 (link posted above) and use it to create a bootable floppy. Although you can load GAG45 into the MBR (Master Boot Record) of your hard drive, I don't recommend it. Instead, follow the directions and load it exclusively on the floppy, and make sure you identify each O/S with a different name (so you don't forget which WinXP is the one with RTW running)

3) Now boot into the secondary O/S using the GAG45 floppy, and load RTW along with any desired patches or mods. To minimize the chance of errors, you probably shouldn't use this O/S for anything other than RTW. Again, you're going to be playing a game without saving, possibly for months, so don't add complexity.

4) Start up your game, play as long as you like, and then minimize it to the desktop. To go into hibernate, click the "Start" button and select "Turn off computer". The shut down menu will appear - press the "Shift" key and you'll see the yellow "Stand-By" button change to "Hibernate". Click this and your computer will shut down.

5) To start up your primary O/S, just boot normally. To get into the secondary O/S (with RTW), boot with the GAG45 floppy and then select either O/S, and the system will go through the "restore from hibernate" start-up sequence. When your desktop appears, voila, there's RTW just as you left it!


Issues/Problems:

Of course, there HAVE to be problems, right? Probably not a complete list, but here they are:

1) Sounds disappear. As in, you now play the whole game with no sounds! Maybe the load-AI bug ain't so bad after all? Anyway, others have reported this, and I'm not sure if it's directly related to hibernate or just a consequence of tabbing down to the desktop. If anyone has a fix, please post!

2) After rebooting into your primary O/S, Windows can tell that something's been going on and insists on running CheckDisk. It won't find a problem, and it can be pretty timeconsuming with big hard drives, so bypass the test immediately. Of course if you aren't paying attention (like me 80% of the time), you won't notice until it's running the test! If anyone knows a safe way to cancel CheckDisk, please post!

3) It seems odd that the GAG45 floppy ignores your O/S selection and just automatically restores from Hibernate (see item 5 above) - and thus removing the floppy is the only way to boot normally into the primary O/S. So this MAY be a unique feature to my system, which means your results might differ.

4) Not all systems come with 3.5 Floppy Drives these days, so I'm not sure what you would do if your system is "floppy-less". Ideas anyone?

5) Be very careful about adding thing to your RTW disk/partition from the primary O/S. When the system reboots from hibernate it expects that things will look a certain way, and something as simple as a different icon on the desktop might cause a problem. I haven't tested this to see what causes crashes and what doesn't, but keep it in mind.

Hopefully others will test this out on their systems, and perhaps we can fine-tune and improve the process, but for now here is a tested, albeit "less-than-perfect", solution to the LoadGame/AI bug. ~:)

A lot of thanks for your detailed answer ~:cheers:

In the hope of discovering another method for running RTW "permanently", I ran a series of tests using "Virtual PC" (as discussed above, this software lets you create a virtual computer, complete with it's own O/S, all resident on your existing PC). Unfortunately the results were very discouraging. I loaded up Virtual PC and found it extremely easy to use and configure. With WinXP installed, patched, and running, RTW loaded up nicely. But when I tried to run the game, it insisted that DirectX 9.0 was not installed. I verifed the presence of 9.0b, and even went so far as to download and install 9.0c, all to no avail - RTW still refused to run. So I uninstalled the game and reinstalled - this time allowing RTW to install the 9.0b drivers, but again, no joy. :inquisitive:

At this point I did some googling and found a blog run by the MS Virtual PC Program Manager. He commented that Virtual PC emulates a "lowly S3 Trio with 8mb VRAM" (not encouraging), but he'd also recently tested a number of games. I decided to try one we had in common (Alpha Centauri, minimum DirectX 5.0), and it worked fine. So then I installed MTW (min DirectX 8.0), and it refused to start, saying it was "unable to initialize Direct 3D". So it appears that Virtual PC can't handle the graphics requirements imposed by the Total War series - although that's just a guess. The only certainty is that Virtual PC is not a viable work-around for the Loadgame/AI bug. :no:

VPC Tested Games: http://blogs.msdn.com/virtual_pc_guy...gory/8225.aspx

I have finally found the time to test _and_ report on my approach to overcoming the loadgame "feature".

Shortly, it consists of having Windows XP installed on a FAT32 partition and a Linux on another one. Theoretically, one could then hibernate Windows with RTW running and then use the hibernation files, stored under Linux, as a kind of savegame files.

Unfortunanetely, this doesn't work.

I have copied hiberfil.sys and the page file under Linux, and then put other files from another game for Windows to use. Windows, however, recognized somehow that files have been altered ("Hibernation file corrupt. Restarting the system").

So I will have to test the approach by Kull, when I find the time to re-install the system.

If you have two disk drives, another way to dual boot is by using the BIOS to switch between boot devices provided the option is available. Also, the second drive can be used to backup the first drive which gives you protection against data loss due to drive failure and other causes as well.