Results 1 to 8 of 8

Thread: Empire Mod Architecture Primer

  1. #1
    CA CA JeromeGrasdyke's Avatar
    Join Date
    Dec 2002
    Location
    At a new top-secret (non-CA) location, surrounded by lots of steel and glass, high atriums, hordes of lovely marketing ladies, and with a new and spacious desk with plenty of room for body-moving.
    Posts
    257

    CA Empire Mod Architecture Primer

    Hi All,

    Since the mod community seems to be moving full speed ahead and you've already got some basic mods up and running I thought I’d explain the way the Empire modding system is intended to work. This should enable you to work with it, not against it, and hopefully avoid breaking things released by CA in the future.

    First of all, the ETW data is meant to stay packed. The pack file system and the virtual file system that it builds internally within the game are central to the patching and modding systems, so please don’t ask users to unpack their data in order to run with your mods.

    So how should you mod the game? Well, it works like this. Empire loads files of type .pack in the data directory in a certain order determined by the pack’s type. There are three basic types – release, patch and mod. The fifth byte in the pack file header is an numerical id which tells the game what a pack is, boot packs are 0, release packs are 1, patch packs are 2, mod packs are 3. The game then loads release packs first, then patch packs, then mod packs.

    At game startup, the file entries in all of these packs are mounted into a virtual file system, and if the game finds an entry in a later pack which has the same name and location as an already existing entry, the new entry replaces the old. This means that a mod pack can replace any file in the vfs and add new ones as well, without asking the person who downloaded the mod to overwrite files on disk and destroy data.

    In order to tell the game to use a mod, you do need to feed the game a script command. Currently the easiest way to do that is to create a user script file. The way to do this is to find your preferences.empire_script.txt file (this will be in Application Data on a Windows XP machine), and create a text file next to it called user.empire_script.txt. Inside that file, for each mod create a line which reads “mod my_pack.pack;” (without the quotes, obviously). The user script gets processed on load, and will tell the game to load your mods in that order. The user script can be used for many other things as well, but we’ll leave those for another day.

    So basically, a mod is intended to be a single “mod pack” file, and installing a mod is just placing that pack inside the game’s data directory and adding the mod command, and uninstalling it is removing those things.

    Now, a note on how to work with the databases. You’ll have noticed that each table in the databases (inside the db folder) is both a folder and a binary file. When the game instantiates the databases, which happens after the virtual file system is built, it scans each directory and parses –all– of the binary files it finds. This means that a mod pack can add a new binary data fragment inside the db/unit_stats_land_table folder which adds several new units, and the game will add them to its land units database. The fragments are processed in the same order as the pack files, and any new database records with the same name as an old record will cause the new to replace the old. This means that you can replace any record within the database without touching the rest of the data or modifying the CA-supplied files on disk.

    For example, say the CA packs contain a unit record for ‘french_dragoons’, which is one entry out of hundreds inside the shipped unit_stats_land_table data fragment, which is used to build the unit table that was available at launch. Now say you create a mod pack which places a new data fragment, say called my_mods_unit_stats_land inside db/ unit_stats_land_table, which contains just one data record for ‘french_dragoons’. What will happen is your new record will end up replacing the current record for that unit in the game data, and the rest will stay as it was originally.

    So… hopefully you’ll now have a good idea of how the modding system was architected, and how you should build tools to work with it. There’s also another interesting consequence of how the system works, which is that it opens the door for mods built on top of other mods…

    One more thing to keep in mind is that we've not yet finished testing the mod support fully - that's part of what's happening here before the mod dev package gets released. We’re aware that there is currently an upper bound of about 6000 files in a single pack, though it’s very possible to create a mod spanning several packs. But be aware that there is some overhead associated with mounting a pack, and a data directory filled with scores of them will make the game startup time significantly longer.

    The other thing I should mention – since you’re going to be experimenting with this – is that the game checks the final database integrity after loading mods. So if you add a pack file which includes a new unit record which doesn’t include a display model or is invalid in some other way, the game will unload that mod pack (though it keeps the rest). This isn’t the main error checking mechanism in the ETW data pipeline, that’s done elsewhere, but as modders you should be aware of it. Unfortunately there currently isn’t any log output on the reasons why the mod failed to load, we’re looking into adding that when the modding dev kit is released.

    Armed with this info you’ll at least be able to work with the system as it was intended, without having to wait for the mod dev package. I have to say it's great seeing you guys making headway so quickly. The pack and database inspection tool was a great start, I think, and really bodes well for the future of the Empire modding scene.

    Good luck!
    Last edited by JeromeGrasdyke; 03-19-2009 at 14:17.
    "All our words are but crumbs that fall down from the feast of the mind."
    -- from 'The Prophet' by Kahlil Gibran

  2. #2
    Stranger in a strange land Moderator Hooahguy's Avatar
    Join Date
    Jun 2007
    Location
    Swampville
    Posts
    11,420

    Default Re: Empire Mod Architecture Primer

    SWEET!
    thanks for the info!
    On the Path to the Streets of Gold: a Suebi AAR
    Visited:
    A man who casts no shadow has no soul.
    Hvil i fred HoreTore

  3. #3
    the G-Diffuser Senior Member pevergreen's Avatar
    Join Date
    Nov 2006
    Location
    Brisbane, Australia
    Posts
    11,585
    Blog Entries
    2

    Default Re: Empire Mod Architecture Primer

    Thank you dear sir, we are in your debt.

    Any tentative release period for the dev kit? (Q2, Q3 etc)

    Understandable if you can't say.

    Keep doing the great job that you are. Impress me enough and I shall find the Brisbane office and shower it with love.
    Quote Originally Posted by TosaInu
    The org will be org until everyone calls it a day.

    Quote Originally Posted by KukriKhan View Post
    but I joke. Some of my best friends are Vietnamese villages.
    Quote Originally Posted by Lemur
    Anyone who wishes to refer to me as peverlemur is free to do so.

  4. #4
    Undercover Lurker Member Mailman653's Avatar
    Join Date
    Nov 2006
    Location
    Austin, TX
    Posts
    1,307

    Default Re: Empire Mod Architecture Primer

    Very cool information, thanks for posting in our little modding corner!

  5. #5
    Member Megas Methuselah's Avatar
    Join Date
    Aug 2007
    Location
    Prairie Grasslands
    Posts
    5,040

    Thumbs up Re: Empire Mod Architecture Primer

    Hey, this is nice to know, thanks!

  6. #6
    The drunken Duke Member Suraknar's Avatar
    Join Date
    Dec 2002
    Location
    Navigating the realm of Ideas
    Posts
    684

    Default Re: Empire Mod Architecture Primer

    Great thank you Jerome!
    Duke Surak'nar
    "Η ΤΑΝ Η ΕΠΙ ΤΑΣ"
    From: Residing:
    Traveled to: Over 70 Countries, most recent: and

    ~ Ask not what modding can do for you, rather ask what you can do for modding ~
    ~ Everyone dies, not everyone really fights ~

  7. #7
    The drunken Duke Member Suraknar's Avatar
    Join Date
    Dec 2002
    Location
    Navigating the realm of Ideas
    Posts
    684

    Default Re: Empire Mod Architecture Primer

    Jerome,

    Any words of wisdom for .esf modding? Like "startpos.esf" ... ? is a HEX editor our only hope?
    Duke Surak'nar
    "Η ΤΑΝ Η ΕΠΙ ΤΑΣ"
    From: Residing:
    Traveled to: Over 70 Countries, most recent: and

    ~ Ask not what modding can do for you, rather ask what you can do for modding ~
    ~ Everyone dies, not everyone really fights ~

  8. #8
    Gognard Member MikeV's Avatar
    Join Date
    Jan 2005
    Location
    Sunny Melbourne (Florida, USA)
    Posts
    203

    Default Re: Empire Mod Architecture Primer

    Quote Originally Posted by JeromeGrasdyke View Post
    The pack file system and the virtual file system that it builds internally within the game
    Sigh. It's bad enough that the kids in Redmond try to re-invent operating systems, and get it wrong. Now we have to deal with a home-brewed "virtual file system." 40 years of systems development lessons down the drain ...

    Where is this stroke of brilliance documented?
    Forums are good for sharing questions, wikis are good for sharing answers:
    Spoiler Alert, click show to read: 
    Check out the Online ETW Data in the Totalwar.org wiki.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Single Sign On provided by vBSSO