Updated formatted modeldb file and associated syntax checker

Printable View

02-15-2007, 06:38
KnightErrant

Updated formatted modeldb file and associated syntax checker

Apologies for the length of this post, it is intended to include enough information to be useful for people who haven't modded the modeldb file before.

Introduction
------------

A couple of weeks ago I released a formatted battle_models.modeldb file which
made editing and hence modding somewhat easier. Since then I have been working on a syntax checker for this file. Along the way I noticed that my original formatted modeldb file (henceforth referred to as the version 1 modeldb file) was not very regular in the way I broke the lines, a situation entirely due to my ignorance of regular expressions. I have now updated the modeldb file (vanilla only, no mods) to be entirely regular and have completed my syntax checker. .I'll term the new formatted modeldb file as version 2. I originally wrote the checker in MatLab, a language I know, and then translated it to Python, a new language for me. Since both are high level languages the program looks pretty much the same in both. I also include the MatLab and Python code which was used to format the original unformatted modeldb file into its new regularized form. The download can be found here:
http://www.twcenter.net/forums/downl...o=file&id=1180

The .zip file is called formatted_modeldb_version2.zip

This zip contains the following files:
readme.txt - brief descriptions of the files
formatted_battle_models.modeldb - the new version of the formatted modeldb file
formatmodeldbfile.m - MatLab script that formats the unformatted modeldb file into the new format
formatmodeldbfile.py - Python script which does the same thing
checkmodeldbsyntax.m - MatLab script that checks the syntax of the new formatted modeldb file
checkmodeldbsyntax.py - Python script which does the same as the above

The formatmodeldbfile.m and formatmodeldbfile.py are included just for interest, you don't really need them unless you want to modify them to create your own format.

You will need MatLab to run the .m file versions and Python 2.5 for Windows to run the .py file versions. MatLab costs big bucks and Python is downloadable for free. You decide. Python is available here:

http://www.python.org/

Look for the Python 2.5 Windows Installer.

Structure of the new formatted battle_models.modeldb file
---------------------------------------------------------

Before I can talk about the syntax checker and how to install it, I need to ntroduce the terminology needed to talk about the formatted modeldb file. My goal was to introduce carriage return/linefeed codes to break the lines so that each line contained only one logical information piece. For example, an integer string count followed by a filename is one logical piece of information. An integer string count followed by a unit name followed by 1 and a mesh file count constitutes two logical pieces of information and should be split into two lines. Doing the formatting this way simplified the coding of the checker.

Let us take english_knights as an example of the formatting. My comments are enclosed in square brackets, they are not part of the file of course.

Code:

15 english_knights [unit name line: string count and string] 1 3 [mesh count line: second number is number of .mesh files to come] 60 unit_models/_Units/EN_Pplate_Plate/english_knights_lod0.mesh 121 [mesh file line: string count, filename, distance threshold] 60 unit_models/_Units/EN_Pplate_Plate/english_knights_lod1.mesh 1225 [mesh file line: string count, filename, distance threshold] 60 unit_models/_Units/EN_Pplate_Plate/english_knights_lod2.mesh 6400 [mesh file line: string count, filename, distance threshold] 2 [faction count line: single number giving number of faction entries to follow] 7 england [faction name line: string count and faction name] 74 unit_models/_Units/EN_Pplate_Plate/textures/mtw2_EN_Pplate_england.texture [texture file line: string count and filename] 73 unit_models/_Units/EN_Pplate_Plate/textures/mtw2_EN_Pplate_normal.texture [texture file line: string count and filename] 54 unit_sprites/england_Dummy_EN_Plate_Knights_sprite.spr [sprite file line: string count and filename] 5 slave [start of another faction entry] 73 unit_models/_Units/EN_Pplate_Plate/textures/mtw2_EN_Pplate_rebels.texture 73 unit_models/_Units/EN_Pplate_Plate/textures/mtw2_EN_Pplate_normal.texture 52 unit_sprites/slave_Dummy_EN_Plate_Knights_sprite.spr 2 [faction count line for AttachmentSets textures] 7 england 67 unit_models/AttachmentSets/Final Special Heads_england_diff.texture 67 unit_models/AttachmentSets/Final Special Heads_england_norm.texture 0 [the 0 is one of kleemann's deadly double space zeros] 5 slave [don't know what they are for but never remove] 65 unit_models/AttachmentSets/Final Special Heads_slave_diff.texture [the double spaces after them] 65 unit_models/AttachmentSets/Final Special Heads_slave_norm.texture 0 1 [From here on I term this the rest of entry] 5 Horse [section. Other than string count followed] 13 MTW2_HR_Lance [by a string, I don't know of any syntax rules] 13 MTW2_HR_Sword 2 [for entries here so I just check the counts] 21 MTW2_HR_Lance_Primary [and move on.] 14 fs_test_shield 2 17 MTW2_Mace_Primary 14 fs_test_shield 16 -0.090000004 0 0 -0.34999999 0.80000001 0.60000002 [unit definition termination delimiter]

All units follow this basic structure except for the mount units at the very
beginning. They only have the body textures and not the AttachmentSets
textures. Thus they have to be processed slightly differently by the code.

I don't know the significance of the deadly double space 0's so I deliberately
chose to break AFTER the double space and otherwise leave them strictly alone, i.e. I don't break them onto a separate line. I know you can't see it, but every line ends with a single space character (ASCII code 0x20 or decimal 32) except for the double space 0's and except for the very last line of the file
which does not have a space at all.

In outline form a unit definition entry would be:

# unitname
1 mesh count
mesh file section
faction count
body textures section
faction count
AttachmentSets section
rest of entry section
termination delimiter

I'll use this terminology in the following discussions.

Note that 16 -0.090000004 0 0 -0.34999999 0.80000001 0.60000002 is not the only entry
termination string. Mount units are terminated by
0 0 0 0 -1 0 0 0 0 0 0 0 0 [ for mount_pony, the very first entry in the file ]
0 0 -1 0 0 0 0 0 0 [ for the other mount units ]
Also
0 -1 0 0 0 0 0 0 [ for elephant_crew and other crew units starting around line 17000]
as well as
0 -1 0 0 0 0 0 0 [ for captain, general, and king units starting around line 22500]

Please don't ask what the significance of these two different termination strings are, I haven't the foggiest notion. I thought at first it had something to do with the non-presence of AttachmentSets for mounts but elephant_crew and the captains, generals, and king units all have AttachmentSets textures so that's not it.

Installation and Use
--------------------

Before installing into your /data/unit_models directory I would recommend that you simply
extract formatted_battle_models.modeldb and checkmodeldbsyntax.py into a test folder.
Rename formatted_battle_models.modeldb to battle_models.modeldb (the real name is
hard-coded into the script) and simply double click on the Python script. You will
see a DOS box come up and a bunch of text will scroll by very fast then the DOS box
will disappear and the script will launch notepad on the error log
(named modeldbsyntaxerrors.txt). There won't be any errors so you should just see
the following

Code:

Unit count at top of file = 701 Number of processed unit models = 701 It is advisable that these two numbers should match. Total errors: 0 String count errors: 0 Faction count errors: 0

Now edit battle_models.modeldb and deliberately change a string count number and/or
a faction count number to something wrong, save, and run again. This time notepad will
come up with some real errors telling you what line number they were found on and what
the conflicting errors were like mismatched string lengths or incorrect faction counts.

If you decide this might be a handy tool, THEN you can move the new battle_models.modeldb
and checkmodeldbsyntax.py files over to your /data/unit_models directory. Please back up
your working version of battle_models.modeldb first. Remember that this is only the
vanilla modeldb file. If you have modded in new units in the EDU, EDB, and use them
in descr_strat.txt, then the modeldb file will load (you won't get the black screen)
but after the first logo screen you will probably CTD. So to test this file out you
will have to comment out the offending units in those three files. (I think this is all
you have to do, at least it was all I had to do because those were the only files I
modded other than the modeldb file itself.)

Syntax Checker: What is does (and doesn't do)
---------------------------------------------

As alluded to in the previous section, the syntax checking is really just a first
crack at the problem, it just does the basic things that people have to do by hand now. Namely, it checks every string against the count number that precedes the string to verify that they match. It also reads the faction count number that precedes the body textures section of a unit definition as well as the faction count number that precedes the AttachmentSet textures section and verifies that the number of faction entries agrees with the number specified.

It is a common error (at least common for me) to add a new faction entry into a unit and then forget to increment the faction count, especially in both places: the body textures section and the AttachmentSets textures section. It's also common to copy a texture file and then rename it and then forget to count (or to count wrong) the new number of characters for that file name. This is what the syntax checker is supposed to help you with.

The code also reads the total unit count number in the first line of the
formatted modeldb file. This line looks like

22 serialization::archive 3 0 0 0 0 701 0 0

The number 701 is the number of units defined in the vanilla modeldb file.
(Thanks to tornnight for this discovery.)
The Python script will also count the number of unit entries you have and
print both numbers out in the summary. In Musashi's tutorial thread scsscsfanfan pointed out that he added a unit at the end of the modeldb file and it didn't show up so I would speculate that the unit number needs to be incremented when you add new units.

I have also put in code to do file existance checking but it is currently flagged out by setting the flag CHECKFILEEXISTS_FLAG = False (in the Python script). Let me explain why. Many people unpack their files into the vanilla /data directory however some people, like me, chose to unpack into a completely separate directory and then only move what files they wished to mod into the appropriately created subdirectory. This allows us to keep track of what we have modded without having to check time/date stamps in the huge number of subdirectories under /data. Thus I'm not able to turn on file existance checking on my system because I read most of the files out of the packs. In other words, this is untested code.

Thus: I CANNOT GUARENTEE THAT THIS CODE WILL WORK IF THE CHECKFILEEXISTS_FLAG IS SET TO True.

With that disclaimer in place let me describe what the file existance checking code SHOULD do. Everywhere that I check the string length of a file,
be it a .mesh, .texture, or a .spr, I take the file name and prepend it with
the string '../' which moves you up to the /data parent directory. (All paths in the modeldb file are relative to the /data directory, which is slightly odd since the modeldb file resides in the /unit_models subdirectory, hence the need for the '../' string.)

If filename is the string read from the modeldb file and tpath is the new string
formed by concatenation

tpath = '../' + filename

then the Python command

exist_flag = os.path.lexists( tpath )

will get either True or False depending on whether the file exists or not. This
might catch an error where you've misspelled a filename or forgotten to create a new texture file for a faction that you've modded into a unit entry.

At present, this is all the syntax checker attempts to do.

If anybody tries out the file existance code could you post back here if it worked or if it just gave a tremendous number of file does not exist errors? I'll try to make it work if I can get enough information on how or why it's failing.
:help:

--------------------------------------------------------------------------------------------
Addendum for modders who've added a new faction
--------------------------------------------------------------------------------------------

If you have added a new faction then the code will fail when it reads the name of your new faction (not saxons or normans, they're already included). Let's say your new faction is named:

smurfwarriors

Then you'll need to edit checkmodeldbsyntax.py and find the function (or def in Python) named isfactionname. It looks like this

def isfactionname( string ) :
if string == 'england' :
return True
elif string == 'scotland' :
return True
elif string == 'france' :
return True
elif string == 'hre' :
return True
elif string == 'denmark' :
return True
elif string == 'spain' :
return True
elif string == 'portugal' :
return True
elif string == 'milan' :
return True
elif string == 'venice' :
return True
elif string == 'papal_states' :
return True
elif string == 'sicily' :
return True
elif string == 'poland' :
return True
elif string == 'russia' :
return True
elif string == 'hungary' :
return True
elif string == 'byzantium' :
return True
elif string == 'moors' :
return True
elif string == 'egypt' :
return True
elif string == 'turks' :
return True
elif string == 'mongols' :
return True
elif string == 'timurids' :
return True
elif string == 'aztecs' :
return True
elif string == 'slave' :
return True
elif string == 'merc' :
return True
elif string == 'saxons' :
return True
elif string == 'normans' :
return True
else :
return False

return False

Just add in an elif statement like

elif string == 'smurfwarriors' :
return True

and now the code will recognize your faction. (NOTE: html removes the spacing that Python requires for defining blocks. Edit the file to see what the required spacing is.)

--------------------------------------------------------------------------------------------
Addendum for modders who've already added lots of units
--------------------------------------------------------------------------------------------

For those modders who have already made extensive edits and who don't relish starting over again with a vanilla formatted modeldb file, I can only say I feel your pain. First off, if you've already modded in 10 to 20 units you probably already got this whole count checking thing down so maybe you don't even need this tool. If you still are thinking about it then read on.

I have diff'd the version 2 modeldb file against the version 1 file and can give a list of all the units that are "nonconforming" i.e. they just need a few line breaks to bring them into conformance with the new regularized format.

The good news is that most of these units require only one enter keystroke to bring them into conformance. :2thumbsup:

The bad news is that there are 95 units that require these enter keystrokes. :wall:

As always, back up your working battle_models.modeldb file first and then only do a few edits, save, and then test until you get a feeling that this is going to work out allright. The units are listed in the order that they appear in the modeldb file so you will be working from top to bottom to make these edits.

There are four types of edits that need to be made:

(1) those where the mesh file count line is joined to the unit name line, for instance in the mounts near the top of the version 1 modeldb file you will see

15 mount_fast_pony 1.12 3

position the cursor at the beginning of the 1.12 number and hit enter.
Similarly for regular units like

15 english_knights 1 3

position the cursor on the 1 of the 1 3 part and hit enter

(2) faction counts that appear on the last mesh file entry or on the last sprite
file entry in the body textures section. Here you have to make two enter
keystrokes to bring the faction count onto its own line at both spots.
That is, the faction count before the body textures, and the faction count
before the AttachmentSets textures.

(3) lines where the termination string 0 -1 0 0 0 0 0 0 is on the same line as
the last line of a model entry. I have given the name of the unit that
immediately follows these cases so you can easily do a search to find
them. Just put the cursor on the first zero of the string 0 -1 0 0 0 0 0 0
and hit enter.

(4) three muslim units that either have an apostrophe in their name, these are
mutatawwi'a and khan's_guard, or one unit that uses their textures which
is mongol_bodyguard. For these three units I have simply listed what they
should look like after being reformatted.

Here follows all the unitnames that need to be reformatted.

The first mount entry at the top of the file needs two edits.
mount_pony 1.12 0 0 3 0 0 Split this at the 1.12 number.

The last mesh file has this sequence with the faction count, 24, in it
0 0 24 0 0 Split it at the very first 0.

The following units only need the mesh count lines split from the unit names.

mount_fast_pony 1.12 3
mount_heavy_horse 1.12 3
mount_barded_horse 1.12 3
mount_mailed_horse 1.12 4
mount_armoured_horse 1.12 3
mount_eastern_armoured_horse 1.12 4
english_knights 1 3
italian_maa_ug1 1 3
famiglia_ducale 1 3
royal_banderium 1 3
conquistadores 1 4

The following need two edits, split the faction count 10 left on the last mesh file before the body textures section and split the faction count 10 left on the last .spr sprite file before the AttachmentSets section.

ne_bodyguard (faction count of 10 left on last mesh file line, faction count of 10 left on last sprite file line )
ne_bodyguard_ug1 (faction count of 10 left on last mesh file line, faction count of 10 left on last sprite file line )
knights_templar (faction count of 10 left on last mesh file line, faction count of 10 left on last sprite file line )
knights_templar_ug1 (faction count of 10 left on last mesh file line, faction count of 10 left on last sprite file line )
knights_hospitaller (faction count of 10 left on last mesh file line, faction count of 10 left on last sprite file line )
knights_hospitaller_ug1 (faction count of 10 left on last mesh file line, faction count of 10 left on last sprite file line )
knights_hospitaller_ug2 (faction count of 10 left on last mesh file line, faction count of 10 left on last sprite file line )

These require only the mesh count split.

teutonic_knights 1 4
mercenary_frankish_knights 1 4
serbian_knights 1 3
crusader_knights 1 4
border_horse_ug1 1 4
albanian_cavalry 1 4
boyar_sons_ug1 1 4
hungarian_nobles 1 4
lithuanian cavalry_ug1 1 3
noble_highland_archers_ug1 1 4
peasant_crossbowmen_ug1 1 4
aventurier_ug1 1 4
ee_peasant_archers_ug1 1 3
cossack_musketeers_ug1 1 3
arquebusiers_ug1 1 4
lithuanian_archers_ug1 1 3
norse_archers 1 4
woodsmen_ug1 1 3
croat_axemen_ug1 1 4
transilvanian_peasants 1 3
levy_spearmen 1 4
town_militia_ug2 1 4
spear_militia 1 4
pike_militia_ug2 1 4

These two require two faction count splits.

papal_guard (faction count of 1 left on last mesh file line, faction count of 1 left on last sprite file line)
papal_guard_ug1 (faction count of 1 left on last mesh file line, faction count of 1 left on last sprite file line)

Back to mesh count splits.

swiss_pikemen 1 4
zweihander 1 3
noble_swordsmen 1 3
bill_militia_ug2 1 4
english_huscarls 1 4

This one requires two faction count splits.

swiss_guard (faction count of 1 left on last mesh file line, faction count of 1 left on last sprite file line)

Back to mesh count splits.

dismounted_polish_knights 1 3
dismounted_latinkon_ug1 1 4
dismounted_mercenary_german_knights 1 3
turkish_horse_archers_ug1 1 4
arab_cavalry_ug2 1 3
granadine_jinetes_ug1 1 3
granadine_cb_cav 1 3
camel_gunners 1 4

Split the termination strings onto their own lines for these three units.

0 -1 0 0 0 0 0 0 above elephant_artillery_crew
0 -1 0 0 0 0 0 0 above elephant_rocket_crew
0 -1 0 0 0 0 0 0 above kwarizmian_cavalry

Back to mesh count splits.

me_bodyguard_ug1 1 4
turkish_archers 1 4
ottoman_infantry 1 3
nubian_archers 1 4
me_peasants_ug1 1 4
azabs 1 4
me_town_militia 1 4

mutatawwi'a (lots of reformatting for this)
This should wind up looking like the following:

Code:

11 mutatawwi'a 1 4 58 unit_models/_Units/ME_Burmous_Padded/mutatawwi'a_lod0.mesh 121 58 unit_models/_Units/ME_Burmous_Padded/mutatawwi'a_lod1.mesh 900 58 unit_models/_Units/ME_Burmous_Padded/mutatawwi'a_lod2.mesh 2500 58 unit_models/_Units/ME_Burmous_Padded/mutatawwi'a_lod3.mesh 6400 1 4 merc 70 unit_models/_Units/ME_Burmous_Padded/textures/ME_burmous_mercs.texture 71 unit_models/_Units/ME_Burmous_Padded/textures/ME_burmous_normal.texture 40 unit_sprites/merc_Mutatawwi'a_sprite.spr 1 4 merc 59 unit_models/AttachmentSets/Final ME Heavy_merc_diff.texture 59 unit_models/AttachmentSets/Final ME Heavy_merc_norm.texture 0 1 4 None 16 MTW2_Fast_2H_Axe 0 1 19 MTW2_2H_Axe_primary 0 16 -0.090000004 0 0 -0.34999999 0.80000001 0.60000002

Two more mesh count splits.

janissary_heavy_inf 1 3
tabardariyya_ug1 1 4

khan's_guard (lots of reformatting for this)
Should look like:

Code:

12 khan's_guard 1 4 59 unit_models/_Units/AS_Lamellar_Heavy/khan's_guard_lod0.mesh 121 59 unit_models/_Units/AS_Lamellar_Heavy/khan's_guard_lod1.mesh 900 59 unit_models/_Units/AS_Lamellar_Heavy/khan's_guard_lod2.mesh 2500 59 unit_models/_Units/AS_Lamellar_Heavy/khan's_guard_lod3.mesh 6400 2 7 mongols 73 unit_models/_Units/AS_Lamellar_Heavy/textures/AS_Lamellar_mongols.texture 72 unit_models/_Units/AS_Lamellar_Heavy/textures/AS_Lamellar_normal.texture 44 unit_sprites/mongols_Khan's_Guard_sprite.spr 8 timurids 74 unit_models/_Units/AS_Lamellar_Heavy/textures/AS_Lamellar_timurids.texture 72 unit_models/_Units/AS_Lamellar_Heavy/textures/AS_Lamellar_normal.texture 45 unit_sprites/timurids_Khan's_Guard_sprite.spr 2 7 mongols 59 unit_models/AttachmentSets/Final Asian_mongols_diff.texture 59 unit_models/AttachmentSets/Final Asian_mongols_norm.texture 0 8 timurids 60 unit_models/AttachmentSets/Final Asian_timurids_diff.texture 60 unit_models/AttachmentSets/Final Asian_timurids_norm.texture 0 1 5 Horse 13 MTW2_HR_Lance 12 MTW2_HR_Mace 2 21 MTW2_HR_Lance_Primary 14 fs_test_shield 2 17 MTW2_Mace_Primary 14 fs_test_shield 16 -0.090000004 0 0 -0.34999999 0.80000001 0.60000002

mongol_bodyguard (reformatting because it uses Khan's_Guard textures)
Should look like:

Code:

16 mongol_bodyguard 1 4 63 unit_models/_Units/AS_Lamellar_Heavy/mongol_bodyguard_lod0.mesh 121 63 unit_models/_Units/AS_Lamellar_Heavy/mongol_bodyguard_lod1.mesh 900 63 unit_models/_Units/AS_Lamellar_Heavy/mongol_bodyguard_lod2.mesh 2500 63 unit_models/_Units/AS_Lamellar_Heavy/mongol_bodyguard_lod3.mesh 6400 2 7 mongols 73 unit_models/_Units/AS_Lamellar_Heavy/textures/AS_Lamellar_mongols.texture 72 unit_models/_Units/AS_Lamellar_Heavy/textures/AS_Lamellar_normal.texture 44 unit_sprites/mongols_Khan's_Guard_sprite.spr 8 timurids 74 unit_models/_Units/AS_Lamellar_Heavy/textures/AS_Lamellar_timurids.texture 72 unit_models/_Units/AS_Lamellar_Heavy/textures/AS_Lamellar_normal.texture 45 unit_sprites/timurids_Khan's_Guard_sprite.spr 2 7 mongols 59 unit_models/AttachmentSets/Final Asian_mongols_diff.texture 59 unit_models/AttachmentSets/Final Asian_mongols_norm.texture 0 8 timurids 60 unit_models/AttachmentSets/Final Asian_timurids_diff.texture 60 unit_models/AttachmentSets/Final Asian_timurids_norm.texture 0 1 5 Horse 13 MTW2_HR_Lance 12 MTW2_HR_Mace 2 21 MTW2_HR_Lance_Primary 14 fs_test_shield 2 17 MTW2_Mace_Primary 14 fs_test_shield 16 -0.090000004 0 0 -0.34999999 0.80000001 0.60000002

Just mesh count splits for these.

dismounted_archers_ug1 1 4
greek_trebuchet_crew 1 4
ne_monster_ribault_crew 1 4
me_ballista_crew 1 3
me_catapult_crew 1 3

Termination splits for these.

0 -1 0 0 0 0 0 0 above aztec_general
0 -1 0 0 0 0 0 0 above eastern_captain
0 -1 0 0 0 0 0 0 above eastern_general
0 -1 0 0 0 0 0 0 above northern_captain
0 -1 0 0 0 0 0 0 above northern_general
0 -1 0 0 0 0 0 0 above norman_general
0 -1 0 0 0 0 0 0 above norman_king
0 -1 0 0 0 0 0 0 above saxon_general
0 -1 0 0 0 0 0 0 above saxon_king also break the 1 4 on the same line as saxon_king
0 -1 0 0 0 0 0 0 above dummy_en_spearmen

Just mesh count splits for these.

dummy_ne_crew 1 4
baltest_spearman 1 4
baltest_knifeman 1 4
baltest_hr_sword 1 4
baltest_hr_spear 1 4
baltest_hr_lance 1 4

And you are done, the syntax checker should be able to process your modeldb file.
02-20-2007, 20:20
Andromachus Theodoulos

Re: Updated formatted modeldb file and associated syntax checker

Knight Errant,

You continue to do us all great service in the modding community. Your efforts have proved most valuable.

Thanks so much, :bow:

AT
02-20-2007, 20:25
KnightErrant

Re: Updated formatted modeldb file and associated syntax checker

You're very welcome AT. And thank you for posting, I wasn't sure
if anyone was interested in this topic.
02-27-2007, 22:52
isellj0epnuts

Re: Updated formatted modeldb file and associated syntax checker

This thread enabled me to finally create a beta ZoR system for M2TW. Your efforts at translating the modeldb are applauded, KnightErrant.
03-09-2007, 06:00
SigniferOne

Re: Updated formatted modeldb file and associated syntax checker

KE, I wanna say, really good job here. :)
03-09-2007, 17:52
KnightErrant

Re: Updated formatted modeldb file and associated syntax checker

Thanks guys.~:)
05-13-2008, 01:33
MikeV

Re: Updated formatted modeldb file and associated syntax checker

First off, thanks KnightErrant for all your hard work! :applause: Without your explanation, I doubt the rest of us could have figured out this file.

Quote:

Originally Posted by KnightErrant

I don't know the significance of the deadly double space 0's so I deliberately
chose to break AFTER the double space and otherwise leave them strictly alone

As it turns out, these are very important. They represent a NULL value, which tells the code reading the file "nothing interesting here, move along." That observation, plus the following regular structure:

Quote:

Originally Posted by KnightErrant

In outline form a unit definition entry would be:

# unitname
1 mesh count
mesh file section
faction count
body textures section
faction count
AttachmentSets section
rest of entry section
termination delimiter

... helped me decode the structure of the entire file! :idea2:
The bad news: whoever wrote the parser for this file made it very naive and rigid (that is, as we've observed, very human un-friendly). :furious3:
The good news: the regular structure should make it possible to write more tools to validate it -- less frustration for modders! :rtwyes:

Quote:

Originally Posted by KnightErrant

Please don't ask what the significance of these two different termination strings are, I haven't the foggiest notion. I thought at first it had something to do with the non-presence of AttachmentSets for mounts but elephant_crew and the captains, generals, and king units all have AttachmentSets textures so that's not it.

My guess is the last line contains timing information for the animations (all of this file's content addresses meshes, textures, and animations ...). If the line starts with -1, the other values are ignored. If the line starts with 16 (binary: 1111), the other values are floating-point numbers. Someone more familiar with the animation software can explain them.

I'm going to go over what I've learned, hopefully clearing up any remaining mystery about this file.
05-13-2008, 02:25
MikeV

Deciphering the battle_models.modeldb file
The file data/unit_models/battle_models.modeldb (the "Model DB" file) is key to driving the animations of the little sim soldiers on the tactical battle display. For many (most?) of us Total War addicts, this is one of the cooler aspects of the game system -- and the focus of many modders' attention. So, understanding how to update this file correctly is critical to getting a mod to work.

I think I've finally cracked the code! In these posts, I'm going to present what I've learned, in the hopes that it will help reduce the frustrations of aspiring modders everywhere. :furious3: That way, they can publish their results sooner, and we can all have more fun! :applause:

There are three key concepts for understanding this file:
1. Counted-length Lists (the number of items in the list, followed by the items themselves. A String is a List of characters).
2. Record structures (different Record types can have different structure).
3. Placeholders (NULL values, also known as 'nil' in some places).
With these, we can explain the whole file.

As KnightErrant explained, the file is structured. In fact, it is one long counted-length List of Records. Each of these top-level Records has the same structure:
1. Name
2. Number
3. List of mesh Records
4. Faction List of texture/sprite Records
5. (Parallel) faction List of texture/sprite Records for AttachmentSets
6. List of animation Records
Where things get a little confusing is that the last two sometimes omit values, and leave a "0 " (representing NULL) placeholder -- because the Record structure expects something there (even if it's "nothing"). :stars:

Here's a short example:

Code:

16 albanian_cavalry 1 4 60 unit_models/_Units/LB_Plain_Fancy/albanian_cavalry_lod0.mesh 121 60 unit_models/_Units/LB_Plain_Fancy/albanian_cavalry_lod1.mesh 900 60 unit_models/_Units/LB_Plain_Fancy/albanian_cavalry_lod2.mesh 2500 60 unit_models/_Units/LB_Plain_Fancy/albanian_cavalry_lod3.mesh 6400 1 4 merc 71 unit_models/_Units/LB_Plain_Fancy/textures/LB_plain_fancy_mercs.texture 72 unit_models/_Units/LB_Plain_Fancy/textures/LB_plain_fancy_normal.texture 45 unit_sprites/merc_Albanian_Cavalry_sprite.spr 1 4 merc 65 unit_models/AttachmentSets/Final European Light_merc_diff.texture 65 unit_models/AttachmentSets/Final European Light_merc_norm.texture 0 1 5 Horse 13 MTW2_HR_Spear 12 MTW2_HR_Mace 1 21 MTW2_HR_spear_Primary 1 17 MTW2_Mace_Primary 16 -0.090000004 0 0 -0.34999999 0.80000001 0.60000002

What this means:
- Name: "albanian_cavalry"
- The 1 means ... something (height? scaling? mass?). For mounts, this is always 1.12.
- The mesh Record follows. It is a List of 4 Records, each always has 2 items:
  1. a mesh file path String (the path is relative to the data directory) and
  2. an Integer (the distance, in terrain units, at which to use the mesh)
- 1 faction texture/sprite Record follows, they always have 4 items:
  1. a String (the faction's name). In this case, it's Mercenary-only unit (faction merc).
  2. the faction-specific texture file path String (again, relative file paths)
  3. the "normal" texture file path String (also relative)
  4. 1 sprite file path String (also relative)
- 1 (parallel) faction texture/sprite Record follows, this time for the AttachmentSet(s).
  NOTE: the 3rd (sprite) path String is always NULL for the AttachmentSets.
- 1 animation Record follows, they always have 4 items:
  1. a Record containing 3 Strings naming the animations to use:
    
    Mount (or "none"). In this case, a Horse.
    Primary Weapon. In this case, a Horse Rider (HR_, Camel Riders are CR_, and Elephant Riders are ER_) Spear.
    Secondary Weapon (or NULL, if the unit has only a Primary). In this case, a Right-Handed Mace.
  2. a List of Primary Weapon animations (usually 2: offensive and shield). In this case, only the Spear -- no shield.
  3. a List of Secondary Weapon animations (if present, same structure as Primary). In this case, only the Mace -- no shield.
  4. animation timing settings (unless the line starts with -1, in which case the values will all be zero)
Once you realize that the non-zero numbers are just the length of the things to follow, you almost stop noticing them. When you understand the structure, and format the text accordingly, you can easily spot errors.

Then, you can focus on the important details: does the name of the (primary, secondary) animation match the corresponding name of the weapon? Do these agree with the values in the export_descr_unit.txt ("EDU") file? Do the named files exist at the right locations? etc.

I've uploaded a copy of the re-formatted file I'm using to: http://www.twcenter.net/forums/downl...o=file&id=1979
05-13-2008, 02:34
MikeV

Re: Deciphering the battle_models.modeldb file

Here's another example:

Code:

28 mercenary_pavise_crossbowmen 1 4 77 unit_models/_Units/LN_Brigandine_heavy/mercenary_pavise_crossbowmen_lod0.mesh 121 77 unit_models/_Units/LN_Brigandine_heavy/mercenary_pavise_crossbowmen_lod1.mesh 900 77 unit_models/_Units/LN_Brigandine_heavy/mercenary_pavise_crossbowmen_lod2.mesh 2500 77 unit_models/_Units/LN_Brigandine_heavy/mercenary_pavise_crossbowmen_lod3.mesh 6400 1 4 merc 76 unit_models/_Units/LN_Brigandine_heavy/textures/LN_ Brigandine_mercs.texture 77 unit_models/_Units/LN_Brigandine_heavy/textures/LN_ Brigandine_normal.texture 57 unit_sprites/merc_Mercenary_Pavise_Crossbowmen_sprite.spr 1 4 merc 66 unit_models/AttachmentSets/Final European CB Gun_merc_diff.texture 66 unit_models/AttachmentSets/Final European CB Gun_merc_norm.texture 0 1 4 None 20 MTW2_Pavise_Crossbow 14 MTW2_Swordsman 2 21 MTW2_Crossbow_Primary 14 fs_test_shield 2 18 MTW2_Sword_Primary 14 fs_test_shield 16 -0.090000004 0 0 -0.34999999 0.80000001 0.60000002

The mercenary_pavise_crossbowmen is a foot soldier, and so lists "None" for the Mount. It has 2 weapons (Crossbow, then Sword), and a shield (seems like all units with shield animations use the fs_test_shield. Has anyone made any others?).

Another example:

Code:

11 highlanders 1 4 54 unit_models/_Units/EN_Highlander/highlanders_lod0.mesh 121 54 unit_models/_Units/EN_Highlander/highlanders_lod1.mesh 900 54 unit_models/_Units/EN_Highlander/highlanders_lod2.mesh 2500 54 unit_models/_Units/EN_Highlander/highlanders_lod3.mesh 6400 2 8 scotland 72 unit_models/_Units/EN_Highlander/textures/en_highlander_scotland.texture 70 unit_models/_Units/EN_Highlander/textures/en_highlander_normal.texture 44 unit_sprites/scotland_Highlanders_sprite.spr 5 slave 70 unit_models/_Units/EN_Highlander/textures/en_highlander_rebels.texture 70 unit_models/_Units/EN_Highlander/textures/en_highlander_normal.texture 41 unit_sprites/slave_Highlanders_sprite.spr 2 8 scotland 69 unit_models/AttachmentSets/Final European Light_scotland_diff.texture 69 unit_models/AttachmentSets/Final European Light_scotland_norm.texture 0 5 slave 66 unit_models/AttachmentSets/Final European Light_slave_diff.texture 66 unit_models/AttachmentSets/Final European Light_slave_norm.texture 0 1 4 None 14 MTW2_Fast_Mace 0 2 17 MTW2_Mace_Primary 14 fs_test_shield 0 16 -0.090000004 0 0 -0.34999999 0.80000001 0.60000002

The highlanders is another foot unit, available to the faction scotland and as rebels (the slave faction). They only have a Primary weapon (a Mace), and a shield.
05-13-2008, 02:46
MikeV

Re: Deciphering the battle_models.modeldb file

A more interesting example:

Code:

27 mount_teutonic_barded_horse 1.12 3 60 unit_models/Mounts/Barded_Horse/mount_barded_horse_lod0.mesh 121 60 unit_models/Mounts/Barded_Horse/mount_barded_horse_lod1.mesh 1225 60 unit_models/Mounts/Barded_Horse/mount_barded_horse_lod2.mesh 10000 1 3 hre 77 unit_models/Mounts/Barded_Horse/textures/horse_barding_teutonic_order.texture 69 unit_models/Mounts/Barded_Horse/textures/horse_barding_normal.texture 57 unit_sprites/teutonic_order_Mount_Barded_Horse_sprite.spr 0 1 5 Horse 14 fs_heavy_horse 0 0 0 -1 0 0 0 0 0 0

The mount_teutonic_barded_horse entry (available only the hre faction) is meant to be used from another entry in this file. Although the animation Record lists the mount name as "Horse" and Primary weapon as "fs_heavy_horse", is has neither a Primary nor a Secondary weapon animation. Note that the animation timing line starts with a -1 and has all zeros. Presumably, the code "knows" what to do with this situation.

Interestingly, even the upgraded Teutonic Knights unit only mentions a generic Horse as a mount:

Code:

20 teutonic_knights_ug2 1 3 65 unit_models/_Units/EN_Pplate_Plate/teutonic_knights_ug2_lod0.mesh 121 65 unit_models/_Units/EN_Pplate_Plate/teutonic_knights_ug2_lod1.mesh 1225 65 unit_models/_Units/EN_Pplate_Plate/teutonic_knights_ug2_lod2.mesh 6400 1 3 hre 70 unit_models/_Units/EN_Pplate_Plate/textures/mtw2_EN_Pplate_hre.texture 73 unit_models/_Units/EN_Pplate_Plate/textures/mtw2_EN_Pplate_normal.texture 45 unit_sprites/hre_NE_Late_Bodyguard_sprite.spr 1 3 hre 64 unit_models/AttachmentSets/Final Heater Special_hre_diff.texture 64 unit_models/AttachmentSets/Final Heater Special_hre_norm.texture 0 1 5 Horse 13 MTW2_HR_Lance 12 MTW2_HR_Mace 2 21 MTW2_HR_Lance_Primary 14 fs_test_shield 2 17 MTW2_Mace_Primary 14 fs_test_shield 16 -0.090000004 0 0 -0.34999999 0.80000001 0.60000002

:dizzy2:
05-13-2008, 03:09
MikeV

Re: Deciphering the battle_models.modeldb file

How about siege units and their crews? Here's a Greek Trebuchet crew's definition:

Code:

20 greek_trebuchet_crew 1 4 70 unit_models/_Units/ES_Greek_Greek_Heavy/greek_trebuchet_crew_lod0.mesh 121 70 unit_models/_Units/ES_Greek_Greek_Heavy/greek_trebuchet_crew_lod1.mesh 900 70 unit_models/_Units/ES_Greek_Greek_Heavy/greek_trebuchet_crew_lod2.mesh 2500 70 unit_models/_Units/ES_Greek_Greek_Heavy/greek_trebuchet_crew_lod3.mesh 6400 1 9 byzantium 80 unit_models/_Units/ES_Greek_Greek_Heavy/textures/mtw2_es_greek_byzantium.texture 77 unit_models/_Units/ES_Greek_Greek_Heavy/textures/mtw2_es_greek_normal.texture 53 unit_sprites/byzantium_Greek_Ballista_Crew_sprite.spr 1 9 byzantium 70 unit_models/AttachmentSets/Final European Light_byzantium_diff.texture 70 unit_models/AttachmentSets/Final European Light_byzantium_norm.texture 0 1 4 None 14 MTW2_Trebuchet 0 1 18 MTW2_Knife_Primary 0 16 -0.090000004 0 0 -0.34999999 0.80000001 0.60000002

They have only a Knife for a Primary weapon animation, and no Secondary -- clearly, no match for a serious melee unit!

But note that their Primary weapon is listed as a Trebuchet. Hmm... :inquisitive:

Over in the export_descr_units.txt ("EDU") file, this entry references the Greek_Trebuchet_Crew:

Code:

type GR Trebuchet dictionary GR_Trebuchet ; Trebuchet category siege class missile voice_type Heavy banner faction main_missile banner holy crusade soldier Greek_Trebuchet_Crew, 16, 2, 1 engine trebuchet attributes sea_faring, can_withdraw, hardy, artillery formation 1.5, 1.5, 3, 3, 3, square stat_health 1, 0 stat_pri 7, 3, no, 0, 0, melee, artillery_mechanical, piercing, knife, 25, 1.8 ;stat_pri_ex 0, 0, 0 stat_pri_attr no stat_sec 55, 3, trebuchet, 285, 30, siege_missile, artillery_mechanical, blunt, none, 25, 1 ;stat_sec_ex 0, 0, 0 stat_sec_attr ap, bp, area, launching stat_ter 12, 3, cow_carcass, 215, 3, siege_missile, artillery_mechanical, blunt, none, 25, 1 ;stat_ter_ex 0, 0, 0 stat_ter_attr ap, bp, area, launching, stat_pri_armour 2, 5, 0, leather ;stat_armour_ex 2, 0, 0, 0, 5, 0, 0, leather stat_sec_armour 0, 0, flesh stat_heat 1 stat_ground 0, 0, 0, 0 stat_mental 9, disciplined, highly_trained stat_charge_dist 30 stat_fire_delay 0 stat_food 60, 300 stat_cost 1, 430, 250, 60, 140, 430, 1, 120 armour_ug_levels 1 armour_ug_models Greek_Trebuchet_Crew

The Trebuchet is one of the few EDU entries to have a 3rd (tertiary) weapon (the cow_carcass -- so much fun! :laugh: ).

And we have this comment in that file about the stat_pri entry:

Quote:

Details of unit's primary weapon. If the unit has a missile weapon it must be the primary

So, apparently, the "_crew" unit type is a special case.
05-13-2008, 03:19
MikeV

Re: Deciphering the battle_models.modeldb file

Here's another interesting example, the basis for General units:

Code:

16 northern_general 1 4 78 unit_models/_Generals_and_Captains/Northern_General/northern_general_lod0.mesh 121 78 unit_models/_Generals_and_Captains/Northern_General/northern_general_lod1.mesh 900 78 unit_models/_Generals_and_Captains/Northern_General/northern_general_lod2.mesh 2500 78 unit_models/_Generals_and_Captains/Northern_General/northern_general_lod3.mesh 6400 18 7 england 89 unit_models/_Generals_and_Captains/Northern_General/textures/late_general_england.texture 88 unit_models/_Generals_and_Captains/Northern_General/textures/late_general_normal.texture 0 ... 18 7 england 61 unit_models/AttachmentSets/Final General_england_diff.texture 61 unit_models/AttachmentSets/Final General_england_norm.texture 0 ... 4 4 None 15 MTW2_Non_Shield 0 1 18 MTW2_Sword_Primary 0 5 horse 18 MTW2_HR_Non_Shield 0 1 18 MTW2_Sword_Primary 0 8 elephant 18 MTW2_Elephant_Crew 0 1 18 MTW2_Sword_Primary 0 5 camel 18 MTW2_HR_Non_Shield 0 1 18 MTW2_Sword_Primary 0 -1 0 0 0 0 0 0

Note that there are no sprite files. And the animation section has a List of 4 entries -- one for dismounted, and one for each known type of mount. But no animation timing. Looks like another special case, or a stub of earlier work no longer used?

For comparison, here's the entry for a Northern European Bodyguard unit:

Code:

12 ne_bodyguard 1 4 56 unit_models/_Units/EN_Lmail_Hmail/ne_bodyguard_lod0.mesh 121 56 unit_models/_Units/EN_Lmail_Hmail/ne_bodyguard_lod1.mesh 900 56 unit_models/_Units/EN_Lmail_Hmail/ne_bodyguard_lod2.mesh 2500 56 unit_models/_Units/EN_Lmail_Hmail/ne_bodyguard_lod3.mesh 6400 10 7 england 73 unit_models/_Units/EN_Lmail_Hmail/textures/EN_Lmail_Hmail_england.texture 72 unit_models/_Units/EN_Lmail_Hmail/textures/EN_Lmail_Hmail_normal.texture 44 unit_sprites/england_NE_Bodyguard_sprite.spr ... 7 england 60 unit_models/AttachmentSets/Final Heater_england_diff.texture 60 unit_models/AttachmentSets/Final Heater_england_norm.texture 0 ... 1 5 Horse 13 MTW2_HR_Lance 13 MTW2_HR_Sword 2 21 MTW2_HR_Lance_Primary 14 fs_test_shield 2 18 MTW2_Sword_Primary 14 fs_test_shield 16 -0.090000004 0 0 -0.34999999 0.80000001 0.60000002

This time there's an entry for a sprite file, and sensible values for Primary and Secondary (Horse Rider) weapon animations.
05-13-2008, 03:25
MikeV

Re: Deciphering the battle_models.modeldb file

Finally, you should know that there are various "test" entries in the file. Here's one example, for a 2-handed Swordsman unit:

Code:

19 baltest_2hswordsman 1 4 63 unit_models/_Units/EN_Lmail_Hmail/baltest_2hswordsman_lod0.mesh 121 63 unit_models/_Units/EN_Lmail_Hmail/baltest_2hswordsman_lod1.mesh 900 63 unit_models/_Units/EN_Lmail_Hmail/baltest_2hswordsman_lod2.mesh 2500 63 unit_models/_Units/EN_Lmail_Hmail/baltest_2hswordsman_lod3.mesh 6400 2 7 england 73 unit_models/_Units/EN_Lmail_Hmail/textures/EN_Lmail_Hmail_england.texture 72 unit_models/_Units/EN_Lmail_Hmail/textures/EN_Lmail_Hmail_normal.texture 0 6 france 72 unit_models/_Units/EN_Lmail_Hmail/textures/EN_Lmail_Hmail_france.texture 72 unit_models/_Units/EN_Lmail_Hmail/textures/EN_Lmail_Hmail_normal.texture 0 2 7 england 60 unit_models/AttachmentSets/Final Heater_england_diff.texture 60 unit_models/AttachmentSets/Final Heater_england_norm.texture 0 6 france 59 unit_models/AttachmentSets/Final Heater_france_diff.texture 59 unit_models/AttachmentSets/Final Heater_france_norm.texture 0 1 4 None 16 MTW2_2HSwordsman 0 1 24 MTW2_2HSwordsman_Primary 0 16 -0.090000004 0 0 -0.34999999 0.80000001 0.60000002

It's useful as a starting "blank," adding your own factions, textures, etc. You'll need to add the right sprite file, though.

--
That's it, for now. If anyone deciphers other details of this file, please feel free to add your comments.

Happy modding ... ~:cheers:
05-13-2008, 05:47
KnightErrant

Re: Updated formatted modeldb file and associated syntax checker

Hi MikeV,

Thanks for the bump, this post has slept for a long time.
You probably know this but up at the top is a unit count as well,
that's the 701. When you add entries you need to increment this
or they don't get read. That is, if you add at the bottom.
You can cut and paste entries in the middle and those baltest
units roll off the bottom and you don't get in trouble until you add
enough units to hit those in game. Also, check out descr_skeleton.txt
which defines the animation families that the modeldb file uses for the
units.

Anyway, not the point of the reply. Your user name carries programmer
under it. Do you have any experience with binary files? What do you like
to program in? C, Java, Python, any of the above? I have a question in
mind, would you be interested in looking at settlement files, specifically
the .world files that define the structures you see on the battlefield like
villages, towns, large towns, etc. I've spent several months looking at these
but just don't have the time to carry it through. Don't commit now but
have a look at these threads to look at the issue.

https://forums.totalwar.org/vb/showthread.php?t=88241

https://forums.totalwar.org/vb/showthread.php?t=89573

No kidding, these .world files are huge and complex so I don't think any
sort of complete understanding is possible. However, they do define the
the structures used by the game for making the buildings and this is
understood. The hope is to pull this information out, import to 3ds Max
and modify it, export out again, and reassemble the .world file. A lot of
modders would love to be able to get at this data and mod it so see
what you think.

Best Regards,

KE
05-13-2008, 10:45
Lusted

Re: Updated formatted modeldb file and associated syntax checker

Quote:

-0.34999999 0.80000001 0.60000002

Those are the torch offset values(hence why set to 0 for mounts).
05-14-2008, 04:33
KnightErrant

Re: Updated formatted modeldb file and associated syntax checker

Ah, thanks Lusted. Yes, that reminds me, we were given the
xml layout for this. It's in ER's thread:

https://forums.totalwar.org/vb/showthread.php?t=74083

This gives the way CA views the creation of modeldb entries
and shows what those termination floats represent for offsets.
05-14-2008, 08:16
Lusted

Re: Updated formatted modeldb file and associated syntax checker

Yup that's how I worked out what those values were(though forgot it included the whole of that line and not just the last 3 values).
08-04-2008, 23:48
Emperor Basil

Re: Updated formatted modeldb file and associated syntax checker

Is this compatible with the latest version of M2TW?
I'm having a little trouble getting it to run with my mod and was wondering if this might be a problem.
08-06-2008, 04:59
KnightErrant

Re: Updated formatted modeldb file and associated syntax checker

Almost for sure not, if we are talking about the formatted .modeldb
file. I think I did this around the 1.1 patch so surely outdated now
and certainly for Kingdoms. You can try running the Python script on
your .modeldb file and see if you can produce a formatted file that is
compatible. Some users have posted patch 1.2 etc. modeldbs so you
can also search for those (try TWcenter if you can't find them here).

If you still have problems upload your version of the modeldb file and I'll
take a shot at running the Python script and see if I can get a viable
file for you. (I stopped at 1.2 so I probably don't have the right modeldb
version.)
09-29-2008, 20:06
ivanhoex

Re: Updated formatted modeldb file and associated syntax checker

hi

i want to better understand this:

lusted Quote:
-0.34999999 0.80000001 0.60000002
Those are the torch offset values(hence why set to 0 for mounts).

this changes what the unit individuals are aiming for or the "torch" nickname has confused me?

because in that case this may be the clue to fix those units like pikes that suddenly show their back on the enemy they are attacking.