Updated formatted modeldb file and associated syntax checker
Apologies for the length of this post, it is intended to include enough information to be useful for people who haven't modded the modeldb file before.
Introduction
------------
A couple of weeks ago I released a formatted battle_models.modeldb file which
made editing and hence modding somewhat easier. Since then I have been working on a syntax checker for this file. Along the way I noticed that my original formatted modeldb file (henceforth referred to as the version 1 modeldb file) was not very regular in the way I broke the lines, a situation entirely due to my ignorance of regular expressions. I have now updated the modeldb file (vanilla only, no mods) to be entirely regular and have completed my syntax checker. .I'll term the new formatted modeldb file as version 2. I originally wrote the checker in MatLab, a language I know, and then translated it to Python, a new language for me. Since both are high level languages the program looks pretty much the same in both. I also include the MatLab and Python code which was used to format the original unformatted modeldb file into its new regularized form. The download can be found here:
http://www.twcenter.net/forums/downl...o=file&id=1180
The .zip file is called formatted_modeldb_version2.zip
This zip contains the following files:
readme.txt - brief descriptions of the files
formatted_battle_models.modeldb - the new version of the formatted modeldb file
formatmodeldbfile.m - MatLab script that formats the unformatted modeldb file into the new format
formatmodeldbfile.py - Python script which does the same thing
checkmodeldbsyntax.m - MatLab script that checks the syntax of the new formatted modeldb file
checkmodeldbsyntax.py - Python script which does the same as the above
The formatmodeldbfile.m and formatmodeldbfile.py are included just for interest, you don't really need them unless you want to modify them to create your own format.
You will need MatLab to run the .m file versions and Python 2.5 for Windows to run the .py file versions. MatLab costs big bucks and Python is downloadable for free. You decide. Python is available here:
http://www.python.org/
Look for the Python 2.5 Windows Installer.
Structure of the new formatted battle_models.modeldb file
---------------------------------------------------------
Before I can talk about the syntax checker and how to install it, I need to ntroduce the terminology needed to talk about the formatted modeldb file. My goal was to introduce carriage return/linefeed codes to break the lines so that each line contained only one logical information piece. For example, an integer string count followed by a filename is one logical piece of information. An integer string count followed by a unit name followed by 1 and a mesh file count constitutes two logical pieces of information and should be split into two lines. Doing the formatting this way simplified the coding of the checker.
Let us take english_knights as an example of the formatting. My comments are enclosed in square brackets, they are not part of the file of course.
Code:15 english_knights [unit name line: string count and string] 1 3 [mesh count line: second number is number of .mesh files to come] 60 unit_models/_Units/EN_Pplate_Plate/english_knights_lod0.mesh 121 [mesh file line: string count, filename, distance threshold] 60 unit_models/_Units/EN_Pplate_Plate/english_knights_lod1.mesh 1225 [mesh file line: string count, filename, distance threshold] 60 unit_models/_Units/EN_Pplate_Plate/english_knights_lod2.mesh 6400 [mesh file line: string count, filename, distance threshold] 2 [faction count line: single number giving number of faction entries to follow] 7 england [faction name line: string count and faction name] 74 unit_models/_Units/EN_Pplate_Plate/textures/mtw2_EN_Pplate_england.texture [texture file line: string count and filename] 73 unit_models/_Units/EN_Pplate_Plate/textures/mtw2_EN_Pplate_normal.texture [texture file line: string count and filename] 54 unit_sprites/england_Dummy_EN_Plate_Knights_sprite.spr [sprite file line: string count and filename] 5 slave [start of another faction entry] 73 unit_models/_Units/EN_Pplate_Plate/textures/mtw2_EN_Pplate_rebels.texture 73 unit_models/_Units/EN_Pplate_Plate/textures/mtw2_EN_Pplate_normal.texture 52 unit_sprites/slave_Dummy_EN_Plate_Knights_sprite.spr 2 [faction count line for AttachmentSets textures] 7 england 67 unit_models/AttachmentSets/Final Special Heads_england_diff.texture 67 unit_models/AttachmentSets/Final Special Heads_england_norm.texture 0 [the 0 is one of kleemann's deadly double space zeros] 5 slave [don't know what they are for but never remove] 65 unit_models/AttachmentSets/Final Special Heads_slave_diff.texture [the double spaces after them] 65 unit_models/AttachmentSets/Final Special Heads_slave_norm.texture 0 1 [From here on I term this the rest of entry] 5 Horse [section. Other than string count followed] 13 MTW2_HR_Lance [by a string, I don't know of any syntax rules] 13 MTW2_HR_Sword 2 [for entries here so I just check the counts] 21 MTW2_HR_Lance_Primary [and move on.] 14 fs_test_shield 2 17 MTW2_Mace_Primary 14 fs_test_shield 16 -0.090000004 0 0 -0.34999999 0.80000001 0.60000002 [unit definition termination delimiter]
All units follow this basic structure except for the mount units at the very
beginning. They only have the body textures and not the AttachmentSets
textures. Thus they have to be processed slightly differently by the code.
I don't know the significance of the deadly double space 0's so I deliberately
chose to break AFTER the double space and otherwise leave them strictly alone, i.e. I don't break them onto a separate line. I know you can't see it, but every line ends with a single space character (ASCII code 0x20 or decimal 32) except for the double space 0's and except for the very last line of the file
which does not have a space at all.
In outline form a unit definition entry would be:
# unitname
1 mesh count
mesh file section
faction count
body textures section
faction count
AttachmentSets section
rest of entry section
termination delimiter
I'll use this terminology in the following discussions.
Note that 16 -0.090000004 0 0 -0.34999999 0.80000001 0.60000002 is not the only entry
termination string. Mount units are terminated by
0 0 0 0 -1 0 0 0 0 0 0 0 0 [ for mount_pony, the very first entry in the file ]
0 0 -1 0 0 0 0 0 0 [ for the other mount units ]
Also
0 -1 0 0 0 0 0 0 [ for elephant_crew and other crew units starting around line 17000]
as well as
0 -1 0 0 0 0 0 0 [ for captain, general, and king units starting around line 22500]
Please don't ask what the significance of these two different termination strings are, I haven't the foggiest notion. I thought at first it had something to do with the non-presence of AttachmentSets for mounts but elephant_crew and the captains, generals, and king units all have AttachmentSets textures so that's not it.
Installation and Use
--------------------
Before installing into your /data/unit_models directory I would recommend that you simply
extract formatted_battle_models.modeldb and checkmodeldbsyntax.py into a test folder.
Rename formatted_battle_models.modeldb to battle_models.modeldb (the real name is
hard-coded into the script) and simply double click on the Python script. You will
see a DOS box come up and a bunch of text will scroll by very fast then the DOS box
will disappear and the script will launch notepad on the error log
(named modeldbsyntaxerrors.txt). There won't be any errors so you should just see
the following
Now edit battle_models.modeldb and deliberately change a string count number and/orCode:Unit count at top of file = 701 Number of processed unit models = 701 It is advisable that these two numbers should match. Total errors: 0 String count errors: 0 Faction count errors: 0
a faction count number to something wrong, save, and run again. This time notepad will
come up with some real errors telling you what line number they were found on and what
the conflicting errors were like mismatched string lengths or incorrect faction counts.
If you decide this might be a handy tool, THEN you can move the new battle_models.modeldb
and checkmodeldbsyntax.py files over to your /data/unit_models directory. Please back up
your working version of battle_models.modeldb first. Remember that this is only the
vanilla modeldb file. If you have modded in new units in the EDU, EDB, and use them
in descr_strat.txt, then the modeldb file will load (you won't get the black screen)
but after the first logo screen you will probably CTD. So to test this file out you
will have to comment out the offending units in those three files. (I think this is all
you have to do, at least it was all I had to do because those were the only files I
modded other than the modeldb file itself.)
Syntax Checker: What is does (and doesn't do)
---------------------------------------------
As alluded to in the previous section, the syntax checking is really just a first
crack at the problem, it just does the basic things that people have to do by hand now. Namely, it checks every string against the count number that precedes the string to verify that they match. It also reads the faction count number that precedes the body textures section of a unit definition as well as the faction count number that precedes the AttachmentSet textures section and verifies that the number of faction entries agrees with the number specified.
It is a common error (at least common for me) to add a new faction entry into a unit and then forget to increment the faction count, especially in both places: the body textures section and the AttachmentSets textures section. It's also common to copy a texture file and then rename it and then forget to count (or to count wrong) the new number of characters for that file name. This is what the syntax checker is supposed to help you with.
The code also reads the total unit count number in the first line of the
formatted modeldb file. This line looks like
22 serialization::archive 3 0 0 0 0 701 0 0
The number 701 is the number of units defined in the vanilla modeldb file.
(Thanks to tornnight for this discovery.)
The Python script will also count the number of unit entries you have and
print both numbers out in the summary. In Musashi's tutorial thread scsscsfanfan pointed out that he added a unit at the end of the modeldb file and it didn't show up so I would speculate that the unit number needs to be incremented when you add new units.
I have also put in code to do file existance checking but it is currently flagged out by setting the flag CHECKFILEEXISTS_FLAG = False (in the Python script). Let me explain why. Many people unpack their files into the vanilla /data directory however some people, like me, chose to unpack into a completely separate directory and then only move what files they wished to mod into the appropriately created subdirectory. This allows us to keep track of what we have modded without having to check time/date stamps in the huge number of subdirectories under /data. Thus I'm not able to turn on file existance checking on my system because I read most of the files out of the packs. In other words, this is untested code.
Thus: I CANNOT GUARENTEE THAT THIS CODE WILL WORK IF THE CHECKFILEEXISTS_FLAG IS SET TO True.
With that disclaimer in place let me describe what the file existance checking code SHOULD do. Everywhere that I check the string length of a file,
be it a .mesh, .texture, or a .spr, I take the file name and prepend it with
the string '../' which moves you up to the /data parent directory. (All paths in the modeldb file are relative to the /data directory, which is slightly odd since the modeldb file resides in the /unit_models subdirectory, hence the need for the '../' string.)
If filename is the string read from the modeldb file and tpath is the new string
formed by concatenation
tpath = '../' + filename
then the Python command
exist_flag = os.path.lexists( tpath )
will get either True or False depending on whether the file exists or not. This
might catch an error where you've misspelled a filename or forgotten to create a new texture file for a faction that you've modded into a unit entry.
At present, this is all the syntax checker attempts to do.
If anybody tries out the file existance code could you post back here if it worked or if it just gave a tremendous number of file does not exist errors? I'll try to make it work if I can get enough information on how or why it's failing.
--------------------------------------------------------------------------------------------
Addendum for modders who've added a new faction
--------------------------------------------------------------------------------------------
If you have added a new faction then the code will fail when it reads the name of your new faction (not saxons or normans, they're already included). Let's say your new faction is named:
smurfwarriors
Then you'll need to edit checkmodeldbsyntax.py and find the function (or def in Python) named isfactionname. It looks like this
def isfactionname( string ) :
if string == 'england' :
return True
elif string == 'scotland' :
return True
elif string == 'france' :
return True
elif string == 'hre' :
return True
elif string == 'denmark' :
return True
elif string == 'spain' :
return True
elif string == 'portugal' :
return True
elif string == 'milan' :
return True
elif string == 'venice' :
return True
elif string == 'papal_states' :
return True
elif string == 'sicily' :
return True
elif string == 'poland' :
return True
elif string == 'russia' :
return True
elif string == 'hungary' :
return True
elif string == 'byzantium' :
return True
elif string == 'moors' :
return True
elif string == 'egypt' :
return True
elif string == 'turks' :
return True
elif string == 'mongols' :
return True
elif string == 'timurids' :
return True
elif string == 'aztecs' :
return True
elif string == 'slave' :
return True
elif string == 'merc' :
return True
elif string == 'saxons' :
return True
elif string == 'normans' :
return True
else :
return False
return False
Just add in an elif statement like
elif string == 'smurfwarriors' :
return True
and now the code will recognize your faction. (NOTE: html removes the spacing that Python requires for defining blocks. Edit the file to see what the required spacing is.)
--------------------------------------------------------------------------------------------
Addendum for modders who've already added lots of units
--------------------------------------------------------------------------------------------
For those modders who have already made extensive edits and who don't relish starting over again with a vanilla formatted modeldb file, I can only say I feel your pain. First off, if you've already modded in 10 to 20 units you probably already got this whole count checking thing down so maybe you don't even need this tool. If you still are thinking about it then read on.
I have diff'd the version 2 modeldb file against the version 1 file and can give a list of all the units that are "nonconforming" i.e. they just need a few line breaks to bring them into conformance with the new regularized format.
The good news is that most of these units require only one enter keystroke to bring them into conformance.
The bad news is that there are 95 units that require these enter keystrokes.
As always, back up your working battle_models.modeldb file first and then only do a few edits, save, and then test until you get a feeling that this is going to work out allright. The units are listed in the order that they appear in the modeldb file so you will be working from top to bottom to make these edits.
There are four types of edits that need to be made:
(1) those where the mesh file count line is joined to the unit name line, for instance in the mounts near the top of the version 1 modeldb file you will see
15 mount_fast_pony 1.12 3
position the cursor at the beginning of the 1.12 number and hit enter.
Similarly for regular units like
15 english_knights 1 3
position the cursor on the 1 of the 1 3 part and hit enter
(2) faction counts that appear on the last mesh file entry or on the last sprite
file entry in the body textures section. Here you have to make two enter
keystrokes to bring the faction count onto its own line at both spots.
That is, the faction count before the body textures, and the faction count
before the AttachmentSets textures.
(3) lines where the termination string 0 -1 0 0 0 0 0 0 is on the same line as
the last line of a model entry. I have given the name of the unit that
immediately follows these cases so you can easily do a search to find
them. Just put the cursor on the first zero of the string 0 -1 0 0 0 0 0 0
and hit enter.
(4) three muslim units that either have an apostrophe in their name, these are
mutatawwi'a and khan's_guard, or one unit that uses their textures which
is mongol_bodyguard. For these three units I have simply listed what they
should look like after being reformatted.
Here follows all the unitnames that need to be reformatted.
The first mount entry at the top of the file needs two edits.
mount_pony 1.12 0 0 3 0 0 Split this at the 1.12 number.
The last mesh file has this sequence with the faction count, 24, in it
0 0 24 0 0 Split it at the very first 0.
The following units only need the mesh count lines split from the unit names.
mount_fast_pony 1.12 3
mount_heavy_horse 1.12 3
mount_barded_horse 1.12 3
mount_mailed_horse 1.12 4
mount_armoured_horse 1.12 3
mount_eastern_armoured_horse 1.12 4
english_knights 1 3
italian_maa_ug1 1 3
famiglia_ducale 1 3
royal_banderium 1 3
conquistadores 1 4
The following need two edits, split the faction count 10 left on the last mesh file before the body textures section and split the faction count 10 left on the last .spr sprite file before the AttachmentSets section.
ne_bodyguard (faction count of 10 left on last mesh file line, faction count of 10 left on last sprite file line )
ne_bodyguard_ug1 (faction count of 10 left on last mesh file line, faction count of 10 left on last sprite file line )
knights_templar (faction count of 10 left on last mesh file line, faction count of 10 left on last sprite file line )
knights_templar_ug1 (faction count of 10 left on last mesh file line, faction count of 10 left on last sprite file line )
knights_hospitaller (faction count of 10 left on last mesh file line, faction count of 10 left on last sprite file line )
knights_hospitaller_ug1 (faction count of 10 left on last mesh file line, faction count of 10 left on last sprite file line )
knights_hospitaller_ug2 (faction count of 10 left on last mesh file line, faction count of 10 left on last sprite file line )
These require only the mesh count split.
teutonic_knights 1 4
mercenary_frankish_knights 1 4
serbian_knights 1 3
crusader_knights 1 4
border_horse_ug1 1 4
albanian_cavalry 1 4
boyar_sons_ug1 1 4
hungarian_nobles 1 4
lithuanian cavalry_ug1 1 3
noble_highland_archers_ug1 1 4
peasant_crossbowmen_ug1 1 4
aventurier_ug1 1 4
ee_peasant_archers_ug1 1 3
cossack_musketeers_ug1 1 3
arquebusiers_ug1 1 4
lithuanian_archers_ug1 1 3
norse_archers 1 4
woodsmen_ug1 1 3
croat_axemen_ug1 1 4
transilvanian_peasants 1 3
levy_spearmen 1 4
town_militia_ug2 1 4
spear_militia 1 4
pike_militia_ug2 1 4
These two require two faction count splits.
papal_guard (faction count of 1 left on last mesh file line, faction count of 1 left on last sprite file line)
papal_guard_ug1 (faction count of 1 left on last mesh file line, faction count of 1 left on last sprite file line)
Back to mesh count splits.
swiss_pikemen 1 4
zweihander 1 3
noble_swordsmen 1 3
bill_militia_ug2 1 4
english_huscarls 1 4
This one requires two faction count splits.
swiss_guard (faction count of 1 left on last mesh file line, faction count of 1 left on last sprite file line)
Back to mesh count splits.
dismounted_polish_knights 1 3
dismounted_latinkon_ug1 1 4
dismounted_mercenary_german_knights 1 3
turkish_horse_archers_ug1 1 4
arab_cavalry_ug2 1 3
granadine_jinetes_ug1 1 3
granadine_cb_cav 1 3
camel_gunners 1 4
Split the termination strings onto their own lines for these three units.
0 -1 0 0 0 0 0 0 above elephant_artillery_crew
0 -1 0 0 0 0 0 0 above elephant_rocket_crew
0 -1 0 0 0 0 0 0 above kwarizmian_cavalry
Back to mesh count splits.
me_bodyguard_ug1 1 4
turkish_archers 1 4
ottoman_infantry 1 3
nubian_archers 1 4
me_peasants_ug1 1 4
azabs 1 4
me_town_militia 1 4
mutatawwi'a (lots of reformatting for this)
This should wind up looking like the following:
Two more mesh count splits.Code:11 mutatawwi'a 1 4 58 unit_models/_Units/ME_Burmous_Padded/mutatawwi'a_lod0.mesh 121 58 unit_models/_Units/ME_Burmous_Padded/mutatawwi'a_lod1.mesh 900 58 unit_models/_Units/ME_Burmous_Padded/mutatawwi'a_lod2.mesh 2500 58 unit_models/_Units/ME_Burmous_Padded/mutatawwi'a_lod3.mesh 6400 1 4 merc 70 unit_models/_Units/ME_Burmous_Padded/textures/ME_burmous_mercs.texture 71 unit_models/_Units/ME_Burmous_Padded/textures/ME_burmous_normal.texture 40 unit_sprites/merc_Mutatawwi'a_sprite.spr 1 4 merc 59 unit_models/AttachmentSets/Final ME Heavy_merc_diff.texture 59 unit_models/AttachmentSets/Final ME Heavy_merc_norm.texture 0 1 4 None 16 MTW2_Fast_2H_Axe 0 1 19 MTW2_2H_Axe_primary 0 16 -0.090000004 0 0 -0.34999999 0.80000001 0.60000002
janissary_heavy_inf 1 3
tabardariyya_ug1 1 4
khan's_guard (lots of reformatting for this)
Should look like:
Code:12 khan's_guard 1 4 59 unit_models/_Units/AS_Lamellar_Heavy/khan's_guard_lod0.mesh 121 59 unit_models/_Units/AS_Lamellar_Heavy/khan's_guard_lod1.mesh 900 59 unit_models/_Units/AS_Lamellar_Heavy/khan's_guard_lod2.mesh 2500 59 unit_models/_Units/AS_Lamellar_Heavy/khan's_guard_lod3.mesh 6400 2 7 mongols 73 unit_models/_Units/AS_Lamellar_Heavy/textures/AS_Lamellar_mongols.texture 72 unit_models/_Units/AS_Lamellar_Heavy/textures/AS_Lamellar_normal.texture 44 unit_sprites/mongols_Khan's_Guard_sprite.spr 8 timurids 74 unit_models/_Units/AS_Lamellar_Heavy/textures/AS_Lamellar_timurids.texture 72 unit_models/_Units/AS_Lamellar_Heavy/textures/AS_Lamellar_normal.texture 45 unit_sprites/timurids_Khan's_Guard_sprite.spr 2 7 mongols 59 unit_models/AttachmentSets/Final Asian_mongols_diff.texture 59 unit_models/AttachmentSets/Final Asian_mongols_norm.texture 0 8 timurids 60 unit_models/AttachmentSets/Final Asian_timurids_diff.texture 60 unit_models/AttachmentSets/Final Asian_timurids_norm.texture 0 1 5 Horse 13 MTW2_HR_Lance 12 MTW2_HR_Mace 2 21 MTW2_HR_Lance_Primary 14 fs_test_shield 2 17 MTW2_Mace_Primary 14 fs_test_shield 16 -0.090000004 0 0 -0.34999999 0.80000001 0.60000002
mongol_bodyguard (reformatting because it uses Khan's_Guard textures)
Should look like:
Just mesh count splits for these.Code:16 mongol_bodyguard 1 4 63 unit_models/_Units/AS_Lamellar_Heavy/mongol_bodyguard_lod0.mesh 121 63 unit_models/_Units/AS_Lamellar_Heavy/mongol_bodyguard_lod1.mesh 900 63 unit_models/_Units/AS_Lamellar_Heavy/mongol_bodyguard_lod2.mesh 2500 63 unit_models/_Units/AS_Lamellar_Heavy/mongol_bodyguard_lod3.mesh 6400 2 7 mongols 73 unit_models/_Units/AS_Lamellar_Heavy/textures/AS_Lamellar_mongols.texture 72 unit_models/_Units/AS_Lamellar_Heavy/textures/AS_Lamellar_normal.texture 44 unit_sprites/mongols_Khan's_Guard_sprite.spr 8 timurids 74 unit_models/_Units/AS_Lamellar_Heavy/textures/AS_Lamellar_timurids.texture 72 unit_models/_Units/AS_Lamellar_Heavy/textures/AS_Lamellar_normal.texture 45 unit_sprites/timurids_Khan's_Guard_sprite.spr 2 7 mongols 59 unit_models/AttachmentSets/Final Asian_mongols_diff.texture 59 unit_models/AttachmentSets/Final Asian_mongols_norm.texture 0 8 timurids 60 unit_models/AttachmentSets/Final Asian_timurids_diff.texture 60 unit_models/AttachmentSets/Final Asian_timurids_norm.texture 0 1 5 Horse 13 MTW2_HR_Lance 12 MTW2_HR_Mace 2 21 MTW2_HR_Lance_Primary 14 fs_test_shield 2 17 MTW2_Mace_Primary 14 fs_test_shield 16 -0.090000004 0 0 -0.34999999 0.80000001 0.60000002
dismounted_archers_ug1 1 4
greek_trebuchet_crew 1 4
ne_monster_ribault_crew 1 4
me_ballista_crew 1 3
me_catapult_crew 1 3
Termination splits for these.
0 -1 0 0 0 0 0 0 above aztec_general
0 -1 0 0 0 0 0 0 above eastern_captain
0 -1 0 0 0 0 0 0 above eastern_general
0 -1 0 0 0 0 0 0 above northern_captain
0 -1 0 0 0 0 0 0 above northern_general
0 -1 0 0 0 0 0 0 above norman_general
0 -1 0 0 0 0 0 0 above norman_king
0 -1 0 0 0 0 0 0 above saxon_general
0 -1 0 0 0 0 0 0 above saxon_king also break the 1 4 on the same line as saxon_king
0 -1 0 0 0 0 0 0 above dummy_en_spearmen
Just mesh count splits for these.
dummy_ne_crew 1 4
baltest_spearman 1 4
baltest_knifeman 1 4
baltest_hr_sword 1 4
baltest_hr_spear 1 4
baltest_hr_lance 1 4
And you are done, the syntax checker should be able to process your modeldb file.
Reply With Quote
Re: Updated formatted modeldb file and associated syntax checker
Without your explanation, I doubt the rest of us could have figured out this file.
Originally Posted by KnightErrant
Deciphering the battle_models.modeldb file
). 
Bookmarks