Cracking the .sd files

[alpaca]

Well, did some work on those yesterday since we need to edit them if we actually want to add new factions (not only replace them).

In case you don't know a thing about them, the .sd files can be found in the data/ui folder and describe the images associated with ui elements (i.e. they describe which part of one of the stratpages/battlepages a ui element can be found in).

Note: This thread is meant first of all as a note to myself, but I thought somebody might be interested in it

Well, a short file definiton:

bytes			content

0-3			long: 6, probably a version id or something
4-7			long: the number of pages in this file
8-11			long: the number of entries for ui elements

n times:		files
0-3			long: string length for the file name
variable		filename
0			0x00
1-4			long: width of the page
5-8			long: height of the page
9-12			long: length of the following bitmap
width*height/8	monochrome bitmap to signify cursor state (have a bitwise 1 here, you get a highlight)

m times:		ui elements
0-3			long: string length
variable		ui tag
0-1			short: file id of the page they are specified in
2-3			short: x position of the top left corner in the page
4-5			short: x position of the bottom right corner
6-7			short: y position (top left)
8-9			short: y position (top right)
10-15			0x010000000000

[shifty157]

So by editting the .sd files we can recreate the UI elements however we want unlike in RTW where we were limited to the size and arrangement of the vanilla elements?

[alpaca]

Yup, iirc RTW had these, too, but since we could just overwrite the vanilla stuff it wasn't that important
However, I don't think we can change the placement of the ui elements so we might not actually be able to change the menu layout (i.e. the new icons and stuff would have to be the same size).

[serenius4tsd]

How did you open these files? In notepad?

[alpaca]

No, in a hex editor. If you don't know what that is, I presume it will be hard for you to understand this, too ;)

[serenius4tsd]

Dont worry, i know what a hex editor is! :lol:

[roy_d_merkel]

Hey, I've done some work in c++ with these files, and I've got a mini class that can parse the files.

right now I'm working on figuring out how the bitmap file format works.

I've also noticed something odd.

Apparently, the 0x010000000000 magic number is not always correct, for example I found that the

TEXT_BUTTON_BG_MID uses the number: 0x010013001800,
POINTER_TOP_LEFT uses the number: 0x010101000300,
and POINTER_TOP_RIGHT uses the number: 0x01012f000200,

as three examples.

I've also noticed that in shared.sd, my hexdump gave me an extra 0 byte at the end of the file.

It would also appear that the ui elements actually use xPos for the left side of the element, and width to represent the right side of the
element.

And yPos for the top of the element, and height for the bottom of the element, because, for example with NUMERAL_1

xPos = 126, width = 139,
yPos = 262, height = 272.


Tell me what you think, Roy.

BTW, should I post my code also for every one to look at?

[roy_d_merkel]

okay, used the what I've found to produce a couple of small programs for editing .sd files, one adds a file resource to a .sd file, and the other one adds a variable.

Successfully used them to add a coa for wales for a mini mod I'm working on. I think I'll post the windows binaries later, I want to do some changes to my code to make them easier to use, and to remove some bugs.

More later.


Tell me what you think, Roy.

[alpaca]

Hey, I've done some work in c++ with these files, and I've got a mini class that can parse the files.

right now I'm working on figuring out how the bitmap file format works.

I've also noticed something odd.

Apparently, the 0x010000000000 magic number is not always correct, for example I found that the

TEXT_BUTTON_BG_MID uses the number: 0x010013001800,
POINTER_TOP_LEFT uses the number: 0x010101000300,
and POINTER_TOP_RIGHT uses the number: 0x01012f000200,

as three examples.

I've also noticed that in shared.sd, my hexdump gave me an extra 0 byte at the end of the file.

It would also appear that the ui elements actually use xPos for the left side of the element, and width to represent the right side of the
element.

And yPos for the top of the element, and height for the bottom of the element, because, for example with NUMERAL_1

xPos = 126, width = 139,
yPos = 262, height = 272.


Tell me what you think, Roy.

BTW, should I post my code also for every one to look at?

Well, maybe do a bit of research about what that number does then.

You're right about the width/height thing, it defines the bottom right corner of the image.

Edit (about your code): Well you may post it, I think the best way to edit these files would be to write a text-file list that contains each tga and the UI element names for that tga (with coordinates) and automatically generates mouse-over info from the alpha channel of the tga.

[roy_d_merkel]

Uploaded my current code, to 3ddownloads, waiting for them to accept.

I'm trying to figure out a good way to divine the purpose, and format of the mystery number put at the end of each ui element, because what my code does right now is put on the original 0x010000000000 that was mentioned, and it seems to be happy, but I'm curious to know why CA uses different numbers sometimes. (could these be flags of some kind? maybe 0x010000000000 represents some kind of normal state, while the other numbers represent flags specific to certain kinds of cursors?)


Your correct about the ui fixes, that would be a whole lot easier then what I have right now, basically the executables want:


addVariableToSd <.sd file (name only, with .sd, not full path> <tag (the name of the variable)> <fileid (the id of the .tga file, offset from 0 from the beginning of the .sd file)> <xpos> <width> <ypos> <height> [<medieval total war 2 path >]

addFileToSd <.sd file (name only, with .sd not full path> <filename to add (filename only with .tga, not full path) > [<medieval total war 2 path>]

readSdFile

(sd file contents need to be sent to the readSdFile)




The gui would make a whole lot more sense, but that will take a little time, because I've never done any windows gui programming outside of vb 6.0.


So far though the concept appears sound, because I have successfully used the programs to produce something, so its just a matter of figuring out the mystery numbers, figureing out the black and white bitmap format and creating a user interface that people can actually use.


Roy.

[alpaca]

Well, we have 12 bytes for the mystical number, it looks like the first two bytes at least are 01 always.

Can you maybe search for more examples of different numbers, so you could see if there's any connection?
Furthermore, it's perfectly possible that this is just data junk, so for now I'd just go with the mostly used example, and see if any problems occur in-game.

[roy_d_merkel]

Well, we have 12 bytes for the mystical number, it looks like the first two bytes at least are 01 always.

Can you maybe search for more examples of different numbers, so you could see if there's any connection?
Furthermore, it's perfectly possible that this is just data junk, so for now I'd just go with the mostly used example, and see if any problems occur in-game.

actually its 6 bytes like you originally said, and it always appears to begin with the first byte being 0x01 (I was using hex notation, sorry for the confusion) .

Here is the entire list of examples that I could find of unique values:



file name hex value
shared.sd CURSOR_PH 0x010101000200
shared.sd TEXT_BUTTON_BG_MID 0x010013001800
shared.sd POINTER_TOP_LEFT 0x010101000300
shared.sd POINTER_TOP_RIGHT 0x01012f000200
shared.sd POINTER_BOTTOM_LEFT 0x010101003100
shared.sd POINTER_BOTTOM_RIGHT 0x01012e003000
shared.sd STRAT_CARD_BACKGROUND 0x000000000000
shared.sd TOOLBAR_BG 0x000000000000
strategy.sd MARBLE_BG 0x000000000000


As you notice almost all of them are in shared.sd, with the only otherone being MARBLE_BG.

I notice that STRAT_CARD_BACKGROUND, TOOLBAR_BG, and MARBLE_BG are all null, but TEXT_BG is 0x010000000000 (could it be that a value of 0 means that its a background?)

I also notice that all of the Pointers begin with the first two bytes being 1 (hex: 0x0101), and that their right side begins with "0x2X" where X is some hex number, so could (0x01012X) mean a value dealing with the right side of a pointer?

Meanwhile both pointer lefts begin with a hex value of (0x010101), and this is true for CURSOR_PH as well, so could (0x010101) be dealing with the left side of a pointer?

So I'm thinking that these numbers are definately bit fields of some kind, but I'm not exactly sure yet what they mean, I bet I'll probably need to go in later, and massecure some of the variables (makeing the mystery value all "0xFF"s, and see what happens).


Though it appears that all values we are interested in editing right now are not using any unique values. This means that editing coat of arms, should be using the value that you mentioned above for this magic number.
Tell me what you think, Roy.

[alpaca]

Right, 6 bytes, I was a bit tired and confused when writing that XD
Anyways, that means we have 6 byte variables, 3 shorts, or anything in between.

If we assume that what the sd pages actually store is links between tags and the images and mouse-over information, what could these 3-6 values for each tag mean in context?
Looking at the more interesting examples of cursors, I think it's safe to assume they have something to do with user interaction; it's a good bet to say that, for the cursor, they somehow define which point exactly is responsible for clicking when the player clicks her mouse-button.
Under that assumption, I'd interpret the first two bytes as flags for something, the following 4 bytes would be the coordinates (2 shorts) of the click-interaction.

After having a look at the files right now (you got me interested again for a while), and comparing them to the actual tgas, it seems that my assumption for the latter 4 bytes is correct. For cursors, they correspond exactly with the (relative) coordinates of the arrow-tip.

Well, we'll still have to figure out the flags but so far, good work

Edit: Ok, I had another idea. I guess the first byte is passive clickability (i.e. a button vs a background image), if you have a 1 it's clickable if not, it isn't - this may be used for the event system or something. The second byte is active clickability (i.e. a cursor) or some other mouse-interaction (like scrolling).
So for your examples, the button is clickable but can not click itself, the cursors and arrows can click or scroll, the background can neither. Makes sense at least, even in context.

Could you do me the favor of testing it? I don't have access to the game right now ;)
However, iirc, the sd pages don't work in a mod folder unless you put a copy in the vanilla data folder, too. Maybe that's why you didn't experience any ill effects earlier (because your files weren't actually used), I think file_first works poorly with them, too.

[roy_d_merkel]

Right, 6 bytes, I was a bit tired and confused when writing that XD
Anyways, that means we have 6 byte variables, 3 shorts, or anything in between.

If we assume that what the sd pages actually store is links between tags and the images and mouse-over information, what could these 3-6 values for each tag mean in context?
Looking at the more interesting examples of cursors, I think it's safe to assume they have something to do with user interaction; it's a good bet to say that, for the cursor, they somehow define which point exactly is responsible for clicking when the player clicks her mouse-button.
Under that assumption, I'd interpret the first two bytes as flags for something, the following 4 bytes would be the coordinates (2 shorts) of the click-interaction.

After having a look at the files right now (you got me interested again for a while), and comparing them to the actual tgas, it seems that my assumption for the latter 4 bytes is correct. For cursors, they correspond exactly with the (relative) coordinates of the arrow-tip.

Well, we'll still have to figure out the flags but so far, good work

Edit: Ok, I had another idea. I guess the first byte is passive clickability (i.e. a button vs a background image), if you have a 1 it's clickable if not, it isn't - this may be used for the event system or something. The second byte is active clickability (i.e. a cursor) or some other mouse-interaction (like scrolling).
So for your examples, the button is clickable but can not click itself, the cursors and arrows can click or scroll, the background can neither. Makes sense at least, even in context.

Could you do me the favor of testing it? I don't have access to the game right now ;)
However, iirc, the sd pages don't work in a mod folder unless you put a copy in the vanilla data folder, too. Maybe that's why you didn't experience any ill effects earlier (because your files weren't actually used), I think file_first works poorly with them, too.

Nice news is I already was putting my modded .sd files, and other mod data directly in their folders under c:\programs and settings\medieval 2 total war\data directory

and my coa for the wales faction is appearing in the strategy screen

(the result of my program, the two wales icons you see on this screen come from a file added to shared.sd, and strategy.sd, and
the variables FACTION_LOGO_WALES, and SMALL_FACTION_LOGO_WALES)


This also means that I can alter that 6 byte mystery value, and see what happens if I change some values.

I think what I will do is take the variable: FACTION_LOGO_WALES, and make its mystery value 0xFFFFFFFFFFFF, and then 0x000000000000, and see what happens to it.

I'll get back to you after I have done so.

If I see changes between the two extremes, then I can go back through and try the various combinations that were already mentioned

get back to you with more info, Roy.

[roy_d_merkel]

here are the images. (changing the mystery value of FACTION_LOGO_WALES)


normally:


When I make the mystery number 0x000000000000


when I make the mystery number 0xFFFFFFFFFFFF


So the effect of the all zeros value appears to be obvious, but the all f's value seems to be less obvious (I can't tell the difference between normal and all fs, and all buttons seem to have the same clickability, and unmovableness), I'll have to figure out more experiments to try to divine what the difference between the values, although all zeros appears to be an object that has no transparencies.


I think I'll check out what happens if I change the values of POINTER_TOP_LEFT, POINTER_TOP_RIGHT, POINTER_BOTTOM_LEFT, and POINTER_BOTTOM_RIGHT to 0x010000000000, and 0xFFFFFFFFFFFF

(btw. having trouble with the game crashing to desktop when I want to start a battle as anyone, so I think that something got corrupted, I think I messed something up when I set up the textures and files for my new welsh faction.).
Tell me what you think, Roy.

[alpaca]

Interesting, so the first byte is probably a transparency switch (that would also explain why backgrounds have a 0 there, they don't need transparency).
By the way, if it's a boolean switch (i.e. true or false), probably everything that isn't a 0 is true, so FF would be the same as 01 or anything in between, which would explain why you can't see a difference between normal and FF.

I'm still pretty sure that the last four bytes are coordinates for mouse-interaction points for the cursors, it just fits too perfectly. Maybe you could try changing the coordinates (say, you change the cursor coordinates to (40,40) instead of (0,2) or what they are now, and you should have to click with the right-bottom corner of the cursor instead of the arrow-tip). Also try changing the 1 in the second byte to 0 and see what happens then.

As for your CTD, maybe you have to supply some mouse-over info in text files or something? Does the error log show anything?


So far, congratulations on getting a new faction symbol in, afaik you're the first one to do that (well a completely new one, not a replace, anyways)

[roy_d_merkel]

Interesting, so the first byte is probably a transparency switch (that would also explain why backgrounds have a 0 there, they don't need transparency).
By the way, if it's a boolean switch (i.e. true or false), probably everything that isn't a 0 is true, so FF would be the same as 01 or anything in between, which would explain why you can't see a difference between normal and FF.

I'm still pretty sure that the last four bytes are coordinates for mouse-interaction points for the cursors, it just fits too perfectly. Maybe you could try changing the coordinates (say, you change the cursor coordinates to (40,40) instead of (0,2) or what they are now, and you should have to click with the right-bottom corner of the cursor instead of the arrow-tip). Also try changing the 1 in the second byte to 0 and see what happens then.

As for your CTD, maybe you have to supply some mouse-over info in text files or something? Does the error log show anything?


So far, congratulations on getting a new faction symbol in, afaik you're the first one to do that (well a completely new one, not a replace, anyways)

the transparency info makes sense, I should have guessed that CA would have used a boolean variable instead of a bit field.

What I need to do is get the decimal values of the mystery numbers on the special cases, that may help me figure out how to teak them to divine the last 5 mystery bytes.

Let me also revert to a vanilla installation, and edit the .sd files on the vanilla install.

here a is link to my code for editing .sd files, I already have the windows executables compiled, I think you need the cygwin libraries before you can use the executables though

http://www.wikiupload.com/download_page.php?id=94766


CURSOR_PH
byte 2: 1
byte 3: 1
byte 4: 0
byte 5: 2
byte 6: 0

TEXT_BUTTON_BG_MID
byte 2: 0
byte 3: 19
byte 4: 0
byte 5: 24
byte 6: 0

POINTER_TOP_LEFT:
byte 2: 1
byte 3: 1
byte 4: 0
byte 5: 3
byte 6: 0

POINTER_TOP_RIGHT
byte 2: 1
byte 3: 47
byte 4: 0
byte 5: 2
byte 6: 0

POINTER_BOTTOM_LEFT
byte 2: 1
byte 3: 1
byte 4: 0
byte 5: 49
byte 6: 0

POINTER_BOTTOM_RIGHT
byte 2: 1
byte 3: 46
byte 4: 0
byte 5: 48
byte 6: 0

well it looks like you may be right alpaca, bytes 3, 4, 5, 6 do look like they should be coordinates of something, so what I need to figure out is what, so maybe what I'll do is edit these arrows, and change bytes 3, and 5. Maybe make the small numbers larger, and the large numbers smaller, this should cause some sort of change in the large diagonal green arrows (these appear to be the objects that POINTER_TOP_X and POINTER_BOTTOM_X refer to). hypathetically this should change where I can click on the object, correct?

I think the byte 2 is a boolean dealing with resizing, to test this, I plan to blow up my wales coa to extreme sizes, and then toggle the bit, and see what happens.


I'll give you note of what I get once I change them.

BTW. my ctds seem only to occur when I try to manually resolve a battle (I tried autobattling a few times last night, and it let me autobattle without ctds). Looking through system.log.txt, no errors are shown until the "unexpected error" message appears.

and a url to the where I'm holding the current beta of my wales mod, for those who are interested.

http://www.wikiupload.com/download_page.php?id=94759

[Arbaces]

Good work indeed, looking forward to this, best of luck in finishing it.

Arbaces.

[DarkKnight]

Is this project still active? I would hate to see this die as I await the ability to add new logos. I would just like to offer my moral support and let you know that there are people out there who want a converter or editor for these files.

[alpaca]

Is this project still active? I would hate to see this die as I await the ability to add new logos. I would just like to offer my moral support and let you know that there are people out there who want a converter or editor for these files.

Well I don't have time to work on it at the moment. I'll maybe look into it again in a few days.

By the way, as a general statement: Even just stating interest will often increase the morale of the people working on something (especially tools).
When I first posted this, it looked like pretty much nobody was really interested, so I didn't continue to work on it (because I still don't need it yet)... May have happened to roy, too, who knows ;)

Anyways, thanks for the kind words.

[roy_d_merkel]

Nah, still on it, just nursing my hand, my hand was giving me some guff, so I have cut down on my computer time for awhile to let it heal.

Found out something about the error from before, tried a new installation of M2TW, patched it, and unpacked the files, then went into battle, and the game immediately crashed.

So I bet that my battle woes have something to do with the io_flags="file_first", and the unpacked files.

Still not sure why though, I'll get back to you as I learn more.

I've been working on doing some fixes to my code:

1. change the code so that it works with windows without the cygwin libraries.
2. expand the SDFile class so that it contains the functions to add files to the .sd libarary, and variables to the .sd library.
3. fix readSd so that it accepts a filename instead of standard in.
4. finished editSDFile.
5. finish my research.


I'll change the code link once I have 1-4 finished.

(Also spending some time trying to switch my total war 2 installation over to another computer, the game keeps complaining:

13:54:45.859 [data.invalid] [error] Expected localised string 'generic_repair')

more later.

(*insert answer to research questions here*)

[Casuir]

Did you delete the desc_geography file?

[roy_d_merkel]

haven't yet, could that be the cause of my mod problems? Does that file need to be removed?

b.t.w. fixed the error that it was complaining about in the error log file by patching my installation,

but still haven't been able to figure why the battles are still crashing (even in vanilla...).

All I know is that it seems to happen when I patch ME2TW, and unpack the files, I'm wondering if your right that I'm leaving a file in the directories that I shouldn't, I think the unpacker wants me to delete a file in the readme that I haven't deleted yet.

[Casuir]

That would be the cause of yer problems then, theres a timestamp problem with that file which causes a crash with battle mode.

Btw, is there enough space in the tga's for the 9 new faction icons or can they be resized?

[roy_d_merkel]

still doing testing, but what I am doing right now is to create a whole new tga file which contains my small faction icon, and my COA.

Then I currently add that tga file to shared.sd, and strategy.sd, as well as adding new variables:

FACTION_LOGO_WALES, and
SMALL_FACTION_LOGO_WALES

into the files that reference the tga file that was added.


This way I don't have to edit any of the factory tgas, and theoretically the only limit to the number of COAs I can add is the physical limits of the number of files and variables in the .sd files.


as for resizing, I have been wanting to do tests on that, I think now that my hand isn't bothering me as much, I can actually do those tests for you

(right now I suspect that the game will resize for you, but I need to do more testing, what I was going to do was take the file that contains my COA and faction icon, and double its height and width, resizing the images to match, I'll tell you the results when I know them -- I'm hoping to know by tonight, I wanted to test if I've fixed the battle error yet).

Update: deleting descr_geography_new.txt and descr_geography_new.db appears to fix the problems with battles crashing for me...

[Lusted]

So how would you go about making a new .tga with new faction COA and faction icons on it, and referencing it in the .sd?

[alpaca]

Yeah, creating a new tga would also be my favourite approach if the game supports it (we could even make a generic version and call it mod_strat.tga and mod_shared.tga or maybe use the same image for both). I think maximum size is 512*512 though - or at least most of the vanilla files have that size.

@Lusted: Right now by writing a converter tool or heavy uncomfortable hex-editing.

[roy_d_merkel]

I need to make some images for a full tutorial, but here is what I did for my wales faction in wales mod.

1. downloaded the wales coat of arms from online (wikipedia).
2. edited the coat of arms to turn the white background into a transparancy
3. saved this as wales_coa.xcf (gimps format)
4. used that file to produce: faction_symbols/wales.tga, menu/symbols/fe_buttons*/symbol*_wales.tga and menu/symbols/fe_faction_units/wales.tga
5. created a new 128x128 tga file, and put the coa of there in both big form and small form (one to represent the mini symbol, and one to represent the big symbol) -- walesCoa.tga
6. copied this tga file into c:\Programs Files\SEGA\Medieval II Total War\data\ui\southern_european\interface
7. using my code (for which I have a download link on one of my posts, I ran):

./addFileToSd shared.sd walesCoa.tga

./addFileToSd strategy.sd walesCoa.tga

./addVariableToSd shared.sd SMALL_FACTION_LOGO_WALES 3 96 119 89 127

./addVariableToSd.exe strategy.sd FACTION_LOGO_WALES 5 31 93 28 98



tell me what you think, roy.

[alpaca]

Well I think too many console commands for the average modder
Anyways, waiting for your new version because I'm too lazy to install cygwin again.

By the way, I can confirm that our first byte seems to be a boolean flag of whether the game should use the alpha channel or not with this pic (mapping with the sign function: 0 is 0, all else is 1)

[Lusted]

Yeah, creating a new tga would also be my favourite approach if the game supports it (we could even make a generic version and call it mod_strat.tga and mod_shared.tga or maybe use the same image for both). I think maximum size is 512*512 though - or at least most of the vanilla files have that size.

We can fit the 9 faction symbols needed(big and small) in a file that size. Having a generic one with instructions on how to edit it would be good.

Roy, i'll give that a try at some point.