Cracking the .sd files

[alpaca]

Well, did some work on those yesterday since we need to edit them if we actually want to add new factions (not only replace them).

In case you don't know a thing about them, the .sd files can be found in the data/ui folder and describe the images associated with ui elements (i.e. they describe which part of one of the stratpages/battlepages a ui element can be found in).

Note: This thread is meant first of all as a note to myself, but I thought somebody might be interested in it

Well, a short file definiton:

bytes			content

0-3			long: 6, probably a version id or something
4-7			long: the number of pages in this file
8-11			long: the number of entries for ui elements

n times:		files
0-3			long: string length for the file name
variable		filename
0			0x00
1-4			long: width of the page
5-8			long: height of the page
9-12			long: length of the following bitmap
width*height/8	monochrome bitmap to signify cursor state (have a bitwise 1 here, you get a highlight)

m times:		ui elements
0-3			long: string length
variable		ui tag
0-1			short: file id of the page they are specified in
2-3			short: x position of the top left corner in the page
4-5			short: x position of the bottom right corner
6-7			short: y position (top left)
8-9			short: y position (top right)
10-15			0x010000000000

[shifty157]

So by editting the .sd files we can recreate the UI elements however we want unlike in RTW where we were limited to the size and arrangement of the vanilla elements?

[alpaca]

Yup, iirc RTW had these, too, but since we could just overwrite the vanilla stuff it wasn't that important
However, I don't think we can change the placement of the ui elements so we might not actually be able to change the menu layout (i.e. the new icons and stuff would have to be the same size).

[serenius4tsd]

How did you open these files? In notepad?

[alpaca]

No, in a hex editor. If you don't know what that is, I presume it will be hard for you to understand this, too ;)

[serenius4tsd]

Dont worry, i know what a hex editor is! :lol:

[roy_d_merkel]

Hey, I've done some work in c++ with these files, and I've got a mini class that can parse the files.

right now I'm working on figuring out how the bitmap file format works.

I've also noticed something odd.

Apparently, the 0x010000000000 magic number is not always correct, for example I found that the

TEXT_BUTTON_BG_MID uses the number: 0x010013001800,
POINTER_TOP_LEFT uses the number: 0x010101000300,
and POINTER_TOP_RIGHT uses the number: 0x01012f000200,

as three examples.

I've also noticed that in shared.sd, my hexdump gave me an extra 0 byte at the end of the file.

It would also appear that the ui elements actually use xPos for the left side of the element, and width to represent the right side of the
element.

And yPos for the top of the element, and height for the bottom of the element, because, for example with NUMERAL_1

xPos = 126, width = 139,
yPos = 262, height = 272.


Tell me what you think, Roy.

BTW, should I post my code also for every one to look at?

[roy_d_merkel]

okay, used the what I've found to produce a couple of small programs for editing .sd files, one adds a file resource to a .sd file, and the other one adds a variable.

Successfully used them to add a coa for wales for a mini mod I'm working on. I think I'll post the windows binaries later, I want to do some changes to my code to make them easier to use, and to remove some bugs.

More later.


Tell me what you think, Roy.

[alpaca]

Hey, I've done some work in c++ with these files, and I've got a mini class that can parse the files.

right now I'm working on figuring out how the bitmap file format works.

I've also noticed something odd.

Apparently, the 0x010000000000 magic number is not always correct, for example I found that the

TEXT_BUTTON_BG_MID uses the number: 0x010013001800,
POINTER_TOP_LEFT uses the number: 0x010101000300,
and POINTER_TOP_RIGHT uses the number: 0x01012f000200,

as three examples.

I've also noticed that in shared.sd, my hexdump gave me an extra 0 byte at the end of the file.

It would also appear that the ui elements actually use xPos for the left side of the element, and width to represent the right side of the
element.

And yPos for the top of the element, and height for the bottom of the element, because, for example with NUMERAL_1

xPos = 126, width = 139,
yPos = 262, height = 272.


Tell me what you think, Roy.

BTW, should I post my code also for every one to look at?

Well, maybe do a bit of research about what that number does then.

You're right about the width/height thing, it defines the bottom right corner of the image.

Edit (about your code): Well you may post it, I think the best way to edit these files would be to write a text-file list that contains each tga and the UI element names for that tga (with coordinates) and automatically generates mouse-over info from the alpha channel of the tga.

[roy_d_merkel]

Uploaded my current code, to 3ddownloads, waiting for them to accept.

I'm trying to figure out a good way to divine the purpose, and format of the mystery number put at the end of each ui element, because what my code does right now is put on the original 0x010000000000 that was mentioned, and it seems to be happy, but I'm curious to know why CA uses different numbers sometimes. (could these be flags of some kind? maybe 0x010000000000 represents some kind of normal state, while the other numbers represent flags specific to certain kinds of cursors?)


Your correct about the ui fixes, that would be a whole lot easier then what I have right now, basically the executables want:


addVariableToSd <.sd file (name only, with .sd, not full path> <tag (the name of the variable)> <fileid (the id of the .tga file, offset from 0 from the beginning of the .sd file)> <xpos> <width> <ypos> <height> [<medieval total war 2 path >]

addFileToSd <.sd file (name only, with .sd not full path> <filename to add (filename only with .tga, not full path) > [<medieval total war 2 path>]

readSdFile

(sd file contents need to be sent to the readSdFile)




The gui would make a whole lot more sense, but that will take a little time, because I've never done any windows gui programming outside of vb 6.0.


So far though the concept appears sound, because I have successfully used the programs to produce something, so its just a matter of figuring out the mystery numbers, figureing out the black and white bitmap format and creating a user interface that people can actually use.


Roy.

[alpaca]

Well, we have 12 bytes for the mystical number, it looks like the first two bytes at least are 01 always.

Can you maybe search for more examples of different numbers, so you could see if there's any connection?
Furthermore, it's perfectly possible that this is just data junk, so for now I'd just go with the mostly used example, and see if any problems occur in-game.

[roy_d_merkel]

Well, we have 12 bytes for the mystical number, it looks like the first two bytes at least are 01 always.

Can you maybe search for more examples of different numbers, so you could see if there's any connection?
Furthermore, it's perfectly possible that this is just data junk, so for now I'd just go with the mostly used example, and see if any problems occur in-game.

actually its 6 bytes like you originally said, and it always appears to begin with the first byte being 0x01 (I was using hex notation, sorry for the confusion) .

Here is the entire list of examples that I could find of unique values:



file name hex value
shared.sd CURSOR_PH 0x010101000200
shared.sd TEXT_BUTTON_BG_MID 0x010013001800
shared.sd POINTER_TOP_LEFT 0x010101000300
shared.sd POINTER_TOP_RIGHT 0x01012f000200
shared.sd POINTER_BOTTOM_LEFT 0x010101003100
shared.sd POINTER_BOTTOM_RIGHT 0x01012e003000
shared.sd STRAT_CARD_BACKGROUND 0x000000000000
shared.sd TOOLBAR_BG 0x000000000000
strategy.sd MARBLE_BG 0x000000000000


As you notice almost all of them are in shared.sd, with the only otherone being MARBLE_BG.

I notice that STRAT_CARD_BACKGROUND, TOOLBAR_BG, and MARBLE_BG are all null, but TEXT_BG is 0x010000000000 (could it be that a value of 0 means that its a background?)

I also notice that all of the Pointers begin with the first two bytes being 1 (hex: 0x0101), and that their right side begins with "0x2X" where X is some hex number, so could (0x01012X) mean a value dealing with the right side of a pointer?

Meanwhile both pointer lefts begin with a hex value of (0x010101), and this is true for CURSOR_PH as well, so could (0x010101) be dealing with the left side of a pointer?

So I'm thinking that these numbers are definately bit fields of some kind, but I'm not exactly sure yet what they mean, I bet I'll probably need to go in later, and massecure some of the variables (makeing the mystery value all "0xFF"s, and see what happens).


Though it appears that all values we are interested in editing right now are not using any unique values. This means that editing coat of arms, should be using the value that you mentioned above for this magic number.
Tell me what you think, Roy.