PDA

View Full Version : Creative Assembly A start on the .MESH file format



Pages : [1] 2 3

GrumpyOldMan
02-14-2007, 07:13
Hi

I've been looking at the .MESH format and although it's not complete, I thought I'd share what I've found and maybe kickstart others to finish it.

I've mainly been looking at figure meshes (siege engine and other meshes are a bit different) and the information relates to these. All values identified as integers are 4 byte integers.

MESH files are stored as follows:-

.at byte 69 of the initial header - integer value for number of different mesh groups = num_group

.at byte 83 - start of first mesh group, integer showing length of mesh type (ie body ,arms, legs,etc) name string

.mesh type name string

.integer showing length of mesh group name(ie legs 01, arms 04, etc) string

.mesh group name string

.2 bytes of dummy data (only in first mesh group)

.integer showing number of triangles in mesh group = n

.n * 3 * 2 byte shorts for triangle values

.29 byte header/footer (only between first and second mesh groups all others are 23 bytes)

repeating below for 1 to num_group-1

.integer - length of mesh type string

.mesh type string

.integer - length of mesh group name string

.mesh group name string

.integer showing number of triangles in mesh group = n

.n * 3 * 2 byte shorts for triangle values

.23 bytes of header/footer

Then you have another 24 byte of header/footer

.integer - number of vertices = n2

.n2 * 2 * 4 byte float - texture coord u and v values (note here the uv values are based on a composite texture ie 2048 x 1024 - the figure texture on the left and the attachment texture on the right)

.28 byte of header/footer

.integer - number of vertices = n2

.n2 * 2 * 4 byte float - vertex weights (primary weight, secondary weight - from left)

.38 bytes of header/footer

.integer - number of vertices = n2

.n2 * 3 * 4 byte float - vertex vector

.36 bytes of header/footer

.integer - number of vertices = n2

.vertex/bone assignments, information in series of four bytes - (from left) byte1 = 0, byte2 = secondary bone, byte3 = primary bone, byte4 = 0

That's as about as far as I've got so far. There is another block of code following the vertex/bone assignments that I haven't worked out yet. It seems to be in series of three bytes followed by a zero byte - maybe something to do with shader systems - any ideas?

After this block is just a listing of the skeleton bones in hierarchy order. Hopefully somebody can use this to go further.

Cheers

GrumpyOldMan

Casuir
02-14-2007, 20:40
Very nice work mate, keep it up if you can. Should be enough here to try some kitbashing

Casuir
02-15-2007, 06:14
.integer showing number of triangles in mesh group = n

.n * 3 * 2 byte shorts for triangle values



Correct me if I'm wrong here but these arent xyz co-ords but references to the vertices the triangle uses? If so what part are the coords given? I've been trying to hand convert some geometry to obj but I'm getting nowhere slowly.

A polycount of a model based on grumpyoldmans finding for those that are interested ( arquebusier)

Head 246
Body 344
Arms 322
Legs 482
Prim 74
Sec 122

Total of 1590 plus an additional 52 in powderbag and 116 in the strap

Btw again good work, after a couple of hours messing about with the info you've provided i can appreciate how much time effort and patience you put into working it out.

KnightErrant
02-16-2007, 17:53
Really good work guys! I'm still trying to get my head around it.:wall:
Is this how it works?

n2 vertex vectors (each entry a 4-byte float)


1.15 3.47 -1.04 [ 0 ]
2.65 2.54 0.98 [ 1 ]
3.77 2.15 1.44 [ 2 ]
. . . [ . ]
. . . [ . ]
. . . [ . ]
0.15 -2.38 3.72 [n2-1 ]
(The numbers in square brackets are meant to
be 0-based indices.)

n mesh group triples (each entry a 2-byte short int)


0 4 5 [ 0 ]
5 4 3 [ 1 ]
4 0 6 [ 2 ]
. . . [ . ]
. . . [ . ]
. . . [ . ]
7 3 2 [ n-1 ]

So each triple in the mesh group block are three
indices into the vertex vector array and this
defines a facet? I suppose the order of indices is
important as that and the right hand rule would define
an outward pointing vector so a rendering system would
know what the "outside" of the body is.

What would the vertex weights be used for?

Casuir
02-16-2007, 21:55
Guy not guys, this is all grumpyoldmans work and thats how it seems to work. Are you converting this by hand grumpyoldman? I'm trying to write something which will automate at least some parts but I've no prior experience at coding so its an uphill struggle.

KnightErrant
02-16-2007, 22:48
@casuir
Sorry, you're right. Point taken.

I've put together a Python script that
goes up to the bones section but I've
only tested it on a fake .mesh file that
I made according to GrumpyOldMan's
prescription. Do you have Python?

Casuir
02-16-2007, 22:50
Yes, what does it export it too? For mesh extraction the vertex vectors, uv co-ords and tris are all thats really neccessary, the rest of it will be needed for putting stuff in though

KnightErrant
02-16-2007, 23:02
Right now it doesn't export to anything
other than writing out the data to an
ASCII file so I can look at it. I'm not at
home so I haven't looked at a real .mesh
file yet. Probably it will be huge.

Edit: Have to take off, I'll check back
in a couple of hours.

Casuir
02-16-2007, 23:27
They're fairly large alright, even the lowest lod of the smallest file has 818 vertices each with at least 5 values that have to be converted plus a similar number of tris. If its outputting as ascii it shouldnt be to hard to convert it to obj, obj format is a text file anyways.
http://www.eg-models.de/formats/Format_Obj.html

Lusted
02-17-2007, 00:24
Well from the pics the guy posted at TWC it looks like each .mesh contains all the different body/head types for a unit which would account of the very large vertices count.

And it looks like he's working on making it open in 3ds max.

http://www.twcenter.net/forums/showpost.php?p=1518014&postcount=2

KnightErrant
02-17-2007, 01:51
I saw those pictures this afternoon. That's pretty impressive
what he's done. I tried my program on the arquebusiers_lod0.mesh.
It made it through all the tris but borked on the first set of texture
coords.:furious3: That was only 6000 lines of ASCII so not as bad as I
thought. I'll try to figure out the problem and then I need to start
deciding on a scheme to store this in memory for later outputting
to a binary.

GrumpyOldMan
02-17-2007, 05:09
Hi Guys
Thanks for the replies.

Casuir

Yes I've written something that takes the mesh, splits it up into component groups and writes it out in Milkshape ascii format. I was using this format to make sure that I was getting the mesh info out correctly. Unfortunately Milkshape doesn't have a facility for vertex weighting :no: . But I was able to check it against a skeleton I extracted from m2tw and the vertex weighting looked pretty accurate. I'll have a look at combining the skeleton and mesh and saving it in a format that supports vertex weighting. And then see if I can convert it back - as anyone that has taken apart a clock will tell you, the dismantling is easy, it's the reassembly that's the killer :laugh4: .

Lusted

Sorry the screenshot is from Milkshape not 3ds max. Unfortunately I've got no knowledge of Maxscript :embarassed: .

Knight Errant

Vertex weights are used for a more realistic movement of vertices. It's where a vertex can be affected by more than one bone, eg think of a vertex in the middle at the bottom of a tunic, either leg could make it go backward or forward. In RTW all vertices were only affected by one bone only.

Incidentally I've managed to extract info from siege engine files now. There's a pic of the war wagon at https://i150.photobucket.com/albums/s104/grumpyoldman2007/war_wagon.png You should note that UV values for siege engines are for a normal texture ie 1024 x 1024.

I'll go back and look at the mystery block of code now and the header/footer arrangement. If I can I'll try to come up with something simple like a Mongol Arquebusier or Pikeman seeing there's lots of space on their attachment texture.

Cheers

GrumpyOldMan

KnightErrant
02-17-2007, 05:27
@GrumpyOldMan
Thank you for the feedback. I'm trying to use your
information to write a Python parser to pull out the facet
information and see if I can image the meshes in MatLab
just to see what's there. I don't have 3dmax so I'm just
doing this for the fun of it.

PS: With your permission, may I just do @GOM in the future
to direct a reply to you.~:)

GrumpyOldMan
02-17-2007, 06:04
Hi KnightErrant

Yeah you can all me @GOM if it saves your typing :laugh4: . Can I call you @KE :laugh4: ?

Milkshape3d is a reasonable program that has a 30 day trial period, and the ascii file format is pretty easy to understand, maybe consider this?

Cheers

GrumpyOldMan

KnightErrant
02-17-2007, 06:25
Thanks. Actually I like KE because the big dudes around here
seem to go by acronyms of their user names. I'll look for
Milkshape3d tomorrow, getting kinda tired right now. I have to
ask, because I'm sure everyone wants to know, are you going to
make a .mesh to 3dmax-whatever converter? The anim modders are
chomping at the bits for something like this. I must confess I
haven't the first clue about these things, I'm just a radar engineer
who likes cool programming puzzles.:thrasher:

Lusted
02-17-2007, 13:16
Lusted

Sorry the screenshot is from Milkshape not 3ds max. Unfortunately I've got no knowledge of Maxscript .

Sorry thought it was max, i know Max is what CA use to edit the .mesh files with their own script, so iom guessing that we might need to use max for editing .mesh files.

KnightErrant
02-17-2007, 22:35
I finally got my program to process down through the bones
section. When I comment out the lines that write out the
data here's what the output file has showing what the sections
look like for arquebusiers_lod0.mesh.


22serialization::archive
3 4 4 4 8 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 4 0 1 3 1 0 0 0 0 0 7 0 1 0 2 0 0 0 0 0 26 0 0 0 0 0 11 0 1 3 3 0 0 0 6 0 0 0 72 101 num_groups = 26
mesh type name: Helmet
mesh group name: hat_01
n = 236
mesh type name: Head
mesh group name: head_01
n = 246
mesh type name: Attachments
mesh group name: strap_01
n = 116
mesh type name: Body
mesh group name: RN_Light_Body_01
n = 344
mesh type name: Arms
mesh group name: arms_light_01
n = 322
mesh type name: Legs
mesh group name: Legs_light_01
n = 482
mesh type name: Head
mesh group name: head_02
n = 246
mesh type name: Body
mesh group name: RN_Light_Body_02
n = 344
mesh type name: Body
mesh group name: RN_Light_Body_03
n = 344
mesh type name: Legs
mesh group name: Legs_light_02
n = 482
mesh type name: Legs
mesh group name: Legs_light_03
n = 482
mesh type name: Legs
mesh group name: Legs_light_04
n = 482
mesh type name: Head
mesh group name: head_03
n = 246
mesh type name: Head
mesh group name: head_04
n = 246
mesh type name: Head
mesh group name: head_05
n = 246
mesh type name: Head
mesh group name: head_06
n = 246
mesh type name: Helmet
mesh group name: hat_02
n = 300
mesh type name: Helmet
mesh group name: hat_03
n = 180
mesh type name: secondaryactive0
mesh group name: ren sword secondary_12
n = 144
mesh type name: secondaryactive1
mesh group name: ren sword secondary_13
n = 144
mesh type name: primaryactive0
mesh group name: arquebus_32
n = 74
mesh type name: primaryactive1
mesh group name: arquebus_33
n = 74
mesh type name: ramrod0
mesh group name: hand ramrod_42
n = 28
mesh type name: equipment20
mesh group name: powderbag_43
n = 52
mesh type name: equipment20
mesh group name: powderbag_44
n = 52
mesh type name: equipment20
mesh group name: powderbag_45
n = 52
number of vertices in texture coordinates: 4949
number of vertices in vertex weights: 4949
number of vertices in vertex vectors: 4949
number of vertices in vertex bones: 4949

@GOM
I have two questions on header/footer sizes.


.at byte 69 of the initial header - integer value for number of different mesh groups = num_group

.at byte 83 - start of first mesh group, integer showing length of mesh type (ie body ,arms, legs,etc) name string

.mesh type name string

.integer showing length of mesh group name(ie legs 01, arms 04, etc) string

.mesh group name string

.2 bytes of dummy data (only in first mesh group)

.integer showing number of triangles in mesh group = n

.n * 3 * 2 byte shorts for triangle values

.29 byte header/footer (only between first and second mesh groups all others are 23 bytes)

repeating below for 1 to num_group-1

.integer - length of mesh type string

.mesh type string

.integer - length of mesh group name string

.mesh group name string

.integer showing number of triangles in mesh group = n

.n * 3 * 2 byte shorts for triangle values

.23 bytes of header/footer

Then you have another 24 byte of header/footer


Shouldn't the red line be a 34 byte header/footer line?

Second question, right before the bones section you have a
36 byte header/footer but I only see four zero bytes there.
Should it be a 40 byte skip because there I see
85 19 0 0 or little endian 4949.
I'm going to look at the headers some more to see if there's any
information in them.

Thank you very, very much for making this info available:bow: . This will
be a fun weekend.:laugh4:

Casuir
02-18-2007, 01:03
There look to be 3 more blocks of data after the vertex-bone assignments part preceded by the no of vertices in the mesh. All 3 seem to be 4 bytes per vertex 3 values and a blank although some have two values and 2 zero bytes.

KnightErrant
02-18-2007, 02:57
I tried looking at them as shorts as well as bytes but couldn't
see anything more as a pattern that way.

Here's whats in the header/footers, seems to just be count
sequences but in a sort of backwards pattern.
4 3 5
6 5 7
8 7 9
etc

I've no idea what that block of 112 bytes is after the mystery
block but before the bone strings start.


mesh type name: Helmet
mesh group name: hat_01
n = 236
0 0 0 0 0 0 0 0 0 10 0 1 0 4 0 0 0 11 0 3 0 0 0 11 0 5 0 0 0
mesh type name: Head
mesh group name: head_01
n = 246
0 0 0 0 0 10 0 6 0 0 0 11 0 5 0 0 0 11 0 7 0 0 0
mesh type name: Attachments
mesh group name: strap_01
n = 116
1 0 0 0 0 10 0 8 0 0 0 11 0 7 0 0 0 11 0 9 0 0 0
mesh type name: Body
mesh group name: RN_Light_Body_01
n = 344
0 0 0 0 0 10 0 10 0 0 0 11 0 9 0 0 0 11 0 11 0 0 0
mesh type name: Arms
mesh group name: arms_light_01
n = 322
0 0 0 0 0 10 0 12 0 0 0 11 0 11 0 0 0 11 0 13 0 0 0
mesh type name: Legs
mesh group name: Legs_light_01
n = 482
0 0 0 0 0 10 0 14 0 0 0 11 0 13 0 0 0 11 0 15 0 0 0
mesh type name: Head
mesh group name: head_02
n = 246
0 0 0 0 0 10 0 16 0 0 0 11 0 15 0 0 0 11 0 17 0 0 0
mesh type name: Body
mesh group name: RN_Light_Body_02
n = 344
0 0 0 0 0 10 0 18 0 0 0 11 0 17 0 0 0 11 0 19 0 0 0
mesh type name: Body
mesh group name: RN_Light_Body_03
n = 344
0 0 0 0 0 10 0 20 0 0 0 11 0 19 0 0 0 11 0 21 0 0 0
mesh type name: Legs
mesh group name: Legs_light_02
n = 482
0 0 0 0 0 10 0 22 0 0 0 11 0 21 0 0 0 11 0 23 0 0 0
mesh type name: Legs
mesh group name: Legs_light_03
n = 482
0 0 0 0 0 10 0 24 0 0 0 11 0 23 0 0 0 11 0 25 0 0 0
mesh type name: Legs
mesh group name: Legs_light_04
n = 482
0 0 0 0 0 10 0 26 0 0 0 11 0 25 0 0 0 11 0 27 0 0 0
mesh type name: Head
mesh group name: head_03
n = 246
0 0 0 0 0 10 0 28 0 0 0 11 0 27 0 0 0 11 0 29 0 0 0
mesh type name: Head
mesh group name: head_04
n = 246
0 0 0 0 0 10 0 30 0 0 0 11 0 29 0 0 0 11 0 31 0 0 0
mesh type name: Head
mesh group name: head_05
n = 246
0 0 0 0 0 10 0 32 0 0 0 11 0 31 0 0 0 11 0 33 0 0 0
mesh type name: Head
mesh group name: head_06
n = 246
0 0 0 0 0 10 0 34 0 0 0 11 0 33 0 0 0 11 0 35 0 0 0
mesh type name: Helmet
mesh group name: hat_02
n = 300
0 0 0 0 0 10 0 36 0 0 0 11 0 35 0 0 0 11 0 37 0 0 0
mesh type name: Helmet
mesh group name: hat_03
n = 180
0 0 0 0 0 10 0 38 0 0 0 11 0 37 0 0 0 11 0 39 0 0 0
mesh type name: secondaryactive0
mesh group name: ren sword secondary_12
n = 144
0 0 0 0 0 10 0 40 0 0 0 11 0 39 0 0 0 11 0 41 0 0 0
mesh type name: secondaryactive1
mesh group name: ren sword secondary_13
n = 144
0 0 0 0 0 10 0 42 0 0 0 11 0 41 0 0 0 11 0 43 0 0 0
mesh type name: primaryactive0
mesh group name: arquebus_32
n = 74
0 0 0 0 0 10 0 44 0 0 0 11 0 43 0 0 0 11 0 45 0 0 0
mesh type name: primaryactive1
mesh group name: arquebus_33
n = 74
0 0 0 0 0 10 0 46 0 0 0 11 0 45 0 0 0 11 0 47 0 0 0
mesh type name: ramrod0
mesh group name: hand ramrod_42
n = 28
0 0 0 0 0 10 0 48 0 0 0 11 0 47 0 0 0 11 0 49 0 0 0
mesh type name: equipment20
mesh group name: powderbag_43
n = 52
0 0 0 0 0 10 0 50 0 0 0 11 0 49 0 0 0 11 0 51 0 0 0
mesh type name: equipment20
mesh group name: powderbag_44
n = 52
0 0 0 0 0 10 0 52 0 0 0 11 0 51 0 0 0 11 0 53 0 0 0
mesh type name: equipment20
mesh group name: powderbag_45
n = 52
0 0 0 0 0 10 0 54 0 0 0 11 0 53 0 0 0 6 0 1 0 55 0
number of vertices in texture coordinates: 4949
0 0 0 0 17 0 1 0 57 0 0 0 18 0 56 0 0 0 18 0 58 0 0 0 1 0 0 0
number of vertices in vertex weights: 4949
0 0 0 0 17 0 59 0 0 0 18 0 58 0 0 0 0 0 1 0 0 0 0 0 23 0 1 0 60 0 0 0 0 0 0 0 0 0
number of vertices in vertex vectors: 4949
0 0 0 0 22 0 1 0 61 0 0 0 23 0 60 0 0 0 0 0 4 0 0 0 0 0 28 0 1 0 62 0 0 0 2 0 0 0 0 0
number of vertices in vertex bones: 4949
0 0 0 0 27 0 1 0 63 0 0 0 28 0 62 0 0 0 28 0 64 0 0 0 3 0 0 0
number of vertices in mystery block: 4949
104 211 220 0 138 248 166 0 116 251 100 0 146 251 104 0 116 253 139 0 117 251 155 0 132 253 112 0
149 42 34 0 212 35 104 0 157 250 142 0 24 196 97 0 57 42 192 0 150 35 212 0 155 250 145 0
50 196 53 0 0 0 0 0 27 0 69 0 0 0 28 0 68 0 0 0 0 0 0 0 0 0 0 0
0 0 163 234 138 61 211 32 148 62 220 16 18 63 237 205 184 63 0 0 0 0 26 0 0 0 0 0
11 bone_pelvis 0
11 bone_rthigh 1
14 bone_rlowerleg 2
10 bone_rfoot 3
8 bone_abs 4
10 bone_torso 5
9 bone_head 6
8 bone_jaw 7
12 bone_eyebrow 8
14 bone_rclavical 9
14 bone_rupperarm 10
11 bone_relbow 11
10 bone_rhand 12
14 bone_lclavical 13
14 bone_lupperarm 14
11 bone_lelbow 15
10 bone_lhand 16
11 bone_lthigh 17
14 bone_llowerleg 18
10 bone_lfoot 19
13 bone_weapon01 20
11 bone_weapon 20
13 bone_weapon02 21
13 bone_weapon03 22
13 bone_shield01 22
11 bone_shield 22

Casuir
02-18-2007, 03:38
What I meant was the mystery block is actually 3 seperate blocks with similar structure seperated by header/footers

KnightErrant
02-18-2007, 04:06
Oh! Thank you, I totally missed that. I was wondering why there
seemed to be left over stuff from the mystery block. I mean I'm
seeing extra # # # 0 byte blocks afterwards so I must be missing
the header/footers.

KnightErrant
02-18-2007, 23:56
After some searching on the web re 3D formats, could the
mystery block be smoothing groups for the vertices? Some
software uses 24 bits for smoothing groups but 3dsmax uses
32 bits so maybe that's why the fourth byte is 0, its to be
compatible with other software.

Edit: Sorry GOM, reread your first post and you already suggested
this.

Casuir
02-19-2007, 06:37
I've looked at a few other mesh files and the block after the vertex weighting (mystery one) sometimes has a value other than 00 in the fourth byte nor does it follow this sequence right through the block, sometimes theres whole strings of 00s or 00 alternating with another value. Same goes for the other two blocks after it.

Look at as_light_fancy_cuman_archer_lod0.mesh or crossbow_militia_ug2_lod0.mesh and you'll see what I mean

KnightErrant
02-20-2007, 04:29
I'm getting thin on ideas. Here's what I've tried this weekend.
Downloaded milkshape 3D as per GOM suggestion. No info in the
help or in its forums on the ASCII format so I tried the quick
tutorial and exported it to Milkshape 3D ASCII format but the simple
format doesn't really look like the blocks we see in the mesh file.
(Meshes seem to be more than triangles.)

Other tries: the .max format for 3dsmax isn't really a file format, its
more of a capture of the executable at the moment the user saves it
and it isn't really documented in anything I've seen. Drop back and
punt, maybe the old Autodesk format .3ds which is a binary format
which is simple chunking for the blocks we've seen in the .mesh
files. This can be loaded in Milkshape 3D as well as 3dsmax. Here's
a printable link to its format specification:

http://www.martinreddy.net/gfx/3d/3DS.spec

Maybe try this, write a program to extract the blocks from a mesh,
see if Milkshape 3D can read it (GOM already said Milkshape 3D can't
handle vertex weighting so leave this out). If so, use the spec to write
a .3ds file and then ask if somebody with 3dsmax can read it?

My head hurts when the walls are hard.:wall: Did I say that this would
be fun a couple of posts back?:surrender:

Casuir
02-20-2007, 13:44
If ms3d is giving you headaches you could try obj, another asci format thats readable by nearly all 3d progs. Your program would have to output a file structured like this:
First vertex vectors, pull from the 3 floats in the . mesh files and write as follows: one vertex per line, 3 values per vertex seperated by a space and preceded by a lower case v (vertex) , ie:

v 1.0 2.6 3.5
v 3.2 2.3 0.91


next would be u.v. co-ords, same deal sept 2 values, preceded by vt (vertex texture):

vt 0.34 0.67
vt 0.23 0.69

next would be the meshs, pull the mesh name and the 3 integers per tri and write as follows: first line would be g (group) followed by the name of the mesh, next lines would be f (face) and the values for the three vertices in the group and their texture co-ords, v and vt seperated by a / i.e:

g head_01
f 324/324 367/367 356/356
f 356/356 366/366 344/344

first value is the v, second is the vt, they should be the same as each other. Your finished .obj file should look something like this:

v 1.0 2.6 3.5
v 3.2 2.3 0.91
vt 0.34 0.67
vt 0.23 0.69
g head_01
f 324/324 367/367 356/356
f 356/356 366/366 344/344
g head_02
f 533/533 567/567 556/556
f 556/556 566/566 544/544

except much bigger. That should give you an exported model with texture co-ords and seperate meshs for each bodypart.

KnightErrant
02-20-2007, 15:28
Thanks Casuir! I think I was too focused on other things to notice
the thread you posted on post 9. I'll try coding this tonight and see
if Milkshape 3D imports it.

KnightErrant
02-20-2007, 23:24
Processed an arquebusiers_lod1.mesh file. Turns out
the lower res mesh only has 1 block of data in the mystery
block, not 3.

@GOM
Does this information help in determining what the mystery
data is?

GrumpyOldMan
02-21-2007, 01:17
Hi Guys

Still here. Main problem at the moment is I'm having mystery reboots so it means taking my pc apart and trying to find what's wrong. So far eliminated software, dust and power supply as problems, now to look at video card or motherboard (shudder).

Haven't really progressed much myself, but it's really great to see you guys getting involved too.

Milkshape - There is an SDK available for download that includes file formats, it's at http://chumbalum.swissquake.ch/files/ms3dsdk177.zip. Also looking at their forum the next release of Milkshape will have native vertex weight support http://www.chumba.ch/chumbalum-soft/forum/showthread.php?t=18854. The only problem I can see is internally storing the information within Milkshape to get all the information required for Mesh files.

I'm going to concentrate on the header/footers now because if we can understand those then we can at least do cutting and pasting to come up with new figures if the mystery block numbers can be just transported with the mesh bits.

Thoughts on the mystery block -

shadow system data? - why have three separate entries for hi-res figures?
normal data? - why not preprocessed and again three for hi-res figures?

Sorry not much there. Still looking. GOSH A ROAD MAP OF THE FORMAT WOULD BE HANDY AT THIS POINT (Do you think anybody from CA is reading this?). Don't despair if you don't hear from me for a while - might mean I've lived up to my name and put my boot through the pc :furious3:

Cheers

GrumpyOldMan

KnightErrant
02-21-2007, 01:53
Nice to hear from you again. Bummer about your PC:embarassed:.
Casuir got me pointed to the obj format so I'm trying that right now.
There's a guy over at TW center doing a tutorial on 3D graphics like
normal maps and shading. I will try to pester him re shading sets to see
if we can get info that way. I've looked briefly at the wrl format which
Milkshape can export but not import but 3dsmax can read in. Its sort of
a meta 3D graphics format in that it allows all sorts of other things like
jpegs to be input into its "world". If it allows vertex weighting and shading
sets, if thats what they are, it might be a viable target format. It's also
ASCII so simpler that way.

Best of luck with the computer.

Wikiman
02-21-2007, 02:12
I am blown away at how promising this work is. Keep up the good work!!!

-wikiman

KnightErrant
02-21-2007, 02:20
Hello wikiman, welcome. Thank you for your interest.
I think I echo GrumpyOldMan's heartfelt wish, any chance
of a roadmap for the headers/footers for the mesh format
or is this outside your purvue? Just being opportunistic
here, no pressure. Again, a pleasure to see you.

Caliban
02-21-2007, 02:22
Sorry not much there. Still looking. GOSH A ROAD MAP OF THE FORMAT WOULD BE HANDY AT THIS POINT (Do you think anybody from CA is reading this?). Don't despair if you don't hear from me for a while - might mean I've lived up to my name and put my boot through the pc :furious3:
GrumpyOldMan

I'm pretty sure some are :2thumbsup:

One of our programmers sent me this a few mins ago, hope its of some help.
Please feel free to ask another other questions. If you need more info on the mystery block, pm me with details.

// [item] means there is an array of items, usually saved as count followed by items

mesh file =
{
[Meshes],
[Materials],
BoundingSphere
}

Meshes =
{
[MeshVariation],
NumVertices,
[float2 vertex streams],
[float3 vertex streams],
[byte4 vertex streams],
[bool vertex streams], (unused for characters)
BoundingSphere,
BoneMap
}

Materials =
{
Name,
[TextureNames],
ZCompare,
ZWrite,
Culling,
AlphaBlendSettings,
[ColorParameters],
[FloatParameters],
TextureMatrix[2] (unused for characters)
}

MeshVariation =
{
GroupName,
MeshName,
[Triangles],
bool Optional,
[stream of bools] (unused for characters)
}

Triangle =
{
int16 index0, index1, index 2
}

BoneMap =
{
[Bone],
}

Bone =
{
Name,
Index
}

vertex stream
{
Semantic,
[Elements],
Name
}

KnightErrant
02-21-2007, 02:27
Caliban, a pleasure to see you as well! Thank you for the information.
We'll try to digest this and I'm sure we'll have some questions once we
do. Any help is greatly appreciated at this point.~:)

KnightErrant
02-21-2007, 02:58
@Casuir
Well, no luck with the obj format. It keeps crashing Milkshape 3D.
Tried a reboot and it looks like I'm clobbering one of its DLLs, probably
a buffer overrun. I'm not missing a header or anything am I? I did
vertices first:
v float float float
...

then uv coords

vt float float
...

finally triangles

g hat_01
f #/# #/# #/#
...

with the numbers the same on both sides of the slash and including
all the groups. I didn't use any other information to separate things
and of course I didn't try including anything else than this basic information.

I'm going to pursue GOM's thread for file formats. He probably used
Milkshape's native ASCII format and it worked for him. (Of course this one
gave me fits because I couldn't find a spec for it.) If I can find a format
spec I'll try that.

GrumpyOldMan
02-21-2007, 04:38
Hi All

Whoo-hoo, great responses from the CA team, I've pm'd Caliban with details of the mystery blocks so I'll go back and look at the header/footers.

Double Whoo-hoo, 45 mins without a reboot :laugh4: .

@KE

The main thing wrong with the OBJ format is vertex bone assignment and weighting. If you have a look at the Milkshape SDK it has file formats included. It may not be much but I can try to hang a GUI on the software I've written - it at least breaks up the MESH into component parts but with no vertex assignments.

Cheers

GrumpyOldMan

Casuir
02-21-2007, 08:09
That looks ok KE, can you send me the exported obj file?

KnightErrant
02-21-2007, 15:50
@Casuir
Hmmm, I tried but I can't find an attach file button when I hit the PM button
on your post. Could you tell me how to do this?

@GOM
Great news. You wished for a CA dev and you got two!
Can you wish for bags of gold next, please?

Edit: Actually, if Caliban would answer another question,
could you ask what format they use to export out their models
from 3dsmax. Presumably they have some outside program
that converts it to a mesh. Or they may have written their
own plugin to do it it which case the question is moot. If the
question makes sense then that is probably the format to target.

KnightErrant
02-21-2007, 16:43
Quick observation on headers/footers. I turned off data output
in my script and just output the strings and footers for arquebusiers_lod0
and levy_spearmen_ug1_lod0. Here's the output, arquebusiers first:



22serialization::archive
3 4 4 4 8 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 4 0 1 3 1 0 0 0 0 0 7 0 1 0 2 0 0 0 0 0
num_groups = 26
0 0 11 0 1 3 3 0 0 0
mesh type name: Helmet
mesh group name: hat_01
number of triangles, n = 236
0 0 0 0 0 0 0 0 0 10 0 1 0 4 0 0 0 11 0 3 0 0 0 11 0 5 0 0 0
mesh type name: Head
g head_01
0 0 0 0 0 10 0 6 0 0 0 11 0 5 0 0 0 11 0 7 0 0 0
mesh type name: Attachments
g strap_01
1 0 0 0 0 10 0 8 0 0 0 11 0 7 0 0 0 11 0 9 0 0 0mesh type name: Body
g RN_Light_Body_01
0 0 0 0 0 10 0 10 0 0 0 11 0 9 0 0 0 11 0 11 0 0 0
mesh type name: Arms
g arms_light_01
0 0 0 0 0 10 0 12 0 0 0 11 0 11 0 0 0 11 0 13 0 0 0
mesh type name: Legs
g Legs_light_01
0 0 0 0 0 10 0 14 0 0 0 11 0 13 0 0 0 11 0 15 0 0 0
mesh type name: Head
g head_02
0 0 0 0 0 10 0 16 0 0 0 11 0 15 0 0 0 11 0 17 0 0 0
mesh type name: Body
g RN_Light_Body_02
0 0 0 0 0 10 0 18 0 0 0 11 0 17 0 0 0 11 0 19 0 0 0
mesh type name: Body
g RN_Light_Body_03
0 0 0 0 0 10 0 20 0 0 0 11 0 19 0 0 0 11 0 21 0 0 0
mesh type name: Legs
g Legs_light_02
0 0 0 0 0 10 0 22 0 0 0 11 0 21 0 0 0 11 0 23 0 0 0
mesh type name: Legs
g Legs_light_03
0 0 0 0 0 10 0 24 0 0 0 11 0 23 0 0 0 11 0 25 0 0 0
mesh type name: Legs
g Legs_light_04
0 0 0 0 0 10 0 26 0 0 0 11 0 25 0 0 0 11 0 27 0 0 0
mesh type name: Head
g head_03
0 0 0 0 0 10 0 28 0 0 0 11 0 27 0 0 0 11 0 29 0 0 0
mesh type name: Head
g head_04
0 0 0 0 0 10 0 30 0 0 0 11 0 29 0 0 0 11 0 31 0 0 0
mesh type name: Head
g head_05
0 0 0 0 0 10 0 32 0 0 0 11 0 31 0 0 0 11 0 33 0 0 0
mesh type name: Head
g head_06
0 0 0 0 0 10 0 34 0 0 0 11 0 33 0 0 0 11 0 35 0 0 0
mesh type name: Helmet
g hat_02
0 0 0 0 0 10 0 36 0 0 0 11 0 35 0 0 0 11 0 37 0 0 0
mesh type name: Helmet
g hat_03
0 0 0 0 0 10 0 38 0 0 0 11 0 37 0 0 0 11 0 39 0 0 0
mesh type name: secondaryactive0
g ren sword secondary_12
0 0 0 0 0 10 0 40 0 0 0 11 0 39 0 0 0 11 0 41 0 0 0
mesh type name: secondaryactive1
g ren sword secondary_13
0 0 0 0 0 10 0 42 0 0 0 11 0 41 0 0 0 11 0 43 0 0 0
mesh type name: primaryactive0
g arquebus_32
0 0 0 0 0 10 0 44 0 0 0 11 0 43 0 0 0 11 0 45 0 0 0
mesh type name: primaryactive1
g arquebus_33
0 0 0 0 0 10 0 46 0 0 0 11 0 45 0 0 0 11 0 47 0 0 0
mesh type name: ramrod0
g hand ramrod_42
0 0 0 0 0 10 0 48 0 0 0 11 0 47 0 0 0 11 0 49 0 0 0
mesh type name: equipment20
g powderbag_43
0 0 0 0 0 10 0 50 0 0 0 11 0 49 0 0 0 11 0 51 0 0 0
mesh type name: equipment20
g powderbag_44
0 0 0 0 0 10 0 52 0 0 0 11 0 51 0 0 0 11 0 53 0 0 0
mesh type name: equipment20
g powderbag_45
0 0 0 0 0 10 0 54 0 0 0 11 0 53 0 0 0 6 0 1 0 55 0
0 0 7 0 2 0 0 0 85 19 0 0 0 0 2 0 0 0 0 0 18 0 1 0 56 0 0 0 4 0 0 0 0 0 0 0 0 0 17 0 1 0 57 0 0 0 18 0 56 0 0 0 18 0 58 0 0 0 1 0 0 0
number of vertices in vertex weights: 4949
0 0 0 0 17 0 59 0 0 0 18 0 58 0 0 0 0 0 1 0 0 0 0 0 23 0 1 0 60 0 0 0 0 0 0 0 0 0
number of vertices in vertex vectors: 4949
0 0 0 0 22 0 1 0 61 0 0 0 23 0 60 0 0 0 0 0 4 0 0 0 0 0 28 0 1 0 62 0 0 0 2 0 0 0 0 0
number of vertices in vertex bones: 4949
0 0 0 0 27 0 1 0 63 0 0 0 28 0 62 0 0 0 28 0 64 0 0 0 3 0 0 0
number of vertices in mystery block: 4949
0 0 0 0 27 0 65 0 0 0 28 0 64 0 0 0 28 0 66 0 0 0 10 0 0 0
number of vertices in mystery block: 4949
0 0 0 0 27 0 67 0 0 0 28 0 66 0 0 0 28 0 68 0 0 0 11 0 0 0
number of vertices in mystery block: 4949
0 0 0 0 27 0 69 0 0 0 28 0 68 0 0 0 0 0 0 0 0 0 0 0 0 0
0.0678303465247 0.289312928915 0.570569753647 1.44378435612
0 0 0 0 26 0 0 0 0 0
Processing bone strings
11 bone_pelvis 0
11 bone_rthigh 1
14 bone_rlowerleg 2
10 bone_rfoot 3
8 bone_abs 4
10 bone_torso 5
9 bone_head 6
8 bone_jaw 7
12 bone_eyebrow 8
14 bone_rclavical 9
14 bone_rupperarm 10
11 bone_relbow 11
10 bone_rhand 12
14 bone_lclavical 13
14 bone_lupperarm 14
11 bone_lelbow 15
10 bone_lhand 16
11 bone_lthigh 17
14 bone_llowerleg 18
10 bone_lfoot 19
13 bone_weapon01 20
11 bone_weapon 20
13 bone_weapon02 21
13 bone_weapon03 22
13 bone_shield01 22
11 bone_shield 22
3 0 1 0 70 0 0 0 4 0 1 0 0 0 0 0 1 0 0 0 0 0 39 0 1 4 71 0 0 0
number of chars 13
characterlod0
17 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 1 0 0 0 3 0 0 0
101 185 63 0 4 0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 9 0 0 0
0 0 0 64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 8 0 0 0 2 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 2 0 0 0 0 0
128 63 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0
0 0 2 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
128 63 0 0 0 0 0 0 0 0 2 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 2 0 0 0 0 0 128 63 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 2 0
0 0 0 0 128 63 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0
0 0 0 0 0 0 2 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 128 63 0 0 0 0 0 0 0 0 2 0 0 0 0 0 128 63 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 38 0 1 0 72 0
0 0 39 0 71 0 0 0
0.0678303465247 0.289312928915 0.570569753647 1.44378435612tell says we are at 257826
bytecount says we are at 257826
Bytecount is now 257826

Now levy_spearmen_ug1:



22serialization::archive
3 4 4 4 8 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 4 0 1 3 1 0 0 0 0 0 7 0 1 0 2 0 0 0 0 0
num_groups = 25
0 0 11 0 1 3 3 0 0 0
mesh type name: Hands
mesh group name: hands_01
number of triangles, n = 202
0 0 0 0 0 0 0 0 0 10 0 1 0 4 0 0 0 11 0 3 0 0 0 11 0 5 0 0 0
mesh type name: Head
g head_02
0 0 0 0 0 10 0 6 0 0 0 11 0 5 0 0 0 11 0 7 0 0 0
mesh type name: Head
g head_05
0 0 0 0 0 10 0 8 0 0 0 11 0 7 0 0 0 11 0 9 0 0 0
mesh type name: Head
g head_04
0 0 0 0 0 10 0 10 0 0 0 11 0 9 0 0 0 11 0 11 0 0 0
mesh type name: Legs
g Legs_01
0 0 0 0 0 10 0 12 0 0 0 11 0 11 0 0 0 11 0 13 0 0 0
mesh type name: Head
g head_01
0 0 0 0 0 10 0 14 0 0 0 11 0 13 0 0 0 11 0 15 0 0 0
mesh type name: Arms
g Arms_01
0 0 0 0 0 10 0 16 0 0 0 11 0 15 0 0 0 11 0 17 0 0 0
mesh type name: Arms
g Arms_02
0 0 0 0 0 10 0 18 0 0 0 11 0 17 0 0 0 11 0 19 0 0 0
mesh type name: Arms
g Arms_03
0 0 0 0 0 10 0 20 0 0 0 11 0 19 0 0 0 11 0 21 0 0 0
mesh type name: Arms
g Arms_04
0 0 0 0 0 10 0 22 0 0 0 11 0 21 0 0 0 11 0 23 0 0 0
mesh type name: Legs
g Legs_02
0 0 0 0 0 10 0 24 0 0 0 11 0 23 0 0 0 11 0 25 0 0 0
mesh type name: Legs
g Legs_03
0 0 0 0 0 10 0 26 0 0 0 11 0 25 0 0 0 11 0 27 0 0 0
mesh type name: Legs
g Legs_04
0 0 0 0 0 10 0 28 0 0 0 11 0 27 0 0 0 11 0 29 0 0 0
mesh type name: Hands
g hands_02
0 0 0 0 0 10 0 30 0 0 0 11 0 29 0 0 0 11 0 31 0 0 0
mesh type name: Head
g head_06
0 0 0 0 0 10 0 32 0 0 0 11 0 31 0 0 0 11 0 33 0 0 0
mesh type name: Attachments3
g teeth
0 0 0 0 0 10 0 34 0 0 0 11 0 33 0 0 0 11 0 35 0 0 0
mesh type name: Body
g Padded_body_01
0 0 0 0 0 10 0 36 0 0 0 11 0 35 0 0 0 11 0 37 0 0 0
mesh type name: Body
g Padded_body_02
0 0 0 0 0 10 0 38 0 0 0 11 0 37 0 0 0 11 0 39 0 0 0
mesh type name: Body
g Padded_body_03
0 0 0 0 0 10 0 40 0 0 0 11 0 39 0 0 0 11 0 41 0 0 0
mesh type name: primaryactive0
g long spear_40
0 0 0 0 0 10 0 42 0 0 0 11 0 41 0 0 0 11 0 43 0 0 0
mesh type name: primaryactive1
g long spear_41
0 0 0 0 0 10 0 44 0 0 0 11 0 43 0 0 0 11 0 45 0 0 0
mesh type name: shield0
g kite simple_64
0 0 0 0 0 10 0 46 0 0 0 11 0 45 0 0 0 11 0 47 0 0 0
mesh type name: shield0
g kite simple_65
0 0 0 0 0 10 0 48 0 0 0 11 0 47 0 0 0 11 0 49 0 0 0
mesh type name: shield0
g kite simple_66
0 0 0 0 0 10 0 50 0 0 0 11 0 49 0 0 0 11 0 51 0 0 0
mesh type name: shield0
g kite simple_67
0 0 0 0 0 10 0 52 0 0 0 11 0 51 0 0 0 6 0 1 0 53 0
0 0 7 0 2 0 0 0 44 15 0 0 0 0 2 0 0 0 0 0 18 0 1 0 54 0 0 0 4 0 0 0 0 0 0 0 0 0 17 0 1 0 55 0 0 0 18 0 54 0 0 0 18 0 56 0 0 0 1 0 0 0
number of vertices in vertex weights: 3884
0 0 0 0 17 0 57 0 0 0 18 0 56 0 0 0 0 0 1 0 0 0 0 0 23 0 1 0 58 0 0 0 0 0 0 0 0 0
number of vertices in vertex vectors: 3884
0 0 0 0 22 0 1 0 59 0 0 0 23 0 58 0 0 0 0 0 4 0 0 0 0 0 28 0 1 0 60 0 0 0 2 0 0 0 0 0
number of vertices in vertex bones: 3884
0 0 0 0 27 0 1 0 61 0 0 0 28 0 60 0 0 0 28 0 62 0 0 0 3 0 0 0
number of vertices in mystery block: 3884
0 0 0 0 27 0 63 0 0 0 28 0 62 0 0 0 28 0 64 0 0 0 10 0 0 0
number of vertices in mystery block: 3884
0 0 0 0 27 0 65 0 0 0 28 0 64 0 0 0 28 0 66 0 0 0 11 0 0 0
number of vertices in mystery block: 3884
0 0 0 0 27 0 67 0 0 0 28 0 66 0 0 0 0 0 0 0 0 0 0 0 0 0
0.470658034086 0.434560090303 0.114010438323 1.69914269447
0 0 0 0 26 0 0 0 0 0
Processing bone strings
11 bone_pelvis 0
11 bone_rthigh 1
14 bone_rlowerleg 2
10 bone_rfoot 3
8 bone_abs 4
10 bone_torso 5
9 bone_head 6
8 bone_jaw 7
12 bone_eyebrow 8
14 bone_rclavical 9
14 bone_rupperarm 10
11 bone_relbow 11
10 bone_rhand 12
14 bone_lclavical 13
14 bone_lupperarm 14
11 bone_lelbow 15
10 bone_lhand 16
11 bone_lthigh 17
14 bone_llowerleg 18
10 bone_lfoot 19
13 bone_weapon01 20
11 bone_weapon 20
13 bone_weapon02 21
13 bone_weapon03 22
13 bone_shield01 22
11 bone_shield 22
3 0 1 0 68 0 0 0 4 0 1 0 0 0 0 0 1 0 0 0 0 0 39 0 1 4 69 0 0 0
number of chars 13
characterlod0
17 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 1 0 0 0 3 0 0 0
101 185 63 0 4 0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 9 0 0 0
0 0 0 64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 8 0 0 0 2 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 2 0 0 0 0 0
128 63 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0
0 0 2 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
128 63 0 0 0 0 0 0 0 0 2 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 2 0 0 0 0 0 128 63 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 2 0
0 0 0 0 128 63 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0
0 0 0 0 0 0 2 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 128 63 0 0 0 0 0 0 0 0 2 0 0 0 0 0 128 63 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 38 0 1 0 70 0
0 0 39 0 69 0 0 0
0.470658034086 0.434560090303 0.114010438323 1.69914269447
tell says we are at 204563
bytecount says we are at 204563
Bytecount is now 204563

I've put in red some interesting differences. The arquebusiers have ONE
header/footer with a 1 0 0 0 instead of 0 0 0 0 starting the line. Very
strange. After the powderbag entry the arquebusiers have a 85 19 0 0
in the third group of 4 bytes, this is 4949 the count of vertices. The
levy_spearmen_ug1 have 44 15 0 0 which is 3884 or their number of vertices.
This is understandable. Finally each has a set of four floats before the bones
and at the very end of the file, the last 16 bytes. The numbers are the
same in both places but different between the units.

Oh, and the arquebusiers have one more triangle group than the levy_spearmen_ug1 so their sequence counts go up 2 higher.
These are the similarities and differences I can spot right now.
These are both infantry types, I'll try a cavalry unit at lunch to
see if there are more differences there.

Casuir
02-21-2007, 17:42
I've pm'd you with my email address, dont think these forums allow attachments. The format they used to export was .gr2:
http://www.radgametools.com/granny/faq.html
Looking at the files calibans provided and ones that seem to be leftovers in the packfiles I'm guessing all models are exported seperatly as gr2s from max the attachment sets are compiled by another program and then the unit mesh is compiled via xmls like the one caliban provided us with. gr2's a licensed format hence the problems with giving us a model exporter.

KnightErrant
02-21-2007, 18:45
@Casuir
I've e-mailed you the obj file. Thanks for the explanation
on gr2s, this is slowly making sense now.

Casuir
02-21-2007, 19:17
Got it cheers, opens up in 3dexplorer ok but the vertex data is messed up:
https://img235.imageshack.us/img235/8724/objnl8.th.png (https://img235.imageshack.us/my.php?image=objnl8.png)
My guess would be python might be having trouble converting the floats? If you look in the file theres some odd values like: -8.65171934095e-010
Hard to tell if u.v. data is right but it looks like it might be ok, texture needs to be flipped vertically though.

KnightErrant
02-21-2007, 20:07
Sent you another one. Did an if statement on any
vertex point being less than 1.0e-06. If so, made it
0.0. This got rid of all the small floats in scientific
notation. Didn't know what to do about the uv data,
I don't know a Python flip up down command to
reverse it.

Casuir
02-21-2007, 21:19
Fraid its no better. Think I may have been hasty in blaming python, the original file looks fine when viewed as just vertexs, problem seems to be the faces. The first mesh to be extracted says it has 2 two sided polys and 1 one sided, think this may be of this part:
f 46/46 5/5 0/0
f 0/0 41/41 46/46
f 45/45 4/4 3/3
f 3/3 44/44 45/45
f 44/44 3/3 2/2
f 2/2 43/43 44/44
f 43/43 2/2 170/170
f 170/170 169/169 43/43
f 42/42 1/1 6/6
f 6/6 48/48 42/42
f 49/49 7/7 5/5
f 5/5 47/47 49/49
f 41/41 0/0 4/4
Seems the vertexs are numbered in the mesh file 0-4948 while in the obj they're 1-4949, adding 1 to each value in the mesh part should (hopefully) solve the problem. I think you could take out the if statement, doesnt seem to have any noticeable difference on placement but the files missing a vertex and face.

KnightErrant
02-21-2007, 21:25
Ok, this shouldn't take too long.

Casuir
02-21-2007, 22:40
https://img178.imageshack.us/img178/5018/obj2qm0.th.png (https://img178.imageshack.us/my.php?image=obj2qm0.png)
Nice work. :2thumbsup:
Dunno what you'd do about the texture co-ords, the texture wont fit even if you flip the vertical as the co-ords are given for the attachment and unit texture as one. Not really much trouble to flip the tex when you have to resize it anyways.

KnightErrant
02-21-2007, 23:16
Very nice Casuir! A productive day all around.~:)

Caliban
02-22-2007, 00:44
The format they used to export was .gr2:..
I'm guessing all models are exported seperatly as gr2s from max the attachment sets are compiled by another program and then the unit mesh is compiled via xmls like the one caliban provided us with. gr2's a licensed format hence the problems with giving us a model exporter.

Pretty much correct. The models and variations are marked up in 3dsmax along with the material type (unit lod0, unit lod1, unit lod2) and exported out to .gr2 format. We then setup the unit xml files which contain things like the attachment sets, skeletons to be used and convert it via an external conversion proccess.

These images should give you an idea of how it looks in 3dsmax:

https://img151.imageshack.us/my.php?image=3dsmax1zj0.jpg

https://img151.imageshack.us/my.php?image=3dsmax2nm5.jpg

https://img523.imageshack.us/my.php?i...dsmax2asm0.jpg

https://img523.imageshack.us/my.php?i...ertypesth6.jpg

The Mystery Blocks

A reply to Grumpy's question about the mystery blocks:

I read his file decription again and it certainly seems it is the unused bool stream. Just before them there should be number 1 somewhere (number of these streams) followed by number of vertices (this number is the same for all streams) followed by bool values themselves. He should be able to just change that 1 to 0 and then remove count + bool values completely. But I'd have to see the file to be sure, I don't really remember.

I'll have a definate answer on this soon after a bit of investigation.

KnightErrant
02-22-2007, 01:55
Thank you Caliban. Here is a snippet containing the header/footer
bytes (byte values in decimal) just before reading the first block within
the mystery block.


0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0 27 0 1 0 63 0 0 0 28 0 62 0 0 0 28 0 64 0 0 0 3 0 0 0
number of vertices in mystery block: 4949
79 25 186 0
181 17 162 0
159 18 185 0
130 20 196 0
105 23 197 0
54 25 153 0


The zeros before the header/footer are the last of the vertex bones group.
I've marked in red the possible unit flag value. This looks like what your
source is describing.

KnightErrant
02-22-2007, 02:29
@GOM and Casuir
I've got Milkshape 3D working on the arquebusier_lod0
in the obj format. It must have been either the 0-based
indexing or not passing the triangles as #/# #/# #/#
that killed me before. The file I sent Casuir reads just
fine.

GrumpyOldMan
02-22-2007, 03:21
Hi Guys

Things really seem to be rolling along now. Many thanks to Caliban for chasing up those details for us.

@KE

One thing I managed to find was the "1" in the belt header for the arquebusier. I've looked through a number of meshes and it seems to indicate with Attachment type groups that there is only one of that type and it is compulsory.

Also with the mystery group/s I think the four byte "3" just before the number of vertices may be more significant - remember there are three streams in the high lod files. The lower lod files have only one stream but still have a "3" in this spot - maybe same stream used three times?

The four float values repeated near the end of the files, looking at one of the replies from Caliban,


Meshes =
{
[MeshVariation],
NumVertices,
[float2 vertex streams],
[float3 vertex streams],
[byte4 vertex streams],
[bool vertex streams], (unused for characters)
BoundingSphere,
BoneMap
}

indicates that there are bounding sphere values but I haven't got it straight in my head about these values yet.

Currently I'm working on a program to convert Meshes to MS3D binary files so that the vertex weighting can be retained. One disadvantage of this is that Milkshape stores weights as integer percentages so I have to covert a Mesh weight value of .674352 to 67%.

There are a number of options at this point -

1. Go with a Milkshape plugin or utility

2. Go with a 3DS Max plugin or utility (I have no knowledge of Maxscript) :wall:

3. Go for a dedicated Mesh disassembler/assembler

Any thoughts from anybody?

Cheers, keep up the really great work guys!!

GrumpyOldMan

KnightErrant
02-22-2007, 04:52
Well, I take the point of view that you are the lead on this, of course,
so I'm willing to do what I can to help whichever way you want to
procede. Personally, I'd like to see a scheme or utility which makes
mesh creation and inclusion into the game as widely available as practical,
in other words, so people like me who's wives would kill us if we told them
we were buying a $3500 3D graphics application to support our $50 game
habit could still play in the game. (Just a humorous aside there.)

Seriously, of course the target group are the 3dsmax users but if there
is no real technical obstacle to programs like Milkshape 3D (with the
next release supporting vertex weighting) perhaps the best would be
a standalone that goes to a file format that both can import and export.
(I'm not excluding any other 3D programs, I just don't know this area
enough to make any judgements.)

At the moment, it seems like we possess a diode: information can come
out of a .mesh file and can be reworked into any number of formats depending
on the target application. The difficult part, but seemingly quite doable
with information from CA, is reversing the diode to make the information go
back into the mesh file.

I don't know your programming preferences, C, C++, Python, etc? I can do
MatLab (don't laugh), Python, Java, Intel assembler (what's the point), and
C but very little C++. I haven't a clue at what's involved in a plug-in which
is why I lean towards a standalone. I see a user as being a great graphics
artist but not necessarily a computer savant who could open a DOS box
and run a conversion utility to go from a filename.mesh to filename.xxx and
then import into his application. And in reverse filename.xxx to filename.mesh
after he's made his mods. Anyway, just my two cents.:2cents:

SigniferOne
02-22-2007, 08:09
Wow you guys are just amazing! Keep this fantastic work up!

On the other hand :( Caliban, what happened with the official tool? Is it going to turn out that it won't be released at all?

Caliban
02-22-2007, 08:51
Wow you guys are just amazing! Keep this fantastic work up!

On the other hand :( Caliban, what happened with the official tool? Is it going to turn out that it won't be released at all?

Unfortuantly there won't be an official tool anytime soon, my guess is that you guys are going to crack it before we do. I will be helping grumpy and the others out as much as I can in the meantime.

Myrddraal
02-23-2007, 12:58
Really good work guys. You'll make it into the history books of Vercingetorix when you succeed :grin:

Thanks a lot for your help Caliban :bow:

Burns
02-23-2007, 14:30
Reading this has brought joy to my cold Friday morning.

Balbor
02-23-2007, 15:23
are there any downloadable plug ins/scripts for either milkshape or 3Dmax yet?

Andromachus Theodoulos
02-23-2007, 20:06
@ KE and GOM

If possible, having something that would be available to many users would be great...

I have run into a dead end with 3dsmax... I can't even afford the student edition ... and with having five kids, my wife would kill me as well... which last i looked is around 600 or 700$, and I am not totally convinced that that would work with the models (I just revealed how much I know about this 3d animation stuff...)...:sweatdrop:

Anyway... my dream from the beginning was just being able to switch attachments on units... moving helmets around, weapons etc... everything else is there to make new units or change existing ones... textures, animations etc... CA has done great in opening a lot of this up to many modders and aspiring graphics/animation newbies...

I can understand if 3dsmax would be better, but something like MilkShape 3D would be awesome... something that would allow many of us rather than a few to have access to change stuff. (Just my 2 cents worth...)

GOM, KE, Casuir - All of you are doing a great service to the TW Modding community... your sharing of information, and working through this is not only exciting, but also educational.

Forgive me for interjecting this in between the technical work...

Thanks guys!!! :beam:

AT

KnightErrant
02-23-2007, 21:50
Hi AT
I hear you on this one. I too would probably
only want to do customizations and tweaks
and maybe spend a month of weekends doing one
dream unit just to do it.

I'm hoping that Caliban can confirm that the mystery data
really is an unused boolean stream and can be removed
and simply set that 1 to a 0. If this is true then its only the
vertex weighting that's an issue (with Milkshape). Take a look here

http://www.chumba.ch/chumbalum-soft/forum/showthread.php?t=18884

Mete says he's working on adding vertex weights for the 1.7.11 release.

All that's left then is the logic of the header/footers which is what
GrumpyOldMan is looking at now from his last post. His observation
about the floats being the bounding sphere makes sense, it looks
like it might be x,y,z and radius. The x,y,z being sort of the geometric
center of the form and the radius being a trigger for fight animations?
Don't know, just guessing.

Hope to do more this weekend. The real world has been
chewing up my time the last few days.~:)

GrumpyOldMan
02-23-2007, 23:39
Hi Guys

I'm looking more at the Milkshape option now after the comments. The utility I'm planning at the moment will give an option of extracting the whole figure with bones and vertex weightings, or just chopping the .mesh into component parts. The main difficulty with component parts is that, the way Milkshape works, if you merge component parts to a figure you lose the vertex assignments and weightings on the component part but I could spit out a text file with that info and it could be manually updated. Otherwise the other option is a purpose-written .mesh chopshop which retains all the info but will take longer to write.

It's my youngest boy's birthday this weekend so there's not going to be too much progress :juggle2: .

Cheers

GrumpyOldMan

KnightErrant
02-24-2007, 02:53
Sounds really good! Thanks for the update and wish your
youngest a very happy birthday.:balloon2:

Balbor
02-25-2007, 17:56
I have milkshape and 3Dmax 7, how do i import .mesh films to either of these programs?

KnightErrant
02-25-2007, 19:08
Sorry Balbor, we're not there yet. GrumpyOldMan is looking
at the Milkshape option but it will be a bit. I did some experiments
yesterday using a borrowed Milkshape tutorial form and exporting
it to a variety of formats. So far, it looks like the Milkshape
ASCII format and the fbx format are the only ones that retain
bone information so I'm going to try and look at the fbx format today.
It's binary so this will be tedious. I'll post back if I learn anything.

Balbor
02-25-2007, 20:15
Sorry Balbor, we're not there yet. GrumpyOldMan is looking
at the Milkshape option but it will be a bit. I did some experiments
yesterday using a borrowed Milkshape tutorial form and exporting
it to a variety of formats. So far, it looks like the Milkshape
ASCII format and the fbx format are the only ones that retain
bone information so I'm going to try and look at the fbx format today.
It's binary so this will be tedious. I'll post back if I learn anything.

so you're not at the point were it is possible to get M2TW battle map models such as units into Milkshape?

Re Berengario I
02-25-2007, 22:34
Hmmm I was thinking at one thing that maybe could help you guys, maybe not.

MTW2 models are exported meshes from 3ds elaborated to work with Granny, which is a code animation library used on a multitude of games.

Even if in MTW2 models don't have the suffix ".gr2" most of "grannified" models have, maybe their format is not very different from models of other games like NWN2 (another game hit by the granny pestilence) and, always maybe, there are people there who are doing your exact research and decoding work. Make a search at NwVault or Bioware Forums (NWN2 section) could give you some hints.

A plea to CA: I know Granny can spare a lot of coding time but if you want to keep your games opened to modification, until Granny won't open up its copyrighted format, you need to switch your attention to the scripted part of the game, giving us modders more options on that part of the game.

Black unbreakable boxes usually hit quickly the dusty shelves of game shops.

Tsar Stefan Dusan IV
02-26-2007, 01:15
First of all, I want to thank You guys for all the hard work!!!
It is nice to know that someone is making any kind of progress on this matter... Keep up the good work as all comunity will love you as I do! :)

Thanks goes to Caliban, too, for helping on those itchy matters.

I can't really help You about the programing and tech part, but am eagerly waiting for Your first release.

My opinion is vertex weighting isn't really hard, at least for experienced artist in 3d field, so my advise would be to concentrate on the vital info, meshes, then you can build on it at later stages. Convert in and outs for .mesh.

Fbx could prove valuable as a wide range of programs use it. Obj format also.
So user can import/export file info in those programs they are familiar with.

Once again, I sencerely bow, as You made my day with lot's of sunshine....:)

I am willing to help in any way I can, maybe test Your plugins in other 3d programs as I am bound to them by my profession and have acces to a wide range...

Sincerely Yours, Tsar Stefan Dusan IV

KnightErrant
02-27-2007, 06:31
Well, I don't know if this constitutes progress, but this is
where I'm at right now. I've been studying the fbx format this
past weekend. I downloaded a Milkshape tutorial which had a
form with bones, bone assignments, etc. but before keyframing
(which doesn't seem to be a .mesh attribute). It did have assignment
of vertices to bones but no vertex weighting because Milkshape
doesn't have this right now. I exported this model to 8 file formats
but the only ones that supported bone information seem to be the
native Milkshape ASCII and fbx. fbx is owned by Autodesk now,
(used to be Alias) and doesn't seem to have any format specification.

However, there is a free download from Autodesk of an SDK, in C++,
not one of my languages, which I downloaded and installed. No information
there about its actual format. In the Autodesk forums I found a
main() for converting the fbx binaries to fbx ASCII. Here's the main()
for those interested


int main(int argc, char** argv)
{
KFbxSdkManager* lSdkManager = NULL;
MyMemoryAllocator lMyMemoryAllocator;
KFbxScene* lScene = NULL;
bool lResult;

// Specify a custom memory allocator to be used by the FBX SDK
KFbxSdkManager::SetMemoryAllocator(&lMyMemoryAllocator);

// Prepare the FBX SDK.
InitializeSdkObjects(lSdkManager, lScene);

// Create the scene.

lResult = CreateScene(lSdkManager, lScene);

if(lResult == false)
{
printf("\n\nAn error occured while creating the scene...\n");
DestroySdkObjects(lSdkManager);
return 0;
}

// Save the scene.

// The example can take an output file name as an argument.
if(argc > 1)
{
lResult = SaveScene(lSdkManager, lScene, argv[1]);
}
// A default output file name is given otherwise.
else
{
lResult = SaveScene(lSdkManager, lScene, SAMPLE_FILENAME);
}

if(lResult == false)
{
printf("\n\nAn error occured while saving the scene...\n");
DestroySdkObjects(lSdkManager);
return 0;
}

// Destroy all objects created by the FBX SDK.
DestroySdkObjects(lSdkManager);

return 0;
}


Anyway here's the bones entries from the Milkshape 3D ASCII format


Bones: 10
"Main Bone"
""
8 0.133333 -68.428429 0.000000 1.570796 -0.000000 -3.141593
0
0
"Pelvis"
"Main Bone"
8 0.000000 0.000000 61.999996 0.000000 0.000000 0.000000
0
0
"Hip left"
"Pelvis"
8 -6.333340 -2.250002 -0.583334 -3.061560 0.070235 -3.141593
0
0
"Knee left"
"Hip left"
8 0.000000 0.000000 25.016495 -0.079812 0.035746 0.000000
0
0
"Calcaneum left"
"Knee left"
8 0.000000 0.000000 32.519444 -1.232623 0.610880 0.000000
0
0
"Toes left"
"Calcaneum left"
8 0.000000 0.000000 11.230869 0.000000 0.000000 0.000000
0
0
"Hip right"
"Pelvis"
8 6.654752 -2.249992 -0.464289 3.061907 0.059931 0.000000
0
0
"Knee right"
"Hip right"
8 0.000000 0.000000 25.124689 0.079672 0.018985 0.000000
0
0
"Calcaneum right"
"Knee right"
8 0.000000 0.000000 32.527370 1.237598 0.569478 0.000000
0
0
"Toes right"
"Calcaneum right"
13 0.000000 0.000000 11.213799 0.000000 0.000000 0.000000
0
0
GroupComments: 0
MaterialComments: 0
BoneComments: 0
ModelComment: 0


And here's the bones from the fbx ASCII file converted by the
above C++ program (Note: I'm not a C++ programmer, the main
shown above is taken from the Autodesk forums. I just built
it with the Autodesk SDK.)


; Object connections
;------------------------------------------------------------------

Connections: {
Connect: "OO", "Model::Main Bone", "Model::Scene"
Connect: "OO", "Model::DefaultMaterial", "Model::Scene"
Connect: "OO", "Model::jacke_seite_rechts", "Model::Scene"
Connect: "OO", "Model::Regroup01", "Model::Scene"
Connect: "OO", "Model::Duplicate04", "Model::Scene"
Connect: "OO", "Model::Pelvis", "Model::Main Bone"
Connect: "OO", "Model::Hip left", "Model::Pelvis"
Connect: "OO", "Model::Hip right", "Model::Pelvis"
Connect: "OO", "Model::Knee left", "Model::Hip left"
Connect: "OO", "Model::Calcaneum left", "Model::Knee left"
Connect: "OO", "Model::Toes left", "Model::Calcaneum left"
Connect: "OO", "Model::Knee right", "Model::Hip right"
Connect: "OO", "Model::Calcaneum right", "Model::Knee right"
Connect: "OO", "Model::Toes right", "Model::Calcaneum right"
Connect: "OO", "Material::DefaultMaterial", "Model::DefaultMaterial"
Connect: "OO", "Deformer::Skin DefaultMaterial_0", "Model::DefaultMaterial"
Connect: "OO", "SubDeformer::Cluster Pelvis_0", "Deformer::Skin DefaultMaterial_0"
Connect: "OO", "SubDeformer::Cluster Hip left_1", "Deformer::Skin DefaultMaterial_0"
Connect: "OO", "SubDeformer::Cluster Knee left_2", "Deformer::Skin DefaultMaterial_0"
Connect: "OO", "SubDeformer::Cluster Calcaneum left_3", "Deformer::Skin DefaultMaterial_0"
Connect: "OO", "SubDeformer::Cluster Toes left_4", "Deformer::Skin DefaultMaterial_0"
Connect: "OO", "Model::Pelvis", "SubDeformer::Cluster Pelvis_0"
Connect: "OO", "Model::Hip left", "SubDeformer::Cluster Hip left_1"
Connect: "OO", "Model::Knee left", "SubDeformer::Cluster Knee left_2"
Connect: "OO", "Model::Calcaneum left", "SubDeformer::Cluster Calcaneum left_3"
Connect: "OO", "Model::Toes left", "SubDeformer::Cluster Toes left_4"
Connect: "OO", "Material::Material01", "Model::jacke_seite_rechts"
Connect: "OO", "Texture::.\Test2.jpg", "Model::jacke_seite_rechts"
Connect: "OO", "Video::.\Test2.jpg", "Texture::.\Test2.jpg"
Connect: "OO", "Material::Material01", "Model::Regroup01"
Connect: "OO", "Texture::.\Test2.jpg", "Model::Regroup01"
Connect: "OO", "Deformer::Skin Regroup01_1", "Model::Regroup01"
Connect: "OO", "SubDeformer::Cluster Pelvis 1_5", "Deformer::Skin Regroup01_1"
Connect: "OO", "Model::Pelvis", "SubDeformer::Cluster Pelvis 1_5"
Connect: "OO", "Video::.\Test2.jpg", "Texture::.\Test2.jpg"
Connect: "OO", "Material::Default", "Model::Duplicate04"
Connect: "OO", "Deformer::Skin Duplicate04_2", "Model::Duplicate04"
Connect: "OO", "SubDeformer::Cluster Pelvis 2_6", "Deformer::Skin Duplicate04_2"
Connect: "OO", "SubDeformer::Cluster Hip left 1_7", "Deformer::Skin Duplicate04_2"
Connect: "OO", "SubDeformer::Cluster Hip right_8", "Deformer::Skin Duplicate04_2"
Connect: "OO", "SubDeformer::Cluster Knee right_9", "Deformer::Skin Duplicate04_2"
Connect: "OO", "SubDeformer::Cluster Calcaneum right_10", "Deformer::Skin Duplicate04_2"
Connect: "OO", "SubDeformer::Cluster Toes right_11", "Deformer::Skin Duplicate04_2"
Connect: "OO", "Model::Pelvis", "SubDeformer::Cluster Pelvis 2_6"
Connect: "OO", "Model::Hip left", "SubDeformer::Cluster Hip left 1_7"
Connect: "OO", "Model::Hip right", "SubDeformer::Cluster Hip right_8"
Connect: "OO", "Model::Knee right", "SubDeformer::Cluster Knee right_9"
Connect: "OO", "Model::Calcaneum right", "SubDeformer::Cluster Calcaneum right_10"
Connect: "OO", "Model::Toes right", "SubDeformer::Cluster Toes right_11"
}
;Takes and animation section
;----------------------------------------------------


This looks promising doesn't it? The information is all there.

Could Tsar Stefan Dusan IV or another 3dsmax professional answer
this question? Can 3dsmax import fbx in its ASCII form or must it
be binary. Secondly, can it export both or again, must it be binary?

It's late where I am so I have to go to bed. The vertex and face data
is there along with vertex weighting. I don't understand bone assignments
so I can't judge that. I'll look at it more tomorrow night. Sorry this is
so slow but all this stuff is really new to me.:wall:

Tsar Stefan Dusan IV
02-27-2007, 15:21
@Knight Errant
I don't really know, but here is some info (if it is, at all)
LINK (http://usa.autodesk.com/adsk/servlet/usearch/results?la=en&nh=10&siteID=123112&col=wacjprd&col=usuppprd&catID=123155&st=1&qt=fbx&id=2088334), but I guess You already visited this one.

What I know for certain is that it can read binary, I'm not sure about ASCII...

Maybe pdf's I attached can help a bit :book: , I don't know. There is a lot of irrelevant information (for this cause), but maybe you can dig something up... I'm sorry if this doesn't helps, but I'm a noob at programing :embarassed: , I'm an artist... I know how to import/export from different programs, but have no idea how it is actually composed, in program language :shame: .

http://rapidshare.com/files/18540794/fbx.rar.html

KnightErrant
02-27-2007, 16:04
@Tsar Stefan Dusan IV
Thanks very much, I'll look through the stuff in the .rar.
I have looked through the Autodesk forums and
what I've noticed is that every post asking about
the fbx format is never answered by an Autodesk
employee. They'll answer anything else but never
this.

I'll try and figure out how to put mesh data into the
fbx ascii format and then see if I can reverse the converter
code to go backwards to fbx binary and see if Milkshape
can read that.

Caliban
02-28-2007, 00:59
Could Tsar Stefan Dusan IV or another 3dsmax professional answer
this question? Can 3dsmax import fbx in its ASCII form or must it
be binary. Secondly, can it export both or again, must it be binary?


The animators here tell me you can import and export .fbx files to and from 3dsmax - Motion builder and most other 3d packages. It seems pretty widely supported. .obj is a good mesh format as it's also widely supported by the big 3d packages.


..but before keyframing (which doesn't seem to be a .mesh attribute).

The .mesh format should be purely for the mesh data and weighting. The actual animation data and keyframes are stored in the old .cas animation format specified by \data\descr_skeleton.txt which references the .cas animations stored in \data\animations

Foz
02-28-2007, 03:33
While I don't know much about computer graphics, especially the stuff you guys are getting into here, I do know some about C++. That C++ main() that you posted, KE, reads almost like plain text (to me at least, lol). That could be a very good thing, because it means they have basically everything already done for you and packed into nice functions in the SDK. CreateScene() and SaveScene() look to be the important bits that process all the data and get your conversion work actually done. Since those functions are so easy to use, I'd say it's likely your SDK also can quite easily turn the conversion around for you, probably with only 1 or 2 lines different from the main() you posted.

I would endeavor to comment further, but at this point I've become hazy on how exactly you guys envision the flow of data in this project, and what software exists or must be made to facilitate each step. If you assume .fbx will be the working format for the project, can someone outline where the data starts and what happens to it along the way, as well as which parts have software to execute them and which do not as yet? I'm envisioning a flowchart with data state boxes and process action arrows... I just want to make sure the one in my head is what's really going on lol :smile:

KnightErrant
02-28-2007, 16:26
@Caliban
Thank you for that. I think Foz is correct that I can easily
make an fbx converter that goes the other way using the
Alias SDK so my original concern over ASCII versus binary has
gone away.


The Mystery Blocks

A reply to Grumpy's question about the mystery blocks:

I read his file decription again and it certainly seems it is the unused bool stream. Just before them there should be number 1 somewhere (number of these streams) followed by number of vertices (this number is the same for all streams) followed by bool values themselves. He should be able to just change that 1 to 0 and then remove count + bool values completely. But I'd have to see the file to be sure, I don't really remember.

I'll have a definate answer on this soon after a bit of investigation.

Do you have any more information concerning the mystery block(s)?
Can we really ignore it or is it in fact some other type of information?

@Foz
Repeating myself but I think you're right, I just need to change 1 to 0
and 0 to 1 and little bit else and I've got an ASCII to binary fbx converter.
Meant to try this last night but didn't even get on the computer.

I'd have to think longer about it to give you a decent description but let
me just do a short one. From the .mesh file we get:
face data (triangles) grouped together with group names
uv co-ordinates (Nvert by 2 array of floats, x y coords)
vertex weights (Nvert by 2 array of floats, primary weight, secondary weight)
vertices (Nvert by 3 array of floats, x, y, z)
bone assignments (Nvert by 4 bytes)
Mytery block?
bone data

Unfortunately this isn't the way Milkshape ASCII or fbx ASCII groups it.
They organize the vertices, vertex normals (which aren't in the .mesh),
uv coords, triangle data, and UV indices? each in their own named group.
The programming task is then to pull out vertices, normals, and uv data
from the mesh's big list, put them in little lists, reindex them, recompute the
reference indices in the face data for each group, and then write
all this junk out with a bunch of surrounding boiler plate ASCII which
the fbx format requires. I'm just starting to code what I
just described but it'll probably be a slow process.

Andromachus Theodoulos
02-28-2007, 17:51
@KE,

First I want to say... take your time!!! If its a slow process, I am sure it will be worth it. I can certainly wait.

Anyway, to get a feel of what is developing, are we still looking at something that could be used with MilkShape?

I just want to say, you all have had me on the edge of my seat for the last several days, I have been following this thread closely. This would constitute one of the biggest breakthroughs for me personally if I am able to have access to an "affordable" modification tool for the meshes... I might even have something of value to post for others to download... which has not been true for my modifications in the past.

Keep up the good work!!! I am cheering you guys on!

@Caliban...
Thanks for your help on this and your gentle guidance...

AT

KnightErrant
02-28-2007, 18:21
Hi AT

GrumpyOldMan said he was doing the Milkshape binary format .ms3d,
so yes and this is probably the best path to success. I'm just looking
at fbx as an alternative just for a backup and because it's just interesting
in and of itself. 3dsmax doesn't support .ms3d but since Milkshape can
export fbx there's only one extra step to get meshes to the 3dsmax
modellers. GOM said something like "it's easy to take apart a clock, it's
the reassembly that's the killer" which refers to going the other way,
turning stuff back into a mesh file with all those funny header/trailer
byte blocks.

I've got my fingers crossed that he got info from Caliban that
makes that doable.~:)

Casuir
02-28-2007, 21:00
I read his file decription again and it certainly seems it is the unused bool stream. Just before them there should be number 1 somewhere (number of these streams) followed by number of vertices (this number is the same for all streams) followed by bool values themselves. He should be able to just change that 1 to 0 and then remove count + bool values completely. But I'd have to see the file to be sure, I don't really remember.

Been having pc problems so havent been able to test this until now but it seems changing the value mentioned above and removing the relevant blocks has no apparent ill effects ingame. Only tested it with a couple of meshs but it does look like its not neccessary and can be left out.

Re the formats ms3d is an affordable modelling program but it currently doesnt support all the necessary info in the mesh files so it might be wise to pursue other formats like fbx which do and arent program specific

KnightErrant
02-28-2007, 21:25
Thank you Casuir! :2thumbsup: That's what I was hoping.
Just so I'm clear on this you mean the header/trailer right
before the mystery block like



0 0 0 0 27 0 1 0 63 0 0 0 28 0 62 0 0 0 28 0 64 0 0 0 3 0 0 0
number of vertices in mystery block: 4949
79 25 186 0
181 17 162 0

and change the red 1 to a 0?

PS: And best of luck with your PC.

Casuir
02-28-2007, 21:38
Its the 3 right before the number of vertices, 3 being the number of streams (what we're calling blocks) which follow. Be careful here if you're deleting as I've noticed the .vertex/bone assignments block comes after this part rather than before in some .mesh files

KnightErrant
02-28-2007, 22:02
Ah, ok I see the 3. Hmmm, my script is going to
break on those mesh files where the vertex bone assignments are
out of place because both it and the mystery blocks have
the number of vertices as the first int to read so they look
alike. The size of the header/footer lines are different so it will
break right after reading the first mystery block of a lod0 file. I'm not
sure how to code around this.:embarassed:

Can you tell me a unit that has this reversed structure? Maybe there's
a clue in the header/footers.

Casuir
02-28-2007, 22:33
Think most of the meshs in the az_body_suit folder were this way, few others like as_light_cuman_archer_lod0.mesh are the same

KnightErrant
02-28-2007, 22:51
Thanks! I'll look at them when I get home and see if I can
fix the script to handle either case. Then make an fbx
ASCII to bin converter with the Alias SDK and I think that's
all the tools needed. Just thinking aloud, I could probably make
a three vertex, one face model in Milkshape and export it,
convert it to ASCII, and then find out just how much of the
fbx format is just default boilerplate, and which part actually
changes. That ought to zero in on what has to be programmed.

Caliban
03-01-2007, 05:50
Do you have any more information concerning the mystery block(s)?
Can we really ignore it or is it in fact some other type of information?


Engine guy is still on holidays, I'll prod him again next time I see him. My guess is that they are unused and can probably be ignored. A test should be able to tell but I'll find out for sure.


@Caliban...
Thanks for your help on this and your gentle guidance...

Aww shucks :beam:

KnightErrant
03-01-2007, 16:12
Thanks Caliban! Casuir has reported a test a few
posts back, where he removed the mystery blocks and changed
the 3 to a 0 in the header/footer line above where they appeared,
and suffered no ill effects with the game. I'll assume they
can be safely ignored.

Casuir also reported that there is at least one variant mesh format
where the vertex bone assignment block follows the mystery block
rather than proceeds it. I've only partially found all the changes but
the very first header/footer line is different between "regular" units
and these variants so I think it can be programmed around. Is this a
granny thing or just different ways that your guys assemble the
mesh files? Just wondering if we should be on the lookout for other
variant type meshes. (It's the az_body_suit types where he says
most of these variants are.)

GrumpyOldMan
03-02-2007, 03:10
Hi Guys

Lots of great things happening here, great to see!!

I've written some code, at the moment there is an alpha .MESH to .MS3D converter that exports bones and weights as well as the mesh and groups, there is also a pre alpha utility to load a .MESH file to screen and texture it with .TEXTURE files. Here's a screen pic https://i150.photobucket.com/albums/s104/grumpyoldman2007/testingmesh.jpg
:beam: . The texture in the background will be for UV adjustments but the code I've written for that is very slow and cluncky - major rewrite here.

The figures in the Aztec bodysuit folder are different in that the initial header is 4 bytes shorter, could be because there is only a body group and no leg group? - Could be a clue to the make up of the initial header. I've put an error trap in the converter to take care of this. Some of the figures in the folders seem to be WIP or base figures.

I've PM'd a few guys to test the converter, and I'll see what happens with their results.

If you're looking at alternative formats there is the .SMD format that's also supported by milkshape, it has vertex weighting (HL2) and allows export of animations only.

Cheers

GrumpyOldMan

Foz
03-02-2007, 04:46
Holy reverse engineering, Batman!

GOM isn't kidding, there's a lot of great things happening here. I'm a bit amazed at how quickly everything is getting figured out and implemented. I'm probably never going to edit a mesh for this game, and yet you guys have me all fired up for the importer/exporter tool to be done. It's almost like I can feel how badly everybody wants it, and see all the awesome visual mods that'll happen following briskly on the heels of this project.

Keep up the good work so I can keep on being amazed :smile:

KnightErrant
03-02-2007, 05:25
Kudos! congratulations, a balloon:balloon2: , and more.
Really great work!:2thumbsup: I'll look at more meshes
tomorrow but so far no problems at all. If there's anything
you need for the reverse process, unit surveys looking at
header/footers etc. please don't hesitate to ask.
I know the entire modding community is going
to be thrilled at what you've done. :jumping:

Caliban
03-02-2007, 07:59
The figures in the Aztec bodysuit folder are different in that the initial header is 4 bytes shorter, could be because there is only a body group and no leg group? - Could be a clue to the make up of the initial header. I've put an error trap in the converter to take care of this. Some of the figures in the folders seem to be WIP or base figures.

If you're looking at alternative formats there is the .SMD format that's also supported by milkshape, it has vertex weighting (HL2) and allows export of animations only.

Your right about not having leg variations. Models can be setup with many variations, or very few. This particular guy has a few body variations which include the legs all in the same mesh. Some units have seperate body, arm and leg variations.

I'll list the different types of variations and attachment types that we can 'markup' in 3dsmax. Mesh variations are numbered such as 'arms01' 'arm02' 'arm03' in 3dsmax and then simply set by selecting the mesh and then the type of variation with our plugin/exporter tool.

Variation Types:
(all meshes except the bones are marked in 3dsmax as one of the following)

Head
Leg
Arm
Body
Hands
Helmet

Attachment Types:
(these can have different variations per set. Usefull for adding things like pouches, bags etc)

Attachments
Attachments1
Attachments2
Attachments3

Weapons
(this markup is used for exporting weapons and shield seperatley, these are not setup within the character file)

Shield
Sword

Below is a 3dsmax7 character example with variations and attachments. Should work with 3dsmax7 or higher. It may prompt for our plugin, you should be able to simply press open to bypass it.

http://www.filefactory.com/file/64e26c/

.SMD format is not a bad idea, I'm also pretty familiar with it. It's quite widley documented and supported by all the big 3d Packages. I'm pretty sure the guys at Valve wouldn't mind you using it ;)

Casuir
03-02-2007, 13:50
Generals and captains are another example of meshs with few variations. Any hard limit on the number of variations caliban or is it limited to what you can fit on a texture? Cheers for the maxfile too :)

KnightErrant
03-02-2007, 22:25
@ GOM and Casuir
I took your and Caliban's advice and looked at the
.smd format (documented thank goodness!)
I've got my script so it outputs all the uv, vertex weights,
primary bone and secondary bone, plus the vertex normals
in the .smd format by triangles.
I'm confused by the node tree and skeleton pose data though.
I just copied the bonestrings part of the .mesh file and put
it in .smd format like


version 1
! Node tree or bone ID assignment section
nodes
0 "bone_pelvis" -1
1 "bone_rthigh" 0
2 "bone_rlowerleg" 1
3 "bone_rfoot" 2
4 "bone_abs" 3
5 "bone_torso" 4
6 "bone_head" 5
7 "bone_jaw" 6
8 "bone_eyebrow" 7
9 "bone_rclavical" 8
10 "bone_rupperarm" 9
11 "bone_relbow" 10
12 "bone_rhand" 11
13 "bone_lclavical" 12
14 "bone_lupperarm" 13
15 "bone_lelbow" 14
16 "bone_lhand" 15
17 "bone_lthigh" 16
18 "bone_llowerleg" 17
19 "bone_lfoot" 18
20 "bone_weapon01" 19
20 "bone_weapon" 20
21 "bone_weapon02" 20
22 "bone_weapon03" 21
22 "bone_shield01" 22
22 "bone_shield" 22
end

but this probably isn't right since the first number is supposed to
be the bone id and the second number is the parent id. Should
the first number just go up monotonically or should I just remove
duplicate weapon and shield entries?

Second question is the skeleton pose block which is
ID Posx Posy Posy Rotx Roty Rotz
The spec says bones that are children of other bones
report position and rotations in their parent's local space.
The rotations are spec'd as Euler rotations in radians.
I'm assuming this means body frame roll, pitch, and yaw (for x, y, and z).
For now I'm just putting in 0's but this probably won't work.
Can't test until I get home.


!Skeleton pose data
Skeleton
Time 0
0 0.0 0.0 0.0 0.0 0.0 0.0
1 0.0 0.0 0.0 0.0 0.0 0.0
2 0.0 0.0 0.0 0.0 0.0 0.0
3 0.0 0.0 0.0 0.0 0.0 0.0
4 0.0 0.0 0.0 0.0 0.0 0.0
5 0.0 0.0 0.0 0.0 0.0 0.0
6 0.0 0.0 0.0 0.0 0.0 0.0
7 0.0 0.0 0.0 0.0 0.0 0.0
8 0.0 0.0 0.0 0.0 0.0 0.0
9 0.0 0.0 0.0 0.0 0.0 0.0
10 0.0 0.0 0.0 0.0 0.0 0.0
11 0.0 0.0 0.0 0.0 0.0 0.0
12 0.0 0.0 0.0 0.0 0.0 0.0
13 0.0 0.0 0.0 0.0 0.0 0.0
14 0.0 0.0 0.0 0.0 0.0 0.0
15 0.0 0.0 0.0 0.0 0.0 0.0
16 0.0 0.0 0.0 0.0 0.0 0.0
17 0.0 0.0 0.0 0.0 0.0 0.0
18 0.0 0.0 0.0 0.0 0.0 0.0
19 0.0 0.0 0.0 0.0 0.0 0.0
20 0.0 0.0 0.0 0.0 0.0 0.0
20 0.0 0.0 0.0 0.0 0.0 0.0
21 0.0 0.0 0.0 0.0 0.0 0.0
22 0.0 0.0 0.0 0.0 0.0 0.0
22 0.0 0.0 0.0 0.0 0.0 0.0
22 0.0 0.0 0.0 0.0 0.0 0.0
end

It seems like we would lose the triangle mesh type names
and mesh group names in this format. Would this matter?

GrumpyOldMan
03-02-2007, 22:49
Hi All

Caliban

Thanks for the information. I've noticed that as well as Attachment groups there are Equipment type groups - is there any difference? Also (probably getting ahead of myself) but with the shields some are shown as shieldpassive and some as shieldactive. I was looking at the Lithuanian cavalry man in particular and the active shield is strapped to the back and the passive shield is on the left forearm. Is the active shield used with active weapons, or is the active shield used with the primary weapon and the passive shield used with secondary weapon :dizzy2: .

Thanks again for all the information.

Cheers

GrumpyOldMan

GrumpyOldMan
03-02-2007, 23:05
Hi @KE

I've just emailed you the node and skeleton information in smd format, you should be able to cut and paste. It was fairly simple to extract and export it in Milkshape from a converted .ms3d figure. I can give you instructions if you need them. The only drawback with smd and Milkshape at the present is that Milkshape is not importing smd with multiple weights, but this is due to change with the next release.

Cheers

GrumpyOldMan

KnightErrant
03-03-2007, 04:40
@GrumpyOldMan
Thanks! Got your e-mail. I was crashing Milkshape until
I made your changes. I had even seen the term hierarchy
tree a couple of weeks ago when I started following what
you were doing but it didn't register back then. Got the
.smd format working but it doesn't like the vertex weighting.
1.7.11 should resolve that but how did you get them in
with .ms3d?

Anyway, got that out of my system. I'll start looking at more
meshes tonight. Since there are 701 unit models and each have
3 or 4 lods, unless we have a lot of time on our hands we won't
look at all of them.~:)

Thanks again for the info!:bow:

GrumpyOldMan
03-03-2007, 05:44
Hi @KE

Yes the smd format is fairly simple but for now the weighted vertices won't load into Milkshape, but it look's like smd wil be the first cab off the rank for imported formats, Mete Ciragan, Milkshape's owner, programmer, etc had this to say about the changes to 1.7.11 :-


I will update the SMD, PSK and maybe Max Payne2 support. Others like MD3, MD2 are updated automatically. But I start with SMD and look how far I come. Of course with time every format that can support weighted vertices will be updated.

I got the weighted vertices into MS3D by using the binary format rather than the ascii format. This supports the recording of vertex weights. If you're using the binary spec be aware that the byte storing the weight is in the range 0-100 rather than 0 -255 as per the spec. The spec and SDK will be updated with 1.7.11. Mete confirmed the weight storage was the case but didn't mention if the ascii format will be updated.

Good luck with the looking at meshes :dizzy2: , it may sound strange but let me know as soon as possible if you find any that can't be converted :laugh4: .

Edit: Just had a thought on .smd, yes you will lose groups and names if you export a whole figure but why not look at using it to export single groups? The Mesh file is neatly put together so that all vertices and triangles used for any group are in sequential order. So if you wanted to look at any particular group the vertex numbers would range from the Lowest - number of vertices in all preceding groups, to the Highest - highest vertex listed in the triangles of the current group - ( I just read that and I'm not sure if that explains it well enough ). I'll write out some pseudo code if you need any clarification.

Sorry for boring anybody else out there.

Cheers

GrumpyOldMan

KnightErrant
03-03-2007, 06:00
@GOM
Ahhh, now I understand your comment a few posts back
about changing 0.677... to 67. Have you seen anything
on Mete's forums about a release date for 1.7.11?
I posted about his working on 1.7.11 and then going on to working
on 2.0 again but I don't know what 2.0 is supposed to add.
(I'm ashamed to admit I still haven't registered my copy.
I'll rectify this gross injustice to Mete this weekend.:oops: )

KnightErrant
03-03-2007, 06:38
Not boring me.
I was wondering how much the 3dsmax modders
are tied to things like that. If they really wanted grouping
for changing helmuts etc. would they find it hard if that
information got lost. I was probably down on the fbx
format because I made a binary to ASCII converter and
an ASCII to binary converter using their SDK and lost the
bone info. But it did tell me I was converting a 1.6.0 file
to a 1.6.1 format when I went from binary to ASCII in the
first place. (Milkshape uses the 1.6.0 format.) Wasn't eager
to try the experiment again with an older format because
the first SDK chewed up over a GIG of harddrive space.
(I've got plenty of space, just didn't know if this was
going anywhere.)

Strategy for meshes: CA probably did a very thorough
job on western units. (Marketing, no offense Caliban.)
I'll try to work on eastern European units and Byzantines
as well as Middle East units. lod0 through lod4 as a check
here and there but otherwise lod0 on a selection of units.

GrumpyOldMan
03-03-2007, 07:02
Hi @KE


I posted about his working on 1.7.11 and then going on to working
on 2.0 again but I don't know what 2.0 is supposed to add.

As fasr as I can see from the forums, there doesn't seem to be much going on with 2.0. He has stated that he is working on it and the only clue I've found is this:-


Q. Will you be able to model soley in the 3D view if you want to?

Yes, you can maximize one of the viewports and do anything from any projection. That's the goal.


Whoa-oa, a gig for an SDK, it would have to be a really good SDK :beam: .

Thanks for looking at the converted meshes, as well as any that don't convert, I'd really appreciate you looking at patterns of groups, ie does it matter if groups are mixed up - Attachments within body group clusters, group sequences out of order, etc.

Cheers

GrumpyOldMan

Arbaces
03-03-2007, 10:13
Oh! This is what I wanted to see...! Magnificent work GrumpyOldMan, wheter it will be in Milkshape or Max, we'll handle it.

My regards,

Arbaces.

KnightErrant
03-04-2007, 00:58
@GOM
Started on this a little late day. I've converted and opened in
Milkshape the following:

Mounts:


\unit_models\mounts\barded_horse\barded_horse_lod0
\unit_models\mounts\barded_horse\barded_horse_lod1
\unit_models\mounts\barded_horse\barded_horse_lod2

\unit_models\mounts\barded_horse\mount_barded_horse_lod0
\unit_models\mounts\barded_horse\mount_barded_horse_lod1
\unit_models\mounts\barded_horse\mount_barded_horse_lod2

\unit_models\mounts\camel\camel_lod0
\unit_models\mounts\camel\camel_lod1
\unit_models\mounts\camel\camel_lod2
\unit_models\mounts\camel\camel_lod3

\unit_models\mounts\camel\mount_camel_lod0
\unit_models\mounts\camel\mount_camel_lod1
\unit_models\mounts\camel\mount_camel_lod2
\unit_models\mounts\camel\mount_camel_lod3

\unit_models\mounts\elephant\elephant_cannon_lod0
\unit_models\mounts\elephant\elephant_lod0
\unit_models\mounts\elephant\elephant_rocket_lod0
\unit_models\mounts\elephant\mount_elephant_lod0
\unit_models\mounts\elephant\mount_elephant_rocket_lod0

\unit_models\mounts\european_armoured_horse\european_armored_horse_lod0
\unit_models\mounts\european_armoured_horse\mount_armored_horse_lod0

\unit_models\mounts\heavy_horse\heavy_horse_lod0
\unit_models\mounts\heavy_horse\mount_heavy_horse_lod0

\unit_models\mounts\mailed_horse\mailed_horse_lod0
\unit_models\mounts\mailed_horse\mount_mailed_horse_lod0

\unit_models\mounts\me_armoured_horse\me_armoured_horse_lod0
\unit_models\mounts\me_armoured_horse\mount_eastern_armoured_horse_lod0

\unit_models\mounts\medium_horse\medium_horse_lod0
\unit_models\mounts\medium_horse\mount_fast_pony_lod0
\unit_models\mounts\medium_horse\mount_pony_lod0
\unit_models\mounts\medium_horse\mount_pony_lod1
\unit_models\mounts\medium_horse\mount_pony_lod2


_units\as_lamellar_heavy :



\unit_models\_units\as_lamellar_heavy\as_lamellar_heavy_archer_lod0
\unit_models\_units\as_lamellar_heavy\as_lamellar_heavy_basic_lod0
\unit_models\_units\as_lamellar_heavy\as_lamellar_heavy_guard_lod0
\unit_models\_units\as_lamellar_heavy\as_lamellar_iron_lod0
\unit_models\_units\as_lamellar_heavy\as_lamellar_ironarcher_lod0
\unit_models\_units\as_lamellar_heavy\as_lamellar_leather_lod0
\unit_models\_units\as_lamellar_heavy\as_lamellar_leatherarcher_lod0
\unit_models\_units\as_lamellar_heavy\dismounted_heavy_archers_ug1_lod0
\unit_models\_units\as_lamellar_heavy\dismounted_heavy_archers_ug1_lod1
\unit_models\_units\as_lamellar_heavy\dismounted_heavy_archers_ug1_lod2

However on the last few units the converter started refusing to
do more than one unit, i.e. the load mesh button would depress
but the file chooser wouldn't come up. I can close it and restart
and get one more unit but same behavior.

Taking the wife out for dinner so will do more when I get back.
I'll try rebooting before I start again to see if that clears it.

PS: Just noticed this today, when I hit the exit button I get one
of those Microsoft error messages saying this application is not
responding do you want to send a report. I always closed it before
with the X button on the upper right corner and never saw this before.

GrumpyOldMan
03-04-2007, 03:20
Hi @KE

Thanks for this work you're putting in. I must admit I haven't tried it with mounts yet but it's good to see that the conversion works with them too. I see I'll have to extract the skeletons for camels and elephants to go with the horse skeleton I have and add some radio buttons to the converter so that the right skeleton is added to the file.

Forgive the .exe, it knows not that I play with a new GUI library :juggle2: .

Try the new exe I've emailed you with the texture splitting and amended UV values and see if that behaves the same. I may have to go in and see if I'm flushing the GUI_events properly.

NOTE for RTW Modders

I've had a look and there is a Max to MS3D plugin at http://www.mds.mdh.se/~elt01mcg/files/max2ms3d_v112.zip which exports an imported .CAS as mesh but not the bones. You could use this mesh to tie your vertices to a M2TW skeleton or chop off pieces as groups. I'm planning a facility to import single groups as MS3D.

Cheers

GrumpyOldMan

KnightErrant
03-04-2007, 04:24
Hi GOM,

Tried rebooting a few times but didn't fix it.
Got the new .exe and its working fine now.
I've looked at the weightings following your
help and I'm seeing it on the arm to torso and
leg to torso. I'm looking at the groups but I'm
not sure whether I know what you mean by


Thanks for looking at the converted meshes, as well as any that don't convert, I'd really appreciate you looking at patterns of groups, ie does it matter if groups are mixed up - Attachments within body group clusters, group sequences out of order, etc.

Is this the order the groups are listed in Milkshape
vs their mesh file order? Haven't noticed any problems.

I'm going to start skipping more to do a wider but less
detailed survey of the eastern units.

I''m off to Chattanooga tomorrow for my stepmom's 90th
birthday so won't be doing much more until tomorrow night.

Have a good rest of the weekend.:holiday:

KE

KnightErrant
03-04-2007, 05:35
Hi GrumpyOldMan,

Meh, spoke too soon. The behavior came back after doing
5 or so meshes. My environment is I had Eudora up for your
e-mail, Mozilla for the forums, Visual Slickedit where I keep a
record of each unit I work on, Explorer, and Milkshape (plus
your executable). See if converting multiple meshes on your
computer does the same.

Anyway I finished off these additional.

Directory \unit_models\_units\as_lamellar\heavy :


\unit_models\_units\as_lamellar_heavy\dismounted_heavy_lancers_lod0
\unit_models\_units\as_lamellar_heavy\dismounted_light_lancers_lod0
\unit_models\_units\as_lamellar_heavy\khan's_guard_lod0
\unit_models\_units\as_lamellar_heavy\mongol_bodyguard_lod0
\unit_models\_units\as_lamellar_heavy\mongol_heavy_archers_lod0
\unit_models\_units\as_lamellar_heavy\mongol_heavy_lancers_lod0
\unit_models\_units\as_lamellar_heavy\mongol_infantry_lod0
\unit_models\_units\as_lamellar_heavy\mongol_light_lancers_lod0
\unit_models\_units\as_lamellar_heavy\sabadar_militia_lod0


Directory \unit_models\_units\as_light :


\unit_models\_units\as_light\as_light_archer_lod0
\unit_models\_units\as_light\as_light_basic_lod0 % Invalid triangles/groups, killed Milkshape.
\unit_models\_units\as_light\as_light_cuman_archer_lod0 % Invalid triangles/groups, killed Milkshape.
\unit_models\_units\as_light\cuman_horse_archers_lod0
\unit_models\_units\as_light\dismounted_archers_lod0 % Invalid triangles/groups, killed Milkshape.
\unit_models\_units\as_light\kazaks_lod0 % Invalid triangles/groups, killed Milkshape.
\unit_models\_units\as_light\mongol_ballista_crew_lod0 % Invalid triangles/groups, killed Milkshape.


Check out the ones I've marked with an error message. They converted
with no error message but when I tried to open them I got a Milkshape
message of Invalid triangles/groups, do you want to convert them?
No I think meant nothing appeared and Yes killed Milkshape. Don't know
if its a conversion problem or these are just bad meshes to start with.
Thought I would get further than this but I'm out of steam right now.:dizzy2:
I'll check back tomorrow night.

KE

GrumpyOldMan
03-04-2007, 06:29
Hi @KE

My pudgy fingers came up with a typo :oops: which meant a variable wasn't getting refreshed. I've emailed the new version, I've tested it just now by converting the whole as_light folder in one sitting, seems to be working ok now.

I've had a look at the mounts and the UV values are different, it seems to me that the mount texture is a third file in the composite texture, and the mount attachments uses the figure attachments texture but uses a different UV value (negative values ). I'll look into it further after I've finished with figures.

Edit:- Just to keep people interested here's a picture of what we're getting in Milkshape.

https://i150.photobucket.com/albums/s104/grumpyoldman2007/Mongol.jpg

Cheers

GrumpyOldMan

Bwian
03-04-2007, 11:02
Been trying to follow this as time allows, and it certainly is looking promising!

Are we looking at switching to Milkshape 3D to model... or will Milkshape be more of a 'conversion tool' for getting models into a .mesh format? Personally, I would prefer Max as the tool ... but I don't know the first thing about scripting and am just glad someone can move this along!

Realistically, how far are we from being able to export something from MS3D to .mesh? I can see you have made huge strides in getting models out of the mesh format and into a modeller, and I have got renewed enthusiasm for modding MTW2 as a result! As least I know the composition of the models and the poly counts etc. that the stock models have.

Is there any way you could post a slightly more detailed breakdown of what the models are made up of? We have one from Caliban that I am using as a template... but I don't know if the parts it uses are the common makeup, or if they are widely different. I am also interested in knowing how many weapons/heads etc. is the maximum. For my Warhammer project, I want a reasonable degree of variation in units for Chaos .....

If you guys are stuck for time... I am happy to help out with testing this thing. Have access to MS 1.7.8 if that will run it. Will gladly pull models out and start listing components and polycounts etc.

Casuir
03-04-2007, 23:07
The figures in the Aztec bodysuit folder are different in that the initial header is 4 bytes shorter, could be because there is only a body group and no leg group? - Could be a clue to the make up of the initial header. I've put an error trap in the converter to take care of this. Some of the figures in the folders seem to be WIP or base figures.

Sorry havent done much with this but been having major pc probs. Got a few converted off today with no errors and noticed that some are as you said base figures. Seem to be the models that were exported sans primarys and secondarys and recompiled later via the xmls. Compared a couple for changes in the initial header and noticed a few things. These are the headers from the jannisary_gunner and jannisary_musketeers meshs, models are identical bar the missing primary and secondarys:
(leaving out the serialization::archive bit)



Musketeer:
00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 00 00 04 00 01 03 01 00 00
00 00 00 07 00 01 00 02 00 00 00 00 00 15 00 00 00 00 00 0B 00 01 03 03
00 00 00 04 00 00 00

Gunner
xx xx xx xx 00 00 00 00 00 00 00 01 00 00 00 00 00 04 00 01 03 00 00 00
00 00 00 07 00 01 00 01 00 00 00 00 00 0E 00 00 00 00 00 0B 00 01 03 02
00 00 00 04 00 00 00


I've marked the parts that are different in red, x'x at the start of the gunner code are there to represent the 4 bytes that are missing. I had a look at the guys in the az_body_suit folder and the guys in there missing 4 bytes look like base figures, theres corresponding meshs with weapons and the missing 4 bytes for them all, same goes for generals. It looks like every model thats used ingame has them. Only other bytes that change from file to file are the 15/0E which corresponds with the number of seperate meshs in the file and the 04 at the end, doesnt change between these two but I've noticed some mesh files have an 0C here instead. Other than that the header looks to stay the same between files.

@Bwian, dunno about limits but the bosnian archer has 45 seperate meshs with 8 head variations

Casuir
03-04-2007, 23:45
Bit more on the 04/0C value, it seems to relate to the mesh type that follows, all with 0C have an attachment as the first mesh in the file while those with 04 have a body part. Guessing its mesh variation header rather than the actual .mesh file one. Primary weapons have an 0E here, secondarys 10 and shields 07. Guess that makes the rest of the part between mesh variations footer rather than header.

btw the geometry value for weapons and shields is added to the base model, meaning anyone who feels like tackling the hexediting should be able to make weapon kitbashs without needing an importer/exporter.

Caliban
03-05-2007, 02:03
Thanks for the information. I've noticed that as well as Attachment groups there are Equipment type groups - is there any difference? Also (probably getting ahead of myself) but with the shields some are shown as shieldpassive and some as shieldactive.
GrumpyOldMan

I don't think there is a difference for equipment. You can have upto 4 different attachment types.

For example:

gunpowder pouch (attachment)
sword shieth (attachment1)
jewlery (attachment2)
quiver (attachment3)

Each of these attachment types can have multiple variations. There is no limit on variations for any of the meshes. They are simply marked as thier mesh type (attachment, attachment1, head, body etc etc) and then the individual variation meshes are named with a sequential numbering. Head01 head02 head03 head04, pouch01, pouch02, pouch03 etc. The engine then knows to choose one of each variation by its numbering.

Passive and Active shields are exactly what they state. However, I really don't think it makes too much difference. Passive shields are supposed to be for being attached to the back and active attached to the fore-arm. (if they are around the other way it probably doesn't matter)

There are no hard limits on variations. You can have 50 variations or more if you really wanted to but you would have a hard time fitting them onto a texture. It's possible to up the texture size to allow for more variations but you will be increasing memory usage a fair whack as you need to do this with normal maps too.

You can get an .SMD importer/exporter for 3dsmax here: http://www.chaosincarnate.net/cannonfodder/cftools.htm
Created by Cannon Fodder, this tool allows you to import the .smd mesh and then the .smd skeletal file. It will automatically rig the skeleton upto the imported mesh. It's probably alot more complete than the milkshape importer/exporter. Autodesk also offer 30 day trials on 3dsmax if you need to have a play around with it: http://usa.autodesk.com/adsk/servlet/index?siteID=123112&id=5972446

Casuir
03-05-2007, 03:03
My apols GOM, see you already figured out the byte for the number of meshs in the file and what the byte before the mesh name is for, really should pay more attention :embarassed: The rest of the footer seems to just be a sequence of numbers which increases by meshand is constant between files


04 00 00 00 Body Body 00 00 00 0A 00 01 00 04 00 00 00 0B 00 03 00 00 00 0B 00 05 00 00 00
04 00 00 00 Legs Leg 00 00 00 00 00 0A 00 06 00 00 00 0B 00 05 00 00 00 0B 00 07 00 00 00
0B 00 00 00 Attach Cylinder02 01 00 00 00 00 0A 00 08 00 00 00 0B 00 07 00 00 00 0B 00 09 00 00 00
04 00 00 00 Arms Arm 00 00 00 00 00 0A 00 0A 00 00 00 0B 00 09 00 00 00 0B 00 0B 00 00 00
0B 00 00 00 Attach Cylinder03 01 00 00 00 00 0A 00 0C 00 00 00 0B 00 0B 00 00 00 0B 00 0D 00 00 00
0B 00 00 00 Attach Box014 00 00 00 00 00 0A 00 0E 00 00 00 0B 00 0D 00 00 00 0B 00 0F 00 00 00
0B 00 00 00 Attach Plane02 01 00 00 00 00 0A 00 10 00 00 00 0B 00 0F 00 00 00 0B 00 11 00 00 00
0B 00 00 00 Attach Plane04 00 00 00 00 00 0A 00 12 00 00 00 0B 00 11 00 00 00 0B 00 13 00 00 00
06 00 00 00 Helmet Hat04 00 00 00 00 00 0A 00 14 00 00 00 0B 00 13 00 00 00 0B 00 15 00 00 00
04 00 00 00 Head Head1 00 00 00 00 00 0A 00 16 00 00 00 0B 00 15 00 00 00 0B 00 17 00 00 00
05 00 00 00 Hands Hand01 00 00 00 00 00 0A 00 18 00 00 00 0B 00 17 00 00 00 0B 00 19 00 00 00
04 00 00 00 Head Head02 00 00 00 00 00 0A 00 1A 00 00 00 0B 00 19 00 00 00 0B 00 1B 00 00 00
04 00 00 00 Head head03 00 00 00 00 00 0A 00 1C 00 00 00 0B 00 1B 00 00 00 0B 00 1D 00 00 00
04 00 00 00 Head Head04 00 00 00 00 00 0A 00 1E 00 00 00 0B 00 1D 00 00 00 0B 00 1F 00 00 00
10 00 00 00 Secnd Sec_2 00 00 00 00 00 0A 00 20 00 00 00 0B 00 1F 00 00 00 0B 00 21 00 00 00
10 00 00 00 Secnd Sec_6 00 00 00 00 00 0A 00 22 00 00 00 0B 00 21 00 00 00 0B 00 23 00 00 00
10 00 00 00 Secnd Sec_7 00 00 00 00 00 0A 00 24 00 00 00 0B 00 23 00 00 00 0B 00 25 00 00 00
10 00 00 00 Secnd Sec_8 00 00 00 00 00 0A 00 26 00 00 00 0B 00 25 00 00 00 0B 00 27 00 00 00
0E 00 00 00 Prmry Pri_27 00 00 00 00 00 0A 00 28 00 00 00 0B 00 27 00 00 00 0B 00 29 00 00 00
0E 00 00 00 Prmry Pri_26 00 00 00 00 00 0A 00 2A 00 00 00 0B 00 29 00 00 00 0B 00 2B 00 00 00
07 00 00 00 Ramrod RR_57 00 00 00 00 00 0A 00 2C 00 00 00 0B 00 2B 00 00 00 0B 00 2B 00 00 00


The only thing that changes is the first byte and It looks like you're right about it being related to attachments, doesnt seem to signify that its compulsoty though as some of the meshs above marked with it dont appear on all soldiers. My guess is that if theres just one type of mesh variant it gets used for all soldiers. The rest bar the first footer is just an increasing sequence and looks to be the same for all .mesh files I've looked at so far.

KnightErrant
03-05-2007, 03:38
@GOM
OK, I've been testing the latest .exe from yesterday.
Converted over 15 meshes without losing functionality
of the button. I'll continue spot checking more units
without closing it out just to verify it further.

KnightErrant
03-05-2007, 05:19
@GOM
Ok, done with tonight's survey. Did 54 units without closing out
the executable; no problems encountered. Units examined
included the ones that caused problems with bad triangles before,
no problems now (so marked below).


Directory \unit_models\_units\as_light :


\unit_models\_units\as_light\as_light_basic_lod0 % Bad before, good now.
\unit_models\_units\as_light\as_light_cuman_archer_lod0 % Bad before, good now.
\unit_models\_units\as_light\dismounted_archers_lod0 % Bad before, good now.
\unit_models\_units\as_light\kazaks_lod0 % Bad before, good now.
\unit_models\_units\as_light\mongol_ballista_crew_lod0 % Bad before, good now.
\unit_models\_units\as_light\mongol_bombard_crew_lod0
\unit_models\_units\as_light\mongol_cannon_crew_lod0
\unit_models\_units\as_light\mongol_catapult_crew_lod0
\unit_models\_units\as_light\mongol_foot_archers_lod0
\unit_models\_units\as_light\mongol_horse_archers_lod0
\unit_models\_units\as_light\mongol_monster_bombard_crew_lod0
\unit_models\_units\as_light\mongol_trebuchet_crew_lod0


Directory \unit_models\_units\ee_bekhtera_heavy_lamellar :


\unit_models\_units\ee_bekhtera_heavy_lamellar\berdiche_axemen_lod0
\unit_models\_units\ee_bekhtera_heavy_lamellar\boyar_sons_lod0
\unit_models\_units\ee_bekhtera_heavy_lamellar\dismounted_boyar_sons_lod0
\unit_models\_units\ee_bekhtera_heavy_lamellar\dismounted_dvor_lod0
\unit_models\_units\ee_bekhtera_heavy_lamellar\dvor_cavalry_lod0
\unit_models\_units\ee_bekhtera_heavy_lamellar\ee_bekhtera_archer_lod0
\unit_models\_units\ee_bekhtera_heavy_lamellar\ee_heavy_lamellar_archer_lod0
\unit_models\_units\ee_bekhtera_heavy_lamellar\tsars_guard_lod0

Directory \unit_models\_units\ee_peasant_leather :


\unit_models\_units\ee_peasant_leather\alan_light_cavalry_lod0
\unit_models\_units\ee_peasant_leather\ce_wagon_fort_lod0 (In spite of name, this is a crossbowman.)
\unit_models\_units\ee_peasant_leather\cossack_cavalry_lod0
\unit_models\_units\ee_peasant_leather\cossack_musketeers_lod0
\unit_models\_units\ee_peasant_leather\dismounted_lithuanian_cavalry_lod0
\unit_models\_units\ee_peasant_leather\ee_basilisk_crew_lod0
\unit_models\_units\ee_peasant_leather\ee_cavalry_militia_lod0
\unit_models\_units\ee_peasant_leather\ee_crossbow_militia_lod0
\unit_models\_units\ee_peasant_leather\ee_leather_cossack_gun_lod0
\unit_models\_units\ee_peasant_leather\ee_peasants_lod0
\unit_models\_units\ee_peasant_leather\ee_spearmen_lod0
\unit_models\_units\ee_peasant_leather\gulay_gorod_lod0
\unit_models\_units\ee_peasant_leather\slav_mercenaries_lod0
\unit_models\_units\ee_peasant_leather\woodsmen_lod0
\unit_models\_units\ee_peasant_leather\woodsmen_lod1
\unit_models\_units\ee_peasant_leather\woodsmen_lod2

Directory \unit_models\_units\es_greek_greek_heavy :


\unit_models\_units\es_greek_greek_heavy\byzantine_cavalry_lod0
\unit_models\_units\es_greek_greek_heavy\byzantine_infantry_lod0
\unit_models\_units\es_greek_greek_heavy\dismounted_byzantine_lancers_lod0
\unit_models\_units\es_greek_greek_heavy\es_greek_iron_archer_lod0
\unit_models\_units\es_greek_greek_heavy\es_greek_leather_lod0
\unit_models\_units\es_greek_greek_heavy\greek_catapult_crew_lod0
\unit_models\_units\es_greek_greek_heavy\kataphractoi_lod0
\unit_models\_units\es_greek_greek_heavy\varangian_guard_lod0
\unit_models\_units\es_greek_greek_heavy\varangian_guard_lod1
\unit_models\_units\es_greek_greek_heavy\varangian_guard_lod2

Directory \unit_models\_units\es_mail :


\unit_models\_units\es_mail\armenian_cavalry_lod0
\unit_models\_units\es_mail\dismounted_christian_guard_lod0
\unit_models\_units\es_mail\latinkon_lod0
\unit_models\_units\es_mail\norman_knights_lod0

Directory \unit_models\_units\es_peasant_padded :


\unit_models\_units\es_peasant_padded\armenian_archers_lod0
\unit_models\_units\es_peasant_padded\es_peasant_archer_lod0
\unit_models\_units\es_peasant_padded\skythikon_lod0
\unit_models\_units\es_peasant_padded\turkopoles_lod0


Obviously, the survey gets sparser as the evening wore on.~:)

GrumpyOldMan
03-05-2007, 05:31
Hi All

Lots of great things going on. I've got limited time today and a really big thunderstorm is rolling in as I type. More tomorrow, one snippet of news from the Milkshape forum:-


Release delayed
Hi, the new 1.7.11 release is delayed a little. I actually wanted to release it today, but I still have to add the weighted vertices support to the SMD import/export. PSK is already working.

So maybe on Tuesday ...

- Mete

Cheers

GrumpyOldMan

Andromachus Theodoulos
03-06-2007, 18:50
@ GOM and KE,

You guys are doing great work.

I would really like to help out. By testing betas, investigation and research. I am a complete newb with models and animation, but I do modify fairly intricate parts of the game, primarily textures, text files, and some code that is available for us to get to.

I work with computers all day and multiple types of software... primarily CADD programs and Databases.

Let me know if I can help you all in anyway, which maybe I could test stuff for you guys.

I have MilkShape on my machine. Let me know if I need anything else and what I could do to help out.

Thanks

AT

KnightErrant
03-06-2007, 22:06
@All

Been looking at the header/footers and think I've found a plausible
breaking scheme. Let me do a normal unit first, this is
feudal_knights_lod0



22 serialization::archive
3 4 4 4 8 1 0 0
meshtypeint = 256
Regular mesh, reading 4 bytes
0 0 0 0
0 1 0 0 0 0 0 4 0 1 3 1 0 0 0 0 0 7 0 1 0 2 0 0 0 0 0
number of bytes from tell() 69
number of bytes from bytecount 69
num_groups = 34
0 0

11 0 1 3
3 0 0 0 Arms arms_Lmail_01 210 (tris:) 0 0 0 0
0 0 0 0 0 10 0 1 0
4 0 0 0 11 0 3 0 0 0

11 0 5 0 0 0 Legs legs_01 244 (tris:) 0 0 0 0 0 10 0 6 0 0 0 11 0 5 0 0 0
11 0 7 0 0 0 Hands hands_01 202 (tris:) 0 0 0 0 0 10 0 8 0 0 0 11 0 7 0 0 0
11 0 9 0 0 0 Legs legs_02 244 (tris:) 0 0 0 0 0 10 0 10 0 0 0 11 0 9 0 0 0
11 0 11 0 0 0 Arms arms_Lmail_02 210 (tris:) 0 0 0 0 0 10 0 12 0 0 0 11 0 11 0 0 0
11 0 13 0 0 0 Arms arms_Lmail_03 210 (tris:) 0 0 0 0 0 10 0 14 0 0 0 11 0 13 0 0 0
11 0 15 0 0 0 Legs legs_03 244 (tris:) 0 0 0 0 0 10 0 16 0 0 0 11 0 15 0 0 0
11 0 17 0 0 0 Legs legs_04 244 (tris:) 0 0 0 0 0 10 0 18 0 0 0 11 0 17 0 0 0
11 0 19 0 0 0 Arms arms_Lmail_04 210 (tris:) 0 0 0 0 0 10 0 20 0 0 0 11 0 19 0 0 0
11 0 21 0 0 0 Body H_mail_body_01 270 (tris:) 0 0 0 0 0 10 0 22 0 0 0 11 0 21 0 0 0
11 0 23 0 0 0 Head Barrel_Helmet_03 192 (tris:) 0 0 0 0 0 10 0 24 0 0 0 11 0 23 0 0 0
11 0 25 0 0 0 Head Barrel_Helmet_01 176 (tris:) 0 0 0 0 0 10 0 26 0 0 0 11 0 25 0 0 0
11 0 27 0 0 0 Body H_mail_body_02 270 (tris:) 0 0 0 0 0 10 0 28 0 0 0 11 0 27 0 0 0
11 0 29 0 0 0 Body H_mail_body_03 270 (tris:) 0 0 0 0 0 10 0 30 0 0 0 11 0 29 0 0 0
11 0 31 0 0 0 Head Barrel_Helmet_02 186 (tris:) 0 0 0 0 0 10 0 32 0 0 0 11 0 31 0 0 0
11 0 33 0 0 0 Arms arms_Hmail_01 210 (tris:) 0 0 0 0 0 10 0 34 0 0 0 11 0 33 0 0 0
11 0 35 0 0 0 Attachments3 teeth 24 (tris:) 0 0 0 0 0 10 0 36 0 0 0 11 0 35 0 0 0
11 0 37 0 0 0 primaryactive0 light lance_0 78 (tris:) 0 0 0 0 0 10 0 38 0 0 0 11 0 37 0 0 0
11 0 39 0 0 0 secondaryactive0 sword secondary_25 64 (tris:) 0 0 0 0 0 10 0 40 0 0 0 11 0 39 0 0 0
11 0 41 0 0 0 secondaryactive0 sword secondary_26 64 (tris:) 0 0 0 0 0 10 0 42 0 0 0 11 0 41 0 0 0
11 0 43 0 0 0 secondaryactive0 sword secondary_27 64 (tris:) 0 0 0 0 0 10 0 44 0 0 0 11 0 43 0 0 0
11 0 45 0 0 0 secondaryactive1 sword secondary_31 108 (tris:) 0 0 0 0 0 10 0 46 0 0 0 11 0 45 0 0 0
11 0 47 0 0 0 secondaryactive1 sword secondary_32 108 (tris:) 0 0 0 0 0 10 0 48 0 0 0 11 0 47 0 0 0
11 0 49 0 0 0 secondaryactive1 sword secondary_33 108 (tris:) 0 0 0 0 0 10 0 50 0 0 0 11 0 49 0 0 0
11 0 51 0 0 0 shield0 heater pattern_79 70 (tris:) 0 0 0 0 0 10 0 52 0 0 0 11 0 51 0 0 0
11 0 53 0 0 0 shield0 heater pattern_80 70 (tris:) 0 0 0 0 0 10 0 54 0 0 0 11 0 53 0 0 0
11 0 55 0 0 0 shield0 heater pattern_81 70 (tris:) 0 0 0 0 0 10 0 56 0 0 0 11 0 55 0 0 0
11 0 57 0 0 0 shield0 heater pattern_82 70 (tris:) 0 0 0 0 0 10 0 58 0 0 0 11 0 57 0 0 0
11 0 59 0 0 0 shield0 heater pattern_83 70 (tris:) 0 0 0 0 0 10 0 60 0 0 0 11 0 59 0 0 0
11 0 61 0 0 0 shield0 heater pattern_84 70 (tris:) 0 0 0 0 0 10 0 62 0 0 0 11 0 61 0 0 0
11 0 63 0 0 0 shield0 heater pattern_85 70 (tris:) 0 0 0 0 0 10 0 64 0 0 0 11 0 63 0 0 0
11 0 65 0 0 0 shield0 heater pattern_86 70 (tris:) 0 0 0 0 0 10 0 66 0 0 0 11 0 65 0 0 0
11 0 67 0 0 0 shield0 heater pattern_87 70 (tris:) 0 0 0 0 0 10 0 68 0 0 0 11 0 67 0 0 0
11 0 69 0 0 0 shield0 heater pattern_88 70 (tris:) 0 0 0 0 0 10 0 70 0 0 0 11 0 69 0 0 0

6 0 1 0 71 0 0 0 7 0 2 0 0 0 142 17 0 0 0 0 2 0 0 0 0 0

18 0 1 0
72 0 0 0 4 0 0 0 0 0 4494 (uv coords: float pairs) 0 0 0 0 17 0 1 0
73 0 0 0 18 0 72 0 0 0

18 0 74 0 0 0 1 0 0 0 4494 (vert. wts: float pairs) 0 0 0 0 17 0 75 0 0 0 18 0 74 0 0 0 0 0 1 0 0 0 0 0

23 0 1 0
76 0 0 0 0 0 0 0 0 0 4494 (vert. vecs: float triple) 0 0 0 0 22 0 1 0
77 0 0 0 23 0 76 0 0 0 0 0 4 0 0 0 0 0

28 0 1 0
78 0 0 0 2 0 0 0 0 0 4494 (vert. bones: byte quads) 0 0 0 0 27 0 1 0
79 0 0 0 28 0 78 0 0 0

28 0 80 0 0 0 3 0 0 0 4494 (mystery strm: byte quads) 0 0 0 0 27 0 81 0 0 0 28 0 80 0 0 0
28 0 82 0 0 0 10 0 0 0 4494 (mystery strm: byte quads) 0 0 0 0 27 0 83 0 0 0 28 0 82 0 0 0
28 0 84 0 0 0 11 0 0 0 4494 (mystery strm: byte quads) 0 0 0 0 27 0 85 0 0 0 28 0 84 0 0 0
0 0 0 0 0 0 0 0 0 0
Bounding sphere data: 4 floats
0.00446363957599 0.432539612055 0.793390989304 1.68704903126
0 0 0 0
Processing bone strings, number = 26
11 bone_pelvis 0
11 bone_rthigh 1
14 bone_rlowerleg 2
10 bone_rfoot 3
8 bone_abs 4
10 bone_torso 5
9 bone_head 6
8 bone_jaw 7
12 bone_eyebrow 8
14 bone_rclavical 9
14 bone_rupperarm 10
11 bone_relbow 11
10 bone_rhand 12
14 bone_lclavical 13
14 bone_lupperarm 14
11 bone_lelbow 15
10 bone_lhand 16
11 bone_lthigh 17
14 bone_llowerleg 18
10 bone_lfoot 19
13 bone_weapon01 20
11 bone_weapon 20
13 bone_weapon02 21
13 bone_weapon03 22
13 bone_shield01 22
11 bone_shield 22
3 0 1 0 86 0 0 0 4 0 1 0 0 0 0 0 1 0 0 0 0 0 39 0 1 4 87 0 0 0
number of chars 13
characterlod0
17 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 1 0 0 0 3 0 0 0
18 192 63 0 4 0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 9 0 0 0
0 0 0 64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 8 0 0 0 2 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 2 0 0 0 0 0
128 63 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0
0 0 2 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
128 63 0 0 0 0 0 0 0 0 2 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 2 0 0 0 0 0 128 63 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 2 0
0 0 0 0 128 63 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0
0 0 0 0 0 0 2 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 128 63 0 0 0 0 0 0 0 0 2 0 0 0 0 0 128 63 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 38 0 1 0 88 0
0 0 39 0 87 0 0 0
0.00446363957599 0.432539612055 0.793390989304 1.68704903126
Bytecount is now 230546

Ignore the green highlighted stuff for the moment. It looks like you can
make up some "rules":

11 0 is the code for short triples (triangle data)
18 0 is the code for float pairs (uv coords and vertex weights)
23 0 is the code for float triples (vertex vectors and variant mystery streams, seen below in an aztec variant)
28 0 is the code for byte quads (regular mystery data and vertex bone assignments)

whenever you see these codes in the data header, they are repeated
along with a "one less" code in the footer. I.e. an 11 code section
has a 10 and 11 in its footer. This was pretty much my rationale
for breaking them up this way, that is, grouping what seemed to be
related data. The header and footers have running counts
which we've seen before.

There's also this "rule": After a two-byte code, if you get two more
bytes that start with a 1 (these are the red highlighted parts above)
like a 1 3 for the triangles or 1 0 in the vertex part, then you get
an extra two bytes in the footer with 1 0. In the case of the very
first triangle group you also get an extra 0 0 0 0 block as well.

Edit: always forget something in long posts, in the vertex parts when
you have a 1 0 following the code, you also get two extra zero bytes, 0 0,
in the header.

Let's repeat for the aztec variant:



22 serialization::archive
3 4 4 4 8 1 0 0
meshtypeint = 0
Variant mesh, NOT reading 4 bytes
0 1 0 0 0 0 0 4 0 1 3 0 0 0 0 0 0 7 0 1 0 1 0 0 0 0 0
number of bytes from tell() 65
number of bytes from bytecount 65
num_groups = 15
0 0

11 0 1 3
2 0 0 0 Body aztec 876 (tris:) 0 0 0 0
0 0 0 0 0 10 0 1 0
3 0 0 0 11 0 2 0 0 0

11 0 4 0 0 0 Attachments lip 2 (tris:) 1 0 0 0 0 10 0 5 0 0 0 11 0 4 0 0 0
11 0 6 0 0 0 Attachments nose 2 (tris:) 1 0 0 0 0 10 0 7 0 0 0 11 0 6 0 0 0
11 0 8 0 0 0 Hands Hands 190 (tris:) 0 0 0 0 0 10 0 9 0 0 0 11 0 8 0 0 0
11 0 10 0 0 0 Helmet Hat 116 (tris:) 0 0 0 0 0 10 0 11 0 0 0 11 0 10 0 0 0
11 0 12 0 0 0 Attachments Ear Things 40 (tris:) 1 0 0 0 0 10 0 13 0 0 0 11 0 12 0 0 0
11 0 14 0 0 0 Head Head 396 (tris:) 0 0 0 0 0 10 0 15 0 0 0 11 0 14 0 0 0
11 0 16 0 0 0 Head Head01 372 (tris:) 0 0 0 0 0 10 0 17 0 0 0 11 0 16 0 0 0
11 0 18 0 0 0 Head Head02 400 (tris:) 0 0 0 0 0 10 0 19 0 0 0 11 0 18 0 0 0
11 0 20 0 0 0 Head Head03 404 (tris:) 0 0 0 0 0 10 0 21 0 0 0 11 0 20 0 0 0
11 0 22 0 0 0 Head Head04 404 (tris:) 0 0 0 0 0 10 0 23 0 0 0 11 0 22 0 0 0
11 0 24 0 0 0 Helmet Head Dress 24 (tris:) 1 0 0 0 0 10 0 25 0 0 0 11 0 24 0 0 0
11 0 26 0 0 0 Body aztec01 876 (tris:) 0 0 0 0 0 10 0 27 0 0 0 11 0 26 0 0 0
11 0 28 0 0 0 Body aztec02 876 (tris:) 0 0 0 0 0 10 0 29 0 0 0 11 0 28 0 0 0
11 0 30 0 0 0 Body aztec03 876 (tris:) 0 0 0 0 0 10 0 31 0 0 0 11 0 30 0 0 0

6 0 1 0 32 0 0 0 7 0 1 0 0 0 224 16 0 0 0 0 2 0 0 0 0 0

18 0 1 0
33 0 0 0 4 0 0 0 0 0 4320 (uv coords: float pairs) 0 0 0 0 17 0 1 0
34 0 0 0 18 0 33 0 0 0

18 0 35 0 0 0 1 0 0 0 4320 (vert. wts: float pairs) 0 0 0 0 17 0 36 0 0 0 18 0 35 0 0 0 0 0 4 0 0 0 0 0

23 0 1 0
37 0 0 0 0 0 0 0 0 0 4320 (vert. vecs: float triple) 0 0 0 0 22 0 1 0
38 0 0 0 23 0 37 0 0 0

23 0 39 0 0 0 3 0 0 0 4320 (var myst: float triples) 0 0 0 0 22 0 40 0 0 0 23 0 39 0 0 0
23 0 41 0 0 0 10 0 0 0 4320 (var myst: float triples) 0 0 0 0 22 0 42 0 0 0 23 0 41 0 0 0
23 0 43 0 0 0 11 0 0 0 4320 (var myst: float triples) 0 0 0 0 22 0 44 0 0 0 23 0 43 0 0 0

0 0 1 0 0 0 0 0

28 0 1 0
45 0 0 0 2 0 0 0 0 0 4320 (vert. bones: byte quads) 0 0 0 0 27 0 1 0
46 0 0 0 28 0 45 0 0 0

0 0 0 0 0 0 0 0 0 0
-0.000567032024264 0.0598796904087 0.0334888845682 1.06927001476
0 0 0 0
Processing bone strings, number = 20
8 bone_abs 0
12 bone_eyebrow 1
9 bone_head 2
8 bone_jaw 3
14 bone_Lclavical 4
11 bone_Lelbow 5
10 bone_Lfoot 6
10 bone_Lhand 7
14 bone_Llowerleg 8
11 bone_LThigh 9
14 bone_Lupperarm 10
11 bone_pelvis 11
14 bone_Rclavical 12
11 bone_Relbow 13
10 bone_Rfoot 14
10 bone_Rhand 15
14 bone_Rlowerleg 16
11 bone_RThigh 17
14 bone_Rupperarm 18
10 bone_torso 19
3 0 1 0 47 0 0 0 4 0 0 0 0 0 0 0 1 0 0 0 0 0 39 0 1 4 48 0 0 0
number of chars 13
characterlod0
17 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 1 0 0 0 3 0 0 0
18 192 63 0 4 0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 9 0 0 0
0 0 0 64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 8 0 0 0 2 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 2 0 0 0 0 0
128 63 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0
0 0 2 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
128 63 0 0 0 0 0 0 0 0 2 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 2 0 0 0 0 0 128 63 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 2 0
0 0 0 0 128 63 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0
0 0 0 0 0 0 2 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 128 63 0 0 0 0 0 0 0 0 2 0 0 0 0 0 128 63 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 128 63 0 0 0 0 0 0 0 0 38 0 1 0 49 0
0 0 39 0 48 0 0 0
0.0324329286814 0.0481732748449 -0.0606130436063 1.08008432388
Bytecount is now 330874

The green highlights differences between the two.
The big header line at the top has 0 1 3 1 and 0 2 0 0 for the feudal knight
and has 0 1 3 0 and 0 1 0 0 for the variant. And as noted it has 4 bytes
less than regular units. The variant's sequence counts are even in the
headers but are odd for regular units.

After the triangle data the regular unit has a sequence 7 0 2 0 but the
variant has 7 0 1 0.

The vertex weights lines are very similar but feudal knights has the
final sequence 0 0 1 0 0 0 0 0 0 while the variant
has 0 0 4 0 0 0 0 0.

The oddest difference is on the vertex vectors line. The regular feudal
knight unit has the extra bytes 0 0 4 0 0 0 0 0 on this line and the
variant doesn't, yet the "codes" for both are identical. I was hoping
to discover local rules for codes controlling what get printed in the footers
but this case seems to rely on a global knowledge that this is a variant
type.

Note that not only does the variant have the mystery blocks before
the bones assignments, the streams aren't byte quad streams as
they are for regular units but float triples and the codes for the
mystery blocks change to reflect that.

Edit: the variant has attachments that have a 1 0 0 0 0 in the footer
(this has been noted in other's posts). However what determines
an attachment? The variant has lips, nose, and ear things as group
names and Attachments as group type as well as head dress of type
helmut that has the 1 0 0 0 0 but the feudal knight has teeth
of type Attachments3 and this just has all zeros 0 0 0 0 0. What's
the difference?

Anyway, that's all I've got for now.~:)

Myrddraal
03-06-2007, 22:46
You're not boring anyone, I'm reading (and not understanding) every word avidly. Well done guys.

Foz
03-07-2007, 01:29
You're not boring anyone, I'm reading (and not understanding) every word avidly. Well done guys.
That's a perfect description of it.

GrumpyOldMan
03-07-2007, 03:57
Hi Everybody

Cleaned up after the storm and rain and ready to get back into it. I'm really impressed with the work @KE and @Cas have been putting into the headers and footers, I was going to suggest we take a look at them but you've already started :2thumbsup: .

One thing I did spot in the Aztec figure was the screwed up skeleton:-


Processing bone strings, number = 20
8 bone_abs 0
12 bone_eyebrow 1
9 bone_head 2
8 bone_jaw 3
14 bone_Lclavical 4
11 bone_Lelbow 5
10 bone_Lfoot 6
10 bone_Lhand 7
14 bone_Llowerleg 8
11 bone_LThigh 9
14 bone_Lupperarm 10
11 bone_pelvis 11
14 bone_Rclavical 12
11 bone_Relbow 13
10 bone_Rfoot 14
10 bone_Rhand 15
14 bone_Rlowerleg 16
11 bone_RThigh 17
14 bone_Rupperarm 18
10 bone_torso 19

Not quite as neat and hierarchical as the '4' regular figures and the weapon/shield bones are missing. I've printed out the excellent stuff you've done and will now sit down and come up with a pattern for extracting stuff stored in the Milkshape file and storing it in a Mesh file. Incidentally, just because the files are stored in a MS3D file format, doesn't mean you have to use Milkshape. The format is becoming available in a number of software products, I'm currently having a look at FragMotion as an alternative modeller and animation creater. MS3D also opens up in LithUnwrapper and Ultimate Unwrap for skinning (and conversion) purposes.

Right, I'll sit down now and come up with some rules and pseudo code for the header/footers. As you guys look through the files could you just take notice of the bounding sphere values and see if they follow any sort of pattern ie dependent on foot vs mounted, or sword vs lance, etc. That may be handy to come up with some default values.

Everybody else that's offered to help, there'll be a place for anybody willing to risk a ctd in the name of progress :laugh4: , if you can make the necessary changes to get a new figure into M2TW that would be a great help too.

Cheers

GrumpyOldMan

Suraknar
03-07-2007, 10:24
Hrm...

I kind of understand now why the Data\Text\export_units.txt is not a .txt file this time around...methinks we were not expected to be making new models.

But, I have to say many of you seem to be doing very great progress here :)

Suraknar
03-07-2007, 12:15
Alrighty,

After a bit of reasearch, I came up with this:

http://www.radgametools.com/granny/download.html

You can freely Download the Granny Viewer, which comes with 2 sample .gr2 files, which may help you ellucidate some things if you can analyse them.

It seems like Radgametools however holds the licences (and they seem a greedy bunch too), many other games have this issue, and I got this from moding forums of AOEIII.

Correct me if I am wrong but from what I understand what your trying to do here is break the .mesh format not the gr2 format, and then if you can do that it could be reversible providing us with a way to do some modeling.

I will be looking for alternatives such as using .xml templates to probably be able to mix and match models (Body of A with head of B) from existing stock models (should not be against any licensing doing that).

I wish success to you :)

KnightErrant
03-07-2007, 16:01
@Suraknar
Yes, what GrumpyOldMan, Casuir, and myself have been calling
header/footer lines are probably granny strings. So we're essentially
trying to reverse engineer the granny strings and understand their
syntax to make a back and forth converter that makes no use of
any proprietary software. This would then be legal and freely
distributable.

Lusted
03-07-2007, 16:14
I kind of understand now why the Data\Text\export_units.txt is not a .txt file this time around...methinks we were not expected to be making new models.

You could say the same for all the fiels which are now .string.bins, and which alpaca has made a converter for.

alpaca
03-07-2007, 17:19
You could say the same for all the fiels which are now .string.bins, and which alpaca has made a converter for.
Nah they were just converted into a format that could be read more quickly by the machine.
The game was probably developed without thinking of modding to any large extent (except for some individuals, such as Caliban and Palamedes) and therefore the designers and programmers just used what yielded the best results in the shortest time possible.

Now it seems that we won't get any more official tools any time soon (maybe Caliban can explain why?) I'm glad to see that you guys are making quite a good deal of progress on this, and even more, provide us with a possible free way of editing models :yes:
I wish I had more time on my hands so I could work a bit on this, too. Anyways, after you cracked this customer, you should also have a look at the settlement packages (it seems there's some cool stuff in there and it's a lot more flexible than RTW) :whip:

Casuir
03-08-2007, 00:12
Question here on weapon bones, a weapon skeleton(s) is specified in the unit xml which is used to compile the finished .mesh so I'm guessing this bone data is pulled from there?


<Skeletons>
<Skeleton>
<Mount>Horse</Mount>
<Primary>MTW2_HR_Spear</Primary>
<Secondary>MTW2_HR_Non_Shield</Secondary>
<PrimaryAttachment>MTW2_HR_spear_Primary</PrimaryAttachment>
<SecondaryAttachment>MTW2_Sword_Primary</SecondaryAttachment>
</Skeleton>
There looks to be a number of different primary/secondary skeletons in the idx, would these affect things or are they just neccessary due to the way ca compiled the models? Also could animated weapons be possible?

Re bounding spheres, theres only 4 values here so its xyz and a radius aye? From what I've seen the radius for dismounted is usually around 1.0-3 and mounted lancers 2.0-1, mounted archers and such are usually the same as infantry so I'm guessing the mounts are handled separately and weapons affect the size.


I will be looking for alternatives such as using .xml templates to probably be able to mix and match models (Body of A with head of B) from existing stock models (should not be against any licensing doing that).

This wont be possible with an xml file afaik, the data is mixed up to much for it to be possible without extracting the parts from the .mesh and writing a completely new file. Changing primary/secondarys is possible with a bit of hex editing but unless you had some way of automatically re-writing the triangle values it could take you some time.

Caliban
03-08-2007, 01:23
I think maybe the knifeman has an animated weapon but I'm told animating the weapon bone can cause some freaky results.

The skeletons below simply tell the game what animation set to use for this unit. The example below means that his primary attack is a Spear animation set and his secondary is non_shield.
You can look up these animation sets by searching for MTW2_HR_Spear in data\descr_skeleton.txt All unit skeleton types are listed in here.
For example if you wanted him to play sword animations for his secondary, you would replace MTW2_HR_Non_Shield with -> MTW2_HR_Sword
If you wanted his primary to be mace animations you would replace MTW2_HR_Spear with -> MTW2_HR_Mace

Hope this helps :)


<Skeletons>
<Skeleton>
<Mount>Horse</Mount>
<Primary>MTW2_HR_Spear</Primary>
<Secondary>MTW2_HR_Non_Shield</Secondary>
<PrimaryAttachment>MTW2_HR_spear_Primary</PrimaryAttachment>
<SecondaryAttachment>MTW2_Sword_Primary</SecondaryAttachment>
</Skeleton>

Suraknar
03-08-2007, 01:53
This wont be possible with an xml file afaik, the data is mixed up to much for it to be possible without extracting the parts from the .mesh and writing a completely new file.

Indeed you are right Casuir,

Working from an Export of an Export is unfortunently not possible the XML way :( as I have come to realise in some tests and recently got it confirmed aswell.

So your efforts, KnightErrant and GrumpyOldman are the best way to go about it :)

Go! Go! Go! :2thumbsup: hehe :)

Casuir
03-08-2007, 02:11
It was more the PrimaryAttachment and SecondaryAttachment skeletons I was wondering about. Looked at the desc_skeletons and theres a few weapons which have animations other than stand_idle or default but they seem to be mainly ready, hold and release animations for missiles. Doesnt look to have any effect ingame though. What I was wondering was would it be possible to animate the actual weapon itself, ie have the arms of the bow draw back when drawn and return when the arrow is released.

Btw theres more than one weapons bone in the mesh, whats the story there?

@Suraknar, GOM and KE are the guys you should hang your hopes on, the major work heres being done by them and my contribution is very limited compared to theirs.

GrumpyOldMan
03-08-2007, 02:33
Hi Guys

With regards to the standard '256' and non-standard '0' meshes I thought I'd go and look at some of the model files (export_descr_unit.txt and battle_models.modeldb ) to see if I could get some clues :idea2: . The non-standard files are NOT used. They are WIPs and anyway the vertex assignments are all screwed up because of the funny skeletons. After I realised this the plaintive cries of "Dohhh" echoed off the hills for a long time, :oops: this shows you shouldn't get too focussed on just the one issue. With this in mind I'll 'lock out' the non-standard meshes since everything they have is available on standard meshes anyway.

I'd just like to make the point that I am not interested in reverse engineering anything, Granny and .gr2 are a mystery to me and will forever remain so. I am just looking at the .MESH format that CA is using and looking at ways we can modify these files. There are lots of bits of data that I don't understand and I don't want to understand but I don't need to as I can just use them. :sweatdrop: Just a clarification in case anybody from Radgames is interested.

Getting closer now.

@Casuir

You said you did some tests on taking out the boolean streams and using the figure in game. What exact blocks did you take out? Just the num_vertex and stream figures, with the header/footer stuff intact or the header/footer stuff as well?

Interesting stuff you pulled out on weapon bones, we'll have to look closer at that stuff to see how we actually get a figure in game.


I'm PMing Caliban with some questions on weapon protocol. It's just about sorted out in my head (and backs of envelopes), it's a matter of getting it into code.

Cheers

GrumpyOldMan

Casuir
03-08-2007, 02:48
They're definitly the exported caracters that are processed by the xmls, think theres a bit more to it than simply adding weapons, theres wip/base meshs in the mounts folder as well, geometry seems to be the same but the ingame file is considerably smaller than the base one, havent looked at them other than that though. Dont think we'll be able to replicate the method CA used to make the models anyways.

As for the booleans, I changed the 03 byte to 00 then tested that, then re-wrote most of the values with 00's then tested that, finally I deleted from the 2 byte num_vertex to the last byte of the third data stream (inclusive) and tested that. I also edited the texture on the model in question so I'd be sure the game was using the edited files.

Suraknar
03-08-2007, 03:17
Well guys, in reality I dont understand a third of what is beiing said here, even if I understand what the effort is all about.

I am no programmer, the Bits & Bytes you are talking about just go over my head :dizzy2:

It is only in the spirit of modding & cooperation and what I can think to help from having past moding experience (not only the small tweaks to this series) that I try to share, in hopes maybe that it can provide some insight to someone.

Modding is most of the time a guessing game, trial & error, and lots of patience and frustration at times (and I am sure you all know these things).

So that is that, again, wishing success to all :)

GrumpyOldMan
03-08-2007, 03:58
Hi Casuir

I've looked at the mounts as well, the meshes that are prefixed "mount_" look to be the in-game ones, the others also seem to have their vertex assignments screwy, does anyone want to try renaming or temporarily removing a non "mount_" mount mesh and then also with a "mount_" mesh and see if the game still runs? I think what we are looking at in the Data folder is a copy of their development and testing folder, after developing and testing were complete, it was easier and faster to export the whole lot to the production release, than to go through and check all the meshes (didn't KE say there were something like 3000 figure meshes alone) to make sure they were needed for the game - hard drive space is cheap now :laugh4: .

No I don't think we can do exactly what CA did to produce their meshes but I'm planning on the abilities to import meshes, import a single group (either exported from another mesh or done from scratch in a modelling program), delete a single group, etc. It may not be whizz-bang but it should have enough functionalty to keep modders going for a while. Mesh manipulation, vertex assignment and UV adjustment should be left in the hands of specialist programs rather than twisted and tortured at the hands of an amateur :laugh4: .

Will now go and have a look at the boolean streams, have to think a bit more about them.

Cheers, keep up the good work everyone.

GrumpyOldMan

Caliban
03-08-2007, 04:52
It was more the PrimaryAttachment and SecondaryAttachment skeletons I was wondering about. Looked at the desc_skeletons and theres a few weapons which have animations other than stand_idle or default but they seem to be mainly ready, hold and release animations for missiles. Doesnt look to have any effect ingame though. What I was wondering was would it be possible to animate the actual weapon itself, ie have the arms of the bow draw back when drawn and return when the arrow is released.

Btw theres more than one weapons bone in the mesh, whats the story there?


Ok, the attachment skeletons are only used with weapons that contain more than one bone. Which is the case for say swords but not for shields.
Sword has an additional sword bone that is attached to a hand, and we use that skeleton to animate it. But the shield is attached directly to the hand bone so there is no need for additional animation.

At one stage there were going to be proper animated weapons such as bow strings etc but there were issues with this and so it never made it in. I don't think it will be possible to animate a bow string straight up, it would properly require a bit of hackery. I'm told animating weapon bones can cause some wierd effects. There is also a lot of junk files in the folders, tests and obsolete units etc, there was simply no way to check what resources the game used and so you might find a few things that were left in.

KnightErrant
03-08-2007, 05:34
Regrets on the reverse engineering remark. I just meant
we want to understand header/footers enough to know
how to cut out things like the mystery blocks and still have
the units play in-game. The thing that worried me was looking
at a lod2 mesh file that only had one mystery block and not
three but still had this header:


18 0 48 0 0 0 1 0 0 0 1924 (vert. wts: float pairs) 0 0 0 0 17 0 49 0 0 0 18 0 48 0 0 0 0 0 1 0 0 0 0 0

23 0 1 0
50 0 0 0 0 0 0 0 0 0 1924 (vert. vecs: float triple) 0 0 0 0 22 0 1 0
51 0 0 0 23 0 50 0 0 0 0 0 2 0 0 0 0 0

28 0 1 0
52 0 0 0 2 0 0 0 0 0 1924 (vert. bones: byte quads) 0 0 0 0 27 0 1 0
53 0 0 0 28 0 52 0 0 0

28 0 54 0 0 0 3 0 0 0 1924 (mystery strm: byte quads) 0 0 0 0 27 0 55 0 0 0 28 0 54 0 0 0
0 0 0 0 0 0 0 0 0 0
Bounding sphere data: 4 floats
0.487427771091 0.393853664398 0.109047822654 1.71645057201


This was the armoured_sergeants_lod2.mesh. I thought that the
field, the one in red, was supposed to mean three blocks followed
and that we could zero it out and remove the data.

So now I'm confused as well, how do we repackage the mesh data
around the excised data? I echo GOM's question for Casuir, what is the
sequence of steps needed to remove mystery data? This might
help thrashing out the header/footers; I'm not convinced I split them
right, I just thought it was plausible.

Side note: I just took the unit count of 701 from the top of the modeldb file
and multiplied by 4 for the number of meshes to get something like
3000. Not every unit has 4 meshes, a lot have 3, and there are some
leftovers in the modeldb like the baltest units and some en_dummy units
near the end that aren't referenced in the EDU so they're not real units.

Casuir
03-08-2007, 06:41
Well the few bits that were left in have helped us figure stuff out so cheers for that :) Shame about the animated weapons, would have been a nice touch. I'm guessing we'd have seen flails too if it had panned out. What was the main idea behind having attachments sets btw, less resource usage?

@GOM Doubt the other non-standard meshs will work ingame, they get skeletons assigned to them by the xmls used to add the weapons so I'm guessing thats why the vertex datas screwed up. The xmls Caliban provided can be found here incase you havent seen them https://forums.totalwar.org/vb/showthread.php?t=74083
Interesting all the .gr2's referenced here and in the other leftover attsetdb files are found in the unit_models folder but as .mesh's not .gr2

I wouldnt worry about too much functionality, as long as it can export functional .mesh's most people will be happy enough.

@KE

As for the booleans, I changed the 03 byte to 00 then tested that, then re-wrote most of the values with 00's then tested that, finally I deleted from the 2 byte num_vertex to the last byte of the third data stream (inclusive) and tested that. I also edited the texture on the model in question so I'd be sure the game was using the edited files.

Cant test at the mo but I'm guessing if the boolean values arent used then it doesnt matter how many streams it says will follow.

Btw theres 3213 meshs including mounts and generals

KnightErrant
03-08-2007, 07:38
@Casuir
Thanks, I should read (and remember) what you said before.
This will be my number 1 task for tomorrow, I keep all the header/
footers, change 3 to 0, remove the mystery data including its
4-byte int vertex count, append the footer data without change,
and test in-game.

A btw, I thank you for including me with GrumpyOldMan but
that's giving me way too much credit. I'm just following along, confused
by things like everybody else, (except Grumpy).

A humorous side note: Did you see Alpaca's comment?


I wish I had more time on my hands so I could work a bit on this, too. Anyways, after you cracked this customer, you should also have a look at the settlement packages (it seems there's some cool stuff in there and it's a lot more flexible than RTW)


We should start charging don't you think?

zxiang1983
03-08-2007, 10:33
Hi all! I've gone through this whole thread for no less than 10 times and finally I think I understand most of its content, such as the structure of mesh file etc... But I still don't know how to import the mesh file (or part of it) into milkshape 3D. Would someone please tell me so I can join this historic work, too:)

taylorj2
03-08-2007, 18:14
Hi Guys, I tryed out the Rhino Program 3.0 and downloaded the plugin for MESH files, but it doesn't work still. It seems as though it should because even the Rhino program as options for joining triangles and stuff, can anyone help me why I can't import them?

Andromachus Theodoulos
03-08-2007, 19:01
@taylorj2 and zxiang1983

I think the guys (GOM, KE, Cas) are still working hard on trying to establish what exactly is necessary within the mesh files. I think they are also trying to figure out how to get the components repackaged so that they will play in game. They are getting very close. Hopefully it won't be too much longer and we can start making changes to meshes.

Keep up the good work!!!

AT

taylorj2
03-08-2007, 23:18
Hmmm maybe the reason we can't open the MESH files normally is because of the 2 bytes of dummy data? CA could of put them in there so we can't edit them

Probably not the answer but its a idea.

Casuir
03-09-2007, 00:39
Na its a completely different format. Rhino's primarily for modelling in nurbs and isnt really suitable for modelling for games.

Caliban
03-09-2007, 01:55
There are lots of bits of data that I don't understand and I don't want to understand but I don't need to as I can just use them. Just a clarification in case anybody from Radgames is interested.

The .MESH format was written inhouse by our engine programmer (that rhino format is completely different as Casuir pointed out). It's not a radtools format and it doesn't contain any of the granny format. It's simply an optimized binary format for the engine to load. No need to worry on that front :)


What was the main idea behind having attachments sets btw, less resource usage?
We re-use alot of base meshes between units so we didn't want to have to build unique meshes each time. The attachment sets let us add that extra detail without building it into the meshes. This lets us have a greater amount of total unit variations by mixing different body parts and attachments. The attachment sets were to keep the attachments grouped by type (such as gunpowder pouches and belts or jewlery).

GrumpyOldMan
03-09-2007, 02:41
Hi Guys

Not much progress because Milkshape 1.8 has come out and I've been playing with this new toy. Thought I'd try it on a request someone put in for a dismounted Hospitaller, took me about five minutes to delete the lances and rename the groups and update the comments with the details.

https://i150.photobucket.com/albums/s104/grumpyoldman2007/Hosp_dismounted.jpg

and the vertex weighting looks like this:- https://i150.photobucket.com/albums/s104/grumpyoldman2007/Hosp_vertex_weight.jpg

1.8 has one annoying bug in that if you try to regroup two separate groups it crashes but this is still possible in 1.7.10 (save and reopen in 1.8). Both are still available:-

http://www.chumba.ch/chumbalum-soft/files/ms3d180.zip and http://www.chumba.ch/chumbalum-soft/files/ms3d1710.zip . The SDK has been updated also, http://www.chumba.ch/chumbalum-soft/files/ms3dsdk180.zip.

As Caliban says the mesh file being used is not any sort of file that will work with a current plugin, etc. We don't have to follow the CA development path to come up with working mesh files, just something that the engine recognises and will use. We shouldn't try to make this harder than it really is. I've had some feedback from Caliban and this seems to be getting a bit easier.

What I'm planning is:-

1 A .MESH to MS3D converter - this is about 98% complete

2. Load .MESH and export single group as .MS3D - about 65% complete

3. A .MS3D to .MESH converter - A gleam in my eye but shouldn't take too long

Any other thoughts?

Cheers

GrumpyOldMan

zxiang1983
03-09-2007, 03:39
Hi Guys

Not much progress because Milkshape 1.8 has come out and I've been playing with this new toy. Thought I'd try it on a request someone put in for a dismounted Hospitaller, took me about five minutes to delete the lances and rename the groups and update the comments with the details.



Wow, impressive job! My dream comes true! I can't wait to see you release that unit and the converter, too, so we can all DIY:laugh4:

KnightErrant
03-09-2007, 04:12
Nice looking unit there. I tried the Casuir
excision method on feudal_knights_lod0.mesh
and it worked in a custom battle so I'm happy
to see this. Registered now so I've got to
upgrade ASAP to play with the new features.~:)

GrumpyOldMan
03-09-2007, 04:42
Hi @KE


I tried the Casuir excision method on feudal_knights_lod0.mesh and it worked in a custom battle so I'm happy to see this.

Me too, This also means that the engine is strong enough to withstand this sort of punishment. From Casuir's post he also deleted from the 3 (changed to 0) down to the end of the streams and it still worked, so the blip in progression between header/footers was also acceptable.

Cheers

GrumpyOldMan

KnightErrant
03-09-2007, 05:38
Didn't catch that the first time. I left
in the 3 0 0 0 converted to 0 0 0 0 but
deleted from the number of vertices to
end of data. That is encouraging that you
can be a little loose in the format and the game
still keeps track of how to get to its data.

Russ Mitchell
03-09-2007, 05:59
Thanks for your work, gents. I follow the gist, but not the code details, and it's still fascinating.

Andromachus Theodoulos
03-09-2007, 16:32
@All...

This is awesome, I am 36 years old and I feel like a little kid in a hobby shop. Having the ability to add and remove weapons and shields??? Awesome.

@GOM,
Will MilkShape allow us to modify the actual mesh, like say if we wanted to modify a helmet to make it look different. Would we be able to do that??

And I assume that if we can switch shields and weapons around, we can also change stuff like the helmet models as well, because some of these are attachments, I think?

Very nice work.

Can't wait for the converters and other stuff to be ready...

Like most everyone else, the code stuff is over my head right now, but the excitement is almost uncontainable...

Also, another note to all...

Would we be able to use MilkShape to modify the Rome Total War models as well. I know they are in a cas format (I think) what extra programming would be required there???

Again, Thanks for all the hard work, you are going to make a lot of Total War diehards very happy...

AT

Suraknar
03-10-2007, 02:50
As Caliban says the mesh file being used is not any sort of file that will work with a current plugin, etc. We don't have to follow the CA development path to come up with working mesh files, just something that the engine recognises and will use. We shouldn't try to make this harder than it really is. I've had some feedback from Caliban and this seems to be getting a bit easier.

What I'm planning is:-

1 A .MESH to MS3D converter - this is about 98% complete

2. Load .MESH and export single group as .MS3D - about 65% complete

3. A .MS3D to .MESH converter - A gleam in my eye but shouldn't take too long

Any other thoughts?

Cheers

GrumpyOldMan


Yes this is sensible indeed, it will permit new units, and Milkshape is very accessible compared to other 3D packages, I for one am excited!

Great progress all! :)

KnightErrant
03-10-2007, 03:50
Well I'm embarassed:embarassed: , I put my excised unit
in my mod folder, not the true /data/unit_models/_units
etc. last night. I put it in the right place and now I'm crashing
the game. I tried the set
3 to 0 and remove from the number of vertices to the
end of mystery data as well as removing the whole entry
including what I think are the headers and footers but no
luck. Crashes with an unspecified error. Casuir, do you know
what I'm doing wrong?

GrumpyOldMan
03-10-2007, 04:13
Hi All

I just wanted to post that after I put in a bug report on the Regroup groups function in Milkshape that it has been fixed and reuploaded to the Milkshape site. If you've downloaded Milkshape 1.8 before you may want to redownload to get the fixed version.

@Andromachus Theodoulos

Milkshape is a full mesh manipulation tool so you can modify mesh appearance, vertex/bone assignments, etc. Once this all up and running, the possibilities extend well beyond just switching weapons/shields of existing figures. From what I can see and have understood, as long as your figure is understood by the engine it can be used; so this means heads, bodies, arms, legs, weapons, etc will be able to be swapped or exchanged for new ones. As long as they all fit together and have the right values, they'll work.

I could probably do a .CAS to MS3D converter but that may have to wait for a while, although now I think of it I may do one that extracts an unassigned mesh and the M2TW skeleton (different skeletons used). Note to RTW modders, meshes will work with 100% assignments (just not look as purty) but the meshes will have to be sliced up into chunks as groups. Might be easier to use closest fit M2TW body parts and new textures.

Cheers

GrumpyOldMan

KnightErrant
03-10-2007, 06:22
Houston we have a problem............

It's always good to question one's assumptions.
The mystery data is used in the game to control
shading and texturing rendering. Because I couldn't
get the excised file to load I wrote a Python script
to zero out the mystery data but leave the file otherwise
as is. It loads and plays but the graphical results
are disappointing. The unit (hard-coded to feudal_knights,
my favorite test case) flickers in and out of sunlight in
a custom battle and the unit colors also render oddly.

I'm sending an attachment to GOM and Casuir to test
but we need a better answer to what the mystery data
is. I've been working on the .ms3d format and there
doesn't seem to be anywhere to place more data (shader
or whatever this is).

Not to worry, this isn't a showstopper but we need to know
what this is and how to deal with it.

Here's the code if anyone else wants to experiment:
(The getters and putters are from my main script
for converting mesh files, I don't use all of them here.)


import struct

global bytecount
global ms3dbytecount

# -----------------------------------------------------------------------------------
def getbyte( fidin ) :
global bytecount
(thebyte,) = struct.unpack( 'b', fidin.read(1) )
bytecount = bytecount + 1
return thebyte

# -----------------------------------------------------------------------------------
def getubyte( fidin ) :
global bytecount
(thebyte,) = struct.unpack( 'B', fidin.read(1) )
bytecount = bytecount + 1
return thebyte

# -----------------------------------------------------------------------------------
def getshort( fidin ) :
global bytecount
(theshort,) = struct.unpack( 'h', fidin.read(2) )
bytecount = bytecount + 2
return theshort

# -----------------------------------------------------------------------------------
def getushort( fidin ) :
global bytecount
(theshort,) = struct.unpack( 'H', fidin.read(2) )
bytecount = bytecount + 2
return theshort

# -----------------------------------------------------------------------------------
def getint( fidin ) :
global bytecount
(theint,) = struct.unpack( 'i', fidin.read(4) )
bytecount = bytecount + 4
return theint

# -----------------------------------------------------------------------------------
def getuint( fidin ) :
global bytecount
(theint,) = struct.unpack( 'I', fidin.read(4) )
bytecount = bytecount + 4
return theint

# -----------------------------------------------------------------------------------
def getfloat( fidin ) :
global bytecount
(thefloat,) = struct.unpack( 'f', fidin.read(4) )
bytecount = bytecount + 4
return thefloat

# -----------------------------------------------------------------------------------
def putbyte( thebyte, fidout ) :
global ms3dbytecount
fidout.write( struct.pack( 'b', thebyte ) )
ms3dbytecount = ms3dbytecount + 1
return

# -----------------------------------------------------------------------------------
def putubyte( thebyte, fidout ) :
global ms3dbytecount
fidout.write( struct.pack( 'B', thebyte ) )
ms3dbytecount = ms3dbytecount + 1
return

# -----------------------------------------------------------------------------------
def putshort( theshort, fidout ) :
global ms3dbytecount
fidout.write( struct.pack( 'h', theshort ) )
ms3dbytecount = ms3dbytecount + 2
return

# -----------------------------------------------------------------------------------
def putushort( theshort, fidout ) :
global ms3dbytecount
fidout.write( struct.pack( 'H', theshort ) )
ms3dbytecount = ms3dbytecount + 2
return

# -----------------------------------------------------------------------------------
def putint( theint, fidout ) :
global ms3dbytecount
fidout.write( struct.pack( 'i', theint ) )
ms3dbytecount = ms3dbytecount + 4
return

# -----------------------------------------------------------------------------------
def putuint( theint, fidout ) :
global ms3dbytecount
fidout.write( struct.pack( 'I', theint ) )
ms3dbytecount = ms3dbytecount + 4
return

# -----------------------------------------------------------------------------------
def putfloat( thefloat, fidout ) :
global ms3dbytecount
fidout.write( struct.pack( 'f', thefloat ) )
ms3dbytecount = ms3dbytecount + 4
return

# -----------------------------------------------------------------------------------
def putzerobytes( n, fidout ) :
global ms3dbytecount
for ii in range(n) :
putubyte( 0, fidout )
ms3dbytecount = ms3dbytecount + 1
return


# Main.

global bytecount
global ms3dbytecount
bytecount = 0
ms3dbytecount = 0

fnin = 'feudal_knights_lod0_test.mesh'
fnout = 'feudal_knights_lod0_testchop.mesh'
fidin = open( fnin, 'rb' )
fidout = open( fnout, 'wb' )

for ii in range(2*175) :
fidout.write( fidin.read(500) )

fidout.write( fidin.read(477) )

# We're at the mystery blocks.
nvert = getuint( fidin )
putuint( nvert, fidout )
print 'nvert = ' + str( nvert )

for ii in range( nvert ) :
sdummy = getushort( fidin )
sdummy = getushort( fidin )
putushort( 0, fidout )
putushort( 0, fidout )

nheaderfooter = 26
for ii in range( nheaderfooter ) :
thebyte = getubyte( fidin )
putubyte( thebyte, fidout )

# 2nd mystery block.
nvert = getuint( fidin )
putuint( nvert, fidout )
print 'nvert = ' + str( nvert )

for ii in range( nvert ) :
sdummy = getushort( fidin )
sdummy = getushort( fidin )
putushort( 0, fidout )
putushort( 0, fidout )

nheaderfooter = 26
for ii in range( nheaderfooter ) :
thebyte = getubyte( fidin )
putubyte( thebyte, fidout )

# 3rd mystery block.
nvert = getuint( fidin )
putuint( nvert, fidout )
print 'nvert = ' + str( nvert )

for ii in range( nvert ) :
sdummy = getushort( fidin )
sdummy = getushort( fidin )
putushort( 0, fidout )
putushort( 0, fidout )

#fidin.seek( 229468, 0 )

fidout.write( fidin.read(500) )
fidout.write( fidin.read(600) )

fidin.close()
fidout.close()

Casuir
03-10-2007, 06:25
Mkay, looks like I messed up here and tested the base figure file instead of the actual ingame one :embarassed: Guess I should have been a bit more thoroug. Did some more testing and it looks like these values are necessary, changing the 3 to 0 crashs the game as does removing any of the values. Changing all the values to 00 in the first stream gave this result:
https://img100.imageshack.us/img100/2325/image1ut0.jpg (https://imageshack.us)
As you can see it has some effect on the lighting, unfortunatly my graphics card is on the way out and doesnt like the battle mode so I cant get a proper look at whats happening, can somebody else have a look at these and see whats happening, preferably someone whos machine can handle all the high-end bells and whistles. Arrow warriors have the first block zeroed out, eagle the second and cuahchiqueh the third.
http://www.wikiupload.com/download_page.php?id=99774
Scrrenshots would be nice

Looks like ya bet me to it KE

GrumpyOldMan
03-10-2007, 08:26
Hi

@KE and Casuir

KE, I tried your example but I couldn't get much difference from the stock figure, I tried it with different combinations of shader system, shadow quality and anything else I could tinker with in the Options menu. Could this be card related - I'm using a 9800pro.

Casuir, I tried the download link you sent but I just got caught in a loop with not much being downloaded.

Thoughts - KE, I'm not familiar with Python, and seeing you've already got it set up :beam: could you try your script with the non zero values being replaced by 127 or 255 or random 0-255?

KE, with your permission I'd like to expand the test on the modified mesh.

Would people like to try out the feudal knight mesh to see what happens on their system? I've just tried it about twenty times on my system and there have been no crashes or other strange behaviour. If KE is happy I can post a download link.

Cheers

GrumpyOldMan

Casuir
03-10-2007, 08:45
Try this one, the first one wont let me dl either
http://www.megaupload.com/?d=O8YR7OVW

Managed to get a bit more messing done and changed the values around a bit
https://img98.imageshack.us/img98/899/blueoe9.th.jpg (https://img98.imageshack.us/my.php?image=blueoe9.jpg)
https://img101.imageshack.us/img101/7858/reded1.th.jpg (https://img101.imageshack.us/my.php?image=reded1.jpg)
first one is 00 00 ff 00, second one is 00 ff 00 00
Thought they might have been rgb values at first but colours change depending on the weather, the lighting effect on the models there seems to be produced from the odd cubemaps that are in globallighting. Definitly looks like one of the values looks to calculate that environment mapping type effect is messed up. Changing the second and third blocks doesnt seem to have any effect that I've noticed, but I seem to be missing a bit lately.

I've got a 9600gt so you should be able to reproduce anything I can, are you sure you have the game set up to use modded files? I find using --io.file_first in a bat file works better than a cfg file.

KnightErrant
03-10-2007, 16:37
@GOM

Of course, you have my permission to do anything
that helps figure things out.

This Saturday is one of those errands and yardwork days
(spring has come to Huntsville) but I'll start back again
tonight looking at things. Maybe I should make my cobbled
together script more general so it searches for the start of
the mystery block for any unit rather than a hard-coded
byte value. Any thoughts on whether that would be helpful?

@Casuir
The first picture you posted looks exactly like what I'm
seeing, shadows and stuff on different guys in the unit
depending on lighting.

KE

GrumpyOldMan
03-11-2007, 02:11
Hi Guys

I finally made my in-game stuff look like yours. Looks like the mystery blocks are not just harmless boolean streams as we were led to believe. I've sent a PM to Caliban and hopefully we can get some more information.

What this means -

Can we still go ahead? - Yes, with the restriction that we can only use M2TW groups until we figure out how to replicate the shading data. However, creative texturing and use of alpha maps will give some leeway for variations.

Is Milkshape still an option? Yes, I can store the mystery blocks within the file and parse it out on conversion to Mesh.

All is not lost and the way ahead is clearer now :yes:

Off to mow the lawn, then rewrite the converter :juggle2: and read up on shader streams :book: . :dizzy2: :dizzy2: :dizzy2: :dizzy2:

Cheers

GrumpyOldMan

alpaca
03-11-2007, 14:47
Heh you guys got me interested. Is any of you up to brief me a bit on this via MSN, ICQ or IRC later next week?
I could try to help you figure out the mystery values afterwards (well I have no idea about 3d programming but I'm good at puzzles) - and I prefer chat to forums :shame:

KnightErrant
03-11-2007, 16:13
@alpaca
Well GrumpyOldMan is the most knowledgable person on the
team about 3D graphics but if he's too busy I can give you
my less in-depth understanding. You'd have to tell me how to do
the protocols you mentioned; I've haven't used chat.

@anybody
This is a noobie request but I never learned how to post screenies
and this will be necessary for dealing with the mystery data.
Could someone point me to a tutorial or give a brief description.
Please include something about hosting services.
Thanks.

zxiang1983
03-11-2007, 17:29
@KE, Let me do this easy job:)
1) I suggest this site to host your image:
https://imageshack.us/

2) The webpage is very user-friendly. Click "browse" button on the right and choose the image you want to upload and then click "host it!" at the bottom

3) After all is done you will get several address. Just copy the "Hotlink for forums" line and paste it in you post. Then others can see your image.

KnightErrant
03-11-2007, 17:50
@zxiang1983
Many thanks, my friend. I'll
try a test this afternoon.

edit: that was fast trying it now.

https://img178.imageshack.us/img178/9957/testfeudalknightiz2.jpg (https://imageshack.us)

Wow! That was easy! Thanks again!

This isn't the best picture of the mystery data effect but some of the guys are shadowed
because of the zeroed out mystery data.

Foz
03-11-2007, 20:54
Could it have something to do with reflections? It occurs to me that there's been no discussion yet of how the engine determines the reflective properties of a given mesh, yet that is almost certainly something you'd want to include, as the various other mesh aspects you guys have discussed can't really tell you the reflective properties of the materials involved.

Re Berengario I
03-11-2007, 21:07
Which parts of the mesh are reflective is handled by the alpha channel of the normal map texture, but those "shadows" seems related at what color is reflected (in 3ds max would be the specular color). Usually an environment map is used for reflections, the testing is easy though, just change the normal texture and see what it happens to the new reflective parts.

Caliban
03-12-2007, 00:41
Those screenshots you posted Casuir look like the envmap (cubemap) at full strength. If you turn the unit's normal map alpha to white, you will get the same effect. When we create meshes in 3dsmax, we also specify the material shader per lod. Each lod has it's own shader so we can control the amount of pixel/vertex shaders on each lod for performance reasons. When these are set, shaders are turned down or off depending on the distance lod.

I have sent a bit of info off to our programmers to find out more about the mystery data.

KnightErrant
03-12-2007, 00:49
This baby ain't cracked yet....But it's a crackin'

Maybe doing yardwork was a good idea this weekend, I finally found the
misplaced 2 byte in the granny strings. Most frustrating side-by-side
hexedit session ever. Anyway I got the conversion from .mesh to .ms3d
done last Wednesday and pulled in the feudal_kinghts_lod0.mesh file and
changed the lance as so:

https://img463.imageshack.us/img463/6977/milkshapemoddedfeudalknzw6.jpg


Today I finally got the reverse process to work. Here's the modded
file in a custom battle. Look at the lances in the foreground.

https://img463.imageshack.us/img463/7052/moddedfeudalknightslod0kh1.jpg


Note that most of the bone info is hard coded into the
script. The .mesh to .ms3d relies on the hierarchy tree that GOM
sent me for the .smd format. The other way is also hard coded and
very brittle. You can't make the slightest error with the header/footers
or granny strings.

Please regard this as a proof of concept only. There's a lot more work
ahead but like GrumpyOldMan said, things are looking clearer. If Caliban
cracks the whip to get info on the mystery data I'm very hopeful.

Permit me a little happy modder's dance here: :dancing:

KE

Edit: Hi Caliban, nice to see you. Yes, please more info on the mystery data!

Caliban
03-12-2007, 01:37
Well I should have checked my email first thing this morning, I had already been sent a bit of info about this problem:


Character models use this vertex format:
Position : float[3]
Normal : byte[4] // only first 3 bytes are used for xyz, last one is 0
TexCoord : float[2]
Tangent : byte[4] // same format as normat
Binormal : byte[4] // same format as normat
BoneIndiciesAndWeight : byte[4]

so there are streams that each mesh should have. They need to find out which they know of and the mystery blocks will likely be the rest of them.

These are the IDs that should be used to mark them in the file:

Position: 0
BoneIndiciesAndWeight : 2
Normal : 3
TexCoord : 4
Tangent : 10
Binormal : 11

Without analyzing their data I've noticed they've been highlighting number 3 somewhere, which could mean it's normals.
I think only lod0 has all that info, other lods probably drop tangent and binormal, the rest should be there. In theory they could mark even lod0 to use lod1 material and get away with not having these two components, but that would remove per pixel normal mapping and replace it by just vertex lighting.

To get more info about what tangent and binormal is, they should look up tangent space normal mapping as that's what we're using. Mesh needs to contain valid tangent space (normal, tangent & binormal). :juggle2:

Nice lances KE, are they for charging around corners :)

KnightErrant
03-12-2007, 01:52
Hi Caliban,
Don't know, I was too excited to let the battle play out.:jumping:
Maybe they just play bumper cars with them.

Thanks for the info. The 3 precedes the first mystery
block and the 10 and 11 precede the following two from what I've seen.
So that's normal, tangent, and binormal. Well I've been computing
vertex normals to go into the .ms3d format so that leaves tangent
and binormals. I thing I'm relieved this is geometry stuff rather than
something else because we should be able to compute that. ~:)

Edit: Wait, I don't know what the bleep I'm talking about. Those are
byte streams so are they just low resolution vectors, meaning we
compute a real live vertex normal in float and then coarse map it
to 0-255 for each x,y,z component?

Re Berengario I
03-12-2007, 02:43
No, normal vertices using normal maps have native values ranging from 0 to 255 because they used the RGB channells of a bitmap (the "normal map" exactly) to store the data.

If you open a normal map of MTW2 you'll see by yourself what I'm talking about...

This is an example taken from this forums

https://img201.imageshack.us/img201/689/t4by0.jpg

KnightErrant
03-12-2007, 02:55
Yes, I've opened one of them before but didn't understand
what I was seeing (still don't). But anyway, I think you answered
my question, I don't know what Caliban just told me. We have
normal, tangent, and binormal data in the mystery blocks and these
are indices into the "normal map" but I don't know what that is.
Someone tried to start a tutorial on the normal map over at twcenter
but I think I scared him off by asking too many questions.
I think I'm the most "fish out of water" player in this whole problem; I'm working
on 3D graphics with absolutely no knowledge of 3D graphics.

Edit: Accidentally posted early trying to preview. Meant to follow up with:
Can you recommend a good reference on this subject so I can educate
myself?

Re Berengario I
03-12-2007, 03:01
Ok a normal map is simply a way to fake geometries which don't exist.

Pratically a normal map "displaces" the normal of a vertex using the values contained in a RGB image so to achieve a 3D bumping effect.

The RGB colors (each 8 bits/1 byte in a 24 bit image) are in reality the XYZ of the normal that is used to create the effect.

From Wikipedia:
the red channel should be the relief of the material when lit from the right, the green channel should be the relief of the material when lit from below, and the blue channel should be the relief of the material when lit from the front(practically, full except on the "slopes"); or, to put it another way, the XYZ coordinates of the face normals are placed in the RGB values of the normal map. If a material is classified as being reflective, the albedo is usually encoded in the alpha channel if one exists.

This is the easiest way I know to explain what a normal map is.

KnightErrant
03-12-2007, 03:08
Ok, becoming clearer now. Most of the mystery data
has three byte values followed by a zero byte but Casuir reported
about a few instances where the fourth byte was non-zero,
should I assume that might be a case where the albedo
is being encoded in the alpha channel, assumed to be
the fourth byte?

Suraknar
03-12-2007, 03:12
hrm...

This looks like Vertex Shading, Shininess and refraction in relation to weather lighting and environment, so it is information for the engine's rendering instructions.

Just a thought, are these mystery values the same accross the board for all units?

If they are then maybe the Converter could just add(insert) this information to the final .mesh file upon export?

EDIT: NVM, I posted this after reading page 5 of the thread, did not notice there was Page 6, which I am reading now :P

KnightErrant
03-12-2007, 03:17
Yes, the format seems to be the same for all units.
The aztec variants that Casuir found had float triple
streams for the mystery data but those units aren't
used in-game (don't appear in the EDU) so were they
mistakes, early prototypes with a different format, etc. ?

Did a quick google on normal map tutorials and there
are a few so maybe I should just read up for a few evenings
until I can catch up.:embarassed:

Re Berengario I
03-12-2007, 03:25
For what you are interested in a normal map it's just another texture because most of the 3d programs know how to handle it. It's not geometry for them.

From the Caliban post I understood that the mystery block values are the tangent and biNormal (bitangent) vectors of the model to support normal mappin in per pixel lighting through shaders.

Re Berengario I
03-12-2007, 03:28
Ok, because I need to be forgiven for the last cryptic post this is a good and simple tutorial about the argument:

http://www.3dkingdoms.com/tutorial.htm

and this for tangent space normal mapping

http://www.3dkingdoms.com/weekly/weekly.php?a=37

KnightErrant
03-12-2007, 03:35
Well when GrumpyOldMan comes online he'll understand this
so I guess I shouldn't be too worried about my ignorance.
What I would like is an assurance of is this data computable
from the vertex vectors, weights, and uv coords so that when
we pull all of that out of a mesh and put it into the .ms3d
format (which wants the vertex normals) can we go back the
other way and put it all back into the mesh format and compute
all the required mystery data. (I realize we may be "talking past each other"
on this question, I see this problem from data in a file format and you
guys who actually make models may be seeing this as something
you apply in a given application.)

Edit: You're too fast, I didn't even get a chance to see that.
I'm going off to study this stuff and then I'll understand it better.
Many thanks for being patient with me.

2nd Edit: Yes, that's the tutorial I found. Reading it now.

3rd Edit: THANK YOU! This has math in it. Now I'll understand what we've
been talking about.

Re Berengario I
03-12-2007, 03:40
I don't possess deep knowledge on normal mapping as I usually let the 3d application handle it but in this case they use directly a shader so those values (if I'm not completely wrong) are a 3x3 matrix for each mesh vertex.

(From the second link above:)
To use tangent space normal mapping, we need to send the tangent and bitangent vectors to the vertex shader. Then we can create a 3x3 rotation matrix from them that can be used to rotate vectors such as the light vector and half-angle vector into tangent space.
Now we can dot these vectors with a normal from our tangent space normal map, using the same pixel shader as with object space normal mapping.

From Caliban's post:

To get more info about what tangent and binormal is, they should look up tangent space normal mapping as that's what we're using. Mesh needs to contain valid tangent space (normal, tangent & binormal).

KnightErrant
03-12-2007, 04:01
Exactly! from the same article


void ComputeTangentBasis(
const Vec3& P1, const Vec3& P2, const Vec3& P3,
const Vec2& UV1, const Vec2& UV2, const Vec2& UV3,
Vec3 &tangent, Vec3 &bitangent )
{
Vec3 Edge1 = P2 - P1;
Vec3 Edge2 = P3 - P1;
Vec2 Edge1uv = UV2 - UV1;
Vec2 Edge2uv = UV3 - UV1;

float cp = Edge1uv.y * Edge2uv.x - Edge1uv.x * Edge2uv.y;

if ( cp != 0.0f ) {
float mul = 1.0f / cp;
tangent = (Edge1 * -Edge2uv.y + Edge2 * Edge1uv.y) * mul;
bitangent = (Edge1 * -Edge2uv.x + Edge2 * Edge1uv.x) * mul;

tangent.Normalize();
bitangent.Normalize();
}
}

This is exactly the education I was looking for. Thank you very much
Re Berengario I. I'm am deeply in your debt.

Suraknar
03-12-2007, 04:01
KE,

Just as a small test here,

when you compare feudal_knightslod0 and feudal_knights_lod1, is the mystery bytes information the same?

KnightErrant
03-12-2007, 04:11
No, in all the "normal units" I've looked (a very small subset unfortunately)
the lod1, lod2, and lod3 only have one mystery block, set off with a
granny int of 3 0 0 0 which, according to Caliban's info, means normal
info only, no tangent or binormal. Is that significant?

Edit: Sorry, if you're asking if the first mystery block is the same for
each lod I don't know. I can find out but I have to run some examples.

Re Berengario I
03-12-2007, 04:15
There's probably no normal mapping on low resolution meshes (it wouldn't make any sense anyway), so tangent space vectors are not needed.

Casuir
03-12-2007, 04:18
Casuir reported
about a few instances where the fourth byte was non-zero,
should I assume that might be a case where the albedo
is being encoded in the alpha channel, assumed to be
the fourth byte?

Just having a look back on the files I looked at before some of them are indeed the base figures, others are ingame and while they do have blanks in one of the first three values I havent seen anything in the fourth yet, its possible I was looking at a bone stream as I dont think I'd noticed that they came last in some files at that time.

So we have all the info now to make a mesh? Where do we stand on formats, is milkshape able to handle the normal/tangent/binormals?

The first block shouldnt be the same if the geometry is different btw

KnightErrant
03-12-2007, 04:35
@Casuir
Sorry, I might be misattributing from my faulty memory.
I got the transformation to go both ways but its really
highly tied to feudal_knights_lod0. In my slightly dazed
condition from all this texture mapping information people
have been giving me I can't make a determination but
I THINK if we sort it out we, I mean GrumpyOldMan, can
make a converter that goes both ways. The mystery
data seems computable, I do vertex normals, and a few
posts back was some C++ math about the tangents and
binormals that seems doable so maybe if I can compute this
and stuff it back into the mesh we have a demo of the
process?

Could you try a different unit and see if it crashes or works.
The bones stuff is way out of my league, I keep seeing funny
blue lines for bones so I don't know if this is my basic ignorance
of Milkshape or if I've reversed the vertex normals in my conversions.

Suraknar
03-12-2007, 05:12
@ KE



Edit: Sorry, if you're asking if the first mystery block is the same for
each lod I don't know. I can find out but I have to run some examples.

Yes, reading above from what Caliban has posted in combination with the excelent information Re Berengario linked, thank you RB :), I would assume that the block of information in question here would indicate a somewhat different information for eahc LOD.

Now, if this is indeed the case, then a second peace of information here would be to confirm if that information is the same between all lod0's, lod1's etc...

In other words, if the info from one unit's lod0 is the same with the info of another units lod0, and the info of one unit's lod1 is the same with anotehr unit's lod1, while being different from another lod level.

It would mean that the engine uses a uniform scheme for each lod across the board, which can simplify the task of the converter to reproducing (copy-paste) these values for each lod instead of trying to calculate them on a per modded model.

As I said before I am no programmer, so I cant verify the validity of all this but from what RB linked, I understand the process conceptually. So before continuing on what I am trying to say I'll post the relevent info.

----

So this..



What are the tangent and bitangent vectors used in tangent space normal mapping? A plane's normal is perpendicular to the plane, while the tangent and bitangent are parallel to the plane. Their direction is determined by the UV coordinates, one points in the direction of U-axis in 3d space, the other in the direction of the V-axis.

The tangent space normal map stores in each component the distance along each of these vectors. For example, one component of the normal map (usually stored in the blue part) points directly away from the mesh triangle, ie. along the triangle normal. This allows the same normal map to be reused on different parts of a mesh.

with this...



Blending the Tangent/Bitangent Vectors

Let's say you are creating smoothed vertex normals, by adding the face normals of adjacent triangles for each vertex, then normalizing the vector. You can do the same thing with tangent and bitangent vectors.

Note that the 3 vectors (normal, tangent, bitangent) may not form an orthogonal basis since the tangent & bitangent directions are based on texture uv coordinates and possibly averaged from multiple vectors. What you can do is use Gram-Schmidt orthogonalization. This is performed simply by subtracting the part of a vector in the direction of another vector then renormalizing. For example the code to make the tangent orthogonal to the normal is:

tangent -= normal * tangent.dot( normal );
tangent.normalize();

Once you have two orthogonal vectors you can use the cross product to compute the 3rd. The orthogonalization step seems to be mainly necessary on complex curved models or places with texture seems. Many texture seems however will cause weird lighting no matter what, so if the tangent/bitangent vectors are too far apart you'll probably be better off leaving a split at the seem and not smoothing them together.

And finally this...



Using Tangent Space with Shaders

To use tangent space normal mapping, we need to send the tangent and bitangent vectors to the vertex shader. Then we can create a 3x3 rotation matrix from them that can be used to rotate vectors such as the light vector and half-angle vector into tangent space.

float3x3 objToTangentSpace = float3x3( IN.Tangent, IN.Bitangent, IN.Normal );
OUT.halfVec = mul(objToTangentSpace, halfVec );
OUT.lightVec = mul(objToTangentSpace, lightVec);

Now we can dot these vectors with a normal from our tangent space normal map, using the same pixel shader as with object space normal mapping.

Are important steps to come to this information in the .mesh format that you see, as I understand it.

Added to all this is a process that the documentation calls normalize in order to better use Tangent Space with vertex and pixel shading.

----

Now returning to the idea here, that after reading page 6 of this thread I think GOM also had.

If, it is proven that the game uses a uniform scheme for handling the units and their various lod's.

Then in the converter we need to not worry about calculating and producing these values ourselves.

The converter could simply insert that information in the appropriate lod.

So the modder would start by loading an lod0.mesh file, make their desired changes, and save as .ms3d.

Then when converting to .mesh, the converter could produce 4 files instead of one.

lod0, lod1 and lod2 and lod3 respectivelly with the appropriate tangent space mapping information to each level.

maybe this is a simpler approach for all?

Casuir
03-12-2007, 05:51
@KE I wont be able to do anything with the script you sent me till later but I'll try it out on a few other models with different makeup later and see if anything looks odd.

@Suraknar, the normal/tangent/binormals are calculated per vertex so two different models are going to have two different sets of values, i.e pasting the data from a file with 4000 vertices into a file with 5000 vertices will crash the game.

Suraknar
03-12-2007, 06:42
@Suraknar, the normal/tangent/binormals are calculated per vertex so two different models are going to have two different sets of values, i.e pasting the data from a file with 4000 vertices into a file with 5000 vertices will crash the game.

So in other words that mysterious byte information is the actual per vertex calculations and not the information required by the rendering engine on how to handle them.

I was under this impression that this information was pertaining to instruct the engine on how to handle the per vertex calculations whose information is stored elsewhere in the file and would depend on each different model (obviously).

Well in that case yes it is not feasable. Continuing to read with interest here, I am a 3D artist(Maya and the older SoftImage), I understand what we talking about, yet, not necessarilly the coding parts.

KnightErrant
03-13-2007, 04:44
Okay, I want to return to tangent space normal mapping but
I'll leave out the mathematics.
Many thanks to Re Berengario I and Suraknar for pointing out
tutorials on this and data cube normalization. Caliban has told
us that the 3 0 0 0 mystery block is vertex normals, 10 0 0 0
is tangents, and 11 0 0 0 are binormals. The vertex normals
are computed by averaging over the face normals by iterating
over the triangles, the tangents are computed by conditioning
the face tangents by the uv coords according to the formulas
in the refs in the previous posts, and the binormals can be obtained
from the cross products. The confusing part is mapping to RGB
values. Basically you take all the normalized vectors, add 1.0
and divide by 2.0 to map the components onto a [0,1] RGB float
space. You then multiply by 255 and truncate to map to a
byte-valued RGB space on [0,255]. This is called cube normalization
(learned at lunch time). This is the format of the mystery
data's block 1, 2, and 3 (along with a fourth byte that looks
like an unused albedo value).

Anyway, if you do this you can get the following .mesh to .ms3d
and .ms3d to .mesh conversion for the feudal_knights_lod0.mesh.

(I'm repeating bits of post #160 and I'll use thumbnails this time,
still learning the fine points here.)

First step: Convert mesh to ms3d and modify in Milkshape:
All I've done is broken the lances.

https://img463.imageshack.us/img463/6977/milkshapemoddedfeudalknzw6.th.jpg (https://img463.imageshack.us/my.php?image=milkshapemoddedfeudalknzw6.jpg)


First try, zero out the mystery data:

https://img463.imageshack.us/img463/7052/moddedfeudalknightslod0kh1.th.jpg (https://img463.imageshack.us/my.php?image=moddedfeudalknightslod0kh1.jpg)


Okay, you can see the Milkshape mod has shown up in the game but
the shadows on the lod0 rendered knights shows the mystery data isn't
being applied.

Finally, use the data cube normalization along with the vertex normals,
tangents, and binormals to compute the mystery data out of the
data pulled from the .ms3d file and put into the .mesh format to get


https://img175.imageshack.us/img175/9865/computedtangentspacefeurz8.th.jpg (https://img175.imageshack.us/my.php?image=computedtangentspacefeurz8.jpg)


and there you have a forward and backward mesh conversion.



Slight but solvable problem: People might notice the lighting looks wrong.
That's because there's still an unresolved coordinate system problem.
Here's the vertex normals (in tangent space normal mapping) from
the original feudal_knights_lod0.mesh

(first ten of them anyway)


189 238 116 0
253 127 106 0
5 95 110 0
136 3 100 0
251 116 98 0
193 236 129 0
209 31 108 0
139 1 115 0
50 25 118 0
2 108 115 0

and here's my computed ones:


116 238 190 0
107 128 253 0
111 95 5 0
103 5 100 0
99 116 251 0
130 236 194 0
109 36 215 0
114 29 208 0
119 26 51 0
116 108 2 0

You can see that the x and z coords are reversed and that they
don't exactly match. Too tired to work this out tonight, I'll hit
it later. Must be a coordinate system thing.

Anyway, the mechanics of this are getting really clear now.
More stuff tomorrow.

GrumpyOldMan
03-13-2007, 04:57
Hi Guys

Lots of real life things happening :juggle2: , The first un-mystery block is normals, stored byte fashion as z y x 0, the algorithm to extract them is MS3D normal(x,y or z)*127.5+127.5. Remember to reverse any Milkshape 'x' values to take into account the different axes. It's not exact but close enough. This means that with the right frame work we can now extract lod1 and below meshes from Milkshape - including any new groups :beam: . M2TW will run without lod0 meshes but you won't get the eye candy.

@KE and Casuir - The Milkshape meshes that I've been extracting don't have valid normal info, I've been relying on Milkshape to update them. To update them, after loading the MS3D, go to Menu/Tools/Model Information V1.7 and after it comes up, hit the Close button rather than the 'x' in the top righthand corner and this will update the normals.

I'm going to work on the tangent and binormal/bitangent using the first triangle/vertices of the Feudal_knights_lod0.mesh. The info is as follow:-

155 154 143 (vertex number)
0.794555 0.558303 -0.0330388 0.330422 0.127918 (x,y,z,u,v)
0.794555 0.431023 -0.0131331 0.315904 0.127895
0.794555 0.482091 -0.0767073 0.322253 0.127879

I'll get out the tangent and bitangent figures later, I'm assuming that it runs z,y,x,0 as per normals.



Well when GrumpyOldMan comes online he'll understand this
so I guess I shouldn't be too worried about my ignorance.

:laugh4: :laugh4: :laugh4: You wouldn't have been so confident if you'd known that I had hot flushes after I was dragged kicking and screaming from my DX7 comfort zone, with shader stuff!!:laugh4: :laugh4: :laugh4: :laugh4:

Cheers

GrumpyOldMan

KnightErrant
03-13-2007, 05:03
Hi GrumpyOldMan!

We almost simultaneously posted! RE I and Suraknar straightened
me out and steered me to the right tutorials for tangent space stuff
and cube normalization. Caliban really came through for us on the
mystery data. I'm feeling really good about getting this to work now.
Is my coordinate space problem because Milkshape uses a system I
didn't pay attention to?

Suraknar
03-13-2007, 05:11
Many thanks to Re Berengario I and Suraknar for pointing out
tutorials on this and data cube normalization.

I did not do much really, except read the docs RB linked and point to various areas, while making assumptive hypothesis. :P

The great thing here is that there is much progress which is extraordinary!:2thumbsup: :2thumbsup: :2thumbsup:

KnightErrant
03-13-2007, 05:19
Yes, I'm dancing off the walls at this moment thinking we've
got the procedures almost nailed down. ~:cheers:
For me, I think if GOM has a plan for handling the bones
stuff by auxilliary files to handle variants like mounts
and artillery which probably don't follow the canonical
unit assignments, then I don't see any technical obstacles
like I worried about with the mystery data.

Woo Hoo! exponential [BIG NUMBER}

GrumpyOldMan
03-13-2007, 06:49
Hi KE


Hi GrumpyOldMan!

We almost simultaneously posted! RE I and Suraknar straightened
me out and steered me to the right tutorials for tangent space stuff
and cube normalization. Caliban really came through for us on the
mystery data. I'm feeling really good about getting this to work now.
Is my coordinate space problem because Milkshape uses a system I
didn't pay attention to?

The coordinate issue relates to how the MESH normal data is written, it's similar to littleendian integers, shorts, etc., with the bytes being written from right to left. Looking from the right it's 0,x,y,z. But if we try to read it from the left like English it looks like z,y,x,0.

I wouldn't be too concerned about the numbers not exactly matching, from what I can see there is usually only a difference of 1 which means a 1.4 degrees difference in a 0-255 world. We can have a look at it but it's not really significant.

With the picture of the reconverted feudal knight, did you have the original 2nd and 3rd blocks of un-mystery code reinserted, your own converted values or is it running with just the 1st block?

Damn, this is the third edit already to this post, shouldn't type when I'm tired. @KE I came across http://graphics.cs.lth.se/theses/projects/pycgfx/report.pdf on a Python CG mode when I was looking around, Haven't looked through it yet but it might be interesting.

Cheers

GrumpyOldMan

SirAnen
03-13-2007, 09:53
Hello , i have been watching this subject form the begining and i like to help.
As a discoverd the system is based on matrix factors.

The coordinate issue relates to how the MESH normal data is written, it's similar to littleendian integers, shorts, etc., with the bytes being written from right to left. Looking from the right it's 0,x,y,z. But if we try to read it from the left like English it looks like z,y,x,0.

This the problem of 3d modeling. UW warp have x,y,z but as you know when you do a mirror you get the same x,y,z but backwords. 0 is the point of the model.

If you need any help with matrix , give ma a sign :)

Btw. I would be pleased to send me some model beacuse i don't have EN edition , my PL one dosn't want to be edited. I would be pleased with some arm men with chainmails and some crusedres units.

Sorry for my english.

KnightErrant
03-13-2007, 15:10
Sorry, I've been falling asleep without logging out the past few nights. :laugh4:

The back converted mesh has all computed values for the normals,
tangents, and binormals for blocks 1, 2, and 3. So it's a consistant
right handed system, it just has a disconnect with Milkshape's convention.
I think I knew this a few days ago because my figures came into Milkshape
lying on their sides so I did a few swaps to make them look right in Milkshape
but then I forgot about this going backwards so the geometry or meshes
are ok but the auxilliary data is reversed. I'll fix it up tonight.
I agree I'm not going to worry too much about a difference of 1 in
the tangent space normal maps because 1/255 = 0.4% so who's going to
even be able to notice that.

I took a quick look at the link and I have to say I never thought Python
would be used to program a shader. It looks like they can make
OpenGL and DirectX calls for vertex shaders, pixel shaders, bump mapping,
environment mapping, etc. Not that we need this but I'm still blown away
that you can do this in an interpreted language. (Actually I think the article
says that Python gets compiled natively at some point, have to read it
carefully later.)

I think I just want to fix up a few things and then I've gone about as far
as my knowledge will take me. I'd like to write up the tangent space
normal mapping stuff to leave some documentation in the Wiki just because
it's so interesting in its own right. Otherwise I'm ready to test stuff
whenever you need it. :smash:

Andromachus Theodoulos
03-13-2007, 17:33
@All...

If you guys don't mind a complete 3D modelling newb test your work as well, I am here...

I work with 3D CADD programs, but I am not sure how similar in function they would be with 3D animation/gaming.

Let me know!!

AT

Bwian
03-13-2007, 18:46
Does this mean what I think it nmeans?

Knight Errant has managed to IMPORT a model from the game, mess about with it in Milkshape 3D, and then EXPORT a completely funtional model with skeleton and texture intact back out and have it work perfectly in game?

Damn fine work!

Assuming this is a stable process, and judging from the tech-speak flying around, you guys definitely seem to understand how the process works, how near are we to a tool that can be tested by us polygon pushers?

I am assuming we can import the skeleton intact and use MS3D to rig a mesh for export? Can't wait to try this out!!!!!

KnightErrant
03-13-2007, 20:08
Hi AT and Bwian,

The only technical thing we were waiting on was what we
called the mystery blocks but now that we know they are
vertex normals, tangents, and binormals in a cube normalized
RGB format that's the last unknown.

My stuff is all hard coded to feudal_knight_lod0 and was only meant to
test the conversion algorithms from the data layout of the .mesh to
the data layout of an .ms2d and back again and getting
the granny strings right. But now we know it works.

I'd say like a week or so, but then again don't let me put words
in GrumpyOldMan's mouth. Stay tuned, we'll hear good things soon!

GrumpyOldMan
03-14-2007, 02:41
Hi Guys


I am assuming we can import the skeleton intact and use MS3D to rig a mesh for export? Can't wait to try this out!!!!!

As a break from wrestling with the Tangent and Binormals, I thought I'd try out the size of RTW meshes with the M2TW skeleton, here's a pic https://i150.photobucket.com/albums/s104/grumpyoldman2007/rtw_2_m2tw.jpg .

The size overall is pretty good but the head needs alignment, especially with the Jaw and Eyebrows. I hope the Warhammer guys don't mind me using one of their .cas files :sweatdrop: . On the whole it would be easier to use M2TW parts and retexture than to completely rerig a skeleton and mesh, but it would be possible.

Well, that was a nice break, I'll go back and see if I can find out what I'm doing wrong :dizzy2: :dizzy2: :dizzy2: .

Cheers

GrumpyOldMan

alpaca
03-14-2007, 15:57
Huh did you write a cas importer for Milkshape then?

@KE: ~:cheers:

Andromachus Theodoulos
03-14-2007, 18:39
@ Alpaca...

It looks as though he did something to get the old cas model into ms3d... revolutionary!!!

@GOM...
I was curious if there was a scale/size difference was between the two models, because I have notice that I have had to scale textures I used from RTW up several times (1.5 if I remember right) to get the to be at the right "scale" with the M2TW models.

I used the lorica segmentata from the western Roman auxilia palatina in RTWBI and merged it with the Italian spear militia texture in M2TW. I have done the same with adding scale mail to the same model... I also did this with the Tercio pikemen (not historically accurate, but looks cool...)

So it is interesting that the models appear to be the same size, which is great for switching textures back and forth, this will save a lot of work, not having to develop things from scratch.

AT

GrumpyOldMan
03-15-2007, 06:47
Hi All

Not much to report, steadily tearing down my walls of ignorance :laugh4: :laugh4: .




@GOM...
I was curious if there was a scale/size difference was between the two models, because I have notice that I have had to scale textures I used from RTW up several times (1.5 if I remember right) to get the to be at the right "scale" with the M2TW models.
AT

The mesh and skeleton were shown as-is, store-bought, off-the-shelf. The mesh retains it's UV texture values so it uses the original texture. After the major work has been done, I'll look at a CAS to MS3D converter possibly reassigning the vertices to the equivalent bones in M2TW. I also have to look at a converter to extract animation data back to M2TW CAS format (I've already got the M2TW CAS to MS3D roughly done).

Cheers

GrumpyOldMan

Bwian
03-15-2007, 09:33
Being able to change animations would be a very nice thing to have the ability to do again! There are lots of things I would like to be able to do with the animations to make unusual units!

I presume that would also open up the possibility of scaling the skeleton and mesh to allow over/undersize units as well.

Mind you.... if it were possible ( and in RTW it wasn't ) to actually re-site the bone joints in MS3D and have that preserved in the export, that would be so absolutely brilliant I would pay money for it ( and my wallet has armed guards on it! ). I am guessing the way RTW handled such things that it was not physically possible. The skeleton seemed to be fixed in the DAT file, and that was it. If we were able to actually re-arrange the bones in a modified skeleton we would be able to make accurate and varied fantasy units, and really make some fabulous mods!

RTW 'Dwarves' were just scaled down, and not anywhere near bulky enough. Goblins and Orcs were re-skinned humans. Yuck.

I am assuming that the actual location of the bones relative to the root must be stored somehow, and that the animations were recording rotational changes in each joints relative to it's parent. In RTW we never got to re-write the start points..... Can this be done for MTW2?

Re Berengario I
03-15-2007, 10:38
With Granny libraries taking care of animations in MTW2 I HIGHLY doubt that we'll ever being able to change animations.

I'll be happy if someone at CA could say I'm wrong but I already saw this movie in other games using Granny.

alpaca
03-15-2007, 12:23
Animations are still in .cas format I think so it might be possible to edit them.

Lusted
03-15-2007, 12:32
Yeah they're still in .cas format though vercs old import script for them doesn't work.

quadrille
03-15-2007, 18:07
I hope the Warhammer guys don't mind me using one of their .cas files :sweatdrop:

Not at all! :)

I'm thrilled by your work, as has been said before you guys will be heroes in the community!

I agree with what Bwian said about animations and skeletons. Not being able to edit bone positions and add new skeletons is part of what killed the warhammer mod. If we were able to do that with M2TW the sky would truly be the limit.

Caliban
03-16-2007, 01:40
Yeah they're still in .cas format though vercs old import script for them doesn't work.

I managed to import the strategy map models general, assasin etc but it fails on "unable to convert: undefined to type: Float" on this section of the script:
rot = ( point3 (ReadFloat fp) (ReadFloat fp) (ReadFloat fp) ) --)--rtw2maxVector(

The .CAS animation format is still the same from RTW. I tried a few tests with Verc's import script and I think it may just need updating to support M2 rigs.
This could be because of the different skeleton setup being used. I'm able to load the Medieval2 animations in the old RTW animation viewer. I'm just speculating here, but Verc's script is probably geared towards only loading specific RTW skeleton setups. Updating the script should allow the M2 animations to be imported again. The models I released in this thread should give some clues about the differences.

GrumpyOldMan
03-16-2007, 02:59
Hi All

As I and Tangent-Binormal head back to our corners ready for the next round, I read through the last couple of posts.



Mind you.... if it were possible ( and in RTW it wasn't ) to actually re-site the bone joints in MS3D and have that preserved in the export, that would be so absolutely brilliant I would pay money for it ( and my wallet has armed guards on it! ). I am guessing the way RTW handled such things that it was not physically possible. The skeleton seemed to be fixed in the DAT file, and that was it. If we were able to actually re-arrange the bones in a modified skeleton we would be able to make accurate and varied fantasy units, and really make some fabulous mods!

RTW 'Dwarves' were just scaled down, and not anywhere near bulky enough. Goblins and Orcs were re-skinned humans. Yuck.

I am assuming that the actual location of the bones relative to the root must be stored somehow, and that the animations were recording rotational changes in each joints relative to it's parent. In RTW we never got to re-write the start points..... Can this be done for MTW2?



I agree with what Bwian said about animations and skeletons. Not being able to edit bone positions and add new skeletons is part of what killed the warhammer mod. If we were able to do that with M2TW the sky would truly be the limit.


The question is whether the skeleton.dat and skeleton.idx take precedence over extracted files (ie does io.file_first work with skeletons)? If not then there may be a chance. The bone locations are stored in the animation folder in the files with no file suffix. There is no central location for bone positions. In RTW all the bone locations were stored within the .CAS files. If you keep the number and name of bones consistent with M2TW you should be able to record a new skeleton, and if that works you'll have to come up with all new animations (standard animations won't work) because inside this file are also the various unit/figure states or actions and the required animations. You'd also have to see if the .MESH bounding sphere affects placement.


I managed to import the strategy map models general, assasin etc but it fails on "unable to convert: undefined to type: Float" on this section of the script:
rot = ( point3 (ReadFloat fp) (ReadFloat fp) (ReadFloat fp) ) --)--rtw2maxVector(

The .CAS animation format is still the same from RTW. I tried a few tests with Verc's import script and I think it may just need updating to support M2 rigs.
This could be because of the different skeleton setup being used. I'm able to load the Medieval2 animations in the old RTW animation viewer. I'm just speculating here, but Verc's script is probably geared towards only loading specific RTW skeleton setups. Updating the script should allow the M2 animations to be imported again. The models I released in this thread should give some clues about the differences.

Hi Caliban

Verc's CAS importation script won't work with M2TW anims without alteration. I had a look at the animations in Oct/Nov when the first demo came out. Here's a quote with the details of the changes to the animation format:-


Hi All

The animation files are in a bit of a different format from RTW ( as well as all biped figures having different skeletons - no cloak bones and additional bones for jaw, eyebrow and two clavicles between torso and upperarms).

The initial part of the animation files are the same, with no. of frames, no. of bones, frames x bones x quat values for rotations, but then there frames x bones x local translations (but this also includes the 'Y' or perpendicular to ground plane movement of the pelvis bone). After this there are frames x 'X' and 'Z' values for movement across the ground plane by the pelvis.


It shouldn't be too much of a job to alter the script by anyone who knows a bit of Maxscript (not me unfortunately).

Back to the forge.

Cheers

GrumpyOldMan

Caliban
03-16-2007, 08:33
The question is whether the skeleton.dat and skeleton.idx take precedence over extracted files (ie does io.file_first work with skeletons)? If not then there may be a chance.

Hi Caliban

Verc's CAS importation script won't work with M2TW anims without alteration. I had a look at the animations in Oct/Nov when the first demo came out. Here's a quote with the details of the changes to the animation format:-

It shouldn't be too much of a job to alter the script by anyone who knows a bit of Maxscript (not me unfortunately).


Hi Grumpy,

The format still loads in our RTW viewer so I'm guessing if there are changes they were minor enough for them to still be compatible with our viewer tool.

When I'm changing descr_skeleton.txt or the animation files directly, I have to delete the .dat and idx files from the data\animation folder for the new changes to take effect. The game re-builds them on startup with the new changes in place. I hope this is still the case with retail version.

Caliban

GrumpyOldMan
03-16-2007, 09:44
Hi Caliban



When I'm changing descr_skeleton.txt or the animation files directly, I have to delete the .dat and idx files from the data\animation folder for the new changes to take effect. The game re-builds them on startup with the new changes in place. I hope this is still the case with retail version.



That's interesting, could be something worth pursuing after this and maybe requires another thread. :juggle2: One thing I was going to ask, does M2TW have a built-in sprite creater like RTW? Hopefully yes, so that new figures can be covered by new sprites.

Cheers

GrumpyOldMan

Bwian
03-16-2007, 12:07
The game definitely does NOT rebuild the DAT and IDX files on startup in retail. This was something that was gone into in depth for RTW, and we all tried it. The game just cannot find the animations and crashes with an error.

It was thought the extra event files were to blame, since they were not present in retail, and these were supplied to us by a CA source ( Jerome Grasdyke) to test this theory. That failed too. You simply cannot run the stock RTW retail exe file without the DAT and IDX files present, and it won't reference the skeleton.txt file to re-create the DAT from available files.

Or...at least... no-one ever actually got it to work!

Lusted
03-16-2007, 12:12
One thing I was going to ask, does M2TW have a built-in sprite creater like RTW? Hopefully yes, so that new figures can be covered by new sprites.

Already asked Caliban that, it is in the .exe, it just needs to be exposed. I asked too late for it to be added in 1.2, but it should be added in the next update.

GrumpyOldMan
03-17-2007, 02:13
Hi all


The game definitely does NOT rebuild the DAT and IDX files on startup in retail. This was something that was gone into in depth for RTW, and we all tried it. The game just cannot find the animations and crashes with an error.

That's a shame, has anybody tried altering the information within the animation and skelton pak and idx files? Something like give spearmen swordmen primary weapon anims. If this works, it means there is no CRC or file length check done within M2TW to make sure these files haven't been altered, and if this is the case then they CAN be altered and possibly new skeletons added. I think V's source code for his Xidx and Xpak are floating around somewhere.


Already asked Caliban that, it is in the .exe, it just needs to be exposed. I asked too late for it to be added in 1.2, but it should be added in the next update.

Shame that means that people are going to have to wait a few months to get the full benefit of any new figures.

Cheers

GrumpyOldMan

KnightErrant
03-17-2007, 05:29
Well not much progress on resolving the tangent space basis. Unweighted vertex normal calculations still show differences from CA's .mesh values. I've found several algorithms for doing weighted vertex normals but they all seem to have little effect on the bad vertices, and make some of the good vertices worse. Here's what I see if I just do unweighted vertex normal computations:


Original RGB Computed RGB Ref. Original tangent space vector Computed tangent space vector
z y x z y x count mag z y x mag z y x Angle
188 119 15 189 120 16 [ 6] +1.00407 ( +0.47451, -0.06667, -0.88235 ) +1.00000 ( +0.48075, -0.06093, -0.87474 ) 0.00000
197 84 29 197 85 30 [ 7] +1.00517 ( +0.54510, -0.34118, -0.77255 ) +1.00000 ( +0.54769, -0.33427, -0.76701 ) 0.00000
246 82 127 234 88 69 [ 2] +0.99558 ( +0.92941, -0.35686, -0.00392 ) +1.00000 ( +0.83236, -0.31026, -0.45926 ) 27.61001
132 137 0 132 137 0 [ 7] +1.00339 ( +0.03529, +0.07451, -1.00000 ) +1.00000 ( +0.03723, +0.07653, -0.99637 ) 0.00000
167 113 7 168 113 7 [ 8] +1.00106 ( +0.30980, -0.11373, -0.94510 ) +1.00000 ( +0.31579, -0.11333, -0.94203 ) 0.00000
209 118 30 209 118 30 [ 7] +0.99946 ( +0.63922, -0.07451, -0.76471 ) +1.00000 ( +0.64171, -0.07339, -0.76343 ) 1.88787
238 121 65 239 121 66 [ 7] +0.99700 ( +0.86667, -0.05098, -0.49020 ) +1.00000 ( +0.87373, -0.04966, -0.48388 ) 4.47212
210 139 30 210 139 31 [ 6] +1.00578 ( +0.64706, +0.09020, -0.76471 ) +1.00000 ( +0.64708, +0.09066, -0.75701 ) 0.00000
72 130 12 73 131 12 [ 6] +1.00523 ( -0.43529, +0.01961, -0.90588 ) +1.00000 ( -0.42872, +0.02385, -0.90312 ) 0.00000
67 157 19 67 157 19 [ 5] +1.00143 ( -0.47451, +0.23137, -0.85098 ) +1.00000 ( -0.47331, +0.23332, -0.84943 ) 0.00000
21 125 56 21 126 57 [ 8] +1.00627 ( -0.83529, -0.01961, -0.56078 ) +1.00000 ( -0.83194, -0.01543, -0.55465 ) 0.00000
79 127 9 79 128 10 [ 6] +1.00425 ( -0.38039, -0.00392, -0.92941 ) +1.00000 ( -0.37902, +0.00296, -0.92538 ) 0.00000
142 241 72 143 241 72 [ 6] +0.99743 ( +0.11373, +0.89020, -0.43529 ) +1.00000 ( +0.12088, +0.89312, -0.43327 ) 4.13269
168 205 35 169 205 35 [ 5] +0.99835 ( +0.31765, +0.60784, -0.72549 ) +1.00000 ( +0.32366, +0.60825, -0.72475 ) 3.30577
192 164 24 192 165 24 [ 7] +0.99841 ( +0.50588, +0.28627, -0.81176 ) +1.00000 ( +0.50739, +0.29189, -0.81078 ) 3.24299
116 205 27 117 206 28 [ 6] +0.99946 ( -0.09020, +0.60784, -0.78824 ) +1.00000 ( -0.08546, +0.61458, -0.78421 ) 1.95207
130 160 4 130 161 4 [ 6] +1.00180 ( +0.01961, +0.25490, -0.96863 ) +1.00000 ( +0.02185, +0.26097, -0.96510 ) 0.00000
137 133 0 138 134 1 [ 7] +1.00370 ( +0.07451, +0.04314, -1.00000 ) +1.00000 ( +0.08135, +0.04980, -0.99544 ) 0.00000
99 232 61 100 233 61 [ 7] +0.99687 ( -0.22353, +0.81961, -0.52157 ) +1.00000 ( -0.21910, +0.82662, -0.51835 ) 4.55719
73 202 39 74 203 40 [ 5] +1.00296 ( -0.42745, +0.58431, -0.69412 ) +1.00000 ( -0.42043, +0.59120, -0.68827 ) 0.00000
69 165 20 69 165 21 [ 6] +1.00394 ( -0.45882, +0.29412, -0.84314 ) +1.00000 ( -0.45738, +0.29634, -0.83844 ) 0.00000
79 237 84 79 237 85 [ 5] +0.99934 ( -0.38039, +0.85882, -0.34118 ) +1.00000 ( -0.38033, +0.86202, -0.33506 ) 2.12102
47 206 67 47 206 68 [ 6] +1.00143 ( -0.63137, +0.61569, -0.47451 ) +1.00000 ( -0.63105, +0.61812, -0.46873 ) 0.00000
26 169 61 27 169 61 [ 6] +1.00584 ( -0.79608, +0.32549, -0.52157 ) +1.00000 ( -0.79021, +0.32606, -0.51890 ) 0.00000
247 85 127 213 98 38 [ 3] +0.99477 ( +0.93725, -0.33333, -0.00392 ) +1.00000 ( +0.67328, -0.23465, -0.70117 ) 44.60200



The RGB values of the vertex normals are shown CA first then my computed values, then translated into float via the inverse cube normalization, again CA first then my computed values. The float normals have their magnitudes shown before the vectors just to verify they are unit normals. The Figure Of Merit is the last column, this is the angle in degrees between CA's original number and what I'm getting just from geometry. (These are for the mailed_knights_lod0.mesh, got tired of looking at the feudal_knights.) The two bad normals are shown in red. The tangents for the first twenty-five vertices are next:


239 127 188 239 128 189 [ 6] +0.99496 ( +0.87451, -0.00392, +0.47451 ) +1.00000 ( +0.87642, +0.00192, +0.48154 ) 5.77339
228 116 204 228 116 205 [ 7] +0.99471 ( +0.78824, -0.09020, +0.60000 ) +1.00000 ( +0.79155, -0.09001, +0.60444 ) 5.89615
127 127 255 183 109 241 [ 2] +1.00002 ( -0.00392, -0.00392, +1.00000 ) +1.00000 ( +0.43474, -0.14849, +0.88823 ) 27.48816
254 130 132 255 131 133 [ 7] +0.99298 ( +0.99216, +0.01961, +0.03529 ) +1.00000 ( +0.99888, +0.02624, +0.03934 ) 6.80755
248 130 167 248 130 168 [ 8] +0.99477 ( +0.94510, +0.01961, +0.30980 ) +1.00000 ( +0.94870, +0.02128, +0.31547 ) 5.86652
225 133 209 225 133 209 [ 7] +0.99761 ( +0.76471, +0.04314, +0.63922 ) +1.00000 ( +0.76685, +0.04576, +0.64019 ) 3.96203
189 128 239 189 129 239 [ 7] +0.99872 ( +0.48235, +0.00392, +0.87451 ) +1.00000 ( +0.48482, +0.00840, +0.87457 ) 2.91006
223 136 210 224 136 211 [ 6] +0.99205 ( +0.74902, +0.06667, +0.64706 ) +1.00000 ( +0.75444, +0.06709, +0.65293 ) 7.23046
242 134 73 243 135 73 [ 6] +0.99589 ( +0.89804, +0.05098, -0.42745 ) +1.00000 ( +0.90249, +0.05707, -0.42692 ) 5.21208
239 128 65 239 129 66 [ 5] +1.00253 ( +0.87451, +0.00392, -0.49020 ) +1.00000 ( +0.87478, +0.01106, -0.48439 ) 0.00000
198 131 21 198 132 21 [ 8] +1.00210 ( +0.55294, +0.02745, -0.83529 ) +1.00000 ( +0.55394, +0.03452, -0.83184 ) 0.00000
245 136 79 245 136 79 [ 6] +0.99922 ( +0.92157, +0.06667, -0.38039 ) +1.00000 ( +0.92321, +0.06987, -0.37790 ) 2.28326
233 146 196 233 147 196 [ 6] +0.99718 ( +0.82745, +0.14510, +0.53725 ) +1.00000 ( +0.82806, +0.14999, +0.54020 ) 4.31214
238 140 188 239 141 188 [ 5] +0.99292 ( +0.86667, +0.09804, +0.47451 ) +1.00000 ( +0.87271, +0.10401, +0.47703 ) 6.83072
233 136 197 234 137 197 [ 7] +0.99310 ( +0.82745, +0.06667, +0.54510 ) +1.00000 ( +0.83349, +0.07262, +0.54775 ) 6.74132
254 141 124 254 141 125 [ 6] +0.99817 ( +0.99216, +0.10588, -0.02745 ) +1.00000 ( +0.99374, +0.10938, -0.02257 ) 3.48514
254 134 132 255 135 132 [ 6] +0.99409 ( +0.99216, +0.05098, +0.03529 ) +1.00000 ( +0.99753, +0.05876, +0.03847 ) 6.24797
254 128 137 255 128 138 [ 7] +0.99496 ( +0.99216, +0.00392, +0.07451 ) +1.00000 ( +0.99665, +0.00455, +0.08168 ) 5.76905
250 141 98 251 142 98 [ 7] +0.99391 ( +0.96078, +0.10588, -0.23137 ) +1.00000 ( +0.96661, +0.11154, -0.23071 ) 6.33577
239 137 67 239 137 67 [ 5] +0.99774 ( +0.87451, +0.07451, -0.47451 ) +1.00000 ( +0.87842, +0.07529, -0.47191 ) 3.86301
240 136 69 240 136 69 [ 6] +0.99675 ( +0.88235, +0.06667, -0.45882 ) +1.00000 ( +0.88585, +0.06924, -0.45877 ) 4.62367
213 128 33 213 129 33 [ 5] +0.99952 ( +0.67059, +0.00392, -0.74118 ) +1.00000 ( +0.67444, +0.01062, -0.73825 ) 1.83152
206 132 27 206 132 28 [ 6] +1.00081 ( +0.61569, +0.03529, -0.78824 ) +1.00000 ( +0.61931, +0.03756, -0.78425 ) 0.00000
199 134 22 199 135 22 [ 6] +1.00088 ( +0.56078, +0.05098, -0.82745 ) +1.00000 ( +0.56412, +0.05617, -0.82378 ) 0.00000
140 164 248 220 128 216 [ 3] +0.99236 ( +0.09804, +0.28627, +0.94510 ) +1.00000 ( +0.72236, +0.00644, +0.69148 ) 43.43275



Again, the problem vertices are in red. Since the tangents are orthogonalized via Gram-Schmidt to the normals, anything wrong in the normals gets transferred to the tangents. Finally, here's the first twenty-five binormals:


131 254 120 131 255 121 [ 6] +0.99428 ( +0.02745, +0.99216, -0.05882 ) +1.00000 ( +0.02766, +0.99814, -0.05432 ) 6.13844
162 247 100 162 247 100 [ 7] +0.99909 ( +0.27059, +0.93725, -0.21569 ) +1.00000 ( +0.27108, +0.93817, -0.21529 ) 2.44175
172 246 127 171 247 126 [ 2] +0.99279 ( +0.34902, +0.92941, -0.00392 ) +1.00000 ( +0.34378, +0.93898, -0.01128 ) 6.91230
123 254 137 124 255 137 [ 7] +0.99558 ( -0.03529, +0.99216, +0.07451 ) +1.00000 ( -0.02916, +0.99672, +0.07547 ) 5.40326
129 254 112 130 254 113 [ 8] +0.99965 ( +0.01176, +0.99216, -0.12157 ) +1.00000 ( +0.01571, +0.99333, -0.11424 ) 1.59825
129 254 116 129 255 117 [ 7] +0.99632 ( +0.01176, +0.99216, -0.09020 ) +1.00000 ( +0.01205, +0.99625, -0.08565 ) 4.92648
132 254 123 133 255 123 [ 7] +0.99341 ( +0.03529, +0.99216, -0.03529 ) +1.00000 ( +0.03936, +0.99873, -0.03142 ) 6.58847
113 254 130 113 254 131 [ 6] +0.99885 ( -0.11373, +0.99216, +0.01961 ) +1.00000 ( -0.10998, +0.99362, +0.02499 ) 2.77893
122 254 133 122 255 133 [ 6] +0.99403 ( -0.04314, +0.99216, +0.04314 ) +1.00000 ( -0.04136, +0.99809, +0.04600 ) 6.26637
140 251 154 141 251 154 [ 5] +0.99551 ( +0.09804, +0.96863, +0.20784 ) +1.00000 ( +0.10362, +0.97234, +0.20934 ) 5.43685
123 254 130 123 255 130 [ 8] +0.99298 ( -0.03529, +0.99216, +0.01961 ) +1.00000 ( -0.03198, +0.99929, +0.02017 ) 6.79705
119 254 131 119 255 131 [ 6] +0.99477 ( -0.06667, +0.99216, +0.02745 ) +1.00000 ( -0.06354, +0.99755, +0.02921 ) 5.86489
57 181 219 58 182 219 [ 6] +0.99841 ( -0.55294, +0.41961, +0.71765 ) +1.00000 ( -0.54745, +0.42407, +0.72143 ) 3.25792
80 227 190 81 228 191 [ 5] +0.99403 ( -0.37255, +0.78039, +0.49020 ) +1.00000 ( -0.36554, +0.78690, +0.49717 ) 6.29087
99 249 153 100 249 154 [ 7] +0.99903 ( -0.22353, +0.95294, +0.20000 ) +1.00000 ( -0.21876, +0.95369, +0.20644 ) 2.56410
118 227 206 118 227 207 [ 6] +0.99681 ( -0.07451, +0.78039, +0.61569 ) +1.00000 ( -0.07191, +0.78123, +0.62009 ) 4.58260
118 250 160 119 250 161 [ 6] +0.99681 ( -0.07451, +0.96078, +0.25490 ) +1.00000 ( -0.06675, +0.96356, +0.25904 ) 4.60343
126 254 133 126 255 134 [ 7] +0.99316 ( -0.01176, +0.99216, +0.04314 ) +1.00000 ( -0.00860, +0.99875, +0.04926 ) 6.71420
144 197 232 144 198 232 [ 7] +0.99279 ( +0.12941, +0.54510, +0.81961 ) +1.00000 ( +0.13290, +0.55159, +0.82346 ) 6.88748
156 229 197 156 230 198 [ 5] +0.99037 ( +0.22353, +0.79608, +0.54510 ) +1.00000 ( +0.22718, +0.80300, +0.55098 ) 7.95746
137 248 165 137 249 165 [ 6] +0.99261 ( +0.07451, +0.94510, +0.29412 ) +1.00000 ( +0.07790, +0.95257, +0.29418 ) 6.97467
208 192 202 208 192 202 [ 5] +0.99798 ( +0.63137, +0.50588, +0.58431 ) +1.00000 ( +0.63283, +0.50676, +0.58542 ) 3.63939
187 227 179 187 228 179 [ 6] +0.99496 ( +0.46667, +0.78039, +0.40392 ) +1.00000 ( +0.46715, +0.78519, +0.40651 ) 5.75704
158 247 156 158 248 157 [ 6] +0.99279 ( +0.23922, +0.93725, +0.22353 ) +1.00000 ( +0.23946, +0.94368, +0.22832 ) 6.88633
167 242 88 148 251 105 [ 3] +0.99922 ( +0.30980, +0.89804, -0.30980 ) +1.00000 ( +0.15774, +0.97206, -0.17384 ) 12.66437


Oddly, even though I've outlined the two problem vertices in red, it's not
really that much of a problem for the binormals, indicating that the
discrepancy is a rotation about the binormal axis, whatever that means.

What does this mean in-game? I don't know, here's a picture of the mailed
knights in custom battle (unimaginative, but the lod0's again are the ones
with the broken lances):

https://img339.imageshack.us/img339/2360/mailedknightslod0ex1oc7.th.jpg (https://img339.imageshack.us/my.php?image=mailedknightslod0ex1oc7.jpg)


They look ok but I'm not sure what I'm looking for. I should mention that
the tangents and binormals for CA's original data contain runs of the RGB
triples 127, 127, 127, which when inverse cube normalized become the zero
vector meaning they have a degenerate tangent space basis. What this
implies for lighting/texturing I have no idea. Since all the higher lods don't
have these bases vectors it probably isn't important.

All I've got for now.

KE

Bwian
03-17-2007, 10:43
GOM...
Gave this a try this morning. Using V's XIDX it is possible to extract the skeleton.dat and idx file, so I did that and swapped over MTW2_Spear and MTW2_Swordsman. REbuilt the file and tried running the game. It crashes with an unspecified error right at the start...so it was obviously reading the PAK and IDX files and finding something it didn't like. Don't know exactly what it was checking...but it checked something!

I tried swapping them back .... but this still crashed.

When I checked the file size against the copies I took before I messed with my files, I noticed the skeleton.dat file had got smaller..... seems the XIDX rebuild doesn't give a usable dat file, making it impossible to actually tell if the test was viable, or if the tool just didn't give a usable end product for MTW2

zxiang1983
03-17-2007, 13:00
@Bwian
Please check my post here (It's a tutorial about extracting animation files):
http://www.twcenter.net/forums/showthread.php?t=74055

The skeleton.idx/dat file is not so easy to extract as it appears.

Andromachus Theodoulos
03-21-2007, 18:29
@ All

Any new updates on this yet???

AT

KnightErrant
03-21-2007, 19:13
Hi AT,

GrumpyOldMan is working on the .ms3d to .mesh part.
I've updated my script to pull all the hard-coded stuff out.
I now have external ASCII files for the standard skeleton,
the horse, camel, and elephant skeletons. Last night I
installed Milkshape 1.8.0 which has a slightly changed format
specification so I made those changes as well. I modded the
armored sergeants and put them in a custom battle and I could
see the mods when I zoomed in on them but the moment I rotated
the camera the game crashed. At first I thought it was something wrong
with how I did the format but that doesn't make any sense because they
wouldn't have loaded at all.

Here's my best guess on this one (it's a guess because I can't check until I
get home). I've noticed for a while one of the bone names in the .ms3d
file was odd, something like bone_weapon01_shield01 or something, not the
original name out of the mesh. I though GOM had renamed it for some reason
but I never asked. It never gave me a problem because I hard-coded the
bone names in my script. I think Mete's program doesn't like joint names
that aren't unique enough and its changing that one. (The weapon/shield
names are like: bone_weapon01, bone_weapon, bone_weapon02,
bone_weapon03, bone_shield01, bone_shield.) Now that I'm not
hard coding I'm getting that bad bone name back in the mesh and the
moment an animation references the unknown name it crashes.

Before I took this out last night, I had been storing the original bone names
and node indices in the model comments along with the boundingsphere
data. But I thought I didn't need to do this anymore. If my guess is
correct then I'll just go back to storing the original names and everything
should work ok just like before. I just need to check with GOM to see
if he thinks this is the right way to handle this or not. The nice thing about
storing stuff in the comments sections is you can see the data in
Milkshape and can update things like bounding sphere numbers and
group types/group names as you add stuff and then you know it will get
written back to the mesh.

Anyway, that's my update. Except for a new episode of "Lost", I've got
a clear evening to work on this stuff.:laugh4:

KE

GrumpyOldMan
03-22-2007, 00:37
Hi KE et al


Hi AT,
Here's my best guess on this one (it's a guess because I can't check until I
get home). I've noticed for a while one of the bone names in the .ms3d
file was odd, something like bone_weapon01_shield01 or something, not the
original name out of the mesh. I though GOM had renamed it for some reason
but I never asked. It never gave me a problem because I hard-coded the
bone names in my script. I think Mete's program doesn't like joint names
that aren't unique enough and its changing that one. (The weapon/shield
names are like: bone_weapon01, bone_weapon, bone_weapon02,
bone_weapon03, bone_shield01, bone_shield.) Now that I'm not
hard coding I'm getting that bad bone name back in the mesh and the
moment an animation references the unknown name it crashes.
KE

I'm sorry, it's one of the things I had to do to get the MESH information into Milkshape, but, unfortunately, didn't explain to anybody. In the MESH file the skeleton's last few bones are stored as below:-


11 bone_pelvis 0
11 bone_rthigh 1
14 bone_rlowerleg 2
10 bone_rfoot 3
8 bone_abs 4
10 bone_torso 5
9 bone_head 6
8 bone_jaw 7
12 bone_eyebrow 8
14 bone_rclavical 9
14 bone_rupperarm 10
11 bone_relbow 11
10 bone_rhand 12
14 bone_lclavical 13
14 bone_lupperarm 14
11 bone_lelbow 15
10 bone_lhand 16
11 bone_lthigh 17
14 bone_llowerleg 18
10 bone_lfoot 19
13 bone_weapon01 20
11 bone_weapon 20
13 bone_weapon02 21
13 bone_weapon03 22
13 bone_shield01 22
11 bone_shield 22

You'll notice that the last three bones all have the same bone index '22', this is used for their attachment and weighting. Nothing would crash Milkshape faster than trying to have 3 bones all with the same index. So what I did was amalgamate the names of all three as a Milkshape bone so they could share the same index. Caliban has told me that the actual attachment of weapon/shield to any particular bone is irrelevant, I've found one attached to the bone_pelvis. It must be parsed in on the class/name string, rather than on the bone attachment. What's important is that vertices are attached to the correct index as per the Mesh format. When the figure is reconverted, the bones and indices as above should be used rather than a reading of the Milkshape skeleton.

Again apologies for not making that clear in the beginning.:embarassed: :embarassed: :embarassed:

At the moment I'm migrating my ms3d to mesh code into the final prog and testing/debugging. The final program has mesh to ms3d (fully operational, unless someone finds a bug) and Merge ms3d (fully operational, unless someone finds a bug) because I wasn't happy with the native one in Milkshape.

Cheers

GrumpyOldman

KnightErrant
03-22-2007, 00:50
Well I found the problem with the bad weapon name,
programming error on my part (usually is).:embarassed: So don't have
to store .mesh bone names in the model comments
after all. Just have to get the right bone indices on
reading out like you said.

GrumpyOldMan
03-22-2007, 01:28
Hi KE


Well I found the problem with the bad weapon name,
programming error on my part (usually is).:embarassed: So don't have
to store .mesh bone names in the model comments
after all. Just have to get the right bone indices on
reading out like you said.

As long as the information is valid, it doesn't seem to make any difference to M2TW, if you have a look at the extracted ms3d figures, most of the weapons including bows are attached to the right hand. This is how they are stored in the Mesh format and why I'm guessing it's parsed out on the class/name data.

Had another thought on the 9 bytes, are you somehow recording the carriage returns with the altered model comment format - just another wild stab in the dark:dizzy2: :dizzy2:

Back to debugging.

Cheers

GrumpyOldMan

KnightErrant
03-22-2007, 03:44
@GOM
My bad, the 9 bytes was the string bone_weapon_weapon01
I copied into the skeleton file without noticing.
The _weapon01 was the 9 bytes.

Anyway, I got armored_spearmen with bent shields, these
were the guys that crashed the game with the above bone name.
Funny thing, it doesn't crash on loading, it seemed to have to wait
until something accessed the bad name.

https://img389.imageshack.us/img389/9/armoredsergeantstx5.th.jpg (https://img389.imageshack.us/my.php?image=armoredsergeantstx5.jpg)

Also did a timurid unit equipped with the deadly Dumbo upgrade

https://img462.imageshack.us/img462/3470/dumbowj0.th.jpg (https://img462.imageshack.us/my.php?image=dumbowj0.jpg)

Was shooting for a hat trick tonight (3 for 3) but the game crashed on the
trebuchet. It looks like the artillery probably requires a skeleton file each.
Haven't looked at anything but the trebuchet but the bones seem very
specific to that unit which makes sense. I'd like to figure out what
crashed the trebuchet, probably misspelled something. I did learn
that weight is sometimes spelled as bone_wieght.:laugh4:

Will test more units to look for any surprises but it seems all the obstacles
are just about done with.

Edit: Yes, I did switch over to the \n delimiter for group comments and
model comments.

GrumpyOldMan
03-22-2007, 05:00
Hi KE

Well done!!!!!!:2thumbsup: :2thumbsup: :2thumbsup: :2thumbsup: :2thumbsup:

I probably won't have anything to show until tomorrow now, Got hit by plague of NaN's, had to go through all my code until I could find the missing '0' trap for a normalization:wall: :wall: :wall: :wall: :wall:

Once I get it all bedded down, I'll try something like adding a head in all LOD levels and see what happens. I'll also try various things like moving Milkshape groups up and down the group order and see if that throws out M2TW.

Be interested to see what you get up to with artillery.

Again, well done!!!!!!!!!!!:2thumbsup: :2thumbsup: :2thumbsup: :2thumbsup: :2thumbsup: :2thumbsup: :2thumbsup: :2thumbsup: :2thumbsup: :2thumbsup: :2thumbsup:

Edit:- Call me a Philistine of Eye Candy but I can't see any huge difference in the shading/bump mapping. Maybe we should release a MESH and get people (who unlike me, don't have tired old eyes and multifocals) to see if there is a difference?

Cheers

GrumpyOldMan

AlphaDelta1
03-22-2007, 06:20
Maybe we should release a MESH and get people (who unlike me, don't have tired old eyes and multifocals) to see if there is a difference?

That sounds like a stunning idea. And if it could be this middle eastern swordsmen mesh (dismounted Qapukulus) (http://media.pc.ign.com/media/800/800327/img_3649419.html) the entire Broken crescent mod (https://forums.totalwar.org/vb/showthread.php?t=80845) team would worship your every step on this earth. :2thumbsup:

Cheers

KnightErrant
03-22-2007, 06:58
Wow! Stunning graphics AlphaDelta1! :2thumbsup: I think I know
what to try tomorrow. The worst vertex normal problems
I've seen are the feudal knights. I'll do (it's after 12:00
my time and my wife is wondering what the h*** I'm doing
up this late) a front shot and back shot of the vanilla guys
versus the converted guys and see if the skinners see anything
that I can't see. That should settle the tangent space basis
question.

Re: granny strings. When you code this could you try out the
feudal knights? I've almost memorized the byte positions of
the blocks and would be happy to look at header/footers.
It's odd how the pattern recognition algorithm works in humans
but I couldn't "see" whoopsies in code, only in side-by-side
hexedit sessions between the back-and-forth meshes and the
originals.

Oh, to have a life again. GOM, are you familiar with the
American colloquialism "stuckee"? Once you release your
converter you're the "stuckee" for the next few months.:laugh4:

Don't plan any vacations, mate.:whip:

KE

GrumpyOldMan
03-22-2007, 07:01
Hi AD1


That sounds like a stunning idea. And if it could be this middle eastern swordsmen mesh (dismounted Qapukulus) (http://media.pc.ign.com/media/800/800327/img_3649419.html) the entire Broken crescent mod (https://forums.totalwar.org/vb/showthread.php?t=80845) team would worship your every step on this earth. :2thumbsup:

Cheers

Followed your link but all I can see is a big ad but no picture.

At this point I'm not talking about a whole new figure in all LODs (that will come as soon as we release the tools) but a modified mesh that can be plugged in straight away and used immediately. This is so we can check on the shader/bumpmapping, that we calculate a little differently (but close) to CA's figures.

I may do a repair job on some of the meshes in the ee_bekhtera_heavy_lamellar folder which since patch 1.1 have only one body variant included in the mesh.

Once the tools are released, coming up with your own meshes will be easier than all the related text entries.

Cheers

GrumpyOldMan

GrumpyOldMan
03-22-2007, 07:13
Hi KE



Re: granny strings. When you code this could you try out the
feudal knights? I've almost memorized the byte positions of
the blocks and would be happy to look at header/footers.
It's odd how the pattern recognition algorithm works in humans
but I couldn't "see" whoopsies in code, only in side-by-side
hexedit sessions between the back-and-forth meshes and the
originals.

Oh, to have a life again. GOM, are you familiar with the
American colloquialism "stuckee"? Once you release your
converter you're the "stuckee" for the next few months.:laugh4:

Don't plan any vacations, mate.:whip:

KE

I was planning on doing the Feudal Knight first up for you to have a look at and then I might do a variation by adding a Mailed Knight head to all LOD levels and see what happens.

Maybe I'll make up some stuckee rules like releasing the tools gradually with a timelimiting feature so that the person has to write a tutorial to use it more :laugh4: :laugh4: :laugh4: :laugh4: :laugh4: :laugh4: :laugh4:

We have to stop cross posting like this :laugh4: :laugh4:

Cheers

GrumpyOldMan

AlphaDelta1
03-22-2007, 09:11
No worries GoM, I was just trying my luck..'Who dares wins' etc etc.

The link should work, you just need to wait for the adverts to finish loading I think..This link might work better. (http://pcmedia.ign.com/pc/image/article/711/711054/medieval-ii-total-war-20060602104105711.jpg)

Cheers and thanks once again for what you are doing!

GrumpyOldMan
03-23-2007, 00:53
Hi AD1


No worries GoM, I was just trying my luck..'Who dares wins' etc etc.

The link should work, you just need to wait for the adverts to finish loading I think..This link might work better. (http://pcmedia.ign.com/pc/image/article/711/711054/medieval-ii-total-war-20060602104105711.jpg)

Cheers and thanks once again for what you are doing!

Saw the picture with the new link, waited for 2 minutes on broadband and still couldn't see anything with the original link.

I thought I'd try and see how long this would take to do in Milkshape, this https://i150.photobucket.com/albums/s104/grumpyoldman2007/dismounted_Quapukulu.jpg took me 6.5 minutes to convert to MS3D, find a figure with suitable scimitars convert and extract and add them to the Quapukulu figure, delete the lance and maces. Probably another 2 minutes tops to update the group class and name text entries and update the bounding sphere text for a foot figure and it's done, Do the same thing for the other three LOD levels and ouila! the new figure is ready to go after converting back to .MESH.

Cheers

GrumpyOldMan

AlphaDelta1
03-23-2007, 01:36
:idea2: :smash: :2thumbsup: Hubba hubba!

Can the community download this to check the shader/bumpmapping yet? During my work on Broken crescent I've been doing extensive work with the .normal files and have been testing my new textures with both shader models, if there's anything wrong, I'll spot it!

Cheers :2thumbsup:

alpaca
03-23-2007, 01:55
Hi AD1



Saw the picture with the new link, waited for 2 minutes on broadband and still couldn't see anything with the original link.

I thought I'd try and see how long this would take to do in Milkshape, this https://i150.photobucket.com/albums/s104/grumpyoldman2007/dismounted_Quapukulu.jpg took me 6.5 minutes to convert to MS3D, find a figure with suitable scimitars convert and extract and add them to the Quapukulu figure, delete the lance and maces. Probably another 2 minutes tops to update the group class and name text entries and update the bounding sphere text for a foot figure and it's done, Do the same thing for the other three LOD levels and ouila! the new figure is ready to go after converting back to .MESH.

Cheers

GrumpyOldMan
Wow, great :)
We need in-battle screens though :whip:

KnightErrant
03-23-2007, 02:40
@alpaca or AlphaDelta1
PM me an e-mail address and I'll send you a converted
feudal_knight_lod0.mesh . This case has normals that don't
agree with the original mesh. Also, the original mesh has
runs of zero vector tangents and binormals while the converted
one has a full basis set at each vertex. Should be a good test case.

AlphaDelta1
03-23-2007, 05:05
PM inbound.

Cheers

KnightErrant
03-23-2007, 05:12
On its way.~:)

GrumpyOldMan
03-23-2007, 05:15
Hi All

I finally got all the converter code working in the one app :dizzy2: :dizzy2: but in the process I broke the Merge MS3D code :furious3: :furious3: . Will have to go back and do this (it's definitely needed).

Here are some in battle shots of converted/modified/reconverted meshes, the lance has been bent on the lod0 and the two body variations have been taken out so they're all wearing the same outfit (just like in the old Robin Hood movies:laugh4: :laugh4: )

https://i150.photobucket.com/albums/s104/grumpyoldman2007/convertedmesh2.jpg https://i150.photobucket.com/albums/s104/grumpyoldman2007/convertedmesh1.jpg https://i150.photobucket.com/albums/s104/grumpyoldman2007/convertedmesh.jpg

Back to debugging the merge code :wall: :wall: :wall:

edit:- Didn't take me long, it's all to do with file names and overwriting files, I may have to put a check in that people can't overwrite files (if they're the same as either of the two original files) - won't be long.

edit2:- The other thing to look out for in KE's meshes is that the vertex weights have been coarsened to only two decimal points but I haven't seen it make any difference in animation.

Cheers

GrumpyOldMan

AlphaDelta1
03-23-2007, 06:39
Recieved the mesh.

Test 1 - Check it loads.

Pass with flying colours, no crashes so far and that bent lance is clear to see.

Test 2 - Check 'LoD pop' between LoD 0 and LoD 1 in shaders version 1.

Pass, with the usual 'LoD pop' associated with a switch between vanilla LoD 0 and LoD 1 models.. No clear change to the appearance of lighting.

I have to run back to work, will run more tests later.

Cheers and good work. :2thumbsup:

GrumpyOldMan
03-23-2007, 06:51
Hi All

Latest pic shows the dismounted Boyar with two bodies added and those bodies altered to use the unused texture variants.

https://i150.photobucket.com/albums/s104/grumpyoldman2007/meshwithaddedbits.jpg

This shows a standard mesh converted to Milkshape, extra pieces added, the uv values of those pieces altered within Milkshape and then exported out to MESH format.

Just doing a check that all lod levels are being converted correctly and adding some GUI controls/checks, etc.

Cheers

GrumpyOldMan

SigniferOne
03-23-2007, 07:54
GOM, would it be possible to have 'pools' of objects, so that you could put all shields into one mesh file, and have all units of the faction take randomly from that shield pool?

GrumpyOldMan
03-23-2007, 08:23
Hi SO


GOM, would it be possible to have 'pools' of objects, so that you could put all shields into one mesh file, and have all units of the faction take randomly from that shield pool?

The short answer is no, if you have all your shields (for example 40 shields) on one attachment set texture then you could put 40 shields on each mesh and they would show up as a random selection of shields ( but that is a pretty big overhead in mesh size). There are no separate shields or weapons in M2TW. All weapons and shields that are being used by a particular figure must be in that particular figures mesh. Don't be confused by the separate weapon/shield entries in the folders, these are CA's development meshes to build final meshes that have all bits attached, the development bits are not used in the retail version of the game.

Cheers

GrumpyOldMan

AlphaDelta1
03-23-2007, 09:42
Another test and a screenshot.

Test 3 - Check the difference between vanilla LoD 0 and modified LoD 0 in shaders version 2.

Pass. As far as I can see, the normals are working correctly, as is specular/shine. The helmet eye slots have a feeling of depth, and the lace on the tunic appears 'raised'. All ok to me.

https://i82.photobucket.com/albums/j278/donlogan/modified_feudal_test.jpg


Also, the original mesh has
runs of zero vector tangents and binormals while the converted
one has a full basis set at each vertex.

What change should we look for in-game?


edit2:- The other thing to look out for in KE's meshes is that the vertex weights have been coarsened to only two decimal points but I haven't seen it make any difference in animation.

I havent noticed any animation errors. A mount with animated fabric, such as the barded or mailed horse might be a better test case. Then again, I dont know much about vertex weighting as my 3D experience is mainly in modelling cold war era aircraft and cockpits.

If you need help testing the software I am happy to. Once I get to know the process I can write a tutorial. I have written tutorial before for such things as uv texture mapping, and mesh editing in 3Dsmax.

Cheers

alpaca
03-23-2007, 14:49
Did anybody try what happens when you charge with that lance btw?
Caliban said that the collision detection is linked to the model, so theoretically these knights shouldn't hit a thing (unless a higher LOD is used for that) :P

KnightErrant
03-23-2007, 14:53
@GrumpyOldMan
Excellent! You must have had less headaches with the
granny bytes than I did...:dizzy2:

@AlphaDelta1
(1) re Zero tangents and binormal runs. These were only like
10 vertices at a time so I'm thinking it would be on small details
like the belt buckle or the little crosses stitched on the tunic
down below the knight's waist. I think I see some detail there
but the lighting and the poses are ever so slightly different who
knows what effect we're seeing. I think the conclusion is definately
the converted meshes are no worse than the originals so the process
is a success!:yes:

(2) The 1.8.0 format allows two types of vertex weighting, the old
way 0-100% or new way 0-255. I left my code with the old way, but this is
a quantization error of 1% versus 0.4%, so subtle it would be hard
to spot. Weighting is used to soften transistions from one bone to
the next. For example without weighting, if the upper arm bone moved
all the vertices grouped to that bone would move and the torso would
stay fixed. This would be very robotic looking. With weighting, you can
tie some of the torso vertices to both bones and some of the upper
arm vertices near the joint to both. This makes the mail or fabric move
more realistically as the bone moves. Like I said, 1% or 0.4%, it would
be hard to tell the difference.

I think there's going to be some very happy modellers out there soon.:2thumbsup:

AlphaDelta1
03-23-2007, 15:28
KE,

Any minor graphical errors and accuracy issues I'm sure can be worked on post release. The main thing is to get a stable and user friendly release out there that shows the community that the .mesh format is in their hands. Hopefully the org will post some coverage of your work on the front page.


alpaca,

Did anybody try what happens when you charge with that lance btw?

I tried, the result was the same as always;

Against infantry - squish

Against cavalry - treadmill running.

I doubt M2TW uses anything like per-poly collision detection, it would be far too slow.

Cheers

KnightErrant
03-23-2007, 15:46
@AlphaDelta1
GrumpyOldMan's release should be user friendly
and have nice features.
re collision detection:
The models carry bounding sphere data, x, y, z, and radius, maybe this
would be the collision detection mechanism when two spheres intersect?

Andromachus Theodoulos
03-23-2007, 17:26
@All...

I am ready to start breaking... uhh I mean testing things...

I really don't have to wait for a tutorial to start poking around, but having one would be a wonderful asset.

How close are we to a beta release???

AT

Trotski12
03-23-2007, 19:17
The models carry bounding sphere data, x, y, z, and radius, maybe this
would be the collision detection mechanism when two spheres intersect?


Maybe a way to make guys fight from closer distance. ("tigth" ranks battles, a sweet dream and a realistic feature...) if this "sphere" is the collision issue.

Well, waiting for the news...

alpaca
03-23-2007, 20:13
Well MTW2 soldiers only score a hit when they actually hit, not when they're somewhere around I think.

So, could the bounding sphere be a preliminary collision detection that checks whether unit1 is near enough to unit2 to attack it and afterwards a more detailed collision detection is activated?

KnightErrant
03-23-2007, 20:55
Yeah, thinking about it more, if per-poly is too complicated
maybe spheres are too simplistic. A pre-trigger for a fight animation
maybe, with a more detailed mechanism for scoring a hit.
Fun experiment: take an infantry model (mounts might complicate
things for this), select all vertices. move upwards only a few
meters, change the z value in the bounding sphere, and put back
in game. Would they float over other units without interacting?:laugh4:
Could modellers change mounts into dragons for a fantasy mod, hmmm...
Well, there goes my Friday night now, got to try this when I get home.

alpaca
03-23-2007, 22:09
Yeah, thinking about it more, if per-poly is too complicated
maybe spheres are too simplistic. A pre-trigger for a fight animation
maybe, with a more detailed mechanism for scoring a hit.
Fun experiment: take an infantry model (mounts might complicate
things for this), select all vertices. move upwards only a few
meters, change the z value in the bounding sphere, and put back
in game. Would they float over other units without interacting?:laugh4:
Could modellers change mounts into dragons for a fantasy mod, hmmm...
Well, there goes my Friday night now, got to try this when I get home.
I'd think that you'd have to do that in the animation files, don't you?

GrumpyOldMan
03-23-2007, 22:50
Hi All

@@@@-KE-@@@@ - Heads up I've just emailed you the final (I hope) alpha code to test. Won't be long now fellas (famous last words?). I haven't put mount support in yet because I had a quick look at the mesh files and all the LOD levels are 'characterlod0', have to look at the files more closely and check normal/tangent/etc blocks and the infamous '4/2' byte.


I'd think that you'd have to do that in the animation files, don't you?

You're right Alpaca, the animation files include a component that shows the height above ground level for the bone_pelvis. I haven't had a look yet but you would have to have a different class of model (flighted?) and have a range of different animations depending on action and opponent. I would think that the class of figure type would be hardcoded. You'd also have to have new animations for all the other figures - shooting up (I mean that in the nicest, military sense of course :laugh4: :laugh4: )

Cheers

GrumpyOldMan

KnightErrant
03-24-2007, 00:23
Awww, too bad.....

KnightErrant
03-24-2007, 05:52
This is another stupid request because I don't
keep track of threads that have really good infomation
in them: what's the line that you put in your config file
to run in window mode? I'm trying to test GrumpyOldMan's
code for mesh conversion and I want to put multiple
Photoshop screencaptures into a .psd file so I want
to switch out of game mode back to the desktop to paste
printscreens. Can anybody enlighten me?:help:

GrumpyOldMan
03-24-2007, 07:18
Hi KE

The rain clouds came and I'm home again.


This is another stupid request because I don't
keep track of threads that have really good infomation
in them: what's the line that you put in your config file
to run in window mode? I'm trying to test GrumpyOldMan's
code for mesh conversion and I want to put multiple
Photoshop screencaptures into a .psd file so I want
to switch out of game mode back to the desktop to paste
printscreens. Can anybody enlighten me?:help:

I found this on a search:-


For some reason the default .cfg file provided by CA is overwritten each time. So create a bat file that uses your custom .cfg file instead, or overwrites the default .cfg file provided by CA each time.

Step 1: copy the default M2TW .cfg file to a file named carl.cfg
Step 2: modify carl.cfg as desired -
[video]
windowed = true
Step 3: create a file called carl.bat with the following in it
medieval2.exe @carl.cfg

run carl.bat file from a shortcut, or the windows command line.

This sounds like what you need. Or you could get Fraps which allows you to take screen shots in full screen but in .bmp only format in the lite version.

Cheers

GrumpyOldMan

alpaca
03-24-2007, 13:32
I usually use

[video]
windowed=1
movies=0
This will disable annoying intros, too.
By the way, when you're using windowed mode you can use Alt+F4 to end the game quickly (e.g. leave from a battle without having to go through the menu)

GrumpyOldMan
03-25-2007, 03:52
Hi All

Still testing. Here are some pics of modified Feudal Knights in battle. I've taken the heads off the Mailed Knights and added them to the Feudal Knights.

https://i150.photobucket.com/albums/s104/grumpyoldman2007/convertedknight1.jpg https://i150.photobucket.com/albums/s104/grumpyoldman2007/convertedknight2.jpg https://i150.photobucket.com/albums/s104/grumpyoldman2007/convertedknight3.jpg

Cheers

GrumpyOldMan