A start on the .MESH file format

[KnightErrant]

@Caliban
Thank you for that. I think Foz is correct that I can easily
make an fbx converter that goes the other way using the
Alias SDK so my original concern over ASCII versus binary has
gone away.

The Mystery Blocks

A reply to Grumpy's question about the mystery blocks:

I read his file decription again and it certainly seems it is the unused bool stream. Just before them there should be number 1 somewhere (number of these streams) followed by number of vertices (this number is the same for all streams) followed by bool values themselves. He should be able to just change that 1 to 0 and then remove count + bool values completely. But I'd have to see the file to be sure, I don't really remember.

I'll have a definate answer on this soon after a bit of investigation.

Do you have any more information concerning the mystery block(s)?
Can we really ignore it or is it in fact some other type of information?

@Foz
Repeating myself but I think you're right, I just need to change 1 to 0
and 0 to 1 and little bit else and I've got an ASCII to binary fbx converter.
Meant to try this last night but didn't even get on the computer.

I'd have to think longer about it to give you a decent description but let
me just do a short one. From the .mesh file we get:
face data (triangles) grouped together with group names
uv co-ordinates (Nvert by 2 array of floats, x y coords)
vertex weights (Nvert by 2 array of floats, primary weight, secondary weight)
vertices (Nvert by 3 array of floats, x, y, z)
bone assignments (Nvert by 4 bytes)
Mytery block?
bone data

Unfortunately this isn't the way Milkshape ASCII or fbx ASCII groups it.
They organize the vertices, vertex normals (which aren't in the .mesh),
uv coords, triangle data, and UV indices? each in their own named group.
The programming task is then to pull out vertices, normals, and uv data
from the mesh's big list, put them in little lists, reindex them, recompute the
reference indices in the face data for each group, and then write
all this junk out with a bunch of surrounding boiler plate ASCII which
the fbx format requires. I'm just starting to code what I
just described but it'll probably be a slow process.

[Andromachus Theodoulos]

@KE,

First I want to say... take your time!!! If its a slow process, I am sure it will be worth it. I can certainly wait.

Anyway, to get a feel of what is developing, are we still looking at something that could be used with MilkShape?

I just want to say, you all have had me on the edge of my seat for the last several days, I have been following this thread closely. This would constitute one of the biggest breakthroughs for me personally if I am able to have access to an "affordable" modification tool for the meshes... I might even have something of value to post for others to download... which has not been true for my modifications in the past.

Keep up the good work!!! I am cheering you guys on!

@Caliban...
Thanks for your help on this and your gentle guidance...

AT

[KnightErrant]

Hi AT

GrumpyOldMan said he was doing the Milkshape binary format .ms3d,
so yes and this is probably the best path to success. I'm just looking
at fbx as an alternative just for a backup and because it's just interesting
in and of itself. 3dsmax doesn't support .ms3d but since Milkshape can
export fbx there's only one extra step to get meshes to the 3dsmax
modellers. GOM said something like "it's easy to take apart a clock, it's
the reassembly that's the killer" which refers to going the other way,
turning stuff back into a mesh file with all those funny header/trailer
byte blocks.

I've got my fingers crossed that he got info from Caliban that
makes that doable.

[Casuir]

I read his file decription again and it certainly seems it is the unused bool stream. Just before them there should be number 1 somewhere (number of these streams) followed by number of vertices (this number is the same for all streams) followed by bool values themselves. He should be able to just change that 1 to 0 and then remove count + bool values completely. But I'd have to see the file to be sure, I don't really remember.

Been having pc problems so havent been able to test this until now but it seems changing the value mentioned above and removing the relevant blocks has no apparent ill effects ingame. Only tested it with a couple of meshs but it does look like its not neccessary and can be left out.

Re the formats ms3d is an affordable modelling program but it currently doesnt support all the necessary info in the mesh files so it might be wise to pursue other formats like fbx which do and arent program specific

[KnightErrant]

Thank you Casuir! That's what I was hoping.
Just so I'm clear on this you mean the header/trailer right
before the mystery block like

   0   0   0   0    27   0   1   0    63   0   0   0    28   0  62   0     0   0  28   0    64   0   0   0     3   0   0   0  
number of vertices in mystery block: 4949
  79  25 186   0
 181  17 162   0

and change the red 1 to a 0?

PS: And best of luck with your PC.

[Casuir]

Its the 3 right before the number of vertices, 3 being the number of streams (what we're calling blocks) which follow. Be careful here if you're deleting as I've noticed the .vertex/bone assignments block comes after this part rather than before in some .mesh files

[KnightErrant]

Ah, ok I see the 3. Hmmm, my script is going to
break on those mesh files where the vertex bone assignments are
out of place because both it and the mystery blocks have
the number of vertices as the first int to read so they look
alike. The size of the header/footer lines are different so it will
break right after reading the first mystery block of a lod0 file. I'm not
sure how to code around this.

Can you tell me a unit that has this reversed structure? Maybe there's
a clue in the header/footers.

[Caliban]

Do you have any more information concerning the mystery block(s)?
Can we really ignore it or is it in fact some other type of information?

Engine guy is still on holidays, I'll prod him again next time I see him. My guess is that they are unused and can probably be ignored. A test should be able to tell but I'll find out for sure.

@Caliban...
Thanks for your help on this and your gentle guidance...

Aww shucks

[KnightErrant]

Thanks Caliban! Casuir has reported a test a few
posts back, where he removed the mystery blocks and changed
the 3 to a 0 in the header/footer line above where they appeared,
and suffered no ill effects with the game. I'll assume they
can be safely ignored.

Casuir also reported that there is at least one variant mesh format
where the vertex bone assignment block follows the mystery block
rather than proceeds it. I've only partially found all the changes but
the very first header/footer line is different between "regular" units
and these variants so I think it can be programmed around. Is this a
granny thing or just different ways that your guys assemble the
mesh files? Just wondering if we should be on the lookout for other
variant type meshes. (It's the az_body_suit types where he says
most of these variants are.)

[GrumpyOldMan]

Hi Guys

Lots of great things happening here, great to see!!

I've written some code, at the moment there is an alpha .MESH to .MS3D converter that exports bones and weights as well as the mesh and groups, there is also a pre alpha utility to load a .MESH file to screen and texture it with .TEXTURE files. Here's a screen pic https://i150.photobucket.com/albums/...estingmesh.jpg
. The texture in the background will be for UV adjustments but the code I've written for that is very slow and cluncky - major rewrite here.

The figures in the Aztec bodysuit folder are different in that the initial header is 4 bytes shorter, could be because there is only a body group and no leg group? - Could be a clue to the make up of the initial header. I've put an error trap in the converter to take care of this. Some of the figures in the folders seem to be WIP or base figures.

I've PM'd a few guys to test the converter, and I'll see what happens with their results.

If you're looking at alternative formats there is the .SMD format that's also supported by milkshape, it has vertex weighting (HL2) and allows export of animations only.

Cheers

GrumpyOldMan

[Foz]

Holy reverse engineering, Batman!

GOM isn't kidding, there's a lot of great things happening here. I'm a bit amazed at how quickly everything is getting figured out and implemented. I'm probably never going to edit a mesh for this game, and yet you guys have me all fired up for the importer/exporter tool to be done. It's almost like I can feel how badly everybody wants it, and see all the awesome visual mods that'll happen following briskly on the heels of this project.

Keep up the good work so I can keep on being amazed

[KnightErrant]

Kudos! congratulations, a balloon , and more.
Really great work! I'll look at more meshes
tomorrow but so far no problems at all. If there's anything
you need for the reverse process, unit surveys looking at
header/footers etc. please don't hesitate to ask.
I know the entire modding community is going
to be thrilled at what you've done.

[Caliban]

The figures in the Aztec bodysuit folder are different in that the initial header is 4 bytes shorter, could be because there is only a body group and no leg group? - Could be a clue to the make up of the initial header. I've put an error trap in the converter to take care of this. Some of the figures in the folders seem to be WIP or base figures.

If you're looking at alternative formats there is the .SMD format that's also supported by milkshape, it has vertex weighting (HL2) and allows export of animations only.

Your right about not having leg variations. Models can be setup with many variations, or very few. This particular guy has a few body variations which include the legs all in the same mesh. Some units have seperate body, arm and leg variations.

I'll list the different types of variations and attachment types that we can 'markup' in 3dsmax. Mesh variations are numbered such as 'arms01' 'arm02' 'arm03' in 3dsmax and then simply set by selecting the mesh and then the type of variation with our plugin/exporter tool.

Variation Types:
(all meshes except the bones are marked in 3dsmax as one of the following)

Head
Leg
Arm
Body
Hands
Helmet

Attachment Types:
(these can have different variations per set. Usefull for adding things like pouches, bags etc)

Attachments
Attachments1
Attachments2
Attachments3

Weapons
(this markup is used for exporting weapons and shield seperatley, these are not setup within the character file)

Shield
Sword

Below is a 3dsmax7 character example with variations and attachments. Should work with 3dsmax7 or higher. It may prompt for our plugin, you should be able to simply press open to bypass it.

http://www.filefactory.com/file/64e26c/

.SMD format is not a bad idea, I'm also pretty familiar with it. It's quite widley documented and supported by all the big 3d Packages. I'm pretty sure the guys at Valve wouldn't mind you using it ;)

[Casuir]

Generals and captains are another example of meshs with few variations. Any hard limit on the number of variations caliban or is it limited to what you can fit on a texture? Cheers for the maxfile too :)

[KnightErrant]

@ GOM and Casuir
I took your and Caliban's advice and looked at the
.smd format (documented thank goodness!)
I've got my script so it outputs all the uv, vertex weights,
primary bone and secondary bone, plus the vertex normals
in the .smd format by triangles.
I'm confused by the node tree and skeleton pose data though.
I just copied the bonestrings part of the .mesh file and put
it in .smd format like

version 1
! Node tree or bone ID assignment section
nodes
0 "bone_pelvis" -1
1 "bone_rthigh" 0
2 "bone_rlowerleg" 1
3 "bone_rfoot" 2
4 "bone_abs" 3
5 "bone_torso" 4
6 "bone_head" 5
7 "bone_jaw" 6
8 "bone_eyebrow" 7
9 "bone_rclavical" 8
10 "bone_rupperarm" 9
11 "bone_relbow" 10
12 "bone_rhand" 11
13 "bone_lclavical" 12
14 "bone_lupperarm" 13
15 "bone_lelbow" 14
16 "bone_lhand" 15
17 "bone_lthigh" 16
18 "bone_llowerleg" 17
19 "bone_lfoot" 18
20 "bone_weapon01" 19
20 "bone_weapon" 20
21 "bone_weapon02" 20
22 "bone_weapon03" 21
22 "bone_shield01" 22
22 "bone_shield" 22
end

but this probably isn't right since the first number is supposed to
be the bone id and the second number is the parent id. Should
the first number just go up monotonically or should I just remove
duplicate weapon and shield entries?

Second question is the skeleton pose block which is
ID Posx Posy Posy Rotx Roty Rotz
The spec says bones that are children of other bones
report position and rotations in their parent's local space.
The rotations are spec'd as Euler rotations in radians.
I'm assuming this means body frame roll, pitch, and yaw (for x, y, and z).
For now I'm just putting in 0's but this probably won't work.
Can't test until I get home.

!Skeleton pose data
Skeleton
Time 0
0 0.0 0.0 0.0 0.0 0.0 0.0
1 0.0 0.0 0.0 0.0 0.0 0.0
2 0.0 0.0 0.0 0.0 0.0 0.0
3 0.0 0.0 0.0 0.0 0.0 0.0
4 0.0 0.0 0.0 0.0 0.0 0.0
5 0.0 0.0 0.0 0.0 0.0 0.0
6 0.0 0.0 0.0 0.0 0.0 0.0
7 0.0 0.0 0.0 0.0 0.0 0.0
8 0.0 0.0 0.0 0.0 0.0 0.0
9 0.0 0.0 0.0 0.0 0.0 0.0
10 0.0 0.0 0.0 0.0 0.0 0.0
11 0.0 0.0 0.0 0.0 0.0 0.0
12 0.0 0.0 0.0 0.0 0.0 0.0
13 0.0 0.0 0.0 0.0 0.0 0.0
14 0.0 0.0 0.0 0.0 0.0 0.0
15 0.0 0.0 0.0 0.0 0.0 0.0
16 0.0 0.0 0.0 0.0 0.0 0.0
17 0.0 0.0 0.0 0.0 0.0 0.0
18 0.0 0.0 0.0 0.0 0.0 0.0
19 0.0 0.0 0.0 0.0 0.0 0.0
20 0.0 0.0 0.0 0.0 0.0 0.0
20 0.0 0.0 0.0 0.0 0.0 0.0
21 0.0 0.0 0.0 0.0 0.0 0.0
22 0.0 0.0 0.0 0.0 0.0 0.0
22 0.0 0.0 0.0 0.0 0.0 0.0
22 0.0 0.0 0.0 0.0 0.0 0.0
end

It seems like we would lose the triangle mesh type names
and mesh group names in this format. Would this matter?

[GrumpyOldMan]

Hi All

Caliban

Thanks for the information. I've noticed that as well as Attachment groups there are Equipment type groups - is there any difference? Also (probably getting ahead of myself) but with the shields some are shown as shieldpassive and some as shieldactive. I was looking at the Lithuanian cavalry man in particular and the active shield is strapped to the back and the passive shield is on the left forearm. Is the active shield used with active weapons, or is the active shield used with the primary weapon and the passive shield used with secondary weapon .

Thanks again for all the information.

Cheers

GrumpyOldMan

[Casuir]

The figures in the Aztec bodysuit folder are different in that the initial header is 4 bytes shorter, could be because there is only a body group and no leg group? - Could be a clue to the make up of the initial header. I've put an error trap in the converter to take care of this. Some of the figures in the folders seem to be WIP or base figures.

Sorry havent done much with this but been having major pc probs. Got a few converted off today with no errors and noticed that some are as you said base figures. Seem to be the models that were exported sans primarys and secondarys and recompiled later via the xmls. Compared a couple for changes in the initial header and noticed a few things. These are the headers from the jannisary_gunner and jannisary_musketeers meshs, models are identical bar the missing primary and secondarys:
(leaving out the serialization::archive bit)

Musketeer:
00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 00 00 04 00 01 03 01 00 00 
00 00 00 07 00 01 00 02 00 00 00 00 00 15 00 00 00 00 00 0B 00 01 03 03
00 00 00 04 00 00 00 

Gunner
xx xx xx xx 00 00 00 00 00 00 00 01 00 00 00 00 00 04 00 01 03 00 00 00
00 00 00 07 00 01 00 01 00 00 00 00 00 0E 00 00 00 00 00 0B 00 01 03 02
00 00 00 04 00 00 00

I've marked the parts that are different in red, x'x at the start of the gunner code are there to represent the 4 bytes that are missing. I had a look at the guys in the az_body_suit folder and the guys in there missing 4 bytes look like base figures, theres corresponding meshs with weapons and the missing 4 bytes for them all, same goes for generals. It looks like every model thats used ingame has them. Only other bytes that change from file to file are the 15/0E which corresponds with the number of seperate meshs in the file and the 04 at the end, doesnt change between these two but I've noticed some mesh files have an 0C here instead. Other than that the header looks to stay the same between files.

@Bwian, dunno about limits but the bosnian archer has 45 seperate meshs with 8 head variations

[Casuir]

Bit more on the 04/0C value, it seems to relate to the mesh type that follows, all with 0C have an attachment as the first mesh in the file while those with 04 have a body part. Guessing its mesh variation header rather than the actual .mesh file one. Primary weapons have an 0E here, secondarys 10 and shields 07. Guess that makes the rest of the part between mesh variations footer rather than header.

btw the geometry value for weapons and shields is added to the base model, meaning anyone who feels like tackling the hexediting should be able to make weapon kitbashs without needing an importer/exporter.

[Casuir]

My apols GOM, see you already figured out the byte for the number of meshs in the file and what the byte before the mesh name is for, really should pay more attention The rest of the footer seems to just be a sequence of numbers which increases by meshand is constant between files

04 00 00 00 Body	Body		00 00 00 0A 00 01 00 04 00 00 00 0B 00 03 00 00 00 0B 00 05 00 00 00
04 00 00 00 Legs	Leg		00 00 00 00 00 0A 00 06 00 00 00 0B 00 05 00 00 00 0B 00 07 00 00 00
0B 00 00 00 Attach	Cylinder02	01 00 00 00 00 0A 00 08 00 00 00 0B 00 07 00 00 00 0B 00 09 00 00 00
04 00 00 00 Arms	Arm		00 00 00 00 00 0A 00 0A 00 00 00 0B 00 09 00 00 00 0B 00 0B 00 00 00
0B 00 00 00 Attach	Cylinder03	01 00 00 00 00 0A 00 0C 00 00 00 0B 00 0B 00 00 00 0B 00 0D 00 00 00
0B 00 00 00 Attach	Box014		00 00 00 00 00 0A 00 0E 00 00 00 0B 00 0D 00 00 00 0B 00 0F 00 00 00
0B 00 00 00 Attach	Plane02		01 00 00 00 00 0A 00 10 00 00 00 0B 00 0F 00 00 00 0B 00 11 00 00 00
0B 00 00 00 Attach 	Plane04		00 00 00 00 00 0A 00 12 00 00 00 0B 00 11 00 00 00 0B 00 13 00 00 00
06 00 00 00 Helmet	Hat04		00 00 00 00 00 0A 00 14 00 00 00 0B 00 13 00 00 00 0B 00 15 00 00 00
04 00 00 00 Head	Head1		00 00 00 00 00 0A 00 16 00 00 00 0B 00 15 00 00 00 0B 00 17 00 00 00
05 00 00 00 Hands	Hand01		00 00 00 00 00 0A 00 18 00 00 00 0B 00 17 00 00 00 0B 00 19 00 00 00
04 00 00 00 Head	Head02		00 00 00 00 00 0A 00 1A 00 00 00 0B 00 19 00 00 00 0B 00 1B 00 00 00
04 00 00 00 Head	head03		00 00 00 00 00 0A 00 1C 00 00 00 0B 00 1B 00 00 00 0B 00 1D 00 00 00
04 00 00 00 Head	Head04		00 00 00 00 00 0A 00 1E 00 00 00 0B 00 1D 00 00 00 0B 00 1F 00 00 00
10 00 00 00 Secnd	Sec_2		00 00 00 00 00 0A 00 20 00 00 00 0B 00 1F 00 00 00 0B 00 21 00 00 00
10 00 00 00 Secnd	Sec_6		00 00 00 00 00 0A 00 22 00 00 00 0B 00 21 00 00 00 0B 00 23 00 00 00
10 00 00 00 Secnd	Sec_7		00 00 00 00 00 0A 00 24 00 00 00 0B 00 23 00 00 00 0B 00 25 00 00 00
10 00 00 00 Secnd	Sec_8		00 00 00 00 00 0A 00 26 00 00 00 0B 00 25 00 00 00 0B 00 27 00 00 00
0E 00 00 00 Prmry	Pri_27		00 00 00 00 00 0A 00 28 00 00 00 0B 00 27 00 00 00 0B 00 29 00 00 00
0E 00 00 00 Prmry	Pri_26		00 00 00 00 00 0A 00 2A 00 00 00 0B 00 29 00 00 00 0B 00 2B 00 00 00
07 00 00 00 Ramrod	RR_57		00 00 00 00 00 0A 00 2C 00 00 00 0B 00 2B 00 00 00 0B 00 2B 00 00 00

The only thing that changes is the first byte and It looks like you're right about it being related to attachments, doesnt seem to signify that its compulsoty though as some of the meshs above marked with it dont appear on all soldiers. My guess is that if theres just one type of mesh variant it gets used for all soldiers. The rest bar the first footer is just an increasing sequence and looks to be the same for all .mesh files I've looked at so far.