A start on the .MESH file format

[Suraknar]

@Suraknar, the normal/tangent/binormals are calculated per vertex so two different models are going to have two different sets of values, i.e pasting the data from a file with 4000 vertices into a file with 5000 vertices will crash the game.

So in other words that mysterious byte information is the actual per vertex calculations and not the information required by the rendering engine on how to handle them.

I was under this impression that this information was pertaining to instruct the engine on how to handle the per vertex calculations whose information is stored elsewhere in the file and would depend on each different model (obviously).

Well in that case yes it is not feasable. Continuing to read with interest here, I am a 3D artist(Maya and the older SoftImage), I understand what we talking about, yet, not necessarilly the coding parts.

[KnightErrant]

Okay, I want to return to tangent space normal mapping but
I'll leave out the mathematics.
Many thanks to Re Berengario I and Suraknar for pointing out
tutorials on this and data cube normalization. Caliban has told
us that the 3 0 0 0 mystery block is vertex normals, 10 0 0 0
is tangents, and 11 0 0 0 are binormals. The vertex normals
are computed by averaging over the face normals by iterating
over the triangles, the tangents are computed by conditioning
the face tangents by the uv coords according to the formulas
in the refs in the previous posts, and the binormals can be obtained
from the cross products. The confusing part is mapping to RGB
values. Basically you take all the normalized vectors, add 1.0
and divide by 2.0 to map the components onto a [0,1] RGB float
space. You then multiply by 255 and truncate to map to a
byte-valued RGB space on [0,255]. This is called cube normalization
(learned at lunch time). This is the format of the mystery
data's block 1, 2, and 3 (along with a fourth byte that looks
like an unused albedo value).

Anyway, if you do this you can get the following .mesh to .ms3d
and .ms3d to .mesh conversion for the feudal_knights_lod0.mesh.

(I'm repeating bits of post #160 and I'll use thumbnails this time,
still learning the fine points here.)

First step: Convert mesh to ms3d and modify in Milkshape:
All I've done is broken the lances.




First try, zero out the mystery data:




Okay, you can see the Milkshape mod has shown up in the game but
the shadows on the lod0 rendered knights shows the mystery data isn't
being applied.

Finally, use the data cube normalization along with the vertex normals,
tangents, and binormals to compute the mystery data out of the
data pulled from the .ms3d file and put into the .mesh format to get





and there you have a forward and backward mesh conversion.



Slight but solvable problem: People might notice the lighting looks wrong.
That's because there's still an unresolved coordinate system problem.
Here's the vertex normals (in tangent space normal mapping) from
the original feudal_knights_lod0.mesh

(first ten of them anyway)

189 238 116   0  
 253 127 106   0  
   5  95 110   0  
 136   3 100   0  
 251 116  98   0  
 193 236 129   0  
 209  31 108   0  
 139   1 115   0  
  50  25 118   0  
   2 108 115   0

and here's my computed ones:

116  238  190  0   
107  128  253  0   
111  95  5  0      
103  5  100  0     
99  116  251  0    
130  236  194  0   
109  36  215  0    
114  29  208  0    
119  26  51  0     
116  108  2  0

You can see that the x and z coords are reversed and that they
don't exactly match. Too tired to work this out tonight, I'll hit
it later. Must be a coordinate system thing.

Anyway, the mechanics of this are getting really clear now.
More stuff tomorrow.

[GrumpyOldMan]

Hi Guys

Lots of real life things happening , The first un-mystery block is normals, stored byte fashion as z y x 0, the algorithm to extract them is MS3D normal(x,y or z)*127.5+127.5. Remember to reverse any Milkshape 'x' values to take into account the different axes. It's not exact but close enough. This means that with the right frame work we can now extract lod1 and below meshes from Milkshape - including any new groups . M2TW will run without lod0 meshes but you won't get the eye candy.

@KE and Casuir - The Milkshape meshes that I've been extracting don't have valid normal info, I've been relying on Milkshape to update them. To update them, after loading the MS3D, go to Menu/Tools/Model Information V1.7 and after it comes up, hit the Close button rather than the 'x' in the top righthand corner and this will update the normals.

I'm going to work on the tangent and binormal/bitangent using the first triangle/vertices of the Feudal_knights_lod0.mesh. The info is as follow:-

155 154 143 (vertex number)
0.794555 0.558303 -0.0330388 0.330422 0.127918 (x,y,z,u,v)
0.794555 0.431023 -0.0131331 0.315904 0.127895
0.794555 0.482091 -0.0767073 0.322253 0.127879

I'll get out the tangent and bitangent figures later, I'm assuming that it runs z,y,x,0 as per normals.

Well when GrumpyOldMan comes online he'll understand this
so I guess I shouldn't be too worried about my ignorance.

You wouldn't have been so confident if you'd known that I had hot flushes after I was dragged kicking and screaming from my DX7 comfort zone, with shader stuff!!

Cheers

GrumpyOldMan

[KnightErrant]

Hi GrumpyOldMan!

We almost simultaneously posted! RE I and Suraknar straightened
me out and steered me to the right tutorials for tangent space stuff
and cube normalization. Caliban really came through for us on the
mystery data. I'm feeling really good about getting this to work now.
Is my coordinate space problem because Milkshape uses a system I
didn't pay attention to?

[Suraknar]

Many thanks to Re Berengario I and Suraknar for pointing out
tutorials on this and data cube normalization.

I did not do much really, except read the docs RB linked and point to various areas, while making assumptive hypothesis. :P

The great thing here is that there is much progress which is extraordinary!

[KnightErrant]

Yes, I'm dancing off the walls at this moment thinking we've
got the procedures almost nailed down.
For me, I think if GOM has a plan for handling the bones
stuff by auxilliary files to handle variants like mounts
and artillery which probably don't follow the canonical
unit assignments, then I don't see any technical obstacles
like I worried about with the mystery data.

Woo Hoo! exponential [BIG NUMBER}

[GrumpyOldMan]

Hi KE

Hi GrumpyOldMan!

We almost simultaneously posted! RE I and Suraknar straightened
me out and steered me to the right tutorials for tangent space stuff
and cube normalization. Caliban really came through for us on the
mystery data. I'm feeling really good about getting this to work now.
Is my coordinate space problem because Milkshape uses a system I
didn't pay attention to?

The coordinate issue relates to how the MESH normal data is written, it's similar to littleendian integers, shorts, etc., with the bytes being written from right to left. Looking from the right it's 0,x,y,z. But if we try to read it from the left like English it looks like z,y,x,0.

I wouldn't be too concerned about the numbers not exactly matching, from what I can see there is usually only a difference of 1 which means a 1.4 degrees difference in a 0-255 world. We can have a look at it but it's not really significant.

With the picture of the reconverted feudal knight, did you have the original 2nd and 3rd blocks of un-mystery code reinserted, your own converted values or is it running with just the 1st block?

Damn, this is the third edit already to this post, shouldn't type when I'm tired. @KE I came across http://graphics.cs.lth.se/theses/pro...gfx/report.pdf on a Python CG mode when I was looking around, Haven't looked through it yet but it might be interesting.

Cheers

GrumpyOldMan

[SirAnen]

Hello , i have been watching this subject form the begining and i like to help.
As a discoverd the system is based on matrix factors.

The coordinate issue relates to how the MESH normal data is written, it's similar to littleendian integers, shorts, etc., with the bytes being written from right to left. Looking from the right it's 0,x,y,z. But if we try to read it from the left like English it looks like z,y,x,0.

This the problem of 3d modeling. UW warp have x,y,z but as you know when you do a mirror you get the same x,y,z but backwords. 0 is the point of the model.

If you need any help with matrix , give ma a sign :)

Btw. I would be pleased to send me some model beacuse i don't have EN edition , my PL one dosn't want to be edited. I would be pleased with some arm men with chainmails and some crusedres units.

Sorry for my english.

[KnightErrant]

Sorry, I've been falling asleep without logging out the past few nights.

The back converted mesh has all computed values for the normals,
tangents, and binormals for blocks 1, 2, and 3. So it's a consistant
right handed system, it just has a disconnect with Milkshape's convention.
I think I knew this a few days ago because my figures came into Milkshape
lying on their sides so I did a few swaps to make them look right in Milkshape
but then I forgot about this going backwards so the geometry or meshes
are ok but the auxilliary data is reversed. I'll fix it up tonight.
I agree I'm not going to worry too much about a difference of 1 in
the tangent space normal maps because 1/255 = 0.4% so who's going to
even be able to notice that.

I took a quick look at the link and I have to say I never thought Python
would be used to program a shader. It looks like they can make
OpenGL and DirectX calls for vertex shaders, pixel shaders, bump mapping,
environment mapping, etc. Not that we need this but I'm still blown away
that you can do this in an interpreted language. (Actually I think the article
says that Python gets compiled natively at some point, have to read it
carefully later.)

I think I just want to fix up a few things and then I've gone about as far
as my knowledge will take me. I'd like to write up the tangent space
normal mapping stuff to leave some documentation in the Wiki just because
it's so interesting in its own right. Otherwise I'm ready to test stuff
whenever you need it.

[Andromachus Theodoulos]

@All...

If you guys don't mind a complete 3D modelling newb test your work as well, I am here...

I work with 3D CADD programs, but I am not sure how similar in function they would be with 3D animation/gaming.

Let me know!!

AT

[Bwian]

Does this mean what I think it nmeans?

Knight Errant has managed to IMPORT a model from the game, mess about with it in Milkshape 3D, and then EXPORT a completely funtional model with skeleton and texture intact back out and have it work perfectly in game?

Damn fine work!

Assuming this is a stable process, and judging from the tech-speak flying around, you guys definitely seem to understand how the process works, how near are we to a tool that can be tested by us polygon pushers?

I am assuming we can import the skeleton intact and use MS3D to rig a mesh for export? Can't wait to try this out!!!!!

[KnightErrant]

Hi AT and Bwian,

The only technical thing we were waiting on was what we
called the mystery blocks but now that we know they are
vertex normals, tangents, and binormals in a cube normalized
RGB format that's the last unknown.

My stuff is all hard coded to feudal_knight_lod0 and was only meant to
test the conversion algorithms from the data layout of the .mesh to
the data layout of an .ms2d and back again and getting
the granny strings right. But now we know it works.

I'd say like a week or so, but then again don't let me put words
in GrumpyOldMan's mouth. Stay tuned, we'll hear good things soon!

[GrumpyOldMan]

Hi Guys

I am assuming we can import the skeleton intact and use MS3D to rig a mesh for export? Can't wait to try this out!!!!!

As a break from wrestling with the Tangent and Binormals, I thought I'd try out the size of RTW meshes with the M2TW skeleton, here's a pic https://i150.photobucket.com/albums/...rtw_2_m2tw.jpg .

The size overall is pretty good but the head needs alignment, especially with the Jaw and Eyebrows. I hope the Warhammer guys don't mind me using one of their .cas files . On the whole it would be easier to use M2TW parts and retexture than to completely rerig a skeleton and mesh, but it would be possible.

Well, that was a nice break, I'll go back and see if I can find out what I'm doing wrong .

Cheers

GrumpyOldMan

[alpaca]

Huh did you write a cas importer for Milkshape then?

@KE:

[Andromachus Theodoulos]

@ Alpaca...

It looks as though he did something to get the old cas model into ms3d... revolutionary!!!

@GOM...
I was curious if there was a scale/size difference was between the two models, because I have notice that I have had to scale textures I used from RTW up several times (1.5 if I remember right) to get the to be at the right "scale" with the M2TW models.

I used the lorica segmentata from the western Roman auxilia palatina in RTWBI and merged it with the Italian spear militia texture in M2TW. I have done the same with adding scale mail to the same model... I also did this with the Tercio pikemen (not historically accurate, but looks cool...)

So it is interesting that the models appear to be the same size, which is great for switching textures back and forth, this will save a lot of work, not having to develop things from scratch.

AT

[GrumpyOldMan]

Hi All

Not much to report, steadily tearing down my walls of ignorance .

@GOM...
I was curious if there was a scale/size difference was between the two models, because I have notice that I have had to scale textures I used from RTW up several times (1.5 if I remember right) to get the to be at the right "scale" with the M2TW models.
AT

The mesh and skeleton were shown as-is, store-bought, off-the-shelf. The mesh retains it's UV texture values so it uses the original texture. After the major work has been done, I'll look at a CAS to MS3D converter possibly reassigning the vertices to the equivalent bones in M2TW. I also have to look at a converter to extract animation data back to M2TW CAS format (I've already got the M2TW CAS to MS3D roughly done).

Cheers

GrumpyOldMan

[Bwian]

Being able to change animations would be a very nice thing to have the ability to do again! There are lots of things I would like to be able to do with the animations to make unusual units!

I presume that would also open up the possibility of scaling the skeleton and mesh to allow over/undersize units as well.

Mind you.... if it were possible ( and in RTW it wasn't ) to actually re-site the bone joints in MS3D and have that preserved in the export, that would be so absolutely brilliant I would pay money for it ( and my wallet has armed guards on it! ). I am guessing the way RTW handled such things that it was not physically possible. The skeleton seemed to be fixed in the DAT file, and that was it. If we were able to actually re-arrange the bones in a modified skeleton we would be able to make accurate and varied fantasy units, and really make some fabulous mods!

RTW 'Dwarves' were just scaled down, and not anywhere near bulky enough. Goblins and Orcs were re-skinned humans. Yuck.

I am assuming that the actual location of the bones relative to the root must be stored somehow, and that the animations were recording rotational changes in each joints relative to it's parent. In RTW we never got to re-write the start points..... Can this be done for MTW2?

[Re Berengario I]

With Granny libraries taking care of animations in MTW2 I HIGHLY doubt that we'll ever being able to change animations.

I'll be happy if someone at CA could say I'm wrong but I already saw this movie in other games using Granny.

[alpaca]

Animations are still in .cas format I think so it might be possible to edit them.

[Lusted]

Yeah they're still in .cas format though vercs old import script for them doesn't work.

[quadrille]

I hope the Warhammer guys don't mind me using one of their .cas files

Not at all! :)

I'm thrilled by your work, as has been said before you guys will be heroes in the community!

I agree with what Bwian said about animations and skeletons. Not being able to edit bone positions and add new skeletons is part of what killed the warhammer mod. If we were able to do that with M2TW the sky would truly be the limit.

[Caliban]

Yeah they're still in .cas format though vercs old import script for them doesn't work.

I managed to import the strategy map models general, assasin etc but it fails on "unable to convert: undefined to type: Float" on this section of the script:
rot = ( point3 (ReadFloat fp) (ReadFloat fp) (ReadFloat fp) ) --)--rtw2maxVector(

The .CAS animation format is still the same from RTW. I tried a few tests with Verc's import script and I think it may just need updating to support M2 rigs.
This could be because of the different skeleton setup being used. I'm able to load the Medieval2 animations in the old RTW animation viewer. I'm just speculating here, but Verc's script is probably geared towards only loading specific RTW skeleton setups. Updating the script should allow the M2 animations to be imported again. The models I released in this thread should give some clues about the differences.

[GrumpyOldMan]

Hi All

As I and Tangent-Binormal head back to our corners ready for the next round, I read through the last couple of posts.

Mind you.... if it were possible ( and in RTW it wasn't ) to actually re-site the bone joints in MS3D and have that preserved in the export, that would be so absolutely brilliant I would pay money for it ( and my wallet has armed guards on it! ). I am guessing the way RTW handled such things that it was not physically possible. The skeleton seemed to be fixed in the DAT file, and that was it. If we were able to actually re-arrange the bones in a modified skeleton we would be able to make accurate and varied fantasy units, and really make some fabulous mods!

RTW 'Dwarves' were just scaled down, and not anywhere near bulky enough. Goblins and Orcs were re-skinned humans. Yuck.

I am assuming that the actual location of the bones relative to the root must be stored somehow, and that the animations were recording rotational changes in each joints relative to it's parent. In RTW we never got to re-write the start points..... Can this be done for MTW2?

I agree with what Bwian said about animations and skeletons. Not being able to edit bone positions and add new skeletons is part of what killed the warhammer mod. If we were able to do that with M2TW the sky would truly be the limit.

The question is whether the skeleton.dat and skeleton.idx take precedence over extracted files (ie does io.file_first work with skeletons)? If not then there may be a chance. The bone locations are stored in the animation folder in the files with no file suffix. There is no central location for bone positions. In RTW all the bone locations were stored within the .CAS files. If you keep the number and name of bones consistent with M2TW you should be able to record a new skeleton, and if that works you'll have to come up with all new animations (standard animations won't work) because inside this file are also the various unit/figure states or actions and the required animations. You'd also have to see if the .MESH bounding sphere affects placement.

I managed to import the strategy map models general, assasin etc but it fails on "unable to convert: undefined to type: Float" on this section of the script:
rot = ( point3 (ReadFloat fp) (ReadFloat fp) (ReadFloat fp) ) --)--rtw2maxVector(

The .CAS animation format is still the same from RTW. I tried a few tests with Verc's import script and I think it may just need updating to support M2 rigs.
This could be because of the different skeleton setup being used. I'm able to load the Medieval2 animations in the old RTW animation viewer. I'm just speculating here, but Verc's script is probably geared towards only loading specific RTW skeleton setups. Updating the script should allow the M2 animations to be imported again. The models I released in this thread should give some clues about the differences.

Hi Caliban

Verc's CAS importation script won't work with M2TW anims without alteration. I had a look at the animations in Oct/Nov when the first demo came out. Here's a quote with the details of the changes to the animation format:-

Hi All

The animation files are in a bit of a different format from RTW ( as well as all biped figures having different skeletons - no cloak bones and additional bones for jaw, eyebrow and two clavicles between torso and upperarms).

The initial part of the animation files are the same, with no. of frames, no. of bones, frames x bones x quat values for rotations, but then there frames x bones x local translations (but this also includes the 'Y' or perpendicular to ground plane movement of the pelvis bone). After this there are frames x 'X' and 'Z' values for movement across the ground plane by the pelvis.

It shouldn't be too much of a job to alter the script by anyone who knows a bit of Maxscript (not me unfortunately).

Back to the forge.

Cheers

GrumpyOldMan

[Caliban]

The question is whether the skeleton.dat and skeleton.idx take precedence over extracted files (ie does io.file_first work with skeletons)? If not then there may be a chance.

Hi Caliban

Verc's CAS importation script won't work with M2TW anims without alteration. I had a look at the animations in Oct/Nov when the first demo came out. Here's a quote with the details of the changes to the animation format:-

It shouldn't be too much of a job to alter the script by anyone who knows a bit of Maxscript (not me unfortunately).

Hi Grumpy,

The format still loads in our RTW viewer so I'm guessing if there are changes they were minor enough for them to still be compatible with our viewer tool.

When I'm changing descr_skeleton.txt or the animation files directly, I have to delete the .dat and idx files from the data\animation folder for the new changes to take effect. The game re-builds them on startup with the new changes in place. I hope this is still the case with retail version.

Caliban

[GrumpyOldMan]

Hi Caliban

When I'm changing descr_skeleton.txt or the animation files directly, I have to delete the .dat and idx files from the data\animation folder for the new changes to take effect. The game re-builds them on startup with the new changes in place. I hope this is still the case with retail version.

That's interesting, could be something worth pursuing after this and maybe requires another thread. One thing I was going to ask, does M2TW have a built-in sprite creater like RTW? Hopefully yes, so that new figures can be covered by new sprites.

Cheers

GrumpyOldMan

[Bwian]

The game definitely does NOT rebuild the DAT and IDX files on startup in retail. This was something that was gone into in depth for RTW, and we all tried it. The game just cannot find the animations and crashes with an error.

It was thought the extra event files were to blame, since they were not present in retail, and these were supplied to us by a CA source ( Jerome Grasdyke) to test this theory. That failed too. You simply cannot run the stock RTW retail exe file without the DAT and IDX files present, and it won't reference the skeleton.txt file to re-create the DAT from available files.

Or...at least... no-one ever actually got it to work!

[Lusted]

One thing I was going to ask, does M2TW have a built-in sprite creater like RTW? Hopefully yes, so that new figures can be covered by new sprites.

Already asked Caliban that, it is in the .exe, it just needs to be exposed. I asked too late for it to be added in 1.2, but it should be added in the next update.

[GrumpyOldMan]

Hi all

The game definitely does NOT rebuild the DAT and IDX files on startup in retail. This was something that was gone into in depth for RTW, and we all tried it. The game just cannot find the animations and crashes with an error.

That's a shame, has anybody tried altering the information within the animation and skelton pak and idx files? Something like give spearmen swordmen primary weapon anims. If this works, it means there is no CRC or file length check done within M2TW to make sure these files haven't been altered, and if this is the case then they CAN be altered and possibly new skeletons added. I think V's source code for his Xidx and Xpak are floating around somewhere.

Already asked Caliban that, it is in the .exe, it just needs to be exposed. I asked too late for it to be added in 1.2, but it should be added in the next update.

Shame that means that people are going to have to wait a few months to get the full benefit of any new figures.

Cheers

GrumpyOldMan

[KnightErrant]

Well not much progress on resolving the tangent space basis. Unweighted vertex normal calculations still show differences from CA's .mesh values. I've found several algorithms for doing weighted vertex normals but they all seem to have little effect on the bad vertices, and make some of the good vertices worse. Here's what I see if I just do unweighted vertex normal computations:

Original RGB  Computed RGB  Ref.            Original tangent space vector               Computed tangent space vector         
  z   y   x     z   y   x   count    mag          z         y        x           mag       z          y           x    Angle  
 188 119  15  189 120  16   [ 6]  +1.00407 ( +0.47451, -0.06667, -0.88235 )    +1.00000 ( +0.48075, -0.06093, -0.87474 ) 0.00000
 197  84  29  197  85  30   [ 7]  +1.00517 ( +0.54510, -0.34118, -0.77255 )    +1.00000 ( +0.54769, -0.33427, -0.76701 ) 0.00000
 246  82 127  234  88  69   [ 2]  +0.99558 ( +0.92941, -0.35686, -0.00392 )    +1.00000 ( +0.83236, -0.31026, -0.45926 ) 27.61001
 132 137   0  132 137   0   [ 7]  +1.00339 ( +0.03529, +0.07451, -1.00000 )    +1.00000 ( +0.03723, +0.07653, -0.99637 ) 0.00000
 167 113   7  168 113   7   [ 8]  +1.00106 ( +0.30980, -0.11373, -0.94510 )    +1.00000 ( +0.31579, -0.11333, -0.94203 ) 0.00000
 209 118  30  209 118  30   [ 7]  +0.99946 ( +0.63922, -0.07451, -0.76471 )    +1.00000 ( +0.64171, -0.07339, -0.76343 ) 1.88787
 238 121  65  239 121  66   [ 7]  +0.99700 ( +0.86667, -0.05098, -0.49020 )    +1.00000 ( +0.87373, -0.04966, -0.48388 ) 4.47212
 210 139  30  210 139  31   [ 6]  +1.00578 ( +0.64706, +0.09020, -0.76471 )    +1.00000 ( +0.64708, +0.09066, -0.75701 ) 0.00000
  72 130  12   73 131  12   [ 6]  +1.00523 ( -0.43529, +0.01961, -0.90588 )    +1.00000 ( -0.42872, +0.02385, -0.90312 ) 0.00000
  67 157  19   67 157  19   [ 5]  +1.00143 ( -0.47451, +0.23137, -0.85098 )    +1.00000 ( -0.47331, +0.23332, -0.84943 ) 0.00000
  21 125  56   21 126  57   [ 8]  +1.00627 ( -0.83529, -0.01961, -0.56078 )    +1.00000 ( -0.83194, -0.01543, -0.55465 ) 0.00000
  79 127   9   79 128  10   [ 6]  +1.00425 ( -0.38039, -0.00392, -0.92941 )    +1.00000 ( -0.37902, +0.00296, -0.92538 ) 0.00000
 142 241  72  143 241  72   [ 6]  +0.99743 ( +0.11373, +0.89020, -0.43529 )    +1.00000 ( +0.12088, +0.89312, -0.43327 ) 4.13269
 168 205  35  169 205  35   [ 5]  +0.99835 ( +0.31765, +0.60784, -0.72549 )    +1.00000 ( +0.32366, +0.60825, -0.72475 ) 3.30577
 192 164  24  192 165  24   [ 7]  +0.99841 ( +0.50588, +0.28627, -0.81176 )    +1.00000 ( +0.50739, +0.29189, -0.81078 ) 3.24299
 116 205  27  117 206  28   [ 6]  +0.99946 ( -0.09020, +0.60784, -0.78824 )    +1.00000 ( -0.08546, +0.61458, -0.78421 ) 1.95207
 130 160   4  130 161   4   [ 6]  +1.00180 ( +0.01961, +0.25490, -0.96863 )    +1.00000 ( +0.02185, +0.26097, -0.96510 ) 0.00000
 137 133   0  138 134   1   [ 7]  +1.00370 ( +0.07451, +0.04314, -1.00000 )    +1.00000 ( +0.08135, +0.04980, -0.99544 ) 0.00000
  99 232  61  100 233  61   [ 7]  +0.99687 ( -0.22353, +0.81961, -0.52157 )    +1.00000 ( -0.21910, +0.82662, -0.51835 ) 4.55719
  73 202  39   74 203  40   [ 5]  +1.00296 ( -0.42745, +0.58431, -0.69412 )    +1.00000 ( -0.42043, +0.59120, -0.68827 ) 0.00000
  69 165  20   69 165  21   [ 6]  +1.00394 ( -0.45882, +0.29412, -0.84314 )    +1.00000 ( -0.45738, +0.29634, -0.83844 ) 0.00000
  79 237  84   79 237  85   [ 5]  +0.99934 ( -0.38039, +0.85882, -0.34118 )    +1.00000 ( -0.38033, +0.86202, -0.33506 ) 2.12102
  47 206  67   47 206  68   [ 6]  +1.00143 ( -0.63137, +0.61569, -0.47451 )    +1.00000 ( -0.63105, +0.61812, -0.46873 ) 0.00000
  26 169  61   27 169  61   [ 6]  +1.00584 ( -0.79608, +0.32549, -0.52157 )    +1.00000 ( -0.79021, +0.32606, -0.51890 ) 0.00000
 247  85 127  213  98  38   [ 3]  +0.99477 ( +0.93725, -0.33333, -0.00392 )    +1.00000 ( +0.67328, -0.23465, -0.70117 ) 44.60200

The RGB values of the vertex normals are shown CA first then my computed values, then translated into float via the inverse cube normalization, again CA first then my computed values. The float normals have their magnitudes shown before the vectors just to verify they are unit normals. The Figure Of Merit is the last column, this is the angle in degrees between CA's original number and what I'm getting just from geometry. (These are for the mailed_knights_lod0.mesh, got tired of looking at the feudal_knights.) The two bad normals are shown in red. The tangents for the first twenty-five vertices are next:

 239 127 188  239 128 189   [ 6]  +0.99496 ( +0.87451, -0.00392, +0.47451 )    +1.00000 ( +0.87642, +0.00192, +0.48154 ) 5.77339
 228 116 204  228 116 205   [ 7]  +0.99471 ( +0.78824, -0.09020, +0.60000 )    +1.00000 ( +0.79155, -0.09001, +0.60444 ) 5.89615
 127 127 255  183 109 241   [ 2]  +1.00002 ( -0.00392, -0.00392, +1.00000 )    +1.00000 ( +0.43474, -0.14849, +0.88823 ) 27.48816
 254 130 132  255 131 133   [ 7]  +0.99298 ( +0.99216, +0.01961, +0.03529 )    +1.00000 ( +0.99888, +0.02624, +0.03934 ) 6.80755
 248 130 167  248 130 168   [ 8]  +0.99477 ( +0.94510, +0.01961, +0.30980 )    +1.00000 ( +0.94870, +0.02128, +0.31547 ) 5.86652
 225 133 209  225 133 209   [ 7]  +0.99761 ( +0.76471, +0.04314, +0.63922 )    +1.00000 ( +0.76685, +0.04576, +0.64019 ) 3.96203
 189 128 239  189 129 239   [ 7]  +0.99872 ( +0.48235, +0.00392, +0.87451 )    +1.00000 ( +0.48482, +0.00840, +0.87457 ) 2.91006
 223 136 210  224 136 211   [ 6]  +0.99205 ( +0.74902, +0.06667, +0.64706 )    +1.00000 ( +0.75444, +0.06709, +0.65293 ) 7.23046
 242 134  73  243 135  73   [ 6]  +0.99589 ( +0.89804, +0.05098, -0.42745 )    +1.00000 ( +0.90249, +0.05707, -0.42692 ) 5.21208
 239 128  65  239 129  66   [ 5]  +1.00253 ( +0.87451, +0.00392, -0.49020 )    +1.00000 ( +0.87478, +0.01106, -0.48439 ) 0.00000
 198 131  21  198 132  21   [ 8]  +1.00210 ( +0.55294, +0.02745, -0.83529 )    +1.00000 ( +0.55394, +0.03452, -0.83184 ) 0.00000
 245 136  79  245 136  79   [ 6]  +0.99922 ( +0.92157, +0.06667, -0.38039 )    +1.00000 ( +0.92321, +0.06987, -0.37790 ) 2.28326
 233 146 196  233 147 196   [ 6]  +0.99718 ( +0.82745, +0.14510, +0.53725 )    +1.00000 ( +0.82806, +0.14999, +0.54020 ) 4.31214
 238 140 188  239 141 188   [ 5]  +0.99292 ( +0.86667, +0.09804, +0.47451 )    +1.00000 ( +0.87271, +0.10401, +0.47703 ) 6.83072
 233 136 197  234 137 197   [ 7]  +0.99310 ( +0.82745, +0.06667, +0.54510 )    +1.00000 ( +0.83349, +0.07262, +0.54775 ) 6.74132
 254 141 124  254 141 125   [ 6]  +0.99817 ( +0.99216, +0.10588, -0.02745 )    +1.00000 ( +0.99374, +0.10938, -0.02257 ) 3.48514
 254 134 132  255 135 132   [ 6]  +0.99409 ( +0.99216, +0.05098, +0.03529 )    +1.00000 ( +0.99753, +0.05876, +0.03847 ) 6.24797
 254 128 137  255 128 138   [ 7]  +0.99496 ( +0.99216, +0.00392, +0.07451 )    +1.00000 ( +0.99665, +0.00455, +0.08168 ) 5.76905
 250 141  98  251 142  98   [ 7]  +0.99391 ( +0.96078, +0.10588, -0.23137 )    +1.00000 ( +0.96661, +0.11154, -0.23071 ) 6.33577
 239 137  67  239 137  67   [ 5]  +0.99774 ( +0.87451, +0.07451, -0.47451 )    +1.00000 ( +0.87842, +0.07529, -0.47191 ) 3.86301
 240 136  69  240 136  69   [ 6]  +0.99675 ( +0.88235, +0.06667, -0.45882 )    +1.00000 ( +0.88585, +0.06924, -0.45877 ) 4.62367
 213 128  33  213 129  33   [ 5]  +0.99952 ( +0.67059, +0.00392, -0.74118 )    +1.00000 ( +0.67444, +0.01062, -0.73825 ) 1.83152
 206 132  27  206 132  28   [ 6]  +1.00081 ( +0.61569, +0.03529, -0.78824 )    +1.00000 ( +0.61931, +0.03756, -0.78425 ) 0.00000
 199 134  22  199 135  22   [ 6]  +1.00088 ( +0.56078, +0.05098, -0.82745 )    +1.00000 ( +0.56412, +0.05617, -0.82378 ) 0.00000
 140 164 248  220 128 216   [ 3]  +0.99236 ( +0.09804, +0.28627, +0.94510 )    +1.00000 ( +0.72236, +0.00644, +0.69148 ) 43.43275

Again, the problem vertices are in red. Since the tangents are orthogonalized via Gram-Schmidt to the normals, anything wrong in the normals gets transferred to the tangents. Finally, here's the first twenty-five binormals:

 131 254 120  131 255 121   [ 6]  +0.99428 ( +0.02745, +0.99216, -0.05882 )    +1.00000 ( +0.02766, +0.99814, -0.05432 ) 6.13844
 162 247 100  162 247 100   [ 7]  +0.99909 ( +0.27059, +0.93725, -0.21569 )    +1.00000 ( +0.27108, +0.93817, -0.21529 ) 2.44175
 172 246 127  171 247 126   [ 2]  +0.99279 ( +0.34902, +0.92941, -0.00392 )    +1.00000 ( +0.34378, +0.93898, -0.01128 ) 6.91230
 123 254 137  124 255 137   [ 7]  +0.99558 ( -0.03529, +0.99216, +0.07451 )    +1.00000 ( -0.02916, +0.99672, +0.07547 ) 5.40326
 129 254 112  130 254 113   [ 8]  +0.99965 ( +0.01176, +0.99216, -0.12157 )    +1.00000 ( +0.01571, +0.99333, -0.11424 ) 1.59825
 129 254 116  129 255 117   [ 7]  +0.99632 ( +0.01176, +0.99216, -0.09020 )    +1.00000 ( +0.01205, +0.99625, -0.08565 ) 4.92648
 132 254 123  133 255 123   [ 7]  +0.99341 ( +0.03529, +0.99216, -0.03529 )    +1.00000 ( +0.03936, +0.99873, -0.03142 ) 6.58847
 113 254 130  113 254 131   [ 6]  +0.99885 ( -0.11373, +0.99216, +0.01961 )    +1.00000 ( -0.10998, +0.99362, +0.02499 ) 2.77893
 122 254 133  122 255 133   [ 6]  +0.99403 ( -0.04314, +0.99216, +0.04314 )    +1.00000 ( -0.04136, +0.99809, +0.04600 ) 6.26637
 140 251 154  141 251 154   [ 5]  +0.99551 ( +0.09804, +0.96863, +0.20784 )    +1.00000 ( +0.10362, +0.97234, +0.20934 ) 5.43685
 123 254 130  123 255 130   [ 8]  +0.99298 ( -0.03529, +0.99216, +0.01961 )    +1.00000 ( -0.03198, +0.99929, +0.02017 ) 6.79705
 119 254 131  119 255 131   [ 6]  +0.99477 ( -0.06667, +0.99216, +0.02745 )    +1.00000 ( -0.06354, +0.99755, +0.02921 ) 5.86489
  57 181 219   58 182 219   [ 6]  +0.99841 ( -0.55294, +0.41961, +0.71765 )    +1.00000 ( -0.54745, +0.42407, +0.72143 ) 3.25792
  80 227 190   81 228 191   [ 5]  +0.99403 ( -0.37255, +0.78039, +0.49020 )    +1.00000 ( -0.36554, +0.78690, +0.49717 ) 6.29087
  99 249 153  100 249 154   [ 7]  +0.99903 ( -0.22353, +0.95294, +0.20000 )    +1.00000 ( -0.21876, +0.95369, +0.20644 ) 2.56410
 118 227 206  118 227 207   [ 6]  +0.99681 ( -0.07451, +0.78039, +0.61569 )    +1.00000 ( -0.07191, +0.78123, +0.62009 ) 4.58260
 118 250 160  119 250 161   [ 6]  +0.99681 ( -0.07451, +0.96078, +0.25490 )    +1.00000 ( -0.06675, +0.96356, +0.25904 ) 4.60343
 126 254 133  126 255 134   [ 7]  +0.99316 ( -0.01176, +0.99216, +0.04314 )    +1.00000 ( -0.00860, +0.99875, +0.04926 ) 6.71420
 144 197 232  144 198 232   [ 7]  +0.99279 ( +0.12941, +0.54510, +0.81961 )    +1.00000 ( +0.13290, +0.55159, +0.82346 ) 6.88748
 156 229 197  156 230 198   [ 5]  +0.99037 ( +0.22353, +0.79608, +0.54510 )    +1.00000 ( +0.22718, +0.80300, +0.55098 ) 7.95746
 137 248 165  137 249 165   [ 6]  +0.99261 ( +0.07451, +0.94510, +0.29412 )    +1.00000 ( +0.07790, +0.95257, +0.29418 ) 6.97467
 208 192 202  208 192 202   [ 5]  +0.99798 ( +0.63137, +0.50588, +0.58431 )    +1.00000 ( +0.63283, +0.50676, +0.58542 ) 3.63939
 187 227 179  187 228 179   [ 6]  +0.99496 ( +0.46667, +0.78039, +0.40392 )    +1.00000 ( +0.46715, +0.78519, +0.40651 ) 5.75704
 158 247 156  158 248 157   [ 6]  +0.99279 ( +0.23922, +0.93725, +0.22353 )    +1.00000 ( +0.23946, +0.94368, +0.22832 ) 6.88633
 167 242  88  148 251 105   [ 3]  +0.99922 ( +0.30980, +0.89804, -0.30980 )    +1.00000 ( +0.15774, +0.97206, -0.17384 ) 12.66437

Oddly, even though I've outlined the two problem vertices in red, it's not
really that much of a problem for the binormals, indicating that the
discrepancy is a rotation about the binormal axis, whatever that means.

What does this mean in-game? I don't know, here's a picture of the mailed
knights in custom battle (unimaginative, but the lod0's again are the ones
with the broken lances):




They look ok but I'm not sure what I'm looking for. I should mention that
the tangents and binormals for CA's original data contain runs of the RGB
triples 127, 127, 127, which when inverse cube normalized become the zero
vector meaning they have a degenerate tangent space basis. What this
implies for lighting/texturing I have no idea. Since all the higher lods don't
have these bases vectors it probably isn't important.

All I've got for now.

KE

[Bwian]

GOM...
Gave this a try this morning. Using V's XIDX it is possible to extract the skeleton.dat and idx file, so I did that and swapped over MTW2_Spear and MTW2_Swordsman. REbuilt the file and tried running the game. It crashes with an unspecified error right at the start...so it was obviously reading the PAK and IDX files and finding something it didn't like. Don't know exactly what it was checking...but it checked something!

I tried swapping them back .... but this still crashed.

When I checked the file size against the copies I took before I messed with my files, I noticed the skeleton.dat file had got smaller..... seems the XIDX rebuild doesn't give a usable dat file, making it impossible to actually tell if the test was viable, or if the tool just didn't give a usable end product for MTW2