A start on the .MESH file format

[zxiang1983]

@Bwian
Please check my post here (It's a tutorial about extracting animation files):
http://www.twcenter.net/forums/showthread.php?t=74055

The skeleton.idx/dat file is not so easy to extract as it appears.

[Andromachus Theodoulos]

@ All

Any new updates on this yet???

AT

[KnightErrant]

Hi AT,

GrumpyOldMan is working on the .ms3d to .mesh part.
I've updated my script to pull all the hard-coded stuff out.
I now have external ASCII files for the standard skeleton,
the horse, camel, and elephant skeletons. Last night I
installed Milkshape 1.8.0 which has a slightly changed format
specification so I made those changes as well. I modded the
armored sergeants and put them in a custom battle and I could
see the mods when I zoomed in on them but the moment I rotated
the camera the game crashed. At first I thought it was something wrong
with how I did the format but that doesn't make any sense because they
wouldn't have loaded at all.

Here's my best guess on this one (it's a guess because I can't check until I
get home). I've noticed for a while one of the bone names in the .ms3d
file was odd, something like bone_weapon01_shield01 or something, not the
original name out of the mesh. I though GOM had renamed it for some reason
but I never asked. It never gave me a problem because I hard-coded the
bone names in my script. I think Mete's program doesn't like joint names
that aren't unique enough and its changing that one. (The weapon/shield
names are like: bone_weapon01, bone_weapon, bone_weapon02,
bone_weapon03, bone_shield01, bone_shield.) Now that I'm not
hard coding I'm getting that bad bone name back in the mesh and the
moment an animation references the unknown name it crashes.

Before I took this out last night, I had been storing the original bone names
and node indices in the model comments along with the boundingsphere
data. But I thought I didn't need to do this anymore. If my guess is
correct then I'll just go back to storing the original names and everything
should work ok just like before. I just need to check with GOM to see
if he thinks this is the right way to handle this or not. The nice thing about
storing stuff in the comments sections is you can see the data in
Milkshape and can update things like bounding sphere numbers and
group types/group names as you add stuff and then you know it will get
written back to the mesh.

Anyway, that's my update. Except for a new episode of "Lost", I've got
a clear evening to work on this stuff.

KE

[GrumpyOldMan]

Hi KE et al

Hi AT,
Here's my best guess on this one (it's a guess because I can't check until I
get home). I've noticed for a while one of the bone names in the .ms3d
file was odd, something like bone_weapon01_shield01 or something, not the
original name out of the mesh. I though GOM had renamed it for some reason
but I never asked. It never gave me a problem because I hard-coded the
bone names in my script. I think Mete's program doesn't like joint names
that aren't unique enough and its changing that one. (The weapon/shield
names are like: bone_weapon01, bone_weapon, bone_weapon02,
bone_weapon03, bone_shield01, bone_shield.) Now that I'm not
hard coding I'm getting that bad bone name back in the mesh and the
moment an animation references the unknown name it crashes.
KE

I'm sorry, it's one of the things I had to do to get the MESH information into Milkshape, but, unfortunately, didn't explain to anybody. In the MESH file the skeleton's last few bones are stored as below:-

11 bone_pelvis 0
11 bone_rthigh 1
14 bone_rlowerleg 2
10 bone_rfoot 3
8 bone_abs 4
10 bone_torso 5
9 bone_head 6
8 bone_jaw 7
12 bone_eyebrow 8
14 bone_rclavical 9
14 bone_rupperarm 10
11 bone_relbow 11
10 bone_rhand 12
14 bone_lclavical 13
14 bone_lupperarm 14
11 bone_lelbow 15
10 bone_lhand 16
11 bone_lthigh 17
14 bone_llowerleg 18
10 bone_lfoot 19
13 bone_weapon01 20
11 bone_weapon 20
13 bone_weapon02 21
13 bone_weapon03 22
13 bone_shield01 22
11 bone_shield 22

You'll notice that the last three bones all have the same bone index '22', this is used for their attachment and weighting. Nothing would crash Milkshape faster than trying to have 3 bones all with the same index. So what I did was amalgamate the names of all three as a Milkshape bone so they could share the same index. Caliban has told me that the actual attachment of weapon/shield to any particular bone is irrelevant, I've found one attached to the bone_pelvis. It must be parsed in on the class/name string, rather than on the bone attachment. What's important is that vertices are attached to the correct index as per the Mesh format. When the figure is reconverted, the bones and indices as above should be used rather than a reading of the Milkshape skeleton.

Again apologies for not making that clear in the beginning.

At the moment I'm migrating my ms3d to mesh code into the final prog and testing/debugging. The final program has mesh to ms3d (fully operational, unless someone finds a bug) and Merge ms3d (fully operational, unless someone finds a bug) because I wasn't happy with the native one in Milkshape.

Cheers

GrumpyOldman

[KnightErrant]

Well I found the problem with the bad weapon name,
programming error on my part (usually is). So don't have
to store .mesh bone names in the model comments
after all. Just have to get the right bone indices on
reading out like you said.

[GrumpyOldMan]

Hi KE

Well I found the problem with the bad weapon name,
programming error on my part (usually is). So don't have
to store .mesh bone names in the model comments
after all. Just have to get the right bone indices on
reading out like you said.

As long as the information is valid, it doesn't seem to make any difference to M2TW, if you have a look at the extracted ms3d figures, most of the weapons including bows are attached to the right hand. This is how they are stored in the Mesh format and why I'm guessing it's parsed out on the class/name data.

Had another thought on the 9 bytes, are you somehow recording the carriage returns with the altered model comment format - just another wild stab in the dark

Back to debugging.

Cheers

GrumpyOldMan

[KnightErrant]

@GOM
My bad, the 9 bytes was the string bone_weapon_weapon01
I copied into the skeleton file without noticing.
The _weapon01 was the 9 bytes.

Anyway, I got armored_spearmen with bent shields, these
were the guys that crashed the game with the above bone name.
Funny thing, it doesn't crash on loading, it seemed to have to wait
until something accessed the bad name.



Also did a timurid unit equipped with the deadly Dumbo upgrade



Was shooting for a hat trick tonight (3 for 3) but the game crashed on the
trebuchet. It looks like the artillery probably requires a skeleton file each.
Haven't looked at anything but the trebuchet but the bones seem very
specific to that unit which makes sense. I'd like to figure out what
crashed the trebuchet, probably misspelled something. I did learn
that weight is sometimes spelled as bone_wieght.

Will test more units to look for any surprises but it seems all the obstacles
are just about done with.

Edit: Yes, I did switch over to the \n delimiter for group comments and
model comments.

[GrumpyOldMan]

Hi KE

Well done!!!!!!

I probably won't have anything to show until tomorrow now, Got hit by plague of NaN's, had to go through all my code until I could find the missing '0' trap for a normalization

Once I get it all bedded down, I'll try something like adding a head in all LOD levels and see what happens. I'll also try various things like moving Milkshape groups up and down the group order and see if that throws out M2TW.

Be interested to see what you get up to with artillery.

Again, well done!!!!!!!!!!!

Edit:- Call me a Philistine of Eye Candy but I can't see any huge difference in the shading/bump mapping. Maybe we should release a MESH and get people (who unlike me, don't have tired old eyes and multifocals) to see if there is a difference?

Cheers

GrumpyOldMan

[AlphaDelta1]

Maybe we should release a MESH and get people (who unlike me, don't have tired old eyes and multifocals) to see if there is a difference?

That sounds like a stunning idea. And if it could be this middle eastern swordsmen mesh (dismounted Qapukulus) the entire Broken crescent mod team would worship your every step on this earth.

Cheers

[KnightErrant]

Wow! Stunning graphics AlphaDelta1! I think I know
what to try tomorrow. The worst vertex normal problems
I've seen are the feudal knights. I'll do (it's after 12:00
my time and my wife is wondering what the h*** I'm doing
up this late) a front shot and back shot of the vanilla guys
versus the converted guys and see if the skinners see anything
that I can't see. That should settle the tangent space basis
question.

Re: granny strings. When you code this could you try out the
feudal knights? I've almost memorized the byte positions of
the blocks and would be happy to look at header/footers.
It's odd how the pattern recognition algorithm works in humans
but I couldn't "see" whoopsies in code, only in side-by-side
hexedit sessions between the back-and-forth meshes and the
originals.

Oh, to have a life again. GOM, are you familiar with the
American colloquialism "stuckee"? Once you release your
converter you're the "stuckee" for the next few months.

Don't plan any vacations, mate.

KE

[GrumpyOldMan]

Hi AD1

That sounds like a stunning idea. And if it could be this middle eastern swordsmen mesh (dismounted Qapukulus) the entire Broken crescent mod team would worship your every step on this earth.

Cheers

Followed your link but all I can see is a big ad but no picture.

At this point I'm not talking about a whole new figure in all LODs (that will come as soon as we release the tools) but a modified mesh that can be plugged in straight away and used immediately. This is so we can check on the shader/bumpmapping, that we calculate a little differently (but close) to CA's figures.

I may do a repair job on some of the meshes in the ee_bekhtera_heavy_lamellar folder which since patch 1.1 have only one body variant included in the mesh.

Once the tools are released, coming up with your own meshes will be easier than all the related text entries.

Cheers

GrumpyOldMan

[GrumpyOldMan]

Hi KE

Re: granny strings. When you code this could you try out the
feudal knights? I've almost memorized the byte positions of
the blocks and would be happy to look at header/footers.
It's odd how the pattern recognition algorithm works in humans
but I couldn't "see" whoopsies in code, only in side-by-side
hexedit sessions between the back-and-forth meshes and the
originals.

Oh, to have a life again. GOM, are you familiar with the
American colloquialism "stuckee"? Once you release your
converter you're the "stuckee" for the next few months.

Don't plan any vacations, mate.

KE

I was planning on doing the Feudal Knight first up for you to have a look at and then I might do a variation by adding a Mailed Knight head to all LOD levels and see what happens.

Maybe I'll make up some stuckee rules like releasing the tools gradually with a timelimiting feature so that the person has to write a tutorial to use it more

We have to stop cross posting like this

Cheers

GrumpyOldMan

[AlphaDelta1]

No worries GoM, I was just trying my luck..'Who dares wins' etc etc.

The link should work, you just need to wait for the adverts to finish loading I think..This link might work better.

Cheers and thanks once again for what you are doing!

[GrumpyOldMan]

Hi AD1

No worries GoM, I was just trying my luck..'Who dares wins' etc etc.

The link should work, you just need to wait for the adverts to finish loading I think..This link might work better.

Cheers and thanks once again for what you are doing!

Saw the picture with the new link, waited for 2 minutes on broadband and still couldn't see anything with the original link.

I thought I'd try and see how long this would take to do in Milkshape, this https://i150.photobucket.com/albums/..._Quapukulu.jpg took me 6.5 minutes to convert to MS3D, find a figure with suitable scimitars convert and extract and add them to the Quapukulu figure, delete the lance and maces. Probably another 2 minutes tops to update the group class and name text entries and update the bounding sphere text for a foot figure and it's done, Do the same thing for the other three LOD levels and ouila! the new figure is ready to go after converting back to .MESH.

Cheers

GrumpyOldMan

[AlphaDelta1]

Hubba hubba!

Can the community download this to check the shader/bumpmapping yet? During my work on Broken crescent I've been doing extensive work with the .normal files and have been testing my new textures with both shader models, if there's anything wrong, I'll spot it!

Cheers

[alpaca]

Hi AD1



Saw the picture with the new link, waited for 2 minutes on broadband and still couldn't see anything with the original link.

I thought I'd try and see how long this would take to do in Milkshape, this https://i150.photobucket.com/albums/..._Quapukulu.jpg took me 6.5 minutes to convert to MS3D, find a figure with suitable scimitars convert and extract and add them to the Quapukulu figure, delete the lance and maces. Probably another 2 minutes tops to update the group class and name text entries and update the bounding sphere text for a foot figure and it's done, Do the same thing for the other three LOD levels and ouila! the new figure is ready to go after converting back to .MESH.

Cheers

GrumpyOldMan

Wow, great :)
We need in-battle screens though

[KnightErrant]

@alpaca or AlphaDelta1
PM me an e-mail address and I'll send you a converted
feudal_knight_lod0.mesh . This case has normals that don't
agree with the original mesh. Also, the original mesh has
runs of zero vector tangents and binormals while the converted
one has a full basis set at each vertex. Should be a good test case.

[AlphaDelta1]

PM inbound.

Cheers

[KnightErrant]

On its way.

[GrumpyOldMan]

Hi All

I finally got all the converter code working in the one app but in the process I broke the Merge MS3D code . Will have to go back and do this (it's definitely needed).

Here are some in battle shots of converted/modified/reconverted meshes, the lance has been bent on the lod0 and the two body variations have been taken out so they're all wearing the same outfit (just like in the old Robin Hood movies )

https://i150.photobucket.com/albums/...ertedmesh2.jpg https://i150.photobucket.com/albums/...ertedmesh1.jpg https://i150.photobucket.com/albums/...vertedmesh.jpg

Back to debugging the merge code

edit:- Didn't take me long, it's all to do with file names and overwriting files, I may have to put a check in that people can't overwrite files (if they're the same as either of the two original files) - won't be long.

edit2:- The other thing to look out for in KE's meshes is that the vertex weights have been coarsened to only two decimal points but I haven't seen it make any difference in animation.

Cheers

GrumpyOldMan

[AlphaDelta1]

Recieved the mesh.

Test 1 - Check it loads.

Pass with flying colours, no crashes so far and that bent lance is clear to see.

Test 2 - Check 'LoD pop' between LoD 0 and LoD 1 in shaders version 1.

Pass, with the usual 'LoD pop' associated with a switch between vanilla LoD 0 and LoD 1 models.. No clear change to the appearance of lighting.

I have to run back to work, will run more tests later.

Cheers and good work.

[GrumpyOldMan]

Hi All

Latest pic shows the dismounted Boyar with two bodies added and those bodies altered to use the unused texture variants.

https://i150.photobucket.com/albums/...haddedbits.jpg

This shows a standard mesh converted to Milkshape, extra pieces added, the uv values of those pieces altered within Milkshape and then exported out to MESH format.

Just doing a check that all lod levels are being converted correctly and adding some GUI controls/checks, etc.

Cheers

GrumpyOldMan

[SigniferOne]

GOM, would it be possible to have 'pools' of objects, so that you could put all shields into one mesh file, and have all units of the faction take randomly from that shield pool?

[GrumpyOldMan]

Hi SO

GOM, would it be possible to have 'pools' of objects, so that you could put all shields into one mesh file, and have all units of the faction take randomly from that shield pool?

The short answer is no, if you have all your shields (for example 40 shields) on one attachment set texture then you could put 40 shields on each mesh and they would show up as a random selection of shields ( but that is a pretty big overhead in mesh size). There are no separate shields or weapons in M2TW. All weapons and shields that are being used by a particular figure must be in that particular figures mesh. Don't be confused by the separate weapon/shield entries in the folders, these are CA's development meshes to build final meshes that have all bits attached, the development bits are not used in the retail version of the game.

Cheers

GrumpyOldMan

[AlphaDelta1]

Another test and a screenshot.

Test 3 - Check the difference between vanilla LoD 0 and modified LoD 0 in shaders version 2.

Pass. As far as I can see, the normals are working correctly, as is specular/shine. The helmet eye slots have a feeling of depth, and the lace on the tunic appears 'raised'. All ok to me.

Also, the original mesh has
runs of zero vector tangents and binormals while the converted
one has a full basis set at each vertex.

What change should we look for in-game?

edit2:- The other thing to look out for in KE's meshes is that the vertex weights have been coarsened to only two decimal points but I haven't seen it make any difference in animation.

I havent noticed any animation errors. A mount with animated fabric, such as the barded or mailed horse might be a better test case. Then again, I dont know much about vertex weighting as my 3D experience is mainly in modelling cold war era aircraft and cockpits.

If you need help testing the software I am happy to. Once I get to know the process I can write a tutorial. I have written tutorial before for such things as uv texture mapping, and mesh editing in 3Dsmax.

Cheers

[alpaca]

Did anybody try what happens when you charge with that lance btw?
Caliban said that the collision detection is linked to the model, so theoretically these knights shouldn't hit a thing (unless a higher LOD is used for that) :P

[KnightErrant]

@GrumpyOldMan
Excellent! You must have had less headaches with the
granny bytes than I did...

@AlphaDelta1
(1) re Zero tangents and binormal runs. These were only like
10 vertices at a time so I'm thinking it would be on small details
like the belt buckle or the little crosses stitched on the tunic
down below the knight's waist. I think I see some detail there
but the lighting and the poses are ever so slightly different who
knows what effect we're seeing. I think the conclusion is definately
the converted meshes are no worse than the originals so the process
is a success!

(2) The 1.8.0 format allows two types of vertex weighting, the old
way 0-100% or new way 0-255. I left my code with the old way, but this is
a quantization error of 1% versus 0.4%, so subtle it would be hard
to spot. Weighting is used to soften transistions from one bone to
the next. For example without weighting, if the upper arm bone moved
all the vertices grouped to that bone would move and the torso would
stay fixed. This would be very robotic looking. With weighting, you can
tie some of the torso vertices to both bones and some of the upper
arm vertices near the joint to both. This makes the mail or fabric move
more realistically as the bone moves. Like I said, 1% or 0.4%, it would
be hard to tell the difference.

I think there's going to be some very happy modellers out there soon.

[AlphaDelta1]

KE,

Any minor graphical errors and accuracy issues I'm sure can be worked on post release. The main thing is to get a stable and user friendly release out there that shows the community that the .mesh format is in their hands. Hopefully the org will post some coverage of your work on the front page.


alpaca,

Did anybody try what happens when you charge with that lance btw?

I tried, the result was the same as always;

Against infantry - squish

Against cavalry - treadmill running.

I doubt M2TW uses anything like per-poly collision detection, it would be far too slow.

Cheers

[KnightErrant]

@AlphaDelta1
GrumpyOldMan's release should be user friendly
and have nice features.
re collision detection:
The models carry bounding sphere data, x, y, z, and radius, maybe this
would be the collision detection mechanism when two spheres intersect?

[Andromachus Theodoulos]

@All...

I am ready to start breaking... uhh I mean testing things...

I really don't have to wait for a tutorial to start poking around, but having one would be a wonderful asset.

How close are we to a beta release???

AT