A start on the .MESH file format

**Andromachus Theodoulos** · 03-09-2007, 16:32

@All...

This is awesome, I am 36 years old and I feel like a little kid in a hobby shop. Having the ability to add and remove weapons and shields??? Awesome.

@GOM,
Will MilkShape allow us to modify the actual mesh, like say if we wanted to modify a helmet to make it look different. Would we be able to do that??

And I assume that if we can switch shields and weapons around, we can also change stuff like the helmet models as well, because some of these are attachments, I think?

Very nice work.

Can't wait for the converters and other stuff to be ready...

Like most everyone else, the code stuff is over my head right now, but the excitement is almost uncontainable...

Also, another note to all...

Would we be able to use MilkShape to modify the Rome Total War models as well. I know they are in a cas format (I think) what extra programming would be required there???

Again, Thanks for all the hard work, you are going to make a lot of Total War diehards very happy...

AT

**Suraknar** · 03-10-2007, 02:50

As Caliban says the mesh file being used is not any sort of file that will work with a current plugin, etc. We don't have to follow the CA development path to come up with working mesh files, just something that the engine recognises and will use. We shouldn't try to make this harder than it really is. I've had some feedback from Caliban and this seems to be getting a bit easier.

What I'm planning is:-

1 A .MESH to MS3D converter - this is about 98% complete

2. Load .MESH and export single group as .MS3D - about 65% complete

3. A .MS3D to .MESH converter - A gleam in my eye but shouldn't take too long

Any other thoughts?

Cheers

GrumpyOldMan

Yes this is sensible indeed, it will permit new units, and Milkshape is very accessible compared to other 3D packages, I for one am excited!

Great progress all! :)

**KnightErrant** · 03-10-2007, 03:50

Well I'm embarassed

, I put my excised unit
in my mod folder, not the true /data/unit_models/_units
etc. last night. I put it in the right place and now I'm crashing
the game. I tried the set
3 to 0 and remove from the number of vertices to the
end of mystery data as well as removing the whole entry
including what I think are the headers and footers but no
luck. Crashes with an unspecified error. Casuir, do you know
what I'm doing wrong?

**GrumpyOldMan** · 03-10-2007, 04:13

Hi All

I just wanted to post that after I put in a bug report on the Regroup groups function in Milkshape that it has been fixed and reuploaded to the Milkshape site. If you've downloaded Milkshape 1.8 before you may want to redownload to get the fixed version.

@Andromachus Theodoulos

Milkshape is a full mesh manipulation tool so you can modify mesh appearance, vertex/bone assignments, etc. Once this all up and running, the possibilities extend well beyond just switching weapons/shields of existing figures. From what I can see and have understood, as long as your figure is understood by the engine it can be used; so this means heads, bodies, arms, legs, weapons, etc will be able to be swapped or exchanged for new ones. As long as they all fit together and have the right values, they'll work.

I could probably do a .CAS to MS3D converter but that may have to wait for a while, although now I think of it I may do one that extracts an unassigned mesh and the M2TW skeleton (different skeletons used). Note to RTW modders, meshes will work with 100% assignments (just not look as purty) but the meshes will have to be sliced up into chunks as groups. Might be easier to use closest fit M2TW body parts and new textures.

Cheers

GrumpyOldMan

**KnightErrant** · 03-10-2007, 06:22

Houston we have a problem............

It's always good to question one's assumptions.
The mystery data is used in the game to control
shading and texturing rendering. Because I couldn't
get the excised file to load I wrote a Python script
to zero out the mystery data but leave the file otherwise
as is. It loads and plays but the graphical results
are disappointing. The unit (hard-coded to feudal_knights,
my favorite test case) flickers in and out of sunlight in
a custom battle and the unit colors also render oddly.

I'm sending an attachment to GOM and Casuir to test
but we need a better answer to what the mystery data
is. I've been working on the .ms3d format and there
doesn't seem to be anywhere to place more data (shader
or whatever this is).

Not to worry, this isn't a showstopper but we need to know
what this is and how to deal with it.

Here's the code if anyone else wants to experiment:
(The getters and putters are from my main script
for converting mesh files, I don't use all of them here.)

Code:

import struct

global bytecount
global ms3dbytecount

# -----------------------------------------------------------------------------------
def getbyte( fidin ) :
    global bytecount
    (thebyte,)             = struct.unpack( 'b', fidin.read(1) )
    bytecount              = bytecount + 1
    return thebyte

# -----------------------------------------------------------------------------------
def getubyte( fidin ) :
    global bytecount
    (thebyte,)             = struct.unpack( 'B', fidin.read(1) )
    bytecount              = bytecount + 1
    return thebyte

# -----------------------------------------------------------------------------------
def getshort( fidin ) :
    global bytecount
    (theshort,)            = struct.unpack( 'h', fidin.read(2) )
    bytecount              = bytecount + 2
    return theshort

# -----------------------------------------------------------------------------------
def getushort( fidin ) :
    global bytecount
    (theshort,)            = struct.unpack( 'H', fidin.read(2) )
    bytecount              = bytecount + 2
    return theshort

# -----------------------------------------------------------------------------------
def getint( fidin ) :
    global bytecount
    (theint,)              = struct.unpack( 'i', fidin.read(4) )
    bytecount              = bytecount + 4
    return theint

# -----------------------------------------------------------------------------------
def getuint( fidin ) :
    global bytecount
    (theint,)              = struct.unpack( 'I', fidin.read(4) )
    bytecount              = bytecount + 4
    return theint

# -----------------------------------------------------------------------------------
def getfloat( fidin ) :
    global bytecount
    (thefloat,)            = struct.unpack( 'f', fidin.read(4) )
    bytecount              = bytecount + 4
    return thefloat

# -----------------------------------------------------------------------------------
def putbyte( thebyte, fidout ) :
    global ms3dbytecount
    fidout.write( struct.pack( 'b', thebyte ) )
    ms3dbytecount          = ms3dbytecount + 1
    return  

# -----------------------------------------------------------------------------------
def putubyte( thebyte, fidout ) :
    global ms3dbytecount
    fidout.write( struct.pack( 'B', thebyte ) )
    ms3dbytecount          = ms3dbytecount + 1
    return  

# -----------------------------------------------------------------------------------
def putshort( theshort, fidout ) :
    global ms3dbytecount
    fidout.write( struct.pack( 'h', theshort ) )
    ms3dbytecount          = ms3dbytecount + 2
    return  

# -----------------------------------------------------------------------------------
def putushort( theshort, fidout ) :
    global ms3dbytecount
    fidout.write( struct.pack( 'H', theshort ) )
    ms3dbytecount          = ms3dbytecount + 2
    return  

# -----------------------------------------------------------------------------------
def putint( theint, fidout ) :
    global ms3dbytecount
    fidout.write( struct.pack( 'i', theint ) )
    ms3dbytecount          = ms3dbytecount + 4
    return  

# -----------------------------------------------------------------------------------
def putuint( theint, fidout ) :
    global ms3dbytecount
    fidout.write( struct.pack( 'I', theint ) )
    ms3dbytecount          = ms3dbytecount + 4
    return  

# -----------------------------------------------------------------------------------
def putfloat( thefloat, fidout ) :
    global ms3dbytecount
    fidout.write( struct.pack( 'f', thefloat ) )
    ms3dbytecount          = ms3dbytecount + 4
    return  

# -----------------------------------------------------------------------------------
def putzerobytes( n, fidout ) :
    global ms3dbytecount
    for ii in range(n) :
        putubyte( 0, fidout )
        ms3dbytecount      = ms3dbytecount + 1
    return  


# Main.

global bytecount
global ms3dbytecount
bytecount                  = 0
ms3dbytecount              = 0

fnin                       = 'feudal_knights_lod0_test.mesh'
fnout                      = 'feudal_knights_lod0_testchop.mesh'
fidin                      = open( fnin, 'rb' )
fidout                     = open( fnout, 'wb' )

for ii in range(2*175) :
    fidout.write( fidin.read(500) )

fidout.write( fidin.read(477) )

# We're at the mystery blocks.
nvert                      = getuint( fidin )
putuint( nvert, fidout )
print 'nvert = ' + str( nvert )

for ii in range( nvert ) :
    sdummy                 = getushort( fidin )
    sdummy                 = getushort( fidin )
    putushort( 0, fidout )
    putushort( 0, fidout )

nheaderfooter              = 26
for ii in range( nheaderfooter ) :
    thebyte                = getubyte( fidin )
    putubyte( thebyte, fidout )

# 2nd mystery block.
nvert                      = getuint( fidin )
putuint( nvert, fidout )
print 'nvert = ' + str( nvert )

for ii in range( nvert ) :
    sdummy                 = getushort( fidin )
    sdummy                 = getushort( fidin )
    putushort( 0, fidout )
    putushort( 0, fidout )

nheaderfooter              = 26
for ii in range( nheaderfooter ) :
    thebyte                = getubyte( fidin )
    putubyte( thebyte, fidout )

# 3rd mystery block.
nvert                      = getuint( fidin )
putuint( nvert, fidout )
print 'nvert = ' + str( nvert )

for ii in range( nvert ) :
    sdummy                 = getushort( fidin )
    sdummy                 = getushort( fidin )
    putushort( 0, fidout )
    putushort( 0, fidout )

#fidin.seek( 229468, 0 )

fidout.write( fidin.read(500) )
fidout.write( fidin.read(600) )

fidin.close()
fidout.close()

**Casuir** · 03-10-2007, 06:25

Mkay, looks like I messed up here and tested the base figure file instead of the actual ingame one

Guess I should have been a bit more thoroug. Did some more testing and it looks like these values are necessary, changing the 3 to 0 crashs the game as does removing any of the values. Changing all the values to 00 in the first stream gave this result:

As you can see it has some effect on the lighting, unfortunatly my graphics card is on the way out and doesnt like the battle mode so I cant get a proper look at whats happening, can somebody else have a look at these and see whats happening, preferably someone whos machine can handle all the high-end bells and whistles. Arrow warriors have the first block zeroed out, eagle the second and cuahchiqueh the third.
http://www.wikiupload.com/download_page.php?id=99774
Scrrenshots would be nice

Looks like ya bet me to it KE

**GrumpyOldMan** · 03-10-2007, 08:26

Hi

@KE and Casuir

KE, I tried your example but I couldn't get much difference from the stock figure, I tried it with different combinations of shader system, shadow quality and anything else I could tinker with in the Options menu. Could this be card related - I'm using a 9800pro.

Casuir, I tried the download link you sent but I just got caught in a loop with not much being downloaded.

Thoughts - KE, I'm not familiar with Python, and seeing you've already got it set up

could you try your script with the non zero values being replaced by 127 or 255 or random 0-255?

KE, with your permission I'd like to expand the test on the modified mesh.

Would people like to try out the feudal knight mesh to see what happens on their system? I've just tried it about twenty times on my system and there have been no crashes or other strange behaviour. If KE is happy I can post a download link.

Cheers

GrumpyOldMan

**Casuir** · 03-10-2007, 08:45

Try this one, the first one wont let me dl either
http://www.megaupload.com/?d=O8YR7OVW

Managed to get a bit more messing done and changed the values around a bit

first one is 00 00 ff 00, second one is 00 ff 00 00
Thought they might have been rgb values at first but colours change depending on the weather, the lighting effect on the models there seems to be produced from the odd cubemaps that are in globallighting. Definitly looks like one of the values looks to calculate that environment mapping type effect is messed up. Changing the second and third blocks doesnt seem to have any effect that I've noticed, but I seem to be missing a bit lately.

I've got a 9600gt so you should be able to reproduce anything I can, are you sure you have the game set up to use modded files? I find using --io.file_first in a bat file works better than a cfg file.

**KnightErrant** · 03-10-2007, 16:37

@GOM

Of course, you have my permission to do anything
that helps figure things out.

This Saturday is one of those errands and yardwork days
(spring has come to Huntsville) but I'll start back again
tonight looking at things. Maybe I should make my cobbled
together script more general so it searches for the start of
the mystery block for any unit rather than a hard-coded
byte value. Any thoughts on whether that would be helpful?

@Casuir
The first picture you posted looks exactly like what I'm
seeing, shadows and stuff on different guys in the unit
depending on lighting.

KE

**GrumpyOldMan** · 03-11-2007, 02:11

Hi Guys

I finally made my in-game stuff look like yours. Looks like the mystery blocks are not just harmless boolean streams as we were led to believe. I've sent a PM to Caliban and hopefully we can get some more information.

What this means -

Can we still go ahead? - Yes, with the restriction that we can only use M2TW groups until we figure out how to replicate the shading data. However, creative texturing and use of alpha maps will give some leeway for variations.

Is Milkshape still an option? Yes, I can store the mystery blocks within the file and parse it out on conversion to Mesh.

All is not lost and the way ahead is clearer now

Off to mow the lawn, then rewrite the converter

and read up on shader streams

.

Cheers

GrumpyOldMan

**Suraknar** · 03-12-2007, 03:12

hrm...

This looks like Vertex Shading, Shininess and refraction in relation to weather lighting and environment, so it is information for the engine's rendering instructions.

Just a thought, are these mystery values the same accross the board for all units?

If they are then maybe the Converter could just add(insert) this information to the final .mesh file upon export?

EDIT: NVM, I posted this after reading page 5 of the thread, did not notice there was Page 6, which I am reading now :P

**KnightErrant** · 03-12-2007, 03:17

Yes, the format seems to be the same for all units.
The aztec variants that Casuir found had float triple
streams for the mystery data but those units aren't
used in-game (don't appear in the EDU) so were they
mistakes, early prototypes with a different format, etc. ?

Did a quick google on normal map tutorials and there
are a few so maybe I should just read up for a few evenings
until I can catch up.