Printable View
Indeed you are right Casuir,Quote:
This wont be possible with an xml file afaik, the data is mixed up to much for it to be possible without extracting the parts from the .mesh and writing a completely new file.
Working from an Export of an Export is unfortunently not possible the XML way :( as I have come to realise in some tests and recently got it confirmed aswell.
So your efforts, KnightErrant and GrumpyOldman are the best way to go about it :)
Go! Go! Go! :2thumbsup: hehe :)
It was more the PrimaryAttachment and SecondaryAttachment skeletons I was wondering about. Looked at the desc_skeletons and theres a few weapons which have animations other than stand_idle or default but they seem to be mainly ready, hold and release animations for missiles. Doesnt look to have any effect ingame though. What I was wondering was would it be possible to animate the actual weapon itself, ie have the arms of the bow draw back when drawn and return when the arrow is released.
Btw theres more than one weapons bone in the mesh, whats the story there?
@Suraknar, GOM and KE are the guys you should hang your hopes on, the major work heres being done by them and my contribution is very limited compared to theirs.
Hi Guys
With regards to the standard '256' and non-standard '0' meshes I thought I'd go and look at some of the model files (export_descr_unit.txt and battle_models.modeldb ) to see if I could get some clues :idea2: . The non-standard files are NOT used. They are WIPs and anyway the vertex assignments are all screwed up because of the funny skeletons. After I realised this the plaintive cries of "Dohhh" echoed off the hills for a long time, :oops: this shows you shouldn't get too focussed on just the one issue. With this in mind I'll 'lock out' the non-standard meshes since everything they have is available on standard meshes anyway.
I'd just like to make the point that I am not interested in reverse engineering anything, Granny and .gr2 are a mystery to me and will forever remain so. I am just looking at the .MESH format that CA is using and looking at ways we can modify these files. There are lots of bits of data that I don't understand and I don't want to understand but I don't need to as I can just use them. :sweatdrop: Just a clarification in case anybody from Radgames is interested.
Getting closer now.
@Casuir
You said you did some tests on taking out the boolean streams and using the figure in game. What exact blocks did you take out? Just the num_vertex and stream figures, with the header/footer stuff intact or the header/footer stuff as well?
Interesting stuff you pulled out on weapon bones, we'll have to look closer at that stuff to see how we actually get a figure in game.
I'm PMing Caliban with some questions on weapon protocol. It's just about sorted out in my head (and backs of envelopes), it's a matter of getting it into code.
Cheers
GrumpyOldMan
They're definitly the exported caracters that are processed by the xmls, think theres a bit more to it than simply adding weapons, theres wip/base meshs in the mounts folder as well, geometry seems to be the same but the ingame file is considerably smaller than the base one, havent looked at them other than that though. Dont think we'll be able to replicate the method CA used to make the models anyways.
As for the booleans, I changed the 03 byte to 00 then tested that, then re-wrote most of the values with 00's then tested that, finally I deleted from the 2 byte num_vertex to the last byte of the third data stream (inclusive) and tested that. I also edited the texture on the model in question so I'd be sure the game was using the edited files.
Well guys, in reality I dont understand a third of what is beiing said here, even if I understand what the effort is all about.
I am no programmer, the Bits & Bytes you are talking about just go over my head :dizzy2:
It is only in the spirit of modding & cooperation and what I can think to help from having past moding experience (not only the small tweaks to this series) that I try to share, in hopes maybe that it can provide some insight to someone.
Modding is most of the time a guessing game, trial & error, and lots of patience and frustration at times (and I am sure you all know these things).
So that is that, again, wishing success to all :)
Hi Casuir
I've looked at the mounts as well, the meshes that are prefixed "mount_" look to be the in-game ones, the others also seem to have their vertex assignments screwy, does anyone want to try renaming or temporarily removing a non "mount_" mount mesh and then also with a "mount_" mesh and see if the game still runs? I think what we are looking at in the Data folder is a copy of their development and testing folder, after developing and testing were complete, it was easier and faster to export the whole lot to the production release, than to go through and check all the meshes (didn't KE say there were something like 3000 figure meshes alone) to make sure they were needed for the game - hard drive space is cheap now :laugh4: .
No I don't think we can do exactly what CA did to produce their meshes but I'm planning on the abilities to import meshes, import a single group (either exported from another mesh or done from scratch in a modelling program), delete a single group, etc. It may not be whizz-bang but it should have enough functionalty to keep modders going for a while. Mesh manipulation, vertex assignment and UV adjustment should be left in the hands of specialist programs rather than twisted and tortured at the hands of an amateur :laugh4: .
Will now go and have a look at the boolean streams, have to think a bit more about them.
Cheers, keep up the good work everyone.
GrumpyOldMan
Ok, the attachment skeletons are only used with weapons that contain more than one bone. Which is the case for say swords but not for shields.Quote:
Originally Posted by Casuir
Sword has an additional sword bone that is attached to a hand, and we use that skeleton to animate it. But the shield is attached directly to the hand bone so there is no need for additional animation.
At one stage there were going to be proper animated weapons such as bow strings etc but there were issues with this and so it never made it in. I don't think it will be possible to animate a bow string straight up, it would properly require a bit of hackery. I'm told animating weapon bones can cause some wierd effects. There is also a lot of junk files in the folders, tests and obsolete units etc, there was simply no way to check what resources the game used and so you might find a few things that were left in.
Regrets on the reverse engineering remark. I just meant
we want to understand header/footers enough to know
how to cut out things like the mystery blocks and still have
the units play in-game. The thing that worried me was looking
at a lod2 mesh file that only had one mystery block and not
three but still had this header:
This was the armoured_sergeants_lod2.mesh. I thought that theCode:18 0 48 0 0 0 1 0 0 0 1924 (vert. wts: float pairs) 0 0 0 0 17 0 49 0 0 0 18 0 48 0 0 0 0 0 1 0 0 0 0 0
23 0 1 0
50 0 0 0 0 0 0 0 0 0 1924 (vert. vecs: float triple) 0 0 0 0 22 0 1 0
51 0 0 0 23 0 50 0 0 0 0 0 2 0 0 0 0 0
28 0 1 0
52 0 0 0 2 0 0 0 0 0 1924 (vert. bones: byte quads) 0 0 0 0 27 0 1 0
53 0 0 0 28 0 52 0 0 0
28 0 54 0 0 0 3 0 0 0 1924 (mystery strm: byte quads) 0 0 0 0 27 0 55 0 0 0 28 0 54 0 0 0
0 0 0 0 0 0 0 0 0 0
Bounding sphere data: 4 floats
0.487427771091 0.393853664398 0.109047822654 1.71645057201
field, the one in red, was supposed to mean three blocks followed
and that we could zero it out and remove the data.
So now I'm confused as well, how do we repackage the mesh data
around the excised data? I echo GOM's question for Casuir, what is the
sequence of steps needed to remove mystery data? This might
help thrashing out the header/footers; I'm not convinced I split them
right, I just thought it was plausible.
Side note: I just took the unit count of 701 from the top of the modeldb file
and multiplied by 4 for the number of meshes to get something like
3000. Not every unit has 4 meshes, a lot have 3, and there are some
leftovers in the modeldb like the baltest units and some en_dummy units
near the end that aren't referenced in the EDU so they're not real units.
Well the few bits that were left in have helped us figure stuff out so cheers for that :) Shame about the animated weapons, would have been a nice touch. I'm guessing we'd have seen flails too if it had panned out. What was the main idea behind having attachments sets btw, less resource usage?
@GOM Doubt the other non-standard meshs will work ingame, they get skeletons assigned to them by the xmls used to add the weapons so I'm guessing thats why the vertex datas screwed up. The xmls Caliban provided can be found here incase you havent seen them https://forums.totalwar.org/vb/showthread.php?t=74083
Interesting all the .gr2's referenced here and in the other leftover attsetdb files are found in the unit_models folder but as .mesh's not .gr2
I wouldnt worry about too much functionality, as long as it can export functional .mesh's most people will be happy enough.
@KE
Cant test at the mo but I'm guessing if the boolean values arent used then it doesnt matter how many streams it says will follow.Quote:
Originally Posted by Casuir
Btw theres 3213 meshs including mounts and generals
@Casuir
Thanks, I should read (and remember) what you said before.
This will be my number 1 task for tomorrow, I keep all the header/
footers, change 3 to 0, remove the mystery data including its
4-byte int vertex count, append the footer data without change,
and test in-game.
A btw, I thank you for including me with GrumpyOldMan but
that's giving me way too much credit. I'm just following along, confused
by things like everybody else, (except Grumpy).
A humorous side note: Did you see Alpaca's comment?
We should start charging don't you think?Quote:
I wish I had more time on my hands so I could work a bit on this, too. Anyways, after you cracked this customer, you should also have a look at the settlement packages (it seems there's some cool stuff in there and it's a lot more flexible than RTW)
Hi all! I've gone through this whole thread for no less than 10 times and finally I think I understand most of its content, such as the structure of mesh file etc... But I still don't know how to import the mesh file (or part of it) into milkshape 3D. Would someone please tell me so I can join this historic work, too:)
Hi Guys, I tryed out the Rhino Program 3.0 and downloaded the plugin for MESH files, but it doesn't work still. It seems as though it should because even the Rhino program as options for joining triangles and stuff, can anyone help me why I can't import them?
@taylorj2 and zxiang1983
I think the guys (GOM, KE, Cas) are still working hard on trying to establish what exactly is necessary within the mesh files. I think they are also trying to figure out how to get the components repackaged so that they will play in game. They are getting very close. Hopefully it won't be too much longer and we can start making changes to meshes.
Keep up the good work!!!
AT
Hmmm maybe the reason we can't open the MESH files normally is because of the 2 bytes of dummy data? CA could of put them in there so we can't edit them
Probably not the answer but its a idea.
Na its a completely different format. Rhino's primarily for modelling in nurbs and isnt really suitable for modelling for games.
The .MESH format was written inhouse by our engine programmer (that rhino format is completely different as Casuir pointed out). It's not a radtools format and it doesn't contain any of the granny format. It's simply an optimized binary format for the engine to load. No need to worry on that front :)Quote:
There are lots of bits of data that I don't understand and I don't want to understand but I don't need to as I can just use them. Just a clarification in case anybody from Radgames is interested.
We re-use alot of base meshes between units so we didn't want to have to build unique meshes each time. The attachment sets let us add that extra detail without building it into the meshes. This lets us have a greater amount of total unit variations by mixing different body parts and attachments. The attachment sets were to keep the attachments grouped by type (such as gunpowder pouches and belts or jewlery).Quote:
What was the main idea behind having attachments sets btw, less resource usage?
Hi Guys
Not much progress because Milkshape 1.8 has come out and I've been playing with this new toy. Thought I'd try it on a request someone put in for a dismounted Hospitaller, took me about five minutes to delete the lances and rename the groups and update the comments with the details.
https://i150.photobucket.com/albums/...dismounted.jpg
and the vertex weighting looks like this:- https://i150.photobucket.com/albums/...tex_weight.jpg
1.8 has one annoying bug in that if you try to regroup two separate groups it crashes but this is still possible in 1.7.10 (save and reopen in 1.8). Both are still available:-
http://www.chumba.ch/chumbalum-soft/files/ms3d180.zip and http://www.chumba.ch/chumbalum-soft/files/ms3d1710.zip . The SDK has been updated also, http://www.chumba.ch/chumbalum-soft/...ms3dsdk180.zip.
As Caliban says the mesh file being used is not any sort of file that will work with a current plugin, etc. We don't have to follow the CA development path to come up with working mesh files, just something that the engine recognises and will use. We shouldn't try to make this harder than it really is. I've had some feedback from Caliban and this seems to be getting a bit easier.
What I'm planning is:-
1 A .MESH to MS3D converter - this is about 98% complete
2. Load .MESH and export single group as .MS3D - about 65% complete
3. A .MS3D to .MESH converter - A gleam in my eye but shouldn't take too long
Any other thoughts?
Cheers
GrumpyOldMan
Quote:
Originally Posted by GrumpyOldMan
Wow, impressive job! My dream comes true! I can't wait to see you release that unit and the converter, too, so we can all DIY:laugh4:
Nice looking unit there. I tried the Casuir
excision method on feudal_knights_lod0.mesh
and it worked in a custom battle so I'm happy
to see this. Registered now so I've got to
upgrade ASAP to play with the new features.~:)
Hi @KE
Me too, This also means that the engine is strong enough to withstand this sort of punishment. From Casuir's post he also deleted from the 3 (changed to 0) down to the end of the streams and it still worked, so the blip in progression between header/footers was also acceptable.Quote:
Originally Posted by KnightErrant
Cheers
GrumpyOldMan
Didn't catch that the first time. I left
in the 3 0 0 0 converted to 0 0 0 0 but
deleted from the number of vertices to
end of data. That is encouraging that you
can be a little loose in the format and the game
still keeps track of how to get to its data.
Thanks for your work, gents. I follow the gist, but not the code details, and it's still fascinating.
@All...
This is awesome, I am 36 years old and I feel like a little kid in a hobby shop. Having the ability to add and remove weapons and shields??? Awesome.
@GOM,
Will MilkShape allow us to modify the actual mesh, like say if we wanted to modify a helmet to make it look different. Would we be able to do that??
And I assume that if we can switch shields and weapons around, we can also change stuff like the helmet models as well, because some of these are attachments, I think?
Very nice work.
Can't wait for the converters and other stuff to be ready...
Like most everyone else, the code stuff is over my head right now, but the excitement is almost uncontainable...
Also, another note to all...
Would we be able to use MilkShape to modify the Rome Total War models as well. I know they are in a cas format (I think) what extra programming would be required there???
Again, Thanks for all the hard work, you are going to make a lot of Total War diehards very happy...
AT
Yes this is sensible indeed, it will permit new units, and Milkshape is very accessible compared to other 3D packages, I for one am excited!Quote:
As Caliban says the mesh file being used is not any sort of file that will work with a current plugin, etc. We don't have to follow the CA development path to come up with working mesh files, just something that the engine recognises and will use. We shouldn't try to make this harder than it really is. I've had some feedback from Caliban and this seems to be getting a bit easier.
What I'm planning is:-
1 A .MESH to MS3D converter - this is about 98% complete
2. Load .MESH and export single group as .MS3D - about 65% complete
3. A .MS3D to .MESH converter - A gleam in my eye but shouldn't take too long
Any other thoughts?
Cheers
GrumpyOldMan
Great progress all! :)
Well I'm embarassed:embarassed: , I put my excised unit
in my mod folder, not the true /data/unit_models/_units
etc. last night. I put it in the right place and now I'm crashing
the game. I tried the set
3 to 0 and remove from the number of vertices to the
end of mystery data as well as removing the whole entry
including what I think are the headers and footers but no
luck. Crashes with an unspecified error. Casuir, do you know
what I'm doing wrong?
Hi All
I just wanted to post that after I put in a bug report on the Regroup groups function in Milkshape that it has been fixed and reuploaded to the Milkshape site. If you've downloaded Milkshape 1.8 before you may want to redownload to get the fixed version.
@Andromachus Theodoulos
Milkshape is a full mesh manipulation tool so you can modify mesh appearance, vertex/bone assignments, etc. Once this all up and running, the possibilities extend well beyond just switching weapons/shields of existing figures. From what I can see and have understood, as long as your figure is understood by the engine it can be used; so this means heads, bodies, arms, legs, weapons, etc will be able to be swapped or exchanged for new ones. As long as they all fit together and have the right values, they'll work.
I could probably do a .CAS to MS3D converter but that may have to wait for a while, although now I think of it I may do one that extracts an unassigned mesh and the M2TW skeleton (different skeletons used). Note to RTW modders, meshes will work with 100% assignments (just not look as purty) but the meshes will have to be sliced up into chunks as groups. Might be easier to use closest fit M2TW body parts and new textures.
Cheers
GrumpyOldMan
Houston we have a problem............
It's always good to question one's assumptions.
The mystery data is used in the game to control
shading and texturing rendering. Because I couldn't
get the excised file to load I wrote a Python script
to zero out the mystery data but leave the file otherwise
as is. It loads and plays but the graphical results
are disappointing. The unit (hard-coded to feudal_knights,
my favorite test case) flickers in and out of sunlight in
a custom battle and the unit colors also render oddly.
I'm sending an attachment to GOM and Casuir to test
but we need a better answer to what the mystery data
is. I've been working on the .ms3d format and there
doesn't seem to be anywhere to place more data (shader
or whatever this is).
Not to worry, this isn't a showstopper but we need to know
what this is and how to deal with it.
Here's the code if anyone else wants to experiment:
(The getters and putters are from my main script
for converting mesh files, I don't use all of them here.)
Code:import struct
global bytecount
global ms3dbytecount
# -----------------------------------------------------------------------------------
def getbyte( fidin ) :
global bytecount
(thebyte,) = struct.unpack( 'b', fidin.read(1) )
bytecount = bytecount + 1
return thebyte
# -----------------------------------------------------------------------------------
def getubyte( fidin ) :
global bytecount
(thebyte,) = struct.unpack( 'B', fidin.read(1) )
bytecount = bytecount + 1
return thebyte
# -----------------------------------------------------------------------------------
def getshort( fidin ) :
global bytecount
(theshort,) = struct.unpack( 'h', fidin.read(2) )
bytecount = bytecount + 2
return theshort
# -----------------------------------------------------------------------------------
def getushort( fidin ) :
global bytecount
(theshort,) = struct.unpack( 'H', fidin.read(2) )
bytecount = bytecount + 2
return theshort
# -----------------------------------------------------------------------------------
def getint( fidin ) :
global bytecount
(theint,) = struct.unpack( 'i', fidin.read(4) )
bytecount = bytecount + 4
return theint
# -----------------------------------------------------------------------------------
def getuint( fidin ) :
global bytecount
(theint,) = struct.unpack( 'I', fidin.read(4) )
bytecount = bytecount + 4
return theint
# -----------------------------------------------------------------------------------
def getfloat( fidin ) :
global bytecount
(thefloat,) = struct.unpack( 'f', fidin.read(4) )
bytecount = bytecount + 4
return thefloat
# -----------------------------------------------------------------------------------
def putbyte( thebyte, fidout ) :
global ms3dbytecount
fidout.write( struct.pack( 'b', thebyte ) )
ms3dbytecount = ms3dbytecount + 1
return
# -----------------------------------------------------------------------------------
def putubyte( thebyte, fidout ) :
global ms3dbytecount
fidout.write( struct.pack( 'B', thebyte ) )
ms3dbytecount = ms3dbytecount + 1
return
# -----------------------------------------------------------------------------------
def putshort( theshort, fidout ) :
global ms3dbytecount
fidout.write( struct.pack( 'h', theshort ) )
ms3dbytecount = ms3dbytecount + 2
return
# -----------------------------------------------------------------------------------
def putushort( theshort, fidout ) :
global ms3dbytecount
fidout.write( struct.pack( 'H', theshort ) )
ms3dbytecount = ms3dbytecount + 2
return
# -----------------------------------------------------------------------------------
def putint( theint, fidout ) :
global ms3dbytecount
fidout.write( struct.pack( 'i', theint ) )
ms3dbytecount = ms3dbytecount + 4
return
# -----------------------------------------------------------------------------------
def putuint( theint, fidout ) :
global ms3dbytecount
fidout.write( struct.pack( 'I', theint ) )
ms3dbytecount = ms3dbytecount + 4
return
# -----------------------------------------------------------------------------------
def putfloat( thefloat, fidout ) :
global ms3dbytecount
fidout.write( struct.pack( 'f', thefloat ) )
ms3dbytecount = ms3dbytecount + 4
return
# -----------------------------------------------------------------------------------
def putzerobytes( n, fidout ) :
global ms3dbytecount
for ii in range(n) :
putubyte( 0, fidout )
ms3dbytecount = ms3dbytecount + 1
return
# Main.
global bytecount
global ms3dbytecount
bytecount = 0
ms3dbytecount = 0
fnin = 'feudal_knights_lod0_test.mesh'
fnout = 'feudal_knights_lod0_testchop.mesh'
fidin = open( fnin, 'rb' )
fidout = open( fnout, 'wb' )
for ii in range(2*175) :
fidout.write( fidin.read(500) )
fidout.write( fidin.read(477) )
# We're at the mystery blocks.
nvert = getuint( fidin )
putuint( nvert, fidout )
print 'nvert = ' + str( nvert )
for ii in range( nvert ) :
sdummy = getushort( fidin )
sdummy = getushort( fidin )
putushort( 0, fidout )
putushort( 0, fidout )
nheaderfooter = 26
for ii in range( nheaderfooter ) :
thebyte = getubyte( fidin )
putubyte( thebyte, fidout )
# 2nd mystery block.
nvert = getuint( fidin )
putuint( nvert, fidout )
print 'nvert = ' + str( nvert )
for ii in range( nvert ) :
sdummy = getushort( fidin )
sdummy = getushort( fidin )
putushort( 0, fidout )
putushort( 0, fidout )
nheaderfooter = 26
for ii in range( nheaderfooter ) :
thebyte = getubyte( fidin )
putubyte( thebyte, fidout )
# 3rd mystery block.
nvert = getuint( fidin )
putuint( nvert, fidout )
print 'nvert = ' + str( nvert )
for ii in range( nvert ) :
sdummy = getushort( fidin )
sdummy = getushort( fidin )
putushort( 0, fidout )
putushort( 0, fidout )
#fidin.seek( 229468, 0 )
fidout.write( fidin.read(500) )
fidout.write( fidin.read(600) )
fidin.close()
fidout.close()
Mkay, looks like I messed up here and tested the base figure file instead of the actual ingame one :embarassed: Guess I should have been a bit more thoroug. Did some more testing and it looks like these values are necessary, changing the 3 to 0 crashs the game as does removing any of the values. Changing all the values to 00 in the first stream gave this result:
https://img100.imageshack.us/img100/2325/image1ut0.jpg
As you can see it has some effect on the lighting, unfortunatly my graphics card is on the way out and doesnt like the battle mode so I cant get a proper look at whats happening, can somebody else have a look at these and see whats happening, preferably someone whos machine can handle all the high-end bells and whistles. Arrow warriors have the first block zeroed out, eagle the second and cuahchiqueh the third.
http://www.wikiupload.com/download_page.php?id=99774
Scrrenshots would be nice
Looks like ya bet me to it KE
Hi
@KE and Casuir
KE, I tried your example but I couldn't get much difference from the stock figure, I tried it with different combinations of shader system, shadow quality and anything else I could tinker with in the Options menu. Could this be card related - I'm using a 9800pro.
Casuir, I tried the download link you sent but I just got caught in a loop with not much being downloaded.
Thoughts - KE, I'm not familiar with Python, and seeing you've already got it set up :beam: could you try your script with the non zero values being replaced by 127 or 255 or random 0-255?
KE, with your permission I'd like to expand the test on the modified mesh.
Would people like to try out the feudal knight mesh to see what happens on their system? I've just tried it about twenty times on my system and there have been no crashes or other strange behaviour. If KE is happy I can post a download link.
Cheers
GrumpyOldMan
Try this one, the first one wont let me dl either
http://www.megaupload.com/?d=O8YR7OVW
Managed to get a bit more messing done and changed the values around a bit
https://img98.imageshack.us/img98/899/blueoe9.th.jpg
https://img101.imageshack.us/img101/7858/reded1.th.jpg
first one is 00 00 ff 00, second one is 00 ff 00 00
Thought they might have been rgb values at first but colours change depending on the weather, the lighting effect on the models there seems to be produced from the odd cubemaps that are in globallighting. Definitly looks like one of the values looks to calculate that environment mapping type effect is messed up. Changing the second and third blocks doesnt seem to have any effect that I've noticed, but I seem to be missing a bit lately.
I've got a 9600gt so you should be able to reproduce anything I can, are you sure you have the game set up to use modded files? I find using --io.file_first in a bat file works better than a cfg file.