Animations

[GrumpyOldMan]

Hi KE et al

EDIT: These are the shields held up in-game.

NOTE This ain't right, everybody has become left-handed. Not that I mind, I'm
left-handed myself.

Ahh, you're a firm but cruel drill sergeant, making your men dislocate their shoulders just so they reverse their arms and pretend to be The Queens Own Left Handed Regiment - specialists in guarding the left flank!! . My guess is that you've misplaced the axes conversion x*-1 somewhere.

Unresolved Problems:

(1) Does no_animdb=1 work? PM into Caliban hasn't been answered.
Can't unpack skeleton.dat into the /animations directory because OS constraints don't allow a file without an extension to have the same
name as a directory. (Repeating xziang1983 here, been down this road.)

Nope, been playing with this, no_animdb is not available in the retail version so we'll have to pack. With the skel name and folder conflicts, my thoughts are that the skeleton names are just handles so if we modify them by the additon of "_skl" and consistently use this across all the relevant files (modeldb,etc) it should take away this bugbear

(2) Unpack in a higher directory? Maybe, haven't tried this.

(3) Unpack in a lower directory? Same answer, try later.

The retail game seems to just run off the idx/dat files, unless the file name and folder conflicts are causing the game to crash

(4) Repacking the pack isn't that bad. Live with it.

Agreed, it's not too much of a price to pay

(5) Maybe the TYPE files have an implied extension we just don't know
about. This would be nice but we need information.

If you have a look at the calls to the skeletons in modeldb just the normal name is used, if they were it would have to be hardcoded but it doesn't help us with name and folder conflicts.

(6) What the heck is the global data? I just saved it in the materials comment
section but need to know what it really is. Also the mystery 8 floats at
the end, also written into the materials comment section.

Once I lash myself into a frenzy of documentation completion (hardest part for me) for the RTW CAS to M2TW converter, I'll go back and have another look at the animation files and tie down the loose ends.

(7) Outside the bounds, but the TYPE files (using the language of
descr_skeleton.txt) have bone info in them, can these be modded
to change the sizes of units re Bwian's question in the mesh thread?

I'm pretty sure that we can modify skeletons adding, moving and rescaling bones as long as we remember tha the engine is looking for various flags for animation ie bone_Rhand, saddle, etc

Cheers

GrumpyOldMan

[KnightErrant]

Yep, it was a missed x -> -x in the animextract that did the
parity inversion on my guys. I was so careful to take these
OUT of the animmerge. I'll do some surveying on the cas's
to see how the globals and final floats vary. It is an addiction,
good times!

Edit: Read up a little on animating, better example of raised shields
with the MTW2_Spear_walk.cas.

[GrumpyOldMan]

Hi KE

Yep, it was a missed x -> -x in the animextract that did the
parity inversion on my guys. I was so careful to take these
OUT of the animmerge. I'll do some surveying on the cas's
to see how the globals and final floats vary. It is an addiction,
good times!

Edit: Read up a little on animating, better example of raised shields
with the MTW2_Spear_walk.cas.

Aw shucks, I kinda miss those sinister left handers

I've gone back and revisited the anim files and this is what I've dug out so far

Cas Anim format

Short - Num_Frame

short - Num_bone

num_frame * num_bone * float*4 (Quaternion rotations)

num_frame * num_bone * float*3 (local positon data - Pelvis in relation to Root Node ie height off ground, others unchanged from rest positions)

num_frame * float*2 (x and incremental z movement for Root Node)

2 * float (0?)

num_frame * float (approximate absolute z position of Root Node but in reverse order compared to animation ??)

num_frame * float *3 (absolute position of root node and pelvis height)

I'll have a look at the rest later.

Cheers

GrumpyOldMan

[KnightErrant]

Aargh, of course, missed that completely. I guess I don't
think in hex after all. So the 14 00 14 in the header after the initial short
count of frames is the bone count for what gets animated.
The variants I saw must have been strat map units which don't
have 20 animated bones... Got a better clue for looking at things
now, many thanks.

[Ashdnazg]

I couldn't see if this was seen earlier, so I post this anyway.
I had a wee look in Verc's script, and the second short in the header is the number of bones.
About the fifth byte, I think he actually skipped it.

Aargh, of course, missed that completely. I guess I don't
think in hex after all. So the 14 00 14 in the header after the initial short
count of frames is the bone count for what gets animated.
The variants I saw must have been strat map units which don't
have 20 animated bones... Got a better clue for looking at things
now, many thanks.

Thank you for ignoring my posts :D

[KnightErrant]

@Ashdnazg
You're absolutely right, I apologize. Went back and looked at
your post and there's the information staring me in the face.
I was so focused on fixing my script then my brain just wasn't
accepting new inputs. Please post any info you have; I promise
I'll pay better attention next time.

@GrumpyOldMan
Ok, got a reply from Caliban. He did the two experiments and none
of those "type files" or "skel files" exist with his unpacked setup.
So I think skeleton.dat/idx is supposed to stay packed and those
names are just look-up handles like you said. Looking at events.dat/idx
right now since Caliban had them unpacked. There's no names in
the index file so I think events and pack have to be unpacked together
since the .evt always matches the .cas path and filename. However,
the mapping is onto but not 1-1, not every cas file has a corresponding
evt file. If I'm seeing the header right, there's 1000 evt files for 2400
cas files.

[Casuir]

evt files look to be sound files so logically there wouldnt be one for every animation, a lot of the entrys in the skeleton.dat look to be just unit_march

[KnightErrant]

I think you're right, the data in events.dat seems to be file names for .wav
and .mp3 files in the sounds directory (although really in sounds.dat).
Looking at descr_skeleton.txt which seems to have much of the real
data in skeletons.dat shows the _basepose.cas, a lot of the _idle .cas's,
and the weapons entries with _Primary don't have evt associations.
I guess weapons and soldiers not doing anything don't need to make sounds.

[Casuir]

Aye and a lot of them would make the same sounds, looking at the engine evts a few of them are blank, if the game requires the evt files to run then theoretically you could get away with copying these and renaming to the required names. I do think they're packed in the skeleton.dat file sans headers, theres definitely stuff in there that matchs up with bits in the events.dat

[Casuir]

Mkay, just been reading some of the old threads on rtw animations problem and hoping someone can explain this a bit better to me, .evt files are missing and the game wont run with the animations extracted because of this? Looking at this post by Jerome it seems the evt files are in the skeleton.dat and this has been known for some time. Vercs extractor doesnt extract them though so I'm wondering if this is an oversight or theres some other problem.
https://forums.totalwar.org/vb/showp...0&postcount=13

Heres the relevant part of skeleton.dat

First byte in yellow is the number of entrys, next byte in purple is the sound catagory, I've identified 01 for SOUND, 02 for SOUND_BANK, 04 for SOUND_VOICE, 05 for SOUND_AMBIENT and 03 must be shockwave, doesnt seem to be any of those types in it though. Next bit in green are the start and end frames in green, followed by the lookup tag. Theres an 00 01 after some entries here which isnt always present, Im guessing this is used to identify the sound as random? Theres also a looping tag in some of the evt files in the effect_evt folder

[KnightErrant]

@Casuir
I've unpacked the cas's in pack.dat and the game won't run
with these alone using the no_animdb = true switch that Caliban
told us about. The jpg of his unpacked /animations directory showed
evt files present so I'm wondering if this is required but I don't know
this. We know that skeleton.dat ISN'T unpacked on his system because
he didn't find any of the type names present for any file extension.

Not really a big problem because its easy to repack the cas's after
modifying them, it's more of a "if it can be done then try doing it" thing.
GrumpyOldMan suspects that the no_animdb switch may not be present
in the retail version and that could very likely be the case.

Missed your post re strat_navy and fs_camel. These are the handles
or type names for different groups of animations. They extract out of
skeleton.dat just like the cas's come out of the pack but they don't
seemed to be used that way. That's why I asked for the experiment
to be done, to see if they were really filenames or just handles like
GrumpyOldMan thought. (I picked those two because the corresponding
cas's aren't named that way, so we could avoid a bunch of false hits
to have to sort through.)

Thanks for digging up the old post by Jerome, there's some food for
thought there. I'll to dig again with this new info.

[Casuir]

The problem here is vercs script doesnt properly extract from skeleton.dat, it just does a binary dump of the different entries taking the name from skeleton.idx. The actual file itself is compiled by the game from descr_skeleton, if it was extracting correctly it should be outputting that and the evt files at the very least. I think GOM is right about the type files being handles, I dont see anything suggesting they are files, am I missing something here? As for Calibans jpg, if hes got the evt's extracted then he's definitely got the skeleton.dat unpacked, becase jerome clearly says they're in there and if we look, we can find data matching their structure.

[GrumpyOldMan]

Hi KE

@GrumpyOldMan
Ok, got a reply from Caliban. He did the two experiments and none
of those "type files" or "skel files" exist with his unpacked setup.
So I think skeleton.dat/idx is supposed to stay packed and those
names are just look-up handles like you said. Looking at events.dat/idx
right now since Caliban had them unpacked. There's no names in
the index file so I think events and pack have to be unpacked together
since the .evt always matches the .cas path and filename. However,
the mapping is onto but not 1-1, not every cas file has a corresponding
evt file. If I'm seeing the header right, there's 1000 evt files for 2400
cas files.

What we really need to see are a few examples of unpacked .evt files. The .evt files look like this in the skel .dat files :-

short - num of variations
short - 0
int - code for 'sound' type??
short - start frame
short - end frame
0 termed string - description/look up table entry??(some different cas use same event - explains number discrepancy)
then usually 10 bytes with 32 00 sometimes appearing and sometimes the start and end frame reappearing.

If we could get a hold of the evt files for one figure we could make a fist of decoding because the 'no_animdb' is sure not working without them. I'll ask Caliban and see if he can help.

I believe now that the two idx and dat files are made up from the folders within animation and that the skeletons we've been so intent on don't really exist. This is bad news for the fantasy people because this means that the skeleton positions are hardcoded and are added in at compiling those idx and dat files.

Cheers

GrumpyOldMan

[SigniferOne]

Guys, Jerome provided all of the .evt files for RTW 1.5, but putting them into the extracted animations folder, so they'd match up with their corresponding animation files, still didn't work, and still crashed. So, Caliban will really have to babywalk us through here, because Jerome spent a lot of time trying to get it happen for us in RTW, and it didn't work, in large part because I think the TW engine is not so successful at loading animations when extracted. It works wonderfully for other files, which don't have to be in packs, but seems to still be a buggy process with animations, and they have to be packed, unless Caliban's animdb switch was truly made to work this time.

Just briefly, if animations have to be repacked, does this mean descr_skeleton.txt is still pretty irrelevant as a file? The one inestimable advantage of getting unpacked anims working is that descr_skeleton.txt would become important, and then we could do a lot of stuff with animations that couldn't be done otherwise.

[GrumpyOldMan]

Hi SigniferOne

Guys, Jerome provided all of the .evt files for RTW 1.5, but putting them into the extracted animations folder, so they'd match up with their corresponding animation files, still didn't work, and still crashed. So, Caliban will really have to babywalk us through here, because Jerome spent a lot of time trying to get it happen for us in RTW, and it didn't work, in large part because I think the TW engine is not so successful at loading animations when extracted. It works wonderfully for other files, which don't have to be in packs, but seems to still be a buggy process with animations, and they have to be packed, unless Caliban's animdb switch was truly made to work this time.

From what Caliban is saying it seems that if no_animdb switch works (and we won't know that until we have the .evt files in place) the skeleton and pack idx and dat files are rebuilt on start from the descr_skeleton.txt file, and then the engine uses those repacked files.

Just briefly, if animations have to be repacked, does this mean descr_skeleton.txt is still pretty irrelevant as a file? The one inestimable advantage of getting unpacked anims working is that descr_skeleton.txt would become important, and then we could do a lot of stuff with animations that couldn't be done otherwise.

I don't think from what Caliban says that we'll be able to work with unpacked files, we'll just use them to create new idx and dat files. It was possible in RTW to get new skeletons and animations in and I think that eventually we'll be able to do the same, but in the meantime its fun to play with it

Cheers

GrumpyOldMan

[Casuir]

Guys, Jerome provided all of the .evt files for RTW 1.5, but putting them into the extracted animations folder, so they'd match up with their corresponding animation files, still didn't work, and still crashed. So, Caliban will really have to babywalk us through here, because Jerome spent a lot of time trying to get it happen for us in RTW, and it didn't work, in large part because I think the TW engine is not so successful at loading animations when extracted. It works wonderfully for other files, which don't have to be in packs, but seems to still be a buggy process with animations, and they have to be packed, unless Caliban's animdb switch was truly made to work this time.

If you re-read what the man said at the time the problem actually lies with vercs .cas extractor. He got it working perfectly with the original files and a retail version of the game, so the problems on our end, not theirs. Verc missed the ball on extracting the evt files from the dat, so its not implausible that something else is getting borked.
https://forums.totalwar.org/vb/showt...ighlight=bones

@gom, theres evt files in the animations/engines folder, not the exact same as the ones missing but you should be able to get the header and file structure from them.

This thread might be of interest as regards the skeletons
https://forums.totalwar.org/vb/showthread.php?t=53763
Be nice to if caliban could confirm that the skeletons are hardcoded, how does the engine know which one to use for a camel and for a man?

[SigniferOne]

Hi SigniferOne



From what Caliban is saying it seems that if no_animdb switch works (and we won't know that until we have the .evt files in place) the skeleton and pack idx and dat files are rebuilt on start from the descr_skeleton.txt file, and then the engine uses those repacked files.



I don't think from what Caliban says that we'll be able to work with unpacked files, we'll just use them to create new idx and dat files. It was possible in RTW to get new skeletons and animations in and I think that eventually we'll be able to do the same, but in the meantime its fun to play with it

Cheers

GrumpyOldMan

Ah right, if the game recreates the packed files is fine, as long as it reads from descr_skeleton.txt and lays my fears to rest :P

[Casuir]

Looking at verc's extracted .cas's they seem to differ a bit from the engine animations and some cas's in unit_models/weapon_testing which look to be animation files. I think whats happening is the .dat has compressed cas's, verc's extractor is extracting them in a format readable by his max script but not by the game, which causes it to crap out. Any chance Caliban could provide an animation thats actually in the pack?

[GrumpyOldMan]

Hi Casuir

Looking at verc's extracted .cas's they seem to differ a bit from the engine animations and some cas's in unit_models/weapon_testing which look to be animation files. I think whats happening is the .dat has compressed cas's, verc's extractor is extracting them in a format readable by his max script but not by the game, which causes it to crap out. Any chance Caliban could provide an animation thats actually in the pack?

The animations are fine, just in a bit of a different format from RTW ( which is why they're apparently appearing a bit wonky in Max - I think the Maxscript is hardwired to the RTW skelton). I can already convert them over to Mikshape format, etc. The main issue is to get the translations from the binary to the text .evt files so we can see if the engine rebuilds the skel and pack idx and dat files or whether we have to repack them ourselves.

Cheers

GrumpyOldMan

[Casuir]

Hmm, well, it looked to me like all the other .cas have bone info at the top of the file. I thought maybe the engine was discarding it or storing it elsewhere when it packed the files. You know more about this than I do so if you say they're ok then I guess they must be.

[GrumpyOldMan]

Hi Casuir

Hmm, well, it looked to me like all the other .cas have bone info at the top of the file. I thought maybe the engine was discarding it or storing it elsewhere when it packed the files. You know more about this than I do so if you say they're ok then I guess they must be.

The change in format that I spoke about was between RTW and M2TW, where the bone info has changed and some of the other information has changed within the file. The main reason that the animations looks wonky is that (I understand) you can only load the animation on top of a figure and since you can only load a RTW figure when you load the anim rotations they go to different bones ie change of skeleton between the programs.

Cheers

GrumpyOldMan

[Casuir]

I'm not talking about differences in format between the games I mean differences in formats between verc's extracted .cas's and CA's original ones. What looks to me is happening here is verc's extractor is taking a binary dump from the pack and saving it as a .cas which his script can read and his tool can repack. The actual game itself when it tries to recompile the files parses the descr_skeleton and writes the idxs and dats. If we look at the skeleton.dat the first thing in there is bones, presumably the basepose? It has to be getting these from somewhere and the .cas's left unpacked all have them at the start of the file. Verc's dont so when the game tries to run without the dats or repack them it craps put. I think the exporter CA uses exports from max with bones in the file then the game discards them on packing as it already has the skeleton in skeletons.dat.

Dunno if this confirms or debunks this

On a side note, the skeleton base pose (Vercinge's bone positions) is normally taken from the first animation which is added to a skeleton, so it's not necessary to supply this information seperately. It is important to make sure that the transforms in an animation are all relative to the same basepose - we did this by making sure that in Max the animations all contain a single frame at the front of the animation which contains the model basepose.

[GrumpyOldMan]

Hi Casuir

I'm not talking about differences in format between the games I mean differences in formats between verc's extracted .cas's and CA's original ones. What looks to me is happening here is verc's extractor is taking a binary dump from the pack and saving it as a .cas which his script can read and his tool can repack. The actual game itself when it tries to recompile the files parses the descr_skeleton and writes the idxs and dats. If we look at the skeleton.dat the first thing in there is bones, presumably the basepose? It has to be getting these from somewhere and the .cas's left unpacked all have them at the start of the file.

Agreed but if you have a look at the basepose cas files they are basically empty data, all quat values are 0,0,0,1 and all bone positions are 0,0,0, the values at the start of the unextracted cas files are the quat rotations. What's really telling now is that the cas files in m2tw don't have the bone names included, so the idx and dat files must be getting them from somewhere, the question is where? I've written to Caliban along these lines and we'll see if we can get any answers. I've seen some values right at the end of the basepose, but there are figures that have no basepose cas and use a normal cas for their default pose . At the moment, I could extract bone positions from a normal cas file but there is nothing I've seen that includes bone names or flags that indicate which skeleton should be used. If the no_animdb works and creates a new skeleton idx and dat then that information must be stored somewhere, hopefully somewhere accessible.

Verc's dont so when the game tries to run without the dats or repack them it craps put. I think the exporter CA uses exports from max with bones in the file then the game discards them on packing as it already has the skeleton in skeletons.dat.

Dunno if this confirms or debunks this

We have to find out where the bone names and positions are being stored, or which flags are used to notify the system to use a paticular skeleton.

Cheers

GrumpyOldMan

[Casuir]

So its not in skeletons.dat then?

Edit: Converted some of the floats in there by hand and it looks to me like the bone positions and heirarchy are there. I'm 100% certain on whats happening GOM, what the extractor is taking out of the pack.dat arent actual cas files, look at any of the unpacked cas files and you'll see they all have bones at the start of the file, strat models, engine animations, missiles. Stands to reason that anything else exported would have the same format. If you look at it the way the game does it makes sense, the game gets the animation name from modeldb, checks that against the skeleton.idx, gets the bone positions, animations and evt files from the dat, then checks the pack index for the animation positions in the dat file and applies the bone data from skeleton.dat to the anims. No need for excess info in the pack.dat, whole point of packing them up is to save space or optimise the process, either way theres no need for them. Quickest way to confirm this would be for one of the CA guys to open up an unpacked animation in notepad or similar and see if theres bone info in the file, so if any of you guys are reading this :)

[KnightErrant]

@Casuir
I think I see what you're saying. If we just unpack the cas files
blindly from pack.dat we just get animation floats. But looking at
siege engines they have their bone sections up front. So should we
be making cas files that are a combination of the bone data from
skeleton.dat and the anim data from pack.dat?

The .evt files that Caliban posted do look like ASCII RTW ones so they should
be easy to extract from the unpacked skeleton files. I'll take a stab
at it at lunch today.

[Bwian]

This is starting to get interesting, nad hopefully will bear some fruit! Now, I am no expert when it comes to hex reading, but I do know a reasonable amount about actually trying to use the old tools..... and they were clearly not extracting data in a reliable fashion. Casuir's explanation would certainly go some way to explaining it.

From what I recall, the process that we were missing before was centred around the skeleton.dat.

We could edit this through a batch file which seemed to be able to change certain human readable parts of the DAT file to change animations. We were not...in actual fact...changing things properly. There was a clear problem actually creating a new skeleton with revised bone positions. We could scale and rename, but no more than that. I asked a lot of questions on this front, but never actually got a clear answer. The CAS file had a skeleton in it, but this was just a reference point for animation. The DAT file had the starting positions in it, but we couldn't re-write them. Never worked out why! I don't hink Vercingetorix ever did either!

The other issue I ran across so many times was the limit to the extreme rotation of parts without corrupting the file. If, as I was lead to believe, the animation files were a series of rotations applied ot the root positions held in the DAT file, there should not be a problem with the amount of rotation. In Metal Mayhem, I bent things REALLY out of shape...so I know how easy it was to break things! Casuir's assertion that we were not playing with real CAS files was again a point worth considering. We have no way of knowing for sure, since the whole pipeline was geared to getting something out of the IDX file that Max could read in to re-position the skeleton he had built up in his max importer. Working from a clean sheet of paper ( as we are for this project ) give us the chance to put aside what went before, and to think it through from first principles.

We need to understand the Skeleton.dat file first, since it holds the key to the starting points. When we can re-locate these points and build a base skeleton, we can then look to work out how the IDX files are changing the thing around. Personally, I prefer them packed...even if we could run with them out of a pack... since it is just a lot neater!

I just wish I could actually understand a bit more of the technical stuff, but I was not designed to think in hexadecimal!

[Bwian]

Anyway..after pausing to rest my fingers...I have a question

please slap me if I am wrrong... but the structure of the skeleton.dat file seems to go like this:

a definition of the starting skeleton
a load of animation links for each of the animation sets stored in the IDX file
a definition of the next skeleton
a load of animations for that

and so on.

Can we change the starting points of the bone positions ( which must be in the bits of data between the bone names in the DAT file ) and actually see the change in game ?

I am assuming that the rotational changes form the IDX file will still be applied ...just to a new starting position...

I have no idea how the various hex numbers actually translate into real co-ordinates in a modelling program...but then I am just a humble polygon pusher

[KnightErrant]

Some success and frustration, bleepin' granny strings again.

I've managed to extract about 15 evt files from the subskeleton
MTW2_Spear. But in the 42 bytes following the null terminated
cas file name is a control int at the sixth int. It seems to control
whether a unit cas entry is terminated "normally" by int 0, short x32
or whether there are more 0 bytes after x32 x00. For instance,
x16=22 has 3 extra zero bytes, x0D=13 has 7 extra zero bytes,
x13=19 has 4 extra zero bytes,
x10=16 has 34 extra zero bytes, and it goes on. I'm just crunching
through and adding elif statements to handle all the different granny
control bytes until I get to end of this subskeleton. Then I'll see
if I can process all the other MTW2_ subskeletons. After that
are the stratmap ones and the fs_ mounts. Fortunately, the
fs_test_ stuff and the _pole entries don't seem to have evt
files.

[KnightErrant]

Well, scratch that controlcode stuff, found counter examples.
Now just gobble zero bytes after the x32 x00 short and then push
back the last byte when a non-zero one is encountered. Managed
to process 6 subskeleton files before bombing, about 243 evt files.
Getting there slowly.

[GrumpyOldMan]

Hi KE et al

@@@@@KE

I've just done a reinstall and update to 1.1 and now my m2tw falls over just as we get into a battle. I had a few cleansing ales last night (nothing to do with why my m2tw won't work incidentally) and woke up at 4.30 this morning with an aha!!! - eureka!!! moment. The cas files haven't drastically changed their format!!, it's just the way they're storing them in the dat files. Instead of each cas file in the dat file having it's own bone names and position information, this is stripped out and stored once only in the skeleton dat file and the rest of the files is then stored in the pack.dat. Edit:- This explains why the basepose files are just empty shells when they're extracted from the .dat files, the bone name and positions have already beem stripped out.

Since I can't get my m2tw to run any more how'd you like to volunteer as an intrepid test pilot. If yes, hide your real skeletons.dat away somewhere safe and then open a copy in hex XVI32 and go to Menu/Search/Replace and replace all instance of 'A4 8F 59 3E' with '00 00 80 3F' and save in the Animations folder. If I'm right this will move all the bone_abs bones from ~.2 to 1 above the bone_pelvis.

Edit2 :-

Don't listen to me, especially if I've had a few cleansing ales the night before. I was just able to get the program to run and then I realised what was wrong with my experiment. Even though the new skeleton would be stored and loaded, the .cas files also store the global final positons of the bones which is what would be used to calculate the vertex vector/rotation matrix positions, so even though the new skeleton is there you won't see anything until we get a whole new series of cas animations.

Cheers

GrumpyOldMan