Settlements: File Formats; Take 2

Quote:

---

Originally Posted by Redmeth

I have no expertise in this area but this seems like a very complex problem that could be easily solved by CA if they were willing.

Does anyone fancy the idea of a petition by modders addressed to CA with the objective of allowing the building in M2TW and Kingdoms to become available for modding?

I am not volunteering to lead this idea but a lot of people that are the best modders of the community seem to be interested in this and allowing the buildings to be moddable would show CA's goodwill towards the community and encourage modders to do even more extraordinary things.

---

Believe me, a petition would have the same effect as swatting a locust in a swarm of two billion.

CA management isn't prone to show any goodwill to modders or even consider the idea. What help we got was because certain individuals (mainly Caliban and Palamedes, with the help of a few programmers) really tried to push things in our favor - but even with this, very little came out of it.

It simply looks like the bigwigs at CA and SEGA aren't interested in modders at all. They tolerate us, but they don't try to help us. If anything, they at least didn't try to stop their subordinates helping us, either.

Hello,

From what I read, this doesn't look to good, but can custom buildings and cities from RTW be brought into M2TW?

Thanks,
RM3

not at the moment, they're totally different formats.

I thought they were both Cas files. So there is no way of editing battle map buildings in M2TW yet?

I think I concur with Alpaca, as much as Caliban has helped we are
probably on our own as regards settlements. Sorry, haven't been
monitoring this thread as I haven't done much. Here's where I've
gotten to: can now read large_towns and it just gets worse. More
data after the complexes/structures, the deployment outline is much
larger etc. I've also been leaning towards the realization that Milkshape
just doesn't have the data structures needed to hold the extra vertex
information that is in the complexes/structures (these are the two
"mystery blocks" that Caliban said were lighting and animations, I forget
the integer codes but they are in an e-mail somewhere).

So I'm stepping back and looking at 3dsmax as the only viable container
for this. I don't own a copy of 3dsmax but Bwian has pointed me to gmax
to learn MAXScript. I've slacked off on settlements to try and support
animations and try and get models into 3dsmax via MAXScript.
If I can just learn how to do a simpler problem then maybe settlements
won't seem so insurmountable. (At least settlements don't have bones,
my current headache in gmax.) However, to me it looks like the best
outcome would be: we should be able to modify an existing structure
(meaning a set of buildings, i.e. change the meshes to get a different
architecture, change textures, etc.) but adding new ones probably
involves changes to all the data the follows the complex/structures
section that I just don't see how it could be done without some magic
CA tutorial. (Not even possible I would think, that data is probably all
granny driven anyway.)

Real life problems, the tensioning spring on my lawn mower broke tonight
so tomorrow evening is a hunt around for repair parts. Check back on
Friday.

KE, I'm concerned about the fact that you're trying to import a whole settlement into Milkshape, and failing to do that given the constraints, find yourself necessary to go to another 3d platform. Remember my suggestion, that milkshape be only used as a kind of 3d reference tool, to know where to place buildings. If I gave you a set of coordinates and a bunch of building models, would you be able to produce a .model file from that? That's all that's really necessary here.

Also why do you trouble yourself with converting all of the scripts to 3dmax form?? If for some reason we NEED 3dmax to do settlement editing, then let it be used only for settlement editing. I would be loathe to think that you wanted to do so much for the community and then got discouraged and left it half way... The modeling and animation scripts work (almost) perfectly (see my post in anims thread), so just merely having a way to do buildings, in 3dmax or otherwise, is the last piece of the puzzle. Because with Vercingetorix script we can already import RTW buildings into 3dmax, and thus being able to insert them into Medieval 2 would be just a dream.

If only we had access to CA's blocksets. This would make it much easier to figure out which part of the file is which.

@KE: I'm currently working on a MAXscript importer/exporter for mesh files. Unfortunately I can only work on it in my vacations from the army so progress is rather slow
If you want my WIP script, or any assistance, PM me and I might be able to help.

@ashdnazg
Actually I'd love some help. I've got bones now using point nodes
and skinops.addbone to add them into the mesh. I've got named
selection sets which seem to be like Milkshape's groups. The problem
right now is I can't do vertex weighting because the skin modifier
I associated with the mesh says it has no vertices even though the
mesh does. Here's the code snippet:

Code:

---

                theskin              = skin()
                theskin.rigid_vertices = false
                select themesh
                max modify mode
                addModifier themesh theskin
                for j = 1 to nbones do
                  (
                  skinOps.addbone theskin (getNodeByName pnodes[j].name) 1
                  )
                nb                  = skinops.getNumberBones theskin
                print( "Number of bones in the skin = " + nb as string )

                nv                  = skinops.getNumberVertices theskin
                print( "Number of vertices in the skin = " + nv as string )

---

I get 26 for the number of bones so that's working but 0 for the number
of vertices. The documentation says it should return the number of its
associated mesh. When I loop over the number of vertices and try
to do

skinOps.SetVertexWeights theskin i boneId_array[i] primaryweight_array[i]

I naturally get an error. This is where I'm stuck.

@SigniferOne
Oh, I agree, no need to redo the Python animation utilities. You can
anim merge and then import the new .ms3d file into 3dsmax so no need
for that one. Would need an exporter though for any animations made
in 3dsmax but that shouldn't be hard (famous last words). The other
export skeleton functions in Python should be fine as is.

Found the magic incantation:

Code:

---

      selbitall            = #{1..themesh.numverts}
      themesh.selectedVerts = selbitall
      skinOps.resetSelectedVerts theskin

---

This forces the skin to update internally and now it
has vertices and the weighting part doesn't die.

N.B. I first tried

Code:

---

skinOps.Invalidate theskin 1

---

because the documentation said it would have the same
effect but this killed gmax dead as a doornail.

I used

Code:

---

classOf themesh

---

to solve this, updates the stuff as well.

but as the saying goes "if it works, don't touch it!"

To moderators: Even though the question seems unrelated to the
thread please leave, it does have relevance.

This question is posed to two groups: those who play with MAXScript
and to modellers. It is the same question posed two ways lets start with
the MAXScript guys. I want to take the group indices that I get out of
an .ms3d file's third section and make a "group". I put it in parentheses
because a group has a somewhat different meaning in GMAX (my cheapo
surrogate for 3dmax). I've tried several MAXscript face selection methods
to make a named "selection group" that I can save in the global
selectionSets array but nothing seems to survive but the name. I don't
get an actual face selection. Any ideas?

Posing the same question to modellers who have experience with 3dsmax.
If you were going to select a set of faces and then "group" them and
give them a name so you could manipulate them later, perhaps by adding
more face to them, how would you do this. Please include excruciating
detail because I want to turn on macro recording and see what commands
your mouse clicks correspond to.

This question is aimed at getting a 3dsmax importer/exporter working but the
ultimate goal is to bring in settlement data that can support lighting arrays
(haven't looked at yet) and animation arrays (don't know what Caliban was
talking about, my books on MAXScript don't cover this.) Any info much
appreciated. :coffeenews:

Well, from a purely modeling perspective (MAXScript is definitely not my cup of tea) and coming from a pure amateur perspective, let me try to help:

Quote:

---

If you were going to select a set of faces and then "group" them and
give them a name so you could manipulate them later, perhaps by adding
more face to them, how would you do this. Please include excruciating
detail because I want to turn on macro recording and see what commands
your mouse clicks correspond to.

---

In MAX, and as far as I know, any set of faces that is selected can only be grouped and named if it stands as his own object outside of the main mesh. The main mesh itself has a specific name. So, trying to remove any ambiguity, if you can name it in MAX, it's a separate object. If you want to make it part of the main mesh it will loose it's name and it's group properties and will be no more than a part of the main mesh.

Now do you want a point-by-point explanation of the commands used?

@Aymar de Bois Mauri,

Thank you, I think you have succinctly explained why what I was trying
to do was so hard: it is, in fact, impossible. In a mesh file you have one
large vertex array but (take armored_sergeants for example) 22 face groups
which index into that one vertex array. In Milkshape you have the same
large single vertex array, a single large face array, and then an auxilliary
set of 22 "groups" that index into the large face array to get the grouping
structure. What I'm hearing is that in Max I need 22 little vertex arrays
accompanied by 22 little face arrays indexing into their separate dedicated
vertex arrays to form the same groups. In other words 22 separate named
meshes.

Yikes, this is going to be an index housekeeping nightmare. Importing seems
straightforward because I know how to grab vertices by index and simply put
them into the 22 new little arrays, perhaps with duplication if the face
groups share a vertex. Exporting is the nightmare. I have to go through each
group in order, pull out the vertices and put into a single large array. Reindex
the faces to point to the same vertex which now has a different index
in the large array, AND remove duplicate entries, all the while keeping the
indexing straight. :dizzy2: I'm going to crawl back under my rock and think
about this for a while.

Ok, back from under my rock. This is just a gmax hack for proof of
concept but if I do like Aymar said and make 22 editable meshes
I do get groups:

https://img152.imageshack.us/img152/...maxhackrt2.jpg
By knighterrant at 2007-10-17


Here I just blew them apart to see them. This isn't the right thing
because I used the full vertex array for each mesh. Not to mention
just how long it would take to do the bone assignments and weighting
over 22 vertex replicas. It takes 2 minutes to do the mesh as a single
entity so over a half an hour this way. Back under the rock to do some
more thinking. :book:

Quote:

---

Originally Posted by KnightErrant

@Aymar de Bois Mauri,

Thank you, I think you have succinctly explained why what I was trying
to do was so hard: it is, in fact, impossible. In a mesh file you have one
large vertex array but (take armored_sergeants for example) 22 face groups
which index into that one vertex array. In Milkshape you have the same
large single vertex array, a single large face array, and then an auxilliary
set of 22 "groups" that index into the large face array to get the grouping
structure. What I'm hearing is that in Max I need 22 little vertex arrays
accompanied by 22 little face arrays indexing into their separate dedicated
vertex arrays to form the same groups. In other words 22 separate named
meshes.

Yikes, this is going to be an index housekeeping nightmare. Importing seems
straightforward because I know how to grab vertices by index and simply put
them into the 22 new little arrays, perhaps with duplication if the face
groups share a vertex. Exporting is the nightmare. I have to go through each
group in order, pull out the vertices and put into a single large array. Reindex
the faces to point to the same vertex which now has a different index
in the large array, AND remove duplicate entries, all the while keeping the
indexing straight. :dizzy2: I'm going to crawl back under my rock and think
about this for a while.

---

Well, that is disturbing to know. An frankly quite an undertaking to cope with. :sad:

I'll try to get help from Salinoc, a new EB member who is a professional 3D artist. Mind though that he mainly works in Maya, so he might not be able to explain all of the functioning of the MAX engine.

Quote:

---

Originally Posted by KnightErrant

Ok, back from under my rock. This is just a gmax hack for proof of
concept but if I do like Aymar said and make 22 editable meshes
I do get groups:

Here I just blew them apart to see them. This isn't the right thing
because I used the full vertex array for each mesh. Not to mention
just how long it would take to do the bone assignments and weighting
over 22 vertex replicas. It takes 2 minutes to do the mesh as a single
entity so over a half an hour this way. Back under the rock to do some
more thinking. :book:

---

:dizzy2:

Ok, point taken, no more babbling about indices.
Upside, Aymar's suggestion seems to be the way
to go. Bwian has tried out the import part and this
seems to work albeit some problems in the weighting
and bone assignment parts for some units. The skin
modifier is trickier to use than first glimpse from the
documentation. The backwards part, meaning exporter,
is sort of complicated. GMAX doesn't support the writing
of ASCII or binary files, I assume so people will buy 3dsmax,
so I can't test the export code. Doing some in situ testing
of reassembling the data in the listener.

I didn't say this earlier but all this is from working off of
Vercingetorix's script which is amazing; I wish I actually
could understand it all. So verify the output matches
the input, try and get animation data in, and resolve
the other problems seen with weighting and bone
assignments. I really had no other plans this weekend.:lam:

Okay, all of this is quite confusing for me. KnightErrant, the "22 face groups" you are talking about... are they Smoothing groups, telling the engine witch edge will be hard and whitch one will be smoothy ? Or are they "simple" groups, whose only purpose is to help the artist to manage his model ?
If it is the 2d answer, can't we get rid of it and simply export a single mesh ? And if it is the 1th answer, maybe another way to export smoothing groups can be found ?

Sorry if my questions seems dumbs, maybe I misundertand the problematic.

Quote:

---

Originally Posted by KnightErrant

Ok, point taken, no more babbling about indices.
Upside, Aymar's suggestion seems to be the way
to go. Bwian has tried out the import part and this
seems to work albeit some problems in the weighting
and bone assignment parts for some units. The skin
modifier is trickier to use than first glimpse from the
documentation. The backwards part, meaning exporter,
is sort of complicated. GMAX doesn't support the writing
of ASCII or binary files, I assume so people will buy 3dsmax,
so I can't test the export code. Doing some in situ testing
of reassembling the data in the listener.

I didn't say this earlier but all this is from working off of
Vercingetorix's script which is amazing; I wish I actually
could understand it all. So verify the output matches
the input, try and get animation data in, and resolve
the other problems seen with weighting and bone
assignments. I really had no other plans this weekend.:lam:

---

This might be a stupid and superfluous suggestion, but have you tried to contact Vercingetorix for some pointers about the programming routines of his import/export .cas plug in? Since he also knows pretty well how to model in MAX he could give you a double perspective (Modeller/Programmer) on the problems at hand. He hasn't posted lately due to his univ work taking too much time out of him.

Quote:

---

Originally Posted by Aymar de Bois Mauri

This might be a stupid and superfluous suggestion, but have you tried to contact Vercingetorix for some pointers about the programming routines of his import/export .cas plug in? Since he also knows pretty well how to model in MAX he could give you a double perspective (Modeller/Programmer) on the problems at hand. He hasn't posted lately due to his univ work taking too much time out of him.

---

Verc wrote that script very long ago. Experience tells me that he probably won't remember what he did back then and would have to go through the whole thing again himself. If I judge him correctly he'd probably do that but I don't know if you want to dump that kind of work on him :sweatdrop:

The groups from the mesh aren't smoothing groups. Grumpy's converter
to ms3d just assigns everybody to Milkshape's smoothing group 1. I think
they are used by modellers to organize their work and they are used by the
game engine to randomize the textures so units have varying uniforms.
That, I believe, is why armoured_sergeants have head_01, head_02, etc
legs_01, legs_02, etc so that each can get different textures. (I may be
a little simplistic in this, I'm basing it on answers I've seen modellers give
in other threads.)

I think I'm making some progress. Bwian said some of the bone assignments
were off or missing and I think I've tracked it down to: If you have a
primary bone Id and secondary bone Id from the mesh but one of them
has a weight of 0, gmax (and I assume 3dsmax) won't let you assign both.
Makes sense of course, why assign a second bone with a 0 weight. But if
you want a 1-1 converter that goes back identically you can assign a weight
of 10^(-5) to the secondary bone and it accepts that. On back conversion
to ms3d the weight is converted back to a percentage byte and gets
rounded back to zero. Voila! 1-1 conversion. (Well, hopefully, I can't test
back conversion because of a little snag, gmax doesn't allow writing ASCII
or binary files.)

I've gotten to importing .ms3d files with animation in them. The problem
here is trying to put in the rotations using coordsys parent (3dsmax guys
should recognize this). This doesn't seem to do the correct thing in
MaxScript so I've switch to doing the anims in the bone node's own system.
When you do that you have to put in differences from the previous rotation
because the node frame keeps changing. This seems to work and I've gotten
a walk anim to work except the guy keeps twirling his spear. Here's what
bone_relbow is doing in Milkshape:

Code:

---

-17.1379418352 +77.8505712919  +7.4188831746 
 -13.0322084034 +82.0132164212 +12.0869657175 
  +4.5573767025 +86.4417539847 +30.3697462954 
 +121.1194986650 +86.2430774421 +148.2408411421
 +137.3846516073 +81.3850985647 +165.5491688763
 +140.9322245415 +79.2293337683 +168.3188379055
 +143.1019024332 +78.7743270564 +168.9871172743
 +143.0572603168 +80.3170140729 +167.4641899950
 +140.8697556313 +82.5764338201 +163.9600570640
 +136.4443121435 +84.4876505235 +158.6667697403
 +133.3265903346 +85.3063579785 +155.2575491179
 +141.0409065115 +82.9760545281 +163.7816935040
 +144.7645070810 +79.9915009174 +168.6940748381
 +145.8186648171 +77.9147819003 +170.7855060827
 +146.0272041528 +77.6578438441 +171.3875735977
 +144.7116687375 +81.5825934846 +168.7966642794
 +119.5874325927 +87.5736051226 +142.6517793065
  -7.6112588676 +84.6621072153 +16.0893011844 
 -18.1540554176 +76.9523740953  +6.9879084832

---

See how the x and z rotation angles jump from line 3 to line 4 and then
back down again near the last two frames. Milkshape handles this fine
but gmax is somehow computing the wrong quaternion and making the spear
twirl at these frames. This is probably due to me using differences and
confusing it. Will investigate this a little more.

Last bit of the puzzle. Werner Von Braun said (just from memory)
"Basic research is what I'm doing when I don't know what I'm doing."
Under that kind definition, I do a LOT of basic research.:embarassed:
Wasted a lot of time investigating the different transformations
between eulers and quats. GOM would have pointed out this
salient feature:

Code:

---

                              ^ y
Milkshape:                    |
                              |
                              |_________> x
                              /
                            /
                            /  z

---

Code:

---

Gmax:                      ^ z
                                |    /  y
                                |  /
                                |/________> x

---

It's just a -90 degrees rotation about the x-axis. If I just use Grumpy's
algorithm to compute quats from eulers and apply the
matrix

( 1 0 0 )
( 0 0 1 )
( 0 -1 0 )

to the spatial part of the quat I get the rule

[ q1, q2, q3, q4 ] ---> [ q1 q3 -q2 q4 ]

and the animations play perfectly upon import. Still have to
use coordsys node and take differences but no twirly spears and
the back conversion reproduces the original Milkshape euler angles.

Just have to clean up the mess left by iterating over the geometry
in the scene (basically all the objects get duplicated with names
like head_05). I think this is one of those "should I use 'instance',
'reference' or just assignment" things the docs talk about but I haven't
read carefully. I want a variable that lets me
access the node information saved in a file that I can still delete afterwards
without wiping out the original information.

KE, speaking of all the turning, I wrote in your anims thread a while ago that there was a problem with exporting modified "turn" animations. When you're satisfied that everything works properly, could you take just a sec to try import, modify, export a turning animation and load it up in the game. Soldiers continue turning back and forth endlessly last I tried it.

@SigniferOne
Apologies, I've been very lazy of late; I haven't monitored my threads
like I should. Could you post a link to get me to it? I'm interested of course
if something is wrong and also if it hints at anything that is causing problems
with Bwian's offset mesh chariot animations, because, frankly I'm stumped at
this point. Gone backwards and forwards with "it's an animations problem,
it's a meshconversion problem, it's an animations problem" but we can't nail
anything down. Maybe this would hint at something.

Just a question, you're not seeing "bad animations", just animations that for
some reason play more than they should, i.e. turning and turning?

KE here, last page:

https://forums.totalwar.org/vb/showt...t=83063&page=9

It seems turning animations have a problem importing/exporting.

But please don't worry about being lazy. If anyone deserves to be lazy it's you and the grumpy grandfather. I mean old man. My biggest concern is about you burning out mid-way, that's why my biggest interest is that you take your time, and that we periodically provide you with motivation and inspiration from the results of what your work helped us accomplish. For instance I've given you big accollades in my newly released animations pack (here).

@SigniferOne
Thanks for the accolades! Congrats on the anims, looks
like people are really pleased with the improvements. Also,
thanks for the link, I undrstand now, I abandoned my original
lthread. Actually happy to have a problem with a definite
error message; this I can deal with. As regards burn-out,
I think this was true in July/August but not from modding for
the game; more just summer real world stuff. Working with
Bwian on the gmax/3dsmax import/export stuff really has me
going strong again.

(Wow, just missed it, Boston just took Game 2, go Sox!)

Haven't thought through this, but now that the importer/exporter
is looking better i.e. might just work, doing a importer/exporter
for .world files doesn't look too bad. The Python reader taught
me enough to find the mesh data for the settlements
meaning the structures for northern european villages, towns,
and large towns. This could be adapted to write out
a .ms3d-like binary file for a chosen structure. This could
then be pulled into 3dsmax since it has: verts, faces, uv coords,
lighting (my worry about Milkshape, it can't do this but
3dsmax can; have to learn this), animations (don't know what
Caliban meant about this, I can ask again at the right time, but I
assume this is maybe a particle system.)

This isn't the same as adding whole new structures, but maybe learning
how to change the existing structures might lead to good questions for
Caliban on how to do more with their world files. Right now, one problem
at a time.~;)

Update:

Sorry, not an update on settlements but we will be getting
back to them soon. Bwian did a test yesterday and we've
been able to do a .ms3d model into 3ds Max and back out
again to .ms3d. The model wasn't animated, just the regular
basepose. 2 more tests needed, (1) back and forth conversion
of a model with animations and (2) Bwian's custom models won't
convert. I suspect custom models don't have the null animation
frames the converters (GOM's and mine) put in and I've borked
the process by assuming something is defined that isn't. If this
is cleared up in a day or two, we'll put out an alpha. This will
also work for Gmax users, it's just that the export will be to the
listener. Gmax guys will have to cut and paste the ASCII output
to a .txt file and then run a Python utility to convert to the .ms3d
file format. (This is how I tested things.)

A murrain upon the Discreet guys who made the clipboard buffer
10 Kbytes in size:furious3: . Not so bad for a lod3 but for a lod0 go get a
cup of coffee and cut/paste, cut/paste, .. cut/paste for 15 minutes
or so. I know, if its free you can't really complain.

Got a new version in the mail from KE today, anbd it's a big step forward.

Imports an MS3D file neatly into Max, with all groups, rigging, UV and comments intact. This includes the messaed about with skeletons in the Dwarves etc. so it is now very tolerant of hacked about stuff!

It exports too, and the resulting object can be opened up safely in MS3D. There is a minor issue with rigging, where it turns all the vertices 'green' to indicate a zero value, rather than the red fully asisgned verts ( looks like a side effect of some work KE had to do to make it accept the slightly rougher models made with the tools we work with! )

You also need to re-assign the materials, but this is no problem at all since the UV mapping is perfectly OK and it only takes a click to have it back again.

Next step will be to get the animations to import/export as well, and we have a major new weapon in the arsenal.

Then, we can get the settlement files doing the same :2thumbsup:

:balloon2: :balloon2: :balloon2: hooray :balloon2: :balloon2: :balloon2:

Excellent, it's going great!