Settlements: File Formats; Take 2

[KnightErrant]

Hi all,

This is a continuation of a thread that had some formatting problems.
Hope this thread doesn't. I'm going to try a quick test in this post,
then post the real question. Here's a chunk of the settlement header,
hopefully it will come out ok with linebreaks:


serialization::archive
3 4 4 4
264 0
267 0 0 0
2 0 184.550750732 0
40 0 -7.09150409698 1.21208238602 -2.87348890305

That's the test; I'll log out and log back in to see how it goes.

[KnightErrant]

Ok, that seems to work! Many thanks to alpaca and to TosaInu for running tests to see what was wrong. Here's where I am: The fun starts at where tell says t
ell says 3863224 That's for the hexedit gurus: this is the byte location for where it starts getting interesting. There seems to be some 4x4 homogeneous coordinate matrices here before the float data starts. (I have been reading up on this.) Here's the float data: Edit: Line-Break test by alpaca:
+1.0000000000 +0.0000000000 +0.0000000000 +0.8681725860
+0.0000000000 +1.0000000000 +0.0000000000 -2.2101776600
+0.0000000000 +0.0000000000 +1.0000000000 +1.2806341648
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+1.0000000000 1 0 0 1 0 0 0 0 0 57 0 0 0 0 0 4 0 0 0 2 0 0 0 0 0 0 0 3 0 0 0 57 0 0 0
+1.0000000000 +0.0000000000 +0.0000000000 +0.1207747832
+0.0000000000 +1.0000000000 +0.0000000000 +0.4997914732
+0.0000000000 +0.0000000000 +1.0000000000 +1.9928042889
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+1.0000000000 0 0 0 0 0 84 0 0 0
+1.0000000000 +0.0000000000 +0.0000000000 +0.8747580051
+0.0000000000 +1.0000000000 +0.0000000000 +0.8698613048
+0.0000000000 +0.0000000000 +1.0000000000 -1.4081792831
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+1.0000000000 0 0 0 0 0 111 0 0 0
+1.0000000000 +0.0000000000 +0.0000000000 -0.1258755475
+0.0000000000 +1.0000000000 +0.0000000000 -0.1756218821
+0.0000000000 +0.0000000000 +1.0000000000 -3.3048241138
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+1.0000000000 0 0 0 0 0 2 0 0 0 0 0 0 0 1 0 0 0 25 0 0 0
+1.0000000000 +0.0000000000 +0.0000000000 -0.9229867458
+0.0000000000 +1.0000000000 +0.0000000000 -4.0490546227
+0.0000000000 +0.0000000000 +1.0000000000 +1.6631650925
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+1.0000000000 1 2 0 0 0 200 1 0 0 49 0 0 0 1 2 0 0 66 0 0 0 2 0 0 0 0 0 0 0 1 0 0 0 139 0 0 0
+1.0000000000 +0.0000000000 +0.0000000000 +0.5850151777
+0.0000000000 +1.0000000000 +0.0000000000 -1.8052126169
+0.0000000000 +0.0000000000 +1.0000000000 +0.7539926171
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+0.6999999881 1 8 0 0 0 202 1 0 0 50 0 0 0 204 1 0 0 51 0 0 0 2 2 0 0 67 0 0 0 3 2 0 0 68 0 0 0 21 2 0 0 76 0 0 0 31 2 0 0 78 0 0 0 203 2 0 0 129 0 0 0 209 2 0 0 130 0 0 0 2 0 0 0 0 0 0 0 3 0 0 0 111 0 0 0
+1.0000000000 +0.0000000000 +0.0000000000 -0.1258755475
+0.0000000000 +1.0000000000 +0.0000000000 -0.1756218821
+0.0000000000 +0.0000000000 +1.0000000000 -3.3048241138
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+1.0000000000 0 0 0 0 0 84 0 0 0
+1.0000000000 +0.0000000000 +0.0000000000 +0.8747580051
+0.0000000000 +1.0000000000 +0.0000000000 +0.8698613048
+0.0000000000 +0.0000000000 +1.0000000000 -1.4081792831
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+1.0000000000 0 0 0 0 0 57 0 0 0
+1.0000000000 +0.0000000000 +0.0000000000 +0.1207747832
+0.0000000000 +1.0000000000 +0.0000000000 +0.4997914732
+0.0000000000 +0.0000000000 +1.0000000000 +1.9928042889
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+1.0000000000 0 0 0 0 0 2 0 0 0 0 0 0 0 1 0 0 0 25 0 0 0
+1.0000000000 +0.0000000000 +0.0000000000 +0.8681725860
+0.0000000000 +1.0000000000 +0.0000000000 -2.2101776600
+0.0000000000 +0.0000000000 +1.0000000000 +1.2806341648
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+1.0000000000 1 1 0 0 0 68 2 0 0 91 0 0 0 2 0 0 0 0 0 0 0 1 0 0 0 157 0 0 0
+1.0000000000 +0.0000000000 +0.0000000000 -0.2069556862
+0.0000000000 +1.0000000000 +0.0000000000 -2.0798547268
+0.0000000000 +0.0000000000 +1.0000000000 -1.6279093027
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+1.0000000000 1 1 0 0 0 69 2 0 0 92 0 0 0 2 0 0 0 0 0 0 0 3 0 0 0 111 0 0 0
+1.0000000000 +0.0000000000 +0.0000000000 +0.1647853851
+0.0000000000 +1.0000000000 +0.0000000000 +0.2269621640
+0.0000000000 +0.0000000000 +1.0000000000 +2.0150504112
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+1.0000000000 0 0 0 0 0 84 0 0 0
+1.0000000000 +0.0000000000 +0.0000000000 -1.0282818079
+0.0000000000 +1.0000000000 +0.0000000000 +0.8543335795
+0.0000000000 +0.0000000000 +1.0000000000 -1.4318115711
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+1.0000000000 0 0 0 0 0 57 0 0 0

+1.0000000000 +0.0000000000 +0.0000000000 +1.2263863087
+0.0000000000 +1.0000000000 +0.0000000000 -0.3336898088
+0.0000000000 +0.0000000000 +1.0000000000 -2.3126564026
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+1.0000000000 0 0 0 0 0 2 0 0 0 0 0 0 0 1 0 0 0 139 0 0 0
+1.0000000000 +0.0000000000 +0.0000000000 +0.2344752252
+0.0000000000 +1.0000000000 +0.0000000000 -1.9429701567
+0.0000000000 +0.0000000000 +1.0000000000 +2.2158527374
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+0.8000000119 1 3 0 0 0 208 1 0 0 52 0 0 0 23 2 0 0 77 0 0 0 37 2 0 0 83 0 0 0 2 0 0 0 0 0 0 0 2 0 0 0 111 0 0 0
+1.0000000000 +0.0000000000 +0.0000000000 +0.2340932637
+0.0000000000 +1.0000000000 +0.0000000000 -0.0248921812
+0.0000000000 +0.0000000000 +1.0000000000 -3.7247326374
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+1.0000000000 0 0 0 0 0 84 0 0 0
+1.0000000000 +0.0000000000 +0.0000000000 -0.4714680314
+0.0000000000 +1.0000000000 +0.0000000000 -0.2671088576
+0.0000000000 +0.0000000000 +1.0000000000 +6.7157201767
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+1.0000000000 0 0 0 0 0 1 0 0 0 1 0 0 0 188 0 0 0
+1.0000000000 +0.0000000000 +0.0000000000 +0.5613649487
+0.0000000000 +1.0000000000 +0.0000000000 +3.7120764256
+0.0000000000 +0.0000000000 +1.0000000000 -0.0595130324
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+0.5000000000 0 0 0 0 0 2 0 0 0 0 0 0 0 1 0 0 0 25 0 0 0
+1.0000000000 +0.0000000000 +0.0000000000 +0.1719831526
+0.0000000000 +1.0000000000 +0.0000000000 -2.7414629459
+0.0000000000 +0.0000000000 +1.0000000000 -5.0158138275
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+1.0000000000 1 1 0 0 0 94 2 0 0 94 0 0 0 2 0 0 0 0 0 0 0 1 0 0 0 212 0 0 0
+1.0000000000 +0.0000000000 +0.0000000000 +0.8394250870
+0.0000000000 +1.0000000000 +0.0000000000 -3.9999992847
+0.0000000000 +0.0000000000 +1.0000000000 +1.1408995390
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+0.3000000119 1 5 0 0 0 101 2 0 0 95 0 0 0 102 2 0 0 96 0 0 0 124 2 0 0 105 0 0 0 16 3 0 0 170 0 0 0 17 3 0 0 171 0 0 0 2 0 0 0 0 0 0 0 1 0 0 0 139 0 0 0
+1.0000000000 +0.0000000000 +0.0000000000 +0.5850151777
+0.0000000000 +1.0000000000 +0.0000000000 -1.8052126169
+0.0000000000 +0.0000000000 +1.0000000000 +0.7539926171
+0.0000000000 +0.0000000000 +0.0000000000 +1.0000000000
+0.6999999881 1 1 0 0 0 104 2 0 0 97 0 0 0 2 0 0 0 0 0 0/Edit Doesn't this look like homogeneous coordinate data with weird granny bytes after it? I'm hoping GrumpyOldMan, Casuir, or zxiang1983 will take pity on me and find a pattern in the granny strings so I can go forward with some guidance but I can't see a pattern here. Hope this post comes out formatted alright. Many thanks for the interest in the subject:laugh4: If nothing else, it would be fun to make villages/towns etc. more tailored to the culture. Really, the interest is of course the fantasy mods with round doors for hobbit villages but better architecture for old (biblical, roman, etc.) mod types would be interesting too. Love this stuff:tredmil: , hope others do as well. KE Edit: Sigh, same problem, no line breaks....don't know what to do about this. I used Mozilla but I can also use IE or the latest Firefox. If anything occurs to people please let me know.

[KnightErrant]

Ok, for humour.

Competititon announced. My birthday is tomorow.
Cleverest entry wins; but no prize is announced.
No credit for "Dude, you're really old" however, clever
stuff like "Did you have hamburgers when you were
growing up" is pretty good. You have to check the
entry section to get the birthday announcements. If you
get it wrong you don't get to play.

Cheers,

KE

[SigniferOne]

KE, How did you find the Model T to handle when younger?

[Ashdnazg]

Yo KE, good sling job on that Goliath dude.
Oh wait, confused you with your cousin, never mind then.

[KnightErrant]

Ok, only two entries in the competition but I liked them both.
Model T handled well but it had nothing on my Stanley Steamer.
Goliath was over-rated, taunting was good, a real clutch player
with a sword but no shield control. If Ashur-Steinbrenner hadn't
traded Marluk the Assyrian for Femnod the effeminant Babylonian
he never would have made it out of the minors. :laugh4:

Tried a few post previews using IE but the data came out all jumbled.
I'll check out a couple of other options tomorrow.

[GrumpyOldMan]

Hi KE



Ok, for humour.

Competititon announced. My birthday is tomorow.
Cleverest entry wins; but no prize is announced.
No credit for "Dude, you're really old" however, clever
stuff like "Did you have hamburgers when you were
growing up" is pretty good. You have to check the
entry section to get the birthday announcements. If you
get it wrong you don't get to play.

Cheers,

KE

I'm sorry I missed your birthday, I was away with the eldest at an athletics meet.

As an act of contrition I do offer you a cyber present though, it's http://www.wsu.edu/~brians/errors/index.html , a web site with literally thousands of common errors (and corrections) in English. We both know that it's the natural state for a man of certain years to be pernickity and pedantic, and this is an admirable tool to assist in that regard:laugh4: :laugh4: .

Back to the settlements, I honestly tried to look at the data you've posted but I get dizzy without line breaks. What are you copying from and pasting into? Sometimes there is a funny thing that happens with line breaks. Have you tried opening it in Notepad, saving it and then copying and pasting?

From the settlement you've looked at, how many total verts and tris would be involved? Is it possible to set the individual models up as groups within a milkshape .ms3d? And positions based on bones/joints? Milkshape has the following limits:-

#define MAX_VERTICES 65534
#define MAX_TRIANGLES 65534
#define MAX_GROUPS 255
#define MAX_MATERIALS 128
#define MAX_JOINTS 128

Cheers

GrumpyOldMan

[KnightErrant]

Hi GrumpyOldMan,

Thanks for the web site and for trying to look at the posts without
line breaks. Kind of reminds me of the vanilla battle_models.modeldb
file. How did people actually mod it that way?

Two thoughts on my problem: Easy way/hard way. (1) Maybe I just
can't start a thread. Solution, hijack an old wiki thread and do settlements
postings there. (2) Put it all in a pdf and just upload it. I'd rather do (1)
because (2) is too hard.

Good news though! I reached the end of the smallest village .world file today.
I just programmatically hard coded through all the weird grannies near the
end that were driving me crazy. ("Programmatically hard coded" is the worst
oxymoron I've ever made up but what I mean is, rather than just put in the right
byte numbers for the grannies between data, I kept making up rules until I
got through the section just by doing a for loop.) I won't try to post anything
because I'm basically where you were when you posted the mesh file stuff
in February, I know where the data IS, I just don't know what all the stuff
around it is. Also, I've made bad assumptions in the early part about what
bytes should be integers/shorts/ etc that I should probably re-think.

Without posting data and recalling Caliban's input the upshot is:
Settlements contains vertex and triangle data that can be pulled out.
The texture file paths are in there like siege weapons. No bone strings but
there are other strings I'm not sure of (near the end). Caliban gave indications
that some of the quad bytes sections are lighting models and animation
place holders. This might be a problem as this doesn't mesh with Milkshape's
allowed data.

What I'd like to do next is just re-look at the data and grannies to make
sure the file provides numbers to read it better; but not try to make sense
of the header/footers yet (always too hard to start with). The data beyond
just the settlement meshes seems to be 4x4 homogeneous coordinate
affine transformations which I would not have recognized before.

Without thinking further, at least it might be possible to modify what's there
but not add new stuff (don't know what would have to change with the grannies). I.e. you have 30 and only 30 village types to choose from, BUT,
you can make those 30 village types anything you want. Fingers crossed
that that might be true.~;)

Regards,

KE

[alpaca]

Hmm these line-break problems are pretty weird. I played around with your above post and if you cut the entries somewhere linebreaks seem to work.

Edit: All right, it seems the "[code]"-tags can't have more than 65 lines each (or there's a maximum number of letters), so you have to split very large ones up.
Don't ask me why, though, maybe the programmer used a fixed-length array.
I pretty arbitrarily split your post into 5-line-packages with 4 of your float entries and/or all the strings. It seems (if you interpreted the float/string stuff correctly) that this stuff usually consist of 17 floats and a granny string? 17 is really weird, being a prime number it doesn't make much sense for coordinates or anything unless one or two of the numbers mean something else

Edit2: Meh damn pattern-detecting brain... For the granny strings, a subdivision into packages of four seems to make most sense, too (skipping the very first one). The first seems to differ most, the second one is often 0, sometimes 1 or 2 and very rarely 3 - and the third and fourth are always 0 I think. So #1 could be an id or a link or something, #2 could be a flag with three options and the other two are unused (in this file).
I also read your post and you mentioned the 4x4 matrices :sweatdrop:
Translation matrix, huh. Makes sense in a way. The question is: What does the 17th float do?

[KnightErrant]

Yes, Thinking about it we've probably seen these 4x4 matrices before in
the mesh files. The book I've got on 3-D graphics has this to say for
homogeneous co-ordinates (but my words not the book's):

You can do rotations in 3D: R v = x
You can do scalings in 3D: Sv = y

but to do translations you have to do v = x + T

No longer a matrix multiplication but a vector add. Once you have adds and
multiplications you have a ring structure (abstract algebra) but not a group
structure (just adds or multiplications, depending on what you are doing.)
Rings have less known about them so groups are better. The cool part, and
the part I missed at university is that in projective geometry you can make
4x4 matrices like:


( R Tvec) ( v ) = ( x )
( 0 1 ) ( 1 ) ( y )

and make translations obey matrix multiplications and so be a group. (This
won't work out well in HTML but this is all I can do right now.) These are
called homogeneous co-ordinates and are probably known to all the modellers
\but where news to me. (My Linear Algebra course skimped on projective
geometry.)

I've seen these in the settlement files but, like you said, there's a 17th float
which doesn't fit (rotations and translations and then an overall scaling?).
At any rate, I have found some overall logic to the .world file. The header
says 40 so you can read the first part: that part defines 31 entries that
have a sequence number/ There happen to be 31 complexes so maybe
that is the magic number. Then come 867 entries. Total mystery except
the header/footers contain a count of the number of homogeneous data
segments but also contain a secondary count of somethingh else. At the
end that number turns out to be 867. Wish I knew what it meant.

I've been refactorizing the code and, except for these oddities, it is almost
suitable for putting in for loops. Maybe a little more study will unlock the
inexplicable. (Or make me buy an X-box: Mario bros. was entertaining when I
played it against my daughter.) :laugh4:

Apologies, I should be fixing animation difficulties but I can't do
that at lunch time; unemployment would result if I installed the game.

@GOM, I've copied almost all of that interesting chapter, just a few pages
more. Sorry, been kinda' rattled with contractors and getting lawn stuff done.

Regards,

KE

[alpaca]

Thanks for the explanation, I already looked it up on wikipedia.

Scaling might make sense, since we're dealing with pre-defined buildings consisting of blocks it could be we have some groups here which have of course to be scaled uniformly (because otherwise they'd be distorted).

Ok looking at the file:

At the start we have some stuff I can't really identify in the header but you might know what it is (bytes 26-51), then we get a float followed by two 0-bytes and your number 40.
Following that, we have a four-byte 0 and then three floats again which might be coordinates.
After that we get 17 integers again (with some -1s in there which means it's probably settings of some sort) until we hit byte 147.
Then nine floats and 17 integers, again followed by 9 floats and so on. I guess these are the entries you described, but I count 40 of them. I'm pretty confident these describe something of our settlements, three points in each entry and followed by a number of integer-settings of some sort (a lot of which are disabled, i.e. -1 - or 2).
Rinse and repeat to byte 4229 where we get the 867 you speak of, followed by 867 almost identical entries (a lot of them are totally identical except for their "id")

Unfortunately I probably can't help you to any huge extent because I don't have that much time and you know the granny file formats much better than I do anyways.

[alpaca]

Hey KE, is any work still going on on this? It'd be a shame if this wasn't pursued further :2thumbsup:

[SigniferOne]

Indeed!!

[KnightErrant]

@alpaca and SigniferOne,

Many apologies, I've really dropped the ball on animation support and
posting anything new on settlements. Probably can't get really active
again until the yard stuff (don't ask) is finished, but here is where I am
on settlements. Made it through the village.world a few weeks ago.
Almost throught town.world lately. (Only getting stuff done at lunch time.)
Lots of new stuff in the town.world file, ArrowTowers, ArrowSlots, gates,
perimeters, etc. Don't know what any of it means yet but it all has to be
coded through. The good news is that I think I've seen all the granny variations
for settlement data. Without going into it here, 95% of the data in a .world
file is mesh models. They are organized as "complexes" with each complex
containing 1 to 8 maybe more "structures". These structures are a self
contained mesh with vertices, tris, normals, tangents, the other stuff
Caliban told me about like uv coords as cube normalized RGB values, and
animation byte quads (don't know yet what these are). Reading the complexes
has finally been reduced to a simple loop with no hard coding. Its all the
data at the end that is still incomplete. I'm sure if I try large_town I'll see
even more stuff I haven't seen yet but it gets easier each time I try another
world and there's only what, city, large_city, huge_city, to go. Even if they
require separate case statements for their own oddities, that isn't out
of the question.

Problem is, even if I leave out all the actual mesh data, the granny strings
and other data still run to ~0.5 Meg in a .txt file. This is probably too much
to post and ask for help on (and I haven't tried lately to see if I can post
code anyway). Maybe the best thing is to get town.world read all the
way through and then upload the Python code so people interested could run
it and see the .txt file output and then post comments on what the
data blocks at the end might mean. I've run into one very odd data structure
that just doesn't seem to have an entry that allows me to read it
programmatically; I have to just crunch through it until its over and then
hard-code it. Would really like some other eyeballs on this problem.

Nowhere near a converter, this is still just reading data to the end.
Hopefully, a converter would pick out a structure for a modeller to work
on and reconstruct the whole .world file with the new structure in place.
The worrisome part of that are the quad byte sections Caliban described
that relate to animations and to lighting sources; Milkshape doesn't have
this so this may be a 3dsmax only thing. Not ready to try this, but remember
when we were doing the mesh file for units and didn't know what the mystery
block were? I could try just putting in arbitrary values and see what happens.

(Reread that and realized it doesn't make any sense if you haven't been
living in these files. Better explanation follows.)
The quad byte blocks must have the same number of entries as the vertices.
Makes sense if they are vertex normals, tangents, or uv coords. If they
are animation or lighting entries, then that means the anims, for instance,
are tied to specific vertices. I'm thinking this is something like smoke from
chimneys. If you mod a structure by adding more vertices, how do you
make sure the smoke still comes out of a chimney and not the side of a
house because the vertices got renumbered? If you can't get the data
into Milkshape you would have to do something else. What that might
be I haven't got to.

That's the update.....:turtle: Yep, this is taking awhile.

KE

[SigniferOne]

KE, much happiness on the progress, and love the turtle. FYI, the chimney smoke is defined in the chimney smoke text file (as regards the density of smoke etc). Maybe the animations could refer to windmills. But from what I remember, windmills don't turn, so maybe it's something CA wanted but later turned back on (which means you can just hardcode to paste it in, without debugging it). Either way, all is nice, hope the yard fixes itself. (yeah!)

[Ashdnazg]

are there any flags blowing in the wind (or at least turning awkwardly)?

[alpaca]

Did you check the animinstances file? I thought it would contain the animation data.

[KnightErrant]

Re: animinstances

Here's what surfaced Friday. I've seen .anim and .animinstances in the
village.world file. Friday I hit the same bit in the town.world file. Going to
try and post the section here:



# ------------------------------------------
# Beginning of .anim strings Section
# ------------------------------------------
0 0
nentries = 12
0 0
ntrans = 3
17 DamageTransition0
106BlockSet/North_European/Animations/Wall_Animations/Town_Wooden_Palisade/palisade_straight_animation_a.anim
17 DamageTransition1
106BlockSet/North_European/Animations/Wall_Animations/Town_Wooden_Palisade/palisade_straight_animation_b.anim
17 DamageTransition2
106BlockSet/North_European/Animations/Wall_Animations/Town_Wooden_Palisade/palisade_straight_animation_c.anim
ntrans = 1
17 DamageTransition0
119BlockSet/North_European/Animations/Tower_Town_Animations/Wooden_Palisade_Animations/palisade_arrow_tower_animation.anim
ntrans = 1
17 DamageTransition0
124BlockSet/North_European/Animations/Tower_Ambient_Animations/Ambient_Stone_Tower_Animations/stone_tower_4m_E_animation_a.anim
ntrans = 1
17 DamageTransition0
124BlockSet/North_European/Animations/Tower_Ambient_Animations/Ambient_Stone_Tower_Animations/stone_tower_4m_B_animation_a.anim
ntrans = 1
17 DamageTransition0
124BlockSet/North_European/Animations/Tower_Ambient_Animations/Ambient_Stone_Tower_Animations/stone_tower_4m_G_animation_a.anim
ntrans = 1
17 DamageTransition0
126BlockSet/North_European/Animations/Tower_Ambient_Animations/Ambient_Thatch_Tower_Animations/thatch_tower_4m_B_animation_a.anim
ntrans = 1
17 DamageTransition0
126BlockSet/North_European/Animations/Tower_Ambient_Animations/Ambient_Thatch_Tower_Animations/thatch_tower_4m_G_animation_a.anim
ntrans = 1
17 DamageTransition0
126BlockSet/North_European/Animations/Tower_Ambient_Animations/Ambient_Thatch_Tower_Animations/thatch_tower_8m_B_animation_a.anim
ntrans = 1
17 DamageTransition0
126BlockSet/North_European/Animations/Tower_Ambient_Animations/Ambient_Thatch_Tower_Animations/thatch_tower_4m_E_animation_a.anim
ntrans = 1
17 DamageTransition0
124BlockSet/North_European/Animations/Tower_Ambient_Animations/Ambient_Stone_Tower_Animations/stone_tower_4m_D_animation_a.anim
ntrans = 1
17 DamageTransition0
126BlockSet/North_European/Animations/Tower_Ambient_Animations/Ambient_Thatch_Tower_Animations/thatch_tower_4m_K_animation_a.anim
ntrans = 8
17 DamageTransition0
120BlockSet/North_European/Animations/Gate_Animations/Wooden_Palisade_gate_anims/NE_wooden_palisade_gate_damage_impact.anim
17 DamageTransition1
123BlockSet/North_European/Animations/Gate_Animations/Wooden_Palisade_gate_anims/NE_wooden_palisade_gate_damage_destroyed.anim
14 DoorCloseAnim0
112BlockSet/North_European/Animations/Gate_Animations/Wooden_Palisade_gate_anims/NE_wooden_palisade_gate_close.anim
14 DoorCloseAnim1
119BlockSet/North_European/Animations/Gate_Animations/Wooden_Palisade_gate_anims/NE_wooden_palisade_gate_damage_close.anim
15 DoorImpactAnim0
113BlockSet/North_European/Animations/Gate_Animations/Wooden_Palisade_gate_anims/NE_wooden_palisade_gate_impact.anim
15 DoorImpactAnim1
120BlockSet/North_European/Animations/Gate_Animations/Wooden_Palisade_gate_anims/NE_wooden_palisade_gate_damage_impact.anim
13 DoorOpenAnim0
111BlockSet/North_European/Animations/Gate_Animations/Wooden_Palisade_gate_anims/NE_wooden_palisade_gate_open.anim
13 DoorOpenAnim1
118BlockSet/North_European/Animations/Gate_Animations/Wooden_Palisade_gate_anims/NE_wooden_palisade_gate_damage_open.anim
79 settlements/North_European/Settlements/Town/north_european_town_A.animInstances


This stuff comes after all the "complex" and "structure" data (my made-up names). The animInstances file always comes last after the .anim files.
Have not looked at any of these other than to see, yes, they are more
binary files. If luck holds then they should have a structure like the .cas
files but I don't know this.

I'll just post this and see how the code tags work for me now. If not this will
just be another code posting with no line breaks.

Edit: Couldn't find a smiley with crossed fingers but this looked ok after
posting. As usual, I'll log out and log back in.

Edit2: Still working!!! Holy moly, I've got a posting jones working here.
Too late tonight for a discussion but I'll shoot for tomorrow.

[alpaca]

Yeah, you just have to keep in mind to not post more than 50 lines or so.

Ok so the anim file apparently contains the damage and open door anims, what's in animinstances?

[KnightErrant]

Aah, so the 50 lines may have been the problem.

Haven't done anything more, BUT life looks managable at work
now. Will try to get to the end of towns and then comparing villages
to towns may reveal something. Will look at animinstances next with
maybe a check on large_towns to see what else may come into play.

At this point why am I doing this?

(1) Hundreds of modellers are waiting to mod settlements?
Naw, hardly think so.

(2) Tens of modellers are waiting to mod settlements?
Naw, hardly think so.

(3) Bwian is waiting to make Hobbit villages?
Probably.

(4) I've bit off more than I can chew and am just too embarrassed
to admit it.
Yeah, that one! That's exactly it!!!

Sheer bloody-mindedness is a strong motivator.:laugh4:

KE

[alpaca]

Yeah I know the feeling. If it makes you happy though I'm also looking forward to edit settlements (I liked playing Lego and what Caliban told me about the process it's a lot like that, it only looks much cooler :laugh4: )

[Redmeth]

I can't speak for everyone, but I for one and I think most of the EB team is also tremendously interested in the possibility of editing settlements.

[SigniferOne]

KE, how can you say that!

https://img357.imageshack.us/img357/8100/roads26vo.th.jpg (https://img357.imageshack.us/my.php?image=roads26vo.jpg) https://img65.imageshack.us/img65/1145/roma15lw.th.jpg (https://img65.imageshack.us/my.php?image=roma15lw.jpg) https://img236.imageshack.us/img236/3899/romawalls111xd.th.jpg (https://img236.imageshack.us/my.php?image=romawalls111xd.jpg) https://img77.imageshack.us/img77/6007/betweencispusandviminalis7tn.th.jpg (https://img77.imageshack.us/my.php?image=betweencispusandviminalis7tn.jpg)

Imagine that in M2 engine.

[KnightErrant]

Ok, point taken. I am encouraged to keep going.
Here's a link to the beta reader.

http://www.twcenter.net/forums/downloads.php?do=file&id=1578

The two files in the zip are animationlibrary.py and settlementutilities.py.
animationlibrary.py is almost the same as the one distributed for the
animation utilites except I put a trap in the function getstring to jump
back out if it gets a requested number of characters more than 1000.
This usually means I'm reading the wrong int and it puts Python into a loop
where it beeps every time it reads the bell code, very annoying to have
happen at work!

settlementutilities.py is the one to double-click on. Just put
north_european_village_a.world and north_european_town_a.world
in the same directory with the two Python files. Currently the .txt
file doesn't have the actual mesh data output to it. If you want this
just search for WRITEDATAFLAG and set it to True. The output file
will be very large!!! You can cut and paste the vertices and tris
to make an obj file. The u and v coordinates are also converted from
RGB encoding into floats if you want them too.

Now for numerology, I'll start. The village has 31 "complexes" containing
a total of 98 "structures" or self contained meshes. The town has 30
complexes with a total of 123 structures.

Villages have a block of data at the top (after the 40 entries block of
which 31 have a positive sequence number that should correspond to the
complexes) that has 867 entries. This number shows up again near the
bottom for another block of data.

For towns the number is 1228.

Question 1: What do these entries correspond to?

For no good reason I kept a running total of the number of triangle groups
defined in all the structures. For villages this number is 1274. Guess what,
there are two blocks of data right at the end (after a block with 867 entries)
that have 1274 entries. For towns this number is 2017 with the same
correspondence. So, of course,

Question 2: What do these entries do or is this a coincidence?

There is a block of data preceded by this comment by me:
# ------------------------------------------
# Beginning of hard coded 17 floats but
# variable ints with 5 byte signatures
# ------------------------------------------
It has 156 entries for villages and 173 entries for towns, BUT, nowhere
can I find that number to read in order to tell me how to read this data.
Right now it is just hard-coded in. This is very unusual for granny strings;
they always tell you how to read themselves so I'm a bit perplexed on this
one. The entries are 17 floats followed by what I read as 5 bytes being
a signature, then followed by a variable number of ints which, by trial and
error, I have made up rules for until I was able to read both cases, village
and town. One entry looks like this:


+1.00000 +0.00000 +0.00000 -0.11595 +0.00000 +1.00000 +0.00000 -4.10875 +0.00000 +0.00000 +1.00000 -4.22434 +0.00000 +0.00000 +0.00000 +1.00000 +1.00000 sig: 1 0 0 2 0 , ints: 0 0 0 0 68 3 0 0 0 0 155 0 0 0 13 4 0 0 220 0 0 0 2 0 0 0 0 0 0 0 3 0 0 0 54 0 0 0


If I rearrange the first 16 floats into a 4x4 matrix I get



+1.00000 +0.00000 +0.00000 -0.11595
+0.00000 +1.00000 +0.00000 -4.10875
+0.00000 +0.00000 +1.00000 -4.22434
+0.00000 +0.00000 +0.00000 +1.00000

+1.00000 sig: 1 0 0 2 0 , ints: 0 0 0 0 68 3 0 0 0 0 155 0 0 0 13 4 0 0 220 0 0 0 2 0 0 0 0 0 0 0 3 0 0 0 54 0 0 0


In homogeneous coordinates this is a translation with no rotation.
All of the entries seem to be translation only.

Question 3: Why 156 and 173 and how can I figure out when I'm done reading
this data block?

This is probably enough for now. If anyone gets a chance to run the program
and look at the grannies and data blocks please post back ideas.

Edit: Aargh! Found a write out bug for villages. After all the complexes are read
there are three strings for .worldTerrain etc. After that my code wrote out

57 0 1 2 [4307] 0
57 0 1 2 [4307] 0 0
57 0 1 2 [4307] 0 0 0
57 0 1 2 [4307] 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
56 0 1 0 [4308] 57 0 [4307] 0 0 255 255 255 255 0 0

That's not what is in the file just this

57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
56 0 1 0 [4308] 57 0 [4307] 0 0 255 255 255 255 0 0

I think my editor auto-indented the write statement to be inside the loop
instead of after it, the Perils of Python.

[alpaca]

Short notice before I leave (and after I return I'll have to do some learning so I probably won't get around to research anything more today):

The 1274 4-byte entries you mention are in my opinion a kind of coordinate-transformation between IDs. They consist of a starting ID and a second int that specifies how many of this structure have that ID (usually 1 but can be 2 or more) so the transformation is between a block ID and a unique ID for each instance.
In fact they're not 1274 entries but 1274 different unique IDs. They're only 867 entries (counting offsets and dividing by 4) and therefore coincide perfectly with your 867.

[SigniferOne]

KE, I hope you haven't taken leave of us, and left us bereft unhappily in the dark!

[alpaca]

KE, I hope you haven't taken leave of us, and left us bereft unhappily in the dark!
I second, although I wouldn't blame you :laugh4:

[KnightErrant]

Hi SigniferOne and Alpaca,

No, no, not at all. I've just been trying to go back to animations
and catch up on some problems people have been having.
TIKO_TWOW_RU has had a animation problem where somehow
in Milkshape he got bones to have different numbers of animation
frames and that killed the game, not suprisingly. Working on that
revealed that the .cas to .txt utility doesn't convert degrees to radians
so that's wrong. Hopefully, no one was using that. It's been an easy fix
but haven't been doing settlements this last week.

Started working on large_towns just to see what is new and the
good news is the complexes and structures were read ALMOST
without any new coding, but right after them (which is fortunately
90% of the file) completely new data structures start unlike anything
in villages or towns. This slows things down of course so I'll be working
on this the coming week. I hope Kingdoms doesn't change things much;
probably won't have anything near a converter till then. (I keep caveating
this on being able to do something about the data blocks that Caliban said
had to do with lighting and animations.) When we get to this. more information
might make this worrisome bugaboo go away.

Probably nothing to report for a week or so. However, any insights on the
village or town data structures is very appreciated.:yes:

KE

[Aymar de Bois Mauri]

I can't speak for everyone, but I for one and I think most of the EB team is also tremendously interested in the possibility of editing settlements.I concur. It would be very sad to see development on this area stopping. Keep up the great work.

[Redmeth]

I have no expertise in this area but this seems like a very complex problem that could be easily solved by CA if they were willing.

Does anyone fancy the idea of a petition by modders addressed to CA with the objective of allowing the building in M2TW and Kingdoms to become available for modding?

I am not volunteering to lead this idea but a lot of people that are the best modders of the community seem to be interested in this and allowing the buildings to be moddable would show CA's goodwill towards the community and encourage modders to do even more extraordinary things.

[alpaca]

I have no expertise in this area but this seems like a very complex problem that could be easily solved by CA if they were willing.

Does anyone fancy the idea of a petition by modders addressed to CA with the objective of allowing the building in M2TW and Kingdoms to become available for modding?

I am not volunteering to lead this idea but a lot of people that are the best modders of the community seem to be interested in this and allowing the buildings to be moddable would show CA's goodwill towards the community and encourage modders to do even more extraordinary things.
Believe me, a petition would have the same effect as swatting a locust in a swarm of two billion.

CA management isn't prone to show any goodwill to modders or even consider the idea. What help we got was because certain individuals (mainly Caliban and Palamedes, with the help of a few programmers) really tried to push things in our favor - but even with this, very little came out of it.

It simply looks like the bigwigs at CA and SEGA aren't interested in modders at all. They tolerate us, but they don't try to help us. If anything, they at least didn't try to stop their subordinates helping us, either.

[Roman_Man#3]

Hello,

From what I read, this doesn't look to good, but can custom buildings and cities from RTW be brought into M2TW?

Thanks,
RM3

[Ashdnazg]

not at the moment, they're totally different formats.

[Roman_Man#3]

I thought they were both Cas files. So there is no way of editing battle map buildings in M2TW yet?

[KnightErrant]

I think I concur with Alpaca, as much as Caliban has helped we are
probably on our own as regards settlements. Sorry, haven't been
monitoring this thread as I haven't done much. Here's where I've
gotten to: can now read large_towns and it just gets worse. More
data after the complexes/structures, the deployment outline is much
larger etc. I've also been leaning towards the realization that Milkshape
just doesn't have the data structures needed to hold the extra vertex
information that is in the complexes/structures (these are the two
"mystery blocks" that Caliban said were lighting and animations, I forget
the integer codes but they are in an e-mail somewhere).

So I'm stepping back and looking at 3dsmax as the only viable container
for this. I don't own a copy of 3dsmax but Bwian has pointed me to gmax
to learn MAXScript. I've slacked off on settlements to try and support
animations and try and get models into 3dsmax via MAXScript.
If I can just learn how to do a simpler problem then maybe settlements
won't seem so insurmountable. (At least settlements don't have bones,
my current headache in gmax.) However, to me it looks like the best
outcome would be: we should be able to modify an existing structure
(meaning a set of buildings, i.e. change the meshes to get a different
architecture, change textures, etc.) but adding new ones probably
involves changes to all the data the follows the complex/structures
section that I just don't see how it could be done without some magic
CA tutorial. (Not even possible I would think, that data is probably all
granny driven anyway.)

Real life problems, the tensioning spring on my lawn mower broke tonight
so tomorrow evening is a hunt around for repair parts. Check back on
Friday.

[SigniferOne]

KE, I'm concerned about the fact that you're trying to import a whole settlement into Milkshape, and failing to do that given the constraints, find yourself necessary to go to another 3d platform. Remember my suggestion, that milkshape be only used as a kind of 3d reference tool, to know where to place buildings. If I gave you a set of coordinates and a bunch of building models, would you be able to produce a .model file from that? That's all that's really necessary here.

Also why do you trouble yourself with converting all of the scripts to 3dmax form?? If for some reason we NEED 3dmax to do settlement editing, then let it be used only for settlement editing. I would be loathe to think that you wanted to do so much for the community and then got discouraged and left it half way... The modeling and animation scripts work (almost) perfectly (see my post in anims thread), so just merely having a way to do buildings, in 3dmax or otherwise, is the last piece of the puzzle. Because with Vercingetorix script we can already import RTW buildings into 3dmax, and thus being able to insert them into Medieval 2 would be just a dream.

[alpaca]

If only we had access to CA's blocksets. This would make it much easier to figure out which part of the file is which.

[Ashdnazg]

@KE: I'm currently working on a MAXscript importer/exporter for mesh files. Unfortunately I can only work on it in my vacations from the army so progress is rather slow
If you want my WIP script, or any assistance, PM me and I might be able to help.

[KnightErrant]

@ashdnazg
Actually I'd love some help. I've got bones now using point nodes
and skinops.addbone to add them into the mesh. I've got named
selection sets which seem to be like Milkshape's groups. The problem
right now is I can't do vertex weighting because the skin modifier
I associated with the mesh says it has no vertices even though the
mesh does. Here's the code snippet:


theskin = skin()
theskin.rigid_vertices = false
select themesh
max modify mode
addModifier themesh theskin
for j = 1 to nbones do
(
skinOps.addbone theskin (getNodeByName pnodes[j].name) 1
)
nb = skinops.getNumberBones theskin
print( "Number of bones in the skin = " + nb as string )

nv = skinops.getNumberVertices theskin
print( "Number of vertices in the skin = " + nv as string )



I get 26 for the number of bones so that's working but 0 for the number
of vertices. The documentation says it should return the number of its
associated mesh. When I loop over the number of vertices and try
to do

skinOps.SetVertexWeights theskin i boneId_array[i] primaryweight_array[i]

I naturally get an error. This is where I'm stuck.

@SigniferOne
Oh, I agree, no need to redo the Python animation utilities. You can
anim merge and then import the new .ms3d file into 3dsmax so no need
for that one. Would need an exporter though for any animations made
in 3dsmax but that shouldn't be hard (famous last words). The other
export skeleton functions in Python should be fine as is.

[KnightErrant]

Found the magic incantation:


selbitall = #{1..themesh.numverts}
themesh.selectedVerts = selbitall
skinOps.resetSelectedVerts theskin


This forces the skin to update internally and now it
has vertices and the weighting part doesn't die.

N.B. I first tried


skinOps.Invalidate theskin 1

because the documentation said it would have the same
effect but this killed gmax dead as a doornail.

[Ashdnazg]

I used

classOf themesh
to solve this, updates the stuff as well.

but as the saying goes "if it works, don't touch it!"

[KnightErrant]

To moderators: Even though the question seems unrelated to the
thread please leave, it does have relevance.

This question is posed to two groups: those who play with MAXScript
and to modellers. It is the same question posed two ways lets start with
the MAXScript guys. I want to take the group indices that I get out of
an .ms3d file's third section and make a "group". I put it in parentheses
because a group has a somewhat different meaning in GMAX (my cheapo
surrogate for 3dmax). I've tried several MAXscript face selection methods
to make a named "selection group" that I can save in the global
selectionSets array but nothing seems to survive but the name. I don't
get an actual face selection. Any ideas?

Posing the same question to modellers who have experience with 3dsmax.
If you were going to select a set of faces and then "group" them and
give them a name so you could manipulate them later, perhaps by adding
more face to them, how would you do this. Please include excruciating
detail because I want to turn on macro recording and see what commands
your mouse clicks correspond to.

This question is aimed at getting a 3dsmax importer/exporter working but the
ultimate goal is to bring in settlement data that can support lighting arrays
(haven't looked at yet) and animation arrays (don't know what Caliban was
talking about, my books on MAXScript don't cover this.) Any info much
appreciated. :coffeenews:

[Aymar de Bois Mauri]

Well, from a purely modeling perspective (MAXScript is definitely not my cup of tea) and coming from a pure amateur perspective, let me try to help:


If you were going to select a set of faces and then "group" them and
give them a name so you could manipulate them later, perhaps by adding
more face to them, how would you do this. Please include excruciating
detail because I want to turn on macro recording and see what commands
your mouse clicks correspond to.In MAX, and as far as I know, any set of faces that is selected can only be grouped and named if it stands as his own object outside of the main mesh. The main mesh itself has a specific name. So, trying to remove any ambiguity, if you can name it in MAX, it's a separate object. If you want to make it part of the main mesh it will loose it's name and it's group properties and will be no more than a part of the main mesh.

Now do you want a point-by-point explanation of the commands used?

[KnightErrant]

@Aymar de Bois Mauri,

Thank you, I think you have succinctly explained why what I was trying
to do was so hard: it is, in fact, impossible. In a mesh file you have one
large vertex array but (take armored_sergeants for example) 22 face groups
which index into that one vertex array. In Milkshape you have the same
large single vertex array, a single large face array, and then an auxilliary
set of 22 "groups" that index into the large face array to get the grouping
structure. What I'm hearing is that in Max I need 22 little vertex arrays
accompanied by 22 little face arrays indexing into their separate dedicated
vertex arrays to form the same groups. In other words 22 separate named
meshes.

Yikes, this is going to be an index housekeeping nightmare. Importing seems
straightforward because I know how to grab vertices by index and simply put
them into the 22 new little arrays, perhaps with duplication if the face
groups share a vertex. Exporting is the nightmare. I have to go through each
group in order, pull out the vertices and put into a single large array. Reindex
the faces to point to the same vertex which now has a different index
in the large array, AND remove duplicate entries, all the while keeping the
indexing straight. :dizzy2: I'm going to crawl back under my rock and think
about this for a while.

[KnightErrant]

Ok, back from under my rock. This is just a gmax hack for proof of
concept but if I do like Aymar said and make 22 editable meshes
I do get groups:

https://img152.imageshack.us/img152/7756/gmaxhackrt2.jpg
By knighterrant (http://profile.imageshack.us/user/knighterrant) at 2007-10-17


Here I just blew them apart to see them. This isn't the right thing
because I used the full vertex array for each mesh. Not to mention
just how long it would take to do the bone assignments and weighting
over 22 vertex replicas. It takes 2 minutes to do the mesh as a single
entity so over a half an hour this way. Back under the rock to do some
more thinking. :book:

[Aymar de Bois Mauri]

@Aymar de Bois Mauri,

Thank you, I think you have succinctly explained why what I was trying
to do was so hard: it is, in fact, impossible. In a mesh file you have one
large vertex array but (take armored_sergeants for example) 22 face groups
which index into that one vertex array. In Milkshape you have the same
large single vertex array, a single large face array, and then an auxilliary
set of 22 "groups" that index into the large face array to get the grouping
structure. What I'm hearing is that in Max I need 22 little vertex arrays
accompanied by 22 little face arrays indexing into their separate dedicated
vertex arrays to form the same groups. In other words 22 separate named
meshes.

Yikes, this is going to be an index housekeeping nightmare. Importing seems
straightforward because I know how to grab vertices by index and simply put
them into the 22 new little arrays, perhaps with duplication if the face
groups share a vertex. Exporting is the nightmare. I have to go through each
group in order, pull out the vertices and put into a single large array. Reindex
the faces to point to the same vertex which now has a different index
in the large array, AND remove duplicate entries, all the while keeping the
indexing straight. :dizzy2: I'm going to crawl back under my rock and think
about this for a while.Well, that is disturbing to know. An frankly quite an undertaking to cope with. :sad:

I'll try to get help from Salinoc, a new EB member who is a professional 3D artist. Mind though that he mainly works in Maya, so he might not be able to explain all of the functioning of the MAX engine.

[Aymar de Bois Mauri]

Ok, back from under my rock. This is just a gmax hack for proof of
concept but if I do like Aymar said and make 22 editable meshes
I do get groups:

Here I just blew them apart to see them. This isn't the right thing
because I used the full vertex array for each mesh. Not to mention
just how long it would take to do the bone assignments and weighting
over 22 vertex replicas. It takes 2 minutes to do the mesh as a single
entity so over a half an hour this way. Back under the rock to do some
more thinking. :book::dizzy2:

[KnightErrant]

Ok, point taken, no more babbling about indices.
Upside, Aymar's suggestion seems to be the way
to go. Bwian has tried out the import part and this
seems to work albeit some problems in the weighting
and bone assignment parts for some units. The skin
modifier is trickier to use than first glimpse from the
documentation. The backwards part, meaning exporter,
is sort of complicated. GMAX doesn't support the writing
of ASCII or binary files, I assume so people will buy 3dsmax,
so I can't test the export code. Doing some in situ testing
of reassembling the data in the listener.

I didn't say this earlier but all this is from working off of
Vercingetorix's script which is amazing; I wish I actually
could understand it all. So verify the output matches
the input, try and get animation data in, and resolve
the other problems seen with weighting and bone
assignments. I really had no other plans this weekend.:lam:

[Salinoc]

Okay, all of this is quite confusing for me. KnightErrant, the "22 face groups" you are talking about... are they Smoothing groups, telling the engine witch edge will be hard and whitch one will be smoothy ? Or are they "simple" groups, whose only purpose is to help the artist to manage his model ?
If it is the 2d answer, can't we get rid of it and simply export a single mesh ? And if it is the 1th answer, maybe another way to export smoothing groups can be found ?

Sorry if my questions seems dumbs, maybe I misundertand the problematic.

[Aymar de Bois Mauri]

Ok, point taken, no more babbling about indices.
Upside, Aymar's suggestion seems to be the way
to go. Bwian has tried out the import part and this
seems to work albeit some problems in the weighting
and bone assignment parts for some units. The skin
modifier is trickier to use than first glimpse from the
documentation. The backwards part, meaning exporter,
is sort of complicated. GMAX doesn't support the writing
of ASCII or binary files, I assume so people will buy 3dsmax,
so I can't test the export code. Doing some in situ testing
of reassembling the data in the listener.

I didn't say this earlier but all this is from working off of
Vercingetorix's script which is amazing; I wish I actually
could understand it all. So verify the output matches
the input, try and get animation data in, and resolve
the other problems seen with weighting and bone
assignments. I really had no other plans this weekend.:lam:This might be a stupid and superfluous suggestion, but have you tried to contact Vercingetorix for some pointers about the programming routines of his import/export .cas plug in? Since he also knows pretty well how to model in MAX he could give you a double perspective (Modeller/Programmer) on the problems at hand. He hasn't posted lately due to his univ work taking too much time out of him.

[alpaca]

This might be a stupid and superfluous suggestion, but have you tried to contact Vercingetorix for some pointers about the programming routines of his import/export .cas plug in? Since he also knows pretty well how to model in MAX he could give you a double perspective (Modeller/Programmer) on the problems at hand. He hasn't posted lately due to his univ work taking too much time out of him.
Verc wrote that script very long ago. Experience tells me that he probably won't remember what he did back then and would have to go through the whole thing again himself. If I judge him correctly he'd probably do that but I don't know if you want to dump that kind of work on him :sweatdrop:

[KnightErrant]

The groups from the mesh aren't smoothing groups. Grumpy's converter
to ms3d just assigns everybody to Milkshape's smoothing group 1. I think
they are used by modellers to organize their work and they are used by the
game engine to randomize the textures so units have varying uniforms.
That, I believe, is why armoured_sergeants have head_01, head_02, etc
legs_01, legs_02, etc so that each can get different textures. (I may be
a little simplistic in this, I'm basing it on answers I've seen modellers give
in other threads.)

I think I'm making some progress. Bwian said some of the bone assignments
were off or missing and I think I've tracked it down to: If you have a
primary bone Id and secondary bone Id from the mesh but one of them
has a weight of 0, gmax (and I assume 3dsmax) won't let you assign both.
Makes sense of course, why assign a second bone with a 0 weight. But if
you want a 1-1 converter that goes back identically you can assign a weight
of 10^(-5) to the secondary bone and it accepts that. On back conversion
to ms3d the weight is converted back to a percentage byte and gets
rounded back to zero. Voila! 1-1 conversion. (Well, hopefully, I can't test
back conversion because of a little snag, gmax doesn't allow writing ASCII
or binary files.)

I've gotten to importing .ms3d files with animation in them. The problem
here is trying to put in the rotations using coordsys parent (3dsmax guys
should recognize this). This doesn't seem to do the correct thing in
MaxScript so I've switch to doing the anims in the bone node's own system.
When you do that you have to put in differences from the previous rotation
because the node frame keeps changing. This seems to work and I've gotten
a walk anim to work except the guy keeps twirling his spear. Here's what
bone_relbow is doing in Milkshape:


-17.1379418352 +77.8505712919 +7.4188831746
-13.0322084034 +82.0132164212 +12.0869657175
+4.5573767025 +86.4417539847 +30.3697462954
+121.1194986650 +86.2430774421 +148.2408411421
+137.3846516073 +81.3850985647 +165.5491688763
+140.9322245415 +79.2293337683 +168.3188379055
+143.1019024332 +78.7743270564 +168.9871172743
+143.0572603168 +80.3170140729 +167.4641899950
+140.8697556313 +82.5764338201 +163.9600570640
+136.4443121435 +84.4876505235 +158.6667697403
+133.3265903346 +85.3063579785 +155.2575491179
+141.0409065115 +82.9760545281 +163.7816935040
+144.7645070810 +79.9915009174 +168.6940748381
+145.8186648171 +77.9147819003 +170.7855060827
+146.0272041528 +77.6578438441 +171.3875735977
+144.7116687375 +81.5825934846 +168.7966642794
+119.5874325927 +87.5736051226 +142.6517793065
-7.6112588676 +84.6621072153 +16.0893011844
-18.1540554176 +76.9523740953 +6.9879084832

See how the x and z rotation angles jump from line 3 to line 4 and then
back down again near the last two frames. Milkshape handles this fine
but gmax is somehow computing the wrong quaternion and making the spear
twirl at these frames. This is probably due to me using differences and
confusing it. Will investigate this a little more.

[KnightErrant]

Last bit of the puzzle. Werner Von Braun said (just from memory)
"Basic research is what I'm doing when I don't know what I'm doing."
Under that kind definition, I do a LOT of basic research.:embarassed:
Wasted a lot of time investigating the different transformations
between eulers and quats. GOM would have pointed out this
salient feature:

^ y
Milkshape: |
|
|_________> x
/
/
/ z


Gmax: ^ z
| / y
| /
|/________> x

It's just a -90 degrees rotation about the x-axis. If I just use Grumpy's
algorithm to compute quats from eulers and apply the
matrix

( 1 0 0 )
( 0 0 1 )
( 0 -1 0 )

to the spatial part of the quat I get the rule

[ q1, q2, q3, q4 ] ---> [ q1 q3 -q2 q4 ]

and the animations play perfectly upon import. Still have to
use coordsys node and take differences but no twirly spears and
the back conversion reproduces the original Milkshape euler angles.

Just have to clean up the mess left by iterating over the geometry
in the scene (basically all the objects get duplicated with names
like head_05). I think this is one of those "should I use 'instance',
'reference' or just assignment" things the docs talk about but I haven't
read carefully. I want a variable that lets me
access the node information saved in a file that I can still delete afterwards
without wiping out the original information.

[SigniferOne]

KE, speaking of all the turning, I wrote in your anims thread a while ago that there was a problem with exporting modified "turn" animations. When you're satisfied that everything works properly, could you take just a sec to try import, modify, export a turning animation and load it up in the game. Soldiers continue turning back and forth endlessly last I tried it.

[KnightErrant]

@SigniferOne
Apologies, I've been very lazy of late; I haven't monitored my threads
like I should. Could you post a link to get me to it? I'm interested of course
if something is wrong and also if it hints at anything that is causing problems
with Bwian's offset mesh chariot animations, because, frankly I'm stumped at
this point. Gone backwards and forwards with "it's an animations problem,
it's a meshconversion problem, it's an animations problem" but we can't nail
anything down. Maybe this would hint at something.

Just a question, you're not seeing "bad animations", just animations that for
some reason play more than they should, i.e. turning and turning?

[SigniferOne]

KE here, last page:

https://forums.totalwar.org/vb/showthread.php?t=83063&page=9

It seems turning animations have a problem importing/exporting.

But please don't worry about being lazy. If anyone deserves to be lazy it's you and the grumpy grandfather. I mean old man. My biggest concern is about you burning out mid-way, that's why my biggest interest is that you take your time, and that we periodically provide you with motivation and inspiration from the results of what your work helped us accomplish. For instance I've given you big accollades in my newly released animations pack (here (http://www.twcenter.net/forums/showthread.php?t=127708)).

[KnightErrant]

@SigniferOne
Thanks for the accolades! Congrats on the anims, looks
like people are really pleased with the improvements. Also,
thanks for the link, I undrstand now, I abandoned my original
lthread. Actually happy to have a problem with a definite
error message; this I can deal with. As regards burn-out,
I think this was true in July/August but not from modding for
the game; more just summer real world stuff. Working with
Bwian on the gmax/3dsmax import/export stuff really has me
going strong again.

(Wow, just missed it, Boston just took Game 2, go Sox!)

Haven't thought through this, but now that the importer/exporter
is looking better i.e. might just work, doing a importer/exporter
for .world files doesn't look too bad. The Python reader taught
me enough to find the mesh data for the settlements
meaning the structures for northern european villages, towns,
and large towns. This could be adapted to write out
a .ms3d-like binary file for a chosen structure. This could
then be pulled into 3dsmax since it has: verts, faces, uv coords,
lighting (my worry about Milkshape, it can't do this but
3dsmax can; have to learn this), animations (don't know what
Caliban meant about this, I can ask again at the right time, but I
assume this is maybe a particle system.)

This isn't the same as adding whole new structures, but maybe learning
how to change the existing structures might lead to good questions for
Caliban on how to do more with their world files. Right now, one problem
at a time.~;)

[KnightErrant]

Update:

Sorry, not an update on settlements but we will be getting
back to them soon. Bwian did a test yesterday and we've
been able to do a .ms3d model into 3ds Max and back out
again to .ms3d. The model wasn't animated, just the regular
basepose. 2 more tests needed, (1) back and forth conversion
of a model with animations and (2) Bwian's custom models won't
convert. I suspect custom models don't have the null animation
frames the converters (GOM's and mine) put in and I've borked
the process by assuming something is defined that isn't. If this
is cleared up in a day or two, we'll put out an alpha. This will
also work for Gmax users, it's just that the export will be to the
listener. Gmax guys will have to cut and paste the ASCII output
to a .txt file and then run a Python utility to convert to the .ms3d
file format. (This is how I tested things.)

A murrain upon the Discreet guys who made the clipboard buffer
10 Kbytes in size:furious3: . Not so bad for a lod3 but for a lod0 go get a
cup of coffee and cut/paste, cut/paste, .. cut/paste for 15 minutes
or so. I know, if its free you can't really complain.

[Bwian]

Got a new version in the mail from KE today, anbd it's a big step forward.

Imports an MS3D file neatly into Max, with all groups, rigging, UV and comments intact. This includes the messaed about with skeletons in the Dwarves etc. so it is now very tolerant of hacked about stuff!

It exports too, and the resulting object can be opened up safely in MS3D. There is a minor issue with rigging, where it turns all the vertices 'green' to indicate a zero value, rather than the red fully asisgned verts ( looks like a side effect of some work KE had to do to make it accept the slightly rougher models made with the tools we work with! )

You also need to re-assign the materials, but this is no problem at all since the UV mapping is perfectly OK and it only takes a click to have it back again.

Next step will be to get the animations to import/export as well, and we have a major new weapon in the arsenal.

Then, we can get the settlement files doing the same :2thumbsup:

:balloon2: :balloon2: :balloon2: hooray :balloon2: :balloon2: :balloon2:

[SigniferOne]

Excellent, it's going great!

[KnightErrant]

Yes, I think we are very close. Tested animated camels and elephants
last night. There was a oopsy on elephants not in the script but in
the Python converter from text to binary for Gmax users. (Elephant
bones have spaces in their names, I forgot this and wasn't parsing the
names correctly in the joints section for making the .ms3d file.) I think
we just have to sort out the anomaly Bwian is seeing with animated horses and
we are there. The only known deficiency is I can't handle animated
siege engines yet. Siege engines contain their own texture file paths
unlike other units and I store these paths in the joint comments.
Unfortunately, I also store the .cas header, hierarchy, and footer for
animations in the same place so there's a conflict there. I'm going to ignore
this for now and try to fix it up later.

There is also an issue about Gmax making a few extra bone assignments.
If Bwian doesn't see this as well it means its a Gmax "feature" that is fixed
in 3ds Max. (What happens is I do the usual primary and secondary bone
assignments on import. On export from Gmax a few random vertices end up
having three bone assignments, for some reason usually to one of the
weapon or shield bones. Oftentimes this is totally innocuous because
the weight on the third bone is zero but it is still worrisome that Gmax is
doing this.:no: )

[KnightErrant]

I sometimes post good news/bad news sorts of things
but this is just a good news post. Yes, Bwian saw anomalous
bone assignments in a unit but I think this is solved.
I was looking through the Max Script documentation for
how to delete vertex assignments to correct the
anomalous assignments but saw this by serendity:

Edit: Apologies, serendity is usually spelled as serendipity.


skinOps.ReplaceVertexWeights <Skin> <vertex_integer> \

( <vertex_bone_integer> | <vertex_bone_array> ) \

( <weight_float> | <weight_array> )

Sets the influence of the specified bone(s) to the specified vertex. Any influence weights for the bone(s) that are not specified are erased. If the bones and weights are specified as arrays, the arrays must be of the same size.



This is literally a one-liner solution to the extraneous bone
assignments problem. If Bwian has the same results for 3ds Max
as I've had for Gmax this is a done deal. The crux is: any influence for the
bones not specified is erased and the original assignments i.e. those from
the .ms3d file, are enforced. Tried this out and it works for Bwian's custom
skeleton model where problems were noted before. Will post again once this
is confirmed.

[SigniferOne]

KnightErrant, I hope you're not considering leaving us all in dire straits. Your work is beloved and needed!

[KnightErrant]

Hi SigniferOne,

Long time no see in this thread for me I know. Been pursuing
too many other things; the importer/exporter for Gmax and 3ds Max
was intended as the warm-up exercise for the importer/exporter
for settlements. That projected three-week project turned into three
months. Now I'm back looking at animations and making tools for
Gmax and 3ds Max. I'm trying to recover what I did back in June
for battlemap animations for the dwarves and the armored_sergeants
with tails. I remember I got it working, I just need to repeat the
experiments again to get the details down. A wordy way of saying
I do intend to take up settlements again once my to-do list gets shorter.

[SigniferOne]

It's all good to take breaks; modding sometimes can get too much. The most important thing as far as I'm concerned, is to not get burned out. So when I felt it was too much, I stepped away from Paeninsula Italica modding after November, but now am back in full swing to my Classical mod for Medieval (which is all thanks to you and GOM). So take care :)

[KnightErrant]

Absolutely, you are right about burn-out. My problem seems to
go much deeper. I haven't played the game since last
March, I just program. I'm not grousing, I love this. But
tonight I was testing how to put a
new Musketeers anim in game and I couldn't remember how to lower my
camera angle. My guys got shot up for two minutes before
the divide sign suggested itself. I find this funny, modders are
the worst players because we never play. The last time I tried
to play a campaign I couldn't even find Wales as the English.:laugh4:

[SigniferOne]

Numpad + and -!

Agree with you about the playing though, I haven't played a full campaign since the summer... and probably around March like you said. I can't wait till I put in my classical mod in; mostly I just load up the custom battle to test that the new batch of units is ready and working properly.

Whatever fun it is we get from the game, is good enough. I just hope you'll derive some further fun from all the other mods which will pour in with all the tools (Broken Crescent, middle eastern medieval mod with high quality units, seems the first contender so far).

[wolfslair]

You guys do some great work; I've spent a lot of time reading your posts, and learning much. It's fun to play a battle now and then too ;-)

[KnightErrant]

Edit: I like the wolfslair user Id.
You mean play a battle for fun, and not to test out a mod?
I'm not sure I'm familiar with this concept. :laugh4:
<\evil music Join us in the quest for modding glory, hah, hah, hah,
oops, damn, my teeth just fell out. \end evil music>
Thanks for reading our posts, in the end, if we weren't having
fun, we wouldn't do this.

[wolfslair]

What I'm interested in right now i the *..worldpathfinding and .worldvegetation files, with the object of making some of the ambient settlement models more useful in the game, and increasing game performance by reducing and or changing the vegetation level and models in certain areas.

Have you studied their structure any?

[KnightErrant]

Hi wolfslair,

Apologies, I haven't check the Wiki for awhile. Sorry, I've only looked
at the .world files, and only the north_european subdirectory at that,
so I don't know anything about the other files in the settlements folder.
I've followed as best I could the research done over at the TWC but don't
have anything to add. The buildings in the .world files mostly follow the
mesh format, i.e. verts, tris, normals, uvs, but there's always the two
mystery blocks I haven't been able to follow up on that prevent a converter.

[wolfslair]

Thanks for your replay, you've done some great research.

Right now it's the worldpathing file I'm interested in; and how the game reads from map.rwm to create battlemaps in a campaign, and on what basis the ambient scenery is generated and placed.

It looks like descr_landscape_ambient_settlements.txt is unfortunately just a holdover from RTW, I don't think the game reads from it at all when loading.

[SigniferOne]

KE see if you can give it a try again. Take for instance "horse_breeders_guild.world", a simpler file and not something like a whole town. There'll be less of 'animations', less of weird blocks, and a smaller file like that would be easier to grasp about what's going on. Consider giving it just another try.

[Roman_Man#3]

Any help to you guys?

http://www.twcenter.net/forums/showthread.php?t=148717

[ptr]

Anymore progress, sirs?