Settlements: File Formats; Take 2

[alpaca]

Hmm these line-break problems are pretty weird. I played around with your above post and if you cut the entries somewhere linebreaks seem to work.

Edit: All right, it seems the "[code]"-tags can't have more than 65 lines each (or there's a maximum number of letters), so you have to split very large ones up.
Don't ask me why, though, maybe the programmer used a fixed-length array.
I pretty arbitrarily split your post into 5-line-packages with 4 of your float entries and/or all the strings. It seems (if you interpreted the float/string stuff correctly) that this stuff usually consist of 17 floats and a granny string? 17 is really weird, being a prime number it doesn't make much sense for coordinates or anything unless one or two of the numbers mean something else

Edit2: Meh damn pattern-detecting brain... For the granny strings, a subdivision into packages of four seems to make most sense, too (skipping the very first one). The first seems to differ most, the second one is often 0, sometimes 1 or 2 and very rarely 3 - and the third and fourth are always 0 I think. So #1 could be an id or a link or something, #2 could be a flag with three options and the other two are unused (in this file).
I also read your post and you mentioned the 4x4 matrices
Translation matrix, huh. Makes sense in a way. The question is: What does the 17th float do?

[KnightErrant]

Yes, Thinking about it we've probably seen these 4x4 matrices before in
the mesh files. The book I've got on 3-D graphics has this to say for
homogeneous co-ordinates (but my words not the book's):

You can do rotations in 3D: R v = x
You can do scalings in 3D: Sv = y

but to do translations you have to do v = x + T

No longer a matrix multiplication but a vector add. Once you have adds and
multiplications you have a ring structure (abstract algebra) but not a group
structure (just adds or multiplications, depending on what you are doing.)
Rings have less known about them so groups are better. The cool part, and
the part I missed at university is that in projective geometry you can make
4x4 matrices like:

        (  R    Tvec)  ( v )    =   ( x )
         (   0    1    )  ( 1 )         ( y )

and make translations obey matrix multiplications and so be a group. (This
won't work out well in HTML but this is all I can do right now.) These are
called homogeneous co-ordinates and are probably known to all the modellers
\but where news to me. (My Linear Algebra course skimped on projective
geometry.)

I've seen these in the settlement files but, like you said, there's a 17th float
which doesn't fit (rotations and translations and then an overall scaling?).
At any rate, I have found some overall logic to the .world file. The header
says 40 so you can read the first part: that part defines 31 entries that
have a sequence number/ There happen to be 31 complexes so maybe
that is the magic number. Then come 867 entries. Total mystery except
the header/footers contain a count of the number of homogeneous data
segments but also contain a secondary count of somethingh else. At the
end that number turns out to be 867. Wish I knew what it meant.

I've been refactorizing the code and, except for these oddities, it is almost
suitable for putting in for loops. Maybe a little more study will unlock the
inexplicable. (Or make me buy an X-box: Mario bros. was entertaining when I
played it against my daughter.)

Apologies, I should be fixing animation difficulties but I can't do
that at lunch time; unemployment would result if I installed the game.

@GOM, I've copied almost all of that interesting chapter, just a few pages
more. Sorry, been kinda' rattled with contractors and getting lawn stuff done.

Regards,

KE

[alpaca]

Thanks for the explanation, I already looked it up on wikipedia.

Scaling might make sense, since we're dealing with pre-defined buildings consisting of blocks it could be we have some groups here which have of course to be scaled uniformly (because otherwise they'd be distorted).

Ok looking at the file:

At the start we have some stuff I can't really identify in the header but you might know what it is (bytes 26-51), then we get a float followed by two 0-bytes and your number 40.
Following that, we have a four-byte 0 and then three floats again which might be coordinates.
After that we get 17 integers again (with some -1s in there which means it's probably settings of some sort) until we hit byte 147.
Then nine floats and 17 integers, again followed by 9 floats and so on. I guess these are the entries you described, but I count 40 of them. I'm pretty confident these describe something of our settlements, three points in each entry and followed by a number of integer-settings of some sort (a lot of which are disabled, i.e. -1 - or 2).
Rinse and repeat to byte 4229 where we get the 867 you speak of, followed by 867 almost identical entries (a lot of them are totally identical except for their "id")

Unfortunately I probably can't help you to any huge extent because I don't have that much time and you know the granny file formats much better than I do anyways.

[alpaca]

Hey KE, is any work still going on on this? It'd be a shame if this wasn't pursued further

[SigniferOne]

Indeed!!

[KnightErrant]

@alpaca and SigniferOne,

Many apologies, I've really dropped the ball on animation support and
posting anything new on settlements. Probably can't get really active
again until the yard stuff (don't ask) is finished, but here is where I am
on settlements. Made it through the village.world a few weeks ago.
Almost throught town.world lately. (Only getting stuff done at lunch time.)
Lots of new stuff in the town.world file, ArrowTowers, ArrowSlots, gates,
perimeters, etc. Don't know what any of it means yet but it all has to be
coded through. The good news is that I think I've seen all the granny variations
for settlement data. Without going into it here, 95% of the data in a .world
file is mesh models. They are organized as "complexes" with each complex
containing 1 to 8 maybe more "structures". These structures are a self
contained mesh with vertices, tris, normals, tangents, the other stuff
Caliban told me about like uv coords as cube normalized RGB values, and
animation byte quads (don't know yet what these are). Reading the complexes
has finally been reduced to a simple loop with no hard coding. Its all the
data at the end that is still incomplete. I'm sure if I try large_town I'll see
even more stuff I haven't seen yet but it gets easier each time I try another
world and there's only what, city, large_city, huge_city, to go. Even if they
require separate case statements for their own oddities, that isn't out
of the question.

Problem is, even if I leave out all the actual mesh data, the granny strings
and other data still run to ~0.5 Meg in a .txt file. This is probably too much
to post and ask for help on (and I haven't tried lately to see if I can post
code anyway). Maybe the best thing is to get town.world read all the
way through and then upload the Python code so people interested could run
it and see the .txt file output and then post comments on what the
data blocks at the end might mean. I've run into one very odd data structure
that just doesn't seem to have an entry that allows me to read it
programmatically; I have to just crunch through it until its over and then
hard-code it. Would really like some other eyeballs on this problem.

Nowhere near a converter, this is still just reading data to the end.
Hopefully, a converter would pick out a structure for a modeller to work
on and reconstruct the whole .world file with the new structure in place.
The worrisome part of that are the quad byte sections Caliban described
that relate to animations and to lighting sources; Milkshape doesn't have
this so this may be a 3dsmax only thing. Not ready to try this, but remember
when we were doing the mesh file for units and didn't know what the mystery
block were? I could try just putting in arbitrary values and see what happens.

(Reread that and realized it doesn't make any sense if you haven't been
living in these files. Better explanation follows.)
The quad byte blocks must have the same number of entries as the vertices.
Makes sense if they are vertex normals, tangents, or uv coords. If they
are animation or lighting entries, then that means the anims, for instance,
are tied to specific vertices. I'm thinking this is something like smoke from
chimneys. If you mod a structure by adding more vertices, how do you
make sure the smoke still comes out of a chimney and not the side of a
house because the vertices got renumbered? If you can't get the data
into Milkshape you would have to do something else. What that might
be I haven't got to.

That's the update..... Yep, this is taking awhile.

KE

[SigniferOne]

KE, much happiness on the progress, and love the turtle. FYI, the chimney smoke is defined in the chimney smoke text file (as regards the density of smoke etc). Maybe the animations could refer to windmills. But from what I remember, windmills don't turn, so maybe it's something CA wanted but later turned back on (which means you can just hardcode to paste it in, without debugging it). Either way, all is nice, hope the yard fixes itself. (yeah!)