Page 1 of 3 123 LastLast
Results 1 to 30 of 75

Thread: Settlements: File Formats; Take 2

  1. #1
    Member Member KnightErrant's Avatar
    Join Date
    Jan 2007
    Location
    Huntsville, Alabama USA
    Posts
    458

    Default Settlements: File Formats; Take 2

    Hi all,

    This is a continuation of a thread that had some formatting problems.
    Hope this thread doesn't. I'm going to try a quick test in this post,
    then post the real question. Here's a chunk of the settlement header,
    hopefully it will come out ok with linebreaks:

    Code:
    serialization::archive
    3 4 4 4
    264 0
    267 0 0 0
    2 0 184.550750732 0
    40 0 -7.09150409698 1.21208238602 -2.87348890305
    That's the test; I'll log out and log back in to see how it goes.

  2. #2
    Member Member KnightErrant's Avatar
    Join Date
    Jan 2007
    Location
    Huntsville, Alabama USA
    Posts
    458

    Default Re: Settlements: File Formats; Take 2

    Ok, that seems to work! Many thanks to alpaca and to TosaInu for running tests to see what was wrong. Here's where I am: The fun starts at where tell says t
    Code:
    ell says 3863224
    That's for the hexedit gurus: this is the byte location for where it starts getting interesting. There seems to be some 4x4 homogeneous coordinate matrices here before the float data starts. (I have been reading up on this.) Here's the float data: Edit: Line-Break test by alpaca:
    Code:
    +1.0000000000  +0.0000000000  +0.0000000000  +0.8681725860   
    +0.0000000000  +1.0000000000  +0.0000000000  -2.2101776600   
    +0.0000000000  +0.0000000000  +1.0000000000  +1.2806341648   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +1.0000000000 1 0 0 1 0 0 0 0 0 57 0 0 0 0 0 4 0 0 0 2 0 0 0 0 0 0 0 3 0 0 0 57 0 0 0  
    +1.0000000000  +0.0000000000  +0.0000000000  +0.1207747832   
    +0.0000000000  +1.0000000000  +0.0000000000  +0.4997914732   
    +0.0000000000  +0.0000000000  +1.0000000000  +1.9928042889   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +1.0000000000 0 0 0 0 0 84 0 0 0   
    +1.0000000000  +0.0000000000  +0.0000000000  +0.8747580051   
    +0.0000000000  +1.0000000000  +0.0000000000  +0.8698613048   
    +0.0000000000  +0.0000000000  +1.0000000000  -1.4081792831   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +1.0000000000 0 0 0 0 0 111 0 0 0   
    +1.0000000000  +0.0000000000  +0.0000000000  -0.1258755475   
    +0.0000000000  +1.0000000000  +0.0000000000  -0.1756218821   
    +0.0000000000  +0.0000000000  +1.0000000000  -3.3048241138   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +1.0000000000 0 0 0 0 0 2 0 0 0 0 0 0 0 1 0 0 0 25 0 0 0   
    +1.0000000000  +0.0000000000  +0.0000000000  -0.9229867458   
    +0.0000000000  +1.0000000000  +0.0000000000  -4.0490546227   
    +0.0000000000  +0.0000000000  +1.0000000000  +1.6631650925   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +1.0000000000 1 2 0 0 0 200 1 0 0 49 0 0 0 1 2 0 0 66 0 0 0 2 0 0 0 0 0 0 0 1 0 0 0 139 0 0 0    
    +1.0000000000  +0.0000000000  +0.0000000000  +0.5850151777   
    +0.0000000000  +1.0000000000  +0.0000000000  -1.8052126169   
    +0.0000000000  +0.0000000000  +1.0000000000  +0.7539926171   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +0.6999999881 1 8 0 0 0 202 1 0 0 50 0 0 0 204 1 0 0 51 0 0 0 2 2 0 0 67 0 0 0 3 2 0 0 68 0 0 0 21 2 0 0 76 0 0 0 31 2 0 0 78 0 0 0 203 2 0 0 129 0 0 0 209 2 0 0 130 0 0 0 2 0 0 0 0 0 0 0 3 0 0 0 111 0 0 0   
    +1.0000000000  +0.0000000000  +0.0000000000  -0.1258755475   
    +0.0000000000  +1.0000000000  +0.0000000000  -0.1756218821   
    +0.0000000000  +0.0000000000  +1.0000000000  -3.3048241138   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +1.0000000000 0 0 0 0 0 84 0 0 0   
    +1.0000000000  +0.0000000000  +0.0000000000  +0.8747580051   
    +0.0000000000  +1.0000000000  +0.0000000000  +0.8698613048   
    +0.0000000000  +0.0000000000  +1.0000000000  -1.4081792831   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +1.0000000000 0 0 0 0 0 57 0 0 0   
    +1.0000000000  +0.0000000000  +0.0000000000  +0.1207747832   
    +0.0000000000  +1.0000000000  +0.0000000000  +0.4997914732   
    +0.0000000000  +0.0000000000  +1.0000000000  +1.9928042889   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +1.0000000000 0 0 0 0 0 2 0 0 0 0 0 0 0 1 0 0 0 25 0 0 0   
    +1.0000000000  +0.0000000000  +0.0000000000  +0.8681725860   
    +0.0000000000  +1.0000000000  +0.0000000000  -2.2101776600   
    +0.0000000000  +0.0000000000  +1.0000000000  +1.2806341648   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +1.0000000000 1 1 0 0 0 68 2 0 0 91 0 0 0 2 0 0 0 0 0 0 0 1 0 0 0 157 0 0 0   
    +1.0000000000  +0.0000000000  +0.0000000000  -0.2069556862   
    +0.0000000000  +1.0000000000  +0.0000000000  -2.0798547268   
    +0.0000000000  +0.0000000000  +1.0000000000  -1.6279093027   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +1.0000000000 1 1 0 0 0 69 2 0 0 92 0 0 0 2 0 0 0 0 0 0 0 3 0 0 0 111 0 0 0   
    +1.0000000000  +0.0000000000  +0.0000000000  +0.1647853851   
    +0.0000000000  +1.0000000000  +0.0000000000  +0.2269621640   
    +0.0000000000  +0.0000000000  +1.0000000000  +2.0150504112   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +1.0000000000 0 0 0 0 0 84 0 0 0   
    +1.0000000000  +0.0000000000  +0.0000000000  -1.0282818079   
    +0.0000000000  +1.0000000000  +0.0000000000  +0.8543335795   
    +0.0000000000  +0.0000000000  +1.0000000000  -1.4318115711   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +1.0000000000 0 0 0 0 0 57 0 0 0
    Code:
    +1.0000000000  +0.0000000000  +0.0000000000  +1.2263863087   
    +0.0000000000  +1.0000000000  +0.0000000000  -0.3336898088   
    +0.0000000000  +0.0000000000  +1.0000000000  -2.3126564026   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +1.0000000000 0 0 0 0 0 2 0 0 0 0 0 0 0 1 0 0 0 139 0 0 0
    +1.0000000000  +0.0000000000  +0.0000000000  +0.2344752252  
    +0.0000000000  +1.0000000000  +0.0000000000  -1.9429701567   
    +0.0000000000  +0.0000000000  +1.0000000000  +2.2158527374   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +0.8000000119 1 3 0 0 0 208 1 0 0 52 0 0 0 23 2 0 0 77 0 0 0 37 2 0 0 83 0 0 0 2 0 0 0 0 0 0 0 2 0 0 0 111 0 0 0   
    +1.0000000000  +0.0000000000  +0.0000000000  +0.2340932637   
    +0.0000000000  +1.0000000000  +0.0000000000  -0.0248921812   
    +0.0000000000  +0.0000000000  +1.0000000000  -3.7247326374   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +1.0000000000 0 0 0 0 0 84 0 0 0   
    +1.0000000000  +0.0000000000  +0.0000000000  -0.4714680314   
    +0.0000000000  +1.0000000000  +0.0000000000  -0.2671088576   
    +0.0000000000  +0.0000000000  +1.0000000000  +6.7157201767   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +1.0000000000 0 0 0 0 0 1 0 0 0 1 0 0 0 188 0 0 0   
    +1.0000000000  +0.0000000000  +0.0000000000  +0.5613649487   
    +0.0000000000  +1.0000000000  +0.0000000000  +3.7120764256   
    +0.0000000000  +0.0000000000  +1.0000000000  -0.0595130324   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +0.5000000000 0 0 0 0 0 2 0 0 0 0 0 0 0 1 0 0 0 25 0 0 0   
    +1.0000000000  +0.0000000000  +0.0000000000  +0.1719831526   
    +0.0000000000  +1.0000000000  +0.0000000000  -2.7414629459   
    +0.0000000000  +0.0000000000  +1.0000000000  -5.0158138275   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +1.0000000000 1 1 0 0 0 94 2 0 0 94 0 0 0 2 0 0 0 0 0 0 0 1 0 0 0 212 0 0 0   
    +1.0000000000  +0.0000000000  +0.0000000000  +0.8394250870   
    +0.0000000000  +1.0000000000  +0.0000000000  -3.9999992847   
    +0.0000000000  +0.0000000000  +1.0000000000  +1.1408995390   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +0.3000000119 1 5 0 0 0 101 2 0 0 95 0 0 0 102 2 0 0 96 0 0 0 124 2 0 0 105 0 0 0 16 3 0 0 170 0 0 0 17 3 0 0 171 0 0 0 2 0 0 0 0 0 0 0 1 0 0 0 139 0 0 0   
    +1.0000000000  +0.0000000000  +0.0000000000  +0.5850151777   
    +0.0000000000  +1.0000000000  +0.0000000000  -1.8052126169   
    +0.0000000000  +0.0000000000  +1.0000000000  +0.7539926171   
    +0.0000000000  +0.0000000000  +0.0000000000  +1.0000000000   
    +0.6999999881 1 1 0 0 0 104 2 0 0 97 0 0 0 2 0 0 0 0 0 0
    /Edit Doesn't this look like homogeneous coordinate data with weird granny bytes after it? I'm hoping GrumpyOldMan, Casuir, or zxiang1983 will take pity on me and find a pattern in the granny strings so I can go forward with some guidance but I can't see a pattern here. Hope this post comes out formatted alright. Many thanks for the interest in the subject If nothing else, it would be fun to make villages/towns etc. more tailored to the culture. Really, the interest is of course the fantasy mods with round doors for hobbit villages but better architecture for old (biblical, roman, etc.) mod types would be interesting too. Love this stuff , hope others do as well. KE Edit: Sigh, same problem, no line breaks....don't know what to do about this. I used Mozilla but I can also use IE or the latest Firefox. If anything occurs to people please let me know.
    Last edited by alpaca; 08-08-2007 at 21:20.

  3. #3
    Member Member KnightErrant's Avatar
    Join Date
    Jan 2007
    Location
    Huntsville, Alabama USA
    Posts
    458

    Default Re: Settlements: File Formats; Take 2

    Ok, for humour.

    Competititon announced. My birthday is tomorow.
    Cleverest entry wins; but no prize is announced.
    No credit for "Dude, you're really old" however, clever
    stuff like "Did you have hamburgers when you were
    growing up" is pretty good. You have to check the
    entry section to get the birthday announcements. If you
    get it wrong you don't get to play.

    Cheers,

    KE
    Last edited by KnightErrant; 08-04-2007 at 06:16.

  4. #4

    Default Re: Settlements: File Formats; Take 2

    KE, How did you find the Model T to handle when younger?

  5. #5
    feed me! Member Ashdnazg's Avatar
    Join Date
    Dec 2006
    Location
    Haifa, Israel
    Posts
    54

    Default Re: Settlements: File Formats; Take 2

    Yo KE, good sling job on that Goliath dude.
    Oh wait, confused you with your cousin, never mind then.
    a.k.a Lord hokomoko @ the Lordz Modding Collective

  6. #6
    Member Member KnightErrant's Avatar
    Join Date
    Jan 2007
    Location
    Huntsville, Alabama USA
    Posts
    458

    Default Re: Settlements: File Formats; Take 2

    Ok, only two entries in the competition but I liked them both.
    Model T handled well but it had nothing on my Stanley Steamer.
    Goliath was over-rated, taunting was good, a real clutch player
    with a sword but no shield control. If Ashur-Steinbrenner hadn't
    traded Marluk the Assyrian for Femnod the effeminant Babylonian
    he never would have made it out of the minors.

    Tried a few post previews using IE but the data came out all jumbled.
    I'll check out a couple of other options tomorrow.

  7. #7

    Default Re: Settlements: File Formats; Take 2

    Hi KE


    Quote Originally Posted by KnightErrant
    Ok, for humour.

    Competititon announced. My birthday is tomorow.
    Cleverest entry wins; but no prize is announced.
    No credit for "Dude, you're really old" however, clever
    stuff like "Did you have hamburgers when you were
    growing up" is pretty good. You have to check the
    entry section to get the birthday announcements. If you
    get it wrong you don't get to play.

    Cheers,

    KE
    I'm sorry I missed your birthday, I was away with the eldest at an athletics meet.

    As an act of contrition I do offer you a cyber present though, it's http://www.wsu.edu/~brians/errors/index.html , a web site with literally thousands of common errors (and corrections) in English. We both know that it's the natural state for a man of certain years to be pernickity and pedantic, and this is an admirable tool to assist in that regard .

    Back to the settlements, I honestly tried to look at the data you've posted but I get dizzy without line breaks. What are you copying from and pasting into? Sometimes there is a funny thing that happens with line breaks. Have you tried opening it in Notepad, saving it and then copying and pasting?

    From the settlement you've looked at, how many total verts and tris would be involved? Is it possible to set the individual models up as groups within a milkshape .ms3d? And positions based on bones/joints? Milkshape has the following limits:-

    #define MAX_VERTICES 65534
    #define MAX_TRIANGLES 65534
    #define MAX_GROUPS 255
    #define MAX_MATERIALS 128
    #define MAX_JOINTS 128

    Cheers

    GrumpyOldMan

  8. #8
    Member Member KnightErrant's Avatar
    Join Date
    Jan 2007
    Location
    Huntsville, Alabama USA
    Posts
    458

    Default Re: Settlements: File Formats; Take 2

    Hi GrumpyOldMan,

    Thanks for the web site and for trying to look at the posts without
    line breaks. Kind of reminds me of the vanilla battle_models.modeldb
    file. How did people actually mod it that way?

    Two thoughts on my problem: Easy way/hard way. (1) Maybe I just
    can't start a thread. Solution, hijack an old wiki thread and do settlements
    postings there. (2) Put it all in a pdf and just upload it. I'd rather do (1)
    because (2) is too hard.

    Good news though! I reached the end of the smallest village .world file today.
    I just programmatically hard coded through all the weird grannies near the
    end that were driving me crazy. ("Programmatically hard coded" is the worst
    oxymoron I've ever made up but what I mean is, rather than just put in the right
    byte numbers for the grannies between data, I kept making up rules until I
    got through the section just by doing a for loop.) I won't try to post anything
    because I'm basically where you were when you posted the mesh file stuff
    in February, I know where the data IS, I just don't know what all the stuff
    around it is. Also, I've made bad assumptions in the early part about what
    bytes should be integers/shorts/ etc that I should probably re-think.

    Without posting data and recalling Caliban's input the upshot is:
    Settlements contains vertex and triangle data that can be pulled out.
    The texture file paths are in there like siege weapons. No bone strings but
    there are other strings I'm not sure of (near the end). Caliban gave indications
    that some of the quad bytes sections are lighting models and animation
    place holders. This might be a problem as this doesn't mesh with Milkshape's
    allowed data.

    What I'd like to do next is just re-look at the data and grannies to make
    sure the file provides numbers to read it better; but not try to make sense
    of the header/footers yet (always too hard to start with). The data beyond
    just the settlement meshes seems to be 4x4 homogeneous coordinate
    affine transformations which I would not have recognized before.

    Without thinking further, at least it might be possible to modify what's there
    but not add new stuff (don't know what would have to change with the grannies). I.e. you have 30 and only 30 village types to choose from, BUT,
    you can make those 30 village types anything you want. Fingers crossed
    that that might be true.

    Regards,

    KE

  9. #9
    Harbinger of... saliva Member alpaca's Avatar
    Join Date
    Aug 2003
    Location
    Germany
    Posts
    2,767

    Default Re: Settlements: File Formats; Take 2

    Hmm these line-break problems are pretty weird. I played around with your above post and if you cut the entries somewhere linebreaks seem to work.

    Edit: All right, it seems the "[code]"-tags can't have more than 65 lines each (or there's a maximum number of letters), so you have to split very large ones up.
    Don't ask me why, though, maybe the programmer used a fixed-length array.
    I pretty arbitrarily split your post into 5-line-packages with 4 of your float entries and/or all the strings. It seems (if you interpreted the float/string stuff correctly) that this stuff usually consist of 17 floats and a granny string? 17 is really weird, being a prime number it doesn't make much sense for coordinates or anything unless one or two of the numbers mean something else

    Edit2: Meh damn pattern-detecting brain... For the granny strings, a subdivision into packages of four seems to make most sense, too (skipping the very first one). The first seems to differ most, the second one is often 0, sometimes 1 or 2 and very rarely 3 - and the third and fourth are always 0 I think. So #1 could be an id or a link or something, #2 could be a flag with three options and the other two are unused (in this file).
    I also read your post and you mentioned the 4x4 matrices
    Translation matrix, huh. Makes sense in a way. The question is: What does the 17th float do?
    Last edited by alpaca; 08-08-2007 at 21:44.

  10. #10
    Member Member KnightErrant's Avatar
    Join Date
    Jan 2007
    Location
    Huntsville, Alabama USA
    Posts
    458

    Default Re: Settlements: File Formats; Take 2

    Yes, Thinking about it we've probably seen these 4x4 matrices before in
    the mesh files. The book I've got on 3-D graphics has this to say for
    homogeneous co-ordinates (but my words not the book's):

    You can do rotations in 3D: R v = x
    You can do scalings in 3D: Sv = y

    but to do translations you have to do v = x + T

    No longer a matrix multiplication but a vector add. Once you have adds and
    multiplications you have a ring structure (abstract algebra) but not a group
    structure (just adds or multiplications, depending on what you are doing.)
    Rings have less known about them so groups are better. The cool part, and
    the part I missed at university is that in projective geometry you can make
    4x4 matrices like:

    Code:
            (  R    Tvec)  ( v )    =   ( x )
             (   0    1    )  ( 1 )         ( y )
    and make translations obey matrix multiplications and so be a group. (This
    won't work out well in HTML but this is all I can do right now.) These are
    called homogeneous co-ordinates and are probably known to all the modellers
    \but where news to me. (My Linear Algebra course skimped on projective
    geometry.)

    I've seen these in the settlement files but, like you said, there's a 17th float
    which doesn't fit (rotations and translations and then an overall scaling?).
    At any rate, I have found some overall logic to the .world file. The header
    says 40 so you can read the first part: that part defines 31 entries that
    have a sequence number/ There happen to be 31 complexes so maybe
    that is the magic number. Then come 867 entries. Total mystery except
    the header/footers contain a count of the number of homogeneous data
    segments but also contain a secondary count of somethingh else. At the
    end that number turns out to be 867. Wish I knew what it meant.

    I've been refactorizing the code and, except for these oddities, it is almost
    suitable for putting in for loops. Maybe a little more study will unlock the
    inexplicable. (Or make me buy an X-box: Mario bros. was entertaining when I
    played it against my daughter.)

    Apologies, I should be fixing animation difficulties but I can't do
    that at lunch time; unemployment would result if I installed the game.

    @GOM, I've copied almost all of that interesting chapter, just a few pages
    more. Sorry, been kinda' rattled with contractors and getting lawn stuff done.

    Regards,

    KE

  11. #11
    Harbinger of... saliva Member alpaca's Avatar
    Join Date
    Aug 2003
    Location
    Germany
    Posts
    2,767

    Default Re: Settlements: File Formats; Take 2

    Thanks for the explanation, I already looked it up on wikipedia.

    Scaling might make sense, since we're dealing with pre-defined buildings consisting of blocks it could be we have some groups here which have of course to be scaled uniformly (because otherwise they'd be distorted).

    Ok looking at the file:

    At the start we have some stuff I can't really identify in the header but you might know what it is (bytes 26-51), then we get a float followed by two 0-bytes and your number 40.
    Following that, we have a four-byte 0 and then three floats again which might be coordinates.
    After that we get 17 integers again (with some -1s in there which means it's probably settings of some sort) until we hit byte 147.
    Then nine floats and 17 integers, again followed by 9 floats and so on. I guess these are the entries you described, but I count 40 of them. I'm pretty confident these describe something of our settlements, three points in each entry and followed by a number of integer-settings of some sort (a lot of which are disabled, i.e. -1 - or 2).
    Rinse and repeat to byte 4229 where we get the 867 you speak of, followed by 867 almost identical entries (a lot of them are totally identical except for their "id")

    Unfortunately I probably can't help you to any huge extent because I don't have that much time and you know the granny file formats much better than I do anyways.

  12. #12
    Harbinger of... saliva Member alpaca's Avatar
    Join Date
    Aug 2003
    Location
    Germany
    Posts
    2,767

    Default Re: Settlements: File Formats; Take 2

    Hey KE, is any work still going on on this? It'd be a shame if this wasn't pursued further

  13. #13

    Default Re: Settlements: File Formats; Take 2

    Indeed!!

  14. #14
    Member Member KnightErrant's Avatar
    Join Date
    Jan 2007
    Location
    Huntsville, Alabama USA
    Posts
    458

    Default Re: Settlements: File Formats; Take 2

    @alpaca and SigniferOne,

    Many apologies, I've really dropped the ball on animation support and
    posting anything new on settlements. Probably can't get really active
    again until the yard stuff (don't ask) is finished, but here is where I am
    on settlements. Made it through the village.world a few weeks ago.
    Almost throught town.world lately. (Only getting stuff done at lunch time.)
    Lots of new stuff in the town.world file, ArrowTowers, ArrowSlots, gates,
    perimeters, etc. Don't know what any of it means yet but it all has to be
    coded through. The good news is that I think I've seen all the granny variations
    for settlement data. Without going into it here, 95% of the data in a .world
    file is mesh models. They are organized as "complexes" with each complex
    containing 1 to 8 maybe more "structures". These structures are a self
    contained mesh with vertices, tris, normals, tangents, the other stuff
    Caliban told me about like uv coords as cube normalized RGB values, and
    animation byte quads (don't know yet what these are). Reading the complexes
    has finally been reduced to a simple loop with no hard coding. Its all the
    data at the end that is still incomplete. I'm sure if I try large_town I'll see
    even more stuff I haven't seen yet but it gets easier each time I try another
    world and there's only what, city, large_city, huge_city, to go. Even if they
    require separate case statements for their own oddities, that isn't out
    of the question.

    Problem is, even if I leave out all the actual mesh data, the granny strings
    and other data still run to ~0.5 Meg in a .txt file. This is probably too much
    to post and ask for help on (and I haven't tried lately to see if I can post
    code anyway). Maybe the best thing is to get town.world read all the
    way through and then upload the Python code so people interested could run
    it and see the .txt file output and then post comments on what the
    data blocks at the end might mean. I've run into one very odd data structure
    that just doesn't seem to have an entry that allows me to read it
    programmatically; I have to just crunch through it until its over and then
    hard-code it. Would really like some other eyeballs on this problem.

    Nowhere near a converter, this is still just reading data to the end.
    Hopefully, a converter would pick out a structure for a modeller to work
    on and reconstruct the whole .world file with the new structure in place.
    The worrisome part of that are the quad byte sections Caliban described
    that relate to animations and to lighting sources; Milkshape doesn't have
    this so this may be a 3dsmax only thing. Not ready to try this, but remember
    when we were doing the mesh file for units and didn't know what the mystery
    block were? I could try just putting in arbitrary values and see what happens.

    (Reread that and realized it doesn't make any sense if you haven't been
    living in these files. Better explanation follows.)
    The quad byte blocks must have the same number of entries as the vertices.
    Makes sense if they are vertex normals, tangents, or uv coords. If they
    are animation or lighting entries, then that means the anims, for instance,
    are tied to specific vertices. I'm thinking this is something like smoke from
    chimneys. If you mod a structure by adding more vertices, how do you
    make sure the smoke still comes out of a chimney and not the side of a
    house because the vertices got renumbered? If you can't get the data
    into Milkshape you would have to do something else. What that might
    be I haven't got to.

    That's the update..... Yep, this is taking awhile.

    KE

  15. #15

    Default Re: Settlements: File Formats; Take 2

    KE, much happiness on the progress, and love the turtle. FYI, the chimney smoke is defined in the chimney smoke text file (as regards the density of smoke etc). Maybe the animations could refer to windmills. But from what I remember, windmills don't turn, so maybe it's something CA wanted but later turned back on (which means you can just hardcode to paste it in, without debugging it). Either way, all is nice, hope the yard fixes itself. (yeah!)

  16. #16
    feed me! Member Ashdnazg's Avatar
    Join Date
    Dec 2006
    Location
    Haifa, Israel
    Posts
    54

    Default Re: Settlements: File Formats; Take 2

    are there any flags blowing in the wind (or at least turning awkwardly)?
    a.k.a Lord hokomoko @ the Lordz Modding Collective

  17. #17
    Harbinger of... saliva Member alpaca's Avatar
    Join Date
    Aug 2003
    Location
    Germany
    Posts
    2,767

    Default Re: Settlements: File Formats; Take 2

    Did you check the animinstances file? I thought it would contain the animation data.

  18. #18
    Member Member KnightErrant's Avatar
    Join Date
    Jan 2007
    Location
    Huntsville, Alabama USA
    Posts
    458

    Default Re: Settlements: File Formats; Take 2

    Re: animinstances

    Here's what surfaced Friday. I've seen .anim and .animinstances in the
    village.world file. Friday I hit the same bit in the town.world file. Going to
    try and post the section here:


    Code:
    # ------------------------------------------
    #      Beginning of .anim strings Section   
    # ------------------------------------------
      0   0 
    nentries = 12
      0   0 
    ntrans = 3
    17 DamageTransition0
    106BlockSet/North_European/Animations/Wall_Animations/Town_Wooden_Palisade/palisade_straight_animation_a.anim
    17 DamageTransition1
    106BlockSet/North_European/Animations/Wall_Animations/Town_Wooden_Palisade/palisade_straight_animation_b.anim
    17 DamageTransition2
    106BlockSet/North_European/Animations/Wall_Animations/Town_Wooden_Palisade/palisade_straight_animation_c.anim
    ntrans = 1
    17 DamageTransition0
    119BlockSet/North_European/Animations/Tower_Town_Animations/Wooden_Palisade_Animations/palisade_arrow_tower_animation.anim
    ntrans = 1
    17 DamageTransition0
    124BlockSet/North_European/Animations/Tower_Ambient_Animations/Ambient_Stone_Tower_Animations/stone_tower_4m_E_animation_a.anim
    ntrans = 1
    17 DamageTransition0
    124BlockSet/North_European/Animations/Tower_Ambient_Animations/Ambient_Stone_Tower_Animations/stone_tower_4m_B_animation_a.anim
    ntrans = 1
    17 DamageTransition0
    124BlockSet/North_European/Animations/Tower_Ambient_Animations/Ambient_Stone_Tower_Animations/stone_tower_4m_G_animation_a.anim
    ntrans = 1
    17 DamageTransition0
    126BlockSet/North_European/Animations/Tower_Ambient_Animations/Ambient_Thatch_Tower_Animations/thatch_tower_4m_B_animation_a.anim
    ntrans = 1
    17 DamageTransition0
    126BlockSet/North_European/Animations/Tower_Ambient_Animations/Ambient_Thatch_Tower_Animations/thatch_tower_4m_G_animation_a.anim
    ntrans = 1
    17 DamageTransition0
    126BlockSet/North_European/Animations/Tower_Ambient_Animations/Ambient_Thatch_Tower_Animations/thatch_tower_8m_B_animation_a.anim
    ntrans = 1
    17 DamageTransition0
    126BlockSet/North_European/Animations/Tower_Ambient_Animations/Ambient_Thatch_Tower_Animations/thatch_tower_4m_E_animation_a.anim
    ntrans = 1
    17 DamageTransition0
    124BlockSet/North_European/Animations/Tower_Ambient_Animations/Ambient_Stone_Tower_Animations/stone_tower_4m_D_animation_a.anim
    ntrans = 1
    17 DamageTransition0
    126BlockSet/North_European/Animations/Tower_Ambient_Animations/Ambient_Thatch_Tower_Animations/thatch_tower_4m_K_animation_a.anim
    ntrans = 8
    17 DamageTransition0
    120BlockSet/North_European/Animations/Gate_Animations/Wooden_Palisade_gate_anims/NE_wooden_palisade_gate_damage_impact.anim
    17 DamageTransition1
    123BlockSet/North_European/Animations/Gate_Animations/Wooden_Palisade_gate_anims/NE_wooden_palisade_gate_damage_destroyed.anim
    14 DoorCloseAnim0
    112BlockSet/North_European/Animations/Gate_Animations/Wooden_Palisade_gate_anims/NE_wooden_palisade_gate_close.anim
    14 DoorCloseAnim1
    119BlockSet/North_European/Animations/Gate_Animations/Wooden_Palisade_gate_anims/NE_wooden_palisade_gate_damage_close.anim
    15 DoorImpactAnim0
    113BlockSet/North_European/Animations/Gate_Animations/Wooden_Palisade_gate_anims/NE_wooden_palisade_gate_impact.anim
    15 DoorImpactAnim1
    120BlockSet/North_European/Animations/Gate_Animations/Wooden_Palisade_gate_anims/NE_wooden_palisade_gate_damage_impact.anim
    13 DoorOpenAnim0
    111BlockSet/North_European/Animations/Gate_Animations/Wooden_Palisade_gate_anims/NE_wooden_palisade_gate_open.anim
    13 DoorOpenAnim1
    118BlockSet/North_European/Animations/Gate_Animations/Wooden_Palisade_gate_anims/NE_wooden_palisade_gate_damage_open.anim
    79 settlements/North_European/Settlements/Town/north_european_town_A.animInstances
    This stuff comes after all the "complex" and "structure" data (my made-up names). The animInstances file always comes last after the .anim files.
    Have not looked at any of these other than to see, yes, they are more
    binary files. If luck holds then they should have a structure like the .cas
    files but I don't know this.

    I'll just post this and see how the code tags work for me now. If not this will
    just be another code posting with no line breaks.

    Edit: Couldn't find a smiley with crossed fingers but this looked ok after
    posting. As usual, I'll log out and log back in.

    Edit2: Still working!!! Holy moly, I've got a posting jones working here.
    Too late tonight for a discussion but I'll shoot for tomorrow.
    Last edited by KnightErrant; 08-26-2007 at 05:43.

  19. #19
    Harbinger of... saliva Member alpaca's Avatar
    Join Date
    Aug 2003
    Location
    Germany
    Posts
    2,767

    Default Re: Settlements: File Formats; Take 2

    Yeah, you just have to keep in mind to not post more than 50 lines or so.

    Ok so the anim file apparently contains the damage and open door anims, what's in animinstances?

  20. #20
    Member Member KnightErrant's Avatar
    Join Date
    Jan 2007
    Location
    Huntsville, Alabama USA
    Posts
    458

    Default Re: Settlements: File Formats; Take 2

    Aah, so the 50 lines may have been the problem.

    Haven't done anything more, BUT life looks managable at work
    now. Will try to get to the end of towns and then comparing villages
    to towns may reveal something. Will look at animinstances next with
    maybe a check on large_towns to see what else may come into play.

    At this point why am I doing this?

    (1) Hundreds of modellers are waiting to mod settlements?
    Naw, hardly think so.

    (2) Tens of modellers are waiting to mod settlements?
    Naw, hardly think so.

    (3) Bwian is waiting to make Hobbit villages?
    Probably.

    (4) I've bit off more than I can chew and am just too embarrassed
    to admit it.
    Yeah, that one! That's exactly it!!!

    Sheer bloody-mindedness is a strong motivator.

    KE

  21. #21
    Harbinger of... saliva Member alpaca's Avatar
    Join Date
    Aug 2003
    Location
    Germany
    Posts
    2,767

    Default Re: Settlements: File Formats; Take 2

    Yeah I know the feeling. If it makes you happy though I'm also looking forward to edit settlements (I liked playing Lego and what Caliban told me about the process it's a lot like that, it only looks much cooler )

  22. #22
    Closet Celtophile Member Redmeth's Avatar
    Join Date
    Mar 2007
    Location
    Bucharest, Romania
    Posts
    3,740

    Default Re: Settlements: File Formats; Take 2

    I can't speak for everyone, but I for one and I think most of the EB team is also tremendously interested in the possibility of editing settlements.

  23. #23

    Default Re: Settlements: File Formats; Take 2

    KE, how can you say that!



    Imagine that in M2 engine.
    Last edited by SigniferOne; 08-28-2007 at 17:30.

  24. #24
    Member Member KnightErrant's Avatar
    Join Date
    Jan 2007
    Location
    Huntsville, Alabama USA
    Posts
    458

    Default Re: Settlements: File Formats; Take 2

    Ok, point taken. I am encouraged to keep going.
    Here's a link to the beta reader.

    http://www.twcenter.net/forums/downl...o=file&id=1578

    The two files in the zip are animationlibrary.py and settlementutilities.py.
    animationlibrary.py is almost the same as the one distributed for the
    animation utilites except I put a trap in the function getstring to jump
    back out if it gets a requested number of characters more than 1000.
    This usually means I'm reading the wrong int and it puts Python into a loop
    where it beeps every time it reads the bell code, very annoying to have
    happen at work!

    settlementutilities.py is the one to double-click on. Just put
    north_european_village_a.world and north_european_town_a.world
    in the same directory with the two Python files. Currently the .txt
    file doesn't have the actual mesh data output to it. If you want this
    just search for WRITEDATAFLAG and set it to True. The output file
    will be very large!!! You can cut and paste the vertices and tris
    to make an obj file. The u and v coordinates are also converted from
    RGB encoding into floats if you want them too.

    Now for numerology, I'll start. The village has 31 "complexes" containing
    a total of 98 "structures" or self contained meshes. The town has 30
    complexes with a total of 123 structures.

    Villages have a block of data at the top (after the 40 entries block of
    which 31 have a positive sequence number that should correspond to the
    complexes) that has 867 entries. This number shows up again near the
    bottom for another block of data.

    For towns the number is 1228.

    Question 1: What do these entries correspond to?

    For no good reason I kept a running total of the number of triangle groups
    defined in all the structures. For villages this number is 1274. Guess what,
    there are two blocks of data right at the end (after a block with 867 entries)
    that have 1274 entries. For towns this number is 2017 with the same
    correspondence. So, of course,

    Question 2: What do these entries do or is this a coincidence?

    There is a block of data preceded by this comment by me:
    # ------------------------------------------
    # Beginning of hard coded 17 floats but
    # variable ints with 5 byte signatures
    # ------------------------------------------
    It has 156 entries for villages and 173 entries for towns, BUT, nowhere
    can I find that number to read in order to tell me how to read this data.
    Right now it is just hard-coded in. This is very unusual for granny strings;
    they always tell you how to read themselves so I'm a bit perplexed on this
    one. The entries are 17 floats followed by what I read as 5 bytes being
    a signature, then followed by a variable number of ints which, by trial and
    error, I have made up rules for until I was able to read both cases, village
    and town. One entry looks like this:

    Code:
    +1.00000 +0.00000 +0.00000 -0.11595 +0.00000 +1.00000 +0.00000 -4.10875 +0.00000 +0.00000 +1.00000 -4.22434 +0.00000 +0.00000 +0.00000 +1.00000 +1.00000 sig:   1 0 0 2 0 , ints:   0 0 0 0 68 3 0 0 0 0 155 0 0 0 13 4 0 0 220 0 0 0 2 0 0 0 0 0 0 0 3 0 0 0 54 0 0 0
    If I rearrange the first 16 floats into a 4x4 matrix I get


    Code:
    +1.00000 +0.00000 +0.00000 -0.11595 
    +0.00000 +1.00000 +0.00000 -4.10875 
    +0.00000 +0.00000 +1.00000 -4.22434 
    +0.00000 +0.00000 +0.00000 +1.00000 
    
    +1.00000 sig:   1 0 0 2 0 , ints:   0 0 0 0 68 3 0 0 0 0 155 0 0 0 13 4 0 0 220 0 0 0 2 0 0 0 0 0 0 0 3 0 0 0 54 0 0 0
    In homogeneous coordinates this is a translation with no rotation.
    All of the entries seem to be translation only.

    Question 3: Why 156 and 173 and how can I figure out when I'm done reading
    this data block?

    This is probably enough for now. If anyone gets a chance to run the program
    and look at the grannies and data blocks please post back ideas.

    Edit: Aargh! Found a write out bug for villages. After all the complexes are read
    there are three strings for .worldTerrain etc. After that my code wrote out

    57 0 1 2 [4307] 0
    57 0 1 2 [4307] 0 0
    57 0 1 2 [4307] 0 0 0
    57 0 1 2 [4307] 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    56 0 1 0 [4308] 57 0 [4307] 0 0 255 255 255 255 0 0

    That's not what is in the file just this

    57 0 1 2 [4307] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    56 0 1 0 [4308] 57 0 [4307] 0 0 255 255 255 255 0 0

    I think my editor auto-indented the write statement to be inside the loop
    instead of after it, the Perils of Python.
    Last edited by KnightErrant; 08-28-2007 at 21:53.

  25. #25
    Harbinger of... saliva Member alpaca's Avatar
    Join Date
    Aug 2003
    Location
    Germany
    Posts
    2,767

    Default Re: Settlements: File Formats; Take 2

    Short notice before I leave (and after I return I'll have to do some learning so I probably won't get around to research anything more today):

    The 1274 4-byte entries you mention are in my opinion a kind of coordinate-transformation between IDs. They consist of a starting ID and a second int that specifies how many of this structure have that ID (usually 1 but can be 2 or more) so the transformation is between a block ID and a unique ID for each instance.
    In fact they're not 1274 entries but 1274 different unique IDs. They're only 867 entries (counting offsets and dividing by 4) and therefore coincide perfectly with your 867.

  26. #26

    Default Re: Settlements: File Formats; Take 2

    KE, I hope you haven't taken leave of us, and left us bereft unhappily in the dark!

  27. #27
    Harbinger of... saliva Member alpaca's Avatar
    Join Date
    Aug 2003
    Location
    Germany
    Posts
    2,767

    Default Re: Settlements: File Formats; Take 2

    Quote Originally Posted by SigniferOne
    KE, I hope you haven't taken leave of us, and left us bereft unhappily in the dark!
    I second, although I wouldn't blame you

  28. #28
    Member Member KnightErrant's Avatar
    Join Date
    Jan 2007
    Location
    Huntsville, Alabama USA
    Posts
    458

    Default Re: Settlements: File Formats; Take 2

    Hi SigniferOne and Alpaca,

    No, no, not at all. I've just been trying to go back to animations
    and catch up on some problems people have been having.
    TIKO_TWOW_RU has had a animation problem where somehow
    in Milkshape he got bones to have different numbers of animation
    frames and that killed the game, not suprisingly. Working on that
    revealed that the .cas to .txt utility doesn't convert degrees to radians
    so that's wrong. Hopefully, no one was using that. It's been an easy fix
    but haven't been doing settlements this last week.

    Started working on large_towns just to see what is new and the
    good news is the complexes and structures were read ALMOST
    without any new coding, but right after them (which is fortunately
    90% of the file) completely new data structures start unlike anything
    in villages or towns. This slows things down of course so I'll be working
    on this the coming week. I hope Kingdoms doesn't change things much;
    probably won't have anything near a converter till then. (I keep caveating
    this on being able to do something about the data blocks that Caliban said
    had to do with lighting and animations.) When we get to this. more information
    might make this worrisome bugaboo go away.

    Probably nothing to report for a week or so. However, any insights on the
    village or town data structures is very appreciated.

    KE

  29. #29
    Father of the EB Isle Member Aymar de Bois Mauri's Avatar
    Join Date
    Nov 2003
    Location
    Staring West at the setting sun, atop the Meneltarma
    Posts
    11,561

    Default Re: Settlements: File Formats; Take 2

    Quote Originally Posted by Redmeth
    I can't speak for everyone, but I for one and I think most of the EB team is also tremendously interested in the possibility of editing settlements.
    I concur. It would be very sad to see development on this area stopping. Keep up the great work.

  30. #30
    Closet Celtophile Member Redmeth's Avatar
    Join Date
    Mar 2007
    Location
    Bucharest, Romania
    Posts
    3,740

    Default Re: Settlements: File Formats; Take 2

    I have no expertise in this area but this seems like a very complex problem that could be easily solved by CA if they were willing.

    Does anyone fancy the idea of a petition by modders addressed to CA with the objective of allowing the building in M2TW and Kingdoms to become available for modding?

    I am not volunteering to lead this idea but a lot of people that are the best modders of the community seem to be interested in this and allowing the buildings to be moddable would show CA's goodwill towards the community and encourage modders to do even more extraordinary things.

Page 1 of 3 123 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Single Sign On provided by vBSSO