M:TW maps: imports and exports

[Kraellin]

ok, based on some short conversations with tosa and barocca, i decided to start looking at the file structure of the mtw maps and the possibility of altering existing ones (before the full game comes out with editor) and with importing existing stw/we/mi maps into the demo for use. like i say, tosa and barocca have both had some success in this already.

so, last night, i opened up the jaffa map and decided to see if i could tweak a couple things. the long and the short of it is that i couldnt....so far. it's been a while since i played with this stuff, but i believe these maps are using a checksum count for integrity, so i might have to use some other tools before i can alter existing mtw maps. just not sure yet...was only a cursory look last night.

however, i started this post for anyone and everyone to post data concerning the maps themselves. if you've tried stw/we/mi maps in the demo, post your results here, good or bad. if you can read and alter the file structure, post what you've found here. some of this may well become moot once we have the full game and editor, but let's start now anyways.

and a note to CA, if you havent already included a converter for making stw/we/mi maps playable in mtw, could you possibly do so? seems a shame to simply toss away all those old maps.

K.


------------------
The only absolute is that there are no absolutes.

[barocca]

DryRiverbed1 works
some minor graphical glitches
30x30 map

Ridges2 works
minor graphical glithces
30x30 map

===
by minor graphical glitches i mean some tiles are not correct,
no models/buildings,
some terrain elevations different,
but if i mention them here they are playable and not too weird


[This message has been edited by barocca (edited 08-07-2002).]

[Choco]

It would be great to have new maps from STW to use to do demos for medieval

The demo's maps are getting old quickly

[barocca]

I have been experimenting,

so far up to tile 86

tiles 4, 5 and 6 are the same between WEMI and Medieval,
all others are different

[barocca]

and using tiles 4,5 and 6 - plus Trees which also work ...

Totomi, with a few more trees, to compensate for the tiles i had to delete,
http://doragonbarocca.homestead.com/...tw/totomi3.jjm

and Waterloo - minus the models,
http://doragonbarocca.homestead.com/.../waterloo3.jjm


both are best for arid or lush terrain

If you want trees for a rocky desert terain you have to place them manually, the tree tiles 4 and 5 showup as 'dirty' terrain in desert conditions,


Now if anyone wants to trawl through the 2 tiles sets and tell us which tile number to use from stw to get valid effects in mtw...

------------------
Clan Doragons Medieval Website
Mods, Unit Descriptions and more
DoragonBarocca of Clan Doragon

[Kraellin]

i'm afraid i kinda gave up on this project, bar, after target's post. seemed easier to just wait for the editor and try and load and tweak the old maps in there.

K.


------------------
The only absolute is that there are no absolutes.

[barocca]

they moved the release date back,
6 September here,
and i have serious Deja Vu'
(remember Warlords? - every time they got to 14 days before release they tacked 2 more weeks onto it...)

what i did was opened a map in stw editor,
then placed a row of consecutive numbered tiles - mark every 5th tile with a double,
saved the map.
did not close stw,
copy map to MTW, then edit a bdf to 'load' the map,
run mtw,
then i can tab back and forth and 'see' the differences,
my problem is i haven't made ANY maps, so it's not clear to me exactly what i am looking at,
but tiles 4, 5 and 6 are the same and the trees work,

time is one thing i have in short supply at the moment,

if i could edit the tiles that are displayed for selection along the bottom in STW editor, but i can't find them...
(I was thinking of taking a screenshot and making a 'representaion' of the MTW tiles for the selection thingy in stw editor)

alternately someone who is far more familiar with the editor could write a list,

[Kraellin]

bar,

stw or we/mi? we do have two editors currently.

the textures for the we/mi editor are all in one large proprietary compressed file. i forget the name. at least, that's my best guess.

what i was going to do was write a program to convert the actual file on a more automatic basis. the file structure is fairly straight forward and shld be fairly easy to write a simple conversion program without ever having to load a map in the an editor. if you know where each texture is stored and what texture number it is for we/mi and then find the appropriate texture in mtw and it's corresponding number, you'd simply make a converter to change the we/mi numbers to mtw numbers.

the models, however, simply wont translate. they just arent the same thing. but, they would be fairly easy to erase from the file on an automatic basis, since they are in ascii form in the file. i suppose a few would translate over ok, in which case you could write a converting routine for that, but most are going to be lost.

K.


------------------
The only absolute is that there are no absolutes.

[barocca]

the wemi editor,

there are plenty of new textures in MTW,
and quite possibly some of the wemi textures were 'discarded',

(when i experiment as above, i get some interesting and unexpected results)


why could they not have simply left the tiles as they were, and simply added new ones...

[Kraellin]

hehe, i aint touching that last line, bar ;)

oh, and tosa made a texture display map at one time. i forget the name, but that might help ya out.

i also see you've converted totomi and waterloo. nice :)

K.


------------------
The only absolute is that there are no absolutes.

[barocca]

WEMI Tiles 4, 5 and 6 work 100%
WEMI tile 7 has to be rotated 180 degrees to work,
WEMI tile 8 works 100%

So for tile 7, once you have placed the tile, simply rotate it 180 degrees and it will work nicely in mtw!

The following tiles WILL look weird when placed using the WEMI editor, the description is what MTW displays them as

WEMI tile 143 also works as a substitue for tile 8, giving a reduction in forest density,
note 143 is a 'blend' tile, meant to blend in with a dark rough grass texture across one corner, but as long as you block it with 5's, 6's and 7's the effect is fine.

WEMI 145 and 144 are also possible to replace 8, also giving a reduction in trees and the increase of dark rough grass texture giving an interesting forest floor effect.

(WEMI 145 lots of rough - sporadic tree density)
(WEMI 144 half rough - light tree density)
(WEMI 143 one corner rough - medium tree density)

waterloo and totomi will be updated again soon,

any requests for maps you'd like me to try?
I will edit in only the above tiles so far.
If I have more time i will try to work out more,
but the tile orientations are painfull to get correct.
B

[This message has been edited by barocca (edited 08-09-2002).]

[TosaInu]

Konnichiwa,

We need to be able to tell MTW jjm maps to use a custom set of folders. All that's required then are STW WE/MI textures in TGA format, the STW WE/MI models (already available) and a simple convertor that adds the custom paths in hex format to the STW WE/MI jjm.

Converting maps by hand takes too much time.
The proposed method also opens the road to real custom maps.

------------------
Ja mata
Toda MizuTosaInu
Daimyo Takiyama Shi

http://www.takiyama.cjb.net

[Kraellin]

bar,

you're getting me curious again. i may have to take up the gauntlet once again. good work :)

K.


------------------
The only absolute is that there are no absolutes.

[GilJaysmith]

I've got a tool that dumps out a .JJM file as text. If you're very nice then I'll give you the source.

IIRC the .JJM format didn't change from the Shogun v5 format, so, subject to my having forgotten something crucial here, you'll only have to deal with missing models and the different texture numbers. You probably know this already, but:
- Unrecognised models will just get dropped (I think!)
- The different texture numbers will have implications for cliffs - most likely they'll vanish - and forests - which will vanish, with new forests springing up on other textures.

Gil ~ CA

[barocca]

GilJaysmith,
the utility would be very handy,

can you send the utility itself?
(zip file format please)

If you can only send the source what would be required to compile it?

please send to barocca_x@hotmail.com

[Kraellin]

gil,

i'm assuming here that you dont just mean something like notepad which can display a .jjm in assci.

and yes, i'd appreciate this also. email is starfire@apex.net

K.


------------------
The only absolute is that there are no absolutes.

[GilJaysmith]

Quote Originally posted by barocca:
If you can only send the source what would be required to compile it?
[/QUOTE]

A C++ compiler. We use MSVC++ for everything at CA, but it should need only minor changes, if any, to build under other compilers.

But I'll send you the EXE too.

Gil ~ CA

[barocca]

Kraellin,

this is a sample of what i am working on,



The first row are from Warlords editor,
the second row are the corresponding tiles as displayed in mtw, they are taken from screenshots of a specially constructed map...(thats why a little blurry)

(up to tile 40 so far...)

and
GilJaysmith

that would be very neat - i'll keep checking my mail..



[This message has been edited by barocca (edited 08-10-2002).]

[TosaInu]

Konnichiwa Gil Jaysmith sama,

Please enlighten me here, what's the use of having the jjm file in txt format?

Is it possible to change the txt and then convert it to a working jjm file again?

Barocca san, very nice. But as you can see some textures are totally different. If you want STW maps to be used with MTW you've to remake them. A tedious job, making a tool to do that requires quite some skill and is limited to only STW maps. While the method described above, also allows custom texture directories.

------------------
Ja mata
Toda MizuTosaInu
Daimyo Takiyama Shi

http://www.takiyama.cjb.net

[barocca]

Konnichiwa Tosa san,

sadly you are correct,
however we have no means at our disposal to implement your directory control idea,
also we have no tool available to convert maps,
therefore we must convert them by hand and/or make 'fresh' maps from scratch.

Tiles 1 and 3 in stw/wemi produce a tile (in mtw) very similar to stw tile 35, tile 1 is a little bit 'dirtier than tile 3.
neither mtw1 or mtw3 match stw35 exactly!
NOTE many of the stw/wemi tile have no exact match in mtw that i can 'see', a converter will have to do a 'best guess'.
In order for anyone to write such a converter they will need a visual guide, in order to 'code' the tile changes.
Hence my current project.


Knowing what tile in stw/wemi will produce what terrain in mtw will allow (i hope) more creativity in maps for use with the demo,

Trying to create or convert maps and remembering which tiles produce what from memory is very error prone, being able to 'see' tiles and default orintation will at the very least make life easier for me to convert maps.

While i acknowledge your idea for directory control is far superior, and that the release may be mere weeks away, I admit to wanting a little more variety in maps for custom battles.

ja mata
B.

[TosaInu]

Konnichiwa Barocca san,

I agree that your method is the best, and only, short term solution.

The directory control is what MTW maps should have, only the developers can make this. A game like UT has maps that control custom textures, models and sounds. Red Alert maps allowed scripting.

The special map Kraellin mentioned is simply 'iron bord' having all STW textures (old STW). You can load this map (error message pops up but you can click this away) and see all textures at once (grouped in series of 5).
Unfortunately loading green like maps in MTW fails. I believe you know a workaround for that?

Good luck.

Edit:http://www.totalwar.org/maps/Tools.shtml
4th file from the bottom, Texturetool.

------------------
Ja mata
Toda MizuTosaInu
Daimyo Takiyama Shi
http://www.takiyama.cjb.net

[This message has been edited by TosaInu (edited 08-10-2002).]

[GilJaysmith]

Quote Originally posted by TosaInu:
Konnichiwa Gil Jaysmith sama,
Please enlighten me here, what's the use of having the jjm file in txt format?
Is it possible to change the txt and then convert it to a working jjm file again?
[/QUOTE]

The main use will be to anyone who feels like modifying the source to do interesting stuff to the map representation and then write out a new .JJM file. With a tool like this, a man... could rule... the world! Or at least automatically convert an STW map to look OK in MTW without having to load it into the editor.

Gil ~ CA

[TosaInu]

Konnichiwa Gil Jaysmith sama,

Is this used at the CA office too? What's the benefit of this tool over the mapeditor that CA uses?


------------------
Ja mata
Toda MizuTosaInu
Daimyo Takiyama Shi

http://www.takiyama.cjb.net

[barocca]

watch this space

17 tiles to go and I will have the visual tile map completed,
i'm just taking a quick break to rest my eyes,
Then anyone will be able to see which tiles in stw represent which in mtw!

[barocca]

Done

I suspect there are more than 161 tiles in mtw, at this moment i cannot access these additional tiles, all I have displayed are the 161 tiles i can currently 'reach'


sample image


download here
http://doragonbarocca.homestead.com/...TileMap_mq.zip
(268k)

a very high quality version is available, but it's a full meg...



===
this took quite a while to make, if i've made any errors - due to lack of sleep - let me know asap!, and i'll fix them


------------------
Clan Doragons Medieval Website
Mods, Unit Descriptions and more
DoragonBarocca of Clan Doragon

[This message has been edited by barocca (edited 08-11-2002).]

[This message has been edited by barocca (edited 08-11-2002).]

[barocca]

mapdump command line,

msdos prompt >>

mapdump mapname.jjm >output.txt




[This message has been edited by barocca (edited 08-12-2002).]

[Kraellin]

yup bar,

just checked my mail and got it also. unzipped and ran just fine. this is cool. this shows the entire file structure, especially when compared to a hex dump of a map. very easy to discern what everything is now.

shld be fairly easy to make a VB converter to turn the .txt file back into a .jjm, or is that what some of the c++ files are for?

also sent a copy to tosa. and gil, i was right. he had it running before i did ;)

even without a VB converter, i can now see all of the file structure and could do edits from a hex editor or even notepad, prolly, though that would take considerably more time.

gil said: 'With a tool like this, a man... could rule... the world! Or at least automatically convert an STW map to look OK in MTW without having to load it into the editor.' he's right. the height data is all there, the texture array is there and the model data is there. even the header info is there. with a byte by byte read > edit > write routine, it shld be possible to alter maps on an automatic basis.

and, it ..might.. be possible to alter map sizes as well. alter the header, add in height data, alter the array...a bit tricky, but may be possible.

K.



------------------
The only absolute is that there are no absolutes.

[barocca]

Well Kraellin,

creating such a VB program is far beyond my skills,

after all I just spent a day and a half taking screenshots of the stw/mtw tile sets,
only to find the mtw tile set tucked away in a folder...
i could have saved half a day if i had found them first,

Note
Tile to be displayed is refered to numerically,
Is it possible to create a VB program that will allow us to use tiles numbers BEYOND the current range?

If mtw does not allow custom tile sets, then perhaps we can create and load tiles outside the current numerical range of the tilset.
(3 sets of 170 tiles are included in the demo)
If we cannot get mtw to load tiles from a custom folder my other hope is to go 'beyond' tile 170 (or what ever is the final number of tiles in the mtw release) and create our custom tiles there.

We would temporarily replace a current tile set, create our map, and then renumber our custom tiles, drop them into a terrain folder and use such a utility to adjust the tile reference numbers in the map file!

Don't try and create such a utility just yet, just tell me if it wold be feasable,


Tosa
I know How to create custom tile sets,
I beleive I can even get mtw to load them,

the problem is they will (at this stage) be at the expense of an existing tile set,

there are 3 tilesets with the demo!
all of them currently 170 tiles (more or less), Lush tTile set tries to look for tile 171 when loading, and fails - it then stops looking for more tiles,
perhaps we can place more tiles beyond 170,


I'm just going to test my load custom tile set theory,
i cannot test my place/number tiles beyond 170 theory, because I don not fully understand the jjm file format, and thus cannot tell the jjm to look for tiles beyond 161,
Kraellin?
help?

[barocca]

We can force any tile set into any folder



ALSO
We have 2 more terrain folders that MTW Demo can recognise and use,
(it took a few loads and deciphering the failures to get these to work, thank heavens for the watcher utility)

The Folders are Temperate and Desert, I have not tested thoroughly for backdrops, but for summer they are both working fine.

what this means is if anyone wants to create custom Tile Sets, they can


If the difference is going to be dramatic then I would suggest sending me a copy of the tile set and I can create a custom TileMap for that set to allow anyone to create their own maps,

ALSO any custom tile set will have to be made available for download if you want people to be able to play on it.

However the standard tile sets zip to 25Meg,
so you need to be concious of that and try to make your set as small as possible,

Here's How to Implement the additional Tile Set folders

Temperate Tile set

Create the folder here
..\Medieval - Total War (Demo Version)\Textures\Ground
call the folder Temperate
in the BDF call the Terrain::TEMPERATE

copy a set of the tree files in this folder
D:\Games\Medieval - Total War (Demo Version)\Textures\Trees
(or create your own)
rename them
Conifer0.lbm
Conifer1.lbm
Conifer2.lbm
Conifer3.lbm


Desert Tile Set

Create the folder here
..\Medieval - Total War (Demo Version)\Textures\Ground
call the folder Desert
in the BDF call the Terrain::SAND_DESERT

The Sandy Desert uses the Desert Palms tree files
despalm0.LBM, despalm1.LBM, despalm2.LBM, despalm3.LBM
so no need to create them, unless you want to change the palms
(NOTE:- This Terrain Choice automatically deletes tree models from forest terrain types)



one theory proven, one to go...

====
edit
====
I forgot to mention,
if you look back at the sample TileMap earlier in this thread you will see 13 White Tiles - these are Actually BLANKS,
meaning you can create 13 NEW tiles without having to create the new folders!

Arid and Lush both contain 13 Blank Tiles,
numbered 88 to 96 AND 98 to 101.
RockyDesert is a Full Tile Set


Tiles 162 to 170 of all 3 sets contain aditional Tiles that I have not yet figured out how to get the WEMI editor to reference.


[This message has been edited by barocca (edited 08-13-2002).]

[Kraellin]

bar,

lol. i thought you were using the textures in the demo to do your map, otherwise i'd have spoken up. btw, nice job!

ok. lots of stuff there. lemme see what i can cover.

texture numbers beyond the current range... can only guess on this one. they're prolly using a single byte for referencing the textures, so # 255 is prolly as high as it goes. in fact, when researching map making during the stw demo/editor, i ran into a phenomenon where the texture numbers 'rolled over'. in there, the numbers only went up to 127 and then rolled over to a new set of 128 numbers, but the second set was completely unusable. still, that made 256 textures, so i suspect it's still the same...256 max textures for any one set. but, only a test will prove whether we can access the higher numbers or not. if the .exe isnt set up for it, i'm not sure. just have to try it.

the same, based on what you've found out about 'temperate' and 'desert', may be true for additional texture sets. it ..may.. be possible to have an unlimited number of texture sets or it may be something hardwired into the .exe. i suspect those additional sets you found are projected sets that will be released with the full game, so, it depends on how things get called and referenced. CA's track record suggests that it will be hardwired, but, of late, they've been loosening up on the hard wiring, so i dunno. it might be possible to tweak something and get addtional sets to work.

the thing is, something has to call up the proper textures for the given map. if that data is in the .adf then we're good to go. we could just name our sets anything we wanted and make the maps that would store the texture set internally............

oh hell, i just looked at the .bdf. they're way ahead of us on this one. the .bdf stores both the map name and the terrain type (texture set). that ..shld.. mean that we can make any number of sets, store them the textures folder and reference them from the .bdf. so you ..shld.. be able to make a texture set named 'ice cream', if you wanted to, stuff it in the texture folder and simply alter your .bdf to call it up.

now, backtracking a bit, it's the map editor that's going to determine if this can be done on an automatic basis or not. if the editor allows for importing custom sets then it's a piece of cake. if not, then we shld still be able to edit by hand. you'd make your map in the normal way with a default set of textures, save the map and then edit the .bdf.

but, that only works for historical and any other forms of map that use and are referenced by the .bdf. so, it may not work for multi and custom games. i dont know where the call is for pulling up maps for those. i'm afraid those may be hardwired in the .exe and CA would have to pull that out and make it available to us to use....unless....i havent done a map dump on an mtw map yet. if they put the texture reference within the mtw maps, then we may be good to go.

bah, i just did a map dump of jaffa.jjm and i dont see a texture set reference. however, some of these byte numbers dont make sense, given a single byte system. some of the texture bytes are double bytes, which might indicate both a texture number and a reference to a texture set. what i would expect to see is simply a 2 digit hex number, representing numbers from 0 to 255, but a number of the textures are 4 digits, or 2 bytes and the first 2 digits are very limited in what they are, indicating a possible reference to a texture set?

i dunno, i'm too tired right now to work it out.

i can tell you what the map dump data means, however. the stuff at the top is header information. simple stuff. i'll explain that in more detail later. the next stuff is the height information, the 'contour' stuff. this is laid out in rows, prolly starting in the upper left hand corner of the map. next is the actual texture numbers and below that is the model data. btw, this map dump stuff matches up just fine with a hex editor result, except the map dump reverses the ibm style of reversing bytes in 'words'.

yeah, i saw the blank textures also. i'd be careful of those. CA may have some use later on for them or they may do something weird or they may have just not been finished at the time of release. dunno, but i wouldnt get your hopes up of being able to use those.

K.


------------------
The only absolute is that there are no absolutes.