Log file reader program

[MrWhipple]

( edit 9 Oct. 2003) This thread started in the Main Hall and was moved here. There is a parallel thread going on in the Jousting Fields at: Jousting Fields thread (http://www.totalwar.org/cgi-bin/forum/ikonboard.cgi?act=ST;f=18;t=11669)
(end edit)

I recently discovered the joys of log files. For those of you who haven’t found out about this pastime, they are text files that are produced for each battle. They will show up in the Logfiles directory if you have the option selected on the options:game menu. You can look at them with Notepad or any text processor.

The problem is that they are just text files and are a bit hard to extract the good bits to get any meaningful data that will help your game-play. Therefore I have decided to write a modest program in Python to massage the text a bit and add up some columns and do some analysis on the data. Like what is the mode and median of the valor of the remains of a unit, and would it be profitable to merge it with another unit. Friendly fire is another concern.

I know that others are working on such a program but I am just an old (and I mean OLD ) disabled hacker with a new language (Python) and nothing much to do all day except hang out on Forums and play MTW .

What I want to know is: What would every one else like to see in such a program. Please jump in and throw around a few ideas. I plan to share the results as freeware if I can get it looking good enough for visitors. *franticly cleans the office as the doorbell rings* http://www.totalwar.org/forum/non-cgi/emoticons/wacko.gif

Thanx in advance.

[motorhead]

There is another online log reader (http://www.thiep.com/cgi-bin/ffmtw/logread.pl) out there that seems to regurgitate just about everything in the logs on a per unit basis. However, as much as i appreciate this writer's efforts, this log reader seems to have difficulty with more than half of the logs i try to analyze with it, larger logs especially - which are the ones that i would really like to see. I guess what i'm looking for is a log reader that shows it all (unit type,cost,valor,weapon upgrade,armor upgrade,kills,etc., but on a per unit basis) and can also handle the larger files accurately.

[MrWhipple]

Thanx for the link Moterhead. It was a good start, but I plan for mine to be much more extensive and be more usefull. Also mine will be an .exe file that you can download. I was not to keen on the idea of a website prowling around on my system. (I know that this paranoia is unfounded, but hey, just 'cuz you are paranoid dosen't mean that they are NOT out to get ya) http://www.totalwar.org/forum/non-cgi/emoticons/eek.gif

Thanx again and keep those cards and letters comming in. http://www.totalwar.org/forum/non-cgi/emoticons/wacko.gif

[Kongamato]

Perhaps you could find a way to read every logfile on the computer to get some career stats and unit effectiveness. You might also be able to find out when you get your millionth kill.

[HopAlongBunny]

Sounds like a great idea http://www.totalwar.org/forum/non-cgi/emoticons/smile.gif

Some sort of breakdown vis. melee missle cavalry might be nice. As it is, they are all lumped together.
http://www.totalwar.org/forum/non-cgi/emoticons/wave.gif

[Vanya]

GAH

Vanya has existing MTW log reader. Vanya's Olde Shoggy reader had some nice features. Vanya not know if its still around here to download... youz could check that one out and see what it has to offer.

Vanya's new reader took the olde code base and simplified it dramatically... not so heavy on the dependancies any more. Now it makes HTML reports youz can see on browser (though its an EXE). Has nice charts and stuff. And it allows youz to make your own "custom" reports if youz want to.

Furthermore, Vanya's reader, in keeping with the tradition of the Olde Shoggy one, shows you nice reports that cover your aggregate battle experience.

Or, you can view individual battles to see all the gory details. And unlike the previous Shoggy version, this one allows you to view a log without importing it first.

The code will not have to undergo any significant changes to handle RTW unless the logfiles change (no changes at all were needed to VI http://www.totalwar.org/forum/non-cgi/emoticons/smile.gif ).

Vanya sez... There are so many log readers out there, Vanya did not feel compelled to distribute His. Vanya sez... make tool for your own consumption FIRST. If youz think its cool, then give it out to the community.

GAH

[Foreign Devil]

Career stats for a particular general/unit would be very cool. Not sure if this is possible, though. Do the logfiles track individual units?

[MrWhipple]

Quote[/b] (Foreign Devil @ Oct. 04 2003,21:16)]Do the logfiles track individual units?
Yes, they track by General's name. I plan to try to include such a feature. It would require that you keep all the logfiles in for a game in one directory, or just delete the files in the logfile dir before a new game. It won't be truly accurate because there is no way that the logfile can track merging or rebuilding of units, so it can't keep track of individuals, but it can keep track of battles under a general's command or a single unit that keeps the same general. I hope that makes sense.

[solypsist]

moved to Dungeon

[Vanya]

Quote[/b] (MrWhipple @ Oct. 06 2003,08:58)]
Quote[/b] (Foreign Devil @ Oct. 04 2003,21:16)]Do the logfiles track individual units?
Yes, they track by General's name. I plan to try to include such a feature. It would require that you keep all the logfiles in for a game in one directory, or just delete the files in the logfile dir before a new game. It won't be truly accurate because there is no way that the logfile can track merging or rebuilding of units, so it can't keep track of individuals, but it can keep track of battles under a general's command or a single unit that keeps the same general. I hope that makes sense.
GAH

Vanya sez... this not good approach. Its too time consuming to read through thousands of log files just so you can get a nice agg report.

Vanya's readers do the following, which you could do to facilitate such endeavors:
1) Creates a database to store imported data.
2) Allows users to import logs into the database
3) Agg reports run off imported data within the database, and not the logs themselves.
4) Once you import a battle, you never need to do so again. All you have to do is do periodic imports of your latest files to keep your data up to date.

Vanya allows individual log views too. But for aggregate reports like "total winning percentage when attacking a river" where the report figures span many logs really needs even a modest database to make it easy on youz.

Vanya sez... if you plan to "read" logs on the fly to generate agg reports, to just give up now, as it will be unwieldy to use. For solo views, this is clearly fine.

Vanya sez... you can architect your DB to contain all the data in the log or just a part of it, or even "massaged" data; that is, as you read the file, you can calculate things and store in your repository for future use. Youz can create views of your data to create denormalized recordsets to search and display.

And, once youz have it in a DB, its a piece of cake to slap it into eksemel (friggin' something blocks word out... so have to spell it out... http://www.totalwar.org/forum/non-cgi/emoticons/mad.gif ) if youz want to pop it on a web page or something. Not that you have to do that, as Vanya's reader creates HTML pages for reports... you can just save the blob and slap it on your site And you can give it any kind of look youz want

But, enough of Vanya's reader... We'z be talking about yourz...

Vanya sez... Vanya check back here from time to time... if youz have question for Vanya, He can cut off His grapefruit and offer youz a piece of His surrogate head

GAH

[MrWhipple]

I am starting to lean towards Vanas database aproach. I first I just wanted someting to massage a few numbers, but now I am seeing the utility of taking a much more all encompassing approach.

I also plan to wrap the whole thing in a nice Windows interface. Not because it is really necessary, but because I want the fun of writing a GUI with some new software that I am learning.

Keep those cards and letters comming in. Everyone has been a great help

[MrWhipple]

There has been some intrest both in messages and emails about what tools I am using to build my reader. Some of the following will sound a bit evangelecial but I get excited about my tools especally Python. I am a regular Python Zelot.

I am using Python as the base language. It is a rather new free, cross platform, open-source lanugage. You can download it for free at Python.org. It runs on Windows, Dos, Mac, Unix, Linux, Solaris, and about any system you can think of (even Amiga). You can even get the source and extend or change the language or join in with teams that are doing the same. There is a great community of programmers from newbies to old pros that are willing to help on comp.language.python newsgroup. Python has many modules that extend the language, from HTML to database and web development. You can implement a browser in just a few lines of code. It is an amazing language that can do every thing from writing a system script (like Pearl) in a few lines to large webserver apps or enterprise level apps with GUIs and the whole works (like VBasic or C++). Python also integrates well with C, C++ and Java and other languages. If you program, or are trying to learn programming Python is a great choice. Check it out

I plan to use Boa Constructor as the GUI (Graphical User Interface) develeper. It is a code generator that usese The wxWindows interface (a C++ class lib). It is a visual WISIWIG user interface that builds the code for the GUI and the hooks for things like mouse events and keyboard input. It comes with all the widgets (dialog boxes, radio buttons, sliders, etc.) pre-made, so you just kinda pick and modify the base objects, mostly by dragging things around on a screen.

I hope this answers most of the techie questions, if not feel free to email me or post here.

[CBR]

Hm is it possible to try out Vanya logfile reader?

CBR

[MrWhipple]

I am sure we would all like to see Vanya's reader. The more ways to look at the problem the better. I am trying to compile a list of all working readers. Anybody know of some obscure ones, please let me know.

[ShinKamiizumi]

I would be happy with something that would tell me what set of stats I was playing with and display them not unlike Ugly Polar's Stats page format. Being able to "massage" data would just be icing on the cake.


Edit - Downloaded and installed Vanya's logfile reader. So far, I can only say I like it but is a bit cumbersome for the average player but good for research.

[Wellington]

(I know that this paranoia is unfounded, but hey, just 'cuz you are paranoid dosen't mean that they are NOT out to get ya) http://www.totalwar.org/forum/non-cgi/emoticons/eek.gif

Feelings of paranoia are not always unfounded. It all depends on whether or not such 'instincts' are really paranoic or can be established as having a factual basis.

I 'recently' discovered that a certain International Computer giant were trying to break my own firewall. Why? A long, long story, that goes back to when I used to work for them.

Some time ago I realised someone was trying to persistently 'hack' me. I did a few things and then disabled my firewall. The next attemped entry to my PC established the origin of such an attempted 'hack' (Jeez ... not only was this company trying to hack me ... they were using totally incompetent staff who obviously did'nt appreciate all of the feedback implications&#33http://www.totalwar.org/forum/non-cgi/emoticons/wink.gif.

I did'nt hold much respect when I worked for them - I have even less now. http://www.totalwar.org/forum/non-cgi/emoticons/wink.gif

And this giant? Well, if you ever saw the film "2001: A Space Odessy" ...

... remember the computer name?

Welly

[MrWhipple]

That is a great bit Wellington. http://www.totalwar.org/forum/non-cgi/emoticons/cool.gif I have since found out that the site is Lady Anne's so I am not so freakey about it now.

P.S. And a cookie for the first one to get his computer name reference (hint: add to letters)

[Trousermonkey]

Hey MrWhipple how's your project going? I've been using Python for text processing and other scripted tasks for a while now and I've been thinking it's about time to start a project of my own with the MTW logfiles.

Though this idea isn't technically a logfile reader I will need to parse the logfiles. I've done a logfile reader in C already so I'm interested in how easy it can be done in python. Although I intend to do my prototyping from the interpreter, my plan is to eventually mount a front end on the project using Boa Constructor on wxPython - similiar to your plans.

You wouldn't happen to have any of your parsing code built yet? I'd be interested in knowing how you approached the problem. You can contact me at nkw10tm[et]hotmail.com