Release of the PACK tool (for packing/unpacking/repacking M2TW PACK files)

Hereby I present the first release of the PACK modding tool, and new addition to the growing collection of modding tools created by the Europa Barbarorum team. This tool is the result of me converting johnwhile's research into the PACK file format into a usable little tool which has the following features (usage info):

Code:

---

pack <command> [opts] -- [<PACK> [file...]]
pack <command> [opts] [<PACK> [file...]]

Where <PACK> is the PACK file to operate on and pack the pack program

Commands:
  -a, --add              :  add files to PACK file. If <pack> does not exist
                            it will be created.
  -c, --create          :  create a new PACK file, overwriting <pack> if it
                            already exists. --no-clobber is not supported.
  -h, --help            :  display this help
  -l, --list            :  list files contained by a given PACK file
  -r, --remove          :  remove files from PACK file
  -x, --extract          :  extract files from an existing PACK file
  -v, --version          :  display version information

Options:

  -n, --no-clobber      :  do not overwrite existing files. When extracting
                            files from a PACK file this option guards against
                            overwriting files in the target directory; when
                            adding files to an existing PACK it prevents
                            overwriting files already present in the PACK file.
  -i, --case-insensitive :  [file...] is case insensitive, this option only
                            applies to matching files located in the PACK
                            file 
  -g, --glob            :  [file...] may contain wildcards ('*' and '?'),
                            this option only applies to matching files
                            located in the PACK file.
  -d, --details          :  list details about the contents of a PACK file
  -t, --text            :  read file names from given text file(s), when
                            creating a PACK file
  -0, --xargs            :  read NULL delimited file names from given file(s),
                            when creating a PACK file

---

The program is available as both a source and a binary distribution from our FTP server. Both distributions are packaged as 7zip archives. Programs such as 7-zip can be used to extract these contents. (You can get 7-zip from here: http://www.7-zip.org/ or commandline equivalent p7zip from here: http://p7zip.sourceforge.net/ )

The program is made available under the GPL v.2 or later, copy of which is provided in licence.txt of either distribution. Below follows the summary:

Terms of Use
This software is free and there is no warranty what so ever. Use at your own risk.
See licence.txt for a copy of the program licence.

Copyright (C) 2012 The Europa Barbarorum Team <webmaster at europabarbarorum dot com>

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details at
http://www.fsf.org/licensing/licenses/gpl.txt

Comments, criticism, improvements and bug reports are welcome. Regards,

The Europa Barbarorum Team.

New version, which brings a few code simplifications. Features & behaviour remain the same.

New release. Fixed a crash when creating directories on Windows.

New release, this one fixes some inverted logic (if/else) which causes packing certain files to fail. To trigger the bug you would have to have a file for which only the last "chunk" of data is incompressible.

Update:

Fixes

• Fix missing include (stdio.h) in util.h
• Fix bad handling of the output pointer at slurp(), in util.c. Fortunately, in the case of the PACK tool this bug would not "manifest" itself during runtime.
• Fix opening sentence in the ReadMe, proving nobody ever reads those.

Another update. This one fixes a bug which caused the tool to crash (due to segfault) when unpacking uncompressed data (chunks).

Version 0.7 of the PACK tool is available now. It brings some major improvements to the tool to make your life a lot easier when using it.

Features

• Maintain a consistent ordering of the files in the PACK. Previously this depended on the order in which files are supplied, but it appears that Medieval II: Total War wants them sorted.
• Avoid packing duplicate files. Previously pack -a some.pack duplicate.txt duplicate.txt would pack the file duplicate.txt twice, no longer. It prints a warning about the second duplicate.txt instead. Duplicates are not removed if they already exist in the pack, however.
• Parsing logic for -t, --text, -0 and --xargs options is now more lenient. Empty strings (lines) are silently discarded instead of triggering a failure to pack. Closing delimiters for the last entry in the parsed file are no longer required, either.

Bug fixes

• Fix memory access warning in the parsing logic which deals with the -t/--text options.

Misc

• Output of the -l, --list commands is now sorted and in-order as well.

Version 0.8 brings the following new features:

Features

• Add -z/--no-compression and -s/--small-files options to disable compression. The --small-files option is meant to mimick behaviour of "Medieval II: Total War" PACK files, it disables compression for files of <= 1024 bytes (1kiB).
• Picked a more aggressive compression algorithm (lzo1x_999 vs lzo1x_1), yielding smaller results at the cost of (significant) extra run time and memory consumption while packing.

Misc

• Use the 'safe' LZO decompression algorithm variant which performs sanity checking of arguments passed by the pack program. This should make the program resilient against malformed input data. Also added some logic to print additional error messages indicating the specific error condition when compression/decompression fails.

Turns out there is a nifty optimisation feature built in to the LZO library. Let's use it, in version 0.85 of the PACK tool:

Features

• Use the LZO optimisation feature. This 'optimises' compressed data to make decompression slightly faster (1-3% according to the documentation).

Misc

• Add a section to the readme on compatibility with "Medieval II: Total War"

There is a bugfix for the LZO library used by the PACK tool available which addresses an issue that could be a potential security concern depending on how the library is used. The PACK tool appears not to be affected by this bug because it never attempts to decompress data that is large enough (16MiB) to trigger the bug.

Nevertheless I'd like to update the PACK tool to use the latest LZO. I ran into an issue trying to cross compile it using MinGW (to produce Windows builds of the library), however. As a result I am not able to produce Windows builds of the PACK tool with LZO 2.07 for now. This means that I'll have to hold off on this update to the PACK tool until the issue is resolved. In the meantime, I have filed a bug report.

Normal compilation of LZO using the lzo makefile still works fine. Since the update does not require modification to the PACK sources (only a rebuild with the new LZO is required), users wishing to upgrade to the latest LZO are advised to rebuild the PACK tool from source for now, until a proper update is available.

Version 0.86 of the PACK tool incorporates the latest version of LZO which address the build issue.

Misc

• Bump LZO version to 2.08.

I am such an utter noob when it comes to this command line stuff. What would be the entry in a batch file to extract all animation files from a custom pack file? (Medieval II\...\mymod\data\animations\pack.idx)
Batch file and IDX file in same folder as the pack.exe - I have got a 64bit system so I am assuming it's pack_amd64?