Rumbling from the Shire #9

Its been a while since the last time I wrote one of these posts. This one is part Project Status and part What has been worked on. It's past 40ºC here, so I'll try to keep it simple. First, the Project Status part.

Project Status

You may have seen there has been barely a few commits in the develop branch of the repo since the last release. That's because I wanted to focus on an old problematic feature: compression. So, instead of wiring up multiple different features that people may or may not use, I decided to focus on finishing support for compressed PackFiles. The thing is.... that's a difficult thing to do. Brace yourself, technicalities incoming.

You see, CA PackFile Format has been evolving over time. Changes in the structure of the PackFile where marked with a change in their PackFile Version, a byte in the header of each PackFile. And, since Warhammer 2/, version 5 is the lastest one. RPFM uses this to guess what game to select when opening a PackFile, by the way. And that's why the game changes to Rome 2 sometimes when opening a Warhammer PackFile (both are version 4, and it defaults to the oldest game with it), because the PackFile was saved either for the wrong game, or was saved with PFM, which for some reason was saving version 5 PackFiles as version 4 ones for some time. But anyway, I'm going off rails. Back to the compression thing.

One improvement on version 5 was the use of compression. There is a byte in certain place that, if set to 1, means a PackedFile is compressed. And... here is where things get weird. The compression method used by CA is LZMA1, preset 3, Non-Streamed, with a borked header. Problems with this are:

  • LZMA is a legacy format since 2013.
  • The borked header (they remove 4 bytes, losing +4GB PackFile support, and reorder the rest of the header), due to the way it's borked makes it very hard (maybe impossible, I just realised I forgot to test an specific situation) to use LZMA Streams.

Why is this a problem? The LZMA format, being legacy, hasn't seen much improvements over the years, and only two big libs (may be more smaller ones, but meh) support it: LZMA SDK (Windows only) and XZ2 (Multi-platform). Now, what are the differences? Only LZMA SDK supports Non-Streamed LZMA files. XZ2 supports only decompressing them. So, for linux, the only thing I can do to compress the files is passing them through p7zip, one by one, which is slow as hell, because I have to write each file to disk, compress it, then read it to memory. SLOW AS HELL.

Now, what happens in Windows? There I can try to use the LZMA SDK, but there is no Rust crate (lib) that works with it and support Non-Streamed PackFiles. So I have to come up with some way to get the lib working with them (which require FFI, of which I have little idea). So... yeah, for now using the 7zip trick kinda works, but... I'll see if I can get some improvements for that in the future.

So, in the end I spent three weeks getting the compression to work. At least now it works most of the time. And add that to the fact that hell/summer is comming to Spain, and the fact that I live in the middle of Spain where heat is felt the most, and.... yeah, it's not pleasant to be burned physically and mentally at the same time. So I decided to take a break. After all, I've been working on this since.... mid-october of 2017 almost non-stop (except a few days after a few releases). So, instead of keep developing RPFM, I decided to rest for a few weeks, and use that time to get other stuff done, like that eternal siege tutorial for WH2, or PLAYING SOME GAMES INSTEAD OF JUST MODDING THEM.

What has been worked on

Now, about feature being working on. Before I started with the compression thing, there were a few features half-done:

  • AnimPack/AnimTables/AnimFragments/MatchedCombat support: support for these is kinda done. Works for some files, still needs fixes for others, and some UI.
  • RPFM Lib: The idea is to get the backend out of RPFM, so it can be used in other programs. Thing is, the moment I got it working, the moment it started to become a maintenance burden, just because I had to duplicate a ton of error-checking code, among other things. Also, doing it C-Compatible (so it can be called from other languages) turns out to not be that easy, specially when dealing with Strings.
  • CLI: Depends on the RPFM Lib.
  • Empire Schemas: turns out the table versioning thing in empire was more chaotic than I initially though. There are some new schemas on the making.
  • Three Kingdoms Schemas: I've been working on an... autofiller for schemas, mainly for getting more data from the assembly kit automagically into the schemas, like lookup info, paths,... Turns out this can be kinda retooled to automagically fill holes in the current 3K schema when the Assembly Kit for that game (and in future games) gets released. Which means we maybe get a fully decoded schema full of reference info a few days after the Assembly Kit gets released. As long as the structure of the files hasn't changed from Warhammer 2, of course....
  • Bugfixes: There are a few bugs I got reported that needs fixes. Like yet another CTD on copy/paste in a very specific situation, or the "Close Note" button not working if you open the note in a specific moment.
  • Schema Improvements: After some test, turns out I can use the schemas for more than DB Tables and Locs. So I've started some changes to, among other things, use it to greatly simplify the Anim support on the first point. Also, there are the changes I talked about in the Three Kingdoms point.
  • And probably more things I forgot about.

So, to keep it simple, compression is kinda done, I'm hot as hell, new stuff is coming soon (but not too soon), and I'll take some time to write tutorials, whatch anime, read mangas/novels, and play games.


Tier Benefits
Recent Posts