Re: Precomp - a command-line file precompressor

Hi again,

schnaader@xxxxxx wrote:
The "trial and error" part is what slows down Precomp. There are 9
different compression levels and 9 different memory levels, and they
don't appear in the zLib header, so the program has to check for
recompression 81 times per stream. But there are some optimizations
already that fasten up that part a lot.

Might I suggest gathering statistics while decompressing. Most likely, the deflate block size, and the maximum distance that the matches reach will help narrow down the choices (or at least help choose the most likely ones first).

Also, if you didn't, try Precomp several times on the files, like PDF
-> PCF -> PCF2 -> PCF3... this will lead to better results sometimes.
Precomp will tell you when this didn't lead to further savings.

You could probably make this automatic. Whenever you decompress a stream, you could look for valid deflate streams within it and recurse...

Deflate64 is not supported by zLib, and I'll have to use an
implementation from a different source like the 7-Zip source code. This
will follow when I added gzip support.

This is less likely to be of use, since to the best of my knowledge there are several different deflate64 implementations. WinRK's is custom built, 7zip's is too, and I know nothing about WinRAR's.

Malcolm

PS. Thanks, it is always fun to see something like this emerge! :) I just might have to fit the idea into WinRK... (if I can find the time)
.

Relevant Pages

  • Re: Z_SYNC_FLUSH and adding empty blocks after that (_tr_align)
    ... so you have a partial deflate stream. ... Now you can append bytes yourself to add empty blocks and terminate ...
    (comp.compression)
  • Re: Itterative detection of deflate parameters
    ... I understand there are numerous reasons ... but control deflate in such a way such ... One trick you could add to the toolbox is appending empty deflate ... is at least 4 bytes shorter than the original stream, ...
    (comp.compression)
  • Re: Itterative detection of deflate parameters
    ... You would still need to try all nine compression levels. ... emitted literals/length-distance pairs. ... zlib distribution will do that for you. ... duplicate deflate data. ...
    (comp.compression)
  • Re: Compress::Zlib unable to inflate
    ... Just trying to understand how the Compress ... module's inflate and deflate functions work. ... calling deflate on an input stream will compress ... inflate on that same stream should restore the original stream. ...
    (comp.lang.perl.modules)
  • Re: Seeking into a deflated stream
    ... You can't ignore the sliding dictionary while scanning a deflate ... since the matched string data may propagate through several ... matches from the very beginning of the stream. ... way to seek and avoid processing the deflate stream from the beginning ...
    (comp.compression)