Project and Hutter Prize
This repository has a Rust lab for testing lossless compression ideas and a competitive research track on a separate branch. The lab contains several independent codecs, a command-line program to run them, a small sample input, and optional targets for downloading larger Wikipedia corpora. The research track studies a pinned published compressor. The project has not yet reproduced a full prize-valid score locally.
The prize task
The Hutter Prize uses enwik9, a file of exactly 1,000,000,000 bytes from Wikipedia. A valid result must reproduce those bytes exactly. The primary score is the combined size of the compressor and the self-extracting archive. The official rules also describe allowed separate compressor and decompressor submissions, platform and resource limits, publication requirements, and the minimum improvement needed for an award.
The record and threshold change when a better submission is accepted. Check the official prize page for the current value rather than relying on an old number in a checkout.
What the Rust lab does now
The CLI has separate compress and decompress commands. That is convenient for experiments and round-trip tests, but it is not itself a self-extracting prize archive. stats reports the archive size and size of the running executable to help compare local results. Its total is a development measurement, not a claim that the output meets every prize rule.
The codecs cover Huffman, LZ77, PPM, PPMD-style exclusion, mixed models, match prediction, and WikiMix-5. They are separate archive formats, selected by a four-byte signature during decompression. See the codec reference for every CLI name.
Many codecs read the complete input before coding, and the LZ77 matcher is deliberately simple. A successful round trip on the sample establishes correctness for that input; it does not establish competitive size, memory use, or runtime on enwik9.
What the research track does now
The research/native-mac branch vendors cmix-neif 1.0 as the candidate starting point. It contains a native Apple Silicon development port, a Linux comparison build, article-aware samples, and saved benchmark and parity records. The published upstream score and a matching reference binary hash are known; a local full enwik9 result still needs verification. The native/Linux case study reports a small valid fixture and explains why almost equal archive sizes do not prove compatible bitstreams.
What to do first
Use the tracked data/sample.txt and run a Rust round trip. Then read what compression does. For Rust codec changes, follow the experiments guide; the larger corpora are optional. For prize research, start with the competitive track and its experiment playbook.