Train the small final bit predictor

The compressor has two distinct pretrained neural components. The transformer predicts bytes; a much smaller LSTM adjusts the final bit probability after the other models have mixed their predictions. The second model, called BITLSTM upstream, ships only 7,386 bytes of bfloat16 weights. Improving its predictions could reduce the data without adding another large model. That is a hypothesis, not a measured gain here.

What upstream actually trained

The upstream training record uses 418,837,176 model bits from roughly the first 20% of the coder stream. A model bit is one where this predictor runs; metadata, alphabet/grammar-fixed bits and adaptive-depth skips make this different from a raw input-bit count. The 16-cell model has 3,585 active parameters. Its weights stay frozen through that training range, then begin learning online at a complete training-window boundary.

Upstream reports 32 offline passes taking 3 hours 10 minutes on one CPU core; collecting the first 20% trace took about 14 hours. These are published costs, not measurements on our hardware. It did not train offline on the entire stream. A broader training range or a different compact correction model may help, but could also worsen the already trained portion.

Work	Main cost
Collect predictor inputs	A real compressor forward run, with all upstream model state
Replay and train this small model	CPU work over saved inputs; no transformer forward pass repeated
Ship the result	Model bytes plus executable changes, measured in the final package

The correction never feeds back into the upstream predictor state. A correctly recorded trace can therefore be reused across many bit-model settings. This is why replay could make iteration much faster after the expensive first run.

The missing piece is real

The public package contains the replay trainer and the final float32 checkpoint, but does not contain the original trace writer's development tree or the trace itself. Its trace shell script expects that missing tree. Running it unchanged in this repository will not reproduce the training run.

The published 20% trace consists of 25,130,230,588 bytes of bit records and 14,449,882,600 bytes of channel records. The latter holds transformer and byte-LSTM projections once per eight model bits. Full-stream storage could be on the order of 200 GB if density scaled similarly; that is a planning estimate, and the actual record count must be measured. This is offline research data, separate from the final compressor's temporary-disk budget.

The float32 checkpoint can be converted back to the shipped bfloat16 array. The completed export audit verifies its published hash, geometry, finite values, zero padding and exact embedded bytes. The exported file and compiled array have the same SHA-256: c72957f4…08a8e7d. It does not regenerate the missing trace or reproduce the original training.

Reconstructing a trace writer

A frozen fixture protocol now defines a diagnostic writer against the current runtime. It copies raw mixer inputs and the projections the model already calculated, records each observed bit and saves the corrected integer probability for replay comparison. Channels follow the count of executed model bits, not raw byte boundaries. Files use exclusive creation; a 16-million-model-bit cap fails explicitly if exceeded. The upstream source remains unchanged in Git. A preflight reader review corrected a mistaken four-zone validation limit: the upstream enum defines six zones, including dictionary and title-mask zones. The v2 protocols retain this correction and the original protocol hashes. The compiled writer bytes are unchanged and are verified against their recorded recipe revision; no recording run was replaced to make this correction.

The writer must leave the complete fixture archive unchanged, and encoder and decoder must produce identical trace hashes. A matching file format alone does not establish faithful replay or reproduce the historical training dataset.

The replay protocol uses the public trainer with the exact shipped bfloat16 file. A diagnostic freeze after its first observed bit prevents any complete backward window or optimizer update. The driver compares every corrected integer probability, then deliberately changes the first reference probability and requires a mismatch at bit zero. Passing this checks frozen inference, not the later online-training arithmetic or a new model's compression performance.

Native Linux diagnostic checks pass the compiled writer/driver controls and source-change guards. These are synthetic checks; the real model trace and replay results are below. A separate summary reports the current head's discrete-probability cost by model-bit decile and recorded channel state. This can guide a later experiment, but does not measure an archive with the head removed.

First fixture results

Both Mac recordings preserve their entire baseline archive and restore the original XML. Encoder and decoder produce identical trace/channel/probability hashes. This gives 2,418,045 model bits on the screen and 2,399,939 on the disjoint fixture. All recorded channel zones are zero in these two traces.

Fixture	Archive bytes	Current head's probability-cost change	First trainer mismatch
Screen	119,065, unchanged	−190.29 B	Bit 665,047: live 47,768, replay 47,767
Disjoint	117,438, unchanged	−222.39 B	Bit 281,491: live 59,892, replay 59,893

The trace records and failed replay records are saved with their disjoint counterparts. Replay stops at the first mismatch; these are zero-based positions, not counts of all differences. The deliberate wrong-probability controls also fail as intended. The probability-cost column sums the current head's actual discretized predictions only where it executes; it is not the size of an archive produced without that head.

The math provider explains these Mac mismatches. The compressor links its own exp/log implementations from detmath.cpp; the standalone trainer used system math. A controlled diagnosis reproduces the same first mismatches even with the runtime head, then matches all 4,817,984 probabilities after linking the compressor's math code. The public trainer also matches with that change. Inputs and weights remain unchanged.

The formal v3 checks now pass on both fixtures, including deliberately wrong first probabilities. They explicitly bind the math source and its compile flags, enabling the separately frozen CPU experiment. The earlier failures remain evidence: matching source formulas and floating-point flags alone did not identify the linked math implementation. The native Linux screen and disjoint replay also pass complete archive identity, exact source restoration, matching encoder/decoder traces and every frozen integer probability (2,418,059 and 2,399,837 bits respectively). The deliberately wrong first probability is rejected on both. An earlier Linux system-math run failed much earlier, at bit 910 on the screen and 230 on the disjoint fixture. These Linux results independently support the need to link the runtime math; they do not yet test updated weights or long-stream online training.

A staged experiment

1. Verify the unchanged checkpoint export and preserve its hashes.
2. Reconstruct a writer against the current runtime. Record exactly the model bits that execute, preserve the once-per-eight-model-bits channel rhythm, and keep dictionary pretraining and metadata behavior explicit.
3. On a canonical fixture, require an unchanged archive, exact XML restoration and identical encoder/decoder traces. Replay the shipped weights and compare every corrected integer probability with the live runtime.
4. Only after replay equivalence, freeze a training recipe and compare exported candidates. Training loss is a diagnostic; complete fixture archives decide promotion. Reused fixtures do not become fresh validation by being traces.
5. Collect broader/full-stream data on persistent native Linux hardware before claiming a competitive improvement. Record any revised point where online learning starts, total package cost, full restoration and runtime limits.

A small warm-start protocol is prepared, gated on both fixture replay checks. It tries one screening-only pass at learning rates 0.00003, 0.0003 and 0.003, each starting from the shipped bfloat16 weights with fresh optimizer state. Selection uses frozen, exported bfloat16 cost on the screen. The selected weight hash is saved before checking that one model on the disjoint fixture; a failure does not trigger a switch to a runner-up. Both probability costs must improve by at least one byte before spending time on a real archive trial. Equal raw weight sizes still need packed executable accounting later.

The first CPU experiment is complete:

Learning rate	Frozen screen cost change	Disjoint cost change
0.00003	−11.9 B	Not evaluated
0.0003	−66.4 B	Not evaluated
0.003	−147.7 B	−91.2 B

The selected model passes both probability-cost gates. Its shape and raw 7,386-byte size are unchanged. Each training pass took about 1.9 seconds in this local run; a frozen replay took about 0.9 seconds. The expensive trace is reused across all three settings. These timings describe this small replay workload, not full compression performance.

A separate archive protocol embeds that exact model hash and changes only its weight header. The completed native Mac archive comparisons pass the predeclared promotion gate:

Fixture	Baseline payload	Trained payload	Change	Original XML
Screen	119,065 B	118,917 B	−148 B	Exact restoration
Disjoint	117,438 B	117,347 B	−91 B	Exact restoration

The native Linux screen also improves from 119,067 to 118,931 bytes (−136 B), and the native Linux disjoint fixture improves from 117,426 to 117,333 bytes (−93 B). Each Linux pair was built and run serially on one Ubuntu x86-64 host; both restore the original XML and match the baseline's context counters. Mac and Linux baselines differ by a few bytes, so only same-platform pairs are compared.

All context counters match their baselines. Both Mac development binaries are 873,096 bytes. The model was selected on the screen before the disjoint archive was tested. That second fixture has been used in earlier research, so it is a useful generalization check rather than untouched validation. Both native Linux fixture gates now pass. The packed Linux release comparison also reproduces the published baseline exactly and finds a 16-byte smaller comp9 compressor plus a 16-byte smaller independent decoder prefix. The external transformer-weight, hint and order blocks are identical, so measured release components shrink 32 bytes. A longer stream and the complete enwik9 archive remain the next gates; these savings are far below a demonstrated full-corpus prize margin.

A frozen extra-pass experiment continued the one-pass checkpoint on the same screen trace. At 16 passes the screen's frozen cost improved another 282.3 bytes, but the disjoint cost worsened by 90.6 bytes against the one-pass model. The selected hash was saved before that disjoint check; the predeclared gate rejected it, and no alternative checkpoint was tried on the disjoint fixture. This is direct evidence of small-trace overfitting. Broader training data is a better next step than more epochs on this fixture.

The next input is a separate four-megabyte training range: 416 complete pages from enwik8 bytes 20,013,121–24,015,644. Its preparation and original-XML restoration are verified. The completed paired trace preserves the unchanged 476,220-byte archive, restores exact source XML and matches the encoder and decoder records for 9,746,875 model bits. Frozen replay now matches every live integer probability and rejects a deliberately wrong first probability. A one-pass CPU training protocol was frozen before candidate training. The manifest and hashes are versioned; the raw and prepared files remain locally reproducible. Any trained model will still need frozen screen selection, one disjoint check, real archives and native Linux verification. This range broadens training coverage without making the old evaluation fixtures into new training data.

The first broader-training result did not pass that selection gate. Three one-pass BF16 models trained from the shipped weights all beat those weights on the old screen, but the best still costs 22.9 more bytes than the prior screen-trained checkpoint. The protocol therefore stopped before the disjoint fixture or a changed archive. The prior checkpoint was trained on that screen, which favors it in this comparison; a later transfer experiment needs an independent disjoint win.

A small predeclared interpolation experiment then mixed those separate-range weights with the prior screen-trained weights without changing model size or runtime. The selected 25% blend improves frozen model-bit cost by 8.6 bytes on screen and 31.3 bytes on disjoint versus the prior checkpoint. Selection happened before the disjoint replay; only that one blend was tested there. The real archive protocol is the next gate, because replay cost is not compressed archive size.

The frozen model then passed actual native Mac screen/disjoint and native Linux screen/disjoint archive trials. Relative to the prior checkpoint, it saves 9/32 bytes on Mac and 11/30 bytes on Linux. All variants restore exact XML and keep identical context counts; development executable sizes are unchanged. The native Linux packed-release comparison then reproduced the exact published baseline and prior releases. The selected blend saves another 17 compressor bytes and 20 decoder-prefix bytes versus prior, or 37 measured component bytes including the unchanged duplicate transformer-weight block. Its comp9 SHA-256 is d448abd6ceb1c143aacc1793d119b01700e863fa86bcc081531790179eb96f39. All four upstream packer checks passed for each build, and external weights, hints and order blocks are byte-identical. The source, model, package and log hashes are saved with the comparison. This is still a small fixture and release-component gain; full enwik9 payload, self-extracting decode and official scoring remain open.

The fixed blend also passed a third native Linux article range: 125,879 payload bytes, versus 125,905 for the prior checkpoint and 126,001 untouched. All variants restored exact original XML and kept identical contexts. This range was already used for other research; it is a supporting check, not untouched final validation or a full prize score.

To check length scaling, we froze a separate 4 MB article range before compression. The native Linux comparison produced 489,618/489,418/489,292 payload bytes for untouched, prior and blend. The blend saves 126 bytes versus prior and 326 versus untouched on 4,017,614 raw XML bytes; all three restore exact XML with identical context counts. A straight-line extrapolation of the 126-byte increment to 1 GB is about 31 KB, but full-stream behavior can differ. This model-only gain is promising as a controlled improvement and remains too small to carry a prize entry by itself. The full enwik9 archive, decoder and resource checks are open.

Before training on that range, a frozen native Linux comparison tested the existing one-pass checkpoint there. It reduced the 4 MB range's payload from 476,220 to 475,970 bytes (250 bytes saved); both variants restored every original XML byte and kept identical context counts. This extends evidence for the fixed model, but it does not predict the complete enwik9 package. Once this range trains a new checkpoint, it cannot also validate that checkpoint.

The fixed one-pass checkpoint was also measured once on a third native Mac range: 125,991 baseline payload bytes versus 125,896 with the trained weights (95 bytes saved). Both variants restore the exact 1,000,794-byte original XML and keep identical context counters. This range had been used for an unrelated retrieval test, so the result supports generalization but does not count as fresh final validation or a full prize archive.

The training-row dump is a different dataset: it cannot replace the final mixer's bit-level inputs. A changed transformer also requires a new bit-model trace, because its features and upstream predictions change.