DHOOM: The Wire Format That Knows the Zero Section
Part II — The Engine Room · seven builds, seven receipts
The chapter's experiments are byte-level: you will run the shipped encoder/decoder, count bytes, hex-dump sentinels, and prove the format does what BLD-DHOOM-* claims it does. Receipts are wire bytes and passing test names, not prose.
Run it live — GQL console
Point at the public read-only instance (no key needed) or your own engine
(start it with GIGI_CORS_ORIGIN=* for browser access — dev only).
—
E6.1Reproduce the 292-to-113
Build
Pull gigi at b3f7d90, locate test_roundtrip_sensors in src/dhoom.rs (around line 2128), and write a tiny harness that loads the same three-record fixture, runs encode() and serde_json::to_string() on it, and prints (json_bytes, dhoom_bytes, ratio).
Receipt
Harness output reads 292, 113, 2.58 and the four-line accounting in the chapter (88+5+5+15 = 113) reproduces line-for-line; the harness also asserts decode(encode(d)) == d.
Bonus
Mutate one of the three records' unit from celsius to kelvin and watch the encoder demote the |celsius default — record the new ratio.
E6.2Stress the majority gate
Build
Generate fiber fixtures of \(N \in \{2, 3, 5, 11, 99\}\) records where exactly \(\lceil N/2 \rceil\) share a value in one column and the rest disagree. Run the shipped encoder on each and diff the resulting header.
Receipt
The column gets promoted to a | default at \(\lceil N/2 \rceil + 1\) matches and stays plain at \(\lceil N/2 \rceil\), exactly as src/dhoom.rs lines 1079–1084 (BLD-MODCAT-DEFAULT-FREQ-ORDER) require. Save the headers as a fixture file.
Bonus
Build a three-default fiber and verify the headers are sorted by descending match-count (frequency sort at line 1135).
E6.3Hexdump the sentinel
Build
Forge the three-row posts fiber from the sentinel section — distinct tags arrays per row, embedded commas in some of them — encode it, write the bytes to disk, and xxd the first cell.
Receipt
Bytes 22 1f 5b 22 22 ... appear in the dump, matching the chapter's hex panel byte-for-byte (BLD-DHOOM-X1F-SENTINEL). The 0x1F is the second byte of the cell.
Bonus
Replace one row's tags with a constant array shared across all three rows and watch the sentinel vanish — the encoder promotes the array to a header default and bypasses the inline path. The hexdump now contains no 0x1F anywhere.
E6.4Replay the UTF-8 storm
Build
Check out 52b99bc (the trigger commit, before the hotfix), build it, and run my regression fixtures — café123, naïve42, the CJK and emoji cases — through parse_string_pattern.
Receipt
The pre-hotfix build panics with a UTF-8 boundary error on at least one of the fixtures, exactly as the comment at src/dhoom.rs:281-288 memorializes. Re-checkout fcc74fa, rerun, and the panic is replaced by a clean arithmetic refusal or a T-001-style suffix split; my regression test now passes.
Bonus
Add a five-byte variation-selector emoji case to the regression and confirm the fix survives. Open a PR with the new fixture.
E6.5Benchmark the streaming encoder against the schema lock
Build
Build a synthetic bundle of 50{,}000 records where the first 100 records share modal value X on one column and the remaining 49{,}900 share modal value Y. Pipe it through StreamingDhoomEncoder (one of its four call-sites in src/engine.rs) and measure (a) elapsed time, (b) total wire bytes, (c) deviation-cell count on the column.
Receipt
The header pins column|X from the 100-record sample; the remaining 49{,}900 records each pay a :Y deviation cell. Wire bytes equal \(\text{header} + 49{,}900 \cdot \text{len}(\text{:Y}) + \text{baseline}\) to the byte. Record the number in your harness output.
Bonus
Pre-sort the bundle so Y dominates the first chunk and rerun — the deviation count flips, the wire shrinks, and you have just paid the price the chapter quotes for the streaming encoder's schema lock.
E6.6Refute interning on a column it cannot help
Build
Construct a fiber where the status column has \(N = 9\) records, \(D = 3\) distinct strings, but the strings are so short (a, b, c) that pool + indices exceed 90% of the raw bytes. Encode and inspect the header.
Receipt
The header carries plain status — no & sigil, no pool line on row two — because detect_interned (BLD-MODCAT-INTERNED-GATE, src/dhoom.rs 1430–1489) refused itself. Diff against a fiber of nine longer strings (completed/pending/failed) and watch & reappear with the pool line &status[...].
Bonus
Sweep distinct-count \(D\) from 2 to \(\lceil N/3 \rceil\) and plot the bytes saved against \(D\); mark the crossover where the 90% rule first fires.
E6.7Time the Chern rider against its asserted floor
Build
Run cargo test --features kahler --test dhoom_wire_savings -- --nocapture and capture the four assertions plus the medium- and high-dimensional savings deltas.
Receipt
The output prints compression ratios 1.33, 5.33, and 21.33\(\times\) at dimensions 2, 4, 8 against asserted floors of 1, 5, and 10\(\times\), plus 17.6% and 18.1% improvements with the 0.181 absolute floor cleared (BLD-DHOOM-WIRE-TEST-MEASURED). Median dhoom_wire_savings elapsed time sits near 0.82\,s on release.
Bonus
Bisect: re-run with --no-default-features --features kahler and confirm the default-build dhoom unit median (0.28\,s) is unchanged by the Chern path's added tests — the rider rides for free on the no-feature build.