perf: §12.1's byte-identical levers, built behind flags and bound for measurement

[justin13888]

Summary

Implements spec/PERFORMANCE.md §12.1's byte-identical acceleration levers behind flags, proves each one reproduces the shipped bytes, gives the levers inside the old quantize_and_pack residual their own stage! marks, adds a perf arm for every lever, and binds a new §12.4 of measured speedups through verify:benchmark. The speedup cells stay TBD: they need a quiet-host bounded run, which is split off as #95.

The levers (rust/src/), each a Tunables::accel_* flag, off in Tunables::DEFAULT:

§12.1	flag	where
1, 2	accel_quant_table	encode.rs (AcQuantJob::accelerated/width/deq/quant, a table per bit width keyed on (bits, index)), mulaw.rs (mu_compress_by, compand_quantize_hoisted, dequantize_table)
3	accel_dct_lanes	dct.rs dct_encode_lanes: four coefficients per pass, each lane summing in the scalar order with the scalar term; portable Rust, no intrinsics
4(a)	accel_gamma_lut_cache	decode.rs cached_gamma_lut (OnceLock, one table per transfer curve), used by the render and by average_color
4(b), layout half	accel_flat_cos	dct.rs precompute_cos_table_flat/dct_decode_pixel_flat, decode.rs render_flat
5	accel_sse_early_exit	encode.rs sse_with_delta(…, stop_at)
6	accel_fused_pixels	encode.rs analyze: one tiled pass for linearize + OKLAB + the alpha reduction; the alpha plane is built only for an image with alpha
7	accel_word_bitpack	bitpack.rs write_bits_word

• rust/src/constants.rs: the seven fields and their DEFAULTs.
• The shipped path is held to master's timing, not only its bytes. Measured stage by stage against master's own build on the same host, two lever refactors had moved it: the fused-pass rewrite shortened the linear-RGB planes' lifetime, so later buffers reused freed memory and a 512×512 encode got ~12% faster with every lever off; and the flat-table loop, sharing a function with the shipped render loop, made a tier-4 decode ~12% slower. Both are fixed (analyze allocates and frees in master's order; render_flat is its own #[inline(never)] function), and the shipped path now matches master within noise at 512×512 stage by stage, at 100×100 tier 1 and tier 4, and on tier-1 and tier-4 decode.
• Stage marks (encode.rs): dc_search, ac_quantize, refine, pack now split what quantize_and_pack lumped together. rust/examples/bench_stages.rs prints the remaining residual as unmarked, and bench_decode_stages.rs's comment follows.
• Tests: rust/tests/accel_levers.rs (every encode, decode and capped vector with each lever alone and all on, plus a differential sweep under the shipped and eight non-default Tunables), with a [[test]] entry in rust/Cargo.toml gating it on spec-vectors like spec_vectors; library tests in encode.rs, decode.rs, dct.rs, mulaw.rs, bitpack.rs so the mutation sweep holds every lever; and goldens produced by master's encoder for the lines the levers re-plumbed (band gains, CfL, refinement with a window, shared chroma scale).
• rust/.cargo/mutants.toml: three equivalent mutants excluded, each anchored to its line (the lanes DCT's scale floor; the cfl_bits > 0 guard around two zero-width writes; and replace + with - in quantize_one at encode.rs:720:30, the ac_nearest search over the symmetric offsets {0, ±1, ±2}, where negating the offset scans the same five candidates in another order and the strict < makes order matter only on an exact tie — the reasoning of the existing select_dc_codes entry). That third mutant is not in new code: master has the same line at encode.rs:526 with no exclusion; it enters the incremental sweep here because this PR re-plumbs quantize_one (job.quant/job.deq/job.width) and so puts the line in the diff. The existing mu_compress exclusion's note now says it also covers mu_compress_by.
• rust/examples/encode_stdin.rs and tools/comparison/src/verify-sweep-labels.ts: the seven tune keys, as constants.rs requires.
• tools/comparison/src/perf/matrix.ts, run.ts: the encode levers join TUNE_ARMS (the early exit priced against refine_passes=1), TIER4_ACCEL_ARMS prices the tier-dependent ones at tier 4, and DECODE_ACCEL_ARMS prices the decode levers at tier 1 natural and tier 4 natural and capped, each against a shipped cell in the same block.
• tools/comparison/src/verify-benchmark.ts: §1's new rows; §12.4's three tables, bound with R.us so an unmeasured cell fails as missing rather than passing as unbound; prose bindings for every §1 figure the text now quotes.
• tools/comparison/baselines/perf-stages.json: §1 re-recorded at 4d68c1f with the new marks. Recorded twice back to back and kept only because the pair agreed (every share within 0.4 points, every total within 2.2%); an earlier recording taken during a load burst did not agree and was discarded.
• tools/benchmark/record_stages.py (+ test): writes float exponents the way biome format does (1.8e-6, not 1.8e-06) — the new refine share needed one and format:check:compare rejected the recorder's own output; the encode residual is named unmarked in its comment and fixture.
• spec/PERFORMANCE.md: §1's table, provenance and prose re-derived from the new recording; the figures quoted from it in §5, §6, §10 and §12 updated; §12 and §12.1's prose rewritten now that ac_quantize is measured rather than bounded; §12.3 updated; new §12.4 (what each flag builds, what checks it did not move a byte, and three bound tables of speedups); §0's banner names §12.4's 34 cells among the gate's failures and corrects the checked-value count to 216 (it read 180; the gate reported 204 on master and 216 here).
• TESTING.md, .github/workflows/ci-comparison.yml: the lever tests listed, and the gate's failure count updated the same way (TESTING.md's heading now says 40 cells, six plus §12.4's 34, as its paragraph does).
• spec/PERFORMANCE.md §12.1 and §12.2: every file.rs:line citation re-anchored to this head, since the levers moved the code they name (e.g. average_color's gamma-LUT build decode.rs:546 → 660, mu_compress's ln(1 + mu) mulaw.rs:7 → 6, dct_decode_pixel_separable's cosine read dct.rs:269 → 370, the separable prototype dct.rs:296 → 424). Each citation names the shipped (flag-off) site.

Informal speedups, not published. On this host (load average ~2.5 of 16 threads), Rust only, min of 7 reps × 2 rounds, bench-encode with CHROMAHASH_TUNE: at 100×100 tier 1, accel_quant_table 1.48×, accel_dct_lanes 1.91×, all levers 6.28×; at 512×512, accel_dct_lanes 7.28×, all 7.51×; at 100×100 tier 4, accel_quant_table 1.57×, accel_dct_lanes 2.11×, all 9.06×; accel_fused_pixels 1.04× and accel_word_bitpack ≤1.02× everywhere; the early exit 1.15× on refine_passes=1. These are here to size review attention only. They are not in the document, because §0's rule is that a wall-clock figure is published from a committed run on a host that holds the 10% bar; #95 is that run.

Validation

Re-run at d618396 (the head after the review repairs, which touch only spec/PERFORMANCE.md and TESTING.md): cargo fmt --check, cargo clippy --all-targets --features full -- -D warnings, cargo test --features full, cargo build/test --no-default-features, the comparison tooling's format:check/lint/build, metric-selftest, verify-claims, verify-experiments --strict, verify-sweep-labels, corpus-licenses --check, spec/validate.py, check-versions.sh, and the tools/benchmark ruff and unittest rows all pass; verify-benchmark fails with the same 40 placeholder disagreements and 216 checked values as below; the mutation sweep is as below. rd-gate, determinism-check, the lever on/off hashing and the timing comparison were not re-run for the repairs, which change no code. The rest of this section is the original validation, at a4c5df7.

• cargo fmt --check (rust/) — pass
• cargo clippy --all-targets --features full -- -D warnings — pass (also with bench-internals)
• cargo test --features full — pass, every test binary (159 library tests, the 4 in accel_levers, and the existing suites)
• cargo build --no-default-features / cargo test --no-default-features — pass
• cargo check --target wasm32-unknown-unknown (rust/) — pass; aarch64-unknown-linux-gnu — unavailable (target not installed here)
• cargo mutants -d rust --in-diff <git -C rust diff --relative origin/master> -j 4 (the ci-mutants.yml incremental job, --no-shuffle), re-run at d618396 with all three exclusions in force — 417 mutants tested in 40m: 412 caught, 5 unviable, 0 missed, 0 timeouts. The excluded replace + with - in quantize_one is absent from the tested set; its sibling at the same site, replace + with * in quantize_one (encode.rs:720:30), is caught.
• python3 spec/validate.py — pass; tools/ci/check-versions.sh — pass
• python3 -m unittest discover -s tools/benchmark -p 'test_*.py', uvx ruff format --check tools/benchmark, uvx ruff check tools/benchmark — pass
• mise run build:compare, lint:compare, format:check:compare — pass
• node dist/metric-selftest.js, verify-claims.js, verify-experiments.js --strict, verify-sweep-labels.js, corpus-licenses.js --check — pass
• node dist/verify-benchmark.js — fails, as expected and as on master: 40 disagreements, all placeholders — master's six (§2 tier 3/4 encode, §3's 128/1024 rows) and §12.4's 34 — with 216 values and 28 prose figures checked and agreeing. CI runs this step with continue-on-error.
• node dist/rd-gate.js — PASS, 0.00% on all eight photos.
• node dist/determinism-check.js — PASS (both runs agree byte for byte; the re-run reproduces results/synthesis-window.json).
• All seven levers on vs off, hashed with encode_stdin: the eight rd:gate photos at tiers 0–4 on their encoder-input size and tiers 0–2 on their display-size reference, 64/64 identical; the perf fixtures (gradient, noise, solid) at 100², 256², 512² and 1000², tiers 0–4 (0–2 at 512 and up), 48/48 identical. rd:gate itself cannot take a flag — its adapter strips CHROMAHASH_TUNE by design — so identical hashes are its 0.00%.
• Shipped path vs master's own build, same host, bench_stages/bench_decode_stages: within noise on every stage (figures under Summary).

Coverage gaps, stated:

• No binding exposes Tunables, so the other eight languages' vector suites exercise only the shipped path; a lever reaching them is the future change that makes it the default, and they would then run the same vectors accel_levers.rs already holds every lever to.
• The early exit's row-granular stop is proven equivalent by argument and by output comparison; no test observes how early it stops beyond early_exit_stops_only_at_the_bound.
• 4(b)'s render-loop vectorization is not built (see Decisions).

Risks and rollout

None for callers: every flag is off in Tunables::DEFAULT, no binding can set one, and the shipped path's bytes are pinned by the unchanged spec vectors and its timing was checked against master. Making any lever the default is a later, separate change (a patch release under §12.1), which #95's measurements are for.

Issue

Closes #81. Its steps 3 and 4 — measuring on a quiet host and restating §12.1 from the measurement — are #95, which depends on this.

Decisions taken

1. How the levers are switched: one Tunables bool per lever, off by default, versus a bitmask field or making them unconditional now.
   Taken: seven accel_* bools, false in DEFAULT - each is its own perf arm and its own tune key, and the shipped path stays the reference every lever is timed against, as perf: measure §12.1's byte-identical levers as implemented speedups #81 asks ("behind a flag").
   Rejected: unconditional - would change the shipped encoder before any lever is measured on a quiet host; a bitmask - the repository's precedent (dct_separable) is a named bool per knob, and a mask's arm labels would not say what they measure.
   Reverses: set the chosen fields to true in Tunables::DEFAULT (and drop the flag once adopted).
2. Items 1 and 2 share one flag, accel_quant_table.
   Taken: one flag - §12.1 says the two "have to land together" because the per-index width walk item 2 hoists is the key item 1's table needs.
   Rejected: separate flags - would allow the index-keyed table §12.1 warns dequantizes the high band wrongly.
   Reverses: split the field and gate AcAccel::bits and deq on separate flags.
3. How item 3 is vectorized: portable Rust with four independent accumulators, versus hand-written backends on simd/'s SimdF64.
   Taken: portable Rust - the compiler may map lanes to registers, and IEEE lane arithmetic does not depend on whether it does; there is no intrinsic to contract into fmadd and no hand-written regrouping, the two hazards §12.1 names.
   Rejected: SimdF64 backends - four hand-written modules outside this change's scope, each carrying exactly those hazards, for a gain not yet measured against the portable form.
   Reverses: add a dct_encode_lanes_with::<V: SimdF64> and dispatch it like oklab_forward_batch.
4. Which of §12.1's levers are taken forward: all seven, except the render-loop SIMD half of 4(b).
   Taken: build 1–7 and 4(b)'s layout half; 4(a) included although the plan's list omitted it, since it is in §12.1 and in decode.rs.
   Rejected: vectorizing decode's colour conversion now - §12.1 itself says it needs the differential tests simd/ runs for encode, which decode has no harness for.
   Reverses: file it as its own issue alongside a decode differential harness.
5. Whether the shipped path may get faster as a side effect: keep its allocation order and codegen exactly as master's, versus keep the ~12% 512×512 encode gain the fused-pass refactor produced with every lever off.
   Taken: exactly as master - measured stage by stage against master's build; the fused lever carries the allocation saving, where it is attributed and timed.
   Rejected: keeping the side effect - an unflagged, unmeasured change to the reference would make every lever's speedup relative to an encoder nobody ships.
   Reverses: shorten the lin_*/alpha_pixels lifetimes in analyze's unfused branch.
6. The lever timings: publish figures from this host now, versus bind the cells and defer them.
   Taken: bind all 34 cells as TBD, fail the gate on them, and split the quiet-host run off as perf: fill §12.4's lever speedups from a quiet-host bounded run, and restate §12.1 from them #95 - §0's rule is that wall-clock figures come from a committed run on a host that holds the 10% bar, and this host was under intermittent load (the user anticipated this deferral).
   Rejected: publishing this session's informal ratios - no committed run backs them, and §0 forbids an untraceable figure.
   Reverses: run §0's bounded pair at this head and verify:benchmark -- --fix.
7. §1's stage shares: re-record now with the new marks, versus leave the old recording against code that no longer emits its row set.
   Taken: re-record, twice back to back, keeping the second only because the pair agreed within 0.4 points and 2.2% - shares are what the issue needs to size items 1, 2 and 7, and the repository treats in-process shares as readable where wall clock is not, which a discarded load-burst recording showed is true only with a guard; the totals row is kept, and lands within 0.6% of the quieter e53e6cd recording at 512×512.
   Rejected: keeping the e53e6cd file - its quantize_and_pack row names a stage the recorder no longer emits, so the next re-record would break §1 without warning.
   Reverses: git checkout origin/master -- tools/comparison/baselines/perf-stages.json and restore §1's prior rows.
8. perf: measure §12.1's byte-identical levers as implemented speedups #81's "rd:gate at 0.00% drift" for a lever: rd:gate strips CHROMAHASH_TUNE, so it cannot run with a flag.
   Taken: hash the same eight photos with all levers on and off (64/64 identical) and run rd:gate on the default build - its figure is a function of the hash, so identical hashes are 0.00%.
   Rejected: teaching rd:gate to pass a tune - it strips the variable deliberately, so no stray override can move a gated number.
   Reverses: add a --tune option to rd-gate.ts that sets CHROMAHASH_TUNE explicitly.
9. Mutants the levers make unkillable by output: exclude them by line, versus restructure.
   Taken: restructure where possible (explicit match arms, no redundant guard, bounded for loops, goldens from master for the re-plumbed gain/window lines) and exclude only the three that are equivalent by construction, each anchored to its line as the file's own l_hi entry is: replace < with <= in dct_encode_lanes (dct.rs:347:14, the scale floor no input lands on exactly), replace > with >= in encode_with (encode.rs:1582:19, two zero-width CfL writes at the shipped cfl_bits = 0), and replace + with - in quantize_one (encode.rs:720:30, the symmetric ac_nearest offsets, equivalent for the same reason as the existing select_dc_codes entry; pre-existing on master at encode.rs:526, brought into the incremental sweep by this PR's edit of quantize_one). The third rests on an argument (no exact reconstruction-error tie on the µ-law grid between q - d and q + d), not on a test that asserts that premise.
   Rejected: unanchored patterns - mutants.toml warns they silently widen an exemption.
   Reverses: delete the three exclude_re entries.
10. The recorder's float spelling: normalize exponents in record_stages.py, versus exempt the baseline from biome format.
    Taken: normalize in the recorder, anchored to whole numeric values - the committed file must be byte-for-byte the recorder's output and pass format:check:compare.
    Rejected: a biome ignore - would stop checking a committed file.
    Reverses: revert _PADDED_EXPONENT and add the file to biome's ignore list.