Adopt delayed modular reduction for mixed base x extension kernels

[koko1123]

TODO.md lists "Further use delayed modular reduction (cf. Plonky3 #1592, #1597)". This issue turns that line into a concrete, chunked plan. All file:line references are against current main (f013eb3, 2026-07-22).

What the vendored field stack already has

The base layer of delayed reduction is already in-tree:

• Scalar delayed dot products: MontyField31::dot_product accumulates raw u64 products and defers to a single monty_reduce/large_monty_reduce per group (crates/backend/koala-bear/src/monty_31/monty_31.rs:247-275, with N=5..7 carry-correction variants continuing below).
• SIMD delayed-dot kernels on all three arches: dot_product_2/4/5 on NEON (crates/backend/koala-bear/src/monty_31/aarch64_neon/packing.rs:636/791/875), AVX2 (crates/backend/koala-bear/src/monty_31/x86_64_avx2/packing.rs:477/545), AVX512 (crates/backend/koala-bear/src/monty_31/x86_64_avx512/packing.rs:901/991).
• The quintic ext-x-ext multiply is already internally delayed: Mul for PackedQuinticExtensionField routes each output coordinate through PF::dot_product::<5> (crates/backend/koala-bear/src/quintic_extension/packed_extension.rs:467-471).

What is missing

The mixed base x extension paths -- which dominate the prover's inner loops -- still reduce eagerly on every product:

(a) No mixed_dot_product on the Algebra trait. The trait is empty (crates/backend/field/src/field.rs:611-624). There is an inherent NEON-only PackedMontyField31Neon::mixed_dot_product (packed x scalar-base, monty_31/aarch64_neon/packing.rs:275, used by the Poseidon MDS path), but nothing generic across arches and nothing usable from the base-slice-times-extension-slice loops below. Upstream Plonky3 has Algebra::mixed_dot_product (field.rs:699 upstream) plus the chunked dispatch introduced in Plonky3 #1573. Porting it gives every such loop a single-reduction path.

(b) The packed-quintic per-coordinate override is eager. Mul<PF> for PackedQuinticExtensionField (packed_extension.rs:490-504) does 5 fully reduced packed products per extension-times-base multiply. Upstream's #1597 binomial version (delayed accumulate across coordinates, one reduction each) ports directly with D=5.

(c) ~6 hot call sites accumulate eagerly and can be retiled into K=8 tiles feeding the delayed kernels:

• crates/backend/sumcheck/src/product_computation.rs:264-272 (sumcheck_quadratic) and :302-362 (Pass A/B of run_product_sumcheck_from_round1_delayed) -- the round-coefficient kernel of the product sumcheck, direct analog of the kernel upstream retiled in #1597.
• crates/whir/src/open.rs:798-813 -- WHIR round-0 combine: per-row w += term.0[t] * term.1 and acc += w * e accumulations, every product reduced.
• crates/backend/poly/src/evals.rs:145-152 -- eval_base_packed inner dot (acc += b * a over packed base slices), analog of Plonky3 #1574.
• crates/backend/poly/src/multilinear_utils.rs:71-81 -- finger_print_packed, hot via logup.

(d) NEON dot_product_8 does two reductions. general_dot_product for N=8 falls into the generic branch: dot_product_4 + dot_product_4, each fully reduced (monty_31/aarch64_neon/packing.rs, generic arm of general_dot_product). Upstream has a single-reduction carry-managed 8-wide version, plus the carry-critical tests from Plonky3 #1600 that pin the overflow invariants.

Prior art in this repo

The stale delayed-modular-reduction branch (087b555, 2026-03-30, now 281 commits behind) prototyped exactly this for the base x ext product sumcheck (compute_product_sumcheck_polynomial_base_ext_packed, u128 accumulators), but the code it patched has since been rewritten (rayon -> parallel, ArenaVec, delayed weight folding), so this plan re-derives it against current main rather than reviving the branch.

Proposed chunks (ordered)

1. Bench harness first. The repo currently has zero benches for these kernels (only the ignored-test poseidon benches in crates/backend/koala-bear/src/benchmark_poseidons.rs). Add ignored-test microbenches in the same style for the four hot kernels above, asserting correctness against a reference computation so they double as regression tests.
2. Algebra::mixed_dot_product with default impl + chunked dispatch (port of Plonky3 #1573 shape).
3. Delayed Mul<PF> for PackedQuinticExtensionField (port of #1597 binomial path, D=5).
4. Retile product_computation.rs round kernels into K=8 tiles over mixed_dot_product.
5. Retile WHIR round-0 combine (open.rs:798-813).
6. Retile eval_base_packed (#1574 analog).
7. Retile finger_print_packed.
8. NEON single-reduction dot_product_8 + carry-critical tests (#1600 analog), extended to AVX2/AVX512 if profitable.

Expected impact (honest framing)

Upstream Plonky3 measured about -28% on the identical isolated kernel, diluted to -1% to -2.5% end-to-end in their pipeline. Here I would expect 15-30% on the touched spans and low-single-digit E2E XMSS/s, but it compounds across recursion layers since the aggregation prover re-runs these kernels at every layer.

Risks

• Overflow invariants. The whole scheme rests on u64 lanes holding at most 4 products of (P-1)^2 before a reduction (the existing N=5..7 scalar cases already do explicit carry correction via wrapping_sub((P as u64) << MONTY_BITS)). Every retiled call site must respect the same bound; getting it wrong is silent corruption, not a crash. Chunk 8's carry-critical tests exist precisely for this.
• Per-arch divergence. NEON vs AVX2 vs AVX512 have different widths and different existing kernel shapes; a win on one arch can regress another. Each chunk gets benched per-arch before merging.

Process

I intend to implement this chunk-by-chunk (each chunk a separate PR), each gated on:

• kernel microbenches (chunk 1's harness), and
• the E2E baseline cargo run --release -- xmss --n-signatures 1550 --log-inv-rate 1 (README currently reports 1426-1481 XMSS/s at rate 1/2).

Question for maintainers: does this ordering work for you, or would you rather see a different priority (e.g. mixed_dot_product and the quintic override first, retiling later)? Also happy to rebase the ordering around in-flight WHIR work (#204/#166) if that lands first, since chunk 5 touches open.rs.