merkle_path: single pooled Lagrange denominator inverse (4 Fermat inversions → 1 constant)#10
Draft
SipengXie2024 wants to merge 1 commit into
Draft
Conversation
In char 2 all four degree-3 Lagrange denominators over {0, 1, w, w+1}
collapse to the same constant w*(w+1): each is a product of the three
nonzero pairwise differences, which are {1, w, w+1} in every case.
Replace the four per-call Fermat inversions in lagrange_eval_degree3
with one precomputed inverse (OnceLock), and fold the four L_p(r)
multiplies into a single scaled sum. Field ops are exact, so values are
bit-identical to the reference.
lagrange_eval_degree3 runs once per slot per sumcheck round in
verify_merkle_path_shift, so verifiers of many shift proofs pay it
heavily: on a batched Merkle-path workload (10k proofs) this cut the
shift-verify phase from 578 ms to 214 ms.
Tests: inverse-identity check + bit-equality against the unfactored
reference implementation. Full flock-prover + flock-core suites pass.
tamirhemo
added a commit
that referenced
this pull request
Jul 17, 2026
* field/gf2_128: x86_64 PCLMULQDQ multiply path Port the x86_64 CLMUL multiplication variants from flock-avx (flat-src commits f414709 + 70fe6f6) onto the crates/ workspace layout. Adds a `pub mod x86_64` mirroring the existing `aarch64` module — schoolbook, binius, karatsuba, karatsuba+barrett, and deferred-unreduced products built on `_mm_clmulepi64_si128` — and routes the F128 `Mul` and `ghash_mul_unreduced` dispatch to it under `target_feature = "pclmulqdq"`. Default x86 mul is karatsuba+barrett (throughput win on Zen4). Validated: cargo check on x86_64-apple-darwin (AVX-512 features) and the all_x86_variants_agree test under Rosetta. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * field/gf2_128: x86_64 AVX-512 VPCLMULQDQ 4-lane primitives Add the shared AVX-512 GF(2^128) primitives from flock-avx (commits 2b4e14d + df9616e) into the x86_64 module: `ghash_mul_x4` (4 independent lane-parallel products via `_mm512_clmulepi64_epi128` + two-stage 0x87 reduction), and the `WideGhashX4` 4-lane deferred-reduction accumulator (`zero`/`mul_acc`/`fold`) with the `f128x4_loadu`/`f128x4_set` pack helpers. These are the cross-file hub consumed by the AVX-512 NTT butterfly, ligerito LSB fold, and multilinear round-2 fold. Gated behind `target_feature = "vpclmulqdq"`. Tests ghash_mul_x4_matches_scalar and wide_ghash_x4_matches_scalar_deferred cross-check every lane against the scalar path. Validated with cargo check on x86_64-apple-darwin. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ntt/inv_table: x86_64 SIMD apply (SSE2 + AVX-512) Port the x86 apply kernels for the prover's hottest inv-NTT table from flock-avx (commits 63bb122 + d52b691). Adds `apply_x86_unchecked` (SSE2: 16-byte `_mm_loadu/xor/storeu_si128`, odd-b half-swap via `_mm_shuffle_epi32::<0x4E>`) and `apply_x86_avx512_unchecked` (widens the same kernel to 512-bit with `_mm512_shuffle_i64x2` lane permutes), wiring the `apply` dispatch to pick AVX-512 (ell==64), then SSE, then scalar on x86_64. Validated: x86_64-apple-darwin cargo check plus the apply_x86_matches_apply_scalar test (SSE path) under Rosetta. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * lincheck: x86 register-tiled partial_fold_z Port the x86 register-tiled single-matrix partial fold from flock-avx (commit 655ba15). Adds `process_block_x86` (SSE2 8-accumulator tile kept in xmm registers, 128 bytes/iter — the x86 mirror of process_block_neon_single) and the rayon driver partial_fold_packed_z_x86_tiled_padded, wired into partial_fold_packed_z_best's non-aarch64 branch (x86_64 → tiled, other → scalar fast path). main's evolved NEON oblock/iblock path is untouched. Validated: both targets cargo check + the partial_fold_x86_tiled_matches_serial test under Rosetta. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * zerocheck/univariate_skip: x86_64 GFNI + AVX-512 VBMI round-1 kernels Port the two x86 round-1 hot kernels from flock-avx (commits 1e443f0 + 3599d03). Adds `shift_reduce_inner_ab_x86` (GFNI: `_mm_gf2p8mul_epi8` on the hardwired 0x11B AES polynomial — exactly gf2_8's field — folding Σ_k gf8mul(a_k·b_k, x^k) with no widening/reduction) and `bit_transpose_64bytes_avx512` (VBMI: `_mm512_permutexvar_epi8` byte-gather + 3 masked bit-swap rounds), wired into the shift_reduce_inner_ab and bit_transpose_64bytes dispatchers alongside the NEON paths. No gf2_8 helper dependency — GFNI supplies the field mul directly. Test x86_gfni_inner_matches_scalar_inner cross-checks against the scalar oracle. Both targets cargo check clean; GFNI/VBMI aren't executable under Rosetta so runtime validation is deferred to x86 CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ntt/additive_ntt_f128: enable parallel interleaved NTT on x86_64 Port commit 1428b55: broaden the cfg gate on the rayon + cache-blocked + layer-fused interleaved forward NTT (the RS-encode core of pcs::commit) from aarch64-only to any(aarch64+aes, x86_64+pclmulqdq). The parallel path and its three butterfly helpers are portable F128 arithmetic (no arch intrinsics — an explicit NEON lane-batch was previously removed as it blocked compiler ILP), so widening the gate brings multithreading + cache locality to x86, backed by the ported x86 F128 mul. On x86 the three butterfly helpers are no longer dead code. Validated: both targets cargo check + interleaved_matches_per_lane under Rosetta. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ntt/additive_ntt_f128: AVX-512 VPCLMULQDQ SoA butterfly + fused-4-layer Port commits 2b4e14d + d782bb3. Factor the interleaved butterfly's inner per-lane loop into `butterfly_row_pair` / `butterfly_fused_2layer_rows` dispatchers and add their x86 AVX-512 kernels (`butterfly_row_pair_avx512`, `butterfly_fused_2layer_avx512`) which multiply 4 SoA F128 lanes per VPCLMULQDQ via gf2_128::x86_64::ghash_mul_x4 (broadcasting the shared twiddle once). Add the fused-4-layer top-butterfly path (`fused4_one_r_avx512` + `butterfly_interleaved_fused_4layer_par_rows`, gated by `fused4_ok`) that applies four NTT layers in registers per pass. On non-vpclmulqdq targets every dispatcher falls back to the original scalar per-lane loop, so aarch64/scalar behaviour is byte-identical. Validated: all 11 native NTT tests pass (incl. interleaved_parallel_ matches_scalar exercising the refactored dispatchers), Rosetta fallback green; the AVX-512 path is compile-checked (not Rosetta-executable) and covered on real x86 by the now-widened interleaved_parallel test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * pcs/ring_switch: fused 2-way 16-wide fold on x86_64 Port commit 04557b5. Add `fold_1b_rows_2way_mfr_16wide_padded`, a portable rayon method-of-four-Russians fold that shares the byte-gather and bit-transpose across both k=2 suffix tensors in one scan. The k=2 16-wide dispatch in fold_1b_rows_multi_padded now picks the fused 2-way scan on x86_64 and keeps the two independent 1-way scans on other targets (measured better on M-series, where the fused kernel's register pressure ate the win). No intrinsics — the arch split is a pure tuning choice. Test mfr_fold_16wide_2way_matches_scalar passes natively. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * pcs/ligerito: AVX-512 + NEON initial-sumcheck LSB fold Port commits 8ec54f4 + 5727ba1 (+ the ligerito visibility hunk of 8853c35). Add pub(crate) `fold_lsb_avx512` (deinterleaves even/odd 128-bit lanes of two `__m512i` loads via permutex2var, then even + r·(even+odd) — 4 lanes/step via ghash_mul_x4) and `fold_lsb_neon` (ghash_mul_vec2_neon), and dispatch the parallel initial-sumcheck fold in fold_and_msg_lsb to AVX-512 on x86, NEON on aarch64, scalar elsewhere. Exposed pub(crate) so multilinear can reuse them for the rounds3+ fold. Validated: fold_lsb_neon_matches_scalar + all 45 ligerito roundtrip tests pass natively (fold_and_msg_lsb now runs the NEON fold end-to-end); AVX-512 path compile-checked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * zerocheck/multilinear: AVX-512 round-2 accumulator + delegate rounds3+ fold Port commits df9616e + 8853c35. Restructure fold_and_compute_round_pair_into to split the LSB fold from the message pass: the rounds3+ fold now delegates to the shared ligerito::fold_lsb_avx512 (x86) / fold_lsb_neon (aarch64) / scalar kernels, and the round-2 message accumulation uses the 4-lane deferred-reduction WideGhashX4 accumulators (f128x4_loadu / f128x4_set / mul_acc / fold) on x86, scalar elsewhere. main's prior fold_and_compute_round_pair_into was byte-equivalent to flock-avx's pre-restructure version, so this is a faithful forward port. Validated: all 62 zerocheck tests pass natively (incl. the round-pair padded-vs-dense and end-to-end prove/verify roundtrips exercising the NEON fold path); AVX-512 path compile-checked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf: size the prover thread pool to physical cores on Linux Port commit 581fee1. macOS already sizes rayon's global pool to physical P-cores (hw.perflevel0.physicalcpu); Linux used available_parallelism(), which counts SMT siblings. The hot prove phases are CLMUL-port-bound / DRAM-bandwidth-bound, so 2 threads per physical core only add contention (measured ~16% slower MT prove at 64 vs 32 threads on a 32C/64T Threadripper; the commit-NTT regresses outright past physical-core count). Derive physical cores from /sys topology (unique physical_package_id, core_id) on Linux; RAYON_NUM_THREADS still overrides. No effect on macOS (unchanged path) — verified the native build; the Linux /sys function was type-checked on the macOS target (std-only). This is the MT-scaling half of the x86 support: the SIMD kernels make single-thread fast, this keeps multi-thread from wasting half the cores on SMT siblings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ligerito: x86-gated scratch-pool reuse for initial-sumcheck fold buffers Take the fold_and_msg_lsb outputs from the prewarmed scratch pool and give the consumed buffers back in SumcheckProver::fold, so the initial sumcheck reuses resident pages instead of faulting ~128 MB of fresh memory each round. This is the buffer-pooling half of the measure/portable-perf combo (the induce_sumcheck_poly parallelization is intentionally omitted). Gated to x86_64: on aarch64 this pooling measured slower, so there the original fresh-alloc + direct-move path is kept unchanged. At m=30 (2^16 compressions, MT) this halves pcs::open (95.9 -> 47.4 ms) and takes the full prove from 203 -> 152 ms (322k -> 432k compressions/sec), at a ~50% x86 peak-memory rise from pool retention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: cargo fmt + clippy fixes for the rebased x86 port Formats the ported x86 code to satisfy main's new fmt/clippy CI gate, removes two duplicated doc-comment blocks in lincheck.rs left over from the forward-port, converts manual assign ops to += in the NTT butterfly fallbacks, and renames the 0x16_16_16 test seed to 0x161616 (clippy mistyped_literal_suffixes). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fixes * perf: fuse AVX-512 zerocheck kernels * perf: vectorize x86 zerocheck round two * perf: interleave x86 Merkle hashing * perf: fuse x86 zerocheck fold messages * perf: align inverse NTT table rows --------- Co-authored-by: Ron Rothblum <rothblum@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SipengXie2024
marked this pull request as draft
July 20, 2026 06:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
In char 2, all four degree-3 Lagrange denominators over the evaluation set
{0, 1, ω, ω+1}collapse to the same constantω·(ω+1): each denominator is the product of the three nonzero pairwise differences, which are{1, ω, ω+1}in every case.lagrange_eval_degree3computed four per-call Fermat inversions (127-squaring chains each). This PR replaces them with one precomputed inverse (OnceLock) and folds the fourq_p · L_p(r)multiplies into a single scaled sum. Field ops are exact, so outputs are bit-identical.Why it matters
lagrange_eval_degree3runs once per slot per sumcheck round insideverify_merkle_path_shift, so a verifier checking many shift proofs pays it heavily. On a batched SHA-256 Merkle-path workload on our fork (~10k shift proofs per verify), the shift-verify phase dropped 578 ms → 214 ms.Tests
New: inverse-identity check (
lagrange_inv_denom · ω(ω+1) == 1) and bit-equality of the factored evaluation against the original unfactored reference. Full upstream flock-prover (83) + flock-core (255) suites pass.