diff --git a/docs/protocol-projection-v1.md b/docs/protocol-projection-v1.md new file mode 100644 index 0000000..09c7abe --- /dev/null +++ b/docs/protocol-projection-v1.md @@ -0,0 +1,341 @@ +# zkLoRA `pedersen-projection-v1` — Protocol Note + +Status: **normative for the v3 backend; unaudited.** No Rust prover code may +change the protocol described here without updating this note first. External +review of §6 (range linkage) and §7 (padding soundness) gates the removal of +the legacy rollback hatch and any public security claims (`audit_status` +flips from `"unaudited"` only after that review). + +Notation: `p` is the order of the Pasta base field `Fp` (≈ 2^254); +commitments live in the Vesta group (`EqAffine`), whose scalar field is `Fp`. +`s = 2^scale_bits`. Integers embed into `Fp` as `v ↦ v mod p` with negative +values as `−Fp(|v|)`; `FIELD_SAFE_BITS = 250` is the global headroom bound. + +--- + +## 1. Relation + +Per batch — public `X ∈ Z^{rows×in_dim}`, `D ∈ Z^{rows×out_dim}`, scale `s`, +scaling `num/den` (`den > 0`, `num ≠ 0`); private `A ∈ Z^{rank×in_dim}`, +`B ∈ Z^{out_dim×rank}`, `U`, `mid_D`, remainders — the prover demonstrates +knowledge of integer witnesses satisfying, entrywise: + +1. `X·Aᵀ = s·U + R_u`, with `R_u[i,j] ∈ [−⌊s/2⌋, ⌊(s−1)/2⌋]` +2. `U·Bᵀ = s·mid_D + R_d1`, with `R_d1` in the same canonical interval of `s` +3. `num·mid_D = den·D + R_d2`, with `R_d2 ∈ [−⌊den/2⌋, ⌊(den−1)/2⌋]` + +The canonical interval of width `d` contains exactly `d` integers, so given +the left side, `(q, r)` decompositions are **unique**; identities 1–3 are +therefore exactly the v2 semantics `delta = round_div(round_div(round_div( +X·Aᵀ, s)·Bᵀ, s)·num, den)` per `proof_contract._div_round_to_canonical_interval`. +`raw_U`/`raw_D` never materialize as witnesses. + +**Deliberate contract change vs v2:** v2 additionally rejected witnesses whose +raw accumulators exceeded `intermediate_bits` (`_rescale`). v3 replaces that +witness-side rejection with the §8 composition checks; prover-side parity is +preserved because the Python layer self-checks every row with +`compute_delta_quantized` before building a statement. + +## 2. Public parameters and generators + +- Protocol seed `GENERATOR_SEED_ID = "zklora/v3/gen-seed/v1"` is hardcoded in + verifier code. Artifacts may never supply generators or seeds. +- `G[label][i] = hash_to_curve("zklora-v3:" + seed_id + ":" + label)(le64(i))` + using the Vesta `CurveExt::hash_to_curve` (simplified SWU; per the + pasta_curves implementation the output has no known discrete-log relations + between distinct inputs). +- Labels: witness generator vectors `w:A, w:B, w:U, w:Ru, w:midD, w:Rd1, + w:Rd2, w:cu, w:cb` (globally indexed); range bit vectors `rb:g`, `rb:h` + (length `MAX_RANGE_AGG = 2^22`, shared across aggregates); result generator + `res` (written `G0`); blinding base `blind` (written `H`). +- Security assumption: discrete log in the Vesta group; all generators are + mutually independent under the hash-to-curve random-oracle model. + +## 3. Commitments + +`VectorCommitment(v)` for `v ∈ Fp^n`: chunks of `COMMIT_CHUNK = 2^16` +coordinates; chunk `i` is `C_i = Σ_j v[i·2^16 + j]·G[label][i·2^16 + j] + r_i·H` +with chunk blind `r_i`. `C_total = Σ_i C_i` is binding for the full vector +because every global index has a distinct generator. Chunking only bounds MSM +working sets; all sub-protocols consume `C_total` and `Σ r_i`. + +Blinding: +- **Per-proof witnesses** (`U, R_u, mid_D, R_d1, R_d2, c_u, c_b`, all masks): + fresh `OsRng` scalars per proof. Two proofs over identical witnesses must + differ in every commitment (tested). +- **Manifest commitments** (`A`, `B`): stateless re-derivation at proof time + via `r = wide_reduce(blake2b_keyed(seed, "zklora/v3/blind:" + module_name + + ":" + matrix + ":" + chunk_index + ":" + commitment_nonce))`, where + `commitment_nonce` is fresh 32-byte randomness at each manifest write, + stored publicly in the entry and hashed into `adapter_commitment`. Keying + includes the module name and nonce so blinds never repeat across adapters + in one manifest nor across manifest rewrites; otherwise `C_A − C_A'` would + be an unblinded commitment to a weight difference. A leaked seed destroys + *hiding* only — binding and soundness are unaffected. + +## 4. Linear-form openings with committed results (LFO) + +Given `C_total(v)` and public weights `w`, the prover sends +`Y = ⟨v, w⟩·G0 + r_Y·H` and proves consistency with a blinded inner-product +argument in the style of Hyrax's zk dot-product (Wahby–Tzialla–shelat– +Thaler–Walfish, S&P 2018, §4.1 / Fig. 6) instantiated with the Bulletproofs +folding IPA (Bünz et al., S&P 2018, Protocol 1 with blinding): 2·⌈log2 n⌉ +points plus a constant number of scalars. Knowledge soundness is by the +standard forking extractor for the folded relation; zero-knowledge by the +blinding terms in every round. + +**Inviolable rule:** no opening ever reveals a scalar evaluation. All identity +checks operate homomorphically on the `Y` commitments. (Cleartext openings +would hand the verifier one exact linear functional of `A` per batch; across +~`rank·in_dim` batches those functionals form a solvable linear system.) + +Committed results per batch (fixed transcript order): + +| Symbol | Statement | Vector | Weights | +|---|---|---|---| +| `y_A` | id. 1 | `A` | `γ ⊗ (Xᵀα)` | +| `y_U` | id. 1 | `U` | `α ⊗ γ` | +| `y_Ru` | id. 1 | `R_u` | `α ⊗ γ` | +| `y_cu` / `y_cu'` | id. 2 consistency | `c_u` vector / `U` | `(δ⁰..δ^{R−1})` / `α ⊗ (δ⁰..δ^{rank−1})` | +| `y_cb` / `y_cb'` | id. 2 consistency | `c_b` vector / `B` | `(δ'⁰..δ'^{R−1})` / `β ⊗ (δ'⁰..δ'^{rank−1})` | +| `Y_z` | id. 2 | `⟨u*, b*⟩` | two-vector IPA, length `R` | +| `y_M` | id. 2 & 3 | `mid_D` | `α ⊗ β` | +| `y_Rd1` | id. 2 | `R_d1` | `α ⊗ β` | +| `y_Rd2` | id. 3 | `R_d2` | `α ⊗ β` | + +with `α_vec = (1, α, …, α^{rows−1})`, `β_vec` over `out_dim`, `γ_vec` over +`rank`, and `R = next_pow2(rank)`. + +## 5. Identity checks and projection soundness + +The verifier computes `Xᵀα` and `P₃ = den·(αᵀDβ)` itself and checks, +homomorphically over committed results: + +- (1) `y_A − s·y_U − y_Ru = 0` +- (2a) `y_cu − y_cu' = 0` and (2b) `y_cb − y_cb' = 0` +- (2c) `Y_z − s·y_M − y_Rd1 = 0` +- (3) `num·y_M − y_Rd2 − P₃·G0` opens to zero (`y_M` reused from id. 2) + +**Zero-check batching:** with challenge `ε`, `C* = Σ_i ε^i·(check_i)`; the +prover gives one Schnorr proof of knowledge of `ρ` with `C* = ρ·H`. If any +single check has a non-`H` component, the batched combination does too except +with probability ≤ (#checks)/p over `ε`; a Schnorr proof on `C*` with a +nonzero `G`-component would break DL between `H` and the other generators. + +**Projection soundness.** All witness commitments are absorbed before `α, β, +γ, δ, δ'` are squeezed (§9). Suppose identity 1 fails at some entry: define +the nonzero bivariate polynomial `E(a, c) = Σ_{i,j}(X·Aᵀ − s·U − R_u)[i,j] +a^i c^j` of degree ≤ (rows−1, rank−1); the check passes only if +`E(α, γ) = 0`, which by Schwartz–Zippel happens with probability +≤ (rows + rank − 2)/p. Identities 2 and 3 are bounded analogously by +(rows + out_dim − 2)/p and the id-2 consistency checks by (R−1)/p each. Union +bound over the three identities, the consistency checks, the Schnorr, and the +LFO/IPA knowledge errors stays below 2^−240 at `MAX_BATCH_ROWS`/`MAX_V3_DIM`; +Fiat–Shamir grinding over `< 2^100` transcripts leaves > 128 bits. **One +projection; no repetition** — repetition is a small-field Freivalds artifact. + +A note on extraction order: the LFO extractor pins each `Y` to the committed +vector and weights; the binding of `C_total` pins the vectors before the +challenges; so a passing transcript yields integer matrices (after §8 lifts +field values to integers) satisfying the identities mod p entrywise. + +## 6. Range argument `bp-aggregate-linked-v1` + +Standard aggregated Bulletproofs range proof (BP §4.3) over width classes, +with **one substitution** — the single novel composition point of this +protocol. Textbook BP checks the value-aggregation term `Σ_j z^{2+j}·v_j` +against per-value commitments `Σ_j z^{2+j}·V_j`; v3 has no per-value +commitments, so that term is supplied as a committed result `W` from one LFO +on the `VectorCommitment`s, with weights `z^{2+j}` routed through the class's +index map (weights are zero outside the class's real coordinates). The BP +verification equation uses `W` homomorphically exactly where `Σ z^{2+j}V_j` +would appear. + +**Extraction lemma (review target).** From a convincing prover one extracts: +(i) via BP's extractor, bit vectors and a value vector `v̂` with every +`v̂_j ∈ [0, 2^n)` whose `z`-weighted sum matches the value committed in `W` +for many `z`; (ii) via the LFO extractor, that `W` commits to +`Σ_j z^{2+j}·v_j` for the vector `v` inside the (binding) witness +commitments. Equality of the two `z`-polynomials at enough sampled `z` forces +`v̂_j = v_j` coefficient-wise (Schwartz–Zippel over `z`, degree ≤ m+1), hence +every committed coordinate lies in `[0, 2^n)`. The composition is a standard +sequential argument-of-knowledge composition; the forking tree is +polynomial-size for logarithmic-round protocols. + +Width classes (values shifted to unsigned by public shift constants): + +| Class | Members (shifted) | width n | +|---|---|---| +| `rem` | `R_u + ⌊s/2⌋`, `R_d1 + ⌊s/2⌋` | `scale_bits` (exact: the canonical interval of `s` has exactly `2^scale_bits` values) | +| `u` | `U + B_U` | `bitlen(2·B_U)` | +| `midD` | `mid_D + B_M` | `bitlen(2·B_M)` | +| `rd2` | `R_d2 + ⌊den/2⌋` | `max(1, ⌈log2 den⌉)`, two-sided | +| `ab` (manifest, one-time) | `A + value_bound`, `B + value_bound` | `value_bits` | + +- `B_U = max_i ⌊(Σ_k |X[i,k]|·value_bound + ⌊s/2⌋)/s⌋` and + `B_M = ⌊(den·max|D| + ⌊den/2⌋)/|num|⌋` are derived from **public** data by + both sides identically (BigInt arithmetic). +- Non-power-of-two interval `[0, den)` for `rd2`: two-sided trick — a second + aggregate proves `v + (2^n − den) ∈ [0, 2^n)`; its `W'` derives + homomorphically from `W` (`W' = W + (2^n − den)·(Σ_j z^{2+j})·G0`), so no + extra commitment or LFO is needed. `den = 1` keeps the uniform path with a + zero vector at width 1. +- Aggregates split into sub-aggregates of ≤ `2^22` bits; their value sums add + homomorphically. The `rb:g`/`rb:h` generator vectors are reused across + aggregates (binding is per-instance; reuse is safe because each aggregate's + `A, S` are absorbed into the transcript before its challenges). +- The `ab` class is proven once at manifest creation and verified at pin + time; without it, nothing bounds `A`/`B` and §8 fails. + +## 7. Padding soundness + +Normative discipline: every padded slot or index map is verifier-derived from +statement dims, never prover-supplied, and each appearance carries an +argument: + +- **Rank-IPA padding (forced zero).** `c_u`, `c_b` are committed at full + length `R = next_pow2(rank)`. The δ-consistency check runs over the full + padded length: `⟨c_u, (δ⁰..δ^{R−1})⟩ = ⟨U, α⊗(δ⁰..δ^{rank−1})⟩` is a + polynomial identity in `δ` of degree ≤ R−1; passing at random `δ` forces + `u*_j = (Uᵀα)_j` for `j < rank` **and `u*_j = 0` for `j ≥ rank`** (error ≤ + (R−1)/p). Truncating the δ-powers at `rank` would leave the padding slots + unconstrained while they still contribute `Σ_{j≥rank} u*_j·b*_j` to `Y_z` — + an additive forgery on identity 2 for any `rank < R`. +- **BP aggregate padding (harmless).** Padding slots of a range aggregate + correspond to no committed coordinate and get weight zero in `W`'s LFO, so + `W`'s extracted value covers exactly the real coordinates. A prover placing + nonzero values in padding slots changes `t(x)`'s constant coefficient away + from `W + δ(y,z)`; since `T1, T2, W` are absorbed before `x`, the degree-2 + polynomial identity in `x` then fails except with probability ≤ 2/p. Honest + provers zero the padding for completeness. + +## 8. Integer-vs-field argument (against proved bounds) + +A one-sided range proof `v + shift ∈ [0, 2^n)` establishes +`v ∈ [−shift, 2^n − 1 − shift]`; the composition checks are stated against +the **proved** bounds, never the nominal ones: + +- `P_A = 2^{value_bits−1}` (one more than `value_bound`; documented), + `P_U = 2^{n_u} − 1 − B_U`, `P_M = 2^{n_m} − 1 − B_M`; `rem` is exact at + `scale_bits`; `rd2` is exact via the two-sided proof. + +Verifier asserts (Rust `verify_v3` is sovereign; Python mirrors): + +1. `2·bitlen(P_A) + ⌈log2 in_dim⌉ + 1 ≤ 250` +2. `scale_bits + bitlen(P_U) + 1 ≤ 250` +3. `bitlen(P_U) + bitlen(P_A) + ⌈log2 rank⌉ + 1 ≤ 250` +4. `bitlen(num) + bitlen(P_M) + 1 ≤ 250` and + `bitlen(den) + bitlen(value_bound) + 1 ≤ 250` + +Then every entry of both sides of each identity has magnitude `< 2^250 ≪ p/2` +(public `X`, `D` are bounded by `value_bound` by the v2 quantization +contract), so the mod-p entrywise equality from §5 is equality over `Z`, and +uniqueness of canonical remainders gives exactly the v2 rounding semantics. +The range proofs on `U`/`mid_D` exist **only** for this overflow safety; their +values are already pinned by the canonical ranges of `R_u`/`R_d1`. + +## 9. Fiat–Shamir schedule (merlin, Strobe-128) + +`Transcript::new(b"zklora-projection-v1")`; every prover message is absorbed +before any later challenge; the verifier recomputes the identical schedule +and rejects any deviation: + +1. absorb `circuit_id`, `statement_digest`, `manifest_commitment`, + `batch_transcript_digest`, dims (rows/in/rank/out), fixed-point, scaling, + derived bounds (`B_U`, `B_M`, all widths) +2. absorb `a_commitment`, `b_commitment` chunks (pinned manifest) +3. absorb `C_U, C_Ru, C_midD, C_Rd1, C_Rd2` (all chunks, fixed label order) +4. challenges `alpha`, `beta`, `gamma` (`Fp::from_uniform_bytes(64)`) +5. absorb `c_u`, `c_b` +6. challenges `delta-u`, `delta-b` +7. absorb all committed results `Y_*` in the §4 table order +8. challenge `epsilon`; Schnorr zero-check (absorb `R`, challenge, response) +9. LFO/IPA sub-protocols under fixed instance labels (`lfo:yA`, …) +10. range classes in fixed order; per aggregate: absorb `A, S` → challenges + `y, z` → absorb `T1, T2, W` → challenge `x` → IPA rounds. + +The ordering invariants that the security argument needs: witness commitments +precede `α/β/γ` (projection soundness, §5); `c_u/c_b` precede `δ/δ'` +(consistency, §7); `C` precedes `z` and `W` precedes `x` (linkage extraction, +§6). + +## 10. Zero-knowledge + +All verifier-visible values are: hiding commitments, blinded IPA messages, +committed results, and Schnorr transcripts — each simulatable given the +public statement by standard sigma-protocol simulators (commit to random +values, program challenges), composed across the schedule in §9. Masking +randomness is CSPRNG output, never transcript-derived. **Scope (stated +honestly, and repeated in user docs):** `X` and `D` are public, so the +effective map `ΔW = B·A` is recoverable from ~`in_dim` independent public +rows regardless of the proof system, and the `A/B` factorization is +non-unique (`B·A = (BR)(R^{-1}A)`). Hiding buys early-session privacy and +non-disclosure of the exact quantized weights — nothing more. The verifier is +linear-time (MSM-bound), not succinct. + +## 11. Accounting (rows=256, in=768, rank=16, out=2304, s=2^20) + +| Vector | entries | width | serialized commitments | +|---|---|---|---| +| `A` / `B` | 12,288 / 36,864 | 63 | 1 + 1 pt (manifest, once) | +| `U` / `R_u` | 4,096 / 4,096 | ~35–75 / 20 | 1 + 1 pt | +| `mid_D` / `R_d1` / `R_d2` | 589,824 each | ~31 / 20 / ⌈log2 den⌉ | 9 + 9 + 9 pts | + +≈ 30 commitment points ≈ 1 KB per batch; LFO/IPA/Schnorr ≈ 200–300 points; +range aggregates (~31M committed bits in ≤ 8 sub-aggregates) ≈ 60–110 points +each. Expected proof size 25–60 KB. Verifier ≈ 60–70M point-ops (≈ 2–3 s +multicore); prover ≈ 130M point-ops + BigInt witness build (≈ 10–30 s). +Generator cache ≈ 512 MB at full size (uncompressed affine points are 64 B); +first-use derivation of ~8M hash-to-curve points is parallelized and +benchmarked separately. Performance gates: P0 (M3) proof ≤ 500 KB / verify ≤ +30 s / prove ≤ 120 s / ≤ 8 GB; P1 (M4) 100 KB / 10 s / 60 s. + +## 12. Blocking questions (answered) + +1. *Range proof without per-entry commitments?* §6: BP aggregation with the + `t₀` term supplied by one LFO on the vector commitments — public + commitment count is O(#vectors). +2. *How are `mid_D`/`R_d1`/`R_d2` committed?* Full chunked vector + commitments (9 points each at the representative shape); no per-value + commitments anywhere. +3. *Exact FS order?* §9, with the three ordering invariants named. +4. *Projection soundness error?* §5: ≤ (rows + out_dim)/p per identity, one + projection, > 128-bit margin after FS grinding. +5. *Domain separation?* Distinct merlin labels per challenge and per + sub-protocol instance (§9). +6. *No modular wraparound?* §8, stated against proved bounds `P_*`. +7. *What counts as proof size?* All bytes the verifier needs beyond the + statement, transcript, and pinned manifest — i.e. the `.zklora.proof` + file including all commitments (§11). +8. *What does the verifier need from the manifest?* The pinned schema-3 + manifest file: per-module dims/config, `a_commitment`, `b_commitment`, + `commitment_nonce`, `adapter_commitment`, and the one-time `ab` range + proof, verified at pin time; `manifest_commitment` is recomputed from the + pinned payload, never taken from artifacts. +9. *Padding soundness?* §7 (added as a blocking question after review). + +## 13. Comparison appendix + +- **Same projection relation in Halo2 multi-phase.** Would eliminate the + hand-rolled FS/IPA/BP and the §6 lemma, keeping one audited proof system. + Killer: the ~1.8M remainder/mid values per batch need ~4–5M lookup rows + even at 10-bit limbs → k≈23 on IPA-halo2 → minutes-to-tens-of-minutes + proving and tens of GB — an order of magnitude outside the gates. (For + scale: v2's per-row circuit with bit-decomposed range checks is already + unusable at real shapes, and the zcash halo2 0.3 pin has no multi-phase + challenge API at all.) +- **Forking dalek `bulletproofs` (ristretto255).** Audited A/S/T/IPA + machinery and merlin-native transcripts, but its aggregated-range API + requires per-value commitments — exactly the O(entries) serialization this + design eliminates — so the `W`-linkage would be a fork of audited code: + the novel part stays novel and we'd own a patched fork instead of an owned + crate. Staying on Pasta also reuses the existing integer↔field helpers and + keeps one curve stack in-tree. +- **Sumcheck/MLE track** (previous design iteration): direct multilinear + sumchecks for the matmuls with a logUp/Lasso-style lookup for ranges. + Better asymptotic verifier (√N Hyrax structure), but three bespoke + components (zk-sumcheck masking, MLE-PCS over Pasta, lookup argument) + versus this design's single novel lemma; range volume — the shared + dominant cost — is identical. Reconsider if batch sizes grow ~10× or the + linear verifier MSM becomes the binding constraint (v3.1 may add + cross-batch MSM aggregation first). diff --git a/src/pyproject.toml b/src/pyproject.toml index 6aac03c..5e4fc27 100644 --- a/src/pyproject.toml +++ b/src/pyproject.toml @@ -10,7 +10,10 @@ authors = [ ] description = "A Python library for zero-knowledge proof generation and verification" readme = "../readme.md" # Update path to point to root README -requires-python = ">=3.8" +# Floor matches actual usage: PEP 604 unions evaluated at import time in +# lora_contributor_mpi require 3.10+, and pathlib.Path.is_relative_to +# requires 3.9+. CI exercises 3.11. +requires-python = ">=3.10" classifiers = [ "Programming Language :: Python :: 3", "License :: Other/Proprietary License", diff --git a/src/src/lib.rs b/src/src/lib.rs index 4b95e83..89ba35f 100644 --- a/src/src/lib.rs +++ b/src/src/lib.rs @@ -10,7 +10,7 @@ use halo2_proofs::{ pasta::{vesta, EqAffine, Fp}, plonk::{ create_proof, keygen_pk, keygen_vk, verify_proof, Advice, Circuit, Column, - ConstraintSystem, Error, Instance, Selector, SingleVerifier, + ConstraintSystem, Error, Instance, ProvingKey, Selector, SingleVerifier, VerifyingKey, }, poly::commitment::Params, poly::Rotation, @@ -22,7 +22,9 @@ use num_traits::{One, Signed, Zero}; use rand_core::OsRng; use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256}; +use std::collections::HashMap; use std::convert::TryInto; +use std::sync::{Arc, Mutex, OnceLock}; const ADAPTER_COMMITMENT_DOMAIN: u64 = 0x5a4b4c4f5241; // "ZKLORA" const ADAPTER_COMMITMENT_VERSION: u64 = 1; @@ -30,6 +32,11 @@ const ADAPTER_COMMITMENT_VERSION: u64 = 1; const ARTIFACT_SCHEMA_VERSION: u64 = 2; const FIELD_SAFE_BITS: usize = 250; const POSEIDON_PAIR_ROWS: usize = 96; +// Caps for the legacy backend: artifacts beyond these shapes are rejected before +// any keygen work so a hostile statement cannot stall the verifier. +const MAX_LEGACY_K: u32 = 24; +const MAX_LEGACY_DIM: usize = 16_384; +const MAX_LEGACY_RANK: usize = 1_024; #[derive(Debug, thiserror::Error)] pub enum NativeError { @@ -608,6 +615,21 @@ impl LoraCircuit { )); } let in_dim = self.in_dim(); + if in_dim > MAX_LEGACY_DIM || self.out_dim() > MAX_LEGACY_DIM { + return Err(NativeError::InvalidDimensions(format!( + "legacy artifact exceeds verification caps: dims {}x{} beyond {}", + in_dim, + self.out_dim(), + MAX_LEGACY_DIM + ))); + } + if self.rank() > MAX_LEGACY_RANK { + return Err(NativeError::InvalidDimensions(format!( + "legacy artifact exceeds verification caps: rank {} beyond {}", + self.rank(), + MAX_LEGACY_RANK + ))); + } for row in &self.a { if row.len() != in_dim { return Err(NativeError::InvalidDimensions( @@ -1197,6 +1219,180 @@ fn k_for(circuit: &LoraCircuit) -> u32 { rows.trailing_zeros().max(8) } +/// Cache key covering everything the circuit structure (and therefore the +/// params/keys) depends on: the in-circuit constants are derived solely from +/// dims, fixed-point widths, and scaling; witness values never enter keygen. +type LegacyShapeKey = (u32, usize, usize, usize, u32, u32, u32, i64, i64); + +/// Both caches are bounded: the cache keys span every attacker-influenceable +/// statement field, so an unbounded map fed varying shapes is a slow memory +/// DoS. Entries near k = MAX_LEGACY_K are GB-scale, hence the small caps. +const MAX_LEGACY_KEY_CACHE_ENTRIES: usize = 4; +const MAX_LEGACY_PARAMS_CACHE_ENTRIES: usize = 4; + +/// Minimal LRU map: a HashMap with a monotonically increasing use stamp per +/// entry; inserting beyond capacity evicts the least recently used entry. +struct BoundedLru { + map: HashMap, + counter: u64, + capacity: usize, +} + +impl BoundedLru { + fn new(capacity: usize) -> Self { + BoundedLru { + map: HashMap::new(), + counter: 0, + capacity: capacity.max(1), + } + } + + fn get(&mut self, key: &K) -> Option { + self.counter += 1; + let stamp = self.counter; + self.map.get_mut(key).map(|slot| { + slot.0 = stamp; + slot.1.clone() + }) + } + + fn insert(&mut self, key: K, value: V) { + self.counter += 1; + if !self.map.contains_key(&key) && self.map.len() >= self.capacity { + if let Some(oldest) = self + .map + .iter() + .min_by_key(|(_, (stamp, _))| *stamp) + .map(|(k, _)| k.clone()) + { + self.map.remove(&oldest); + } + } + self.map.insert(key, (self.counter, value)); + } + + #[cfg(test)] + fn len(&self) -> usize { + self.map.len() + } + + #[cfg(test)] + fn contains_key(&self, key: &K) -> bool { + self.map.contains_key(key) + } +} + +/// Verification only ever needs the verifying key; the proving key is built +/// (and cached) lazily the first time a shape is actually proven, so a +/// verifier never pays keygen_pk for hostile or one-off shapes. +enum LegacyKeys { + VerifyOnly { + params: Arc>, + vk: VerifyingKey, + }, + Prover { + params: Arc>, + pk: ProvingKey, + }, +} + +impl LegacyKeys { + fn params(&self) -> &Params { + match self { + LegacyKeys::VerifyOnly { params, .. } => params, + LegacyKeys::Prover { params, .. } => params, + } + } + + fn vk(&self) -> &VerifyingKey { + match self { + LegacyKeys::VerifyOnly { vk, .. } => vk, + LegacyKeys::Prover { pk, .. } => pk.get_vk(), + } + } + + fn pk(&self) -> Option<&ProvingKey> { + match self { + LegacyKeys::VerifyOnly { .. } => None, + LegacyKeys::Prover { pk, .. } => Some(pk), + } + } +} + +static LEGACY_KEY_CACHE: OnceLock>>> = + OnceLock::new(); +static LEGACY_PARAMS_CACHE: OnceLock>>>> = + OnceLock::new(); + +/// Cached values are immutable once inserted (Arc'd keys/params plus LRU +/// bookkeeping), so a panic in another thread cannot leave them torn; +/// recover from poisoning instead of propagating panics through PyO3. +fn lock_recovering(mutex: &Mutex) -> std::sync::MutexGuard<'_, T> { + mutex + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) +} + +fn legacy_cache() -> &'static Mutex>> { + LEGACY_KEY_CACHE.get_or_init(|| Mutex::new(BoundedLru::new(MAX_LEGACY_KEY_CACHE_ENTRIES))) +} + +fn legacy_params_for(k: u32) -> Arc> { + let cache = LEGACY_PARAMS_CACHE + .get_or_init(|| Mutex::new(BoundedLru::new(MAX_LEGACY_PARAMS_CACHE_ENTRIES))); + if let Some(found) = lock_recovering(cache).get(&k) { + return found; + } + // Built outside the lock: Params::new at large k takes seconds and two + // racing builders are deterministic, so last-write-wins is harmless. + let params = Arc::new(Params::::new(k)); + lock_recovering(cache).insert(k, params.clone()); + params +} + +fn legacy_shape_key(circuit: &LoraCircuit, k: u32) -> LegacyShapeKey { + ( + k, + circuit.in_dim(), + circuit.rank(), + circuit.out_dim(), + circuit.fixed_point.scale_bits, + circuit.fixed_point.value_bits, + circuit.fixed_point.intermediate_bits, + circuit.scaling_num, + circuit.scaling_den, + ) +} + +fn legacy_keys_for(circuit: &LoraCircuit, need_pk: bool) -> Result, NativeError> { + let k = k_for(circuit); + if k > MAX_LEGACY_K { + return Err(NativeError::InvalidDimensions(format!( + "legacy artifact exceeds verification caps: k {k} beyond {MAX_LEGACY_K}" + ))); + } + let key = legacy_shape_key(circuit, k); + if let Some(found) = lock_recovering(legacy_cache()).get(&key) { + if !need_pk || found.pk().is_some() { + return Ok(found); + } + } + // Keygen runs outside the lock so concurrent callers on other shapes are + // not serialized behind it; duplicated keygen on the same shape is + // deterministic and last-write-wins. + let params = legacy_params_for(k); + let empty = circuit.without_witnesses(); + let vk = keygen_vk(¶ms, &empty).map_err(|e| NativeError::Halo2(e.to_string()))?; + let entry = if need_pk { + let pk = keygen_pk(¶ms, vk, &empty).map_err(|e| NativeError::Halo2(e.to_string()))?; + Arc::new(LegacyKeys::Prover { params, pk }) + } else { + Arc::new(LegacyKeys::VerifyOnly { params, vk }) + }; + lock_recovering(legacy_cache()).insert(key, entry.clone()); + Ok(entry) +} + fn circuit_from_json(statement_json: &str, witness_json: &str) -> Result { let statement: NativeStatement = serde_json::from_str(statement_json).map_err(|e| NativeError::Json(e.to_string()))?; @@ -1236,16 +1432,14 @@ fn default_scaling_den() -> i64 { pub fn prove_bytes(statement_json: &str, witness_json: &str) -> Result, NativeError> { let circuit = circuit_from_json(statement_json, witness_json)?; - let k = k_for(&circuit); - let params: Params = Params::new(k); - let vk = keygen_vk(¶ms, &circuit).map_err(|e| NativeError::Halo2(e.to_string()))?; - let pk = keygen_pk(¶ms, vk, &circuit).map_err(|e| NativeError::Halo2(e.to_string()))?; + let keys = legacy_keys_for(&circuit, true)?; + let pk = keys.pk().expect("prover cache entry carries a proving key"); let instances = public_inputs(&circuit)?; let instance_refs: Vec<&[Fp]> = vec![instances.as_slice()]; let mut transcript = Blake2bWrite::<_, vesta::Affine, Challenge255<_>>::init(vec![]); create_proof( - ¶ms, - &pk, + keys.params(), + pk, &[circuit], &[instance_refs.as_slice()], &mut OsRng, @@ -1266,16 +1460,14 @@ pub fn verify_bytes(statement_json: &str, proof: &[u8]) -> Result = Params::new(k); - let vk = keygen_vk(¶ms, &circuit).map_err(|e| NativeError::Halo2(e.to_string()))?; + let keys = legacy_keys_for(&circuit, false)?; let instances = public_inputs(&circuit)?; let instance_refs: Vec<&[Fp]> = vec![instances.as_slice()]; let mut transcript = Blake2bRead::<_, vesta::Affine, Challenge255<_>>::init(proof); let result = verify_proof( - ¶ms, - &vk, - SingleVerifier::new(¶ms), + keys.params(), + keys.vk(), + SingleVerifier::new(keys.params()), &[instance_refs.as_slice()], &mut transcript, ); @@ -1405,6 +1597,82 @@ mod tests { assert_ne!(first, adapter_commitment_for_input(&changed).unwrap()); } + #[test] + fn legacy_key_cache_reuses_and_upgrades_entries() { + let circuit = minimal_circuit(); + let verify_only = legacy_keys_for(&circuit, false).unwrap(); + assert!(verify_only.pk().is_none()); + let cached = legacy_keys_for(&circuit, false).unwrap(); + assert!(Arc::ptr_eq(&verify_only, &cached)); + + // A prover call on the same shape upgrades the entry in place... + let prover = legacy_keys_for(&circuit, true).unwrap(); + assert!(prover.pk().is_some()); + // ...and both later verifiers and provers share the upgraded entry. + let reused_verify = legacy_keys_for(&circuit, false).unwrap(); + assert!(Arc::ptr_eq(&prover, &reused_verify)); + let reused_prover = legacy_keys_for(&circuit, true).unwrap(); + assert!(Arc::ptr_eq(&prover, &reused_prover)); + + let key = legacy_shape_key(&circuit, k_for(&circuit)); + assert!(lock_recovering(legacy_cache()).contains_key(&key)); + } + + #[test] + fn bounded_lru_evicts_least_recently_used() { + let mut lru: BoundedLru = BoundedLru::new(2); + lru.insert(1, 10); + lru.insert(2, 20); + assert_eq!(lru.get(&1), Some(10)); // touch 1 so 2 becomes the oldest + lru.insert(3, 30); + assert_eq!(lru.len(), 2); + assert!(lru.contains_key(&1)); + assert!(!lru.contains_key(&2)); + assert!(lru.contains_key(&3)); + + // Re-inserting an existing key must not evict anything. + lru.insert(1, 11); + assert_eq!(lru.len(), 2); + assert_eq!(lru.get(&1), Some(11)); + assert!(lru.contains_key(&3)); + } + + #[test] + fn legacy_caps_reject_oversized_dimensions() { + let fixed_point = FixedPointConfig { + scale_bits: 1, + value_bits: 8, + intermediate_bits: 16, + }; + let wide = LoraCircuit { + a: vec![vec![0; MAX_LEGACY_DIM + 1]], + b: vec![vec![0]], + x: vec![0; MAX_LEGACY_DIM + 1], + delta: vec![0], + fixed_point: fixed_point.clone(), + scaling_num: 1, + scaling_den: 1, + adapter_commitment: "0".to_string(), + statement_digest: "22".repeat(32), + }; + let err = wide.validate().unwrap_err(); + assert!(err.to_string().contains("exceeds verification caps")); + + let deep = LoraCircuit { + a: vec![vec![0]; MAX_LEGACY_RANK + 1], + b: vec![vec![0; MAX_LEGACY_RANK + 1]], + x: vec![0], + delta: vec![0], + fixed_point, + scaling_num: 1, + scaling_den: 1, + adapter_commitment: "0".to_string(), + statement_digest: "22".repeat(32), + }; + let err = deep.validate().unwrap_err(); + assert!(err.to_string().contains("exceeds verification caps")); + } + #[test] fn mock_prover_accepts_valid_lora_relation() { let circuit = valid_circuit(); diff --git a/src/zklora/__init__.py b/src/zklora/__init__.py index 700e6a2..5ea18fa 100644 --- a/src/zklora/__init__.py +++ b/src/zklora/__init__.py @@ -13,6 +13,7 @@ "generate_proofs": ("zklora.zk_proof_generator", "generate_proofs"), "adapter_manifest_entry": ("zklora.proof_contract", "adapter_manifest_entry"), "write_adapter_manifest": ("zklora.proof_contract", "write_adapter_manifest"), + "expand_statement_rows": ("zklora.proof_v3", "expand_statement_rows"), "commit_activations": ("zklora.polynomial_commit", "commit_activations"), "verify_commitment": ("zklora.polynomial_commit", "verify_commitment"), } diff --git a/src/zklora/lora_contributor_mpi/__init__.py b/src/zklora/lora_contributor_mpi/__init__.py index cdf1032..b33d7d9 100644 --- a/src/zklora/lora_contributor_mpi/__init__.py +++ b/src/zklora/lora_contributor_mpi/__init__.py @@ -8,7 +8,7 @@ from transformers import AutoModelForCausalLM from peft import PeftModel -from ..zk_proof_generator import generate_proofs +from ..zk_proof_generator import generate_proofs, prover_backend from ..base_model_user_mpi import _recv_json_message, _send_json_message from ..proof_contract import ( FixedPointConfig, @@ -19,6 +19,14 @@ quantize_nested, write_adapter_manifest, ) +from ..proof_v3 import ( + adapter_manifest_entry_v3, + adapter_manifest_payload_v3, + ensure_secret_outside_artifacts, + load_or_create_contributor_secret, + resolve_contributor_secret_path, + write_adapter_manifest_v3, +) def read_file_as_bytes(path: str) -> bytes: @@ -46,10 +54,19 @@ def __init__( lora_model_id: str, out_dir: str, fixed_point: FixedPointConfig | None = None, + manifest_secret_path: str | None = None, ): self.out_dir = out_dir self.fixed_point = fixed_point or FixedPointConfig() os.makedirs(self.out_dir, exist_ok=True) + # Contributor secret seed for hiding adapter commitments. It must never + # live inside out_dir, which is handed to the verifier as-is. + self.manifest_secret_path = resolve_contributor_secret_path( + manifest_secret_path + ) + ensure_secret_outside_artifacts(self.manifest_secret_path, self.out_dir) + self._manifest_entries: list | None = None + self._manifest_payload: dict | None = None # 1) Load model, disable cache => no 'past_key_values' base_model = AutoModelForCausalLM.from_pretrained(base_model_name) @@ -82,6 +99,15 @@ def list_lora_injection_points(self): return list(self.submodules.keys()) def adapter_manifest_entries(self): + legacy = prover_backend() == "legacy-halo2" + if not legacy and self._manifest_entries is not None: + # Schema-3 entries carry a fresh commitment nonce; the same entries + # must back both the pinned manifest and every later proof, so they + # are built once per server run. + return self._manifest_entries + secret = None + if not legacy: + secret = load_or_create_contributor_secret(self.manifest_secret_path) entries = [] for sub_name, module in self.submodules.items(): a_matrix, b_matrix, scaling_num, scaling_den = lora_matrices_and_scaling( @@ -93,20 +119,40 @@ def adapter_manifest_entries(self): b_quantized = quantize_nested( b_matrix.detach().cpu().numpy().tolist(), self.fixed_point ) - entries.append( - adapter_manifest_entry( - sub_name, - a_quantized, - b_quantized, - scaling_num, - scaling_den, - self.fixed_point, + if legacy: + entries.append( + adapter_manifest_entry( + sub_name, + a_quantized, + b_quantized, + scaling_num, + scaling_den, + self.fixed_point, + ) ) - ) + else: + entries.append( + adapter_manifest_entry_v3( + sub_name, + a_quantized, + b_quantized, + scaling_num, + scaling_den, + self.fixed_point, + secret, + ) + ) + if not legacy: + self._manifest_entries = entries return entries def write_adapter_manifest(self, path: str): - write_adapter_manifest(path, self.adapter_manifest_entries()) + entries = self.adapter_manifest_entries() + if prover_backend() == "legacy-halo2": + write_adapter_manifest(path, entries) + self._manifest_payload = None + else: + self._manifest_payload = write_adapter_manifest_v3(path, entries) def apply_lora( self, @@ -190,11 +236,25 @@ def finalize_proofs_and_collect(self, session_id: str | None = None): else: records = self.session_data.pop(session_id, []) - proof_res = generate_proofs( - records=records, - output_dir=self.out_dir, - verbose=True, - ) + if prover_backend() == "legacy-halo2": + proof_res = generate_proofs( + records=records, + output_dir=self.out_dir, + verbose=True, + ) + else: + if self._manifest_payload is None: + # Pin the same entries the verifier will receive out-of-band. + self._manifest_payload = adapter_manifest_payload_v3( + self.adapter_manifest_entries() + ) + proof_res = generate_proofs( + records=records, + output_dir=self.out_dir, + verbose=True, + adapter_manifest=self._manifest_payload, + manifest_secret_path=self.manifest_secret_path, + ) if not proof_res: print("[A] No proofs generated or something went wrong.") diff --git a/src/zklora/proof_v3.py b/src/zklora/proof_v3.py new file mode 100644 index 0000000..5807647 --- /dev/null +++ b/src/zklora/proof_v3.py @@ -0,0 +1,1330 @@ +"""Schema-3 projection-backend artifact layer. + +One artifact set covers a contiguous batch of module invocations (rows) instead +of a single row. Statements are digest-only: full ``x``/``delta`` rows live in +the verifier-recorded transcript and are bound through per-row digests plus a +batch transcript digest. The cryptographic relation is proven by the native +``pedersen-projection-v1`` backend; this module owns statements, batching, +manifests, coverage, and dispatch between schema versions. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import secrets +from dataclasses import asdict, dataclass, field +from pathlib import Path +from typing import Any, Iterable + +from .proof_contract import ( + FixedPointConfig, + InvocationWitness, + ProofContractError, + TranscriptEntry, + canonical_json, + compute_delta_quantized, + digest_hex, + load_json, + load_transcript, + module_slug, +) +from . import proof_contract as _v2 + +SCHEMA_VERSION_V3 = 3 +BACKEND_ID_V3 = "zklora-projection-v1" +PROOF_KIND_V3 = "pedersen-projection-v1" +PROOF_SYSTEM_V3 = "pedersen-projection-ipa-bp" +COMMITMENT_SCHEME_V3 = "pedersen-vector-vesta-v1" +RANGE_ARGUMENT_V3 = "bp-aggregate-linked-v1" +GENERATOR_SEED_ID = "zklora/v3/gen-seed/v1" +ENCODING_V3 = "row-major-exact-len-v1" +FIAT_SHAMIR_V3 = "merlin-strobe128-v1" +INVOCATION_STRATEGY_V3 = "one-proof-per-row-batch-v1" +SECURITY_LEVEL_BITS_V3 = 128 +AUDIT_STATUS_V3 = "unaudited" + +DEFAULT_TARGET_CHUNK_ROWS = 256 +MAX_BATCH_ROWS = 4096 +# Python mirror of the Rust-enforced DoS caps (Rust verify_v3 is sovereign). +MAX_V3_DIM = 65_536 +MAX_V3_RANK = 1_024 +FIELD_SAFE_BITS = 250 + +_CHUNK_ROWS_ENV = "ZKLORA_CHUNK_ROWS" +_SECRET_ENV = "ZKLORA_CONTRIBUTOR_SECRET" +_SECRET_DEFAULT = Path("~") / ".zklora" / "contributor.secret.json" +_SECRET_GLOB = "*.secret.json" + +_STATEMENT_SUFFIX = ".zklora.statement.json" + + +def _native_v3(): + # Resolved through the module attribute so test fakes that patch + # proof_contract._native_module cover both schema paths. + native = _v2._native_module() + if native is None: + raise ProofContractError( + "native projection prover is unavailable; build/install zklora with " + "maturin, or unset ZKLORA_PROVER_BACKEND to use the default legacy " + "backend" + ) + missing = [ + name + for name in ( + "prove_v3", + "verify_v3", + "adapter_commit_v3", + "verify_adapter_manifest_v3", + ) + if not hasattr(native, name) + ] + if missing: + raise ProofContractError( + "installed native module does not support the opt-in projection " + f"backend (missing {', '.join(missing)}); rebuild zklora with a " + "native module that ships v3 support, or unset " + "ZKLORA_PROVER_BACKEND to use the default legacy backend" + ) + return native + + +def _ceil_log2(value: int) -> int: + if value <= 1: + return 0 + return (value - 1).bit_length() + + +def target_chunk_rows() -> int: + raw = os.environ.get(_CHUNK_ROWS_ENV) + if raw is None: + return DEFAULT_TARGET_CHUNK_ROWS + try: + value = int(raw) + except ValueError as exc: + raise ProofContractError(f"invalid {_CHUNK_ROWS_ENV}: {raw!r}") from exc + if value < 1 or value > MAX_BATCH_ROWS: + raise ProofContractError( + f"{_CHUNK_ROWS_ENV} must be within [1, {MAX_BATCH_ROWS}], got {value}" + ) + return value + + +# --------------------------------------------------------------------------- +# Contributor secret handling +# --------------------------------------------------------------------------- + + +def resolve_contributor_secret_path( + explicit: str | os.PathLike[str] | None = None, +) -> Path: + if explicit is not None: + return Path(explicit).expanduser() + env = os.environ.get(_SECRET_ENV) + if env: + return Path(env).expanduser() + return _SECRET_DEFAULT.expanduser() + + +def ensure_secret_outside_artifacts( + secret_path: Path, output_dir: str | os.PathLike[str] +) -> None: + secret = secret_path.expanduser().resolve() + artifacts = Path(output_dir).expanduser().resolve() + if secret == artifacts or secret.is_relative_to(artifacts): + raise ProofContractError( + "contributor secret must not live inside the shared artifact " + f"directory: {secret} is inside {artifacts}; the artifact directory " + "is handed to the verifier and a leaked seed destroys hiding of the " + "adapter commitments" + ) + + +def _read_contributor_secret(path: Path) -> str: + data = load_json(path) + seed = str(data.get("seed", "")) + if len(seed) != 64 or any(c not in "0123456789abcdef" for c in seed.lower()): + raise ProofContractError(f"malformed contributor secret at {path}") + return seed + + +def load_or_create_contributor_secret(path: Path) -> str: + path = path.expanduser() + if path.exists(): + return _read_contributor_secret(path) + path.parent.mkdir(parents=True, exist_ok=True) + seed = secrets.token_hex(32) + try: + fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + except FileExistsError: + # Lost a concurrent-creation race; the winner's seed is authoritative. + return _read_contributor_secret(path) + with os.fdopen(fd, "w", encoding="utf-8") as f: + f.write(canonical_json({"seed": seed}) + "\n") + return seed + + +def reject_secrets_in_artifact_dir(proof_dir: str | os.PathLike[str]) -> None: + leaked = sorted(Path(proof_dir).glob(_SECRET_GLOB)) + if leaked: + names = ", ".join(p.name for p in leaked) + raise ProofContractError( + f"refusing to verify: contributor secret file(s) found among proof " + f"artifacts ({names}); the seed must never be shared with the verifier" + ) + + +# --------------------------------------------------------------------------- +# Batching +# --------------------------------------------------------------------------- + + +@dataclass(frozen=True) +class BatchWitness: + session_id: str + module_name: str + start_invocation_index: int + rows: list[InvocationWitness] = field(default_factory=list) + + @property + def count(self) -> int: + return len(self.rows) + + @property + def first(self) -> InvocationWitness: + return self.rows[0] + + @property + def x_rows(self) -> list[list[int]]: + return [[int(v) for v in row.x] for row in self.rows] + + @property + def delta_rows(self) -> list[list[int]]: + return [[int(v) for v in row.delta] for row in self.rows] + + +def _group_key(record: InvocationWitness): + return ( + record.session_id, + record.module_name, + record.in_dim, + record.rank, + record.out_dim, + record.fixed_point, + int(record.scaling_num), + int(record.scaling_den), + ) + + +def build_batches( + records: Iterable[InvocationWitness], target_rows: int | None = None +) -> list[BatchWitness]: + target = target_chunk_rows() if target_rows is None else int(target_rows) + if target < 1 or target > MAX_BATCH_ROWS: + raise ProofContractError( + f"target chunk rows must be within [1, {MAX_BATCH_ROWS}], got {target}" + ) + + groups: dict[Any, list[InvocationWitness]] = {} + order: list[Any] = [] + for record in records: + if len(record.x) != record.in_dim: + raise ProofContractError( + f"record x length {len(record.x)} does not match in_dim " + f"{record.in_dim} for {record.module_name}" + ) + if len(record.delta) != record.out_dim: + raise ProofContractError( + f"record delta length {len(record.delta)} does not match out_dim " + f"{record.out_dim} for {record.module_name}" + ) + key = _group_key(record) + if key not in groups: + groups[key] = [] + order.append(key) + groups[key].append(record) + + batches: list[BatchWitness] = [] + for key in order: + rows = sorted(groups[key], key=lambda r: int(r.invocation_index)) + reference = rows[0] + for row in rows[1:]: + if row.a != reference.a or row.b != reference.b: + raise ProofContractError( + "batch group has inconsistent adapter weights for " + f"{reference.module_name}; one batch must cover one adapter" + ) + indices = [int(row.invocation_index) for row in rows] + for prev, cur in zip(indices, indices[1:]): + if cur == prev: + raise ProofContractError( + f"duplicate invocation index {cur} for {reference.module_name}" + ) + if cur != prev + 1: + raise ProofContractError( + "non-contiguous invocation indices for " + f"{reference.module_name}: {prev} -> {cur}" + ) + for offset in range(0, len(rows), target): + chunk = rows[offset : offset + target] + batches.append( + BatchWitness( + session_id=reference.session_id, + module_name=reference.module_name, + start_invocation_index=int(chunk[0].invocation_index), + rows=list(chunk), + ) + ) + return batches + + +# --------------------------------------------------------------------------- +# Digests, identifiers, statements +# --------------------------------------------------------------------------- + + +def row_digest( + *, + session_id: str, + module_name: str, + invocation_index: int, + input_shape: list[int], + output_shape: list[int], + x_row: list[int], + delta_row: list[int], + fixed_point: FixedPointConfig, + scaling_num: int, + scaling_den: int, + rank: int, + in_dim: int, + out_dim: int, + adapter_commitment: dict[str, Any], + manifest_commitment: str, +) -> str: + payload = { + "schema_version": SCHEMA_VERSION_V3, + "backend": BACKEND_ID_V3, + "proof_kind": PROOF_KIND_V3, + "session_id": session_id, + "module_name": module_name, + "invocation_index": int(invocation_index), + "input_shape": [int(v) for v in input_shape], + "output_shape": [int(v) for v in output_shape], + "x_row": [int(v) for v in x_row], + "delta_row": [int(v) for v in delta_row], + "fixed_point": asdict(fixed_point), + "scaling": {"num": int(scaling_num), "den": int(scaling_den)}, + "rank": int(rank), + "in_dim": int(in_dim), + "out_dim": int(out_dim), + "adapter_commitment": adapter_commitment, + "manifest_commitment": manifest_commitment, + } + return digest_hex(payload) + + +def batch_transcript_digest(row_digests: list[str]) -> str: + return digest_hex(list(row_digests)) + + +def circuit_id_v3( + in_dim: int, + rank: int, + out_dim: int, + fixed_point: FixedPointConfig, + scaling_num: int, + scaling_den: int, + chunk_rows: int, +) -> str: + payload = { + "backend": BACKEND_ID_V3, + "proof_kind": PROOF_KIND_V3, + "commitment_scheme": COMMITMENT_SCHEME_V3, + "generator_seed_id": GENERATOR_SEED_ID, + "encoding": ENCODING_V3, + "range_argument": RANGE_ARGUMENT_V3, + "fiat_shamir": FIAT_SHAMIR_V3, + "in_dim": int(in_dim), + "rank": int(rank), + "out_dim": int(out_dim), + "fixed_point": asdict(fixed_point), + "scaling": {"num": int(scaling_num), "den": int(scaling_den)}, + "target_chunk_rows": int(chunk_rows), + } + return digest_hex(payload) + + +def vk_fingerprint_v3(circuit: str) -> str: + return digest_hex({"backend": BACKEND_ID_V3, "vk_for_circuit": circuit}) + + +def _statement_digest_payload(statement: dict[str, Any]) -> dict[str, Any]: + return {key: value for key, value in statement.items() if key != "statement_digest"} + + +def derive_bounds( + x_rows: list[list[int]], + delta_rows: list[list[int]], + fixed_point: FixedPointConfig, + scaling_num: int, + scaling_den: int, +) -> dict[str, int]: + """Public per-batch bounds and range widths, identical on both sides. + + The composition checks below are stated against the *proved* bounds + ``P = 2^n - 1 - shift`` (what a one-sided ``[0, 2^n)`` range proof actually + establishes), never against the nominal bounds. + """ + + scale = fixed_point.scale + value_bound = fixed_point.value_bound + b_u = 0 + for row in x_rows: + raw = sum(abs(int(v)) for v in row) * value_bound + b_u = max(b_u, (raw + scale // 2) // scale) + d_max = max((abs(int(v)) for row in delta_rows for v in row), default=0) + num = int(scaling_num) + den = int(scaling_den) + if den <= 0 or num == 0: + raise ProofContractError("scaling must have positive den and nonzero num") + b_m = (den * d_max + den // 2) // abs(num) + + n_u = max(1, (2 * b_u).bit_length()) + n_m = max(1, (2 * b_m).bit_length()) + proved_a = 1 << (fixed_point.value_bits - 1) + proved_u = (1 << n_u) - 1 - b_u + proved_m = (1 << n_m) - 1 - b_m + return { + "value_bound": value_bound, + "b_u": b_u, + "b_m": b_m, + "n_u": n_u, + "n_m": n_m, + "n_rem": int(fixed_point.scale_bits), + "n_rd2": max(1, _ceil_log2(den)), + "proved_a": proved_a, + "proved_u": proved_u, + "proved_m": proved_m, + } + + +def check_bounds_composition( + bounds: dict[str, int], + in_dim: int, + rank: int, + fixed_point: FixedPointConfig, + scaling_num: int, + scaling_den: int, +) -> None: + proved_a = bounds["proved_a"] + proved_u = bounds["proved_u"] + proved_m = bounds["proved_m"] + checks = [ + 2 * proved_a.bit_length() + _ceil_log2(max(1, in_dim)) + 1, + int(fixed_point.scale_bits) + proved_u.bit_length() + 1, + proved_u.bit_length() + proved_a.bit_length() + _ceil_log2(max(1, rank)) + 1, + abs(int(scaling_num)).bit_length() + proved_m.bit_length() + 1, + int(scaling_den).bit_length() + bounds["value_bound"].bit_length() + 1, + ] + if any(bits > FIELD_SAFE_BITS for bits in checks): + raise ProofContractError( + "fixed-point config and batch dimensions exceed Pasta field-safe " + "integer bounds for the projection relation" + ) + + +def _check_v3_dims(in_dim: int, rank: int, out_dim: int, count: int) -> None: + if in_dim < 1 or out_dim < 1 or in_dim > MAX_V3_DIM or out_dim > MAX_V3_DIM: + raise ProofContractError( + f"projection artifact exceeds verification caps: dims {in_dim}x{out_dim}" + ) + if rank < 1 or rank > MAX_V3_RANK: + raise ProofContractError( + f"projection artifact exceeds verification caps: rank {rank}" + ) + if count < 1 or count > MAX_BATCH_ROWS: + raise ProofContractError( + f"projection artifact exceeds verification caps: row count {count}" + ) + + +def statement_from_batch( + batch: BatchWitness, + manifest_entry: dict[str, Any], + manifest_commitment: str, + chunk_rows: int, +) -> dict[str, Any]: + first = batch.first + rank = first.rank + in_dim = first.in_dim + out_dim = first.out_dim + _check_v3_dims(in_dim, rank, out_dim, batch.count) + if batch.count > int(chunk_rows): + raise ProofContractError( + f"batch of {batch.count} rows exceeds target chunk {chunk_rows}" + ) + _check_manifest_entry_matches( + manifest_entry, + module_name=batch.module_name, + rank=rank, + in_dim=in_dim, + out_dim=out_dim, + fixed_point=first.fixed_point, + scaling_num=first.scaling_num, + scaling_den=first.scaling_den, + ) + + for row in batch.rows: + # The statement pins input_shape/output_shape to the flat [in_dim] / + # [out_dim] form and verification requires every transcript row to + # match it exactly; reject other recorded shapes here so a drifting + # capture path fails at generation time with a clear error instead of + # producing artifacts that can never verify. + if list(row.input_shape) != [in_dim] or list(row.output_shape) != [out_dim]: + raise ProofContractError( + "schema-3 statements require per-row shapes " + f"[{in_dim}]/[{out_dim}]; {batch.module_name}" + f"#{row.invocation_index} recorded input_shape=" + f"{row.input_shape} output_shape={row.output_shape}" + ) + expected = compute_delta_quantized( + first.a, + first.b, + row.x, + first.scaling_num, + first.scaling_den, + first.fixed_point, + ) + if expected != row.delta: + raise ProofContractError( + "witness delta does not match fixed-point LoRA relation for " + f"{batch.module_name}#{row.invocation_index}" + ) + + adapter_commitment = manifest_entry["adapter_commitment"] + digests = [ + row_digest( + session_id=batch.session_id, + module_name=batch.module_name, + invocation_index=row.invocation_index, + input_shape=row.input_shape, + output_shape=row.output_shape, + x_row=row.x, + delta_row=row.delta, + fixed_point=first.fixed_point, + scaling_num=first.scaling_num, + scaling_den=first.scaling_den, + rank=rank, + in_dim=in_dim, + out_dim=out_dim, + adapter_commitment=adapter_commitment, + manifest_commitment=manifest_commitment, + ) + for row in batch.rows + ] + bounds = derive_bounds( + batch.x_rows, + batch.delta_rows, + first.fixed_point, + first.scaling_num, + first.scaling_den, + ) + check_bounds_composition( + bounds, in_dim, rank, first.fixed_point, first.scaling_num, first.scaling_den + ) + + circuit = circuit_id_v3( + in_dim, + rank, + out_dim, + first.fixed_point, + first.scaling_num, + first.scaling_den, + chunk_rows, + ) + statement = { + "schema_version": SCHEMA_VERSION_V3, + "backend": BACKEND_ID_V3, + "proof_kind": PROOF_KIND_V3, + "session_id": batch.session_id, + "module_name": batch.module_name, + "start_invocation_index": batch.start_invocation_index, + "count": batch.count, + "target_chunk_rows": int(chunk_rows), + "input_shape": [in_dim], + "output_shape": [out_dim], + "row_digests": digests, + "batch_transcript_digest": batch_transcript_digest(digests), + "fixed_point": asdict(first.fixed_point), + "scaling": {"num": int(first.scaling_num), "den": int(first.scaling_den)}, + "adapter_commitment": adapter_commitment, + "manifest_commitment": manifest_commitment, + "circuit_id": circuit, + "vk_fingerprint": vk_fingerprint_v3(circuit), + } + statement["statement_digest"] = digest_hex(_statement_digest_payload(statement)) + return statement + + +def _check_manifest_entry_matches( + entry: dict[str, Any], + *, + module_name: str, + rank: int, + in_dim: int, + out_dim: int, + fixed_point: FixedPointConfig, + scaling_num: int, + scaling_den: int, +) -> None: + if entry.get("module_name") != module_name: + raise ProofContractError( + f"manifest entry module {entry.get('module_name')!r} != {module_name!r}" + ) + if "a_commitment" not in entry or "commitment_nonce" not in entry: + raise ProofContractError( + f"module {module_name} requires a schema-3 manifest entry with " + "pedersen commitments" + ) + if ( + int(entry.get("rank", -1)) != rank + or int(entry.get("in_dim", -1)) != in_dim + or int(entry.get("out_dim", -1)) != out_dim + or entry.get("fixed_point") != asdict(fixed_point) + or entry.get("scaling") != {"num": int(scaling_num), "den": int(scaling_den)} + ): + raise ProofContractError( + f"statement does not match pinned manifest entry for {module_name}" + ) + scheme = entry.get("adapter_commitment", {}).get("scheme") + if scheme != COMMITMENT_SCHEME_V3: + raise ProofContractError( + f"manifest adapter commitment scheme mismatch for {module_name}" + ) + + +# --------------------------------------------------------------------------- +# Adapter manifest (schema 3) +# --------------------------------------------------------------------------- + + +def adapter_manifest_entry_v3( + module_name: str, + a: list[list[int]], + b: list[list[int]], + scaling_num: int, + scaling_den: int, + fixed_point: FixedPointConfig, + secret_seed_hex: str, +) -> dict[str, Any]: + rank = len(a) + in_dim = len(a[0]) if a else 0 + out_dim = len(b) + _v2.validate_matrix(a, rank, in_dim, "A") + _v2.validate_matrix(b, out_dim, rank, "B") + _check_v3_dims(in_dim, rank, out_dim, 1) + native = _native_v3() + adapter_json = canonical_json( + { + "schema_version": SCHEMA_VERSION_V3, + "module_name": module_name, + "in_dim": in_dim, + "rank": rank, + "out_dim": out_dim, + "fixed_point": asdict(fixed_point), + "scaling_num": int(scaling_num), + "scaling_den": int(scaling_den), + "a": a, + "b": b, + } + ) + commit = json.loads(native.adapter_commit_v3(adapter_json, secret_seed_hex)) + for key in ("a_commitment", "b_commitment", "commitment_nonce", "range_proof"): + if key not in commit: + raise ProofContractError(f"native adapter_commit_v3 omitted {key}") + public = { + "module_name": module_name, + "rank": rank, + "in_dim": in_dim, + "out_dim": out_dim, + "fixed_point": asdict(fixed_point), + "scaling": {"num": int(scaling_num), "den": int(scaling_den)}, + "a_commitment": list(commit["a_commitment"]), + "b_commitment": list(commit["b_commitment"]), + "commitment_nonce": str(commit["commitment_nonce"]), + } + value = digest_hex(public) + entry = dict(public) + entry["adapter_commitment"] = {"scheme": COMMITMENT_SCHEME_V3, "value": value} + entry["range_proof"] = str(commit["range_proof"]) + return entry + + +def adapter_manifest_payload_v3(entries: Iterable[dict[str, Any]]) -> dict[str, Any]: + return { + "schema_version": SCHEMA_VERSION_V3, + "backend": BACKEND_ID_V3, + "commitment_scheme": COMMITMENT_SCHEME_V3, + "adapters": list(entries), + } + + +def write_adapter_manifest_v3( + path: str | os.PathLike[str], entries: Iterable[dict[str, Any]] +) -> dict[str, Any]: + payload = adapter_manifest_payload_v3(entries) + Path(path).write_text(canonical_json(payload) + "\n", encoding="utf-8") + return payload + + +def manifest_commitment_of(payload: dict[str, Any]) -> str: + return digest_hex(payload) + + +_MANIFEST_META_KEY = "__zklora_manifest__" + + +def _index_expected_adapter(index: dict[str, Any], entry: dict[str, Any]) -> None: + module_name = entry["module_name"] + if module_name in index: + raise ProofContractError(f"duplicate expected adapter for {module_name}") + if "a_commitment" in entry: + scheme = entry.get("adapter_commitment", {}).get("scheme") + if scheme != COMMITMENT_SCHEME_V3: + raise ProofContractError( + f"unsupported v3 adapter commitment scheme for {module_name}" + ) + expected_value = digest_hex( + { + "module_name": entry["module_name"], + "rank": int(entry["rank"]), + "in_dim": int(entry["in_dim"]), + "out_dim": int(entry["out_dim"]), + "fixed_point": entry["fixed_point"], + "scaling": entry["scaling"], + "a_commitment": list(entry["a_commitment"]), + "b_commitment": list(entry["b_commitment"]), + "commitment_nonce": str(entry["commitment_nonce"]), + } + ) + if entry["adapter_commitment"].get("value") != expected_value: + raise ProofContractError( + f"manifest adapter commitment mismatch for {module_name}" + ) + _check_v3_dims( + int(entry["in_dim"]), int(entry["rank"]), int(entry["out_dim"]), 1 + ) + native = _native_v3() + if not native.verify_adapter_manifest_v3(canonical_json(entry)): + raise ProofContractError( + f"manifest adapter range proof failed for {module_name}" + ) + index[module_name] = entry + + +def load_expected_adapters_any( + expected_adapters: str + | os.PathLike[str] + | dict[str, Any] + | Iterable[dict[str, Any]], +) -> dict[str, Any]: + """Index adapter entries by module, schema-aware. + + Schema-3 entries (those carrying pedersen commitments) are verified at pin + time via the native one-time A/B range proof; the manifest commitment over + the full pinned payload is stashed under a reserved key for statement + checks. v2 entries pass through untouched so legacy artifacts keep + verifying against them. + """ + + if isinstance(expected_adapters, (str, os.PathLike)): + data: Any = load_json(expected_adapters) + elif isinstance(expected_adapters, dict): + data = expected_adapters + else: + data = {"adapters": list(expected_adapters)} + + adapters = data.get("adapters") if isinstance(data, dict) else None + if adapters is None: + raise ProofContractError("expected adapter manifest must contain adapters") + + index: dict[str, Any] = {} + has_v3 = False + for position, entry in enumerate(adapters): + try: + _index_expected_adapter(index, entry) + except ProofContractError: + raise + except (KeyError, TypeError, IndexError, ValueError, AttributeError) as exc: + raise ProofContractError( + f"malformed expected adapter entry at position {position}: " + f"{type(exc).__name__}: {exc}" + ) from exc + if "a_commitment" in entry: + has_v3 = True + + manifest_commitment = None + if has_v3: + if not isinstance(data, dict) or data.get("schema_version") not in ( + SCHEMA_VERSION_V3, + ): + # Entries supplied without a pinned schema-3 manifest envelope still + # get a commitment over a canonical envelope so statements can bind. + data = adapter_manifest_payload_v3(adapters) + manifest_commitment = manifest_commitment_of( + {k: v for k, v in data.items() if k != _MANIFEST_META_KEY} + ) + index[_MANIFEST_META_KEY] = {"manifest_commitment": manifest_commitment} + return index + + +def _adapters_index_for_v2(index: dict[str, Any]) -> dict[str, Any]: + return {k: v for k, v in index.items() if k != _MANIFEST_META_KEY} + + +# --------------------------------------------------------------------------- +# Artifact writing +# --------------------------------------------------------------------------- + + +def _artifact_prefix( + output_dir: str | os.PathLike[str], statement: dict[str, Any] +) -> Path: + return Path(output_dir) / ( + f"{module_slug(statement['session_id'])}." + f"{module_slug(statement['module_name'])}." + f"{int(statement['start_invocation_index']):04d}" + ) + + +def _rows_json(x_rows: list[list[int]], delta_rows: list[list[int]]) -> str: + return canonical_json({"x_rows": x_rows, "delta_rows": delta_rows}) + + +def write_batch_artifacts( + output_dir: str | os.PathLike[str], + batch: BatchWitness, + manifest_entry: dict[str, Any], + manifest_commitment: str, + secret_seed_hex: str, + chunk_rows: int | None = None, +) -> dict[str, str]: + Path(output_dir).mkdir(parents=True, exist_ok=True) + chunk = target_chunk_rows() if chunk_rows is None else int(chunk_rows) + statement = statement_from_batch(batch, manifest_entry, manifest_commitment, chunk) + prefix = _artifact_prefix(output_dir, statement) + proof_path = Path(f"{prefix}.zklora.proof") + vk_path = Path(f"{prefix}.zklora.vk") + pk_path = Path(f"{prefix}.zklora.pk") + statement_path = Path(f"{prefix}{_STATEMENT_SUFFIX}") + meta_path = Path(f"{prefix}.zklora.meta.json") + + native = _native_v3() + first = batch.first + witness_json = canonical_json( + { + "a": first.a, + "b": first.b, + "secret_seed": secret_seed_hex, + "commitment_nonce": manifest_entry["commitment_nonce"], + } + ) + proof_bytes = native.prove_v3( + canonical_json(statement), + _rows_json(batch.x_rows, batch.delta_rows), + witness_json, + ) + + vk = { + "schema_version": SCHEMA_VERSION_V3, + "backend": BACKEND_ID_V3, + "proof_kind": PROOF_KIND_V3, + "commitment_scheme": COMMITMENT_SCHEME_V3, + "range_argument": RANGE_ARGUMENT_V3, + "generator_seed_id": GENERATOR_SEED_ID, + "circuit_id": statement["circuit_id"], + "vk_fingerprint": statement["vk_fingerprint"], + } + pk = { + "schema_version": SCHEMA_VERSION_V3, + "backend": BACKEND_ID_V3, + "circuit_id": statement["circuit_id"], + "pk_fingerprint": digest_hex({"pk_for_circuit": statement["circuit_id"]}), + } + meta = { + "schema_version": SCHEMA_VERSION_V3, + "backend": BACKEND_ID_V3, + "proof_kind": PROOF_KIND_V3, + "proof_system": PROOF_SYSTEM_V3, + "commitment_scheme": COMMITMENT_SCHEME_V3, + "range_argument": RANGE_ARGUMENT_V3, + "generator_seed_id": GENERATOR_SEED_ID, + "fiat_shamir": FIAT_SHAMIR_V3, + "security_level_bits": SECURITY_LEVEL_BITS_V3, + "audit_status": AUDIT_STATUS_V3, + "proof_file": proof_path.name, + "statement_file": statement_path.name, + "vk_file": vk_path.name, + "pk_file": pk_path.name, + "statement_digest": statement["statement_digest"], + "statement_file_digest": digest_hex(statement), + "proof_digest": hashlib.sha256(proof_bytes).hexdigest(), + "vk_digest": digest_hex(vk), + "pk_digest": digest_hex(pk), + "circuit_id": statement["circuit_id"], + "vk_fingerprint": statement["vk_fingerprint"], + "manifest_commitment": manifest_commitment, + "batch_transcript_digest": statement["batch_transcript_digest"], + } + + proof_path.write_bytes(proof_bytes) + vk_path.write_text(canonical_json(vk) + "\n", encoding="utf-8") + pk_path.write_text(canonical_json(pk) + "\n", encoding="utf-8") + statement_path.write_text(canonical_json(statement) + "\n", encoding="utf-8") + meta_path.write_text(canonical_json(meta) + "\n", encoding="utf-8") + return { + "proof": str(proof_path), + "vk": str(vk_path), + "pk": str(pk_path), + "statement": str(statement_path), + "meta": str(meta_path), + } + + +def generate_batch_proofs( + records: Iterable[InvocationWitness], + output_dir: str | os.PathLike[str], + adapter_manifest: str | os.PathLike[str] | dict[str, Any], + manifest_secret_path: str | os.PathLike[str] | None = None, + verbose: bool = False, +) -> tuple[int, int]: + """Generate schema-3 batch artifacts; returns (artifact_sets, total_params).""" + + if adapter_manifest is None: + raise ProofContractError( + "projection backend requires the pinned adapter manifest " + "(adapter_manifest=...)" + ) + if isinstance(adapter_manifest, (str, os.PathLike)): + manifest_payload = load_json(adapter_manifest) + else: + manifest_payload = adapter_manifest + if manifest_payload.get("schema_version") != SCHEMA_VERSION_V3: + raise ProofContractError( + "projection backend requires a schema-3 adapter manifest" + ) + manifest_commitment = manifest_commitment_of(manifest_payload) + entries_by_module = { + entry["module_name"]: entry for entry in manifest_payload["adapters"] + } + + secret_path = resolve_contributor_secret_path(manifest_secret_path) + ensure_secret_outside_artifacts(secret_path, output_dir) + seed = load_or_create_contributor_secret(secret_path) + + chunk = target_chunk_rows() + batches = build_batches(records, chunk) + if not batches: + raise ProofContractError( + "native zkLoRA proof generation requires captured invocation records" + ) + total_params = 0 + for batch in batches: + entry = entries_by_module.get(batch.module_name) + if entry is None: + raise ProofContractError( + f"module {batch.module_name} missing from pinned adapter manifest" + ) + write_batch_artifacts( + output_dir, batch, entry, manifest_commitment, seed, chunk + ) + first = batch.first + total_params += first.rank * first.in_dim + first.out_dim * first.rank + if verbose: + print( + f"Generated projection batch artifact for {batch.module_name}" + f"[{batch.start_invocation_index}, " + f"{batch.start_invocation_index + batch.count})" + ) + return len(batches), total_params + + +# --------------------------------------------------------------------------- +# Verification +# --------------------------------------------------------------------------- + + +def _expected_statement_name(statement: dict[str, Any]) -> str: + return ( + f"{module_slug(statement['session_id'])}." + f"{module_slug(statement['module_name'])}." + f"{int(statement['start_invocation_index']):04d}{_STATEMENT_SUFFIX}" + ) + + +def _transcript_rows_for_statement( + statement: dict[str, Any], + transcript_index: dict[tuple[str, str, int], TranscriptEntry], +) -> list[TranscriptEntry]: + session = statement["session_id"] + module = statement["module_name"] + start = int(statement["start_invocation_index"]) + count = int(statement["count"]) + rows: list[TranscriptEntry] = [] + for index in range(start, start + count): + entry = transcript_index.get((session, module, index)) + if entry is None: + raise ProofContractError( + f"statement row {module}#{index} is missing from verifier transcript" + ) + rows.append(entry) + return rows + + +def _unique_transcript_index( + entries: Iterable[TranscriptEntry], +) -> dict[tuple[str, str, int], TranscriptEntry]: + index: dict[tuple[str, str, int], TranscriptEntry] = {} + for entry in entries: + key = entry.key() + if key in index: + raise ProofContractError(f"duplicate transcript row for {key}") + index[key] = entry + return index + + +def verify_v3_artifact_set( + statement_path: str | os.PathLike[str], + transcript_index: dict[tuple[str, str, int], TranscriptEntry], + adapters_index: dict[str, Any], +) -> None: + # Artifacts are hostile input: structural surprises (missing keys, wrong + # JSON types, non-numeric strings) must surface as contract errors, not + # raw KeyError/TypeError crashes. ProofContractError subclasses ValueError + # and must pass through unwrapped. + try: + _verify_v3_artifact_set_checked( + statement_path, transcript_index, adapters_index + ) + except ProofContractError: + raise + except (KeyError, TypeError, IndexError, ValueError, AttributeError) as exc: + raise ProofContractError( + f"malformed schema-3 proof artifact {Path(statement_path).name}: " + f"{type(exc).__name__}: {exc}" + ) from exc + + +def _verify_v3_artifact_set_checked( + statement_path: str | os.PathLike[str], + transcript_index: dict[tuple[str, str, int], TranscriptEntry], + adapters_index: dict[str, Any], +) -> None: + statement = load_json(statement_path) + if ( + statement.get("schema_version") != SCHEMA_VERSION_V3 + or statement.get("backend") != BACKEND_ID_V3 + or statement.get("proof_kind") != PROOF_KIND_V3 + ): + raise ProofContractError("unsupported proof artifact schema/backend") + for forbidden in ("x", "delta", "lora_commitment", "invocation_index"): + if forbidden in statement: + raise ProofContractError( + f"schema-3 statement must not carry top-level {forbidden!r}" + ) + if statement.get("statement_digest") != digest_hex( + _statement_digest_payload(statement) + ): + raise ProofContractError("statement digest mismatch") + + statement_path = Path(statement_path) + if statement_path.name != _expected_statement_name(statement): + raise ProofContractError( + f"unexpected statement artifact name: {statement_path.name}" + ) + prefix = str(statement_path)[: -len(_STATEMENT_SUFFIX)] + proof_path = Path(f"{prefix}.zklora.proof") + vk_path = Path(f"{prefix}.zklora.vk") + pk_path = Path(f"{prefix}.zklora.pk") + meta_path = Path(f"{prefix}.zklora.meta.json") + + count = int(statement["count"]) + chunk = int(statement.get("target_chunk_rows", 0)) + if chunk < 1 or chunk > MAX_BATCH_ROWS: + raise ProofContractError("statement target_chunk_rows out of bounds") + if count < 1 or count > chunk: + raise ProofContractError("statement row count exceeds its target chunk") + if len(statement.get("row_digests", [])) != count: + raise ProofContractError("statement row digest count mismatch") + + module = statement["module_name"] + entry = adapters_index.get(module) + if entry is None or _MANIFEST_META_KEY == module: + raise ProofContractError( + "statement module is missing from expected adapter manifest" + ) + fixed_point = FixedPointConfig(**statement["fixed_point"]) + scaling_num = int(statement["scaling"]["num"]) + scaling_den = int(statement["scaling"]["den"]) + in_dim = int(statement["input_shape"][0]) + out_dim = int(statement["output_shape"][0]) + if "a_commitment" not in entry: + raise ProofContractError( + f"schema-3 artifact for {module} requires a schema-3 manifest entry" + ) + rank = int(entry["rank"]) + _check_v3_dims(in_dim, rank, out_dim, count) + _check_manifest_entry_matches( + entry, + module_name=module, + rank=rank, + in_dim=in_dim, + out_dim=out_dim, + fixed_point=fixed_point, + scaling_num=scaling_num, + scaling_den=scaling_den, + ) + if statement["adapter_commitment"] != entry["adapter_commitment"]: + raise ProofContractError("statement adapter commitment mismatch") + manifest_meta = adapters_index.get(_MANIFEST_META_KEY) or {} + pinned_commitment = manifest_meta.get("manifest_commitment") + if pinned_commitment is None: + raise ProofContractError( + "schema-3 artifacts require a pinned schema-3 manifest commitment" + ) + if statement["manifest_commitment"] != pinned_commitment: + raise ProofContractError("statement manifest commitment mismatch") + + expected_circuit = circuit_id_v3( + in_dim, rank, out_dim, fixed_point, scaling_num, scaling_den, chunk + ) + if statement["circuit_id"] != expected_circuit: + raise ProofContractError("statement circuit_id does not match expected circuit") + if statement["vk_fingerprint"] != vk_fingerprint_v3(expected_circuit): + raise ProofContractError( + "statement vk_fingerprint does not match expected circuit" + ) + + rows = _transcript_rows_for_statement(statement, transcript_index) + x_rows: list[list[int]] = [] + delta_rows: list[list[int]] = [] + for offset, row in enumerate(rows): + if ( + row.input_shape != statement["input_shape"] + or row.output_shape != statement["output_shape"] + or asdict(row.fixed_point) != statement["fixed_point"] + or row.scaling_num != scaling_num + or row.scaling_den != scaling_den + ): + raise ProofContractError( + f"transcript row {module}#{row.invocation_index} does not match " + "statement configuration" + ) + recomputed = row_digest( + session_id=row.session_id, + module_name=row.module_name, + invocation_index=row.invocation_index, + input_shape=row.input_shape, + output_shape=row.output_shape, + x_row=row.x, + delta_row=row.delta, + fixed_point=row.fixed_point, + scaling_num=row.scaling_num, + scaling_den=row.scaling_den, + rank=rank, + in_dim=in_dim, + out_dim=out_dim, + adapter_commitment=statement["adapter_commitment"], + manifest_commitment=statement["manifest_commitment"], + ) + if recomputed != statement["row_digests"][offset]: + raise ProofContractError( + f"row digest mismatch for {module}#{row.invocation_index}" + ) + x_rows.append([int(v) for v in row.x]) + delta_rows.append([int(v) for v in row.delta]) + if statement["batch_transcript_digest"] != batch_transcript_digest( + statement["row_digests"] + ): + raise ProofContractError("batch transcript digest mismatch") + + bounds = derive_bounds(x_rows, delta_rows, fixed_point, scaling_num, scaling_den) + check_bounds_composition( + bounds, in_dim, rank, fixed_point, scaling_num, scaling_den + ) + + vk = load_json(vk_path) + if ( + vk.get("schema_version") != SCHEMA_VERSION_V3 + or vk.get("backend") != BACKEND_ID_V3 + or vk.get("circuit_id") != expected_circuit + or vk.get("vk_fingerprint") != statement["vk_fingerprint"] + ): + raise ProofContractError("verification key descriptor mismatch") + pk = load_json(pk_path) + if ( + pk.get("schema_version") != SCHEMA_VERSION_V3 + or pk.get("circuit_id") != expected_circuit + ): + raise ProofContractError("proving key descriptor mismatch") + + meta = load_json(meta_path) + proof_bytes = proof_path.read_bytes() + if ( + meta.get("schema_version") != SCHEMA_VERSION_V3 + or meta.get("backend") != BACKEND_ID_V3 + ): + raise ProofContractError("unsupported metadata schema/backend") + if meta.get("proof_kind") != PROOF_KIND_V3: + raise ProofContractError(f"unsupported proof kind; expected {PROOF_KIND_V3}") + if meta.get("statement_digest") != statement["statement_digest"]: + raise ProofContractError("metadata statement digest mismatch") + if meta.get("statement_file_digest") != digest_hex(statement): + raise ProofContractError("metadata statement file digest mismatch") + if meta.get("proof_digest") != hashlib.sha256(proof_bytes).hexdigest(): + raise ProofContractError("metadata proof digest mismatch") + if meta.get("vk_digest") != digest_hex(vk): + raise ProofContractError("metadata vk digest mismatch") + if meta.get("pk_digest") != digest_hex(pk): + raise ProofContractError("metadata pk digest mismatch") + if meta.get("manifest_commitment") != statement["manifest_commitment"]: + raise ProofContractError("metadata manifest commitment mismatch") + if meta.get("batch_transcript_digest") != statement["batch_transcript_digest"]: + raise ProofContractError("metadata batch transcript digest mismatch") + + native = _native_v3() + entry_public = {k: v for k, v in entry.items()} + if not native.verify_v3( + canonical_json(statement), + _rows_json(x_rows, delta_rows), + canonical_json(entry_public), + proof_bytes, + ): + raise ProofContractError("proof bytes failed projection verification") + + +@dataclass(frozen=True) +class CoverageClaim: + session_id: str + module_name: str + start: int + count: int + source: str + + @property + def indices(self) -> range: + return range(self.start, self.start + self.count) + + +def check_coverage( + claims: list[CoverageClaim], transcript_entries: list[TranscriptEntry] +) -> None: + transcript_keys: dict[tuple[str, str], set[int]] = {} + for entry in transcript_entries: + key = (entry.session_id, entry.module_name) + indices = transcript_keys.setdefault(key, set()) + index = int(entry.invocation_index) + if index in indices: + duplicate = (entry.session_id, entry.module_name, index) + raise ProofContractError(f"duplicate transcript row for {duplicate}") + indices.add(index) + + claim_keys: dict[tuple[str, str], list[CoverageClaim]] = {} + for claim in claims: + claim_keys.setdefault((claim.session_id, claim.module_name), []).append(claim) + + # Keys are enumerated from the transcript first so a module with zero + # artifacts is reported missing, never silently skipped. + all_keys = sorted(set(transcript_keys) | set(claim_keys)) + for key in all_keys: + expected = transcript_keys.get(key, set()) + covered: set[int] = set() + for claim in sorted(claim_keys.get(key, []), key=lambda c: c.start): + overlap = covered.intersection(claim.indices) + if overlap: + raise ProofContractError( + f"duplicate proof coverage for {key} rows {sorted(overlap)} " + f"(claimed again by {claim.source})" + ) + covered.update(claim.indices) + if covered != expected: + missing = sorted(expected - covered) + extra = sorted(covered - expected) + raise ProofContractError( + f"proof transcript coverage mismatch for {key} " + f"missing={missing} extra={extra}" + ) + + +def expand_statement_rows( + statement: dict[str, Any], + transcript: str | os.PathLike[str] | Iterable[Any] | None = None, +) -> list[TranscriptEntry]: + """Resolve a statement (v2 or v3) to its covered transcript rows.""" + + schema = statement.get("schema_version") + if schema == _v2.SCHEMA_VERSION: + return [_v2.transcript_entry_from_statement(statement)] + if schema != SCHEMA_VERSION_V3: + raise ProofContractError(f"unsupported statement schema: {schema}") + if transcript is None: + raise ProofContractError( + "schema-3 statements are digest-only; pass the verifier transcript" + ) + entries = load_transcript(transcript) + index = _unique_transcript_index(entries) + return _transcript_rows_for_statement(statement, index) + + +def verify_artifacts_mixed( + proof_dir: str | os.PathLike[str], + transcript: str | os.PathLike[str] | Iterable[Any], + expected_adapters: str + | os.PathLike[str] + | dict[str, Any] + | Iterable[dict[str, Any]], +) -> tuple[float, int]: + import time + + start = time.time() + reject_secrets_in_artifact_dir(proof_dir) + entries = load_transcript(transcript) + transcript_index = _unique_transcript_index(entries) + adapters_index = load_expected_adapters_any(expected_adapters) + v2_index = _adapters_index_for_v2(adapters_index) + + statement_files = sorted(Path(proof_dir).glob(f"*{_STATEMENT_SUFFIX}")) + claims: list[CoverageClaim] = [] + for statement_file in statement_files: + statement = load_json(statement_file) + schema = statement.get("schema_version") + if schema == _v2.SCHEMA_VERSION: + _v2.verify_artifact_set(statement_file, entries, v2_index) + claims.append( + CoverageClaim( + session_id=statement["session_id"], + module_name=statement["module_name"], + start=int(statement["invocation_index"]), + count=1, + source=str(statement_file), + ) + ) + elif schema == SCHEMA_VERSION_V3: + verify_v3_artifact_set(statement_file, transcript_index, adapters_index) + claims.append( + CoverageClaim( + session_id=statement["session_id"], + module_name=statement["module_name"], + start=int(statement["start_invocation_index"]), + count=int(statement["count"]), + source=str(statement_file), + ) + ) + else: + raise ProofContractError( + f"unsupported proof artifact schema: {schema} in {statement_file}" + ) + + check_coverage(claims, entries) + return time.time() - start, len(statement_files) diff --git a/src/zklora/zk_proof_generator.py b/src/zklora/zk_proof_generator.py index 065adff..84279ad 100644 --- a/src/zklora/zk_proof_generator.py +++ b/src/zklora/zk_proof_generator.py @@ -1,37 +1,82 @@ from __future__ import annotations +import os from pathlib import Path -from typing import Iterable +from typing import Any, Iterable from .proof_contract import ( InvocationWitness, ProofContractError, TranscriptEntry, - verify_artifacts, write_invocation_artifacts, ) +_BACKEND_ENV = "ZKLORA_PROVER_BACKEND" +_BACKEND_PROJECTION = "projection-v1" +_BACKEND_LEGACY = "legacy-halo2" +# The legacy backend stays the default until the native projection prover +# (prove_v3/verify_v3) ships; flipping earlier would break proof generation on +# every real install. The default flips to projection-v1 in the milestone that +# delivers the native backend (M2). +_BACKEND_DEFAULT = _BACKEND_LEGACY + + +def prover_backend() -> str: + """Resolve the active prover backend from ``ZKLORA_PROVER_BACKEND``.""" + + backend = os.environ.get(_BACKEND_ENV, _BACKEND_DEFAULT) + if backend not in (_BACKEND_PROJECTION, _BACKEND_LEGACY): + raise ProofContractError( + f"unknown {_BACKEND_ENV} value {backend!r}; expected " + f"{_BACKEND_PROJECTION!r} or {_BACKEND_LEGACY!r}" + ) + return backend + + +def _generate_proofs_legacy( + record_list: list[InvocationWitness], output_dir: str, verbose: bool +) -> tuple[int, int]: + proofs = 0 + total_params = 0 + for record in record_list: + write_invocation_artifacts(output_dir, record) + total_params += record.rank * record.in_dim + record.out_dim * record.rank + proofs += 1 + if verbose: + print( + f"Generated native zkLoRA proof artifact for " + f"{record.module_name}#{record.invocation_index}" + ) + return proofs, total_params + def generate_proofs( records: Iterable[InvocationWitness] | None = None, output_dir: str = "proof_artifacts", verbose: bool = False, + *, + adapter_manifest: str | os.PathLike[str] | dict[str, Any] | None = None, + manifest_secret_path: str | os.PathLike[str] | None = None, **_legacy_kwargs, ) -> tuple[float, float, float, int, int]: """Generate native zkLoRA proof artifacts for captured LoRA invocations. - The old external-backend implementation scanned model-export directories. The native backend is - transcript-first: callers pass invocation witnesses captured during multi-party - inference. Legacy keyword arguments are accepted so old callers fail with a clear - no-records result instead of importing removed proof backends. + The default ``legacy-halo2`` backend writes one schema-2 artifact set per + invocation row. Setting ``ZKLORA_PROVER_BACKEND=projection-v1`` opts into + the schema-3 batch backend, which writes one artifact set per contiguous + batch of invocations (``proofs`` in the returned tuple then counts + artifact sets, not rows) and requires the pinned schema-3 adapter manifest + plus the contributor secret used to commit it. The projection backend + additionally requires a native module built with v3 support; until that + ships, opting in fails with a clear error. Legacy keyword arguments are + accepted so old callers fail with a clear no-records result instead of + importing removed proof backends. """ import time start = time.time() Path(output_dir).mkdir(parents=True, exist_ok=True) - proofs = 0 - total_params = 0 record_list = list(records or []) if not record_list: @@ -39,15 +84,19 @@ def generate_proofs( "native zkLoRA proof generation requires captured invocation records" ) - for record in record_list: - write_invocation_artifacts(output_dir, record) - total_params += record.rank * record.in_dim + record.out_dim * record.rank - proofs += 1 - if verbose: - print( - f"Generated native zkLoRA proof artifact for " - f"{record.module_name}#{record.invocation_index}" - ) + backend = prover_backend() + if backend == _BACKEND_LEGACY: + proofs, total_params = _generate_proofs_legacy(record_list, output_dir, verbose) + else: + from .proof_v3 import generate_batch_proofs + + proofs, total_params = generate_batch_proofs( + record_list, + output_dir, + adapter_manifest=adapter_manifest, + manifest_secret_path=manifest_secret_path, + verbose=verbose, + ) elapsed = time.time() - start return (0.0, 0.0, elapsed, total_params, proofs) @@ -69,7 +118,9 @@ def batch_verify_proofs( raise ProofContractError( "native zkLoRA verification requires a pre-inference adapter manifest" ) - total_time, count = verify_artifacts(proof_dir, transcript, expected_adapters) + from .proof_v3 import verify_artifacts_mixed + + total_time, count = verify_artifacts_mixed(proof_dir, transcript, expected_adapters) if verbose: print(f"Verified {count} native zkLoRA proof artifacts in {total_time:.2f}s") return total_time, count diff --git a/tests/test_v3_artifacts.py b/tests/test_v3_artifacts.py new file mode 100644 index 0000000..331769c --- /dev/null +++ b/tests/test_v3_artifacts.py @@ -0,0 +1,579 @@ +import base64 +import hashlib +import json +import sys +from dataclasses import replace +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +import zklora.proof_contract as proof_contract # noqa: E402 +from zklora.proof_contract import ( # noqa: E402 + FixedPointConfig, + InvocationWitness, + ProofContractError, + TranscriptEntry, + adapter_manifest_entry, + canonical_json, + compute_delta_quantized, + flatten, + load_json, + quantize_nested, + write_invocation_artifacts, +) +from zklora.proof_v3 import ( # noqa: E402 + BACKEND_ID_V3, + PROOF_KIND_V3, + adapter_manifest_entry_v3, + adapter_manifest_payload_v3, + build_batches, + check_bounds_composition, + derive_bounds, + expand_statement_rows, +) +from zklora.zk_proof_generator import batch_verify_proofs, generate_proofs # noqa: E402 + +MODULE = "transformer.h.0.attn.c_attn" +SECOND_MODULE = "transformer.h.1.attn.c_attn" +FP = FixedPointConfig(scale_bits=2, value_bits=16, intermediate_bits=32) +SEED = "ab" * 32 + + +class FakeNativeBoth: + """Deterministic stand-in for the native module covering v2 and v3.""" + + def adapter_commitment(self, adapter_json): + return str(abs(hash(adapter_json))) + + def prove(self, statement_json, witness_json): + return f"{statement_json}|{witness_json}".encode() + + def verify(self, statement_json, proof): + return proof.startswith(statement_json.encode() + b"|") + + def adapter_commit_v3(self, adapter_json, seed_hex): + base = hashlib.sha256((adapter_json + seed_hex).encode()).hexdigest() + return canonical_json( + { + "a_commitment": [base], + "b_commitment": [base[::-1]], + "commitment_nonce": hashlib.sha256(base.encode()).hexdigest(), + "range_proof": base64.b64encode(b"fake-range-proof").decode(), + } + ) + + def verify_adapter_manifest_v3(self, entry_json): + return True + + def prove_v3(self, statement_json, rows_json, witness_json): + return ( + b"ZKL3" + + hashlib.sha256(statement_json.encode()).digest() + + hashlib.sha256(rows_json.encode()).digest() + + hashlib.sha256(witness_json.encode()).digest() + ) + + def verify_v3(self, statement_json, rows_json, manifest_entry_json, proof): + return ( + proof[:4] == b"ZKL3" + and proof[4:36] == hashlib.sha256(statement_json.encode()).digest() + and proof[36:68] == hashlib.sha256(rows_json.encode()).digest() + ) + + +@pytest.fixture(autouse=True) +def fake_native(monkeypatch): + monkeypatch.setattr(proof_contract, "_native_module", lambda: FakeNativeBoth()) + # These tests exercise the opt-in projection backend; the shipped default + # stays legacy-halo2 until the native v3 prover lands (see + # test_default_backend_is_legacy_until_native_v3_ships). + monkeypatch.setenv("ZKLORA_PROVER_BACKEND", "projection-v1") + monkeypatch.delenv("ZKLORA_CHUNK_ROWS", raising=False) + monkeypatch.delenv("ZKLORA_CONTRIBUTOR_SECRET", raising=False) + + +def _adapter(): + a = quantize_nested([[2.0, -1.0]], FP) + b = quantize_nested([[3.0], [-2.0]], FP) + return a, b + + +def _records(n_rows=5, start=0, session_id="s1", module=MODULE): + a, b = _adapter() + records = [] + for offset in range(n_rows): + x = flatten(quantize_nested([1.5 + offset, -0.5], FP)) + delta = compute_delta_quantized(a, b, x, 1, 2, FP) + records.append( + InvocationWitness( + session_id=session_id, + module_name=module, + invocation_index=start + offset, + input_shape=[2], + output_shape=[2], + x=x, + delta=delta, + a=a, + b=b, + scaling_num=1, + scaling_den=2, + adapter_metadata={"rank": 1}, + fixed_point=FP, + ) + ) + return records + + +def _transcript(records): + return [ + TranscriptEntry( + session_id=r.session_id, + module_name=r.module_name, + invocation_index=r.invocation_index, + input_shape=r.input_shape, + output_shape=r.output_shape, + x=r.x, + delta=r.delta, + fixed_point=r.fixed_point, + scaling_num=r.scaling_num, + scaling_den=r.scaling_den, + ) + for r in records + ] + + +def _v3_manifest(extra_entries=()): + a, b = _adapter() + entry = adapter_manifest_entry_v3(MODULE, a, b, 1, 2, FP, SEED) + return adapter_manifest_payload_v3([entry, *extra_entries]) + + +def _generate(tmp_path, records, payload, chunk, monkeypatch): + monkeypatch.setenv("ZKLORA_CHUNK_ROWS", str(chunk)) + secret = tmp_path / "secrets" / "contributor.secret.json" + out_dir = tmp_path / "artifacts" + generate_proofs( + records, + output_dir=str(out_dir), + adapter_manifest=payload, + manifest_secret_path=secret, + ) + return out_dir + + +def test_v3_batch_artifacts_roundtrip(tmp_path, monkeypatch): + records = _records(5) + payload = _v3_manifest() + out_dir = _generate(tmp_path, records, payload, chunk=2, monkeypatch=monkeypatch) + + names = sorted(p.name for p in out_dir.glob("*.zklora.statement.json")) + assert names == [ + f"s1.{MODULE}.0000.zklora.statement.json", + f"s1.{MODULE}.0002.zklora.statement.json", + f"s1.{MODULE}.0004.zklora.statement.json", + ] + statement = load_json(out_dir / names[0]) + assert statement["schema_version"] == 3 + assert statement["backend"] == BACKEND_ID_V3 + assert statement["count"] == 2 + for forbidden in ("x", "delta", "lora_commitment", "invocation_index"): + assert forbidden not in statement + meta = load_json(out_dir / f"s1.{MODULE}.0000.zklora.meta.json") + assert meta["proof_kind"] == PROOF_KIND_V3 + assert meta["audit_status"] == "unaudited" + + elapsed, count = batch_verify_proofs( + proof_dir=str(out_dir), + transcript=_transcript(records), + expected_adapters=payload, + ) + assert elapsed >= 0 + assert count == 3 + + +def test_nonzero_start_and_short_final_chunk(tmp_path, monkeypatch): + records = _records(5, start=3) + payload = _v3_manifest() + out_dir = _generate(tmp_path, records, payload, chunk=4, monkeypatch=monkeypatch) + + names = sorted(p.name for p in out_dir.glob("*.zklora.statement.json")) + assert names == [ + f"s1.{MODULE}.0003.zklora.statement.json", + f"s1.{MODULE}.0007.zklora.statement.json", + ] + assert load_json(out_dir / names[0])["count"] == 4 + assert load_json(out_dir / names[1])["count"] == 1 + _, count = batch_verify_proofs( + proof_dir=str(out_dir), + transcript=_transcript(records), + expected_adapters=payload, + ) + assert count == 2 + + +def test_missing_batch_rejected(tmp_path, monkeypatch): + records = _records(5) + payload = _v3_manifest() + out_dir = _generate(tmp_path, records, payload, chunk=2, monkeypatch=monkeypatch) + (out_dir / f"s1.{MODULE}.0002.zklora.statement.json").unlink() + + with pytest.raises(ProofContractError, match="coverage mismatch"): + batch_verify_proofs( + proof_dir=str(out_dir), + transcript=_transcript(records), + expected_adapters=payload, + ) + + +def test_overlapping_coverage_rejected(tmp_path, monkeypatch): + records = _records(5) + payload = _v3_manifest() + out_dir = _generate(tmp_path, records, payload, chunk=2, monkeypatch=monkeypatch) + # Regenerate with a different chunking into the same directory: the new + # 5-row batch at start 0 overlaps the surviving 2-row batches. + _generate(tmp_path, records, payload, chunk=5, monkeypatch=monkeypatch) + + with pytest.raises(ProofContractError, match="duplicate proof coverage"): + batch_verify_proofs( + proof_dir=str(out_dir), + transcript=_transcript(records), + expected_adapters=payload, + ) + + +def test_transcript_tamper_and_reorder_rejected(tmp_path, monkeypatch): + records = _records(4) + payload = _v3_manifest() + out_dir = _generate(tmp_path, records, payload, chunk=2, monkeypatch=monkeypatch) + + tampered = _transcript(records) + tampered[1] = replace(tampered[1], x=[tampered[1].x[0] + 1, *tampered[1].x[1:]]) + with pytest.raises(ProofContractError, match="row digest mismatch"): + batch_verify_proofs( + proof_dir=str(out_dir), + transcript=tampered, + expected_adapters=payload, + ) + + reordered = _transcript(records) + first, second = reordered[0], reordered[1] + reordered[0] = replace(first, x=second.x, delta=second.delta) + reordered[1] = replace(second, x=first.x, delta=first.delta) + with pytest.raises(ProofContractError, match="row digest mismatch"): + batch_verify_proofs( + proof_dir=str(out_dir), + transcript=reordered, + expected_adapters=payload, + ) + + +def test_duplicate_transcript_rows_rejected(tmp_path, monkeypatch): + records = _records(1) + payload = _v3_manifest() + out_dir = _generate(tmp_path, records, payload, chunk=1, monkeypatch=monkeypatch) + + transcript = _transcript(records) + duplicate = replace(transcript[0], x=[transcript[0].x[0] + 1, *transcript[0].x[1:]]) + with pytest.raises(ProofContractError, match="duplicate transcript row"): + batch_verify_proofs( + proof_dir=str(out_dir), + transcript=[duplicate, *transcript], + expected_adapters=payload, + ) + + +def test_zero_artifact_module_reported_missing(tmp_path, monkeypatch): + records = _records(2) + payload = _v3_manifest() + out_dir = _generate(tmp_path, records, payload, chunk=2, monkeypatch=monkeypatch) + + ghost = _records(1, module=SECOND_MODULE) + with pytest.raises(ProofContractError, match="coverage mismatch"): + batch_verify_proofs( + proof_dir=str(out_dir), + transcript=_transcript(records + ghost), + expected_adapters=payload, + ) + + +def test_secret_path_guards(tmp_path, monkeypatch): + records = _records(2) + payload = _v3_manifest() + out_dir = tmp_path / "artifacts" + with pytest.raises(ProofContractError, match="must not live inside"): + generate_proofs( + records, + output_dir=str(out_dir), + adapter_manifest=payload, + manifest_secret_path=out_dir / "contributor.secret.json", + ) + + out_dir = _generate(tmp_path, records, payload, chunk=2, monkeypatch=monkeypatch) + (out_dir / "leaked.secret.json").write_text("{}", encoding="utf-8") + with pytest.raises(ProofContractError, match="refusing to verify"): + batch_verify_proofs( + proof_dir=str(out_dir), + transcript=_transcript(records), + expected_adapters=payload, + ) + + +def test_mixed_v2_and_v3_directory(tmp_path, monkeypatch): + a, b = _adapter() + v2_witness = InvocationWitness( + session_id="s1", + module_name=SECOND_MODULE, + invocation_index=0, + input_shape=[2], + output_shape=[2], + x=flatten(quantize_nested([1.5, -0.5], FP)), + delta=compute_delta_quantized( + a, b, flatten(quantize_nested([1.5, -0.5], FP)), 1, 2, FP + ), + a=a, + b=b, + scaling_num=1, + scaling_den=2, + adapter_metadata={"rank": 1}, + fixed_point=FP, + ) + v2_entry = adapter_manifest_entry(SECOND_MODULE, a, b, 1, 2, FP) + payload = _v3_manifest(extra_entries=[v2_entry]) + + records = _records(3) + out_dir = _generate(tmp_path, records, payload, chunk=2, monkeypatch=monkeypatch) + write_invocation_artifacts(out_dir, v2_witness) + + transcript = _transcript(records + [v2_witness]) + _, count = batch_verify_proofs( + proof_dir=str(out_dir), + transcript=transcript, + expected_adapters=payload, + ) + assert count == 3 # two v3 batches + one v2 single-row artifact + + +def test_expand_statement_rows_v2_and_v3(tmp_path, monkeypatch): + records = _records(4) + payload = _v3_manifest() + out_dir = _generate(tmp_path, records, payload, chunk=3, monkeypatch=monkeypatch) + transcript = _transcript(records) + + statement = load_json(out_dir / f"s1.{MODULE}.0000.zklora.statement.json") + rows = expand_statement_rows(statement, transcript) + assert [r.invocation_index for r in rows] == [0, 1, 2] + assert rows[0].x == records[0].x + + v2_statement = proof_contract.statement_from_witness(records[0]) + v2_rows = expand_statement_rows(v2_statement) + assert len(v2_rows) == 1 + assert v2_rows[0].x == records[0].x + + +def test_legacy_backend_generates_v2_artifacts(tmp_path, monkeypatch): + monkeypatch.setenv("ZKLORA_PROVER_BACKEND", "legacy-halo2") + records = _records(3) + a, b = _adapter() + out_dir = tmp_path / "artifacts" + generate_proofs(records, output_dir=str(out_dir)) + + statements = sorted(out_dir.glob("*.zklora.statement.json")) + assert len(statements) == 3 + assert all(load_json(p)["schema_version"] == 2 for p in statements) + + monkeypatch.delenv("ZKLORA_PROVER_BACKEND") + _, count = batch_verify_proofs( + proof_dir=str(out_dir), + transcript=_transcript(records), + expected_adapters=[adapter_manifest_entry(MODULE, a, b, 1, 2, FP)], + ) + assert count == 3 + + +def test_default_backend_is_legacy_until_native_v3_ships(tmp_path, monkeypatch): + # With ZKLORA_PROVER_BACKEND unset, proof generation must keep working on + # installs whose native module predates the projection backend, so the + # default stays legacy-halo2 until prove_v3/verify_v3 ship. + monkeypatch.delenv("ZKLORA_PROVER_BACKEND", raising=False) + records = _records(2) + out_dir = tmp_path / "artifacts" + generate_proofs(records, output_dir=str(out_dir)) + + statements = sorted(out_dir.glob("*.zklora.statement.json")) + assert len(statements) == 2 + assert all(load_json(p)["schema_version"] == 2 for p in statements) + + +def test_unknown_backend_rejected(tmp_path, monkeypatch): + monkeypatch.setenv("ZKLORA_PROVER_BACKEND", "sumcheck-mle") + with pytest.raises(ProofContractError, match="unknown ZKLORA_PROVER_BACKEND"): + generate_proofs(_records(1), output_dir=str(tmp_path / "artifacts")) + + +def test_statement_and_proof_tamper_rejected(tmp_path, monkeypatch): + records = _records(2) + payload = _v3_manifest() + out_dir = _generate(tmp_path, records, payload, chunk=2, monkeypatch=monkeypatch) + statement_path = out_dir / f"s1.{MODULE}.0000.zklora.statement.json" + + statement = load_json(statement_path) + statement["count"] = 1 + statement["row_digests"] = statement["row_digests"][:1] + statement_path.write_text( + json.dumps(statement, sort_keys=True, separators=(",", ":")) + "\n", + encoding="utf-8", + ) + # Anchored: a failed digest check is a genuine contract error and must not + # be re-wrapped by the malformed-artifact guard (ProofContractError is a + # ValueError subclass). + with pytest.raises(ProofContractError, match="^statement digest mismatch"): + batch_verify_proofs( + proof_dir=str(out_dir), + transcript=_transcript(records), + expected_adapters=payload, + ) + + out_dir = _generate( + tmp_path / "second", records, payload, chunk=2, monkeypatch=monkeypatch + ) + (out_dir / f"s1.{MODULE}.0000.zklora.proof").write_bytes(b"ZKL3tampered") + with pytest.raises(ProofContractError, match="metadata proof digest mismatch"): + batch_verify_proofs( + proof_dir=str(out_dir), + transcript=_transcript(records), + expected_adapters=payload, + ) + + +def test_manifest_commitment_binds_statements(tmp_path, monkeypatch): + records = _records(2) + payload = _v3_manifest() + out_dir = _generate(tmp_path, records, payload, chunk=2, monkeypatch=monkeypatch) + + a, b = _adapter() + other_entry = adapter_manifest_entry_v3(SECOND_MODULE, a, b, 1, 2, FP, SEED) + drifted = adapter_manifest_payload_v3([payload["adapters"][0], other_entry]) + with pytest.raises(ProofContractError, match="manifest commitment mismatch"): + batch_verify_proofs( + proof_dir=str(out_dir), + transcript=_transcript(records), + expected_adapters=drifted, + ) + + +def test_batch_grouping_contract(): + records = _records(3) + bad = records + _records(1, start=5) + with pytest.raises(ProofContractError, match="non-contiguous"): + build_batches(bad, target_rows=4) + + duplicated = records + [records[-1]] + with pytest.raises(ProofContractError, match="duplicate invocation index"): + build_batches(duplicated, target_rows=4) + + a, b = _adapter() + drifted_b = quantize_nested([[3.0], [-1.0]], FP) + mixed = records + [ + InvocationWitness( + **{ + **records[-1].__dict__, + "invocation_index": 3, + "b": drifted_b, + "delta": compute_delta_quantized(a, drifted_b, records[-1].x, 1, 2, FP), + } + ) + ] + with pytest.raises(ProofContractError, match="inconsistent adapter weights"): + build_batches(mixed, target_rows=4) + + +def test_shape_drift_rejected_at_generation(tmp_path, monkeypatch): + records = _records(2) + records[1] = replace(records[1], input_shape=[1, 2]) + payload = _v3_manifest() + with pytest.raises(ProofContractError, match="require per-row shapes"): + _generate(tmp_path, records, payload, chunk=2, monkeypatch=monkeypatch) + + +def test_malformed_statement_rejected_cleanly(tmp_path, monkeypatch): + from zklora.proof_contract import digest_hex + + records = _records(2) + payload = _v3_manifest() + out_dir = _generate(tmp_path, records, payload, chunk=2, monkeypatch=monkeypatch) + statement_path = out_dir / f"s1.{MODULE}.0000.zklora.statement.json" + + # A hostile statement with a self-consistent digest but the wrong JSON + # type must fail as a contract error, not a raw TypeError/KeyError. + statement = load_json(statement_path) + statement["fixed_point"] = "not-a-mapping" + statement["statement_digest"] = digest_hex( + {k: v for k, v in statement.items() if k != "statement_digest"} + ) + statement_path.write_text( + json.dumps(statement, sort_keys=True, separators=(",", ":")) + "\n", + encoding="utf-8", + ) + with pytest.raises(ProofContractError, match="malformed schema-3 proof artifact"): + batch_verify_proofs( + proof_dir=str(out_dir), + transcript=_transcript(records), + expected_adapters=payload, + ) + + +def test_malformed_manifest_entry_rejected_cleanly(tmp_path, monkeypatch): + records = _records(2) + payload = _v3_manifest() + out_dir = _generate(tmp_path, records, payload, chunk=2, monkeypatch=monkeypatch) + + broken = json.loads(json.dumps(payload)) + del broken["adapters"][0]["rank"] + with pytest.raises(ProofContractError, match="malformed expected adapter entry"): + batch_verify_proofs( + proof_dir=str(out_dir), + transcript=_transcript(records), + expected_adapters=broken, + ) + + +def test_secret_creation_race_returns_winner_seed(tmp_path, monkeypatch): + from zklora import proof_v3 + + path = tmp_path / "secrets" / "contributor.secret.json" + path.parent.mkdir(parents=True) + winner_seed = "cd" * 32 + path.write_text(canonical_json({"seed": winner_seed}) + "\n", encoding="utf-8") + + # Simulate losing the O_EXCL creation race: the existence pre-check says + # the file is absent, but creation finds it already present. + real_exists = Path.exists + monkeypatch.setattr( + Path, + "exists", + lambda self: False if self == path else real_exists(self), + ) + assert proof_v3.load_or_create_contributor_secret(path) == winner_seed + + +def test_bounds_composition_guards(): + bounds = derive_bounds([[4, -2]], [[6, -4]], FP, 1, 2) + assert bounds["n_rem"] == FP.scale_bits + assert bounds["proved_u"] >= bounds["b_u"] + check_bounds_composition( + bounds, in_dim=2, rank=1, fixed_point=FP, scaling_num=1, scaling_den=2 + ) + + huge = FixedPointConfig(scale_bits=20, value_bits=130, intermediate_bits=260) + big_bounds = derive_bounds([[huge.value_bound]], [[huge.value_bound]], huge, 1, 1) + with pytest.raises(ProofContractError, match="field-safe"): + check_bounds_composition( + big_bounds, + in_dim=2, + rank=1, + fixed_point=huge, + scaling_num=1, + scaling_den=1, + )