Feat/prune sealed merkle nodes

[nol4lej]

The problem

A sealed Merkle tree kept ~1,048,574 MerkleNodes entries forever — roughly 72 MiB each, growing without bound across up to 4096 trees, and every full node had to retain all of it.

That storage serves exactly one purpose: handing Merkle paths to wallets so they can build a spend proof. Verified by grep — one reader (MerkleTreeService::get_merkle_path), one writer (insert_leaf). No dispatchable reads it: not unshield, not private_transfer, not validate_unsigned. The chain verifies ZK proofs against a known root and never walks the tree.

So MerkleNodes is not consensus data, it is a query convenience. Dropping it cannot make a note unspendable, as long as SealedTreeRoots keeps the anchor — and that is permanent and tiny, one entry per tree. MerkleLeaves is never pruned, so a recompute always has something to read from.

The approach: partial pruning by level

Nodes concentrate at the bottom. Level 1 holds 524,288 entries (50%); level 10 holds 1,024 (0.1%). Keeping only the upper levels of sealed trees lets a path be rebuilt from the subtree below the cut.

Cut	Entries/tree	Freed	Hashes	Native	Wasm (×3)
8	8,190	99.22%	255	15 ms	45 ms
10	2,046	99.80%	1,023	59 ms	178 ms
12	510	99.95%	4,095	238 ms	714 ms

Level 10 frees 99.8% and keeps the worst case around 180 ms under Wasm. Level 12 frees a further 0.15% for nearly four times the work. Sized from a measured 58.1 µs/hash for Poseidon (release build, two consistent runs at 5k and 20k hashes).

The active tree is never touched: still O(depth), 20 point reads and zero hashes.

What changed

New Config constant SealedTreePrunedBelowLevel (production 10), parameterised rather than fixed because the recompute cost tracks validator hardware. integrity_test rejects a cut outside 1..DEFAULT_TREE_DEPTH.

get_merkle_path rebuilds pruned siblings from MerkleLeaves on demand. Only the sibling subtree is recomputed, never the whole tree, and a sealed tree is immutable so the result is byte-identical to what was stored.

An on_idle hook sweeps sealed trees in bounded batches, parking its position in SealedPruneCursor. It is bounded twice — by the block's leftover weight and by MAX_PRUNED_NODES_PER_BLOCK (512), which caps trie churn on an idle chain. It charges every probe rather than only removals, so a level that is already clean cannot scan for free.

Why consumers are safe

Checked app, ts-sdk and indexer:

• No batching. Nothing requests proofs for many commitments at once; wallet rescan does not touch proofs. Worst case per user action is 8 (a 2-note transfer); typical is 1–2.
• The retry loop does not apply to sealed trees. privateTransfer.ts retries up to 3 times when roots diverge, but a sealed pair converges on the first attempt because its root is immutable.
• No per-call RPC timeout — only connection (15s) and heartbeat (30s). A slower proof lengthens a spinner; it does not fail.
• The SDK is depth-agnostic: merkle.ts derives depth = siblings.length. No hardcoded 20.

The proof is fetched immediately before Groth16 generation, which takes seconds. 178 ms is marginal there, and it sits under a step already labelled "Fetching Merkle proof…".

Benchmarks

prune_sealed_nodes measured on the VPS (AMD EPYC-Genoa, 50 steps, 20 repeats): 12.68 µs per node plus a 0.25 µs base.

The placeholder it replaces charged nothing for execution and leaned entirely on DbWeight, over-declaring the per-node cost by roughly 10x (125 ms). The sweep would have run, just in far smaller batches than a block can afford.

With the real figure, the 512-node ceiling costs ~6.5 ms — about 0.3% of a 2s block — so MAX_PRUNED_NODES_PER_BLOCK needs no adjustment. A full sealed tree (1,046,528 prunable nodes) drains in ~2,044 blocks, around 3.4 hours at 6s.

The other extrinsics move 8–20% higher, which is expected rather than a regression: benchmark amounts now scale from the configured relay fee (1e18 planck) instead of a flat 1e4, so balances encode into more bytes. shield 878ms → 1028ms, unshield 852ms → 987ms, private_transfer 813ms → 919ms per n.

That amount change was itself a fix. Removing MinShieldAmount in an earlier PR took with it the expression the benchmarks used (MinShieldAmount * 10), and a flat 10_000 planck replaced it. But unshield pays the relayer out of amount, and production min_relay_fee is 1e15 — a hundred billion times larger, so the extrinsic could never succeed. It now derives from the fee the runtime configures, so another literal cannot rot the same way.

Migration

None needed. Nothing is pruned until a tree seals, which takes 2^20 leaves. No live chain has reached that — testnet is at ~134k — so the on_idle sweep reaches already-sealed trees on its own whenever they appear.

Verification

Pallet tests	✅ 308 (11 new)
Precompile tests	✅ 65
Runtime + try-runtime + runtime-benchmarks	✅
Dev-node E2E	✅ 14/14

The decisive unit test captures the Merkle path of every leaf in a sealed tree, prunes, and asserts byte-for-byte equality — a single diverging hash would invalidate every proof against that tree. Others cover that the kept levels survive, that the active tree keeps every node, that the sweep respects its budget and resumes from the cursor, and that probes are charged so a clean level cannot be re-walked for free.

The E2E (ts-tests/node/sealed-tree-pruning.test.cjs) covers the config wiring and that the sweep stays idle while nothing has sealed. It cannot seal a tree itself: MaxLeavesPerTree is a compile-time constant, so sealing on-chain would need 2^20 shields. That equivalence is what the unit tests cover, using MaxLeavesPerTree = 8.

Out of scope

A proof endpoint in the indexer. It has the data (shielded_commitments.leaf_index, sealed_trees, merkle_roots) but does not compute proofs — no Poseidon, no tree construction. Separate feature, and since sealed_trees are already served with CACHE.IMMUTABLE it would be CDN-cacheable if ever built.