dash-spv: the initial scan's new-script cascade never re-tests blocks that only match on the prevouts of coins discovered mid-scan, so their spends are missed and the coins stay credited

[llbartekll]

Summary

On a CoinJoin-heavy mainnet wallet, a fresh, uninterrupted import ends with 11 CoinJoin outputs credited (≈ 0.11 DASH) that were spent on-chain 3–300 blocks after they were funded. The 4 blocks that spend them are never matched, requested or applied during the scan — there is no BlockProcessed event for those heights — although each of them contains the spent coin's outpoint as a transaction input. A rescan_filters(from = birth) on the finished wallet re-tests every filter against the wallet's complete script set, matches those blocks, applies them, and the coins are removed (12 398 blocks re-applied in ~50 s, release build). So the filters are fine and the engine's spend logic is fine; what is missing is a pass that tests the committed range against the scripts that exist at the end of the discovery cascade.

The same root cause makes an interrupted initial scan (process killed mid-sweep) resume into the same wrong state, see "Interrupted-sync variant" below.

Reproduced on HashEngineering/rust-dashcore@e5cbb13 (= platform pin 93260bf + #979) through platform-wallet FFI on the iOS Simulator, release profile. Not a #979 regression: the pinned engine ends the same wallet at the same 0.11 after a restart.

What the logs show (fresh import, single session)

• Forward pass over 2.33M filters from height 200000, ~2.2k matched blocks; 3 645 new scripts derived during block application (gap widening); per-batch Rescan filters (…) re-tests for the still-active batches; then one coalesced Rescan committed filters (200000-2534999) … (sweep #1) → found 10265 additional blocks; 12 480 blocks applied in total; no sweep Rust bitcoin master #2; filters_pending_sweep.dat empty at the end.
• 11 outputs (one 0.1 and ten 0.001 denominations) funded in 6 blocks within ~540 blocks of height 2.403M are credited. Their funding blocks were delivered by the per-batch rescans (applied ~0.6–1.0 s after the forward-pass blocks of the same batch).
• Their 4 spending blocks (mixing rounds, 5–7 inputs / 5–7 outputs each) are absent from the log entirely: never found by a batch, a rescan or the sweep.
• A reference run of the same build a few hours earlier (different timing) did match those 4 blocks — from a batch rescan, ~1 s after the funding blocks — and ended with utxo_count=0. Which blocks get re-tested against which scripts therefore depends on the order in which block applications, script derivations and rescans interleave.
• Where a spending block was applied before its funding block (one block in this run), the bug: out-of-order block processing causes SPV wallet to miss UTXO spends #649 observed_spent_outpoints guard worked and the funding outputs were correctly not credited. The 11 leftovers are exactly the outputs whose spending blocks were never applied at all.

Why

A block whose only wallet-relevant element is the prevout script of a coin that did not exist in the wallet when that batch was tested can only be matched by a later re-test that includes that coin's script. The cascade re-tests (a) the still-active batches with the scripts collected from their own block applications, and (b) the committed prefix once, with the scripts accumulated in backward_scripts at the moment the sweep starts. Scripts that enter the wallet after the batch covering the spending block has committed and after the sweep has taken its snapshot are never tested against that height again — and once the coin is credited, nothing else in the engine can remove it (no spender record for #979 to attribute to, observed_spent pruned by the committed height, spent_outpoints only fed by matched transactions).

Suggested fix

After the cascade reaches its fixpoint (last batch committed, backward_scripts and pending_sweep empty), run one final sweep of the committed range with the wallet's complete script set (equivalent to the manual rescan_filters(from birth) that repairs the wallet today), applying matched blocks in height order. Cost on this wallet: ~30 s of filter matching + ~40 s of block re-application from local storage in a release build. A cheaper variant: sweep with the scripts of every output credited during the cascade (those are the prevouts that can still be missing), instead of the full set.

Independently, prune_finalized_observed_spends uses metadata.synced_height (the committed filter height) as the boundary; during a historical scan that height runs ahead of block deliveries from rescans and sweeps, so a spend seen in the forward pass can be forgotten before a rescan delivers the funding block. Bounding the prune by the height below which no further block deliveries are possible for this wallet (the cascade fixpoint) would close that window.

Interrupted-sync variant

If the process is killed while sweep #1 is running, the next launch resumes from filters_pending_sweep.dat and re-runs the sweep, but the outputs credited before the kill were restored into the wallet, so their spending blocks are again matched only if their scripts happen to be in the durable pending set. In our run they were not: 7 spending blocks (all present in local block storage from an earlier run) were never re-applied after the resume, and the wrong balance became durable. The final full-set sweep above covers this case too, provided it also runs after a resumed cascade.

Workaround

rescan_filters(from = wallet birth height): reset_for_rescan clears the processed-block tracker; every matched block is re-applied, the spends now match the credited inputs and the coins are removed. Verified twice today (restart afterwards keeps the correct balance).

Full dash-spv / key-wallet / platform-wallet logs of the reference run, the fresh run and the interrupted+resumed run are available privately.

[llbartekll]

Root cause found and fixed in #1007.

It is not the sweep's coverage as such: with temporary counters in the BlockProcessed handler (release build, same wallet), 3 645 scripts were derived during the fresh scan but only 3 348 reached the committed-range sweep. The missing 297 were derived by 14 blocks that were applied 2–4 times (forward pass, then batch rescans / in-flight re-requests). The handler collected new scripts only when the tracker still held the block's in-flight entry, which the first delivery consumes, so everything derived by the later applications was dropped — not noted in the durable pending-sweep set either. The 11 spent coins' addresses were among those 297; their spending blocks match only on those prevout scripts, hence never found.

With #1007 the same fresh import ends with the sweep running on all 3 645 scripts, 11 169 blocks found (the count of the one earlier run that had converged by luck), utxo_count=0, and a restart restores nothing.

The two secondary points (4× re-application of in-flight blocks; the observed-spent prune bound) stay open here as noted in the PR.

[llbartekll]

The fix is now a standalone PR on dev: #1008 (no dependency on #979; #1007 closed as superseded). Verified again by hand through the app's own delete + import flow: 3 645 scripts swept, 11 169 blocks found, 0 coins left credited.

[ZocoLini]

Closed by #989

[llbartekll]

Fixed by #989 (merged into dev at e4208c9); #1008 closed as a duplicate. Verified on the affected wallet: pinned engine + #989 rebuilds from seed to 0 phantom coins.