Skip to content

fix: remove evicted resolved path hops and preserve relay snapshots - #1966

Merged
efiten merged 7 commits into
Kpa-clawbot:masterfrom
n30nex:codex/fix-resolved-hop-eviction
Sep 6, 2026
Merged

fix: remove evicted resolved path hops and preserve relay snapshots#1966
efiten merged 7 commits into
Kpa-clawbot:masterfrom
n30nex:codex/fix-resolved-hop-eviction

Conversation

@n30nex

@n30nex n30nex commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Eviction removes raw wire hops but leaves resolved full-key entries in byPathHop, retaining expired transmissions and stale relay counts/scopes. Filter every hop bucket once per eviction batch using the existing evicted-ID set, remove duplicate references and empty buckets, and clear discarded pointer slots.

Bulk relay aggregation now owns its bucket snapshots before releasing the read lock, so eviction and raw-path updates cannot mutate an in-flight reader. Three existing handler test fixtures also wait for index readiness or explicitly simulate not-ready state, preserving their original 200/503 assertions.

Fixes #1908.

Validation:

  • Regression commits fail before their corresponding fixes: resolved keys/counts/scopes remain after eviction, and saved relay snapshots change during eviction.
  • Targeted eviction, relay, scope, cache and concurrent-reader checks pass under -race; coverage includes time/cap eviction, missing resolved-path prefetch, disabled membership indexing, duplicate references, retained backing arrays and surviving entries.
  • Local browser smoke: nodes, node details/path attribution and analytics render using the fixture-backed Go server.
  • The last full Windows server race run, before the final snapshot-copy correction, had one remaining DB-only timing failure (TestGetChannelMessagesPerfLargeChannel: 2.198s against a 1.5s budget). The final correction was checked with focused race tests. The unchanged ingestor suite also cannot create one symlink without Windows privileges. These thresholds/assertions were preserved; full Linux Go/E2E results still require upstream CI approval.

Performance tradeoff: cleanup is O(total indexed pointers) per nonempty eviction batch, under the existing write lock. The minute-based ticker pays for one sweep instead of repeated scans of shared raw buckets. No per-transmission string index or dependency is added. Synthetic benchmark medians (three single-iteration runs, shared Windows host):

Transmissions Evicted Before After
30,000 1 1.07 ms 14.19 ms
30,000 3,000 56.27 ms 61.68 ms
30,000 7,500 83.58 ms 65.54 ms
100,000 1 0.30 ms 56.95 ms
100,000 10,000 949.93 ms 190.71 ms
100,000 25,000 1,834.48 ms 320.38 ms

Fixture: eight raw plus eight resolved hops per transmission, two observations, 2,048 relays; 480,000/1,600,000 hop entries. Timing includes acquiring the store lock and omits unrelated secondary indexes. Small batches now pay for the complete sweep; shared-host timing is noisy.

Owning the bulk reader's arrays also has a measured cost on cold/bulk recomputation, rather than cached hits. Snapshot medians from three samples of ten iterations:

Transmissions / relay nodes Before time / bytes per operation After time / bytes per operation
30,000 / 50 0.0068 ms / 5,416 B 23.65 ms / 4,101,435 B
30,000 / 2,000 0.1374 ms / 196,768 B 26.77 ms / 4,274,336 B
100,000 / 2,000 0.1376 ms / 196,768 B 27.89 ms / 13,959,337 B

These are total snapshot costs, comparing the unsafe header-only snapshot with owned pointer arrays. Cleanup guarantees here apply to byPathHop; other indexes and existing periodic bulk-cache freshness are outside this change.

Following #1922, this runtime fix is separate from the release-routing and frontend-runner PRs. Current Go and E2E job results should be assessed separately from workflow-approval or staging-runner state.

@efiten

efiten commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

Reviewed and merging. CI had never run on this: it was sitting in action_required, which I have now approved for all seven of your PRs. It came back fully green.

Two things I checked rather than took on trust, both because you flagged them yourself.

The remaining test failure you disclosed is a timing flake, not a break. TestGetChannelMessagesPerfLargeChannel passes three consecutive runs on this branch and also on a branch off current master, so it is not something this change introduced. Thank you for naming it instead of leaving it out; that is what made it cheap to check.

The performance tradeoff is real and worth stating in the merge, not just the PR. From your own table, the small-batch case gets materially worse:

100,000 tx,      1 evicted:   0.30 ms ->  56.95 ms
100,000 tx, 10,000 evicted: 949.93 ms -> 190.71 ms

The minute ticker means most real batches are small, so the common case is the slower one. 57 ms under the write lock once a minute is an acceptable price for not retaining expired transmissions and stale relay counts indefinitely, and the large-batch case is five times faster, which is where an operator would actually notice a stall. But if anyone later sees a periodic write-lock hiccup on a large instance, this is the change to look at first, and the numbers to compare against are right here.

Full cmd/server suite green locally as well.

The bulk-relay change, taking bucket snapshots before releasing the read lock so eviction cannot mutate an in-flight reader, is the part I would have missed. Owning the snapshot rather than the map is the right fix and it is the kind of race that only shows up as wrong numbers, never as a crash.

@efiten
efiten merged commit cf67a5e into Kpa-clawbot:master Sep 6, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: eviction never removes resolved full-pubkey entries from byPathHop — evicted transmissions stay referenced

2 participants