fix: remove evicted resolved path hops and preserve relay snapshots - #1966
Conversation
|
Reviewed and merging. CI had never run on this: it was sitting in Two things I checked rather than took on trust, both because you flagged them yourself. The remaining test failure you disclosed is a timing flake, not a break. The performance tradeoff is real and worth stating in the merge, not just the PR. From your own table, the small-batch case gets materially worse: The minute ticker means most real batches are small, so the common case is the slower one. 57 ms under the write lock once a minute is an acceptable price for not retaining expired transmissions and stale relay counts indefinitely, and the large-batch case is five times faster, which is where an operator would actually notice a stall. But if anyone later sees a periodic write-lock hiccup on a large instance, this is the change to look at first, and the numbers to compare against are right here. Full The bulk-relay change, taking bucket snapshots before releasing the read lock so eviction cannot mutate an in-flight reader, is the part I would have missed. Owning the snapshot rather than the map is the right fix and it is the kind of race that only shows up as wrong numbers, never as a crash. |
Eviction removes raw wire hops but leaves resolved full-key entries in
byPathHop, retaining expired transmissions and stale relay counts/scopes. Filter every hop bucket once per eviction batch using the existing evicted-ID set, remove duplicate references and empty buckets, and clear discarded pointer slots.Bulk relay aggregation now owns its bucket snapshots before releasing the read lock, so eviction and raw-path updates cannot mutate an in-flight reader. Three existing handler test fixtures also wait for index readiness or explicitly simulate not-ready state, preserving their original 200/503 assertions.
Fixes #1908.
Validation:
-race; coverage includes time/cap eviction, missing resolved-path prefetch, disabled membership indexing, duplicate references, retained backing arrays and surviving entries.TestGetChannelMessagesPerfLargeChannel: 2.198s against a 1.5s budget). The final correction was checked with focused race tests. The unchanged ingestor suite also cannot create one symlink without Windows privileges. These thresholds/assertions were preserved; full Linux Go/E2E results still require upstream CI approval.Performance tradeoff: cleanup is O(total indexed pointers) per nonempty eviction batch, under the existing write lock. The minute-based ticker pays for one sweep instead of repeated scans of shared raw buckets. No per-transmission string index or dependency is added. Synthetic benchmark medians (three single-iteration runs, shared Windows host):
Fixture: eight raw plus eight resolved hops per transmission, two observations, 2,048 relays; 480,000/1,600,000 hop entries. Timing includes acquiring the store lock and omits unrelated secondary indexes. Small batches now pay for the complete sweep; shared-host timing is noisy.
Owning the bulk reader's arrays also has a measured cost on cold/bulk recomputation, rather than cached hits. Snapshot medians from three samples of ten iterations:
These are total snapshot costs, comparing the unsafe header-only snapshot with owned pointer arrays. Cleanup guarantees here apply to
byPathHop; other indexes and existing periodic bulk-cache freshness are outside this change.Following #1922, this runtime fix is separate from the release-routing and frontend-runner PRs. Current Go and E2E job results should be assessed separately from workflow-approval or staging-runner state.