Skip to content

Lite: the route cache's background reload stays running while it still has a load to do - #6711

Merged
erikdarlingdata merged 3 commits into
devfrom
fix/route-cache-idle-gap
Oct 24, 2026
Merged

erikdarlingdata merged 3 commits into
devfrom
fix/route-cache-idle-gap

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Claude posting for Erik Darling.

No issue. PR #6705's CI (run 40082928834, Test leg 5/16) failed in PerformanceMonitorLite.Tests.NotificationRouteCacheBackgroundTests.ManyInvalidatesDuringOneReload_CostOneMoreLoad_NotOnePerCall: Assert.Equal expected 2, actual 1, at line 70.

Why

BackgroundReloadLoop in Lite/Services/NotificationRouteCache.cs cleared _backgroundRunning after every load, then checked whether an Invalidate during that load needed one more pass, then set the flag again with a compare-exchange. Between the clear and the set, BackgroundReloadRunning read false while the second load was still to come. The test's WaitForIdleAsync polls that flag, so a runner that paused the reload thread in that gap ended the wait before the second load, and the count read 1.

What changes

  • The flag stays set while the loop still has a load to do. The loop runs Current until MoreToLoad() reports nothing stale, and clears the flag only after that.
  • After the clear, the loop checks MoreToLoad() again and re-takes the flag with the same compare-exchange as before, so an Invalidate that landed in between still gets its load.
  • MoreToLoad() is a new private member next to the loop. It holds the three conditions the loop already used: a loader is set, the loaded version differs from the current version, and the 30 second retry window is not open.
  • One file: Lite/Services/NotificationRouteCache.cs, 18 insertions and 9 deletions. The method sits at about line 189 on origin/dev. No Darling copy of this loop exists: git grep finds BackgroundReloadLoop only in that file, so there is no twin to mirror.

Behaviour: no behaviour change for an alert. The same loads happen, each only while the routes are stale and outside the retry window. The only difference is that BackgroundReloadRunning no longer drops to false between two loads of one reload, so WaitForIdleAsync now waits for the whole reload.

Why the inner loop cannot spin: the loop runs on a thread-pool thread (Task.Run), where _isUiThread() is false, so Current takes its gate path. That path reloads under the same three conditions MoreToLoad() checks. On success it sets the loaded version to the one it read, and on failure it opens the retry window, so MoreToLoad() turns false unless another Invalidate arrived.

CHANGELOG

None (internal: a test-visible flag)

Test plan

Not run here. The brief says no build and no test run on this laptop, and CI decides.

  • CI: NotificationRouteCacheBackgroundTests passes, including ManyInvalidatesDuringOneReload_CostOneMoreLoad_NotOnePerCall.
  • CI: the Lite build reports 0 warnings.
  • Local, without a build: the edited file has 30 open and 30 close braces (29 and 29 before the edit, plus the one new inner do block), and MoreToLoad sits inside the class. git grep finds no other file that names the loop or reads its source text.

Second commit (3b6389c)

The flag is now cleared with Interlocked.Exchange instead of Volatile.Write. A release store can let the MoreToLoad() read after it run before the clear is seen. An Invalidate whose version increment and compare-exchange land in that window starts nothing, and the loop stops with that change unloaded until the next Current read. The original code had the same ordering. A full fence closes it.

Considered, not added

What the coordinator should double-check

  • Whether to fold in the Interlocked.Exchange change above. It is one token in the same method.
  • That Current reloads on the loop's thread under the same conditions as MoreToLoad(): read Current in Lite/Services/NotificationRouteCache.cs (about lines 211 to 263 on origin/dev).

@erikdarlingdata
erikdarlingdata marked this pull request as ready for review October 24, 2026 10:35
Comment on lines +204 to +209
Interlocked.Exchange(ref _backgroundRunning, 0);

/* Look again after clearing the flag: an Invalidate that landed between the last look and the clear saw the flag set
and started nothing, so this loop goes on (or that Invalidate won the race and started its own loop). */
}
while (Volatile.Read(ref _loader) is not null
&& Volatile.Read(ref _loadedVersion) != Volatile.Read(ref _version)
&& DateTime.UtcNow.Ticks >= Volatile.Read(ref _retryAfterTicks)
&& Interlocked.CompareExchange(ref _backgroundRunning, 1, 0) == 0);
while (MoreToLoad() && Interlocked.CompareExchange(ref _backgroundRunning, 1, 0) == 0);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: the flag-false-with-a-load-pending window is narrowed, not closed.

Take this interleaving:

  1. inner MoreToLoad() at :200 reads false (_loadedVersion == _version) and the inner loop exits;
  2. a writer's Invalidate() increments _version, then StartBackgroundReload CASes, sees the flag still 1 and starts nothing;
  3. :204 clears the flag;
  4. :209 reads MoreToLoad() true, re-takes the flag and loads.

Between (3) and (4) BackgroundReloadRunning is false while a load of this same reload is still to come — the shape the PR fixes, now two atomic instructions wide instead of one DuckDB read wide. Any single-flag design has it; closing it needs a separate "pending" count the observer reads with the flag. Not worth changing for the test this PR unflakes (ManyInvalidatesDuringOneReload_… issues every Invalidate while load 1 is in flight, so they are all absorbed by the inner loop at :196-200 and never reach this gap), but the body's "BackgroundReloadRunning no longer drops to false between two loads of one reload" reads as absolute and is not.

Comment on lines +196 to +200
do
{
_ = Current;
}
while (MoreToLoad());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking, on the "cannot spin" argument in the body. It is sound as written — on a pool thread OnApplicationDispatcher() (:105-106) is false, so Current takes the gate path and either advances _loadedVersion (:261) or arms _retryAfterTicks (:268), and MoreToLoad() goes false either way.

What the argument now rests on that it did not before is the injected predicate. NotificationRouteCacheCore takes isUiThread as a ctor parameter, and if one ever returns true on the loop's own thread, Current returns at :239-243 without loading and without arming the retry window, so MoreToLoad() stays true and this inner loop spins hot. The old shape spun there too, so this is not a regression — but it spun with the flag toggling, where this one spins with the flag pinned at 1, which turns a CPU burn into a hang in every while (BackgroundReloadRunning) poll (NotificationRouteCacheBackgroundTests.cs:35, NotificationRouteFirstStartTests.cs:97, the latter uncapped).

No current caller does it — both injected predicates compare against the test thread id (NotificationRouteCacheBackgroundTests.cs:136 and :169) and the loop runs under Task.Run (:186) — so nothing to fix here. It is just worth knowing that the loop's termination proof is now a property of the predicate, not of this method.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: LGTM

Claims

  • VERIFIED — The old loop cleared _backgroundRunning after every load and re-took it in the while condition, so BackgroundReloadRunning read false while a second load was still to come — the removed hunk (Volatile.Write(ref _backgroundRunning, 0); ahead of a condition whose last conjunct is the CAS), the flag's only reader at Lite/Services/NotificationRouteCache.cs:219, and the poll the test does on it at Lite.Tests/NotificationRouteCacheBackgroundTests.cs:35.
  • VERIFIED — The flag now stays set while the loop still has a load to do; the loop runs Current until MoreToLoad() reports nothing stale and clears only after that — Lite/Services/NotificationRouteCache.cs:196-204.
  • VERIFIED — After the clear the loop re-checks MoreToLoad() and re-takes the flag with the same compare-exchange as before — Lite/Services/NotificationRouteCache.cs:209.
  • VERIFIED — MoreToLoad() holds the three conditions the loop already used (loader set, loaded version differs, retry window not open) — Lite/Services/NotificationRouteCache.cs:213-216 is conjunct-for-conjunct the removed while condition minus its CAS.
  • VERIFIED — No Darling copy of this loop exists — BackgroundReloadLoop, _backgroundRunning and BackgroundReloadRunning grep to Lite/Services/NotificationRouteCache.cs alone across both SKUs and both test projects; Darling reads routes through NotificationRouteWriter / ViewerDataService.NotificationRoutes.cs with no lazily-loaded cache.
  • VERIFIED — No behaviour change for an alert; the same loads happen, each only while stale and outside the retry window — Current (:222-277) is the only entry point that calls the loader, and the predicate gating entry into it is unchanged.
  • VERIFIED — The inner loop cannot spin, because the loop runs on a thread-pool thread where _isUiThread() is false and Current takes its gate path, setting the loaded version on success or opening the retry window on failure — Task.Run at :186, OnApplicationDispatcher at :105-106, _loadedVersion = version at :261, _retryAfterTicks at :268. (One caveat on what the proof rests on: inline comment at :196.)
  • VERIFIED — Second commit: a release store can let the MoreToLoad() read after it run before the clear is seen, and the original code had the same ordering — Volatile.Write is a release store, which orders prior writes but not a following load, and the removed condition read _version / _loadedVersion before its CAS, so the old shape had the same StoreLoad hole; Interlocked.Exchange at :204 is a full fence and closes it.
  • VERIFIED — The edited file has 30 open and 30 close braces, and MoreToLoad sits inside the class — counted on the tree; :213 is inside NotificationRouteCacheCore, between BackgroundReloadLoop and BackgroundReloadRunning.
  • VERIFIED — The failing assertion is Assert.Equal expected 2 at line 70 — Lite.Tests/NotificationRouteCacheBackgroundTests.cs:70 is Assert.Equal(2, Volatile.Read(ref calls));, and walking the new shape through that test's ordering still yields exactly 2 loads with the flag held across both.
  • UNVERIFIED — PR #6705's CI run 40082928834, Test leg 5/16, failed that test with actual 1 — the run log is outside a read-only tree and outside this pass's command list.
  • UNVERIFIED — "18 insertions and 9 deletions" — gh pr view reports the PR at +21/−10 for both commits together; splitting that per commit needs a per-commit diff this pass does not run. The body discloses the second commit directly below the figure, so this reads as a stale first-commit number rather than a wrong one.
  • UNVERIFIED — "Not run here", and both CI boxes (the test passes, the Lite build reports 0 warnings) — needs a build and a test run.

Riskiest lines

  1. Lite/Services/NotificationRouteCache.cs:200 — the new inner loop is unbounded, and its termination now rests entirely on Current either advancing _loadedVersion or arming the retry window on every path it can return by — covered by AFailingBackgroundReload_KeepsTheRoutesInForce_AndDoesNotSpin (a loader that throws without arming the window leaves this loop spinning with the flag pinned, and WaitForIdleAsync trips its 3000-tick cap).
  2. Lite/Services/NotificationRouteCache.cs:204 — the fence; swap it back to Volatile.Write and the lost wakeup the second commit closes comes back, leaving an edit unloaded until the next Current read — UNCOVERED (a memory-model reorder cannot be made to fail deterministically, and on x86 TSO the StoreLoad this prevents is the one reorder the hardware does not perform, so a test would be green on the defect on every CI runner this repo uses).
  3. Lite/Services/NotificationRouteCache.cs:209 — the hand-off; without the CAS, an Invalidate landing after the clear could start its own loop while this one also continues, and two loops would run against one flag — UNCOVERED (ManyInvalidatesDuringOneReload_CostOneMoreLoad_NotOnePerCall pins the load count only for its own ordering, in which every Invalidate lands during load 1 and is absorbed by the inner loop; the duplicate-loop race is not reachable from it).

Traps near this diff

  • "Lite and Darling ship in parity (CONTRIBUTING's two-store rule): a fix to one SKU's reader, tool, alert or description without the counterpart's is a finding, not a nit" — does not apply — the loop has no twin: BackgroundReloadLoop and _backgroundRunning exist only at Lite/Services/NotificationRouteCache.cs:186 and :96, and the cache exists because Lite's DuckDB store is single-process; Darling's viewer reads the shared PostgreSQL store directly.
  • "Ruling (#5833): tests are load-bearing. … never answer a trap with a test that reads source text or asserts how code is spelled … A trap is answered with a behaviour test that fails on the defect and passes on the fix, or with one sentence in the PR body saying why none is needed" — does not apply — no test is added, the body says under "Considered, not added" that the existing behaviour test is the coverage, and the brace count stayed a local check rather than becoming an assertion.
  • "Our wrong test is a bug we introduced, never a flake: a test that dev's own push broke … gets an issue and a fix, and a line in .github/ci-quarantine.txt only while that fix lands" — does not apply — the PR takes the fix path, not the quarantine path: the product code changes at Lite/Services/NotificationRouteCache.cs:196-216, no line is added to .github/ci-quarantine.txt, and no retry or skip is added to the test, so the trap's issue requirement (which attaches to a quarantine line) is not engaged.
  • "A test that reads the clock twice measures the CI scheduler: the delta-seeder tests failed at 123 and 243 on unrelated PRs (#3676). One anchor owns both ends of an interval" — does not apply — the clock read this diff adds is in product code, not a test, and it is one read against a stored deadline, not two reads bounding an interval: _retryAfterTicks is anchored once at Lite/Services/NotificationRouteCache.cs:268 and compared at :216.

Divergence (blind read vs PR body)

Three, all narrow; none changes the verdict.

  1. "BackgroundReloadRunning no longer drops to false between two loads of one reload" reads as absolute and is not. An Invalidate that lands between the inner MoreToLoad() at :200 and the clear at :204 sees the flag set and starts nothing; the flag is then false from :204 until the CAS at :209 re-takes it, with a load of that same reload still to come. The window goes from one DuckDB read wide to two atomic instructions wide — a real narrowing, and closing it entirely needs a pending count rather than a flag — but the claim as written overstates it. Inline comment at :204-209.
  2. The "why the inner loop cannot spin" argument is correct for every caller that exists, but it omits what it now depends on. NotificationRouteCacheCore takes isUiThread as a ctor parameter, and an injected predicate returning true on the loop's own thread makes Current return at :239-243 without loading and without arming the retry window, so MoreToLoad() never goes false. The old shape spun there too, so it is not a regression — but it spun with the flag toggling, where this one spins with the flag pinned at 1, turning a CPU burn into a hang in NotificationRouteCacheBackgroundTests.cs:35 and the uncapped poll at NotificationRouteFirstStartTests.cs:97. Inline comment at :196-200.
  3. The "+18/−9" figure describes the first commit while the PR stands at +21/−10; the body discloses the second commit immediately below it, so the two accounts differ only in which snapshot the number belongs to.

Reviewer checklist

  • Rerun NotificationRouteCacheBackgroundTests under load (several parallel runs, or a runner pinned to few cores) rather than once — the defect this fixes only appears when the reload thread is descheduled inside the old gap, so a single green run is not evidence the race is gone.
  • Confirm the Lite build is 0 warnings; nothing in a read-only tree compiles the nested do/while or the new MoreToLoad member.
  • If you keep the injectable isUiThread seam, decide whether the loop should guard its own termination (an iteration cap, or re-reading the predicate once outside the loop) rather than relying on every future caller passing a predicate that is false on a pool thread.

@claude

claude Bot commented Oct 24, 2026

Copy link
Copy Markdown

Follow-up pass on 3b6389cc9..688dccca9.

688dccca9 is Merge branch 'dev' into fix/route-cache-idle-gap, parents 3b6389cc9 (this PR's own prior tip, already reviewed) and 2bec9c4bf (dev's tip). Its three "new" commits in that range — 471707e69 (test-helper Elapse timing fix), d488fbd91 (test-collection rename), 2bec9c4bf (Darling blocking baseline/self-wait filtering, #6690/#6705) — are all dev history the merge brought in, not commits authored for this PR. Confirmed two ways: gh pr diff 6711 --name-only shows only Lite/Services/NotificationRouteCache.cs, and gh pr view 6711 --json commits lists exactly three commits for the PR (d34d755a, 3b6389cc9, 688dccca9) — no commit of this PR's own touches any of the files the merge brought in.

None of the merged-in content interacts with this PR's fix: it touches Darling's blocking-baseline SQL, TestTimeHelpers.cs, and NotificationRouteFirstStartTests.cs's xunit [Collection] attribute — none of it references NotificationRouteCache, BackgroundReloadLoop, _backgroundRunning, or NotificationRouteCacheBackgroundTests, which has no [Collection] attribute to begin with. Per this review's instructions, content a base-branch merge brings in without interacting with the PR's own change is left alone.

Nothing new to review; the prior full-pass verdict on 3b6389cc9 stands.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: LGTM

Claims

  • VERIFIED — PR body describes only the route-cache background-reload fix (NotificationRouteCache.cs); it makes no claims about the follow-up range's content — gh pr diff 6711 --name-only confirms the PR's actual diff is that one file.

Riskiest lines

  1. n/a — the only commit in the follow-up range (688dccca9) is a merge of dev into this branch; it introduces no lines of its own and the base-branch content it brings in does not interact with this PR's change (see posted comment) — UNCOVERED (not applicable: nothing of this PR's own authorship to cover).
  2. n/a
  3. n/a

Traps near this diff

  • "Lite and Darling ship in parity (CONTRIBUTING's two-store rule)" — does not apply — no commit authored for this PR touches a reader, tool, alert or description in either SKU; the parity-relevant work in the merged-in range (2bec9c4bf's self-wait filtering) is dev's own already-landed change, not this PR's.

Divergence (blind read vs PR body)

None — the PR body's account covers only the route-cache fix, and the follow-up range's only new commit is a base-branch merge that brings in unrelated, already-landed dev work with no interaction with that fix.

Reviewer checklist

  • None beyond the prior full-pass review's checklist (rerun NotificationRouteCacheBackgroundTests under load, confirm 0-warning build) — nothing in this follow-up range changes what needs hand-verification.

@erikdarlingdata
erikdarlingdata merged commit 68565b6 into dev Oct 24, 2026
11 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/route-cache-idle-gap branch October 24, 2026 11:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant