Repository navigation
Lite: the route cache's background reload stays running while it still has a load to do - #6711
Conversation
…l has a load to do
| Interlocked.Exchange(ref _backgroundRunning, 0); | ||
|
|
||
| /* Look again after clearing the flag: an Invalidate that landed between the last look and the clear saw the flag set | ||
| and started nothing, so this loop goes on (or that Invalidate won the race and started its own loop). */ | ||
| } | ||
| while (Volatile.Read(ref _loader) is not null | ||
| && Volatile.Read(ref _loadedVersion) != Volatile.Read(ref _version) | ||
| && DateTime.UtcNow.Ticks >= Volatile.Read(ref _retryAfterTicks) | ||
| && Interlocked.CompareExchange(ref _backgroundRunning, 1, 0) == 0); | ||
| while (MoreToLoad() && Interlocked.CompareExchange(ref _backgroundRunning, 1, 0) == 0); |
There was a problem hiding this comment.
Non-blocking: the flag-false-with-a-load-pending window is narrowed, not closed.
Take this interleaving:
- inner
MoreToLoad()at :200 reads false (_loadedVersion == _version) and the inner loop exits; - a writer's
Invalidate()increments_version, thenStartBackgroundReloadCASes, sees the flag still1and starts nothing; - :204 clears the flag;
- :209 reads
MoreToLoad()true, re-takes the flag and loads.
Between (3) and (4) BackgroundReloadRunning is false while a load of this same reload is still to come — the shape the PR fixes, now two atomic instructions wide instead of one DuckDB read wide. Any single-flag design has it; closing it needs a separate "pending" count the observer reads with the flag. Not worth changing for the test this PR unflakes (ManyInvalidatesDuringOneReload_… issues every Invalidate while load 1 is in flight, so they are all absorbed by the inner loop at :196-200 and never reach this gap), but the body's "BackgroundReloadRunning no longer drops to false between two loads of one reload" reads as absolute and is not.
| do | ||
| { | ||
| _ = Current; | ||
| } | ||
| while (MoreToLoad()); |
There was a problem hiding this comment.
Non-blocking, on the "cannot spin" argument in the body. It is sound as written — on a pool thread OnApplicationDispatcher() (:105-106) is false, so Current takes the gate path and either advances _loadedVersion (:261) or arms _retryAfterTicks (:268), and MoreToLoad() goes false either way.
What the argument now rests on that it did not before is the injected predicate. NotificationRouteCacheCore takes isUiThread as a ctor parameter, and if one ever returns true on the loop's own thread, Current returns at :239-243 without loading and without arming the retry window, so MoreToLoad() stays true and this inner loop spins hot. The old shape spun there too, so this is not a regression — but it spun with the flag toggling, where this one spins with the flag pinned at 1, which turns a CPU burn into a hang in every while (BackgroundReloadRunning) poll (NotificationRouteCacheBackgroundTests.cs:35, NotificationRouteFirstStartTests.cs:97, the latter uncapped).
No current caller does it — both injected predicates compare against the test thread id (NotificationRouteCacheBackgroundTests.cs:136 and :169) and the loop runs under Task.Run (:186) — so nothing to fix here. It is just worth knowing that the loop's termination proof is now a property of the predicate, not of this method.
There was a problem hiding this comment.
Verdict: LGTM
Claims
- VERIFIED — The old loop cleared
_backgroundRunningafter every load and re-took it in thewhilecondition, soBackgroundReloadRunningread false while a second load was still to come — the removed hunk (Volatile.Write(ref _backgroundRunning, 0);ahead of a condition whose last conjunct is the CAS), the flag's only reader atLite/Services/NotificationRouteCache.cs:219, and the poll the test does on it atLite.Tests/NotificationRouteCacheBackgroundTests.cs:35. - VERIFIED — The flag now stays set while the loop still has a load to do; the loop runs
CurrentuntilMoreToLoad()reports nothing stale and clears only after that —Lite/Services/NotificationRouteCache.cs:196-204. - VERIFIED — After the clear the loop re-checks
MoreToLoad()and re-takes the flag with the same compare-exchange as before —Lite/Services/NotificationRouteCache.cs:209. - VERIFIED —
MoreToLoad()holds the three conditions the loop already used (loader set, loaded version differs, retry window not open) —Lite/Services/NotificationRouteCache.cs:213-216is conjunct-for-conjunct the removedwhilecondition minus its CAS. - VERIFIED — No Darling copy of this loop exists —
BackgroundReloadLoop,_backgroundRunningandBackgroundReloadRunninggrep toLite/Services/NotificationRouteCache.csalone across both SKUs and both test projects; Darling reads routes throughNotificationRouteWriter/ViewerDataService.NotificationRoutes.cswith no lazily-loaded cache. - VERIFIED — No behaviour change for an alert; the same loads happen, each only while stale and outside the retry window —
Current(:222-277) is the only entry point that calls the loader, and the predicate gating entry into it is unchanged. - VERIFIED — The inner loop cannot spin, because the loop runs on a thread-pool thread where
_isUiThread()is false andCurrenttakes its gate path, setting the loaded version on success or opening the retry window on failure —Task.Runat:186,OnApplicationDispatcherat:105-106,_loadedVersion = versionat:261,_retryAfterTicksat:268. (One caveat on what the proof rests on: inline comment at:196.) - VERIFIED — Second commit: a release store can let the
MoreToLoad()read after it run before the clear is seen, and the original code had the same ordering —Volatile.Writeis a release store, which orders prior writes but not a following load, and the removed condition read_version/_loadedVersionbefore its CAS, so the old shape had the same StoreLoad hole;Interlocked.Exchangeat:204is a full fence and closes it. - VERIFIED — The edited file has 30 open and 30 close braces, and
MoreToLoadsits inside the class — counted on the tree;:213is insideNotificationRouteCacheCore, betweenBackgroundReloadLoopandBackgroundReloadRunning. - VERIFIED — The failing assertion is
Assert.Equalexpected 2 at line 70 —Lite.Tests/NotificationRouteCacheBackgroundTests.cs:70isAssert.Equal(2, Volatile.Read(ref calls));, and walking the new shape through that test's ordering still yields exactly 2 loads with the flag held across both. - UNVERIFIED — PR #6705's CI run 40082928834, Test leg 5/16, failed that test with actual 1 — the run log is outside a read-only tree and outside this pass's command list.
- UNVERIFIED — "18 insertions and 9 deletions" —
gh pr viewreports the PR at +21/−10 for both commits together; splitting that per commit needs a per-commit diff this pass does not run. The body discloses the second commit directly below the figure, so this reads as a stale first-commit number rather than a wrong one. - UNVERIFIED — "Not run here", and both CI boxes (the test passes, the Lite build reports 0 warnings) — needs a build and a test run.
Riskiest lines
Lite/Services/NotificationRouteCache.cs:200— the new inner loop is unbounded, and its termination now rests entirely onCurrenteither advancing_loadedVersionor arming the retry window on every path it can return by — covered byAFailingBackgroundReload_KeepsTheRoutesInForce_AndDoesNotSpin(a loader that throws without arming the window leaves this loop spinning with the flag pinned, andWaitForIdleAsynctrips its 3000-tick cap).Lite/Services/NotificationRouteCache.cs:204— the fence; swap it back toVolatile.Writeand the lost wakeup the second commit closes comes back, leaving an edit unloaded until the nextCurrentread — UNCOVERED (a memory-model reorder cannot be made to fail deterministically, and on x86 TSO the StoreLoad this prevents is the one reorder the hardware does not perform, so a test would be green on the defect on every CI runner this repo uses).Lite/Services/NotificationRouteCache.cs:209— the hand-off; without the CAS, anInvalidatelanding after the clear could start its own loop while this one also continues, and two loops would run against one flag — UNCOVERED (ManyInvalidatesDuringOneReload_CostOneMoreLoad_NotOnePerCallpins the load count only for its own ordering, in which everyInvalidatelands during load 1 and is absorbed by the inner loop; the duplicate-loop race is not reachable from it).
Traps near this diff
- "Lite and Darling ship in parity (CONTRIBUTING's two-store rule): a fix to one SKU's reader, tool, alert or description without the counterpart's is a finding, not a nit" — does not apply — the loop has no twin:
BackgroundReloadLoopand_backgroundRunningexist only atLite/Services/NotificationRouteCache.cs:186and:96, and the cache exists because Lite's DuckDB store is single-process; Darling's viewer reads the shared PostgreSQL store directly. - "Ruling (#5833): tests are load-bearing. … never answer a trap with a test that reads source text or asserts how code is spelled … A trap is answered with a behaviour test that fails on the defect and passes on the fix, or with one sentence in the PR body saying why none is needed" — does not apply — no test is added, the body says under "Considered, not added" that the existing behaviour test is the coverage, and the brace count stayed a local check rather than becoming an assertion.
- "Our wrong test is a bug we introduced, never a flake: a test that dev's own push broke … gets an issue and a fix, and a line in
.github/ci-quarantine.txtonly while that fix lands" — does not apply — the PR takes the fix path, not the quarantine path: the product code changes atLite/Services/NotificationRouteCache.cs:196-216, no line is added to.github/ci-quarantine.txt, and no retry or skip is added to the test, so the trap's issue requirement (which attaches to a quarantine line) is not engaged. - "A test that reads the clock twice measures the CI scheduler: the delta-seeder tests failed at 123 and 243 on unrelated PRs (#3676). One anchor owns both ends of an interval" — does not apply — the clock read this diff adds is in product code, not a test, and it is one read against a stored deadline, not two reads bounding an interval:
_retryAfterTicksis anchored once atLite/Services/NotificationRouteCache.cs:268and compared at:216.
Divergence (blind read vs PR body)
Three, all narrow; none changes the verdict.
- "
BackgroundReloadRunningno longer drops to false between two loads of one reload" reads as absolute and is not. AnInvalidatethat lands between the innerMoreToLoad()at:200and the clear at:204sees the flag set and starts nothing; the flag is then false from:204until the CAS at:209re-takes it, with a load of that same reload still to come. The window goes from one DuckDB read wide to two atomic instructions wide — a real narrowing, and closing it entirely needs a pending count rather than a flag — but the claim as written overstates it. Inline comment at:204-209. - The "why the inner loop cannot spin" argument is correct for every caller that exists, but it omits what it now depends on.
NotificationRouteCacheCoretakesisUiThreadas a ctor parameter, and an injected predicate returning true on the loop's own thread makesCurrentreturn at:239-243without loading and without arming the retry window, soMoreToLoad()never goes false. The old shape spun there too, so it is not a regression — but it spun with the flag toggling, where this one spins with the flag pinned at 1, turning a CPU burn into a hang inNotificationRouteCacheBackgroundTests.cs:35and the uncapped poll atNotificationRouteFirstStartTests.cs:97. Inline comment at:196-200. - The "+18/−9" figure describes the first commit while the PR stands at +21/−10; the body discloses the second commit immediately below it, so the two accounts differ only in which snapshot the number belongs to.
Reviewer checklist
- Rerun
NotificationRouteCacheBackgroundTestsunder load (several parallel runs, or a runner pinned to few cores) rather than once — the defect this fixes only appears when the reload thread is descheduled inside the old gap, so a single green run is not evidence the race is gone. - Confirm the Lite build is 0 warnings; nothing in a read-only tree compiles the nested
do/whileor the newMoreToLoadmember. - If you keep the injectable
isUiThreadseam, decide whether the loop should guard its own termination (an iteration cap, or re-reading the predicate once outside the loop) rather than relying on every future caller passing a predicate that is false on a pool thread.
|
Follow-up pass on
None of the merged-in content interacts with this PR's fix: it touches Darling's blocking-baseline SQL, Nothing new to review; the prior full-pass verdict on |
There was a problem hiding this comment.
Verdict: LGTM
Claims
- VERIFIED — PR body describes only the route-cache background-reload fix (
NotificationRouteCache.cs); it makes no claims about the follow-up range's content —gh pr diff 6711 --name-onlyconfirms the PR's actual diff is that one file.
Riskiest lines
- n/a — the only commit in the follow-up range (
688dccca9) is a merge ofdevinto this branch; it introduces no lines of its own and the base-branch content it brings in does not interact with this PR's change (see posted comment) — UNCOVERED (not applicable: nothing of this PR's own authorship to cover). - n/a
- n/a
Traps near this diff
- "Lite and Darling ship in parity (CONTRIBUTING's two-store rule)" — does not apply — no commit authored for this PR touches a reader, tool, alert or description in either SKU; the parity-relevant work in the merged-in range (
2bec9c4bf's self-wait filtering) is dev's own already-landed change, not this PR's.
Divergence (blind read vs PR body)
None — the PR body's account covers only the route-cache fix, and the follow-up range's only new commit is a base-branch merge that brings in unrelated, already-landed dev work with no interaction with that fix.
Reviewer checklist
- None beyond the prior full-pass review's checklist (rerun
NotificationRouteCacheBackgroundTestsunder load, confirm 0-warning build) — nothing in this follow-up range changes what needs hand-verification.
Claude posting for Erik Darling.
No issue. PR #6705's CI (run 40082928834, Test leg 5/16) failed in
PerformanceMonitorLite.Tests.NotificationRouteCacheBackgroundTests.ManyInvalidatesDuringOneReload_CostOneMoreLoad_NotOnePerCall:Assert.Equalexpected 2, actual 1, at line 70.Why
BackgroundReloadLoopinLite/Services/NotificationRouteCache.cscleared_backgroundRunningafter every load, then checked whether an Invalidate during that load needed one more pass, then set the flag again with a compare-exchange. Between the clear and the set,BackgroundReloadRunningread false while the second load was still to come. The test'sWaitForIdleAsyncpolls that flag, so a runner that paused the reload thread in that gap ended the wait before the second load, and the count read 1.What changes
CurrentuntilMoreToLoad()reports nothing stale, and clears the flag only after that.MoreToLoad()again and re-takes the flag with the same compare-exchange as before, so an Invalidate that landed in between still gets its load.MoreToLoad()is a new private member next to the loop. It holds the three conditions the loop already used: a loader is set, the loaded version differs from the current version, and the 30 second retry window is not open.Lite/Services/NotificationRouteCache.cs, 18 insertions and 9 deletions. The method sits at about line 189 on origin/dev. No Darling copy of this loop exists:git grepfindsBackgroundReloadLooponly in that file, so there is no twin to mirror.Behaviour: no behaviour change for an alert. The same loads happen, each only while the routes are stale and outside the retry window. The only difference is that
BackgroundReloadRunningno longer drops to false between two loads of one reload, soWaitForIdleAsyncnow waits for the whole reload.Why the inner loop cannot spin: the loop runs on a thread-pool thread (
Task.Run), where_isUiThread()is false, soCurrenttakes its gate path. That path reloads under the same three conditionsMoreToLoad()checks. On success it sets the loaded version to the one it read, and on failure it opens the retry window, soMoreToLoad()turns false unless another Invalidate arrived.CHANGELOG
None (internal: a test-visible flag)
Test plan
Not run here. The brief says no build and no test run on this laptop, and CI decides.
NotificationRouteCacheBackgroundTestspasses, includingManyInvalidatesDuringOneReload_CostOneMoreLoad_NotOnePerCall.doblock), andMoreToLoadsits inside the class.git grepfinds no other file that names the loop or reads its source text.Second commit (3b6389c)
The flag is now cleared with
Interlocked.Exchangeinstead ofVolatile.Write. A release store can let theMoreToLoad()read after it run before the clear is seen. An Invalidate whose version increment and compare-exchange land in that window starts nothing, and the loop stops with that change unloaded until the nextCurrentread. The original code had the same ordering. A full fence closes it.Considered, not added
What the coordinator should double-check
Interlocked.Exchangechange above. It is one token in the same method.Currentreloads on the loop's thread under the same conditions asMoreToLoad(): readCurrentinLite/Services/NotificationRouteCache.cs(about lines 211 to 263 on origin/dev).