Skip to content

Lite: cheaper High Impact read, FinOps loads wait for the first show, one skip line per cycle (#9233) - #9235

Merged
erikdarlingdata merged 5 commits into
devfrom
fix/9233-high-impact-read-lock
Nov 21, 2026
Merged

erikdarlingdata merged 5 commits into
devfrom
fix/9233-high-impact-read-lock

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

Claude posting for Erik Darling.

Fixes #9233

Release note: Lite no longer loads the FinOps panels at window start for a tab nobody has opened, the High Impact read does less work, and a skipped database-state maintenance cycle logs one line naming every server it skipped.

Cause

Three things met at window start.

  1. FinOpsTab.Initialize selected the first server (which fired the selection handler and a whole per-server load) and then called RefreshData() (a second whole load). Each is fifteen store reads, High Impact among them, all at once, on a tab that is not visible and against a store the first collection has not written to. The tab's first show already re-runs the whole load whatever the age of any earlier one (FinOpsShowReloadPolicy), so the start-up loads were pure cost. A server-list change that moved the selection while the tab was hidden ran the same load again.
  2. The High Impact statement has two operators that sort the window. The grant lookup is a per-hash LEFT JOIN LATERAL ... ORDER BY ... LIMIT 1 over the materialized window, and the sample text is a ROW_NUMBER sort that carries the text of every row of the kept hashes.
  3. The maintenance skip logged once per server, so one held lock read as six identical lines.

Fix

  • Read (LocalDataService.FinOps.Workload.cs): the grant is one grouped arg_max over the window, keyed on a struct. A struct key ranks a NULL field above every value in DuckDB (Lite MCP: get_query_store_top with no database filter and get_analysis_facts take over 5 s on a small install, and the facts reply passes the budget with no note #9195), so each key that sorted NULLS LAST is mapped with COALESCE to a value below every real one. The sample text is one grouped arg_min over a negated key, NULLs mapped to the largest value, with the text as the last member so a full tie still goes to the smallest text. The rows returned are the same. The one deliberate difference: when rows tie on all three grant keys but hold different grants, the old pick was whichever the sort met first and the new one is the largest grant.
  • Loads (FinOpsTab.xaml.cs, FinOpsShowReloadPolicy.ShouldLoadOnSelectionChange): Initialize no longer calls RefreshData(). It still starts the fleet inventory load. A selection that moves while the tab is hidden loads nothing and sets _firstShowPending, so the next show runs the whole load. A selection while the tab is shown loads as before.
  • Log (LocalDataService.DatabaseStates.cs): the first skip starts a one-shot timer and later skips join its pending list. The line is written the write-lock budget plus 5 s after that FIRST skip, so a lock that stays stuck still logs once a window, and it names every server that skipped in the window (servers 'A', 'B', 'C': skipped this cycle's ...). A single server keeps the old text. The per-server transition latch and the recovery line are unchanged.

Measured

A temp DuckDB store of the issue's shape: six servers, 21 h, one query_stats collection per minute per server with 400 rows, 4000 hashes per server, about 1.2 KB of text per row, 3.0 M rows, rows of the six servers interleaved by time (the layout a live store has, so a server filter prunes nothing). Run with memory_limit 2 GB as Lite sets it. I used the DuckDB 1.4 Python engine for the timing because the Lite process cannot be launched here. The statement text is the one in the source.

Operator profile, one thread, old statement: table scan of query_stats 1.26 s, the second scan for the text and its join 0.26 s, the LATERAL's window sort 0.17 s, the sample text's window sort 0.13 s. The scan is the cost that stays.

old new
one thread, interleaved store 2.3 s 1.9 s
8 threads, interleaved store 0.8 to 1.1 s 0.85 to 1.0 s
one thread, one server's rows contiguous 1.4 s 0.75 s
6 copies at once, 8 threads 5.3 s 4.4 s
12 copies at once, 8 threads 9.0 s 9.7 s

So the statement is cheaper by the two sorts, but on a store where the filter prunes nothing it is scan bound, and the new statement is not much faster. I could not reproduce a 12 s hold from one statement. What reproduced the shape is concurrency: twelve of this one statement at once take 9 s on a 32-core machine, and the window-start load runs fifteen reads together on a smaller one. That is why the load change is the part that addresses the issue, and the read change is the smaller part. The same results were checked on the seeded store: the old and new statements returned identical rows.

Tests

  • HighImpactReadEqualityTests: the old statement is kept in the test as the oracle and compared with the shipped one, every column of every row, at keep counts 1000, 10 and 3. The seed has NULL creation_time, NULL delta_worker_time, NULL count, a NULL grant, an empty and a NULL text, ties on collection_time, several plans, several databases, and a zero-execution newest row. A second test reads the shipped statement and asserts the grant and sample picked for each case, so a seed that stops exercising a NULL key fails there.
  • Revert proof: with the three COALESCE sentinels flipped (a NULL ranking above every value), all four tests failed. Restored, all pass.
  • StoreLockTimeoutLogTests: three servers skip under one held lock and one line names all three. The next burst writes its own line without the earlier servers. The two existing single-server tests now flush the burst and still pass.
  • FinOpsShowReloadPolicyTests.ASelectionChange_LoadsOnlyWhileTheTabIsShown: a hidden selection does not load and the first show reloads.
  • Not covered by a behaviour test: the WPF wiring in FinOpsTab (that Initialize no longer loads and that the hidden branch sets _firstShowPending). WPF cannot run in the suite. The existing source checks on the selection handler and the show handler still pass.

Gate (GATE STOP, the selection is the whole suite, so the Guard lines and my classes are the evidence):

[guard-darling]    Darling.Tests  Total: 4980, Errors: 0, Failed: 0, Skipped: 0, Not Run: 0
[guard-lite]    Lite.Tests  Total: 1244, Errors: 0, Failed: 0, Skipped: 0, Not Run: 0
classes: darling 3240, lite 1140

Classes run after merging dev, all green: HighImpactReadEqualityTests, StoreLockTimeoutLogTests, SkippedMaintenanceIsReportedTests, StoreBusyReportTests, FinOpsShowReloadPolicyTests, FinOpsHealthBadgeNoCpuTests, FinOpsStorageGrowthBaselineTests, FinOpsUtilizationNoCpuTests, FinOpsHighImpactBoundedReadTests, FinOpsSupersededLoadTests, FinOpsTests, LiteFinOpsDroppedDatabaseTests, ArchiveWindowReadScalingTests, ArchivedHistoryReadersTests. Not run: the full suite and Installer.Tests.

Darling parity

Darling's High Impact statement (DarlingFinOpsHighImpactReader.HighImpactQueriesSql) is not the same defect. Its per-hash LATERAL probes collect.query_stats itself, a table with indexes, filtered by server and hash. Lite's LATERAL scanned a materialized CTE, which has no index. So no Darling change is needed unless a measurement shows that read slow.

Not done, and why

  • The read that ran 14 minutes in with the tab never opened. The only caller of the read is the FinOps tab load, and I found no timer that runs it. The hidden-tab paths I found (start-up selection, RefreshData at start, a server-list change that moves the selection) are the ones now closed. A load while the tab is shown (a collection finishing, a show) is by design.

Considered, not added

  • A cycle-complete signal from the alert engine for the skip line. The engine is shared with Darling and has no cycle object. The fixed-window flush needs no change there. Cost: the line appears the write-lock budget plus 5 s after the first skip of a burst.
  • Bounding the read-lock hold (slicing the read, or copying the window to a temp table under a short lock and aggregating outside it). FinOps now loads only while shown, so the window-start pile-up of concurrent copies is gone, and a single copy of the read took about 2 s on the seeded store. A copy of the window under a short lock waits for a field measurement after this ships.

Fix round 1

  • Defect: each skip disposed and re-created the skip-log timer, so the quiet period restarted on every skip. While the store write lock stays out of reach, skips keep coming, and with enough servers they arrive less than one quiet period apart. A long-held or stuck read lock is that case: the timer never fired and the warning never logged, which is when it matters most.
  • Fix (LocalDataService.DatabaseStates.cs, NoteMaintenanceSkipped): the window is fixed from the first skip of a burst. When a timer is already pending, the skip joins the list and the timer is left alone. The flush still writes one line naming every server that skipped in the window, and the next skip after a flush starts a new window.
  • Test (StoreLockTimeoutLogTests.SkipsThatKeepArriving_StillLogWithinOneWindowOfTheFirst_EachLineNamingItsOwnServers): nine servers skip under one held lock, half a quiet period apart (1 s period), four and a half periods in all. It asserts the first line is written by the fifth skip, at least three lines are written, and every server is named in exactly one line. It counts skips and does not read a clock, so the timing source check holds.
  • Revert proof: with the timer reset on every skip restored, that test failed (no line was written until the final flush). With the fix, it passes. The other StoreLockTimeoutLogTests and SkippedMaintenanceIsReportedTests pass unchanged, twice in a row.
  • Gate after merging dev (GATE STOP, the selection is the whole suite, so the Guard lines and my classes are the evidence):
[guard-darling]    Darling.Tests  Total: 4980, Errors: 0, Failed: 0, Skipped: 0, Not Run: 0
[guard-lite]    Lite.Tests  Total: 1244, Errors: 0, Failed: 0, Skipped: 0, Not Run: 0
classes: darling 3247, lite 1140

… one skip line per cycle (#9233)

The High Impact statement picked each hash's grant with a per-hash LATERAL that sorted the whole window, and its sample text with a ROW_NUMBER sort over the text of every row of the kept hashes. Both are now one grouped pass with the same rows (equality test against the old statement).

The FinOps tab ran its whole per-server load at window start, twice, for a tab nobody had opened. A selection that moves while the tab is hidden now loads nothing, and the first show runs the load.

A burst of skipped database-state maintenance sweeps writes one line naming every server it skipped.
…xed (#9233)

A skip used to restart the quiet period. A stuck lock keeps skipping, so with enough servers the skips arrived less than a period apart and the line never logged. The first skip now starts the window, later skips only join it, and the next skip after the line starts a new one.
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review November 21, 2026 07:38
lock (_skipLogGate)
{
_skipsThisCycle.Add((server, holder));
_skipLogTimer ??= new System.Threading.Timer(_ => FlushMaintenanceSkipLog(), null, MaintenanceSkipLogQuietPeriod, System.Threading.Timeout.InfiniteTimeSpan);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking nit. Nothing flushes or disposes this timer when the service shuts down, so a burst whose first skip lands less than budget + 5 s (≈10 s) before the app closes never writes its Warn line. Before this PR that line was written synchronously. There's also an ordering quirk: the per-server recovery Info at line 301 is still written immediately. If a server skips and then recovers inside one window, the log shows "running again" before the "skipped" line it follows. With the 30 s overview sweep that's unlikely, so no change is needed. One option is to call FlushMaintenanceSkipLog() from the service's shutdown path, if it has one.

claude[bot]
claude Bot previously requested changes Nov 21, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: CHANGES REQUESTED

The code looks correct to me. The one blocking item is the PR body: its Fix → Log bullet still describes the sliding timer that Fix round 1 removed (see Claims below). Please rewrite that bullet; there is no code change to make. The inline comment on LocalDataService.DatabaseStates.cs:64 is non-blocking.

Claims

  • VERIFIED — Initialize no longer calls RefreshData() and still starts the fleet inventory load — Lite/Controls/FinOpsTab.xaml.cs:166-167
  • VERIFIED — a selection that moves while the tab is hidden loads nothing and sets _firstShowPending, so the next show runs the whole load — FinOpsTab.xaml.cs:1448-1452, with ReloadUnfinishedSizeGridsOnShow at :880-884
  • VERIFIED — the grant is one grouped arg_max keyed on a struct, and each NULLS LAST key is mapped with COALESCE to a value below every real one — Lite/Services/LocalDataService.FinOps.Workload.cs:600-608 (-infinity, -9223372036854775807; the columns are Timestamp/BigInt per QueryStatsCollector.cs:989,1011,1026-1027)
  • VERIFIED — the sample text is one grouped arg_min over negated keys, NULLs mapped to the largest value, with the text as the last member — Workload.cs:627-637
  • VERIFIED — the rows returned are the same; the only difference is that a full three-key tie with different grants now picks the largest grant — I traced the key mapping against the old ORDER BY and found them equivalent. HighImpactReadEqualityTests.TheGroupedPasses_ReturnTheRowsTheLateralAndTheRowNumberSortReturned compares the shipped statement against the old one, kept as an oracle, at keep counts 1000, 10 and 3
  • VERIFIED — a single server keeps the old text, and a burst names every server — LocalDataService.DatabaseStates.cs:204-206; StoreLockTimeoutLogTests.ASkippedCycle_WritesOneLine_NamingEveryServerItSkipped
  • VERIFIED — the per-server transition latch and the recovery line are unchanged — DatabaseStates.cs:292-303
  • VERIFIED — (Fix round 1) the window is fixed from the first skip and later skips only join it — DatabaseStates.cs:64 (_skipLogTimer ??=); StoreLockTimeoutLogTests.SkipsThatKeepArriving_StillLogWithinOneWindowOfTheFirst_EachLineNamingItsOwnServers
  • VERIFIED — the test FinOpsShowReloadPolicyTests.ASelectionChange_LoadsOnlyWhileTheTabIsShown exists — the class is at Lite.Tests/FinOpsFirstVisitAndBaselineTests.cs:31
  • VERIFIED — Darling's LATERAL probes the indexed collect.query_stats, not a materialized CTE — DarlingFinOpsHighImpactReader.cs:111-122
  • VERIFIED — the WPF wiring is not covered by a behaviour test — the only new FinOps test exercises the policy function, not the handler
  • UNVERIFIED — the timing table (2.3 s → 1.9 s, and the others) and the operator profile — these are live DuckDB measurements on a seeded 3 M-row store
  • UNVERIFIED — the revert proofs and the gate totals (Lite 1244 / Darling 4980 green) — checking them needs a build and a test run
  • UNVERIFIED — old and new statements return identical rows on the 3 M-row seeded store — that needs the rig; the in-tree oracle test covers the small seed only
  • REFUTED — Fix → Log: "a skip joins a pending list and (re)starts a one-shot timer. The line is written once no further server has skipped for the write-lock budget plus 5 s" — DatabaseStates.cs:64 creates the timer only when none is pending and never restarts it, so the line is written budget + 5 s after the FIRST skip. The body's own Fix round 1 says the sliding window was removed. The Fix bullet still describes it.

Riskiest lines

  1. Lite/Services/LocalDataService.FinOps.Workload.cs:600 — the struct-keyed arg_max replaces the LATERAL pick; a wrong sentinel inverts NULL ranking — covered by HighImpactReadEqualityTests.TheShippedStatement_PicksTheGrantAndTheSampleTheNullKeysCallFor (0xA, 0xB, 0xE, 0xF) and the oracle equality theory
  2. Lite/Services/LocalDataService.FinOps.Workload.cs:627 — the negated-key arg_min for the sample text; a wrong sign or sentinel picks the wrong text — covered by HighImpactReadEqualityTests.TheShippedStatement_PicksTheGrantAndTheSampleTheNullKeysCallFor (0xH, 0xI, 0xJ, 0xK, 0xL, 0xM)
  3. Lite/Controls/FinOpsTab.xaml.cs:1448 — the hidden-tab gate plus _firstShowPending = true; if the flag is dropped, a hidden reselect shows stale or empty grids until the age policy allows a reload — UNCOVERED. The body gives the reason (WPF does not run in the suite); ASelectionChange_LoadsOnlyWhileTheTabIsShown exercises only the identity policy, not the handler

Traps near this diff

  • "A test that reads the clock twice measures the CI scheduler" — does not apply — StoreLockTimeoutLogTests.SkipsThatKeepArriving_… measures no interval: it counts skips, and a slow scheduler only widens the gaps, which produces more lines and still satisfies lines.Count >= 3. There is a narrow race where a timer callback holds the snapshot while the final drain runs, but nothing in it reads the clock twice
  • "MAX(max_dop) grouped by query_hash is the highest DOP ANY plan for that statement text ever ran at since it was cached … An aggregate across plans needs provenance (which plan, when) or the newest plan's reading." — does not apply — the grant is still the newest row's reading (collection_time, then creation_time first in the arg_max key), not a MAX across plans — Workload.cs:600-604
  • "Lite and Darling ship in parity (CONTRIBUTING's two-store rule): a fix to one SKU's reader, tool, alert or description without the counterpart's is a finding" — does not apply — the operator fixed here is DuckDB-specific (a LATERAL over an unindexed MATERIALIZED CTE). Darling's LATERAL is an index probe (DarlingFinOpsHighImpactReader.cs:109-122), and Darling has no database-state maintenance skip line (grep for "skipped this cycle's database-state maintenance" under Darling/ finds nothing)
  • "Ruling (#5833): tests are load-bearing … never answer a trap with a test that reads source text or asserts how code is spelled" — does not apply — the new tests execute SQL against DuckDB and drive the skip logging through a held lock (HighImpactReadEqualityTests.cs, StoreLockTimeoutLogTests.cs:110-230). None of them reads source text
  • "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy." — does not apply — the oracle in HighImpactReadEqualityTests.cs:43 is a frozen copy of the pre-#9233 statement, not a second live definition. A drift in the shipped statement fails the comparison on its own

Divergence (blind read vs PR body)

  • The Fix → Log bullet describes a sliding, restarting timer ("once no further server has skipped for …"). The code at DatabaseStates.cs:64 implements a fixed window that opens at the first skip, as Fix round 1 describes. Anyone who reads only the Fix section (or reuses it for the release note) gets the wrong mechanism. Please rewrite that bullet to match :64.
  • The body does not mention one effect of the change: the skip Warn is now deferred by budget + 5 s (≈10 s) and is not flushed at shutdown. The per-server recovery Info at :301 is still written immediately, so a skip that resolves inside one window logs "running again" before "skipped". There is an inline comment on :64; it is non-blocking.
  • The body says the hidden-selection test shows "the first show reloads". Its second assert, ShouldReloadOnShow(null, Now, firstShowSinceStart: true), tests behaviour that already existed; nothing ties the handler's _firstShowPending = true to it. The body admits this under "Not covered".

Reviewer checklist

  • Run HighImpactReadEqualityTests and StoreLockTimeoutLogTests on CI and confirm they pass, and that SkipsThatKeepArriving_… stays stable across a few reruns
  • Launch Lite with several servers and check that the FinOps tab issues no store reads at window start until it is first shown, and that the first show fills every grid
  • With the FinOps tab hidden, edit the server list so the selection moves, then show the tab and confirm the new server's data loads
  • On a real store, hold the write lock (archival) across one alert sweep and check that a single "servers 'A', 'B', …" Warn line appears about 10 s after the first skip

@erikdarlingdata
erikdarlingdata dismissed claude[bot]’s stale review November 21, 2026 08:01

Claude posting for Erik Darling. The only blocking item was the Fix > Log bullet in the PR body. It now describes the fixed window at DatabaseStates.cs:64. No code change was requested.

@erikdarlingdata
erikdarlingdata merged commit e2dda12 into dev Nov 21, 2026
24 of 28 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/9233-high-impact-read-lock branch November 21, 2026 08:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant