Repository navigation
Lite: cheaper High Impact read, FinOps loads wait for the first show, one skip line per cycle (#9233) - #9235
Conversation
… one skip line per cycle (#9233) The High Impact statement picked each hash's grant with a per-hash LATERAL that sorted the whole window, and its sample text with a ROW_NUMBER sort over the text of every row of the kept hashes. Both are now one grouped pass with the same rows (equality test against the old statement). The FinOps tab ran its whole per-server load at window start, twice, for a tab nobody had opened. A selection that moves while the tab is hidden now loads nothing, and the first show runs the load. A burst of skipped database-state maintenance sweeps writes one line naming every server it skipped.
…xed (#9233) A skip used to restart the quiet period. A stuck lock keeps skipping, so with enough servers the skips arrived less than a period apart and the line never logged. The first skip now starts the window, later skips only join it, and the next skip after the line starts a new one.
| lock (_skipLogGate) | ||
| { | ||
| _skipsThisCycle.Add((server, holder)); | ||
| _skipLogTimer ??= new System.Threading.Timer(_ => FlushMaintenanceSkipLog(), null, MaintenanceSkipLogQuietPeriod, System.Threading.Timeout.InfiniteTimeSpan); |
There was a problem hiding this comment.
Non-blocking nit. Nothing flushes or disposes this timer when the service shuts down, so a burst whose first skip lands less than budget + 5 s (≈10 s) before the app closes never writes its Warn line. Before this PR that line was written synchronously. There's also an ordering quirk: the per-server recovery Info at line 301 is still written immediately. If a server skips and then recovers inside one window, the log shows "running again" before the "skipped" line it follows. With the 30 s overview sweep that's unlikely, so no change is needed. One option is to call FlushMaintenanceSkipLog() from the service's shutdown path, if it has one.
There was a problem hiding this comment.
Verdict: CHANGES REQUESTED
The code looks correct to me. The one blocking item is the PR body: its Fix → Log bullet still describes the sliding timer that Fix round 1 removed (see Claims below). Please rewrite that bullet; there is no code change to make. The inline comment on LocalDataService.DatabaseStates.cs:64 is non-blocking.
Claims
- VERIFIED —
Initializeno longer callsRefreshData()and still starts the fleet inventory load —Lite/Controls/FinOpsTab.xaml.cs:166-167 - VERIFIED — a selection that moves while the tab is hidden loads nothing and sets
_firstShowPending, so the next show runs the whole load —FinOpsTab.xaml.cs:1448-1452, withReloadUnfinishedSizeGridsOnShowat:880-884 - VERIFIED — the grant is one grouped
arg_maxkeyed on a struct, and each NULLS LAST key is mapped with COALESCE to a value below every real one —Lite/Services/LocalDataService.FinOps.Workload.cs:600-608(-infinity,-9223372036854775807; the columns are Timestamp/BigInt perQueryStatsCollector.cs:989,1011,1026-1027) - VERIFIED — the sample text is one grouped
arg_minover negated keys, NULLs mapped to the largest value, with the text as the last member —Workload.cs:627-637 - VERIFIED — the rows returned are the same; the only difference is that a full three-key tie with different grants now picks the largest grant — I traced the key mapping against the old ORDER BY and found them equivalent.
HighImpactReadEqualityTests.TheGroupedPasses_ReturnTheRowsTheLateralAndTheRowNumberSortReturnedcompares the shipped statement against the old one, kept as an oracle, at keep counts 1000, 10 and 3 - VERIFIED — a single server keeps the old text, and a burst names every server —
LocalDataService.DatabaseStates.cs:204-206;StoreLockTimeoutLogTests.ASkippedCycle_WritesOneLine_NamingEveryServerItSkipped - VERIFIED — the per-server transition latch and the recovery line are unchanged —
DatabaseStates.cs:292-303 - VERIFIED — (Fix round 1) the window is fixed from the first skip and later skips only join it —
DatabaseStates.cs:64(_skipLogTimer ??=);StoreLockTimeoutLogTests.SkipsThatKeepArriving_StillLogWithinOneWindowOfTheFirst_EachLineNamingItsOwnServers - VERIFIED — the test
FinOpsShowReloadPolicyTests.ASelectionChange_LoadsOnlyWhileTheTabIsShownexists — the class is atLite.Tests/FinOpsFirstVisitAndBaselineTests.cs:31 - VERIFIED — Darling's LATERAL probes the indexed
collect.query_stats, not a materialized CTE —DarlingFinOpsHighImpactReader.cs:111-122 - VERIFIED — the WPF wiring is not covered by a behaviour test — the only new FinOps test exercises the policy function, not the handler
- UNVERIFIED — the timing table (2.3 s → 1.9 s, and the others) and the operator profile — these are live DuckDB measurements on a seeded 3 M-row store
- UNVERIFIED — the revert proofs and the gate totals (Lite 1244 / Darling 4980 green) — checking them needs a build and a test run
- UNVERIFIED — old and new statements return identical rows on the 3 M-row seeded store — that needs the rig; the in-tree oracle test covers the small seed only
- REFUTED — Fix → Log: "a skip joins a pending list and (re)starts a one-shot timer. The line is written once no further server has skipped for the write-lock budget plus 5 s" —
DatabaseStates.cs:64creates the timer only when none is pending and never restarts it, so the line is written budget + 5 s after the FIRST skip. The body's own Fix round 1 says the sliding window was removed. The Fix bullet still describes it.
Riskiest lines
Lite/Services/LocalDataService.FinOps.Workload.cs:600— the struct-keyedarg_maxreplaces the LATERAL pick; a wrong sentinel inverts NULL ranking — covered byHighImpactReadEqualityTests.TheShippedStatement_PicksTheGrantAndTheSampleTheNullKeysCallFor(0xA, 0xB, 0xE, 0xF) and the oracle equality theoryLite/Services/LocalDataService.FinOps.Workload.cs:627— the negated-keyarg_minfor the sample text; a wrong sign or sentinel picks the wrong text — covered byHighImpactReadEqualityTests.TheShippedStatement_PicksTheGrantAndTheSampleTheNullKeysCallFor(0xH, 0xI, 0xJ, 0xK, 0xL, 0xM)Lite/Controls/FinOpsTab.xaml.cs:1448— the hidden-tab gate plus_firstShowPending = true; if the flag is dropped, a hidden reselect shows stale or empty grids until the age policy allows a reload — UNCOVERED. The body gives the reason (WPF does not run in the suite);ASelectionChange_LoadsOnlyWhileTheTabIsShownexercises only the identity policy, not the handler
Traps near this diff
- "A test that reads the clock twice measures the CI scheduler" — does not apply —
StoreLockTimeoutLogTests.SkipsThatKeepArriving_…measures no interval: it counts skips, and a slow scheduler only widens the gaps, which produces more lines and still satisfieslines.Count >= 3. There is a narrow race where a timer callback holds the snapshot while the final drain runs, but nothing in it reads the clock twice - "
MAX(max_dop)grouped byquery_hashis the highest DOP ANY plan for that statement text ever ran at since it was cached … An aggregate across plans needs provenance (which plan, when) or the newest plan's reading." — does not apply — the grant is still the newest row's reading (collection_time, then creation_time first in thearg_maxkey), not a MAX across plans —Workload.cs:600-604 - "Lite and Darling ship in parity (CONTRIBUTING's two-store rule): a fix to one SKU's reader, tool, alert or description without the counterpart's is a finding" — does not apply — the operator fixed here is DuckDB-specific (a LATERAL over an unindexed MATERIALIZED CTE). Darling's LATERAL is an index probe (
DarlingFinOpsHighImpactReader.cs:109-122), and Darling has no database-state maintenance skip line (grep for "skipped this cycle's database-state maintenance" underDarling/finds nothing) - "Ruling (#5833): tests are load-bearing … never answer a trap with a test that reads source text or asserts how code is spelled" — does not apply — the new tests execute SQL against DuckDB and drive the skip logging through a held lock (
HighImpactReadEqualityTests.cs,StoreLockTimeoutLogTests.cs:110-230). None of them reads source text - "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy." — does not apply — the oracle in
HighImpactReadEqualityTests.cs:43is a frozen copy of the pre-#9233 statement, not a second live definition. A drift in the shipped statement fails the comparison on its own
Divergence (blind read vs PR body)
- The Fix → Log bullet describes a sliding, restarting timer ("once no further server has skipped for …"). The code at
DatabaseStates.cs:64implements a fixed window that opens at the first skip, as Fix round 1 describes. Anyone who reads only the Fix section (or reuses it for the release note) gets the wrong mechanism. Please rewrite that bullet to match:64. - The body does not mention one effect of the change: the skip Warn is now deferred by budget + 5 s (≈10 s) and is not flushed at shutdown. The per-server recovery Info at
:301is still written immediately, so a skip that resolves inside one window logs "running again" before "skipped". There is an inline comment on:64; it is non-blocking. - The body says the hidden-selection test shows "the first show reloads". Its second assert,
ShouldReloadOnShow(null, Now, firstShowSinceStart: true), tests behaviour that already existed; nothing ties the handler's_firstShowPending = trueto it. The body admits this under "Not covered".
Reviewer checklist
- Run
HighImpactReadEqualityTestsandStoreLockTimeoutLogTestson CI and confirm they pass, and thatSkipsThatKeepArriving_…stays stable across a few reruns - Launch Lite with several servers and check that the FinOps tab issues no store reads at window start until it is first shown, and that the first show fills every grid
- With the FinOps tab hidden, edit the server list so the selection moves, then show the tab and confirm the new server's data loads
- On a real store, hold the write lock (archival) across one alert sweep and check that a single "servers 'A', 'B', …" Warn line appears about 10 s after the first skip
Claude posting for Erik Darling. The only blocking item was the Fix > Log bullet in the PR body. It now describes the fixed window at DatabaseStates.cs:64. No code change was requested.
Claude posting for Erik Darling.
Fixes #9233
Release note: Lite no longer loads the FinOps panels at window start for a tab nobody has opened, the High Impact read does less work, and a skipped database-state maintenance cycle logs one line naming every server it skipped.
Cause
Three things met at window start.
FinOpsTab.Initializeselected the first server (which fired the selection handler and a whole per-server load) and then calledRefreshData()(a second whole load). Each is fifteen store reads, High Impact among them, all at once, on a tab that is not visible and against a store the first collection has not written to. The tab's first show already re-runs the whole load whatever the age of any earlier one (FinOpsShowReloadPolicy), so the start-up loads were pure cost. A server-list change that moved the selection while the tab was hidden ran the same load again.LEFT JOIN LATERAL ... ORDER BY ... LIMIT 1over the materialized window, and the sample text is aROW_NUMBERsort that carries the text of every row of the kept hashes.Fix
LocalDataService.FinOps.Workload.cs): the grant is one groupedarg_maxover the window, keyed on a struct. A struct key ranks a NULL field above every value in DuckDB (Lite MCP: get_query_store_top with no database filter and get_analysis_facts take over 5 s on a small install, and the facts reply passes the budget with no note #9195), so each key that sorted NULLS LAST is mapped with COALESCE to a value below every real one. The sample text is one groupedarg_minover a negated key, NULLs mapped to the largest value, with the text as the last member so a full tie still goes to the smallest text. The rows returned are the same. The one deliberate difference: when rows tie on all three grant keys but hold different grants, the old pick was whichever the sort met first and the new one is the largest grant.FinOpsTab.xaml.cs,FinOpsShowReloadPolicy.ShouldLoadOnSelectionChange):Initializeno longer callsRefreshData(). It still starts the fleet inventory load. A selection that moves while the tab is hidden loads nothing and sets_firstShowPending, so the next show runs the whole load. A selection while the tab is shown loads as before.LocalDataService.DatabaseStates.cs): the first skip starts a one-shot timer and later skips join its pending list. The line is written the write-lock budget plus 5 s after that FIRST skip, so a lock that stays stuck still logs once a window, and it names every server that skipped in the window (servers 'A', 'B', 'C': skipped this cycle's ...). A single server keeps the old text. The per-server transition latch and the recovery line are unchanged.Measured
A temp DuckDB store of the issue's shape: six servers, 21 h, one
query_statscollection per minute per server with 400 rows, 4000 hashes per server, about 1.2 KB of text per row, 3.0 M rows, rows of the six servers interleaved by time (the layout a live store has, so a server filter prunes nothing). Run withmemory_limit2 GB as Lite sets it. I used the DuckDB 1.4 Python engine for the timing because the Lite process cannot be launched here. The statement text is the one in the source.Operator profile, one thread, old statement: table scan of
query_stats1.26 s, the second scan for the text and its join 0.26 s, the LATERAL's window sort 0.17 s, the sample text's window sort 0.13 s. The scan is the cost that stays.So the statement is cheaper by the two sorts, but on a store where the filter prunes nothing it is scan bound, and the new statement is not much faster. I could not reproduce a 12 s hold from one statement. What reproduced the shape is concurrency: twelve of this one statement at once take 9 s on a 32-core machine, and the window-start load runs fifteen reads together on a smaller one. That is why the load change is the part that addresses the issue, and the read change is the smaller part. The same results were checked on the seeded store: the old and new statements returned identical rows.
Tests
HighImpactReadEqualityTests: the old statement is kept in the test as the oracle and compared with the shipped one, every column of every row, at keep counts 1000, 10 and 3. The seed has NULL creation_time, NULL delta_worker_time, NULL count, a NULL grant, an empty and a NULL text, ties on collection_time, several plans, several databases, and a zero-execution newest row. A second test reads the shipped statement and asserts the grant and sample picked for each case, so a seed that stops exercising a NULL key fails there.StoreLockTimeoutLogTests: three servers skip under one held lock and one line names all three. The next burst writes its own line without the earlier servers. The two existing single-server tests now flush the burst and still pass.FinOpsShowReloadPolicyTests.ASelectionChange_LoadsOnlyWhileTheTabIsShown: a hidden selection does not load and the first show reloads.FinOpsTab(thatInitializeno longer loads and that the hidden branch sets_firstShowPending). WPF cannot run in the suite. The existing source checks on the selection handler and the show handler still pass.Gate (
GATE STOP, the selection is the whole suite, so the Guard lines and my classes are the evidence):Classes run after merging dev, all green: HighImpactReadEqualityTests, StoreLockTimeoutLogTests, SkippedMaintenanceIsReportedTests, StoreBusyReportTests, FinOpsShowReloadPolicyTests, FinOpsHealthBadgeNoCpuTests, FinOpsStorageGrowthBaselineTests, FinOpsUtilizationNoCpuTests, FinOpsHighImpactBoundedReadTests, FinOpsSupersededLoadTests, FinOpsTests, LiteFinOpsDroppedDatabaseTests, ArchiveWindowReadScalingTests, ArchivedHistoryReadersTests. Not run: the full suite and Installer.Tests.
Darling parity
Darling's High Impact statement (
DarlingFinOpsHighImpactReader.HighImpactQueriesSql) is not the same defect. Its per-hash LATERAL probescollect.query_statsitself, a table with indexes, filtered by server and hash. Lite's LATERAL scanned a materialized CTE, which has no index. So no Darling change is needed unless a measurement shows that read slow.Not done, and why
RefreshDataat start, a server-list change that moves the selection) are the ones now closed. A load while the tab is shown (a collection finishing, a show) is by design.Considered, not added
Fix round 1
LocalDataService.DatabaseStates.cs,NoteMaintenanceSkipped): the window is fixed from the first skip of a burst. When a timer is already pending, the skip joins the list and the timer is left alone. The flush still writes one line naming every server that skipped in the window, and the next skip after a flush starts a new window.StoreLockTimeoutLogTests.SkipsThatKeepArriving_StillLogWithinOneWindowOfTheFirst_EachLineNamingItsOwnServers): nine servers skip under one held lock, half a quiet period apart (1 s period), four and a half periods in all. It asserts the first line is written by the fifth skip, at least three lines are written, and every server is named in exactly one line. It counts skips and does not read a clock, so the timing source check holds.StoreLockTimeoutLogTestsandSkippedMaintenanceIsReportedTestspass unchanged, twice in a row.GATE STOP, the selection is the whole suite, so the Guard lines and my classes are the evidence):