Repository navigation
Darling blocking baseline leaves out self-waits, like the window it is compared with (#6690) - #6705
Conversation
…s compared with (#6690)
| collection_time, | ||
| count(*) AS event_count | ||
| FROM collect.blocked_process_reports | ||
| WHERE " + PerformanceMonitor.Common.BlockingWaits.NotSelfWaitSql + @" |
There was a problem hiding this comment.
This aggregate serves TWO families, and only one of them has a filtered current window. The doc comment just above says "both leave a parallel query's self-waits out of their current window (#6663), so the baseline they are compared with leaves them out too" — that holds for MetricNames.Blocking, whose current count is PgAnomalyDetector.BlockingWindowSql (lines 317-322, both arms carry NotSelfWaitSql). It does not hold for MetricNames.BlockingPerMinute.
BlockingPerMinute has no anomaly-detector arm. Its only consumers are the Overview blocking lane (CorrelatedTimelineLanesControl.xaml.cs:193 → UpdateBlockingLane, band = mean ± 2σ) and the web Overview strip (DarlingWebOverviewBaselines.cs:39), and the series that band is drawn over is ViewerDataService.BlockingTrends.cs:72 (BlockingTrendSql) — a plain COUNT(*) per minute over v_blocked_process_reports with no NotSelfWaitSql.
So after this change the band excludes self-waits while the bars still count them, which is the mirror image of the defect being fixed. Concretely: a server whose parallel query writes self-wait reports at, say, 6 of every 8 reports per minute gets a band around mean ≈ 2 while the lane keeps drawing bars at 8 — every ordinary minute now plots above the band, and UpdateBlockingLane passes no minAnomalyValue floor (unlike the CPU and I/O lanes), so nothing suppresses it. The anomaly line moves in the under-flagging direction for Blocking and the over-flagging direction for BlockingPerMinute.
Either BlockingTrendSql needs the same predicate, or BlockingPerMinute needs a supply that still counts every report row — but the two cannot disagree, and the comment should not claim they agree.
Related, same root: Lite is now out of parity for this family. Lite/Analysis/BaselineProvider.cs:778 (the BlockingPerMinute arm) is COUNT(*) over StoredEventCopies.BlockedProcessReports("server_id = $1 AND event_time >= $2 AND event_time < $3") with no NotSelfWaitSql — the PR body's "Lite: its baseline query already filters … No change" cites lines 714-719, which is the Blocking arm only. Lite's band and Lite's bars still agree with each other (both unfiltered); Darling's no longer do, and the same lane now bands differently in the two SKUs.
The "about 4 days" in the body and in the CHANGELOG is the wrong tier
Blocked-process reports are purged by the collector-retention path instead, at So the real cost of the drop-and-rebuild on a default store is about 5 days (the 35-day baseline tier minus the 30-day raw horizon), not 31. That is a much easier thing to accept, and it is worth stating correctly rather than conservatively, because the number is going out to users verbatim:
A reader of that line concludes their blocking anomaly detection is effectively off for a month. On a default store it loses the oldest five days of a thirty-day window. Two things that are worth keeping from that paragraph, restated against the right tier:
Not blocking on its own, but the CHANGELOG entry should be corrected before this merges. |
| /* blocked_process_baseline gained the self-wait filter (#6690) -> stale iff the aggregate EXISTS but its | ||
| definition does not mention blocking_spid (the filter's column; the old text had no WHERE at all). | ||
| Nothing is built on it, so the CASCADE drops only the view; the ensure sweep recreates it WITH NO DATA | ||
| and the start path's coverage-gated BackfillBaselineAggregatesAsync (its materialization is empty, so | ||
| its gate fires on this very start) re-materializes it from raw. NULL view_definition is not stale. */ | ||
| (View: BlockedProcessBaselineView, | ||
| StaleCheck: "SELECT EXISTS (SELECT 1 FROM timescaledb_information.continuous_aggregates WHERE view_schema = 'collect' AND view_name = 'blocked_process_baseline' AND position('blocking_spid' IN view_definition) = 0)"), |
There was a problem hiding this comment.
The history this drop forfeits does not read as missing, it reads as zero blocking. The body calls the cost "the hour/day-of-week buckets are thinner (the anomaly check's own minimum-sample rules apply)". They are not thinner — they are full, with fabricated quiet samples in them, which is the #3590 trap ("a retention-tiered store answers a 7-day question from a 4-day raw tier and labels the missing days quiet").
MetricNames.Blocking is the event family, PgBaselineProvider.EventBaselineSql (PgBaselineProvider.cs:1170-1204). Its slots CTE is logged (n = 0) UNION ALL events (n), and the mean is SUM(n) / COUNT(DISTINCT d) over that union:
eventscomes fromblocked_process_baseline— after the drop-and-rebuild it reaches back only as far as rawblocked_process_reportsdoes.loggedcomes fromcollection_log, whose retention isDarlingRetentionHorizons.CollectionLogRetentionDays=DataRetentionBaseDays * 2(DailySummaryHorizon.cs:46) — deliberately twice the base, "so it needs no floor of the kind the raw baseline sources have" (PgBaselineProvider.cs:1160-1162).
So for every day in the baseline window that the rebuilt aggregate no longer covers, collection_log still has the collector's SUCCESS runs, logged contributes that day with n = 0, and EXISTS (SELECT 1 FROM events) is satisfied by the recent days. Those days enter the mean as measured zeros and enter sample_count / distinct_days as full samples. The bucket does not look short to the detector; it looks like a server that reliably did not block on those days.
Net effect on a default store (~30 days raw vs the 35-day baseline tier, see my top-level note about the RawRetentionInterval figure): for roughly the first five days after the upgrade start, each (hour, dow) bucket carries about one fabricated zero out of about four samples, so the blocking mean reads ~25% low and real blocking is over-flagged — the opposite error from the one this PR fixes, for a bounded window. On a store whose operator shortened blocked_process_report retention it is worse and lasts longer: blocked_process_report is not in DarlingRetentionHorizons.BaselineServingRawCollectors (DailySummaryHorizon.cs:107-118), so nothing floors its raw horizon at BaselineMath.BaselineWindowDays the way cpu_utilization and file_io_stats are floored.
This is transient and self-healing, so I am not asking for the reshape to be abandoned. But it should be stated honestly, and there is a cheap mitigation worth considering: have the backfill (or a one-shot alongside it) record the rebuild instant, and have the Blocking arm not count logged days older than the aggregate's own min(bucket) — the same "say which tier served it" discipline RetentionTierRouter already applies elsewhere. At minimum the doc here and the CHANGELOG should say the lost days read as quiet rather than as absent.
There was a problem hiding this comment.
Verdict: CHANGES REQUESTED
The reshape mechanism is right, the predicate is aliased rather than copied, and the two live tests are real behaviour tests that run in PR CI. Two things block: the aggregate serves a SECOND family (BlockingPerMinute) whose current series is not filtered, so the band now reads low against its own bars; and Lite's half of that family is untouched. Details inline.
Claims
- VERIFIED —
CreateBlockedProcessBaselineSqlcarriesWHERE+BlockingWaits.NotSelfWaitSql, referenced not copied —TimescaleSupport.cs:809 - VERIFIED — the plain-PG fallback is derived from the same body, so a fresh plain store gets the filter —
CreateBaselineFallbackViewSqlslices betweenBaselineBodyMarkerand the lastWITH NO DATA,TimescaleSupport.cs:1026-1039 - VERIFIED —
EnsureBaselineFallbackViewsAsyncreplaces a stale PLAIN view only —TimescaleSupport.cs:1079-1091, probe at1125-1136; a non-view returns no row from therelkind='v'filter, so theis truetest is false - VERIFIED — "the ungated stage, so it runs on every store shape" —
DarlingWorker.cs:444, withNeedsTimescaleatDarlingWorker.cs:261 - VERIFIED — the stale entry shares the others' quiesce /
DROP ... CASCADE/ isolation (TimescaleSupport.cs:6251-6252in the loop at6256-6312), and its step isStage.Timescale(DarlingWorker.cs:392-393), so thetimescaledb_informationreference never runs on a plain-PG store and cannot throw 42P01 there - VERIFIED — "nothing is built on
blocked_process_baseline" — only readers arePgBaselineProvider.cs:1424and:1533; only membership isBaselineAggregates(TimescaleSupport.cs:2402) - VERIFIED — the backfill gate reads coverage off the materialization, empty on a fresh aggregate, so it fires that same start —
TimescaleSupport.cs:876+ the doc at1170-1183 - VERIFIED —
BackfillBaselineAggregatesAsyncis launched off the start path after the ensure —DarlingWorker.cs:5312, list doc at344-348, ensure at420 - VERIFIED — the
PgAnomalyDetectorcomment no longer claims the baseline counts self-waits —PgAnomalyDetector.cs:310-312 - VERIFIED —
BaselineSupplyTestsassertion flipped —BaselineSupplyTests.cs:110-116 - VERIFIED — Lite's
Blockingarm already filters —Lite/Analysis/BaselineProvider.cs:714-719 - VERIFIED — the new live tests are not nightly-only:
pr.yml's legs start the bundled cluster,start-test-pg.ps1:39setsDARLING_TEST_PGand:95preloadstimescaledb, so the hardAssert.True(TryEnableAsync(...))has an extension to find - UNVERIFIED — "no-op on a converged store" for the CAGG stale check: nothing else in the tree reads
timescaledb_information.continuous_aggregates.view_definition, so whether it renders theWHEREcannot be decided from a read-only tree. The test's closingAssert.Equal(0, DropStale...)is that check; it needs the rig - UNVERIFIED — "0 errors, 15 warnings" — needs a build
- UNVERIFIED — "49 passed / 0 failed" and "904 passed / 0 failed" — needs the rig
- UNVERIFIED — "they assert counts 2 vs 1 that the old definition cannot produce", offered instead of reverting the fix: plausible from the test text, not demonstrated red on old code
- REFUTED — "#6663 made both blocking families leave self-waits out of their current window": true for
Blocking(PgAnomalyDetector.cs:317-322), false forBlockingPerMinute, whose only series isViewerDataService.BlockingTrends.cs:72(bareCOUNT(*)) feedingCorrelatedTimelineLanesControl.xaml.cs:193andDarlingWebOverviewBaselines.cs:39 - REFUTED — "Lite: its baseline query already filters … No change": the cited lines are the
Blockingarm;Lite/Analysis/BaselineProvider.cs:774-784has noNotSelfWaitSql, so this opens a parity gap in that family - REFUTED — "Raw
blocked_process_reportsretention is 4 days (RawRetentionInterval)":RawTierCoveragehas three members (TimescaleSupport.cs:7824-7841) andRetentionPolicies(7881-7883) arms that interval only on those;8128says "the three raw tables". Reports are purged atCollectorScheduleDefaults.All["blocked_process_report"] = new(1, 30)— 30 days. The forfeit is ~5 days of the 35-day tier, not 31, and the CHANGELOG ships the wrong number (top-level note)
Riskiest lines
Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs:809— oneWHEREon a supply serving two families, only one with a filtered current window — covered forBlockingbyBlockingBaselineBuiltWithoutTheSelfWaitFilter_IsRebuiltFilteredAndBackfilledOverTheWholeWindow_AgainstDevPostgres; UNCOVERED forBlockingPerMinuteDarling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs:6252— ifview_definitionnever holdsblocking_spidthe aggregate is dropped and rebuilt every start and every hourly tick — covered by that test's closingAssert.Equal(0, ...DropStaleContinuousAggregatesAsync...)Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs:1133— a probe that mis-read a CAGG as a plain view wouldCREATE OR REPLACE VIEWover a materialization — covered byPlainBlockingBaselineViewWithoutTheSelfWaitFilter_IsReplacedInPlace_AgainstDevPostgresDarling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs:6251— forfeited days re-enter theBlockingmean as measured zeros viaEventBaselineSql'sloggedCTE instead of dropping out of the sample — UNCOVERED
Traps near this diff
- "Lite and Darling ship in parity (CONTRIBUTING's two-store rule): a fix to one SKU's reader, tool, alert or description without the counterpart's is a finding, not a nit" — applies —
TimescaleSupport.cs:809filtered,Lite/Analysis/BaselineProvider.cs:778not - "A retention-tiered store answers a 7-day question from a 4-day raw tier and labels the missing days quiet unless the read routes by tier and says which tier served it (#3590; the viewer's charts in #3684)" — applies — the rebuilt aggregate's missing days still have
collection_logSUCCESS rows (DailySummaryHorizon.cs:46), soEventBaselineSql'sslotsunion (PgBaselineProvider.cs:1191-1196) enters them atn = 0and counts them insample_count - "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy." — does not apply — referenced at
TimescaleSupport.cs:809, and the fallback derives from the same body at1026-1039 - "Ruling (#5833): tests are load-bearing … never answer a trap with a test that reads source text" — does not apply —
BaselineSupplyTests.cs:116edits a pin that already existed, and the new coverage is two behaviour tests (FleetPartialDatabaseFailureBandTests.cs:617,:706) - "
timescaledb_information.hypertablesnever lists a continuous aggregate's materialization … Enumeratecontinuous_aggregates" — does not apply — both new reads already go throughcontinuous_aggregates(TimescaleSupport.cs:6252,1226) - "A first-level continuous aggregate's materialization is chunked at ten times the raw interval unless
set_chunk_time_intervalsays otherwise (#3620 → #3621)" — does not apply — the recreated aggregate goes back through the convergence list, whoseaggregate compressionstep carries #3620's chunk-width ensure (DarlingWorker.cs:430-431)
Divergence (blind read vs PR body)
Three places, all the same shape: the body reasons about the Blocking family and states the conclusion for both.
- The second family is never separated. The doc at
TimescaleSupport.cs:794-799says itself that this supply servesBlockingANDBlockingPerMinute.Blocking's current window is filtered;BlockingPerMinutehas no anomaly arm at all — its band is drawn overBlockingTrendSql, which is not. "Both blocking families" is the premise of the change and is half true. - Lite. "No change" is right for
Blocking, wrong forBlockingPerMinute. The body visits:714-719and not:774-784, which is the arm that must move for the two SKUs to band the same lane the same way. - The retention tier. "What a store loses (please read)" — the section explicitly asking a reviewer to check it — is built on a tier that does not cover this table, and so is the CHANGELOG line. The correction makes the case stronger (five days, not thirty-one); what the body does not say is that the lost days read as zero blocking rather than as absent.
The parts I was least sure of from the diff alone — no migration needed, no new warm needed because the gate reads the empty materialization — both check out against the tree.
Reviewer checklist
- On the rig:
SELECT view_definition FROM timescaledb_information.continuous_aggregates WHERE view_name = 'blocked_process_baseline'after the ensure, and confirmblocking_spidis in it. If not, the sweep drop-and-rebuilds the blocking baseline on every start and every hourly tick. - On a store with real blocking history, after one start on this build:
min(bucket)of the recreated aggregate vssample_countfor one (hour, dow) bucket of the 30-dayBlockingbaseline — the forfeited days should appear as samples with no events. - Open the Viewer Overview blocking lane and the web Overview blocking strip on a server that produces parallel-query self-wait reports: do the bars now sit above the band?
- Confirm the recreated aggregate's retention policy is re-armed after the CASCADE drop —
EnsureRetentionPoliciesAsyncis start-path-only and its coverage gate will see a short tier.
There was a problem hiding this comment.
Verdict: LGTM
Claims
- VERIFIED — every XE-preferred/DMV-fallback series method (
GetBlockingTrendAsync,GetBlockingDurationStatsAsync,GetBlockingSlicerDataAsync) now appliesBlockingWaits.NotSelfWaitSqlon both arms —Lite/Services/LocalDataService.Blocking.cs:1004,1016,1217,1223,Lite/Services/LocalDataService.BlockingStats.cs:99,111 - VERIFIED —
HasBlockedProcessReportsInWindowAsynctakes the same predicate —Lite/Services/LocalDataService.Blocking.cs:1102 - VERIFIED —
BaselineProvider.cs'sBlockingPerMinutearm filters self-waits, and theBlockingarm's comment now says the Darling aggregate filters too —Lite/Analysis/BaselineProvider.cs:714-715,778 - VERIFIED —
GetEarliestBlockedProcessReportInWindowAsyncis left unfiltered (self-wait still proves XE capture), doc comment reworded, SQL unchanged —Lite/Services/LocalDataService.Blocking.cs:1113-1124 - VERIFIED — the described test scenarios (self-wait+real+zero-pair minute counts 2 across trend/duration/slicer,
HasBlockedProcessReportsInWindowAsynctrue; XE-only-self-wait window falls to the DMV arm and the DMV arm's own self-wait is also dropped) matchBlockingSeries_LeaveSelfWaitsOut_OnTheXeArmandBlockingSeries_OnlySelfWaitReportsInXe_DrawTheDmvArm_AndDmvSelfWaitsGoinLite.Tests/BlockingStatsReaderTests.cs - VERIFIED — "
BlockingPerMinutebaseline over five Tuesdays: 10 samples of mean 1.0 (old SQL: 15 samples)" —TheBlockingPerMinuteBaseline_LeavesSelfWaitsOutinLite.Tests/EventBaselineCoveredDaysTests.csassertsSampleCount=10, Mean=1.0; the old unfiltered SQL would have bucketed all 3 non-empty minutes/day × 5 days = 15 samples, consistent with the claimed delta - UNVERIFIED — the cited local test run counts (247 + 22 tests, 0 failed) — a live build/test run, not checkable from a read-only tree
Riskiest lines
Lite/Services/LocalDataService.Blocking.cs:1217,1223—GetBlockingTrendAsync's bpr/dmv arms now both filter self-waits; a mistake here would silently drop real blocking from the Overview lane and MCP trend tool — covered byBlockingSeries_LeaveSelfWaitsOut_OnTheXeArmandBlockingSeries_OnlySelfWaitReportsInXe_DrawTheDmvArm_AndDmvSelfWaitsGoLite/Analysis/BaselineProvider.cs:778—BlockingPerMinutebaseline predicate change feeds the anomaly-detection ratio directly — covered byTheBlockingPerMinuteBaseline_LeavesSelfWaitsOutLite/Services/LocalDataService.Blocking.cs:1004,1016—GetBlockingSlicerDataAsync's dmv arm filters onv_dmv_blocking_snapshotsusing the sameblocking_spid/blocked_spidcolumns already relied on elsewhere in the file (e.g. line 55-56'sFILTERclauses) — covered byBlockingSeries_OnlySelfWaitReportsInXe_DrawTheDmvArm_AndDmvSelfWaitsGo(asserts slicer sum)
Traps near this diff
- "Lite and Darling ship in parity (CONTRIBUTING's two-store rule): a fix to one SKU's reader, tool, alert or description without the counterpart's is a finding" — does not apply: this commit is the Lite-side counterpart that brings Lite into parity with Darling's already-filtered baseline (
Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs:809, from the earlier #6690/#6663 work); nothing on the Darling side needs a matching change here. - "Our wrong test is a bug we introduced, never a flake" / "A test that encodes the lie" — does not apply: the new tests seed real rows through the service and assert on actual query results (
Lite.Tests/BlockingStatsReaderTests.cs,Lite.Tests/EventBaselineCoveredDaysTests.cs), not source-text or spelling assertions.
Divergence (blind read vs PR body)
None — the "Round 1 fixes (Lite)" section of the PR body matches the diff on every changed site, every left-alone site, and every described test outcome.
Reviewer checklist
- Confirm the cited local test run (
*BlockingStats*,*BaselineProvider*,*AnomalyDetector*, etc., 247 + 22 tests) actually passed, and that the full suite /pre-push.shpass in CI
…lt blocking baseline records where its supply begins (#6705) The Viewer and MCP blocking trend and duration reads, and the existence check that picks their arm, take BlockingWaits.NotSelfWaitSql on both arms. A rebuilt blocked_process_baseline records a supply floor (view comment) on the backfill pass, and the Blocking baseline starts its window no earlier, so days the aggregate cannot back are not counted as quiet.
…waits' into fix/6705-r1a
…d's report count, record the baseline supply floor when the view is created (#6705)
Claude posting for Erik Darling. Every round-1 item is fixed on the branch: the bars and band agree in both products, the supply floor, and the retention figures. The bot's round 2 (5637680663) covered the Lite commit; a separate review covered the Darling commits, and its findings are fixed in 44b8d01.
Claude posting for Erik Darling.
Fixes #6690.
Why
#6663 made both blocking families (Blocking, BlockingPerMinute) leave a parallel query's self-waits out of their current window. The baseline they are compared with,
collect.blocked_process_baseline, was stillcount(*)over every report row. On a server with many self-waits the baseline reads high and a real blocking rise can stay under the anomaly line. Lite already filters its baseline, so Lite and Darling disagreed on the same data.What changes
TimescaleSupport.CreateBlockedProcessBaselineSqlnow hasWHERE+BlockingWaits.NotSelfWaitSql(referenced, not copied). Doc comment updated.CreateBaselineFallbackViewSql), so a fresh plain store gets the filter with no further change. A plain view built earlier is present but stale:EnsureBaselineFallbackViewsAsyncnow replaces it in place (CREATE OR REPLACE VIEW; no data to lose) when it is a plain view (not a continuous aggregate, viaResolveMaterializationAsync) whose definition lacksblocking_spid. This is the ungated stage, so it runs on every store shape.DropStaleContinuousAggregatesAsyncgets ablocked_process_baselineentry: stale iff the aggregate exists and itstimescaledb_information.continuous_aggregates.view_definitionhas noblocking_spid. Same quiesce,DROP MATERIALIZED VIEW ... CASCADE, failure isolation as the other entries. No-op on a converged store and on a fresh store (no row matches). Nothing is built onblocked_process_baseline(onlyPgBaselineProviderreads it), so the CASCADE takes only the view; the ensure sweep recreates it filtered.PgAnomalyDetectorcomment that said the baseline "still counts them until a store migration" now says it does not.BaselineSupplyTestsline 110-116 assert flipped toContains(BlockingWaits.NotSelfWaitSql, CreateBlockedProcessBaselineSql); two live tests added toFleetPartialDatabaseFailureBandTests(it already holds the livecollection_health_hourlyreshape test).Why no migration (replaces the issue's plan)
No
SchemaVersionbump and no migration-queue wait. The startup reshape sweep is the proven mechanism for exactly this (an aggregate whose definition changed;collection_health_hourly#4812, the composer-dimension aggregates): a structural stale check, quiesce, drop, ensure recreates. It is idempotent and self-converging on every start, needs no version sentinel or rollback-list entry, and cannot be skipped by a store that migrated past it.Backfill (the trap) - already covered by the existing start-path backfill, so no new warm
With
materialized_only = falsethe recreated aggregate's watermark sits at the minimum timestamp, so right after the ensure it answers from raw; once the refresh policy refreshes its recent window, older days would read empty.DarlingWorkeralready launchesBackfillBaselineAggregatesAsync(#1757) off the start path on every start, after the ensure.blocked_process_baselineis inBaselineAggregates, and its probe reads coverage from the materialization hypertable (#3653), which a freshly recreated aggregate has empty, so the gate fires on that same start and refreshes frommax(raw oldest, now - BaselineRetentionSpan)(35 days). A second warm would repeat that work, so I added none. The live test below proves it: after the rebuild and a recent-window-only refresh the old day reads 0; afterBackfillBaselineAggregatesAsyncit reads the real report only.What a store loses (please read)
RawRetentionInterval(4 days) does not governblocked_process_reports: it covers query_stats, procedure_stats and query_store_stats only. Blocked process reports are purged at their collector retention,CollectorScheduleDefaults.All["blocked_process_report"]= 30 days (fleet-overridable), bydrop_chunks. The drop-and-rebuild re-materializes what raw still holds, and the baseline window is 30 days, so with the default retention no window day is lost (the aggregate's own retention is 35 days). An operator who setblocked_process_reportretention below 30 days gets an aggregate that begins at the oldest report the rebuild could keep, andcollection_log(kept twice the base) would count the days before that as quiet. The rebuild therefore records a supply floor and the Blocking baseline counts no day before it (see Round 1 fixes (Darling), item 2). This is the cost of the rebuild-in-place route; a migration that recreated the aggregate would have lost the same history, because the old buckets still contain self-waits and cannot be re-filtered from the aggregate itself.Plain PostgreSQL, Lite
PgBaselineProviderreads the relation by name on both store shapes (PgBaselineProvider.cslines 1424 and 1533), so no reader change.Lite/Analysis/BaselineProvider.cs:714-719, withAnomalyDetector.cs:805-807for the current window). No change.Test plan
Darling.Testswith 0 errors (build shows 15 warnings; I did not check whether they predate this branch).FleetPartialDatabaseFailureBandTests+BaselineSupplyTestson a throwaway TimescaleDB rig: 49 passed, 0 failed, including the two new live tests:BackfillBaselineAggregatesAsync-> both days count only the real report -> second sweep drops 0.EnsureBaselineFallbackViewsAsyncreplaces it (count 2 -> 1) -> next pass changes nothing.*Timescale*,*Baseline*,*Census*,*Ratchet*,*AggregateJobQuiesce*,*StoreObjectConvergence*,*LocalClockBucketKey*,*RetiredBaseline*,*AnomalyDetector*with the rig: 904 passed, 0 failed.pre-push.sh(laptop; CI decides). I did not revert the fix to watch the new tests fail on old code; they assert counts 2 vs 1 that the old definition cannot produce.Considered, not added
WarmCollectionHealthHourlyAsync: redundant with the existing coverage-gated baseline backfill (above).SchemaVersionmigration: replaced by the startup reshape (above).CHANGELOG
SECTION: Fixed
ENTRY:
get_blocking_trendand the blocking duration series still counted them, so a server with many of them read a high baseline, drew bars above its own band, and could hide a real rise in blocking. The Darling baseline aggregate, those series and the check that picks their blocked-process-report or DMV source now apply the same filter, in both products. On start, a Darling store that built the old aggregate rebuilds it from the blocked process reports it still keeps (30 days by default, so no baseline day is lost); a store that shortened blocked_process_report retention counts no baseline day before the oldest report the rebuild could keep. A plain-PostgreSQL store's baseline view is replaced in place.REF:
[Darling blocking baseline leaves out self-waits, like the window it is compared with (#6690) #6705]: Darling blocking baseline leaves out self-waits, like the window it is compared with (#6690) #6705
Round 1 fixes (Darling)
Claude posting for Erik Darling.
Answers the two round-1 bot reviews (5637563824, 5637567462). Lane A (Darling). Lite is the section below.
1. The bars and the band agree
Self-waits are not blocking (#6645). Every Darling per-minute blocking COUNT or DURATION series over the XE-preferred / DMV-fallback pair now takes
BlockingWaits.NotSelfWaitSqlon BOTH arms (referenced, never copied; the consts became interpolated raw strings), so the DMV arm is drawn when the filtered XE arm is empty, asPgAnomalyDetector.BlockingWindowSqldoes. Every "has XE reports in the window" check that tells which arm was drawn takes the same predicate.Changed:
ViewerDataService.BlockingTrends.csBlockingTrendSql(the bars under the Overview lane's band and the Blocking Trends chart).ViewerDataService.BlockingStats.csBlockingDurationStatsSql(the duration companion).ViewerDataService.Blocking.csBlockedProcessReportsInWindowSql(HasBlockedProcessReportsInWindowAsync, which also drivesGetBlockingChartDataStartAsync).Mcp/DarlingBlockingTrendReader.csBlockingTrendSql(get_blocking_trend, and the web Overview strip:overview-strips.jsreadsget_blocking_trendfor its bars).Mcp/DarlingDataReader.csBlockingDurationStatsSql(GetBlockingDurationStatsAsync).TimescaleSupport.csCreateBlockedProcessBaselineSql(it now says which series each family is compared with).PgAnomalyDetector.cs's comment already says the baseline and current window agree; left.Left, with why:
BlockedProcessReportsSql, pair rows, chain text, drill-down,PgFactCollectorchain read): they list reports, per the ruling.BlockingWaits.WaitsCteSqlalready excludes self-waits in its WAITS count:FleetTotalsSql, bothServerSummaryBlockingSql,FleetMasterUnscopedSql,DailySummarySql.RangeSql.BlockingChartXeStartSql: its SQL is unchanged on purpose (a self-wait report still proves the XE session was capturing); its doc is reworded below.HasAnyBlockingCaptureSql(ever captured),DarlingHealthReader/ViewerDataService.Overview"newest event" columns: not series.Slicer and doc (this push, lane
6705-slicer):ViewerDataService.BlockingSlicers.csBlockingSlicerSql(the Blocking-tab slicer's hourly overlay) now takesBlockingWaits.NotSelfWaitSqlon both arms, referenced (the const became an interpolated raw string), as Lite'sGetBlockingSlicerDataAsyncdoes. The DMV arm is drawn when the filtered XE arm has no buckets. I grepped Darling product code for other slicer or hourly series over the two views that pick XE over DMV: none; the other readers are per-minute series already covered above, single-report grids, scalar fleet/server counts built onWaitsCteSql, or "newest event" columns.BlockingChartXeStartSqldoc: it no longer says "the same predicate asBlockedProcessReportsInWindowSql"; it says a self-wait report counts here (it proves the XE session was capturing) and this read leaves out the filter that one takes, as Lite'sGetEarliestBlockedProcessReportInWindowAsyncdoc reads. SQL unchanged.BlockingSeries_LeaveSelfWaitsOut_OnTheXeArm: an hour with one self-wait (7000 ms), one real report (1000 ms) and one (0, 0) report (3000 ms) gives 2 events, 4.0 s, 2 blockers, 2 blocked (not 3 events / 11.0 s); a window whose XE rows are only self-waits draws the DMV arm (1 event, 9.999 s). With the slicer's predicate removed from both arms the test fails (reverted once, run, restored).ViewerBlockingDepthTeststext asserts pass unchanged.Four review findings (same push):
EnsureContinuousAggregatesAsyncnow reads whetherblocked_process_baselineexists before itsCREATE ... IF NOT EXISTSand, when it really creates it andcollect.blocked_process_reportshas a row, records the floor (same value: now minus the retention the purge uses; still write-once) before the refresh policy is attached. A service that stops between the reshape and the backfill, with the policy then materializing the last day, keeps its floor. A fresh store (raw empty) records nothing (EnsureOnAFreshStore_RecordsNoSupplyFloor_AgainstDevPostgres). The backfill-time write stays: it covers an existing empty materialization the create step did not make. The floor test now asserts the floor right after the ensure, runs a recent-day refresh before the backfill, and asserts the backfill leaves the floor alone.TryRecordBlockedBaselineSupplyFloorAsync, which catches (not cancellation), logs a warning and lets the refresh run; the next start retries the write. No test:COMMENT ON VIEWfails only for a role that does not own the view, and the live class connects as the owner, so forcing it needs a second role and grants for one catch block.DarlingFleetReader.FleetBlockingSql: thexe_reportsanddmv_reportsCTEs takeBlockingWaits.NotSelfWaitSql(referenced), soCOUNT(*)over them no longer counts a parallel query waiting on itself; the waits count already did throughWaitsCteSql. The earlier sentence thatFleetBlockingSqlalready left self-waits out was wrong for the report count and is corrected above. Lite's equivalent (the Overview and daily-summary report counts,reportCount: trueonWaitsCteSqlinLocalDataService.DailySummary.cs,LocalDataService.Overview.cs) readswait_reportsfrom the chain after itsNotSelfWaitSqlfilter, so it already leaves them out; no Lite change.InsertDmvBlockingSnapshotAsynctakesblockedSpid,blockingSpidandcollectionId; the live test inserts a DMV self-wait (spid 166 on itself, 5555 ms) next to the real DMV row (9999 ms) and asserts it is left out on the Viewer trend, Viewer duration, MCP trend, MCP duration and the slicer (1 event, 9999 ms).Test:
ViewerBlockingStatsLivePostgresTests.SelfWaitReports_AreNotBlocking_OnEveryTrendRead_AndOnlySelfWaitsDrawTheDmvArm_AgainstDevPostgres: a minute with one self-wait report and one real report returns 1 on the Viewer trend, the MCP trend, both duration reads (total 1000 ms, not 8000) and the existence check; with only a self-wait left in XE plus a DMV row, all four draw the DMV arm and the existence check is false. No existing test pinned the old SQL text.2. Measured zeros the rebuilt aggregate cannot back (#3590)
Found first:
drop_after) forblocked_process_baselineisBaselineRetentionInterval= "35 days" (TimescaleSupport.cs:570), attached by theRetentionPoliciesconcat overBaselineAggregates(:8036,TimeColumn: "bucket").drop_chunks(DarlingRetention.cs:546, which records no invalidation; its DELETE fallback after a faileddrop_chunkswould, but the refresh window is 1 day, so that does not reach back either), the baseline views take the hourly refresh policy whosestart_offsetis 1 day (HourlyRefreshStartOffset,:3674), so a refresh never reaches back to rematerialize an older region. So the rebuilt aggregate holds what raw held at the rebuild, then keeps it for 35 days.blocked_process_reportretention below 30 days (not inBaselineServingRawCollectors, so nothing floors it). With the default (30 days) the aggregate covers the whole window.What changed:
BackfillBaselineAggregatesAsync(TimescaleSupport.cs:916): on the pass that findsblocked_process_baseline's materialization EMPTY over a non-empty raw table (the rebuild's backfill, before the refresh),RecordBlockedBaselineSupplyFloorAsyncwritesCOMMENT ON VIEW collect.blocked_process_baseline IS 'supply_floor_utc=<ISO>'= now minus theblocked_process_reportretention the purge uses at that moment (DarlingRetentionHorizons.FleetRetentionDaysSql, the fleet override else the default; the store can learn it, so the oldest-raw fallback was not needed). No migration. A CAGG is arelkind='v'view, so it isCOMMENT ON VIEW, notON MATERIALIZED VIEW. Written once: a recorded floor is never moved by a later start, so a store whose aggregate stays empty (only self-waits in raw) does not creep forward; a rebuild drops the view and its comment.PgBaselineProvider.ComputeBucketsAsync(:906): forMetricNames.Blockingonly,$2(the window start) is raised to the floor when it is later.EventBaselineSql's text is unchanged (the byte-identical Lite twin test passes). Absent or unparsable comment is no floor (ParseBlockedBaselineSupplyFloor). Never bound by the aggregate'smin(bucket)or a per-server oldest event.Tests (
FleetPartialDatabaseFailureBandTests):TheSupplyFloorComment_ParsesOnlyAValidFloor(absent, empty, garbage, date only, valid);BlockingBaseline_OfARebuiltAggregate_CountsNoDayBeforeTheFloor_AndAStoreWithNoFloorReadsAsBefore_AgainstDevPostgres: blocked_process_report retention 10 days, reports 2 and 8 days back, a SUCCESS run at 14:00 on each of 30 days; after drop + ensure + backfill the floor isnow - 10 days(set by the ensure sweep, see A in this push, unchanged by the backfill and a second backfill), the Blocking baseline's covered days (sum ofsample_count) equal the logged days at or after the floor (9 or 10, not 29) and events / covered days gives the two events; with the comment cleared it reads all 29 logged days, as before. I did not revert the code to watch the floor assertions fail (the new tests assert a floor that did not exist and counts 29 vs 9-10); the self-wait test's old-code result is 2 not 1.3. Retention numbers
RawRetentionInterval(4 days) does not governblocked_process_reports(RawTierCoveragelists query_stats, procedure_stats, query_store_stats only). Corrected above in "What a store loses" and in the CHANGELOG entry. No code comment this PR added mentions 4 days or 31 days. Real effect: with the default, no baseline day is lost; with a shorterblocked_process_reportretention the item 2 floor applies.4. CHANGELOG
Rewritten below to cover the trend charts, Overview lane,
get_blocking_trendand the duration series in Darling and Lite.Considered, not added
(c) The same bias on the plain-PostgreSQL fallback view over raw (
IsStalePlainBlockedBaselineViewAsyncpath, a store without TimescaleDB): its event source is raw, so withblocked_process_reportretention R below 30 days thecollection_logruns older than R read as zeros. Magnitude: R = 7 reads about 23 of 30 covered days as fabricated quiet days, a mean about 77% low. It needs a now-based floor (now minus R) plus a plain-view test in the provider (ResolveMaterializationAsyncnull) andDateTime.UtcNowinside the provider. That breaksDarlingEventBaselineCoveredDaysLiveTests, which seeds March 2026 data under a plain view and analyses at a fixed March time (a now-based floor would empty its window), so it needs a provider clock and those tests re-timed: more than the 30 lines allowed. Not a regression from this PR (the plain view read raw before it too); no aggregate case exists (every baseline aggregate keeps 35 days, above the window).A fallback that records the oldest raw
collection_timewhen the retention cannot be learned: the store always learns it, so no second path.A try/catch around the floor read: no concrete failure (the view comment and the schedules table exist on every migrated store); a failed read reaches the provider's existing classified catch, as the other reads there do.
Lite's BlockingPerMinute baseline arm (
BaselineProvider.cs:778): lane B's.A floor later than the window end: the only trigger is a service clock that ran ahead when the floor was written, and the effect is no Blocking baseline until real time catches up.
Tests run (Darling, local rig, TimescaleDB)
6705-slicer, throwaway rig):ViewerBlockingStats*,ViewerBlockingDepthTests,FleetPartialDatabaseFailureBandTests,DatabaseFilterTests,EventTimeWindowSqlShapeTests,DarlingFleetReader*: Total 181, Failed 0, Skipped 0. Also*Census*,*Ratchet*,ViewerBlockingDepthTests,DarlingFleetReader*,*Timescale*,*Baseline*without a database: Total 893, Failed 0, Skipped 64 (the live ones, run against the rig above). Not run:pre-push.sh, the full suite (CI decides). Build: 0 errors; the warnings are in files this push does not touch (MainWindow.xaml.cs).FleetPartialDatabaseFailureBandTests,ViewerBlockingStatsLivePostgresTests,BlockingDatabaseFilterLiveTests,DarlingEventBaselineCoveredDays*,BaselineSupplyTests: Total 80, Failed 0, Skipped 0.DarlingMcpBlockingToolsTests,DatabaseFilterTests,EventWindowedReads*,ViewerBlockingSourceLabelTests,ViewerFileIoBlockingTests,WaitsDatabaseFilterLiveTests,ViewerBlockingDepth*,*EventBaseline*,*Timescale*,*Baseline*,*Census*,*Ratchet*,ViewerWave*,ViewerW2a*,EventTimeWindowSqlShape*: Total 1122, Failed 0, Skipped 0.pre-push.sh, the full suite; CI decides. Build: 0 errors; the warnings are in files this change does not touch (CustomViewValidation.cs,MainWindow.xaml.cs,InvestigationDispatcher.cs,StoreConnectionStringAppendPinTests.cs).Round 1 fixes (Lite)
Claude posting for Erik Darling.
Self-waits are not blocking (#6645), so every Lite per-minute (or per-hour) blocking count or duration series that reads the XE-preferred / DMV-fallback pair now takes
BlockingWaits.NotSelfWaitSqlon both arms, asAnomalyDetectoralready did. The DMV arm is drawn when the filtered XE arm is empty. Grids that list single reports are unchanged.Sites changed:
LocalDataService.Blocking.csGetBlockingTrendAsync(Overview lane, Locking tab trend, MCPget_blocking_trend): both arms.LocalDataService.Blocking.csHasBlockedProcessReportsInWindowAsync: same predicate, so it still tells which arm was drawn.LocalDataService.BlockingStats.csGetBlockingDurationStatsAsync: both arms.LocalDataService.Blocking.csGetBlockingSlicerDataAsync(the hourly slicer over the same pair, event count and wait seconds): both arms. Not in the brief's list; same pair and same count/duration shape, so left as is it would disagree with the trend bars.BaselineProvider.csBlockingPerMinute: predicate inside theBlockedProcessReports(...)filter. The comment at theBlockingarm now says the Darling aggregate leaves self-waits out too.Left, with why:
GetEarliestBlockedProcessReportInWindowAsync: it names when the XE session began capturing, so a self-wait report is evidence of capture. Doc reworded.HasAnyBlockingCaptureAsync,GetBlockingCaptureCountsAsync: existence and collector-run checks, not series.DrillDownCollector.Blocking.cs,LiteInvestigationEvidenceCaptureBlockedReportsSql,BlockingPairRowQuery): list single reports.BlockingWaits.WaitsCteSql(Overview.cs, DailySummary.cs,GetBlockingWaits,DuckDbFactCollector.Waits.cs): the CTE already leaves self-waits out.WhoDroveIt.cs,QueryTag*: pair and tag listings, not per-minute series.Tests (
BlockingStatsReaderTests,EventBaselineCoveredDaysTests; behaviour tests over DuckDB, all three fail on the old SQL and pass now):HasBlockedProcessReportsInWindowAsyncis true.HasBlockedProcessReportsInWindowAsyncis false.BlockingPerMinutebaseline over five Tuesdays: 10 samples of mean 1.0 (old SQL: 15 samples). A self-wait-only minute is not a sample and (0, 0) still counts.*BlockingStats*,*BaselineProvider*,*AnomalyDetector*,*Census*,*Ratchet*,*BlockingChartsDataStart*,*OverviewEventLane*,*StoredEventCopies*and others that name the changed readers: 247 + 22 tests, 0 failed. No live SQL Server classes, no full suite (CI decides).For the CHANGELOG entry (lane A owns the text): Lite's blocking trend, blocking duration chart, slicer and the blocking-per-minute baseline now leave self-waits out too, so Lite and Darling agree.