Skip to content

Darling blocking baseline leaves out self-waits, like the window it is compared with (#6690) - #6705

Merged
erikdarlingdata merged 8 commits into
devfrom
fix/6690-blocking-baseline-self-waits
Oct 24, 2026
Merged

erikdarlingdata merged 8 commits into
devfrom
fix/6690-blocking-baseline-self-waits

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Claude posting for Erik Darling.

Fixes #6690.

Why

#6663 made both blocking families (Blocking, BlockingPerMinute) leave a parallel query's self-waits out of their current window. The baseline they are compared with, collect.blocked_process_baseline, was still count(*) over every report row. On a server with many self-waits the baseline reads high and a real blocking rise can stay under the anomaly line. Lite already filters its baseline, so Lite and Darling disagreed on the same data.

What changes

  • TimescaleSupport.CreateBlockedProcessBaselineSql now has WHERE + BlockingWaits.NotSelfWaitSql (referenced, not copied). Doc comment updated.
  • The plain-PostgreSQL fallback view is derived from the same select body (CreateBaselineFallbackViewSql), so a fresh plain store gets the filter with no further change. A plain view built earlier is present but stale: EnsureBaselineFallbackViewsAsync now replaces it in place (CREATE OR REPLACE VIEW; no data to lose) when it is a plain view (not a continuous aggregate, via ResolveMaterializationAsync) whose definition lacks blocking_spid. This is the ungated stage, so it runs on every store shape.
  • DropStaleContinuousAggregatesAsync gets a blocked_process_baseline entry: stale iff the aggregate exists and its timescaledb_information.continuous_aggregates.view_definition has no blocking_spid. Same quiesce, DROP MATERIALIZED VIEW ... CASCADE, failure isolation as the other entries. No-op on a converged store and on a fresh store (no row matches). Nothing is built on blocked_process_baseline (only PgBaselineProvider reads it), so the CASCADE takes only the view; the ensure sweep recreates it filtered.
  • PgAnomalyDetector comment that said the baseline "still counts them until a store migration" now says it does not.
  • Tests: BaselineSupplyTests line 110-116 assert flipped to Contains(BlockingWaits.NotSelfWaitSql, CreateBlockedProcessBaselineSql); two live tests added to FleetPartialDatabaseFailureBandTests (it already holds the live collection_health_hourly reshape test).

Why no migration (replaces the issue's plan)

No SchemaVersion bump and no migration-queue wait. The startup reshape sweep is the proven mechanism for exactly this (an aggregate whose definition changed; collection_health_hourly #4812, the composer-dimension aggregates): a structural stale check, quiesce, drop, ensure recreates. It is idempotent and self-converging on every start, needs no version sentinel or rollback-list entry, and cannot be skipped by a store that migrated past it.

Backfill (the trap) - already covered by the existing start-path backfill, so no new warm

With materialized_only = false the recreated aggregate's watermark sits at the minimum timestamp, so right after the ensure it answers from raw; once the refresh policy refreshes its recent window, older days would read empty. DarlingWorker already launches BackfillBaselineAggregatesAsync (#1757) off the start path on every start, after the ensure. blocked_process_baseline is in BaselineAggregates, and its probe reads coverage from the materialization hypertable (#3653), which a freshly recreated aggregate has empty, so the gate fires on that same start and refreshes from max(raw oldest, now - BaselineRetentionSpan) (35 days). A second warm would repeat that work, so I added none. The live test below proves it: after the rebuild and a recent-window-only refresh the old day reads 0; after BackfillBaselineAggregatesAsync it reads the real report only.

What a store loses (please read)

RawRetentionInterval (4 days) does not govern blocked_process_reports: it covers query_stats, procedure_stats and query_store_stats only. Blocked process reports are purged at their collector retention, CollectorScheduleDefaults.All["blocked_process_report"] = 30 days (fleet-overridable), by drop_chunks. The drop-and-rebuild re-materializes what raw still holds, and the baseline window is 30 days, so with the default retention no window day is lost (the aggregate's own retention is 35 days). An operator who set blocked_process_report retention below 30 days gets an aggregate that begins at the oldest report the rebuild could keep, and collection_log (kept twice the base) would count the days before that as quiet. The rebuild therefore records a supply floor and the Blocking baseline counts no day before it (see Round 1 fixes (Darling), item 2). This is the cost of the rebuild-in-place route; a migration that recreated the aggregate would have lost the same history, because the old buckets still contain self-waits and cannot be re-filtered from the aggregate itself.

Plain PostgreSQL, Lite

  • Plain PG: covered above (derived fallback view for fresh stores, in-place replacement for stale ones). PgBaselineProvider reads the relation by name on both store shapes (PgBaselineProvider.cs lines 1424 and 1533), so no reader change.
  • Lite: its baseline query already filters (Lite/Analysis/BaselineProvider.cs:714-719, with AnomalyDetector.cs:805-807 for the current window). No change.

Test plan

  • Built Darling.Tests with 0 errors (build shows 15 warnings; I did not check whether they predate this branch).
  • FleetPartialDatabaseFailureBandTests + BaselineSupplyTests on a throwaway TimescaleDB rig: 49 passed, 0 failed, including the two new live tests:
    • old no-WHERE aggregate with one self-wait and one real report on a recent day and on a day 10 days old -> sweep drops exactly 1 -> ensure -> recent-window refresh leaves the old day at 0 -> BackfillBaselineAggregatesAsync -> both days count only the real report -> second sweep drops 0.
    • old plain fallback view -> EnsureBaselineFallbackViewsAsync replaces it (count 2 -> 1) -> next pass changes nothing.
  • *Timescale*, *Baseline*, *Census*, *Ratchet*, *AggregateJobQuiesce*, *StoreObjectConvergence*, *LocalClockBucketKey*, *RetiredBaseline*, *AnomalyDetector* with the rig: 904 passed, 0 failed.
  • Not run: full suite, pre-push.sh (laptop; CI decides). I did not revert the fix to watch the new tests fail on old code; they assert counts 2 vs 1 that the old definition cannot produce.

Considered, not added

  • A separate one-time warm refresh like WarmCollectionHealthHourlyAsync: redundant with the existing coverage-gated baseline backfill (above).
  • A SchemaVersion migration: replaced by the startup reshape (above).

CHANGELOG

SECTION: Fixed
ENTRY:

Round 1 fixes (Darling)

Claude posting for Erik Darling.

Answers the two round-1 bot reviews (5637563824, 5637567462). Lane A (Darling). Lite is the section below.

1. The bars and the band agree

Self-waits are not blocking (#6645). Every Darling per-minute blocking COUNT or DURATION series over the XE-preferred / DMV-fallback pair now takes BlockingWaits.NotSelfWaitSql on BOTH arms (referenced, never copied; the consts became interpolated raw strings), so the DMV arm is drawn when the filtered XE arm is empty, as PgAnomalyDetector.BlockingWindowSql does. Every "has XE reports in the window" check that tells which arm was drawn takes the same predicate.

Changed:

  • ViewerDataService.BlockingTrends.cs BlockingTrendSql (the bars under the Overview lane's band and the Blocking Trends chart).
  • ViewerDataService.BlockingStats.cs BlockingDurationStatsSql (the duration companion).
  • ViewerDataService.Blocking.cs BlockedProcessReportsInWindowSql (HasBlockedProcessReportsInWindowAsync, which also drives GetBlockingChartDataStartAsync).
  • Mcp/DarlingBlockingTrendReader.cs BlockingTrendSql (get_blocking_trend, and the web Overview strip: overview-strips.js reads get_blocking_trend for its bars).
  • Mcp/DarlingDataReader.cs BlockingDurationStatsSql (GetBlockingDurationStatsAsync).
  • Doc comment at TimescaleSupport.cs CreateBlockedProcessBaselineSql (it now says which series each family is compared with). PgAnomalyDetector.cs's comment already says the baseline and current window agree; left.

Left, with why:

  • Single-report grids (BlockedProcessReportsSql, pair rows, chain text, drill-down, PgFactCollector chain read): they list reports, per the ruling.
  • Everything built on BlockingWaits.WaitsCteSql already excludes self-waits in its WAITS count: FleetTotalsSql, both ServerSummaryBlockingSql, FleetMasterUnscopedSql, DailySummarySql.RangeSql.
  • BlockingChartXeStartSql: its SQL is unchanged on purpose (a self-wait report still proves the XE session was capturing); its doc is reworded below.
  • HasAnyBlockingCaptureSql (ever captured), DarlingHealthReader/ViewerDataService.Overview "newest event" columns: not series.

Slicer and doc (this push, lane 6705-slicer):

  • ViewerDataService.BlockingSlicers.cs BlockingSlicerSql (the Blocking-tab slicer's hourly overlay) now takes BlockingWaits.NotSelfWaitSql on both arms, referenced (the const became an interpolated raw string), as Lite's GetBlockingSlicerDataAsync does. The DMV arm is drawn when the filtered XE arm has no buckets. I grepped Darling product code for other slicer or hourly series over the two views that pick XE over DMV: none; the other readers are per-minute series already covered above, single-report grids, scalar fleet/server counts built on WaitsCteSql, or "newest event" columns.
  • BlockingChartXeStartSql doc: it no longer says "the same predicate as BlockedProcessReportsInWindowSql"; it says a self-wait report counts here (it proves the XE session was capturing) and this read leaves out the filter that one takes, as Lite's GetEarliestBlockedProcessReportInWindowAsync doc reads. SQL unchanged.
  • Test: the live self-wait test now also covers the slicer, like Lite's BlockingSeries_LeaveSelfWaitsOut_OnTheXeArm: an hour with one self-wait (7000 ms), one real report (1000 ms) and one (0, 0) report (3000 ms) gives 2 events, 4.0 s, 2 blockers, 2 blocked (not 3 events / 11.0 s); a window whose XE rows are only self-waits draws the DMV arm (1 event, 9.999 s). With the slicer's predicate removed from both arms the test fails (reverted once, run, restored). ViewerBlockingDepthTests text asserts pass unchanged.

Four review findings (same push):

  • A. The supply floor is written when the view is created. EnsureContinuousAggregatesAsync now reads whether blocked_process_baseline exists before its CREATE ... IF NOT EXISTS and, when it really creates it and collect.blocked_process_reports has a row, records the floor (same value: now minus the retention the purge uses; still write-once) before the refresh policy is attached. A service that stops between the reshape and the backfill, with the policy then materializing the last day, keeps its floor. A fresh store (raw empty) records nothing (EnsureOnAFreshStore_RecordsNoSupplyFloor_AgainstDevPostgres). The backfill-time write stays: it covers an existing empty materialization the create step did not make. The floor test now asserts the floor right after the ensure, runs a recent-day refresh before the backfill, and asserts the backfill leaves the floor alone.
  • B. A failed floor write no longer blocks the backfill. Both call sites go through TryRecordBlockedBaselineSupplyFloorAsync, which catches (not cancellation), logs a warning and lets the refresh run; the next start retries the write. No test: COMMENT ON VIEW fails only for a role that does not own the view, and the live class connects as the owner, so forcing it needs a second role and grants for one catch block.
  • C. The fleet band's report count leaves self-waits out. DarlingFleetReader.FleetBlockingSql: the xe_reports and dmv_reports CTEs take BlockingWaits.NotSelfWaitSql (referenced), so COUNT(*) over them no longer counts a parallel query waiting on itself; the waits count already did through WaitsCteSql. The earlier sentence that FleetBlockingSql already left self-waits out was wrong for the report count and is corrected above. Lite's equivalent (the Overview and daily-summary report counts, reportCount: true on WaitsCteSql in LocalDataService.DailySummary.cs, LocalDataService.Overview.cs) reads wait_reports from the chain after its NotSelfWaitSql filter, so it already leaves them out; no Lite change.
  • D. The DMV-arm assertions meet a DMV self-wait. InsertDmvBlockingSnapshotAsync takes blockedSpid, blockingSpid and collectionId; the live test inserts a DMV self-wait (spid 166 on itself, 5555 ms) next to the real DMV row (9999 ms) and asserts it is left out on the Viewer trend, Viewer duration, MCP trend, MCP duration and the slicer (1 event, 9999 ms).

Test: ViewerBlockingStatsLivePostgresTests.SelfWaitReports_AreNotBlocking_OnEveryTrendRead_AndOnlySelfWaitsDrawTheDmvArm_AgainstDevPostgres: a minute with one self-wait report and one real report returns 1 on the Viewer trend, the MCP trend, both duration reads (total 1000 ms, not 8000) and the existence check; with only a self-wait left in XE plus a DMV row, all four draw the DMV arm and the existence check is false. No existing test pinned the old SQL text.

2. Measured zeros the rebuilt aggregate cannot back (#3590)

Found first:

  • The aggregate's own retention (drop_after) for blocked_process_baseline is BaselineRetentionInterval = "35 days" (TimescaleSupport.cs:570), attached by the RetentionPolicies concat over BaselineAggregates (:8036, TimeColumn: "bucket").
  • Nothing removes a materialized row once its raw rows are gone: the raw purge is drop_chunks (DarlingRetention.cs:546, which records no invalidation; its DELETE fallback after a failed drop_chunks would, but the refresh window is 1 day, so that does not reach back either), the baseline views take the hourly refresh policy whose start_offset is 1 day (HourlyRefreshStartOffset, :3674), so a refresh never reaches back to rematerialize an older region. So the rebuilt aggregate holds what raw held at the rebuild, then keeps it for 35 days.
  • Real trigger: an operator who set blocked_process_report retention below 30 days (not in BaselineServingRawCollectors, so nothing floors it). With the default (30 days) the aggregate covers the whole window.

What changed:

  • a. Record. BackfillBaselineAggregatesAsync (TimescaleSupport.cs:916): on the pass that finds blocked_process_baseline's materialization EMPTY over a non-empty raw table (the rebuild's backfill, before the refresh), RecordBlockedBaselineSupplyFloorAsync writes COMMENT ON VIEW collect.blocked_process_baseline IS 'supply_floor_utc=<ISO>' = now minus the blocked_process_report retention the purge uses at that moment (DarlingRetentionHorizons.FleetRetentionDaysSql, the fleet override else the default; the store can learn it, so the oldest-raw fallback was not needed). No migration. A CAGG is a relkind='v' view, so it is COMMENT ON VIEW, not ON MATERIALIZED VIEW. Written once: a recorded floor is never moved by a later start, so a store whose aggregate stays empty (only self-waits in raw) does not creep forward; a rebuild drops the view and its comment.
  • b. Use. PgBaselineProvider.ComputeBucketsAsync (:906): for MetricNames.Blocking only, $2 (the window start) is raised to the floor when it is later. EventBaselineSql's text is unchanged (the byte-identical Lite twin test passes). Absent or unparsable comment is no floor (ParseBlockedBaselineSupplyFloor). Never bound by the aggregate's min(bucket) or a per-server oldest event.
  • c. Not added (see below).

Tests (FleetPartialDatabaseFailureBandTests): TheSupplyFloorComment_ParsesOnlyAValidFloor (absent, empty, garbage, date only, valid); BlockingBaseline_OfARebuiltAggregate_CountsNoDayBeforeTheFloor_AndAStoreWithNoFloorReadsAsBefore_AgainstDevPostgres: blocked_process_report retention 10 days, reports 2 and 8 days back, a SUCCESS run at 14:00 on each of 30 days; after drop + ensure + backfill the floor is now - 10 days (set by the ensure sweep, see A in this push, unchanged by the backfill and a second backfill), the Blocking baseline's covered days (sum of sample_count) equal the logged days at or after the floor (9 or 10, not 29) and events / covered days gives the two events; with the comment cleared it reads all 29 logged days, as before. I did not revert the code to watch the floor assertions fail (the new tests assert a floor that did not exist and counts 29 vs 9-10); the self-wait test's old-code result is 2 not 1.

3. Retention numbers

RawRetentionInterval (4 days) does not govern blocked_process_reports (RawTierCoverage lists query_stats, procedure_stats, query_store_stats only). Corrected above in "What a store loses" and in the CHANGELOG entry. No code comment this PR added mentions 4 days or 31 days. Real effect: with the default, no baseline day is lost; with a shorter blocked_process_report retention the item 2 floor applies.

4. CHANGELOG

Rewritten below to cover the trend charts, Overview lane, get_blocking_trend and the duration series in Darling and Lite.

Considered, not added

  • (c) The same bias on the plain-PostgreSQL fallback view over raw (IsStalePlainBlockedBaselineViewAsync path, a store without TimescaleDB): its event source is raw, so with blocked_process_report retention R below 30 days the collection_log runs older than R read as zeros. Magnitude: R = 7 reads about 23 of 30 covered days as fabricated quiet days, a mean about 77% low. It needs a now-based floor (now minus R) plus a plain-view test in the provider (ResolveMaterializationAsync null) and DateTime.UtcNow inside the provider. That breaks DarlingEventBaselineCoveredDaysLiveTests, which seeds March 2026 data under a plain view and analyses at a fixed March time (a now-based floor would empty its window), so it needs a provider clock and those tests re-timed: more than the 30 lines allowed. Not a regression from this PR (the plain view read raw before it too); no aggregate case exists (every baseline aggregate keeps 35 days, above the window).

  • A fallback that records the oldest raw collection_time when the retention cannot be learned: the store always learns it, so no second path.

  • A try/catch around the floor read: no concrete failure (the view comment and the schedules table exist on every migrated store); a failed read reaches the provider's existing classified catch, as the other reads there do.

  • Lite's BlockingPerMinute baseline arm (BaselineProvider.cs:778): lane B's.

  • A floor later than the window end: the only trigger is a service clock that ran ahead when the floor was written, and the effect is no Blocking baseline until real time catches up.

Tests run (Darling, local rig, TimescaleDB)

  • This push (lane 6705-slicer, throwaway rig): ViewerBlockingStats*, ViewerBlockingDepthTests, FleetPartialDatabaseFailureBandTests, DatabaseFilterTests, EventTimeWindowSqlShapeTests, DarlingFleetReader*: Total 181, Failed 0, Skipped 0. Also *Census*, *Ratchet*, ViewerBlockingDepthTests, DarlingFleetReader*, *Timescale*, *Baseline* without a database: Total 893, Failed 0, Skipped 64 (the live ones, run against the rig above). Not run: pre-push.sh, the full suite (CI decides). Build: 0 errors; the warnings are in files this push does not touch (MainWindow.xaml.cs).
  • FleetPartialDatabaseFailureBandTests, ViewerBlockingStatsLivePostgresTests, BlockingDatabaseFilterLiveTests, DarlingEventBaselineCoveredDays*, BaselineSupplyTests: Total 80, Failed 0, Skipped 0.
  • Wider: DarlingMcpBlockingToolsTests, DatabaseFilterTests, EventWindowedReads*, ViewerBlockingSourceLabelTests, ViewerFileIoBlockingTests, WaitsDatabaseFilterLiveTests, ViewerBlockingDepth*, *EventBaseline*, *Timescale*, *Baseline*, *Census*, *Ratchet*, ViewerWave*, ViewerW2a*, EventTimeWindowSqlShape*: Total 1122, Failed 0, Skipped 0.
  • Not run (as briefed): pre-push.sh, the full suite; CI decides. Build: 0 errors; the warnings are in files this change does not touch (CustomViewValidation.cs, MainWindow.xaml.cs, InvestigationDispatcher.cs, StoreConnectionStringAppendPinTests.cs).

Round 1 fixes (Lite)

Claude posting for Erik Darling.

Self-waits are not blocking (#6645), so every Lite per-minute (or per-hour) blocking count or duration series that reads the XE-preferred / DMV-fallback pair now takes BlockingWaits.NotSelfWaitSql on both arms, as AnomalyDetector already did. The DMV arm is drawn when the filtered XE arm is empty. Grids that list single reports are unchanged.

Sites changed:

  • LocalDataService.Blocking.cs GetBlockingTrendAsync (Overview lane, Locking tab trend, MCP get_blocking_trend): both arms.
  • LocalDataService.Blocking.cs HasBlockedProcessReportsInWindowAsync: same predicate, so it still tells which arm was drawn.
  • LocalDataService.BlockingStats.cs GetBlockingDurationStatsAsync: both arms.
  • LocalDataService.Blocking.cs GetBlockingSlicerDataAsync (the hourly slicer over the same pair, event count and wait seconds): both arms. Not in the brief's list; same pair and same count/duration shape, so left as is it would disagree with the trend bars.
  • BaselineProvider.cs BlockingPerMinute: predicate inside the BlockedProcessReports(...) filter. The comment at the Blocking arm now says the Darling aggregate leaves self-waits out too.

Left, with why:

  • GetEarliestBlockedProcessReportInWindowAsync: it names when the XE session began capturing, so a self-wait report is evidence of capture. Doc reworded.
  • HasAnyBlockingCaptureAsync, GetBlockingCaptureCountsAsync: existence and collector-run checks, not series.
  • Top-5 and report grids (DrillDownCollector.Blocking.cs, LiteInvestigationEvidenceCapture BlockedReportsSql, BlockingPairRowQuery): list single reports.
  • Sites already through BlockingWaits.WaitsCteSql (Overview.cs, DailySummary.cs, GetBlockingWaits, DuckDbFactCollector.Waits.cs): the CTE already leaves self-waits out.
  • WhoDroveIt.cs, QueryTag*: pair and tag listings, not per-minute series.

Tests (BlockingStatsReaderTests, EventBaselineCoveredDaysTests; behaviour tests over DuckDB, all three fail on the old SQL and pass now):

  • A minute with a self-wait, a real report and a (0, 0) report counts 2 in the trend, duration stats (total 4000, max 3000) and slicer; HasBlockedProcessReportsInWindowAsync is true.
  • A window whose XE rows are only self-waits draws the DMV arm in the trend, duration stats and slicer, the DMV self-wait is left out, and HasBlockedProcessReportsInWindowAsync is false.
  • The BlockingPerMinute baseline over five Tuesdays: 10 samples of mean 1.0 (old SQL: 15 samples). A self-wait-only minute is not a sample and (0, 0) still counts.
  • Ran: those classes plus *BlockingStats*, *BaselineProvider*, *AnomalyDetector*, *Census*, *Ratchet*, *BlockingChartsDataStart*, *OverviewEventLane*, *StoredEventCopies* and others that name the changed readers: 247 + 22 tests, 0 failed. No live SQL Server classes, no full suite (CI decides).

For the CHANGELOG entry (lane A owns the text): Lite's blocking trend, blocking duration chart, slicer and the blocking-per-minute baseline now leave self-waits out too, so Lite and Darling agree.

@erikdarlingdata
erikdarlingdata marked this pull request as ready for review October 24, 2026 08:28
collection_time,
count(*) AS event_count
FROM collect.blocked_process_reports
WHERE " + PerformanceMonitor.Common.BlockingWaits.NotSelfWaitSql + @"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This aggregate serves TWO families, and only one of them has a filtered current window. The doc comment just above says "both leave a parallel query's self-waits out of their current window (#6663), so the baseline they are compared with leaves them out too" — that holds for MetricNames.Blocking, whose current count is PgAnomalyDetector.BlockingWindowSql (lines 317-322, both arms carry NotSelfWaitSql). It does not hold for MetricNames.BlockingPerMinute.

BlockingPerMinute has no anomaly-detector arm. Its only consumers are the Overview blocking lane (CorrelatedTimelineLanesControl.xaml.cs:193 → UpdateBlockingLane, band = mean ± 2σ) and the web Overview strip (DarlingWebOverviewBaselines.cs:39), and the series that band is drawn over is ViewerDataService.BlockingTrends.cs:72 (BlockingTrendSql) — a plain COUNT(*) per minute over v_blocked_process_reports with no NotSelfWaitSql.

So after this change the band excludes self-waits while the bars still count them, which is the mirror image of the defect being fixed. Concretely: a server whose parallel query writes self-wait reports at, say, 6 of every 8 reports per minute gets a band around mean ≈ 2 while the lane keeps drawing bars at 8 — every ordinary minute now plots above the band, and UpdateBlockingLane passes no minAnomalyValue floor (unlike the CPU and I/O lanes), so nothing suppresses it. The anomaly line moves in the under-flagging direction for Blocking and the over-flagging direction for BlockingPerMinute.

Either BlockingTrendSql needs the same predicate, or BlockingPerMinute needs a supply that still counts every report row — but the two cannot disagree, and the comment should not claim they agree.

Related, same root: Lite is now out of parity for this family. Lite/Analysis/BaselineProvider.cs:778 (the BlockingPerMinute arm) is COUNT(*) over StoredEventCopies.BlockedProcessReports("server_id = $1 AND event_time >= $2 AND event_time < $3") with no NotSelfWaitSql — the PR body's "Lite: its baseline query already filters … No change" cites lines 714-719, which is the Blocking arm only. Lite's band and Lite's bars still agree with each other (both unfiltered); Darling's no longer do, and the same lane now bands differently in the two SKUs.

@claude

claude Bot commented Oct 24, 2026

Copy link
Copy Markdown

The "about 4 days" in the body and in the CHANGELOG is the wrong tier

Raw blocked_process_reports retention is 4 days (RawRetentionInterval); the aggregate's own retention is 35 days. […] a store loses its blocking baseline history older than about 4 days. The blocking baseline therefore starts from about 4 days of supply and refills one day per day over roughly the next 30 days.

RawRetentionInterval does not govern blocked_process_reports. The 4-day drop_chunks tier is built from RawTierCoverage (TimescaleSupport.cs:7824-7841), which has exactly three members — query_stats, procedure_stats, query_store_stats — and RetentionPolicies (TimescaleSupport.cs:7881-7883) attaches RawRetentionInterval only to those. The doc on the sweep says so in its own words: "The three raw tables drop at RawRetentionInterval" (TimescaleSupport.cs:8128). There is no retention policy on blocked_process_reports in that list at all.

Blocked-process reports are purged by the collector-retention path instead, at CollectorScheduleDefaults.All["blocked_process_report"] = new(1, 30) — 30 days, fleet-overridable through DailySummaryHorizon.ResolveFleetRetentionDays.

So the real cost of the drop-and-rebuild on a default store is about 5 days (the 35-day baseline tier minus the 30-day raw horizon), not 31. That is a much easier thing to accept, and it is worth stating correctly rather than conservatively, because the number is going out to users verbatim:

On start, a store that built the old aggregate rebuilds it and re-materializes it from the raw reports still kept (about 4 days), so the blocking baseline starts from about 4 days of history and fills back out to its full window over the following weeks.

A reader of that line concludes their blocking anomaly detection is effectively off for a month. On a default store it loses the oldest five days of a thirty-day window.

Two things that are worth keeping from that paragraph, restated against the right tier:

  • blocked_process_report is not in DarlingRetentionHorizons.BaselineServingRawCollectors (DailySummaryHorizon.cs:107-118), so unlike cpu_utilization / file_io_stats its raw retention is not floored at BaselineMath.BaselineWindowDays. An operator who shortened it to 7 days for disk reasons really does get a baseline rebuilt from 7 days — and nothing says so. That is the case the paragraph should name.
  • The BackfillBaselineAggregatesAsync clamp is GREATEST(min(collection_time), now − BaselineRetentionSpan) (BaselineBackfillProbeSql, TimescaleSupport.cs:1199-1205), which is what makes "re-materializes only what raw still holds" true in either case.

Not blocking on its own, but the CHANGELOG entry should be corrected before this merges.

Comment on lines +6246 to +6252
/* blocked_process_baseline gained the self-wait filter (#6690) -> stale iff the aggregate EXISTS but its
definition does not mention blocking_spid (the filter's column; the old text had no WHERE at all).
Nothing is built on it, so the CASCADE drops only the view; the ensure sweep recreates it WITH NO DATA
and the start path's coverage-gated BackfillBaselineAggregatesAsync (its materialization is empty, so
its gate fires on this very start) re-materializes it from raw. NULL view_definition is not stale. */
(View: BlockedProcessBaselineView,
StaleCheck: "SELECT EXISTS (SELECT 1 FROM timescaledb_information.continuous_aggregates WHERE view_schema = 'collect' AND view_name = 'blocked_process_baseline' AND position('blocking_spid' IN view_definition) = 0)"),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The history this drop forfeits does not read as missing, it reads as zero blocking. The body calls the cost "the hour/day-of-week buckets are thinner (the anomaly check's own minimum-sample rules apply)". They are not thinner — they are full, with fabricated quiet samples in them, which is the #3590 trap ("a retention-tiered store answers a 7-day question from a 4-day raw tier and labels the missing days quiet").

MetricNames.Blocking is the event family, PgBaselineProvider.EventBaselineSql (PgBaselineProvider.cs:1170-1204). Its slots CTE is logged (n = 0) UNION ALL events (n), and the mean is SUM(n) / COUNT(DISTINCT d) over that union:

  • events comes from blocked_process_baseline — after the drop-and-rebuild it reaches back only as far as raw blocked_process_reports does.
  • logged comes from collection_log, whose retention is DarlingRetentionHorizons.CollectionLogRetentionDays = DataRetentionBaseDays * 2 (DailySummaryHorizon.cs:46) — deliberately twice the base, "so it needs no floor of the kind the raw baseline sources have" (PgBaselineProvider.cs:1160-1162).

So for every day in the baseline window that the rebuilt aggregate no longer covers, collection_log still has the collector's SUCCESS runs, logged contributes that day with n = 0, and EXISTS (SELECT 1 FROM events) is satisfied by the recent days. Those days enter the mean as measured zeros and enter sample_count / distinct_days as full samples. The bucket does not look short to the detector; it looks like a server that reliably did not block on those days.

Net effect on a default store (~30 days raw vs the 35-day baseline tier, see my top-level note about the RawRetentionInterval figure): for roughly the first five days after the upgrade start, each (hour, dow) bucket carries about one fabricated zero out of about four samples, so the blocking mean reads ~25% low and real blocking is over-flagged — the opposite error from the one this PR fixes, for a bounded window. On a store whose operator shortened blocked_process_report retention it is worse and lasts longer: blocked_process_report is not in DarlingRetentionHorizons.BaselineServingRawCollectors (DailySummaryHorizon.cs:107-118), so nothing floors its raw horizon at BaselineMath.BaselineWindowDays the way cpu_utilization and file_io_stats are floored.

This is transient and self-healing, so I am not asking for the reshape to be abandoned. But it should be stated honestly, and there is a cheap mitigation worth considering: have the backfill (or a one-shot alongside it) record the rebuild instant, and have the Blocking arm not count logged days older than the aggregate's own min(bucket) — the same "say which tier served it" discipline RetentionTierRouter already applies elsewhere. At minimum the doc here and the CHANGELOG should say the lost days read as quiet rather than as absent.

claude[bot]
claude Bot previously requested changes Oct 24, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: CHANGES REQUESTED

The reshape mechanism is right, the predicate is aliased rather than copied, and the two live tests are real behaviour tests that run in PR CI. Two things block: the aggregate serves a SECOND family (BlockingPerMinute) whose current series is not filtered, so the band now reads low against its own bars; and Lite's half of that family is untouched. Details inline.

Claims

  • VERIFIED — CreateBlockedProcessBaselineSql carries WHERE + BlockingWaits.NotSelfWaitSql, referenced not copied — TimescaleSupport.cs:809
  • VERIFIED — the plain-PG fallback is derived from the same body, so a fresh plain store gets the filter — CreateBaselineFallbackViewSql slices between BaselineBodyMarker and the last WITH NO DATA, TimescaleSupport.cs:1026-1039
  • VERIFIED — EnsureBaselineFallbackViewsAsync replaces a stale PLAIN view only — TimescaleSupport.cs:1079-1091, probe at 1125-1136; a non-view returns no row from the relkind='v' filter, so the is true test is false
  • VERIFIED — "the ungated stage, so it runs on every store shape" — DarlingWorker.cs:444, with NeedsTimescale at DarlingWorker.cs:261
  • VERIFIED — the stale entry shares the others' quiesce / DROP ... CASCADE / isolation (TimescaleSupport.cs:6251-6252 in the loop at 6256-6312), and its step is Stage.Timescale (DarlingWorker.cs:392-393), so the timescaledb_information reference never runs on a plain-PG store and cannot throw 42P01 there
  • VERIFIED — "nothing is built on blocked_process_baseline" — only readers are PgBaselineProvider.cs:1424 and :1533; only membership is BaselineAggregates (TimescaleSupport.cs:2402)
  • VERIFIED — the backfill gate reads coverage off the materialization, empty on a fresh aggregate, so it fires that same start — TimescaleSupport.cs:876 + the doc at 1170-1183
  • VERIFIED — BackfillBaselineAggregatesAsync is launched off the start path after the ensure — DarlingWorker.cs:5312, list doc at 344-348, ensure at 420
  • VERIFIED — the PgAnomalyDetector comment no longer claims the baseline counts self-waits — PgAnomalyDetector.cs:310-312
  • VERIFIED — BaselineSupplyTests assertion flipped — BaselineSupplyTests.cs:110-116
  • VERIFIED — Lite's Blocking arm already filters — Lite/Analysis/BaselineProvider.cs:714-719
  • VERIFIED — the new live tests are not nightly-only: pr.yml's legs start the bundled cluster, start-test-pg.ps1:39 sets DARLING_TEST_PG and :95 preloads timescaledb, so the hard Assert.True(TryEnableAsync(...)) has an extension to find
  • UNVERIFIED — "no-op on a converged store" for the CAGG stale check: nothing else in the tree reads timescaledb_information.continuous_aggregates.view_definition, so whether it renders the WHERE cannot be decided from a read-only tree. The test's closing Assert.Equal(0, DropStale...) is that check; it needs the rig
  • UNVERIFIED — "0 errors, 15 warnings" — needs a build
  • UNVERIFIED — "49 passed / 0 failed" and "904 passed / 0 failed" — needs the rig
  • UNVERIFIED — "they assert counts 2 vs 1 that the old definition cannot produce", offered instead of reverting the fix: plausible from the test text, not demonstrated red on old code
  • REFUTED — "#6663 made both blocking families leave self-waits out of their current window": true for Blocking (PgAnomalyDetector.cs:317-322), false for BlockingPerMinute, whose only series is ViewerDataService.BlockingTrends.cs:72 (bare COUNT(*)) feeding CorrelatedTimelineLanesControl.xaml.cs:193 and DarlingWebOverviewBaselines.cs:39
  • REFUTED — "Lite: its baseline query already filters … No change": the cited lines are the Blocking arm; Lite/Analysis/BaselineProvider.cs:774-784 has no NotSelfWaitSql, so this opens a parity gap in that family
  • REFUTED — "Raw blocked_process_reports retention is 4 days (RawRetentionInterval)": RawTierCoverage has three members (TimescaleSupport.cs:7824-7841) and RetentionPolicies (7881-7883) arms that interval only on those; 8128 says "the three raw tables". Reports are purged at CollectorScheduleDefaults.All["blocked_process_report"] = new(1, 30) — 30 days. The forfeit is ~5 days of the 35-day tier, not 31, and the CHANGELOG ships the wrong number (top-level note)

Riskiest lines

  1. Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs:809 — one WHERE on a supply serving two families, only one with a filtered current window — covered for Blocking by BlockingBaselineBuiltWithoutTheSelfWaitFilter_IsRebuiltFilteredAndBackfilledOverTheWholeWindow_AgainstDevPostgres; UNCOVERED for BlockingPerMinute
  2. Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs:6252 — if view_definition never holds blocking_spid the aggregate is dropped and rebuilt every start and every hourly tick — covered by that test's closing Assert.Equal(0, ...DropStaleContinuousAggregatesAsync...)
  3. Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs:1133 — a probe that mis-read a CAGG as a plain view would CREATE OR REPLACE VIEW over a materialization — covered by PlainBlockingBaselineViewWithoutTheSelfWaitFilter_IsReplacedInPlace_AgainstDevPostgres
  4. Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs:6251 — forfeited days re-enter the Blocking mean as measured zeros via EventBaselineSql's logged CTE instead of dropping out of the sample — UNCOVERED

Traps near this diff

  • "Lite and Darling ship in parity (CONTRIBUTING's two-store rule): a fix to one SKU's reader, tool, alert or description without the counterpart's is a finding, not a nit" — applies — TimescaleSupport.cs:809 filtered, Lite/Analysis/BaselineProvider.cs:778 not
  • "A retention-tiered store answers a 7-day question from a 4-day raw tier and labels the missing days quiet unless the read routes by tier and says which tier served it (#3590; the viewer's charts in #3684)" — applies — the rebuilt aggregate's missing days still have collection_log SUCCESS rows (DailySummaryHorizon.cs:46), so EventBaselineSql's slots union (PgBaselineProvider.cs:1191-1196) enters them at n = 0 and counts them in sample_count
  • "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy." — does not apply — referenced at TimescaleSupport.cs:809, and the fallback derives from the same body at 1026-1039
  • "Ruling (#5833): tests are load-bearing … never answer a trap with a test that reads source text" — does not apply — BaselineSupplyTests.cs:116 edits a pin that already existed, and the new coverage is two behaviour tests (FleetPartialDatabaseFailureBandTests.cs:617, :706)
  • "timescaledb_information.hypertables never lists a continuous aggregate's materialization … Enumerate continuous_aggregates" — does not apply — both new reads already go through continuous_aggregates (TimescaleSupport.cs:6252, 1226)
  • "A first-level continuous aggregate's materialization is chunked at ten times the raw interval unless set_chunk_time_interval says otherwise (#3620 → #3621)" — does not apply — the recreated aggregate goes back through the convergence list, whose aggregate compression step carries #3620's chunk-width ensure (DarlingWorker.cs:430-431)

Divergence (blind read vs PR body)

Three places, all the same shape: the body reasons about the Blocking family and states the conclusion for both.

  1. The second family is never separated. The doc at TimescaleSupport.cs:794-799 says itself that this supply serves Blocking AND BlockingPerMinute. Blocking's current window is filtered; BlockingPerMinute has no anomaly arm at all — its band is drawn over BlockingTrendSql, which is not. "Both blocking families" is the premise of the change and is half true.
  2. Lite. "No change" is right for Blocking, wrong for BlockingPerMinute. The body visits :714-719 and not :774-784, which is the arm that must move for the two SKUs to band the same lane the same way.
  3. The retention tier. "What a store loses (please read)" — the section explicitly asking a reviewer to check it — is built on a tier that does not cover this table, and so is the CHANGELOG line. The correction makes the case stronger (five days, not thirty-one); what the body does not say is that the lost days read as zero blocking rather than as absent.

The parts I was least sure of from the diff alone — no migration needed, no new warm needed because the gate reads the empty materialization — both check out against the tree.

Reviewer checklist

  • On the rig: SELECT view_definition FROM timescaledb_information.continuous_aggregates WHERE view_name = 'blocked_process_baseline' after the ensure, and confirm blocking_spid is in it. If not, the sweep drop-and-rebuilds the blocking baseline on every start and every hourly tick.
  • On a store with real blocking history, after one start on this build: min(bucket) of the recreated aggregate vs sample_count for one (hour, dow) bucket of the 30-day Blocking baseline — the forfeited days should appear as samples with no events.
  • Open the Viewer Overview blocking lane and the web Overview blocking strip on a server that produces parallel-query self-wait reports: do the bars now sit above the band?
  • Confirm the recreated aggregate's retention policy is re-armed after the CASCADE drop — EnsureRetentionPoliciesAsync is start-path-only and its coverage gate will see a short tier.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: LGTM

Claims

  • VERIFIED — every XE-preferred/DMV-fallback series method (GetBlockingTrendAsync, GetBlockingDurationStatsAsync, GetBlockingSlicerDataAsync) now applies BlockingWaits.NotSelfWaitSql on both arms — Lite/Services/LocalDataService.Blocking.cs:1004,1016,1217,1223, Lite/Services/LocalDataService.BlockingStats.cs:99,111
  • VERIFIED — HasBlockedProcessReportsInWindowAsync takes the same predicate — Lite/Services/LocalDataService.Blocking.cs:1102
  • VERIFIED — BaselineProvider.cs's BlockingPerMinute arm filters self-waits, and the Blocking arm's comment now says the Darling aggregate filters too — Lite/Analysis/BaselineProvider.cs:714-715,778
  • VERIFIED — GetEarliestBlockedProcessReportInWindowAsync is left unfiltered (self-wait still proves XE capture), doc comment reworded, SQL unchanged — Lite/Services/LocalDataService.Blocking.cs:1113-1124
  • VERIFIED — the described test scenarios (self-wait+real+zero-pair minute counts 2 across trend/duration/slicer, HasBlockedProcessReportsInWindowAsync true; XE-only-self-wait window falls to the DMV arm and the DMV arm's own self-wait is also dropped) match BlockingSeries_LeaveSelfWaitsOut_OnTheXeArm and BlockingSeries_OnlySelfWaitReportsInXe_DrawTheDmvArm_AndDmvSelfWaitsGo in Lite.Tests/BlockingStatsReaderTests.cs
  • VERIFIED — "BlockingPerMinute baseline over five Tuesdays: 10 samples of mean 1.0 (old SQL: 15 samples)" — TheBlockingPerMinuteBaseline_LeavesSelfWaitsOut in Lite.Tests/EventBaselineCoveredDaysTests.cs asserts SampleCount=10, Mean=1.0; the old unfiltered SQL would have bucketed all 3 non-empty minutes/day × 5 days = 15 samples, consistent with the claimed delta
  • UNVERIFIED — the cited local test run counts (247 + 22 tests, 0 failed) — a live build/test run, not checkable from a read-only tree

Riskiest lines

  1. Lite/Services/LocalDataService.Blocking.cs:1217,1223 — GetBlockingTrendAsync's bpr/dmv arms now both filter self-waits; a mistake here would silently drop real blocking from the Overview lane and MCP trend tool — covered by BlockingSeries_LeaveSelfWaitsOut_OnTheXeArm and BlockingSeries_OnlySelfWaitReportsInXe_DrawTheDmvArm_AndDmvSelfWaitsGo
  2. Lite/Analysis/BaselineProvider.cs:778 — BlockingPerMinute baseline predicate change feeds the anomaly-detection ratio directly — covered by TheBlockingPerMinuteBaseline_LeavesSelfWaitsOut
  3. Lite/Services/LocalDataService.Blocking.cs:1004,1016 — GetBlockingSlicerDataAsync's dmv arm filters on v_dmv_blocking_snapshots using the same blocking_spid/blocked_spid columns already relied on elsewhere in the file (e.g. line 55-56's FILTER clauses) — covered by BlockingSeries_OnlySelfWaitReportsInXe_DrawTheDmvArm_AndDmvSelfWaitsGo (asserts slicer sum)

Traps near this diff

  • "Lite and Darling ship in parity (CONTRIBUTING's two-store rule): a fix to one SKU's reader, tool, alert or description without the counterpart's is a finding" — does not apply: this commit is the Lite-side counterpart that brings Lite into parity with Darling's already-filtered baseline (Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs:809, from the earlier #6690/#6663 work); nothing on the Darling side needs a matching change here.
  • "Our wrong test is a bug we introduced, never a flake" / "A test that encodes the lie" — does not apply: the new tests seed real rows through the service and assert on actual query results (Lite.Tests/BlockingStatsReaderTests.cs, Lite.Tests/EventBaselineCoveredDaysTests.cs), not source-text or spelling assertions.

Divergence (blind read vs PR body)

None — the "Round 1 fixes (Lite)" section of the PR body matches the diff on every changed site, every left-alone site, and every described test outcome.

Reviewer checklist

  • Confirm the cited local test run (*BlockingStats*, *BaselineProvider*, *AnomalyDetector*, etc., 247 + 22 tests) actually passed, and that the full suite / pre-push.sh pass in CI

…lt blocking baseline records where its supply begins (#6705)

The Viewer and MCP blocking trend and duration reads, and the existence check that picks their arm, take BlockingWaits.NotSelfWaitSql on both arms. A rebuilt blocked_process_baseline records a supply floor (view comment) on the backfill pass, and the Blocking baseline starts its window no earlier, so days the aggregate cannot back are not counted as quiet.
…d's report count, record the baseline supply floor when the view is created (#6705)
@erikdarlingdata
erikdarlingdata dismissed claude[bot]’s stale review October 24, 2026 09:47

Claude posting for Erik Darling. Every round-1 item is fixed on the branch: the bars and band agree in both products, the supply floor, and the retention figures. The bot's round 2 (5637680663) covered the Lite commit; a separate review covered the Darling commits, and its findings are fixed in 44b8d01.

@erikdarlingdata
erikdarlingdata enabled auto-merge (squash) October 24, 2026 09:47
@erikdarlingdata
erikdarlingdata merged commit 2bec9c4 into dev Oct 24, 2026
24 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/6690-blocking-baseline-self-waits branch October 24, 2026 11:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant