Skip to content

A parallel query waiting on itself is no longer counted as blocking (#6645) - #6663

Merged
erikdarlingdata merged 5 commits into
devfrom
fix/6645-intra-query-waits
Oct 24, 2026
Merged

erikdarlingdata merged 5 commits into
devfrom
fix/6645-intra-query-waits

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Claude posting for Erik Darling.

Closes #6645.

Why

The threads of a parallel query write blocked-process reports with blocked_spid = blocking_spid (different ecid): the query waiting on itself. Nothing dropped them, so the wait counts, the head-blocker count, the chain and the anomaly count all included them.

What changes

A report whose blocked and blocking spid are the same is not a blocking wait. Rows with no spid (the collector stores 0, never NULL, for a report that names no session; NULL spids appear only in some test fixtures) are still blocking events, so the rule is "equal and not 0 (and not NULL)", not IS DISTINCT FROM as the brief wrote (that would drop NULL/NULL marker rows; an existing Lite fixture with unset spids also caught it).

  • PerformanceMonitor.Common/BlockingWaits.cs: WaitsCteSql filters BlockingWaits.NotSelfWaitSql (NOT COALESCE(blocked_spid = blocking_spid AND blocked_spid <> 0, FALSE), one constant every SQL surface uses) in the _marked CTE (PostgreSQL and DuckDB; also lowers wait_reports); Collapse<T> skips equal non-zero spids. This covers BLOCKING_EVENTS (Pg and DuckDB), BlockingWaitRows, fleet tile, daily summary, health reader, Overview, Lite LocalDataService.Blocking.cs.
  • PerformanceMonitor.Analysis/BlockingChainReconstructor.cs: Reconstruct skips a row with equal spids before any node is added, so no chain and no cycle (Pg, Lite and the Viewer share the file).
  • Anomaly counts: all four COUNT subqueries in PgAnomalyDetector.BlockingWindowSql / BlockingSkippingSeparateCountSql, and the two in Lite/Analysis/AnomalyDetector.cs.
  • Current-wait snapshot: DarlingAlertReadAdapter.CurrentBlockingWaitSql and Lite GetCurrentBlockingWaitAsync use FILTER (WHERE ...) on the SUM and the COUNT(DISTINCT), not a WHERE, so a stored snapshot made only of self-wait rows still reads as a snapshot (its time and freshness) with nothing blocked, instead of as no snapshot. The DMV collector (DmvBlockingSnapshotCollector.cs:207) already refuses to write self-wait rows, so this only matters for rows stored before; the state is not reachable from new collection.

Test plan

  • New tests, each failing on the old code and passing on the fix (source reverted, rebuilt, both suites run): BlockingWaitCountTests.ParallelQuerySelfWaits_AreNotWaits (Collapse and BlockingWaitRows, plus a spid 0/0 report kept), BlockingWaitCountTests.PostgresSql_LeavesOutParallelQuerySelfWaits_AndCountsTheRealOne (live PG: 1 wait, 1 head blocker, wait_reports 5), BlockingWaitDuckDbTests.ParallelQuerySelfWaits_AreNotBlocking (fact event_count 1, distinct_head_blockers 1, overview 1, current-wait snapshot 0 sessions), BlockingChainReconstructorTests.ParallelQuerySelfWaits_MakeNoChainAndNoCycle (no chain, CycleDetected false; real chain beside it unchanged).
  • Darling: *BlockingWaitCountTests*, *Census*, *Ratchet*, *Anomaly*, *AlertEngineTests*, *DarlingSelfAlertTests*, *BlockingChain*, *DarlingFleetReader*, *Blocking* on a throwaway PostgreSQL: Total 1580, Failed 0, Skipped 1.
  • Lite: *Census*, *Ratchet*, *Anomaly*, *Blocking*, *Overview*, *LiteAlert*, *McpBlocking*: Total 479, Failed 0.
  • The Darling FILTER aggregate checked by hand on the throwaway PostgreSQL (self row dropped, NULL/NULL row kept; the 0/0 row the collector really stores is kept by the round-1 predicate, see below).
  • Not run: full suite, pre-push.sh (brief says CI decides). The PG anomaly SQL and the Darling current-wait SQL have no behavior test of their own (the anomaly SQL is only pinned as a string by an existing test); their Lite twins are exercised.

Considered, not added

  • intra_query = true on get_blocking rows (optional in the brief): get_blocking keeps the rows unchanged; adding the flag needs a column through three readers, more than a few lines.

CHANGELOG entry

SECTION: Fixed
ENTRY:

Bot review round 1 (changes requested)

  • spid 0 versus NULL (blocking): the SQL dropped (0, 0) rows (0 = 0 is true) that Collapse keeps. BlockingWaits.NotSelfWaitSql is now NOT COALESCE(blocked_spid = blocking_spid AND blocked_spid <> 0, FALSE), used by WaitsCteSql (all eleven surfaces), both FILTER current-wait reads, the Pg BlockingWindowSql / BlockingSkippingSeparateCountSql (now $@ const strings) and the Lite AnomalyDetector window. The NULL comments are corrected. New behaviour tests: (0, 0) and an equal-spid wait added to PostgresSql_AgreesWithTheCSharpRule_OnTheSameReports (7 waits, PostgreSQL), BlockingWaitDuckDbTests.AReportThatNamedNoSession_StoredAsSpidZeroZero_IsStillAWait (fact, overview and C# agree) and ADmvSnapshotOfSpidZeroZero_IsStillBlockedInTheCurrentWaitRead (DuckDB).
  • Anomaly ratio: Lite BaselineProvider Blocking baseline is a query, so it now filters with the same predicate and the 3x ratio compares like with like. The Darling baseline is the stored continuous aggregate collect.blocked_process_baseline (TimescaleSupport.CreateBlockedProcessBaselineSql, count(*) AS event_count over collect.blocked_process_reports); changing it needs a store migration that recreates and backfills it, which this PR does not add. Until then the Darling baseline keeps self-waits (a server with many of them has a high baseline and real blocking is under-flagged); a comment at PgAnomalyDetector.BlockingWindowSql says so. The Lite BlockingPerMinute baseline (a chart series, not used by the anomaly ratio) is left as it was.
  • Refuted lines corrected: spid 0, not NULL, is what the collector stores (above). The self-wait-only snapshot is not reachable from the DMV collector (DmvBlockingSnapshotCollector.cs:207); the FILTER rationale and the LocalDataService.Blocking.cs comment now say it covers stored rows only.
  • Alert XE-or-DMV choice (divergence Add check for updates feature #1): AlertContextBuilders.BlockingWaitRows now decides hasXeReports from XE reports that are not self-waits (equal non-zero spids), so a window whose XE reports are all self-waits falls back to the DMV waits like the SQL surfaces. Test: BlockingWaitCountTests.XeReportsThatAreAllSelfWaits_FallBackToTheDmvWaits. BlockingChainReconstructor still compares raw spids (a (0, 0) row makes no node, as before this PR).
  • Not run locally (CI decides): no build or test on this laptop; the PostgreSQL tests need DARLING_TEST_PG.

@erikdarlingdata
erikdarlingdata marked this pull request as ready for review October 23, 2026 23:43
@erikdarlingdata
erikdarlingdata enabled auto-merge (squash) October 23, 2026 23:43
Comment thread Lite/Analysis/AnomalyDetector.cs Outdated
SELECT
COALESCE(NULLIF(
(SELECT COUNT(*) FROM " + StoredEventCopies.BlockedProcessReports("server_id = $1 AND event_time >= $2 AND event_time < $3{SCOPE}") + @" AS ev), 0),
(SELECT COUNT(*) FROM " + StoredEventCopies.BlockedProcessReports("server_id = $1 AND event_time >= $2 AND event_time < $3 AND NOT COALESCE(blocked_spid = blocking_spid, FALSE){SCOPE}") + @" AS ev), 0),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The blocking anomaly's current window and its baseline are now in different units.

currentBlocking here (and in PgAnomalyDetector.BlockingWindowSql / BlockingSkippingSeparateCountSql) counts blocked-process report rows minus self-waits. The baseline it is divided by at Lite/Analysis/AnomalyDetector.cs:855 (currentBlockingPerHour / Math.Max(baselineBlockingRate, 1) >= 3) is MetricNames.Blocking, which is still a count of every report row:

  • Lite: Lite/Analysis/BaselineProvider.cs:716-717 — EventBaselineSql("blocked_process_report", …, StoredEventCopies.BlockedProcessReports("server_id = $1 AND event_time >= $2 AND event_time < $3"), "COUNT(*)"), no spid predicate.
  • Darling: PgBaselineProvider.cs:1424 reads blocked_process_baseline, whose body is count(*) over collect.blocked_process_reports with no row-level filter (TimescaleSupport.cs:797-806, and its own doc line says "both have NO row-level filter").

Failure scenario: a server where a parallel query's self-waits are most of the reports (the #6645 server) keeps a 30-day baseline mean built from all of them, while the current window now counts only the real ones. If 80% of reports were self-waits, the ratio is deflated ~5x and a genuine blocking storm no longer reaches the 3x spike bar — the detector goes quiet on exactly the servers this fix was written for, and stays quiet for as long as the self-waits keep being written (the baseline never washes out).

The same file already documents the one place it accepts a skew of this shape, deliberately and with its direction stated (:790-793, the separately-monitored-databases filter: "Only the current-window counts are filtered; the baseline stays server-wide, which can only make a master spike less likely, an accepted trade"). This one is not mentioned anywhere in the diff or the PR body, and unlike the master case it applies to every server. Either filter the baseline arms too, or state the trade and its direction here and in the body.

/* #6645: a parallel query's threads write reports with blocked_spid = blocking_spid (different ecid). That is
the query waiting on itself, not blocking: it is no wait, no head blocker and no report. A report that
names no session (NULL spids) is kept: the comparison is not true for it. */
WHERE NOT COALESCE(s.blocked_spid = s.blocking_spid, FALSE)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two forms of the rule disagree on the report that names no session — and it is the SQL form's comment that is wrong about why.

The comment above says a report naming no session "is kept: the comparison is not true for it", and Collapse at :97 keeps it by an explicit && blockedSpid(row) != 0. That holds only when the store actually has NULL in both columns. The collector writes 0, never NULL: BlockedProcessReportCollector.cs:1115 / :1117 are int.TryParse(...) ? bs : 0 on a non-nullable int BlockedSpid, and the writer binds it straight through (:1050 .Value(row.BlockedSpid)).

So for a stored (0, 0) row: 0 = 0 is TRUE here and the row is dropped, while Collapse keeps it. That is the one thing the design says cannot happen — "The same rule is written twice … A test runs both over the same rows and checks they agree" (:42-44) — and the agreement tests do not cover it: PostgresSql_AgreesWithTheCSharpRule_OnTheSameReports, …OnSubMillisecondStartsAndNullTimes and …OnDmvSnapshotRows carry no row with both spids 0, and the new ParallelQuerySelfWaits_AreNotWaits pins only the C# side of the 0/0 case (BlockingWaitCountTests.cs:113). The hand-check in the test plan ("self row dropped, NULL/NULL row kept") tested NULL/NULL, which is the shape only the test fixtures produce (e.g. Lite.Tests/AnomalyTileWindowEndTests.cs:71, which omits the spid columns).

Consequence if such a row is ever written: the alert value counts it as one blocking event while the overview tile, the fleet tile, the BLOCKING_EVENTS fact and the daily summary do not. Either add the 0 exception to the SQL (NOT COALESCE(s.blocked_spid = s.blocking_spid AND s.blocked_spid <> 0, FALSE)) or drop it from Collapse and say the marker row is not a wait — but not one each way, and the comment should name 0, not NULL, as the shape the collector stores.

@claude

claude Bot commented Oct 23, 2026

Copy link
Copy Markdown

The alert's XE-or-DMV choice is still made on the self-waits, so the DMV arm it is supposed to fall back to is suppressed

This is the one finding I'd call blocking. It sits in a file the diff does not touch, which is why it is here rather than inline.

AlertContextBuilders.BlockingWaitRows decides XE-or-DMV before the rows are collapsed:

var hasXeReports = events.Any(r => r.Source == BlockedProcessAlertRow.XeReportSource);   // :502

var waits = BlockingWaits.Collapse(FilterBlocking(events, excludedDatabases), …);        // :504 — self-waits dropped here

return hasXeReports
    ? waits.Where(r => r.Source == BlockedProcessAlertRow.XeReportSource).ToList()       // :516-518
    : waits;

A parallel query's self-wait reports are blocked-process reports, so they still set hasXeReports. After this change they are gone from waits, and the .Where(Source == Xe) then throws away the DMV-snapshot rows the merge appended (BlockedProcessReportMerge.AppendDmvFallbackRows, called at DarlingAlertReadAdapter.cs:566 and LocalDataService.Blocking.cs:930). The result is zero, not the DMV waits.

Failure scenario. A server whose blocked-process-report session captures a parallel query waiting on itself (the #6645 shape), and whose real lock blocking in the same hour is shorter than the blocked process threshold, so only dmv_blocking_snapshots caught it. Before: the alert fired on the phantom self-waits. After: effectiveBlockingCount (AlertEngine.cs:1433) is 0 and BuildBlockingContext returns null (:64-65) — the real blocking is never alerted on. The watermark also drops to 0, so the gate re-arms as if nothing had happened.

Why I read this as a defect rather than the intended trade. Every SQL surface in this same diff falls back correctly, because there the filter is inside the counted subquery and NULLIF(…, 0) sees the post-filter count:

  • Lite/Services/LocalDataService.Overview.cs:134-136 — COALESCE(NULLIF((SELECT COUNT(*) FROM xe_waits), 0), (SELECT COUNT(*) FROM dmv_waits)), where xe_waits is now self-wait-free, so an all-self-wait hour falls through to dmv_waits.
  • Lite/Analysis/AnomalyDetector.cs:804-807 and PgAnomalyDetector.BlockingWindowSql / BlockingSkippingSeparateCountSql — same shape, same fallback.
  • ViewerDataService.Overview.cs:173-181 — the caller applies "use XE when it has any row, else DMV" to the already-filtered xe_waits count.

So after this PR the alert engine is the only surface that answers 0 where the overview card, the fleet tile and the anomaly count all answer with the DMV waits — the exact disagreement #5644 W1/W2 set out to make impossible ("BlockingWaitRows is the one place that set is built", AlertEngine.cs:1423; "so the alert's value and its detail cannot disagree", AlertContextBuilders.cs:488).

The excluded-database case that :512-515 documents as deliberate is not the same thing: there, suppression is the rule being honoured. Here the ruling is that a self-wait is not a blocking event at all, so it should not be able to vote on which source counts. Deciding hasXeReports on rows that survive the self-wait test — e.g. events.Any(r => r.Source == XeReportSource && (r.BlockedSpid != r.BlockingSpid || r.BlockedSpid == 0)), or hoisting the test into a shared predicate used by both lines — restores the fallback while leaving the excluded-database behaviour exactly as it is.

No behaviour test covers it: ParallelQuerySelfWaits_AreNotWaits asserts the all-self-wait case returns empty with no DMV rows present (BlockingWaitCountTests.cs:110), and NoXeReportsAtAll_CountsTheDmvWaits_OutsideTheExcludedDatabases has no XE rows at all. A test that seeds XE self-waits plus a real DmvSnapshotSource wait and expects 1 would fail today.


Non-blocking notes

1. The DMV-side half of this change cannot bite on anything the collector writes. DmvBlockingSnapshotCollector.cs:207 is already AND wt.blocking_session_id <> wt.session_id, so dmv_blocking_snapshots has never held a self-wait row. The four DMV NOT COALESCE(...) predicates and the FILTER-not-WHERE reasoning on CurrentBlockingWaitSql / GetCurrentBlockingWaitAsync are therefore inert for current data (harmless as belt-and-braces for rows already in the store, but the PR body reads as though these rows arrive). It also makes SeedWaitAsync(59, 59, Ago(30), reports: 3, dmv: true) in BlockingWaitDuckDbTests.ParallelQuerySelfWaits_AreNotBlocking a fixture of a shape no live run produces — the .github/REVIEW-TRAPS.md line "A test that constructs a 'four maintenance anomalies in one run' fixture is testing a shape no live run produces (#3709) … Fixtures should carry the population the engine actually emits, or say they do not." The XE half of that test is the real one; worth a word in the test saying the DMV half covers only already-stored rows.

2. A comment the change falsified. ViewerDataService.Overview.cs:163 still says "the CTE's *_marked relation holds every source row" of the two report-count columns the health band reads. After the new WHERE in {name}_marked it holds every source row except self-waits. The lowering is intended ("also lowers wait_reports"), so this is just the sentence going stale — same for PgFactCollector.Waits.cs:305-307's "the report rows before they collapse to waits".

3. Nine copies of one predicate. NOT COALESCE(blocked_spid = blocking_spid, FALSE) is now written out nine times across five files, with a tenth, deliberately different spelling in C# (:97, with the != 0 exception). The traps file: "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy." The repo already has the seam for this (StoredEventCopies, SeparatelyMonitoredScope.BprFilter); a BlockingWaits.SelfWaitExcludedSql(string alias = "") would make the next edit one edit — and would have made the 0-vs-NULL divergence in my other comment impossible to write.

4. Dead guard. BlockingChainReconstructor.cs:233-234 (if (blocker.Equals(blocked)) continue; // a session cannot block itself) is now unreachable: SessionKey equality needs equal spids, and equal spids never get past the new :217. Worth removing with a note, or leaving with one.

5. For the checklist, not a finding: the blocking grids still list self-wait rows (no filter on the grid reads at LocalDataService.Blocking.cs:772 / :896 or DarlingAlertReadAdapter.cs:521), but clicking one now reconstructs nothing, so BlockingChainViewerProjection.BuildModelForSession returns null and the window says "No blocking was captured for this session in the selected time range" (BlockingChainViewerProjection.cs:18-20) about a row the user is looking at. That may well be the right answer, but it is a sentence a person should read in the real app before it ships.

claude[bot]
claude Bot previously requested changes Oct 23, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: CHANGES REQUESTED

Two inline comments and one top-level note carry the detail.

Claims

  • VERIFIED — "WaitsCteSql filters NOT COALESCE(blocked_spid = blocking_spid, FALSE) in the _marked CTE (both dialects; also lowers wait_reports)" — BlockingWaits.cs:200-203; wait_reports descends from the filtered _marked (:190, :218-226) and PgFactCollector.Waits.cs:307 reads waits_marked.
  • VERIFIED — "Collapse<T> skips equal non-zero spids" — BlockingWaits.cs:97-100, pinned by BlockingWaitCountTests.ParallelQuerySelfWaits_AreNotWaits.
  • VERIFIED — "covers BLOCKING_EVENTS (Pg and DuckDB), BlockingWaitRows, fleet tile, daily summary, health reader, Overview, Lite LocalDataService.Blocking.cs" — all eleven WaitsCteSql call sites go through that one CTE (PgFactCollector.Waits.cs:278, DuckDbFactCollector.Waits.cs:257, DarlingFleetReader.cs:373, DailySummarySql.cs:243, LocalDataService.DailySummary.cs:55, DarlingHealthReader.cs:93, ViewerDataService.Overview.cs:166, .Fleet.cs:92, .AzureMasterScope.cs:189, LocalDataService.Overview.cs:127, .Blocking.cs:619).
  • VERIFIED — "Reconstruct skips a row with equal spids before any node is added, so no chain and no cycle" — BlockingChainReconstructor.cs:215-218, above the allNodes adds; BlockingChainReconstructorTests.ParallelQuerySelfWaits_MakeNoChainAndNoCycle.
  • VERIFIED — "all four COUNT subqueries in Pg, and the two in Lite" — PgAnomalyDetector.cs:315, 318, 331, 335; Lite/Analysis/AnomalyDetector.cs:805, 807.
  • VERIFIED — "FILTER (WHERE …) on the SUM and the COUNT(DISTINCT), not a WHERE" — DarlingAlertReadAdapter.cs:592-593, LocalDataService.Blocking.cs:54-55; the snapshot stays present with 0 blocked, which AlertEngine.cs:1739-1789 reads as a fresh non-breaching sample, as BlockingWaitDuckDbTests.cs:154-157 asserts.
  • VERIFIED — "get_blocking keeps the rows unchanged" — no MCP/grid blocking reader in the diff; LocalDataService.Blocking.cs:772/:896 and DarlingAlertReadAdapter.cs:521 carry no spid predicate.
  • VERIFIED — the CHANGELOG's "could draw a false cycle" — the old loop skipped only identical SessionKeys (:233-234), so 59:2 ← 59:1 plus 59:1 ← 59:2 made two edges and HasCycle fired.
  • UNVERIFIED — "each failing on the old code and passing on the fix (source reverted, rebuilt)" — needs a build and two runs.
  • UNVERIFIED — "Darling … Total 1580, Failed 0, Skipped 1" — needs a live store; DARLING_TEST_PG is unset here, so every PostgresSql_* test self-skips (BlockingWaitCountTests.cs:298, 328, 406, 482, 522).
  • UNVERIFIED — "Lite … Total 479, Failed 0" — needs the run.
  • UNVERIFIED — "the Darling FILTER aggregate checked by hand (self row dropped, NULL/NULL row kept)" — a live read; it checked NULL/NULL, not the 0/0 the collector stores.
  • UNVERIFIED — PostgresSql_LeavesOutParallelQuerySelfWaits_AndCountsTheRealOne's "1 wait, 1 head blocker, wait_reports 5" — PostgreSQL-only, skipped.
  • REFUTED — "rows with no spid … stored with NULL spids and read as 0 … are still blocking events", and BlockingWaits.cs:200-202's "A report that names no session (NULL spids) is kept" — the collector stores 0, never NULL (BlockedProcessReportCollector.cs:1115, 1117 are int.TryParse(…) ? x : 0 on non-nullable int, bound at :1050), and 0 = 0 is TRUE, so the SQL drops what Collapse keeps. NULL spids are the fixtures' shape (Lite.Tests/AnomalyTileWindowEndTests.cs:71). Inline on BlockingWaits.cs:203.
  • REFUTED — "a newest snapshot made only of self-wait rows" as reachable state — DmvBlockingSnapshotCollector.cs:207 is already AND wt.blocking_session_id <> wt.session_id, so the DMV-side predicates bite only on rows already stored.

Riskiest lines

  1. PerformanceMonitor.Common/BlockingWaits.cs:203 — one WHERE re-defines eleven read surfaces at once — covered by PerformanceMonitorLite.Tests.BlockingWaitDuckDbTests.ParallelQuerySelfWaits_AreNotBlocking (real DuckDB: fact event_count, distinct_head_blockers, overview count, current-wait snapshot).
  2. PerformanceMonitor.Common/BlockingWaits.cs:97 — the != 0 exception the SQL half lacks; the C# side is covered by Darling.Tests.BlockingWaitCountTests.ParallelQuerySelfWaits_AreNotWaits, the C#-vs-SQL disagreement on a stored (0, 0) row is UNCOVERED (no PostgresSql_AgreesWithTheCSharpRule_* case carries one).
  3. Lite/Analysis/AnomalyDetector.cs:805 (twins PgAnomalyDetector.cs:315, 331) — filtered numerator over the unfiltered MetricNames.Blocking baseline (BaselineProvider.cs:716-717, TimescaleSupport.cs:797-806) in the 3x ratio at :855 — UNCOVERED (DarlingAnomalyBaselineTests.cs:91 is a string pin, not coverage).

Traps near this diff

  • "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy." — applies — nine spellings (BlockingWaits.cs:203, LocalDataService.Blocking.cs:54-55, DarlingAlertReadAdapter.cs:592-593, AnomalyDetector.cs:805, 807, PgAnomalyDetector.cs:315, 318, 331, 335) plus a tenth, different one in C# at BlockingWaits.cs:97 — and the two already disagree.
  • "A threshold that means different units at different surfaces … State the unit at every surface that shows the number (#3631)." — applies — AnomalyDetector.cs:805 / PgAnomalyDetector.cs:315 count non-self reports against a baseline of all reports; the health band's tiers read wait_reports / report_count (PgFactCollector.Waits.cs:305-307, ViewerDataService.Overview.cs:180-181), measured in the unfiltered unit.
  • "A test that constructs a … fixture is testing a shape no live run produces (#3709). Fixtures should carry the population the engine actually emits, or say they do not." — applies — BlockingWaitDuckDbTests.cs:146 seeds a self-wait DMV snapshot that DmvBlockingSnapshotCollector.cs:207 forbids.
  • "Lite and Darling ship in parity … a fix to one SKU's reader, tool, alert or description without the counterpart's is a finding (#3594)." — does not apply — AnomalyDetector.cs:805 ↔ PgAnomalyDetector.cs:315, LocalDataService.Blocking.cs:54 ↔ DarlingAlertReadAdapter.cs:592, rest shared.
  • "Ruling (#5833) … never answer a trap with a test that reads source text." — does not apply — all four new tests are behaviour tests (BlockingWaitDuckDbTests.cs:143, BlockingWaitCountTests.cs:97/:325, BlockingChainReconstructorTests.cs:62).
  • "An empty analysis window is 'unavailable', not an all-clear (#3553)." — does not apply — the FILTER keeps the row, so the read stays non-null with its collection_time (LocalDataService.Blocking.cs:53-55) and AlertEngine.cs:1739-1748's freeze is never entered.
  • "A 'latest snapshot' read anchored on = (SELECT MAX(collection_time) …) must project the anchor (#3637)." — does not apply — both reads still project and group by collection_time (LocalDataService.Blocking.cs:53, DarlingAlertReadAdapter.cs:591).
  • "Every MCP filter is part of the query, not a post-filter on the page (#3641)." — does not apply — every store surface filters in the query; the one in-memory drop runs over the engine's whole 200-row fetch, and the MCP tools are untouched by design.

Divergence (blind read vs PR body)

Four, all the same direction — the body describes the rows dropped, not what the surfaces around them then do.

  1. The alert's XE-or-DMV choice still counts the dropped rows. AlertContextBuilders.cs:502 computes hasXeReports before the collapse, so self-wait reports still elect the XE arm and :516-518 then discards the merged DMV waits. An hour whose only blocked-process reports are self-waits, with real DMV-only blocking beside them, now counts 0 where every SQL surface in this diff falls back through NULLIF(…, 0) (LocalDataService.Overview.cs:134-136, AnomalyDetector.cs:804-807, ViewerDataService.Overview.cs:173-181). That leaves the alert engine the only surface disagreeing with the overview card and the fleet tile — what #5644 W1/W2 exists to prevent (AlertEngine.cs:1419-1433). Suggested predicate in the top-level note; no test covers it.
  2. The anomaly baseline goes unmentioned — filtered current window, unfiltered 30-day mean. Inline on AnomalyDetector.cs:805.
  3. The DMV premise — the FILTER-over-WHERE rationale describes a state DmvBlockingSnapshotCollector.cs:207 prevents.
  4. NULL versus 0 — the body and two new comments name NULL as the stored shape; the collector writes 0, and the two forms of the rule part ways there.

Reviewer checklist

  • Run the two live-PostgreSQL tests with DARLING_TEST_PG set, and confirm COUNT(DISTINCT blocked_spid) FILTER (…) plans and returns as expected against the real dmv_blocking_snapshots (the FILTER form is new here).
  • On a store that already holds self-wait rows, read the blocking anomaly fact before and after: how far does current_blocking fall while the baseline mean stays put? That number decides whether the baseline arms need the filter or a documented trade.
  • In the real app, click a self-wait row in the blocking grid (Lite and Viewer): BuildModelForSession now returns null, so the window says "No blocking was captured for this session in the selected time range" about a row on screen.
  • Check a live server's health band: wait_reports / report_count now exclude self-waits, so a band sized on the old report volume can change tier with no workload change.

…ng baseline the same way (#6645)

The collector stores 0, never NULL, for a report that names no session. The C# rule keeps those
rows, but the SQL predicate dropped them (0 = 0 is true). One BlockingWaits.NotSelfWaitSql now
carries the rule for every SQL surface. The Lite blocking baseline counts the same reports as the
current window.
@claude

claude Bot commented Oct 24, 2026

Copy link
Copy Markdown

Follow-up review (range 8636733cbd85..6d29768add71). The merge commit (8adebda6e) only brings in unrelated dev history (V215 indexes, admin-write, notification-channels, etc.) — nothing in it interacts with blocking/anomaly code, so the only commit graded here is 6d29768ad.

That commit does exactly what round 1 asked for the spid-0-vs-NULL defect: PerformanceMonitor.Common/BlockingWaits.cs:56 now carries one NotSelfWaitSql constant (NOT COALESCE(blocked_spid = blocking_spid AND blocked_spid <> 0, FALSE)) that every SQL surface aliases instead of copying — WaitsCteSql's _marked CTE, PgAnomalyDetector.BlockingWindowSql/BlockingSkippingSeparateCountSql (now $@ interpolated consts), DarlingAlertReadAdapter.CurrentBlockingWaitSql, Lite's AnomalyDetector/LocalDataService.Blocking.cs/BaselineProvider. That also answers round 1's "nine copies of one predicate" trap finding directly.

Verified the collector claim at the source: BlockedProcessReportCollector.cs:1115,1117 is int.TryParse(...) ? x : 0 into a non-nullable int, so the collector really does store 0, never NULL — the new comments are now correct where round 1 flagged them as REFUTED.

New tests are solid, real behaviour tests, not spelling pins: BlockingWaitCountTests.PostgresSql_AgreesWithTheCSharpRule_OnTheSameReports now seeds a (0,0) row and a (59,59) self-wait beside the existing rows and checks the live-Postgres CTE count (7) against Collapse; BlockingWaitDuckDbTests.AReportThatNamedNoSession_StoredAsSpidZeroZero_IsStillAWait checks the C# rule, the BLOCKING_EVENTS fact and the Overview card all agree (2), and ADmvSnapshotOfSpidZeroZero_IsStillBlockedInTheCurrentWaitRead exercises the FILTER-based current-wait read the same way.

The Darling anomaly-baseline parity gap round 1 flagged is not closed here — the stored continuous aggregate still counts self-waits until a migration rebuilds it — but this round is honest about that: the Lite twin (BaselineProvider.cs) got the filter, and a new comment on PgAnomalyDetector.BlockingWindowSql spells out the gap and its effect (baseline runs high, real blocking under-flagged) rather than silently leaving it. That's a documented, deliberate trade-off, not a hidden one.

The hasXeReports/AlertContextBuilders.cs:502 defect from round 1 is untouched by this commit, and the PR body says so explicitly under "Not changed (outside this round's list)" — not re-litigating it here since nothing in this round's diff touches that path either way.

No new defects found in this round's commit.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: LGTM

Claims

  • VERIFIED — "the SQL dropped (0, 0) rows (0 = 0 is true) that Collapse keeps" — BlockingWaits.cs:97-100 (C# keeps (0,0)) vs. the old SQL NOT COALESCE(blocked_spid = blocking_spid, FALSE) (0=0 true, dropped); fixed by the new predicate at BlockingWaits.cs:56.
  • VERIFIED — "BlockingWaits.NotSelfWaitSql is now NOT COALESCE(blocked_spid = blocking_spid AND blocked_spid <> 0, FALSE), used by WaitsCteSql (all eleven surfaces), both FILTER current-wait reads, the Pg BlockingWindowSql / BlockingSkippingSeparateCountSql (now $@ const strings) and the Lite AnomalyDetector window" — BlockingWaits.cs:56,213, PgAnomalyDetector.cs:315,318,331,335, DarlingAlertReadAdapter.cs:592-593, Lite/Analysis/AnomalyDetector.cs:805,807.
  • VERIFIED — "New behaviour tests: (0,0) and an equal-spid wait added to PostgresSql_AgreesWithTheCSharpRule_OnTheSameReports (7 waits, PostgreSQL)… AReportThatNamedNoSession_StoredAsSpidZeroZero_IsStillAWait… ADmvSnapshotOfSpidZeroZero_IsStillBlockedInTheCurrentWaitRead" — BlockingWaitCountTests.cs:307-310,321-322; Lite.Tests/BlockingWaitDuckDbTests.cs:162-182.
  • VERIFIED — "Lite BaselineProvider Blocking baseline is a query, so it now filters with the same predicate" — Lite/Analysis/BaselineProvider.cs:716-719.
  • VERIFIED — "The Darling baseline is the stored continuous aggregate… changing it needs a store migration… this PR does not add [it]… a comment at PgAnomalyDetector.BlockingWindowSql says so" — PgAnomalyDetector.cs:308-312 (new comment), no migration file in this diff.
  • VERIFIED — "The Lite BlockingPerMinute baseline (a chart series, not used by the anomaly ratio) is left as it was" — BaselineProvider.cs:773-783 carries no NotSelfWaitSql filter.
  • VERIFIED — "spid 0, not NULL, is what the collector stores" — BlockedProcessReportCollector.cs:1115,1117 (int.TryParse(...) ? x : 0 into non-nullable int); comments corrected at BlockingWaits.cs:102-106,210-213.
  • VERIFIED — "Not changed (outside this round's list): the alert's XE-or-DMV choice in AlertContextBuilders.cs:502…" — AlertContextBuilders.cs is absent from this commit's changed-file list.
  • UNVERIFIED — "source reverted, rebuilt, both suites run"; Darling/Lite test totals — need a build and a live run.
  • UNVERIFIED — "the Darling FILTER aggregate checked by hand on the throwaway PostgreSQL" — a live read this tree can't reproduce.

Riskiest lines

  1. PerformanceMonitor.Common/BlockingWaits.cs:56 — the one predicate every SQL surface now aliases; a future edit to it moves every surface at once — covered by BlockingWaitCountTests.PostgresSql_AgreesWithTheCSharpRule_OnTheSameReports (live Postgres, 7 waits) and BlockingWaitDuckDbTests.AReportThatNamedNoSession_StoredAsSpidZeroZero_IsStillAWait.
  2. Darling/PerformanceMonitor.Darling.Analysis/PgAnomalyDetector.cs:331,335 (BlockingSkippingSeparateCountSql) — same substitution as BlockingWindowSql but over the excluded-database arm — UNCOVERED (no live-PG test seeds a (0,0) row through this specific statement; it was already a string-pin-only surface before this round per the prior review).
  3. Lite/Services/LocalDataService.Blocking.cs:54-55 — COUNT(DISTINCT blocked_spid) FILTER (WHERE NotSelfWaitSql) now can count blocked_spid = 0 as a distinct "session" — covered by ADmvSnapshotOfSpidZeroZero_IsStillBlockedInTheCurrentWaitRead, which asserts BlockedSessionCount == 1 for a (0,0) snapshot, so the behaviour is intentional and pinned, not accidental.

Traps near this diff

  • "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy." — does not apply (anymore) — this commit is the fix for exactly that trap: nine inline spellings collapse to one BlockingWaits.NotSelfWaitSql constant (BlockingWaits.cs:56) referenced from both SKUs.
  • "A test that constructs a … fixture is testing a shape no live run produces (#3709). Fixtures should carry the population the engine actually emits, or say they do not." — applies, pre-existing and unchanged by this round — BlockingWaitDuckDbTests.cs seeds dmv: true self-wait/(0,0) snapshot rows that DmvBlockingSnapshotCollector.cs:207's own join predicate never produces; this round adds one more such fixture (ADmvSnapshotOfSpidZeroZero_IsStillBlockedInTheCurrentWaitRead) without a note saying so, same as the pre-existing (59,59) DMV fixture already flagged in round 1.
  • "Lite and Darling ship in parity … a fix to one SKU's reader, tool, alert or description without the counterpart's is a finding (#3594)." — does not apply — every surface touched has its twin touched in the same commit (AnomalyDetector.cs:805,807 ↔ PgAnomalyDetector.cs:315,318,331,335; LocalDataService.Blocking.cs:54-55 ↔ DarlingAlertReadAdapter.cs:592-593; BlockingWaitDuckDbTests.cs ↔ BlockingWaitCountTests.cs), and the one deliberately-NOT-ported piece (Darling's stored baseline aggregate) is called out in a comment rather than silently skipped.
  • "Ruling (#5833) … never answer a trap with a test that reads source text." — does not apply — all four new/changed assertions in this round are behaviour tests over real DuckDB/Postgres reads, not text pins.

Divergence (blind read vs PR body)

None — the PR body's "Bot review round 1" section describes exactly what this commit's diff does (the NotSelfWaitSql constant, the corrected 0-vs-NULL comments, the Lite baseline fix, the documented-not-fixed Darling baseline gap, and the explicit "not changed" list for AlertContextBuilders.cs/BlockingChainReconstructor), and nothing in the diff goes further than that account.

Reviewer checklist

  • Run the two live-PostgreSQL tests with DARLING_TEST_PG set (PostgresSql_AgreesWithTheCSharpRule_OnTheSameReports, PostgresSql_LeavesOutParallelQuerySelfWaits_AndCountsTheRealOne) and the Darling/Lite suites named in the PR body, since none of them ran in this read-only tree.
  • The hasXeReports/AlertContextBuilders.cs:502 defect and the Darling stored-baseline parity gap from round 1 remain open by the author's own account — track them for a follow-up commit or issue; they are unaffected (neither fixed nor worsened) by this round's diff.

@erikdarlingdata
erikdarlingdata dismissed claude[bot]’s stale review October 24, 2026 03:32

Claude posting for Erik Darling. Round 2 is LGTM on 6d29768, and the alert's XE-or-DMV finding is fixed in 09b386e.

Lite's blocking baseline arm now filters self-waits with
BlockingWaits.NotSelfWaitSql (6d29768), so the expected event source
at line 228 of DarlingEventBaselineCoveredDaysTests.cs follows it.
Test expectation only; no behavior change.
@erikdarlingdata

Copy link
Copy Markdown
Owner Author

Claude posting for Erik Darling.

Lane report for the one-line pin behind the CI failure in TheTwoProducts_CarryTheSameEventBaselineBody_AndTheSameCollectors. Posted as a comment because the brief said not to edit the PR body.

  • Pushed b85ad464f to fix/6645-intra-query-waits. It is a fast-forward on 09b386ee5, one commit, with nothing new on origin to merge. PR A parallel query waiting on itself is no longer counted as blocking (#6645) #6663 head is now b85ad464f. It is still open, not a draft, and auto-merge is still armed.
  • Change: Darling/Darling.Tests/DarlingEventBaselineCoveredDaysTests.cs line 228. The expected Lite Blocking event source is now StoredEventCopies.BlockedProcessReports("server_id = $1 AND event_time >= $2 AND event_time < $3 AND " + PerformanceMonitor.Common.BlockingWaits.NotSelfWaitSql) + " AS ev". The diff is one line.
  • Check (no build and no test run, per the brief): a Python script applied the test's ArmCall regex to Lite/Analysis/BaselineProvider.cs and captured the Blocking arm's third argument as raw source. It compared that with the unescaped literal on line 228. Result: EQUAL True.
  • Doc comment: the brief's conditional sentence was not added. The comment above the test does not say the blocking arms filter alike, so the condition is not met.
  • Other references: no other file under Darling/Darling.Tests or Lite.Tests pins the Lite arm text. AnomalyTileWindowEndTests.cs matches event_time < $3 only in an unrelated OpenEnds list.
  • Not run: the build, the tests and the pre-push gate. CI decides.
  • Attribution: the commit has no attribution lines.
  • Defects found: none. Issues filed: none. Considered, not added: none.

For the coordinator: check CI on b85ad464f, mainly DarlingEventBaselineCoveredDaysTests. Nothing else was changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant