Skip to content

Read the plan regression live half from one interval length below the first unbuilt day (#9225) - #9234

Merged
erikdarlingdata merged 5 commits into
devfrom
fix/9225-plan-agg-floor
Nov 21, 2026
Merged

erikdarlingdata merged 5 commits into
devfrom
fix/9225-plan-agg-floor

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Claude posting for Erik Darling.

Fixes #9225

Cause

The plan regression read of a pass anchored at an instant (PlanRegressionDailyAsOfSql) takes closed days from plan_regression_daily and the rest from query_store_interval_latest. The live half started one whole day below the first day that is not built. A built day's rows are already in the totals, so the live half drops every row of that floor day except the few that cross midnight into the first unbuilt day. But no index finds those few rows, so the whole floor day's partition was read and thrown away. For an as_of a few hours back this was the largest read in the statement.

Fix

The live floor is now one interval length below the first unbuilt day, not a whole day.

  • A live row's last execution is at or after the first unbuilt day's midnight, and its first execution is at most one Query Store interval earlier. So a floor one interval length below the midnight reads every row that can be live.
  • QueryStoreIntervalWide.ResolveLongestIntervalSpanAsync finds that length. It runs the same statement and the same trust test as ResolveEdgeMarginAsync (no capture, a database with wide rows and no length, a capture that does not reach back a day before the read, a length of zero). ResolveEdgeMarginAsync now shares the one private method with it, so the two cannot drift.
  • When the length cannot be trusted, the floor stays a whole day, as before. The length is also never taken as more than a day, which is the longest an interval spans.
  • A row whose interval is longer than the length found is the one case that could be dropped. It can only exist if the server changed its interval length. A capture of the old length inside the window makes the longest length reported a day, and a change the captures did not see makes the trust test fail. Both leave the whole-day floor.
  • The length is looked up only when the unbuilt day sets the floor. A partial built day whose watermark is already lower still sets it exactly, as before, and a store that is current (watermark floor) does no extra read.
  • The interval table half of the statement is unchanged apart from the floor value. The same rows come back in the same order.

Measured

A seeded store with three servers, 17 days of intervals, closed days built through three days back, the server reporting 60 minute intervals, and a read anchored at 22:00 the previous day (heap table, parallelism off):

  • the live half read 24,001 interval rows with the whole-day floor and 18,001 with the resolved floor. The floor day of every server is gone, and every row the answer needs is still read.
  • the blocks of those scans were 3,957 and 3,949. The heap table only has an index that leads with the first execution time, so blocks move little here.

The diagnosis in the issue measured the same change on a day-partitioned store with one busy server: raising the floor to one hour below the first unbuilt day pruned the floor partition and took the plan from about 57,700 to 44,700 blocks, with the same answer. I did not re-seed that store.

Tests (PlanRegressionAsOfLiveFloorLiveTests, all live, own scratch store)

  • A trusted 60 minute length: the resolved floor is one hour below the first unbuilt day, the live half reads at least a thousand fewer rows than from a whole day below (counted from the executed plan), and the rows returned are identical, including the regressed query whose interval starts at 23:40 the day before and ends at 00:10, which sits between the new floor and midnight.
  • No capture, captures that start too late, and a capture that reports a one day interval: the floor stays a whole day and the rows are the same.
  • A partial built day's watermark below the resolved floor still sets the floor.
  • The fact itself for an anchored pass, end to end through PgFactCollector: the same fact metadata with and without health captures (so with and without the narrower floor), and the daily statement touches fewer blocks with the trusted length (from pg_stat_statements). This is the test that proves the collector uses the resolved floor.
  • Revert proof: with the resolve step made to return the whole-day floor, the trusted-length, partial-day and fact tests fail and the three untrusted cases pass. Reverted after.

Not covered by a behaviour test: the drill-down (PgDrillDownCollector) still binds the whole-day floor. It reads the offenders' rows through the offender-filtered index (#5957), not the whole floor day, so it was left alone.

local_rows and snaps

  • local_rows (PgFindingStore.GetPriorOccurrencesSql): I ran it with EXPLAIN (ANALYZE, BUFFERS) on a stand-in table with the real table's key columns and idx_analysis_findings_time. It reads exactly the window's rows for the one server through that index (12,096 rows, 333 blocks, no rows thrown away). No cause beyond a cold cache, so no change.
  • snaps was judged cold only in the issue (the extension's SkipScan reads two index entries). No change.

Lite

Lite does not run an equivalent statement. Its PLAN_REGRESSION fact reads query_store_stats in DuckDB with one last_execution_time >= window start - 14 days bound, and has no built-days table or live half with a day floor. Nothing to change.

Considered, not added

  • A fallback if the interval-length lookup itself fails inside the read's transaction. The statement reads store tables ResolveEdgeMarginAsync already reads in the same way, and no input I can name makes it fail short of a cancel or a timeout, which the read would hit anyway.

Gate

(filled in below)

Release note: analyze_server with an as_of a few hours back reads less of the Query Store interval table on a busy store, and returns the same plan regressions.

@erikdarlingdata
erikdarlingdata marked this pull request as ready for review November 21, 2026 07:03

async Task<(string Fact, long Blocks)> RunAsync()
{
await Exec(rig.Connection, "SELECT pg_stat_statements_reset()", ct);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test reliability: this is the only test that shows the collector binds the resolved floor, and it measures through pg_stat_statements. The repo already documents that instrument as unreliable for a per-test count.

  1. pg_stat_statements_reset() with no arguments clears the whole cluster. This class is deliberately not in a collection, so it runs in parallel with other live classes. StoreSizeCacheLiveTests and OverviewFleetHealthSingleFlightLiveTests scope their reset to their own database (pg_stat_statements_reset(0, $1, 0)). PgStatementStatsNoTextReadLiveTests (Darling.Tests: a full local run leaves state that fails later runs #4981) brackets its reads against exactly this kind of full reset from a neighbour.
  2. Another class can also run a full reset between RunAsync's fact and its view read. Separately, eviction can drop this test's entry: WaitRateTileReadCountLiveTests' header measured about three evictions a second under the scratch-database classes, which replay migrations. In either case the untrusted run reads 0 blocks, trusted.Blocks < 0 is false, and the test fails without any defect in the product.
  3. If the library isn't in shared_preload_libraries, CREATE EXTENSION IF NOT EXISTS still succeeds, and this reset then throws 55000. That is a hard failure, not a skip (see the same header).
  4. The margin is thin. The PR body measured 3,957 vs 3,949 blocks for the live-half scans. That is an 8-block gap inside a whole-statement sum that also includes the daily-totals half.

Suggested fix: read the bound floor directly instead. Use EXPLAIN (ANALYZE, BUFFERS) through a test seam, as ReadAsync already does. Alternatively, capture the $4 the collector binds through a seam like TestOnlyAfterBuiltDaysRead, and assert it equals FirstUnbuilt - 1h. That gives a behaviour assertion that fails on the defect and doesn't depend on the cluster-wide view.

var connection = new NpgsqlConnection(scratch.ConnectionString);
try
{
await connection.OpenAsync(ct);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trap: "A test that reads the clock twice measures the CI scheduler … One anchor owns both ends of an interval." This rig sets today from LiveClock.Now() (line 180) and end from a second read, LiveClock.AnchoredEndNotAfterNow(), which calls Now() and DateTime.UtcNow again.

Here's how it breaks without an override, if UTC midnight falls between the two reads:

  • today is N, but end is N 22:00 rather than (N-1) 22:00.
  • FirstUnbuilt becomes N-1, while the built days still end at N-3.
  • windowFloor moves up a day, so Assert.Equal(13, built.Count) sees 12 and fails.

The window is small, but it's exactly the shape the trap names, and the fix is one line: compute end first and set today = end.Date.AddDays(1). Under the pinned 00:30 pass the override makes Now() stable. I traced that case and the arithmetic holds there, because wholeBy drops T-2. So only the real-clock race is open.

public static async Task<TimeSpan> ResolveLongestIntervalSpanAsync(
NpgsqlConnection connection, NpgsqlTransaction? transaction, int serverId, DateTime readStart, DateTime end,
int commandTimeoutSeconds, CancellationToken cancellationToken)
{

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking, but worth a sentence in the PR body: the trust test was built for reads of query_store_interval_wide, and here it bounds a read of query_store_interval_latest.

LongestIntervalSpanSql builds the set of databases to check from a skip scan of query_store_interval_wide. Only those databases trigger the "no capture" and "non-positive length" fallbacks. query_store_interval_latest is kept for 15 days (DarlingRetention.QueryStoreIntervalLatestRetentionDays), and the wide table for about 10 (WideHorizonDays).

So a database can still have latest rows after its wide rows are gone, for example one dropped about 11 days ago. If it also has no health capture in the range, the trust test ignores it, and the other databases' 60-minute length is trusted. For a non-anchored pass whose first unbuilt day falls in that gap, a row from that database's longer interval that crosses the unbuilt midnight would sit below the new floor and be dropped.

That needs several things at once (a build gap old enough to reach it, a database with no captures, and an interval longer than the others), so I'm not blocking on it. It does contradict the docstring's "a database with no capture … the floor keeps a whole day … never dropped". Either narrow that sentence, or build the database set from query_store_interval_latest for this caller.

claude[bot]
claude Bot previously requested changes Nov 21, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: CHANGES REQUESTED

Claims

  • VERIFIED — The live floor is one interval length below the first unbuilt day, not a whole day — PgFactCollector.QueryPerf.cs:742 (cursor - (unbuiltReach ?? IntervalSpanMargin)), bound at :1096.
  • VERIFIED — ResolveLongestIntervalSpanAsync uses the same statement and trust test as ResolveEdgeMarginAsync, through one shared private method — QueryStoreIntervalWide.cs:794, :808, :816-838.
  • VERIFIED — An untrusted length keeps the whole-day floor, and a length is never taken as more than a day — QueryStoreIntervalWide.cs:810 (trusted < IntervalSpanMargin ? trusted : IntervalSpanMargin).
  • VERIFIED — A server interval-length change, seen in a capture or missed by the captures, leaves the whole-day floor — LongestIntervalSpanSql takes the MAX over [readStart − 2d, end + 1d], and :832 returns null when the oldest capture is newer than readStart − 1d.
  • VERIFIED — The length is looked up only when the unbuilt day sets the floor; a current store (watermark floor) does no extra read — PgFactCollector.QueryPerf.cs:766-770.
  • VERIFIED — The interval-table half is unchanged apart from the floor value, and the same rows come back — the diff has no SQL hunk; WithATrustedIntervalLength_… asserts Assert.Equal(wide.Rows, narrow.Rows).
  • VERIFIED — The trusted, untrusted (three cases) and partial-watermark tests assert what the body says — PlanRegressionAsOfLiveFloorLiveTests.cs:47-118.
  • VERIFIED — The drill-down still binds the whole-day floor — PgDrillDownCollector.Queries.cs:1360.
  • VERIFIED — Lite has no equivalent statement — no live floor or plan_regression_daily anywhere under Lite/.
  • UNVERIFIED — 24,001 vs 18,001 live-half rows read, and 3,957 vs 3,949 blocks — a live seeded rig is needed.
  • UNVERIFIED — The issue's 57,700 → 44,700 blocks on a day-partitioned store — the author's rig, not re-seeded.
  • UNVERIFIED — Revert proof: the trusted, partial and fact tests fail with the resolve step reverted — a live run is needed.
  • UNVERIFIED — The local_rows EXPLAIN (12,096 rows, 333 blocks) — the author's stand-in table.
  • UNVERIFIED — The fact test "is the test that proves the collector uses the resolved floor" — it does assert trusted.Blocks < untrusted.Blocks, but it measures through the cluster-wide pg_stat_statements with a full reset, under eviction, on an 8-block margin (inline comment on line 126). Whether it proves this reliably can't be settled from the tree.

Riskiest lines

  1. Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.QueryPerf.cs:742 — the new floor; set too high, it silently drops live rows — covered by WithATrustedIntervalLength_TheFloorIsOneIntervalBelowTheFirstUnbuiltDay_ReadsFarFewerRows_AndReturnsTheSameRows (live; the 23:40→00:10 row and the row-set equality fail on an over-narrow floor).
  2. Darling/PerformanceMonitor.Darling.Storage/QueryStoreIntervalWide.cs:810 (and the trust test at :826-835) — trusting a length that shouldn't be trusted narrows the floor below what's safe — covered by WithoutATrustedIntervalLength_TheFloorStaysAWholeDay_AndTheRowsAreTheSame for the named fallbacks. A database present in query_store_interval_latest but not in query_store_interval_wide is UNCOVERED (inline comment).
  3. Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.QueryPerf.cs:1096 — the collector actually binding the resolved floor inside the REPEATABLE READ snapshot — covered only by TheFact_OfAPassAnchoredBack_ReadsLessOfTheTableWithATrustedLength_AndReportsTheSameRegression, whose pg_stat_statements instrument can fail with no defect present; in practice UNCOVERED until it asserts the bound floor directly.

Traps near this diff

  • "A test that reads the clock twice measures the CI scheduler: the delta-seeder tests failed at 123 and 243 on unrelated PRs (#3676). One anchor owns both ends of an interval." — applies — PlanRegressionAsOfLiveFloorLiveTests.cs:180-182: today and end come from separate clock reads. If UTC midnight falls between them, Assert.Equal(13, built.Count) fails. Derive today from end.
  • "Our wrong test is a bug we introduced, never a flake" — applies — PlanRegressionAsOfLiveFloorLiveTests.cs:126: an unscoped pg_stat_statements_reset() plus a cluster-wide block count is a failure this PR introduces, both for this class and for neighbours that scope their own reset (StoreSizeCacheLiveTests.cs:136).
  • "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy." — does not apply — the drill-down's whole-day floor (PgDrillDownCollector.Queries.cs:1360) is a read bound, not a copy pinned equal; a lower floor reads a superset, and the shared statement is unchanged.
  • "Lite and Darling ship in parity (CONTRIBUTING's two-store rule): a fix to one SKU's reader, tool, alert or description without the counterpart's is a finding" — does not apply — Lite has no built-days live floor (no LiveFloor or plan_regression_daily under Lite/), and its PLAN_REGRESSION reads DuckDB query_store_stats with one window bound.
  • "Ruling (#5833): tests are load-bearing. … never answer a trap with a test that reads source text or asserts how code is spelled" — does not apply — the new tests execute the statement and compare row sets (PlanRegressionAsOfLiveFloorLiveTests.cs:70); none reads source text.

Divergence (blind read vs PR body)

  • The body says a row of a longer interval "can only exist if the server changed its interval length", and the new docstring says "a database with no capture … the floor keeps a whole day". Both overstate what the trust test checks. It builds its database set from query_store_interval_wide (about 10 days of retention), but the floor now bounds query_store_interval_latest (15 days). A database with latest rows, no wide rows and no capture is not checked, even if its interval is longer than the length the others report. This is narrow and non-blocking, but the body doesn't mention it.
  • The body presents the fact test as the proof that the collector uses the resolved floor. The test's instrument is the one WaitRateTileReadCountLiveTests documents as wrong for a per-test count, and the reset it runs is cluster-wide. The body's own measurement puts the block difference at 8 of about 3,950.
  • The new resolve statement runs inside the read's snapshot transaction, and a failure there costs the whole fact. The body documents this as deliberate under "Considered, not added", so it is not counted as a finding.

Reviewer checklist

  • Run PlanRegressionAsOfLiveFloorLiveTests with DARLING_TEST_PG set, alone and in the full parallel suite, including the pinned 23:30 and 00:30 pass. Confirm the fact test doesn't fail from pg_stat_statements eviction or resets, and that its unscoped reset doesn't disturb the neighbouring classes.
  • Confirm the CI Postgres preloads pg_stat_statements; otherwise the reset throws 55000 instead of skipping.
  • On a day-partitioned production-shaped store, re-measure an as_of pass a few hours back with the resolved floor against the whole-day floor, to confirm the partition pruning the issue claims.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: LGTM

Claims

  • VERIFIED — Live floor stays a whole day when the first unbuilt day is at or below the wide-table horizon — PgFactCollector.QueryPerf.cs:775 (guard returns wholeDay before ResolveLongestIntervalSpanAsync).
  • VERIFIED — One clock anchor: utcNow is passed in and DateTime.UtcNow is read once at the call site — PgFactCollector.QueryPerf.cs:1108; the only callers of the changed signature are that site and PlanRegressionAsOfLiveFloorLiveTests.cs.
  • VERIFIED — The bound floor is checked through a test seam, not a pg_stat_statements count — TestOnlyLiveFloorBound at PgFactCollector.QueryPerf.cs:~790, invoked just before AddWithValue; asserted in TheFact_OfAPassAnchoredBack_BindsTheResolvedFloor_AndReportsTheSameRegression.
  • UNVERIFIED — The live-Postgres tests pass — they need a Postgres rig, which this read-only tree does not have.

Riskiest lines

  1. Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.QueryPerf.cs:775 — the horizon comparison, including the margin and its boundary — covered by AFirstUnbuiltDayPastTheWideTablesHorizon_KeepsTheWholeDayFloor_AndTheSameRows (live, which asserts both sides of the boundary).
  2. Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.QueryPerf.cs:1108 — the single DateTime.UtcNow anchor and the seam call before the bind — covered by TheFact_OfAPassAnchoredBack_BindsTheResolvedFloor_AndReportsTheSameRegression.
  3. Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.QueryPerf.cs:~790 — the AsyncLocal seam is null in production, so the added cost is one null-check — covered by the same test.

Traps near this diff

  • None of the traps I checked apply — the change adds an early-return guard and a test seam, with no new SQL and no Lite counterpart (the plan-regression floor exists only in the Darling collector).

Divergence (blind read vs PR body)

None found. I did not re-read the PR body for this pass. The commit message matches the diff. The merges in the range were left alone.

Reviewer checklist

  • Run the PlanRegressionAsOfLiveFloorLiveTests suite against a real Postgres rig.

@github-actions
github-actions Bot dismissed claude[bot]’s stale review November 21, 2026 07:33

Superseded by claude[bot]'s LGTM on 4e21a06: #9234 (review)

@erikdarlingdata
erikdarlingdata merged commit 8e47828 into dev Nov 21, 2026
18 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/9225-plan-agg-floor branch November 21, 2026 07:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant