Repository navigation
Read the plan regression live half from one interval length below the first unbuilt day (#9225) - #9234
Conversation
|
|
||
| async Task<(string Fact, long Blocks)> RunAsync() | ||
| { | ||
| await Exec(rig.Connection, "SELECT pg_stat_statements_reset()", ct); |
There was a problem hiding this comment.
Test reliability: this is the only test that shows the collector binds the resolved floor, and it measures through pg_stat_statements. The repo already documents that instrument as unreliable for a per-test count.
pg_stat_statements_reset()with no arguments clears the whole cluster. This class is deliberately not in a collection, so it runs in parallel with other live classes.StoreSizeCacheLiveTestsandOverviewFleetHealthSingleFlightLiveTestsscope their reset to their own database (pg_stat_statements_reset(0, $1, 0)).PgStatementStatsNoTextReadLiveTests(Darling.Tests: a full local run leaves state that fails later runs #4981) brackets its reads against exactly this kind of full reset from a neighbour.- Another class can also run a full reset between
RunAsync's fact and its view read. Separately, eviction can drop this test's entry:WaitRateTileReadCountLiveTests' header measured about three evictions a second under the scratch-database classes, which replay migrations. In either case the untrusted run reads 0 blocks,trusted.Blocks < 0is false, and the test fails without any defect in the product. - If the library isn't in
shared_preload_libraries,CREATE EXTENSION IF NOT EXISTSstill succeeds, and this reset then throws 55000. That is a hard failure, not a skip (see the same header). - The margin is thin. The PR body measured 3,957 vs 3,949 blocks for the live-half scans. That is an 8-block gap inside a whole-statement sum that also includes the daily-totals half.
Suggested fix: read the bound floor directly instead. Use EXPLAIN (ANALYZE, BUFFERS) through a test seam, as ReadAsync already does. Alternatively, capture the $4 the collector binds through a seam like TestOnlyAfterBuiltDaysRead, and assert it equals FirstUnbuilt - 1h. That gives a behaviour assertion that fails on the defect and doesn't depend on the cluster-wide view.
| var connection = new NpgsqlConnection(scratch.ConnectionString); | ||
| try | ||
| { | ||
| await connection.OpenAsync(ct); |
There was a problem hiding this comment.
Trap: "A test that reads the clock twice measures the CI scheduler … One anchor owns both ends of an interval." This rig sets today from LiveClock.Now() (line 180) and end from a second read, LiveClock.AnchoredEndNotAfterNow(), which calls Now() and DateTime.UtcNow again.
Here's how it breaks without an override, if UTC midnight falls between the two reads:
todayis N, butendis N 22:00 rather than (N-1) 22:00.FirstUnbuiltbecomes N-1, while the built days still end at N-3.windowFloormoves up a day, soAssert.Equal(13, built.Count)sees 12 and fails.
The window is small, but it's exactly the shape the trap names, and the fix is one line: compute end first and set today = end.Date.AddDays(1). Under the pinned 00:30 pass the override makes Now() stable. I traced that case and the arithmetic holds there, because wholeBy drops T-2. So only the real-clock race is open.
| public static async Task<TimeSpan> ResolveLongestIntervalSpanAsync( | ||
| NpgsqlConnection connection, NpgsqlTransaction? transaction, int serverId, DateTime readStart, DateTime end, | ||
| int commandTimeoutSeconds, CancellationToken cancellationToken) | ||
| { |
There was a problem hiding this comment.
Non-blocking, but worth a sentence in the PR body: the trust test was built for reads of query_store_interval_wide, and here it bounds a read of query_store_interval_latest.
LongestIntervalSpanSql builds the set of databases to check from a skip scan of query_store_interval_wide. Only those databases trigger the "no capture" and "non-positive length" fallbacks. query_store_interval_latest is kept for 15 days (DarlingRetention.QueryStoreIntervalLatestRetentionDays), and the wide table for about 10 (WideHorizonDays).
So a database can still have latest rows after its wide rows are gone, for example one dropped about 11 days ago. If it also has no health capture in the range, the trust test ignores it, and the other databases' 60-minute length is trusted. For a non-anchored pass whose first unbuilt day falls in that gap, a row from that database's longer interval that crosses the unbuilt midnight would sit below the new floor and be dropped.
That needs several things at once (a build gap old enough to reach it, a database with no captures, and an interval longer than the others), so I'm not blocking on it. It does contradict the docstring's "a database with no capture … the floor keeps a whole day … never dropped". Either narrow that sentence, or build the database set from query_store_interval_latest for this caller.
There was a problem hiding this comment.
Verdict: CHANGES REQUESTED
Claims
- VERIFIED — The live floor is one interval length below the first unbuilt day, not a whole day —
PgFactCollector.QueryPerf.cs:742(cursor - (unbuiltReach ?? IntervalSpanMargin)), bound at:1096. - VERIFIED —
ResolveLongestIntervalSpanAsyncuses the same statement and trust test asResolveEdgeMarginAsync, through one shared private method —QueryStoreIntervalWide.cs:794,:808,:816-838. - VERIFIED — An untrusted length keeps the whole-day floor, and a length is never taken as more than a day —
QueryStoreIntervalWide.cs:810(trusted < IntervalSpanMargin ? trusted : IntervalSpanMargin). - VERIFIED — A server interval-length change, seen in a capture or missed by the captures, leaves the whole-day floor —
LongestIntervalSpanSqltakes the MAX over [readStart − 2d, end + 1d], and:832returns null when the oldest capture is newer than readStart − 1d. - VERIFIED — The length is looked up only when the unbuilt day sets the floor; a current store (watermark floor) does no extra read —
PgFactCollector.QueryPerf.cs:766-770. - VERIFIED — The interval-table half is unchanged apart from the floor value, and the same rows come back — the diff has no SQL hunk;
WithATrustedIntervalLength_…assertsAssert.Equal(wide.Rows, narrow.Rows). - VERIFIED — The trusted, untrusted (three cases) and partial-watermark tests assert what the body says —
PlanRegressionAsOfLiveFloorLiveTests.cs:47-118. - VERIFIED — The drill-down still binds the whole-day floor —
PgDrillDownCollector.Queries.cs:1360. - VERIFIED — Lite has no equivalent statement — no live floor or
plan_regression_dailyanywhere underLite/. - UNVERIFIED — 24,001 vs 18,001 live-half rows read, and 3,957 vs 3,949 blocks — a live seeded rig is needed.
- UNVERIFIED — The issue's 57,700 → 44,700 blocks on a day-partitioned store — the author's rig, not re-seeded.
- UNVERIFIED — Revert proof: the trusted, partial and fact tests fail with the resolve step reverted — a live run is needed.
- UNVERIFIED — The
local_rowsEXPLAIN (12,096 rows, 333 blocks) — the author's stand-in table. - UNVERIFIED — The fact test "is the test that proves the collector uses the resolved floor" — it does assert
trusted.Blocks < untrusted.Blocks, but it measures through the cluster-widepg_stat_statementswith a full reset, under eviction, on an 8-block margin (inline comment on line 126). Whether it proves this reliably can't be settled from the tree.
Riskiest lines
Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.QueryPerf.cs:742— the new floor; set too high, it silently drops live rows — covered byWithATrustedIntervalLength_TheFloorIsOneIntervalBelowTheFirstUnbuiltDay_ReadsFarFewerRows_AndReturnsTheSameRows(live; the 23:40→00:10 row and the row-set equality fail on an over-narrow floor).Darling/PerformanceMonitor.Darling.Storage/QueryStoreIntervalWide.cs:810(and the trust test at:826-835) — trusting a length that shouldn't be trusted narrows the floor below what's safe — covered byWithoutATrustedIntervalLength_TheFloorStaysAWholeDay_AndTheRowsAreTheSamefor the named fallbacks. A database present inquery_store_interval_latestbut not inquery_store_interval_wideis UNCOVERED (inline comment).Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.QueryPerf.cs:1096— the collector actually binding the resolved floor inside the REPEATABLE READ snapshot — covered only byTheFact_OfAPassAnchoredBack_ReadsLessOfTheTableWithATrustedLength_AndReportsTheSameRegression, whosepg_stat_statementsinstrument can fail with no defect present; in practice UNCOVERED until it asserts the bound floor directly.
Traps near this diff
- "A test that reads the clock twice measures the CI scheduler: the delta-seeder tests failed at 123 and 243 on unrelated PRs (#3676). One anchor owns both ends of an interval." — applies —
PlanRegressionAsOfLiveFloorLiveTests.cs:180-182:todayandendcome from separate clock reads. If UTC midnight falls between them,Assert.Equal(13, built.Count)fails. Derivetodayfromend. - "Our wrong test is a bug we introduced, never a flake" — applies —
PlanRegressionAsOfLiveFloorLiveTests.cs:126: an unscopedpg_stat_statements_reset()plus a cluster-wide block count is a failure this PR introduces, both for this class and for neighbours that scope their own reset (StoreSizeCacheLiveTests.cs:136). - "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy." — does not apply — the drill-down's whole-day floor (
PgDrillDownCollector.Queries.cs:1360) is a read bound, not a copy pinned equal; a lower floor reads a superset, and the shared statement is unchanged. - "Lite and Darling ship in parity (CONTRIBUTING's two-store rule): a fix to one SKU's reader, tool, alert or description without the counterpart's is a finding" — does not apply — Lite has no built-days live floor (no
LiveFloororplan_regression_dailyunderLite/), and its PLAN_REGRESSION reads DuckDBquery_store_statswith one window bound. - "Ruling (#5833): tests are load-bearing. … never answer a trap with a test that reads source text or asserts how code is spelled" — does not apply — the new tests execute the statement and compare row sets (
PlanRegressionAsOfLiveFloorLiveTests.cs:70); none reads source text.
Divergence (blind read vs PR body)
- The body says a row of a longer interval "can only exist if the server changed its interval length", and the new docstring says "a database with no capture … the floor keeps a whole day". Both overstate what the trust test checks. It builds its database set from
query_store_interval_wide(about 10 days of retention), but the floor now boundsquery_store_interval_latest(15 days). A database with latest rows, no wide rows and no capture is not checked, even if its interval is longer than the length the others report. This is narrow and non-blocking, but the body doesn't mention it. - The body presents the fact test as the proof that the collector uses the resolved floor. The test's instrument is the one
WaitRateTileReadCountLiveTestsdocuments as wrong for a per-test count, and the reset it runs is cluster-wide. The body's own measurement puts the block difference at 8 of about 3,950. - The new resolve statement runs inside the read's snapshot transaction, and a failure there costs the whole fact. The body documents this as deliberate under "Considered, not added", so it is not counted as a finding.
Reviewer checklist
- Run
PlanRegressionAsOfLiveFloorLiveTestswithDARLING_TEST_PGset, alone and in the full parallel suite, including the pinned 23:30 and 00:30 pass. Confirm the fact test doesn't fail frompg_stat_statementseviction or resets, and that its unscoped reset doesn't disturb the neighbouring classes. - Confirm the CI Postgres preloads
pg_stat_statements; otherwise the reset throws 55000 instead of skipping. - On a day-partitioned production-shaped store, re-measure an
as_ofpass a few hours back with the resolved floor against the whole-day floor, to confirm the partition pruning the issue claims.
There was a problem hiding this comment.
Verdict: LGTM
Claims
- VERIFIED — Live floor stays a whole day when the first unbuilt day is at or below the wide-table horizon — PgFactCollector.QueryPerf.cs:775 (guard returns wholeDay before ResolveLongestIntervalSpanAsync).
- VERIFIED — One clock anchor: utcNow is passed in and DateTime.UtcNow is read once at the call site — PgFactCollector.QueryPerf.cs:1108; the only callers of the changed signature are that site and PlanRegressionAsOfLiveFloorLiveTests.cs.
- VERIFIED — The bound floor is checked through a test seam, not a pg_stat_statements count — TestOnlyLiveFloorBound at PgFactCollector.QueryPerf.cs:~790, invoked just before AddWithValue; asserted in TheFact_OfAPassAnchoredBack_BindsTheResolvedFloor_AndReportsTheSameRegression.
- UNVERIFIED — The live-Postgres tests pass — they need a Postgres rig, which this read-only tree does not have.
Riskiest lines
- Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.QueryPerf.cs:775 — the horizon comparison, including the margin and its boundary — covered by AFirstUnbuiltDayPastTheWideTablesHorizon_KeepsTheWholeDayFloor_AndTheSameRows (live, which asserts both sides of the boundary).
- Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.QueryPerf.cs:1108 — the single DateTime.UtcNow anchor and the seam call before the bind — covered by TheFact_OfAPassAnchoredBack_BindsTheResolvedFloor_AndReportsTheSameRegression.
- Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.QueryPerf.cs:~790 — the AsyncLocal seam is null in production, so the added cost is one null-check — covered by the same test.
Traps near this diff
- None of the traps I checked apply — the change adds an early-return guard and a test seam, with no new SQL and no Lite counterpart (the plan-regression floor exists only in the Darling collector).
Divergence (blind read vs PR body)
None found. I did not re-read the PR body for this pass. The commit message matches the diff. The merges in the range were left alone.
Reviewer checklist
- Run the PlanRegressionAsOfLiveFloorLiveTests suite against a real Postgres rig.
Superseded by claude[bot]'s LGTM on 4e21a06: #9234 (review)
Claude posting for Erik Darling.
Fixes #9225
Cause
The plan regression read of a pass anchored at an instant (
PlanRegressionDailyAsOfSql) takes closed days fromplan_regression_dailyand the rest fromquery_store_interval_latest. The live half started one whole day below the first day that is not built. A built day's rows are already in the totals, so the live half drops every row of that floor day except the few that cross midnight into the first unbuilt day. But no index finds those few rows, so the whole floor day's partition was read and thrown away. For anas_ofa few hours back this was the largest read in the statement.Fix
The live floor is now one interval length below the first unbuilt day, not a whole day.
QueryStoreIntervalWide.ResolveLongestIntervalSpanAsyncfinds that length. It runs the same statement and the same trust test asResolveEdgeMarginAsync(no capture, a database with wide rows and no length, a capture that does not reach back a day before the read, a length of zero).ResolveEdgeMarginAsyncnow shares the one private method with it, so the two cannot drift.Measured
A seeded store with three servers, 17 days of intervals, closed days built through three days back, the server reporting 60 minute intervals, and a read anchored at 22:00 the previous day (heap table, parallelism off):
The diagnosis in the issue measured the same change on a day-partitioned store with one busy server: raising the floor to one hour below the first unbuilt day pruned the floor partition and took the plan from about 57,700 to 44,700 blocks, with the same answer. I did not re-seed that store.
Tests (
PlanRegressionAsOfLiveFloorLiveTests, all live, own scratch store)PgFactCollector: the same fact metadata with and without health captures (so with and without the narrower floor), and the daily statement touches fewer blocks with the trusted length (frompg_stat_statements). This is the test that proves the collector uses the resolved floor.Not covered by a behaviour test: the drill-down (
PgDrillDownCollector) still binds the whole-day floor. It reads the offenders' rows through the offender-filtered index (#5957), not the whole floor day, so it was left alone.local_rows and snaps
local_rows(PgFindingStore.GetPriorOccurrencesSql): I ran it with EXPLAIN (ANALYZE, BUFFERS) on a stand-in table with the real table's key columns andidx_analysis_findings_time. It reads exactly the window's rows for the one server through that index (12,096 rows, 333 blocks, no rows thrown away). No cause beyond a cold cache, so no change.snapswas judged cold only in the issue (the extension's SkipScan reads two index entries). No change.Lite
Lite does not run an equivalent statement. Its PLAN_REGRESSION fact reads
query_store_statsin DuckDB with onelast_execution_time >= window start - 14 daysbound, and has no built-days table or live half with a day floor. Nothing to change.Considered, not added
ResolveEdgeMarginAsyncalready reads in the same way, and no input I can name makes it fail short of a cancel or a timeout, which the read would hit anyway.Gate
(filled in below)
Release note: analyze_server with an as_of a few hours back reads less of the Query Store interval table on a busy store, and returns the same plan regressions.