Repository navigation
Darling on plain PostgreSQL: the blocking and deadlock baselines stop at the purge horizon (#6706) - #6712
Conversation
| internal static string IsContinuousAggregateSql(string view) | ||
| => $"SELECT EXISTS (SELECT 1 FROM timescaledb_information.continuous_aggregates WHERE view_schema = 'collect' AND view_name = '{view}')"; |
There was a problem hiding this comment.
Two copies of one definition — the #3684 trap in .github/REVIEW-TRAPS.md: "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy."
IsContinuousAggregateSql is byte-identical to the public helper that already exists in the assembly this method already calls:
// Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs:2142
public static string BaselineRelationIsContinuousAggregateSql(string view)
=> $"SELECT EXISTS (SELECT 1 FROM timescaledb_information.continuous_aggregates WHERE view_schema = 'collect' AND view_name = '{view}')";
Same view_schema, same view_name, same EXISTS. PgBaselineProvider already references TimescaleSupport three lines up (TimescaleSupport.BlockedProcessBaselineView, line 863) and again at line 984 (ReadBlockedBaselineSupplyFloorAsync), so the alias costs nothing. Failure mode is the trap's: the next change to the aggregate predicate — a WHERE the ownership question forces, a schema that stops being hard-coded 'collect' — lands on one copy, and the clamp keeps answering from the other with nothing failing to build.
| internal static string IsContinuousAggregateSql(string view) | |
| => $"SELECT EXISTS (SELECT 1 FROM timescaledb_information.continuous_aggregates WHERE view_schema = 'collect' AND view_name = '{view}')"; | |
| /* #3684: ONE definition of the aggregate predicate. TimescaleSupport owns it (its doc carries the | |
| "valid only once DetectAsync has answered true" condition the gate above honors); this is an alias. */ | |
| internal static string IsContinuousAggregateSql(string view) | |
| => TimescaleSupport.BaselineRelationIsContinuousAggregateSql(view); |
Worth noting the stronger version of the same point: lines 868–919 re-implement a question TimescaleSupport already answers end-to-end. ResolveMaterializationAsync (TimescaleSupport.cs:1321) is DetectAsync + that same aggregate probe, and returns null "when the relation is not one (a plain fallback view, or a store without the extension)" — which is exactly this method's ContinuousAggregatesExistSql + IsContinuousAggregateSql pair. IsStalePlainBlockedBaselineViewAsync (TimescaleSupport.cs:1133) already asks the plain-view question that way:
if (await ResolveMaterializationAsync(connection, BlockedProcessBaselineView, cancellationToken) is not null)
{
return false;
}Routing through it would leave this clamp with the pg_class existence probe (IsPlainBaselineViewSql, genuinely new here) plus one call, and would delete both new SQL constants.
| retentionDays = Convert.ToInt64(await retention.ExecuteScalarAsync(cancellationToken), System.Globalization.CultureInfo.InvariantCulture); | ||
| } | ||
|
|
||
| var horizon = DateTime.SpecifyKind(_time.GetUtcNow().UtcDateTime.AddDays(-retentionDays), DateTimeKind.Unspecified); |
There was a problem hiding this comment.
Non-blocking, and the body already names half of it: the horizon is anchored on the wall clock, while every other input of this compute is anchored on windowEnd — the cache key's own instant.
The PR body says a cached entry "is valid for the UTC day like every other input of the compute". It isn't quite like the others. windowEnd = RoundedKeyTime(...) is the key; windowStart, the clock resolve and the $2/$3 binds all derive from it, which is what lets this class's own doc claim (lines 38–42, restated at 951–956) that "the window ends on the analysis HOUR the cache keys on, so a shared entry is the answer a fresh compute would give." With the horizon on _time.GetUtcNow(), two computes inside one cached UTC day — the worker's pass at 01:00 and an MCP analyze_server at 23:00 on a cache miss, sharing BaselineCache — ask for windows 22 hours apart at the start. Whichever lands first wins the day, so a shared entry is no longer the answer a fresh compute would give. Bounded (at most the oldest day's hours) and in the safe direction, so not a correctness complaint.
windowEnd would restore it, and it is the instant the purge horizon is actually being compared against — the window the arm reads ends there, so "what did raw still hold over this window" is a question about windowEnd, not about now:
var horizon = windowEnd.AddDays(-retentionDays);That also makes the whole TimeProvider seam unnecessary: the fixed clock the six existing live tests now have to be handed (DarlingEventBaselineCoveredDaysTests.cs:278) exists only because the horizon reads a clock the test cannot set otherwise, and a windowEnd-anchored floor is already deterministic from AnalysisTime. On a default-retention store it is also exactly the "rounded down to the UTC day" variant the body offers, for free, since windowEnd for these two daily arms is a UTC midnight (RoundedDay, line 479) — windowEnd - 30d == windowStart, so the default store stops changing at all rather than losing the oldest day's early hours.
The one behavioural difference to weigh before taking it: for an analysis whose analysisTime is historical (context.TimeRangeStart is now - hoursBack, so windowEnd trails now by up to a day at the default and further on a wide hours_back), a windowEnd anchor floors the window where raw stood then rather than now, which admits days the purge has since taken. Both readings are defensible; the clock-anchored one is the more conservative, the key-anchored one is the one the cache contract is written for.
There was a problem hiding this comment.
Verdict: CHANGES REQUESTED
One substantive finding (inline): IsContinuousAggregateSql is a byte-identical second copy of TimescaleSupport.BaselineRelationIsContinuousAggregateSql — the #3684 trap. The rest reads correct.
Claims
- VERIFIED — clamp called in
ComputeBucketsAsyncright afterChooseSupplyAsync, unkeyed only —PgBaselineProvider.cs:973-974. - VERIFIED — Blocking/Deadlock only —
:861-866; they are the only arms with aloggedside (:1515,:1518), andBlockingPerMinutehas nocollection_logCTE (:1620-1631) so it needs none. - VERIFIED — raises the start to now minus the retention in force when later —
:927-928. - VERIFIED — retention is the purge's rule —
FleetRetentionDaysSql,DailySummaryHorizon.cs:137-156; both names resolve inCollectorScheduleDefaults.All(:61,:73), and neither overridesPrefixTimeColumnName(CollectorDefinitionBase.cs:34), so the purge ages raw on the column theeventsCTE filters. - VERIFIED —
EventBaselineSqluntouched, so the Lite byte-identity pin holds —:1261-1295; both CTEs still bound on$2/$3, which is what makes raising$2coherent rather than one-sided. - VERIFIED — detection is
pg_classrelkindvincollectplus thetimescaledb_informationhalf only when that view exists —:875-879,:901-919;to_regclassreads NULL on an absent schema, so an extension-free store short-circuits at:911. - VERIFIED — an aggregate changes nothing —
:916, and behaviourallyFleetPartialDatabaseFailureBandTests.cs:774, which sets a 10-day override on a real aggregate and still asserts 30 covered days (:862). - VERIFIED — new optional
TimeProvider, nothing else takes it —:155-160; the three service/web/viewer sites pass none, andPgTargetBaselineProvider'sbase(...)(:84) is unaffected. - VERIFIED — no cache key change —
CacheKeyFor,RoundedHour/RoundedDay,IsDailyCacheMetricuntouched. - VERIFIED — Lite unaffected —
collection_logis archived in the same catalog-driven set as the event tables (ArchiveService.cs:264,DuckDbInitializer.cs:881) under one 3-month horizon (RetentionService.cs:34), so its log cannot outlive its events. - VERIFIED — the merge note: both clamps present and composing, each only raising
windowStart(:974,:983-988), and #6705's retention read is private (TimescaleSupport.cs:1171). - VERIFIED — "about 77%" — arithmetic on
SUM(n) / COUNT(DISTINCT d)(:1290): 7/30 = 0.233. - VERIFIED — the default-retention note's premise — these are daily arms, so
windowEndisRoundedDay(:479), a UTC midnight, and the floor lands inside the oldest window day. - UNVERIFIED — "disabling the clamp makes both cases fail" — needs a live
DARLING_TEST_PGstore; the test isAssert.SkipWhenon it. - UNVERIFIED — the two runs (884; 94/1 skipped) — a live rig and a test host.
- UNVERIFIED — "Build: 0 errors, no new warnings" — no build available here.
Riskiest lines
Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs:974— mutateswindowStartahead of the clock resolve (:997) and the #6705 max (:985); a wrong anchor shortens every plain-view store's baseline silently — covered byLive_RetentionBelowTheWindow_CountsOnlyTheDaysTheRawTableStillHolds, which fails (3/5 where 3.0 is expected) on a clamp that does not fire.Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs:914— the aggregate escape hatch; misread it and every TimescaleDB store's Blocking window shrinks to the raw retention — covered byBlockingBaseline_OfARebuiltAggregate_CountsNoDayBeforeTheFloor_AndAStoreWithNoFloorReadsAsBefore_AgainstDevPostgres(FleetPartialDatabaseFailureBandTests.cs:862).Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs:927— the wall-clock anchor, the one input not derived from the cache key — UNCOVERED: the new test pins its clock towindowEndexactly (DarlingEventBaselineCoveredDaysTests.cs:278), so both anchors agree there.
Traps near this diff
- "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy." — applies —
:878-879is byte-for-byteTimescaleSupport.cs:2142, public, in an assembly this method already calls at:863and:984;ResolveMaterializationAsync(:1321) andIsStalePlainBlockedBaselineViewAsync(:1133) already ask the whole question. - "A retention-tiered store answers a 7-day question from a 4-day raw tier and labels the missing days quiet (#3590)." — applies, as the defect closed rather than one introduced — 30 days of
collection_log(DarlingRetention.cs:878-888) against 7 of raw events; #6705's half stays beside it (:983-988). - "Lite and Darling ship in parity: a fix to one SKU's reader without the counterpart's is a finding." — does not apply — Lite reads events over hot∪parquet views (
Lite/Analysis/BaselineProvider.cs:718-723) with the log on the same horizon, and a plain fallback view standing in for an aggregate has no DuckDB analogue. - "Never answer a trap with a test that reads source text or asserts how code is spelled." (#5833) — does not apply — the one test added is a live behaviour test that fails on the defect.
- "
job_historyis ownership-filtered whilejobsandjob_statshave noWHEREat all. A read of that view must state whose eyes it has." (#3585) — does not apply — the new read is ofcontinuous_aggregates, which this repo reads from the MCP role's paths without a visibility caveat (TimescaleSupport.cs:6664,:9203;Darling/README.md:1662singles outjob_history). On the checklist anyway: no row reads as "plain view". - "An empty analysis window is 'unavailable', not an all-clear (#3553)." — does not apply — a clamp past
windowEndyields no buckets, which theis_new/IsZeroHistorypath words as "no baseline", and that read was already empty becauseslotsgates onWHERE EXISTS (SELECT 1 FROM events)(:1284).
Divergence (blind read vs PR body)
Two, neither a correctness disagreement. "What changes" presents the plain-view detection as new work in its entirety; half of it already exists in TimescaleSupport, and the merge note's "there is no second read of the same kind to share" is true of the retention read it was written about but reads as if it settled reuse for the whole method. Only the pg_class probe is genuinely new. And "a cached entry is valid for the UTC day like every other input of the compute" does more work than the tree supports: every other input derives from windowEnd, the cache key (:957, :1003-1004), which is what the class's shared-cache claim rests on (:38-42, :951-956); the horizon is the first that moves with the wall clock inside a cache period, so two computes in one cached day no longer ask for the same window. Bounded and safe in direction, hence inline and non-blocking.
Reviewer checklist
- Run the two live classes with
blocked_process_baselinein BOTH shapes — plain fallback view and real continuous aggregate.LivePostgresStoreFixturedeliberately establishes neither (:61-66), and the aggregate shape would makeshort7read 3/5 rather than 3.0. - On a managed store that HAS the aggregate, read
timescaledb_information.continuous_aggregatesas the role the analysis pass connects as (mcp,admin) and confirm the row is visible — a blind read is indistinguishable from "plain view" at:914. - The build and the two test runs the body reports.
- On a plain-PostgreSQL store with a shortened retention, confirm the Blocking fact's wording: days that leave the window should leave the covered-day count, not return as a thinner baseline still treated as trustworthy.
… clamp's aggregate check (#6706)
There was a problem hiding this comment.
Verdict: LGTM
Claims
- VERIFIED —
ClampToPlainViewPurgeHorizonAsyncnow asksTimescaleSupport.ResolveMaterializationAsyncinstead of its ownto_regclass+continuous_aggregatesstatements —Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs:905(diff vs21b2c8050..f6b7a8d3b). - VERIFIED —
ContinuousAggregatesExistSqlandIsContinuousAggregateSqlare deleted; no other reference existed — confirmed bygrep -rn "IsContinuousAggregateSql\|ContinuousAggregatesExistSql", which now returns only the unrelated, pre-existingTimescaleSupport.BaselineRelationIsContinuousAggregateSql. - UNVERIFIED — "Targeted classes … pass on a throwaway rig: 698 and 21 tests, 0 failed, 0 skipped" — a live-rig test run; not checkable from a read-only tree.
Riskiest lines
Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs:905— ifResolveMaterializationAsyncever answered non-null for a genuine plain view, the retention clamp would silently stop firing, reintroducing #6706 — covered byDarlingEventBaselineCoveredDaysTests.Live_RetentionBelowTheWindow_CountsOnlyTheDaysTheRawTableStillHolds, which fails (mean 0.6 instead of 3.0) when the clamp doesn't apply.Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs:905— the other direction, a real continuous aggregate misread as plain — same escape-hatch risk the prior review flagged at this spot, now routed through the shared, already-exercisedResolveMaterializationAsync(used byIsStalePlainBlockedBaselineViewAsync, the materialization-hole repair paths, etc.) rather than a bespoke pair of statements — covered byFleetPartialDatabaseFailureBandTests.BlockingBaseline_OfARebuiltAggregate_CountsNoDayBeforeTheFloor_AndAStoreWithNoFloorReadsAsBefore_AgainstDevPostgres, which runsPgBaselineProvider.GetBucketMapAsyncagainst a real, established continuous aggregate.Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs:873-874(IsPlainBaselineViewSql) — untouched by this commit, still the first gate before the aggregate check; no regression here, included only because it borders the changed lines.
Traps near this diff
- "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy." — applied and resolved by this commit — the duplicate
IsContinuousAggregateSql/ContinuousAggregatesExistSqlpair (byte-identical toTimescaleSupport.BaselineRelationIsContinuousAggregateSqlper the prior review) is deleted in favor of the sharedTimescaleSupport.ResolveMaterializationAsyncatPgBaselineProvider.cs:905. - "Lite and Darling ship in parity…" — does not apply — this commit touches only Darling's PostgreSQL/TimescaleDB-specific aggregate detection; Lite has no TimescaleDB analogue.
- "Never answer a trap with a test that reads source text or asserts how code is spelled." (#5833) — does not apply — no test was added or changed by this commit; the existing behaviour tests above are unaffected by the refactor.
Divergence (blind read vs PR body)
None — the "Round 1 fixes" bullet's account of this commit (what was deleted, why, and that nothing else referenced the deleted symbols) matches the diff exactly.
Reviewer checklist
- Confirm the reported live-rig run (698 + 21 tests, 0 failed, 0 skipped) actually executed against this commit, and the build has 0 errors / no new warnings as claimed.
Claude posting for Erik Darling. Addressed in f6b7a8d; round 2 LGTM.
Claude posting for Erik Darling.
Fixes #6706.
Why
The Blocking and Deadlock baselines count events per covered day, where a day is covered by a successful
collection_logrun.collection_logis kept twice the base retention. On a store without TimescaleDB the event side (blocked_process_baseline,deadlock_baseline) is a plain view over the raw table, which is purged at the collector's retention. With retention below 30 days, every day between the purge horizon and 30 days back has logged runs and no events, so it counts as a measured zero. At 7 days about 23 of 30 days read as quiet, the mean falls about 77%, and normal blocking or deadlocks flag as anomalies. It lasts as long as the shorter retention is in force.What changes
PgBaselineProvider.ClampToPlainViewPurgeHorizonAsync(Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs), called inComputeBucketsAsyncright afterChooseSupplyAsyncfor unkeyed computes. For Blocking and Deadlock only, when the arm's relation is an ordinary view (not a continuous aggregate, not absent), it raises the window start ($2) to now minus the collector's retention in force, when that is later. Retention comes fromDarlingRetentionHorizons.FleetRetentionDaysSql(collector), the read the purge uses (fleet override else default).EventBaselineSql's text is unchanged (the byte-identity pin with Lite holds).pg_classrelkindvin schemacollect(the only new check; a continuous aggregate is also relkindv), and not a continuous aggregate, which reusesTimescaleSupport.ResolveMaterializationAsync(null for a plain view or a store without the extension). With a continuous aggregate (35-day aggregates) nothing changes.TimeProvider? timeProvider = null(defaults toTimeProvider.System). Nothing else takes it.StoredEventCopies(Lite/Analysis/BaselineProvider.cs:718-723), over views that include the parquet archive, andv_collection_logis archived in the same set (Lite/Database/DuckDbInitializer.cs:881,Lite/Services/ArchiveService.cs:264). Lite has one retention horizon for every archive,RetentionService.ArchiveRetentionMonths = 3(Lite/Services/RetentionService.cs:34), so events and the log both outlive the 30-day window and the log cannot reach past the events.Default-retention note (differs slightly from "no change"): the window ends at the UTC midnight and the floor is
now - 30d, so at default retention the floor is later than the window start by the time of day. Only the hours of the oldest window day before that time drop out of the covered days. That is correct, since the purge may already have removed those hours' events, but a default store's oldest-day sample counts can drop by one for those hours. If the coordinator wants the floor rounded down to the UTC day instead, it is a one-line change (a partial-day undercount returns for retention below 30).Round 1 fixes
ClampToPlainViewPurgeHorizonAsyncnow asksTimescaleSupport.ResolveMaterializationAsyncinstead of its ownto_regclass+continuous_aggregatesstatements (The MCP trend reader's tier ladder and hourly SQL were pinned EQUAL to the Storage definitions #3666 moved them to, which fails only after two copies drift — they are now aliases; and the Query Store duration chart, the one Performance Trends chart left untitled, says what served it and where the rollup's floor cut the window's head (#3653) #3684, alias do not copy).ContinuousAggregatesExistSqlandIsContinuousAggregateSqlare deleted; no other reference existed. Targeted classes (event baseline covered days, baseline provider, Timescale, Census, Ratchet, baseline supply, and the neighbouring live baseline classes) pass on a throwaway rig: 698 and 21 tests, 0 failed, 0 skipped.windowEndinstead of the wall clock. An analysis with anas_ofin the past has awindowEnddays behind now; awindowEndanchor would floor its window where raw stood then, and so read the days the purge has since taken as quiet days, which is Darling on plain PostgreSQL: a blocked-process or deadlock retention under 30 days makes the anomaly baseline read the purged days as quiet #6706's bug again for every historical run. The wall clock is what the purge measures from. The cost is real and bounded: see the Cache bullet above.Test plan
DarlingEventBaselineCoveredDaysLiveTestsnow gives the provider a fixed clock (Mar 4 2026 00:00 UTC, the window end); all existing assertions stand unchanged.Live_RetentionBelowTheWindow_CountsOnlyTheDaysTheRawTableStillHolds(blocking and deadlock): retention 7 days with logged runs back 35 days and events only on the last 7 days; sample_count is 1 and mean 3.0 for Tuesday 14:00 (Mar 3 alone). Same data at the default 30 days: 5 days, mean 0.6. Disabling the clamp makes both cases fail (mean 0.6 where 3.0 is expected); restored.DARLING_TEST_PG):*DarlingEventBaselineCoveredDays*,*BaselineProvider*,*Baseline*,*Timescale*,*Census*,*Ratchet*,*AnalysisShutdownResidue*,*LocalClockBucketKey*: 884 tests, 0 failed. Also the other classes that build a provider and read these arms (AnomalyDetectorErrorGuardLive,DarlingEventBaselineCoverageReadPlanShape,EventTimeAnalysisReadsLive,CollectionLogSegmentByLive,SharedBaselineCache,MeasurementContractCensus,PgTargetBlocking,CountFamilyZeroHistory,AnomalyNullWindow): 94 tests, 0 failed, 1 skipped.pre-push.sh, the full suite (CI decides). Build: 0 errors; no new warnings (2 pre-existing, in other files).Merge note
#6705 (
fix/6690-blocking-baseline-self-waits) inserts its supply-floor clamp at the same spot inComputeBucketsAsync(right after theif (keys is null)block that callsChooseSupplyAsync). Whichever merges second gets a trivial adjacent-lines conflict; keep both clamps. They compose: each only raiseswindowStart. #6705's retention read is private to its floor recorder, so this PR usesDarlingRetentionHorizons.FleetRetentionDaysSqldirectly; there is no second read of the same kind to share.Considered, not added
CHANGELOG
SECTION: Fixed
ENTRY:
REF:
[Darling on plain PostgreSQL: the blocking and deadlock baselines stop at the purge horizon (#6706) #6712]: Darling on plain PostgreSQL: the blocking and deadlock baselines stop at the purge horizon (#6706) #6712