Skip to content

Darling on plain PostgreSQL: the blocking and deadlock baselines stop at the purge horizon (#6706) - #6712

Merged
erikdarlingdata merged 3 commits into
devfrom
fix/6706-plain-pg-event-floor
Oct 24, 2026
Merged

erikdarlingdata merged 3 commits into
devfrom
fix/6706-plain-pg-event-floor

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Claude posting for Erik Darling.

Fixes #6706.

Why

The Blocking and Deadlock baselines count events per covered day, where a day is covered by a successful collection_log run. collection_log is kept twice the base retention. On a store without TimescaleDB the event side (blocked_process_baseline, deadlock_baseline) is a plain view over the raw table, which is purged at the collector's retention. With retention below 30 days, every day between the purge horizon and 30 days back has logged runs and no events, so it counts as a measured zero. At 7 days about 23 of 30 days read as quiet, the mean falls about 77%, and normal blocking or deadlocks flag as anomalies. It lasts as long as the shorter retention is in force.

What changes

  • PgBaselineProvider.ClampToPlainViewPurgeHorizonAsync (Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs), called in ComputeBucketsAsync right after ChooseSupplyAsync for unkeyed computes. For Blocking and Deadlock only, when the arm's relation is an ordinary view (not a continuous aggregate, not absent), it raises the window start ($2) to now minus the collector's retention in force, when that is later. Retention comes from DarlingRetentionHorizons.FleetRetentionDaysSql(collector), the read the purge uses (fleet override else default). EventBaselineSql's text is unchanged (the byte-identity pin with Lite holds).
  • Plain-view detection: pg_class relkind v in schema collect (the only new check; a continuous aggregate is also relkind v), and not a continuous aggregate, which reuses TimescaleSupport.ResolveMaterializationAsync (null for a plain view or a store without the extension). With a continuous aggregate (35-day aggregates) nothing changes.
  • Clock: new optional constructor parameter TimeProvider? timeProvider = null (defaults to TimeProvider.System). Nothing else takes it.
  • Cache: no key change. The per-(server, metric, UTC day) cache key already carries the window end; the floor is computed at compute time from the wall clock, which is what the purge measures from. Within one cache period the floor moves with the clock, so two computes for the same key can ask for slightly different windows. The direction is safe: an entry computed earlier counted only days raw still held at that time. A retention change shows at the next day's compute.
  • Lite: not affected. Lite's Blocking and Deadlock arms read events through StoredEventCopies (Lite/Analysis/BaselineProvider.cs:718-723), over views that include the parquet archive, and v_collection_log is archived in the same set (Lite/Database/DuckDbInitializer.cs:881, Lite/Services/ArchiveService.cs:264). Lite has one retention horizon for every archive, RetentionService.ArchiveRetentionMonths = 3 (Lite/Services/RetentionService.cs:34), so events and the log both outlive the 30-day window and the log cannot reach past the events.

Default-retention note (differs slightly from "no change"): the window ends at the UTC midnight and the floor is now - 30d, so at default retention the floor is later than the window start by the time of day. Only the hours of the oldest window day before that time drop out of the covered days. That is correct, since the purge may already have removed those hours' events, but a default store's oldest-day sample counts can drop by one for those hours. If the coordinator wants the floor rounded down to the UTC day instead, it is a one-line change (a partial-day undercount returns for retention below 30).

Round 1 fixes

Test plan

  • DarlingEventBaselineCoveredDaysLiveTests now gives the provider a fixed clock (Mar 4 2026 00:00 UTC, the window end); all existing assertions stand unchanged.
  • New Live_RetentionBelowTheWindow_CountsOnlyTheDaysTheRawTableStillHolds (blocking and deadlock): retention 7 days with logged runs back 35 days and events only on the last 7 days; sample_count is 1 and mean 3.0 for Tuesday 14:00 (Mar 3 alone). Same data at the default 30 days: 5 days, mean 0.6. Disabling the clamp makes both cases fail (mean 0.6 where 3.0 is expected); restored.
  • Classes run on a throwaway PostgreSQL (DARLING_TEST_PG): *DarlingEventBaselineCoveredDays*, *BaselineProvider*, *Baseline*, *Timescale*, *Census*, *Ratchet*, *AnalysisShutdownResidue*, *LocalClockBucketKey*: 884 tests, 0 failed. Also the other classes that build a provider and read these arms (AnomalyDetectorErrorGuardLive, DarlingEventBaselineCoverageReadPlanShape, EventTimeAnalysisReadsLive, CollectionLogSegmentByLive, SharedBaselineCache, MeasurementContractCensus, PgTargetBlocking, CountFamilyZeroHistory, AnomalyNullWindow): 94 tests, 0 failed, 1 skipped.
  • Not run: pre-push.sh, the full suite (CI decides). Build: 0 errors; no new warnings (2 pre-existing, in other files).

Merge note

#6705 (fix/6690-blocking-baseline-self-waits) inserts its supply-floor clamp at the same spot in ComputeBucketsAsync (right after the if (keys is null) block that calls ChooseSupplyAsync). Whichever merges second gets a trivial adjacent-lines conflict; keep both clamps. They compose: each only raises windowStart. #6705's retention read is private to its floor recorder, so this PR uses DarlingRetentionHorizons.FleetRetentionDaysSql directly; there is no second read of the same kind to share.

Considered, not added

  • A cached daily entry does not notice a retention change until the next UTC day: same as every other config input of the compute.

CHANGELOG

SECTION: Fixed
ENTRY:

@erikdarlingdata
erikdarlingdata marked this pull request as ready for review October 24, 2026 11:16
Comment on lines +878 to +879
internal static string IsContinuousAggregateSql(string view)
=> $"SELECT EXISTS (SELECT 1 FROM timescaledb_information.continuous_aggregates WHERE view_schema = 'collect' AND view_name = '{view}')";

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two copies of one definition — the #3684 trap in .github/REVIEW-TRAPS.md: "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy."

IsContinuousAggregateSql is byte-identical to the public helper that already exists in the assembly this method already calls:

// Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs:2142
public static string BaselineRelationIsContinuousAggregateSql(string view)
    => $"SELECT EXISTS (SELECT 1 FROM timescaledb_information.continuous_aggregates WHERE view_schema = 'collect' AND view_name = '{view}')";

Same view_schema, same view_name, same EXISTS. PgBaselineProvider already references TimescaleSupport three lines up (TimescaleSupport.BlockedProcessBaselineView, line 863) and again at line 984 (ReadBlockedBaselineSupplyFloorAsync), so the alias costs nothing. Failure mode is the trap's: the next change to the aggregate predicate — a WHERE the ownership question forces, a schema that stops being hard-coded 'collect' — lands on one copy, and the clamp keeps answering from the other with nothing failing to build.

Suggested change
internal static string IsContinuousAggregateSql(string view)
=> $"SELECT EXISTS (SELECT 1 FROM timescaledb_information.continuous_aggregates WHERE view_schema = 'collect' AND view_name = '{view}')";
/* #3684: ONE definition of the aggregate predicate. TimescaleSupport owns it (its doc carries the
"valid only once DetectAsync has answered true" condition the gate above honors); this is an alias. */
internal static string IsContinuousAggregateSql(string view)
=> TimescaleSupport.BaselineRelationIsContinuousAggregateSql(view);

Worth noting the stronger version of the same point: lines 868–919 re-implement a question TimescaleSupport already answers end-to-end. ResolveMaterializationAsync (TimescaleSupport.cs:1321) is DetectAsync + that same aggregate probe, and returns null "when the relation is not one (a plain fallback view, or a store without the extension)" — which is exactly this method's ContinuousAggregatesExistSql + IsContinuousAggregateSql pair. IsStalePlainBlockedBaselineViewAsync (TimescaleSupport.cs:1133) already asks the plain-view question that way:

if (await ResolveMaterializationAsync(connection, BlockedProcessBaselineView, cancellationToken) is not null)
{
    return false;
}

Routing through it would leave this clamp with the pg_class existence probe (IsPlainBaselineViewSql, genuinely new here) plus one call, and would delete both new SQL constants.

retentionDays = Convert.ToInt64(await retention.ExecuteScalarAsync(cancellationToken), System.Globalization.CultureInfo.InvariantCulture);
}

var horizon = DateTime.SpecifyKind(_time.GetUtcNow().UtcDateTime.AddDays(-retentionDays), DateTimeKind.Unspecified);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking, and the body already names half of it: the horizon is anchored on the wall clock, while every other input of this compute is anchored on windowEnd — the cache key's own instant.

The PR body says a cached entry "is valid for the UTC day like every other input of the compute". It isn't quite like the others. windowEnd = RoundedKeyTime(...) is the key; windowStart, the clock resolve and the $2/$3 binds all derive from it, which is what lets this class's own doc claim (lines 38–42, restated at 951–956) that "the window ends on the analysis HOUR the cache keys on, so a shared entry is the answer a fresh compute would give." With the horizon on _time.GetUtcNow(), two computes inside one cached UTC day — the worker's pass at 01:00 and an MCP analyze_server at 23:00 on a cache miss, sharing BaselineCache — ask for windows 22 hours apart at the start. Whichever lands first wins the day, so a shared entry is no longer the answer a fresh compute would give. Bounded (at most the oldest day's hours) and in the safe direction, so not a correctness complaint.

windowEnd would restore it, and it is the instant the purge horizon is actually being compared against — the window the arm reads ends there, so "what did raw still hold over this window" is a question about windowEnd, not about now:

var horizon = windowEnd.AddDays(-retentionDays);

That also makes the whole TimeProvider seam unnecessary: the fixed clock the six existing live tests now have to be handed (DarlingEventBaselineCoveredDaysTests.cs:278) exists only because the horizon reads a clock the test cannot set otherwise, and a windowEnd-anchored floor is already deterministic from AnalysisTime. On a default-retention store it is also exactly the "rounded down to the UTC day" variant the body offers, for free, since windowEnd for these two daily arms is a UTC midnight (RoundedDay, line 479) — windowEnd - 30d == windowStart, so the default store stops changing at all rather than losing the oldest day's early hours.

The one behavioural difference to weigh before taking it: for an analysis whose analysisTime is historical (context.TimeRangeStart is now - hoursBack, so windowEnd trails now by up to a day at the default and further on a wide hours_back), a windowEnd anchor floors the window where raw stood then rather than now, which admits days the purge has since taken. Both readings are defensible; the clock-anchored one is the more conservative, the key-anchored one is the one the cache contract is written for.

claude[bot]
claude Bot previously requested changes Oct 24, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: CHANGES REQUESTED

One substantive finding (inline): IsContinuousAggregateSql is a byte-identical second copy of TimescaleSupport.BaselineRelationIsContinuousAggregateSql — the #3684 trap. The rest reads correct.

Claims

  • VERIFIED — clamp called in ComputeBucketsAsync right after ChooseSupplyAsync, unkeyed only — PgBaselineProvider.cs:973-974.
  • VERIFIED — Blocking/Deadlock only — :861-866; they are the only arms with a logged side (:1515, :1518), and BlockingPerMinute has no collection_log CTE (:1620-1631) so it needs none.
  • VERIFIED — raises the start to now minus the retention in force when later — :927-928.
  • VERIFIED — retention is the purge's rule — FleetRetentionDaysSql, DailySummaryHorizon.cs:137-156; both names resolve in CollectorScheduleDefaults.All (:61, :73), and neither overrides PrefixTimeColumnName (CollectorDefinitionBase.cs:34), so the purge ages raw on the column the events CTE filters.
  • VERIFIED — EventBaselineSql untouched, so the Lite byte-identity pin holds — :1261-1295; both CTEs still bound on $2/$3, which is what makes raising $2 coherent rather than one-sided.
  • VERIFIED — detection is pg_class relkind v in collect plus the timescaledb_information half only when that view exists — :875-879, :901-919; to_regclass reads NULL on an absent schema, so an extension-free store short-circuits at :911.
  • VERIFIED — an aggregate changes nothing — :916, and behaviourally FleetPartialDatabaseFailureBandTests.cs:774, which sets a 10-day override on a real aggregate and still asserts 30 covered days (:862).
  • VERIFIED — new optional TimeProvider, nothing else takes it — :155-160; the three service/web/viewer sites pass none, and PgTargetBaselineProvider's base(...) (:84) is unaffected.
  • VERIFIED — no cache key change — CacheKeyFor, RoundedHour/RoundedDay, IsDailyCacheMetric untouched.
  • VERIFIED — Lite unaffected — collection_log is archived in the same catalog-driven set as the event tables (ArchiveService.cs:264, DuckDbInitializer.cs:881) under one 3-month horizon (RetentionService.cs:34), so its log cannot outlive its events.
  • VERIFIED — the merge note: both clamps present and composing, each only raising windowStart (:974, :983-988), and #6705's retention read is private (TimescaleSupport.cs:1171).
  • VERIFIED — "about 77%" — arithmetic on SUM(n) / COUNT(DISTINCT d) (:1290): 7/30 = 0.233.
  • VERIFIED — the default-retention note's premise — these are daily arms, so windowEnd is RoundedDay (:479), a UTC midnight, and the floor lands inside the oldest window day.
  • UNVERIFIED — "disabling the clamp makes both cases fail" — needs a live DARLING_TEST_PG store; the test is Assert.SkipWhen on it.
  • UNVERIFIED — the two runs (884; 94/1 skipped) — a live rig and a test host.
  • UNVERIFIED — "Build: 0 errors, no new warnings" — no build available here.

Riskiest lines

  1. Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs:974 — mutates windowStart ahead of the clock resolve (:997) and the #6705 max (:985); a wrong anchor shortens every plain-view store's baseline silently — covered by Live_RetentionBelowTheWindow_CountsOnlyTheDaysTheRawTableStillHolds, which fails (3/5 where 3.0 is expected) on a clamp that does not fire.
  2. Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs:914 — the aggregate escape hatch; misread it and every TimescaleDB store's Blocking window shrinks to the raw retention — covered by BlockingBaseline_OfARebuiltAggregate_CountsNoDayBeforeTheFloor_AndAStoreWithNoFloorReadsAsBefore_AgainstDevPostgres (FleetPartialDatabaseFailureBandTests.cs:862).
  3. Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs:927 — the wall-clock anchor, the one input not derived from the cache key — UNCOVERED: the new test pins its clock to windowEnd exactly (DarlingEventBaselineCoveredDaysTests.cs:278), so both anchors agree there.

Traps near this diff

  • "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy." — applies — :878-879 is byte-for-byte TimescaleSupport.cs:2142, public, in an assembly this method already calls at :863 and :984; ResolveMaterializationAsync (:1321) and IsStalePlainBlockedBaselineViewAsync (:1133) already ask the whole question.
  • "A retention-tiered store answers a 7-day question from a 4-day raw tier and labels the missing days quiet (#3590)." — applies, as the defect closed rather than one introduced — 30 days of collection_log (DarlingRetention.cs:878-888) against 7 of raw events; #6705's half stays beside it (:983-988).
  • "Lite and Darling ship in parity: a fix to one SKU's reader without the counterpart's is a finding." — does not apply — Lite reads events over hot∪parquet views (Lite/Analysis/BaselineProvider.cs:718-723) with the log on the same horizon, and a plain fallback view standing in for an aggregate has no DuckDB analogue.
  • "Never answer a trap with a test that reads source text or asserts how code is spelled." (#5833) — does not apply — the one test added is a live behaviour test that fails on the defect.
  • "job_history is ownership-filtered while jobs and job_stats have no WHERE at all. A read of that view must state whose eyes it has." (#3585) — does not apply — the new read is of continuous_aggregates, which this repo reads from the MCP role's paths without a visibility caveat (TimescaleSupport.cs:6664, :9203; Darling/README.md:1662 singles out job_history). On the checklist anyway: no row reads as "plain view".
  • "An empty analysis window is 'unavailable', not an all-clear (#3553)." — does not apply — a clamp past windowEnd yields no buckets, which the is_new/IsZeroHistory path words as "no baseline", and that read was already empty because slots gates on WHERE EXISTS (SELECT 1 FROM events) (:1284).

Divergence (blind read vs PR body)

Two, neither a correctness disagreement. "What changes" presents the plain-view detection as new work in its entirety; half of it already exists in TimescaleSupport, and the merge note's "there is no second read of the same kind to share" is true of the retention read it was written about but reads as if it settled reuse for the whole method. Only the pg_class probe is genuinely new. And "a cached entry is valid for the UTC day like every other input of the compute" does more work than the tree supports: every other input derives from windowEnd, the cache key (:957, :1003-1004), which is what the class's shared-cache claim rests on (:38-42, :951-956); the horizon is the first that moves with the wall clock inside a cache period, so two computes in one cached day no longer ask for the same window. Bounded and safe in direction, hence inline and non-blocking.

Reviewer checklist

  • Run the two live classes with blocked_process_baseline in BOTH shapes — plain fallback view and real continuous aggregate. LivePostgresStoreFixture deliberately establishes neither (:61-66), and the aggregate shape would make short7 read 3/5 rather than 3.0.
  • On a managed store that HAS the aggregate, read timescaledb_information.continuous_aggregates as the role the analysis pass connects as (mcp, admin) and confirm the row is visible — a blind read is indistinguishable from "plain view" at :914.
  • The build and the two test runs the body reports.
  • On a plain-PostgreSQL store with a shortened retention, confirm the Blocking fact's wording: days that leave the window should leave the covered-day count, not return as a thinner baseline still treated as trustworthy.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: LGTM

Claims

  • VERIFIED — ClampToPlainViewPurgeHorizonAsync now asks TimescaleSupport.ResolveMaterializationAsync instead of its own to_regclass + continuous_aggregates statements — Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs:905 (diff vs 21b2c8050..f6b7a8d3b).
  • VERIFIED — ContinuousAggregatesExistSql and IsContinuousAggregateSql are deleted; no other reference existed — confirmed by grep -rn "IsContinuousAggregateSql\|ContinuousAggregatesExistSql", which now returns only the unrelated, pre-existing TimescaleSupport.BaselineRelationIsContinuousAggregateSql.
  • UNVERIFIED — "Targeted classes … pass on a throwaway rig: 698 and 21 tests, 0 failed, 0 skipped" — a live-rig test run; not checkable from a read-only tree.

Riskiest lines

  1. Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs:905 — if ResolveMaterializationAsync ever answered non-null for a genuine plain view, the retention clamp would silently stop firing, reintroducing #6706 — covered by DarlingEventBaselineCoveredDaysTests.Live_RetentionBelowTheWindow_CountsOnlyTheDaysTheRawTableStillHolds, which fails (mean 0.6 instead of 3.0) when the clamp doesn't apply.
  2. Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs:905 — the other direction, a real continuous aggregate misread as plain — same escape-hatch risk the prior review flagged at this spot, now routed through the shared, already-exercised ResolveMaterializationAsync (used by IsStalePlainBlockedBaselineViewAsync, the materialization-hole repair paths, etc.) rather than a bespoke pair of statements — covered by FleetPartialDatabaseFailureBandTests.BlockingBaseline_OfARebuiltAggregate_CountsNoDayBeforeTheFloor_AndAStoreWithNoFloorReadsAsBefore_AgainstDevPostgres, which runs PgBaselineProvider.GetBucketMapAsync against a real, established continuous aggregate.
  3. Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs:873-874 (IsPlainBaselineViewSql) — untouched by this commit, still the first gate before the aggregate check; no regression here, included only because it borders the changed lines.

Traps near this diff

  • "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy." — applied and resolved by this commit — the duplicate IsContinuousAggregateSql/ContinuousAggregatesExistSql pair (byte-identical to TimescaleSupport.BaselineRelationIsContinuousAggregateSql per the prior review) is deleted in favor of the shared TimescaleSupport.ResolveMaterializationAsync at PgBaselineProvider.cs:905.
  • "Lite and Darling ship in parity…" — does not apply — this commit touches only Darling's PostgreSQL/TimescaleDB-specific aggregate detection; Lite has no TimescaleDB analogue.
  • "Never answer a trap with a test that reads source text or asserts how code is spelled." (#5833) — does not apply — no test was added or changed by this commit; the existing behaviour tests above are unaffected by the refactor.

Divergence (blind read vs PR body)

None — the "Round 1 fixes" bullet's account of this commit (what was deleted, why, and that nothing else referenced the deleted symbols) matches the diff exactly.

Reviewer checklist

  • Confirm the reported live-rig run (698 + 21 tests, 0 failed, 0 skipped) actually executed against this commit, and the build has 0 errors / no new warnings as claimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant