Skip to content

Brains-review campaign: deferred structural residue (from #3538 / #3539 / #3540 / #3541) #3653

Closed

Description

@erikdarlingdata

Status: a work queue, being worked easiest-first. Nothing here waits on a ruling or a prerequisite — no ruling from the maintainer is pending, no prerequisite PR is open, and every item below is independent of the others. Each item is a lane of its own (S/M for the mechanical ones, multi-day for the four marked structural). It stays open until its checklist is empty; items marked structural carry a one-line reason for staying and are the last to go.

The 2026-09 brains-review campaign (five adversarial lanes graded against the OtterTune corpus; wave 1 #3523–#3563, wave 2 #3538–#3541 + #3526/#3540 keystone) closed its four lane issues on 2026-09-18 with 21 wave-2 PRs merged in one day. This issue holds what was deliberately deferred — every item below is structural or multi-day by the reviews' own sizing, or is a frozen-twin mirror batched here so it lands as one faithful diff. Nothing here is forgotten; each line names its evidence and its shape. Measured inputs (sanitized) are in the wave-2 measurement record on the design bus (pm-issue-queue-inbox/design/), filed beside the PG-analysis v1 plan.

Analysis engine (from #3538)

Alert semantics (from #3539)

Measurement layer (from #3540)

MCP payload contract (from #3541)

Frozen-twin mirror batch (deprecated/Dashboard, bug-fix support — one lane, smallest faithful diff each)

Batch landed in #3658 (c7ca1ad): items 1–4 and the anomaly-detector markers; the Waits.cs divisor stays declined and open as stated. compare_analysis ±0.1 band / confidence_basis deliberately not mirrored (semantics upgrades, not bug fixes on a frozen twin).

Not here

Activity

  1. erikdarlingdata commented on Sep 19, 2026

    @erikdarlingdata
    OwnerAuthor

    Dispatch table — 2026-09-19 (every open item RUNS or names its blocker; "parked" is retired). Edited in place as lanes land. Vocabulary: RUNNING (branch) → MERGED #N; QUEUED (wave N) = dispatches tonight when the named file/rung frees; BLOCKED: lane 9 — RETIRED at 03:45Z: feat/3542-pg-anomalies-baselines merged as #3689 (f887409), so its fence (AnomalyThresholds, MetricNames, the PgTarget anomaly/baseline/Cpu files and their tests) is open; the items it held are re-classified below; BLOCKED: ruling = one question the maintainer answers in one line; BLOCKED: dependency = a named prerequisite; DECLINED = stays as marked.

    Lanes are file-disjoint by construction (one lane per file set); each PR is PARTIAL #3653, review-dark protocol (no auto-merge arm; coordinator reads the diff and arms). A merge hold is in force until 04:00Z for the nightly cut. Merge hold lifted 04:00Z; dev green at 829773d. Wave 1 dispatched 03:58Z: fix/3653-tempdb-persistence, fix/3653-trend-rates, fix/3653-running-jobs-name, fix/3653-identity-epoch.

    # Item Disposition PR / blocker
    1 A8 Peak-vs-per-sample baseline bias BLOCKED: ruling (lane 9 merged; file fence gone.) Q1: first slice = gate every detector on window peak AND window mean under the existing cutoffs (no new storage; IO moves off AVG to the same pair) — approve, or go straight to per-window-peak-per-bucket baselines (a new aggregate family)?
    2 A10 Config → outcome attribution BLOCKED: ruling Q2: approve CONFIG_CHANGED as an analysis finding on both SKUs, rooted at Information, computed by ComparePeriodsAsync over ±4 h around each server_config_changes row inside the pass window (database config + trace flags as slice two) — yes, or a different surface (e.g. a compare tool argument)? A worker cannot pick the surface: it is a new fact key + story on the SQL Server engine and the worker's AN3 block.
    3 A9 structural half SPLIT MERGED #3693 (97de1b3) = RUNNING_JOBS carries the job name (Fact.ObjectName, both collectors; the advice names it). Recurrence scoring + moving maintenance window: BLOCKED: ruling Q3: a finding recurring at the same hour×dow for ≥3 consecutive weeks — DISCOUNT its severity, or LABEL it "recurring at this hour" at unchanged severity? (lane 9 merged; no file fence remains)
    4 BLOCKING_EVENTS scorer (10, 50) vs alerting (5, 20) BLOCKED: dependency needs the fleet read the item says to decide on: per-server distribution of the BLOCKING_EVENTS fact value per 4 h analysis window over 14 d (p50/p90/p99) — the monitor seat's read; workers have no production access.
    5 Wait-profile ratio ramp / DefaultRatioThreshold; scheduler-count normalisation BLOCKED: dependency needs the extra fleet wait-pass column (per-server-window ratio distribution joined to scheduler_count) — the monitor seat's read; AnomalyThresholds.cs:79 is free now that lane 9 merged, but the number cannot be written without the measurement.
    6 A5 Persistence-gate rollout SPLIT MERGED #3692 (7e216ad) = tempdb Space behind the shared AlertPersistenceGate, K = 3 consecutive COLLECTIONS (keyed on collection_time), persisted under its own metric name, no-data freezes the gate. Blocking Wait Time: BLOCKED: ruling Q4: fire on a SINGLE snapshot when blocking wait ≥ N× the configured Blocking Wait Time threshold (N = ?), else require K = 3 consecutive — or leave blocking ungated? Long-Running Query: BLOCKED: ruling Q5: opt-out coverage means a new knob (exclude by program_name/login) — allowed? and the "what are they" breakdown is a production read — authorised for the monitor seat?
    7 PG blocking band BLOCKED: dependency #3646's lock_wait log events must exist on the PG fleet: needs the next release installed there, then ≥7 days of accumulation before a report-rate band can be sized.
    8 A5 Identity-epoch detection MERGED #3694 (3386f2b) Landed = the mechanical slice: ReconcileServers definition-changed branch → ClearServer; sqlserver_start_time / PG stats_reset epoch change → ClearServer + a logged discontinuity; AG role change → ClearServer; the PG statement delta key's datid question (A11) rides here as the same epoch class — with corrections from the lane: the SQL Server carrier is the CPU collector's second result set (sqlserver_start_time, @@SERVERNAME) compared to the pair persisted in collector_state; the PG epoch is pg_stat_statements_info.stats_reset (postmaster restarts keep the counters); the AG role change is deliberately NOT a clear (a node registration stays the same instance; a listener registration is caught by the name/start-time pair). Marker home exists without a rung: the carrier run's collection_log note (identity_epoch_changes=1, the #3161 seam) + old/new pair in collector_state — rendering it is the read-layer follow-up. Residuals (five families scheduled before the carrier subtract once on the epoch pass; PgWaitStatsCollector (Aurora) has no carrier) → MERGED #3705 (8a90d63): wait_stats (first in the order on both hosts) carries the SQL Server identity pair as a second result set observed before anything subtracts; pg_wait_stats carries pg_postmaster_start_time() as its own epoch (a clean restart zeroes aurora_stat_system_waits() while stats_reset stays continuous); the CPU carrier stays as fallback and ServerEpoch remembers per calculator/server the identity it last forgot to, so two carriers forget once. Both census rosters empty. No Azure SQL DB epoch (ruling stands); marker still unrendered.
    9 A6 CAGG follow-up MERGED #3698 (c7c973e) — clause 3 remains perfmon_interval_baseline / wait_stats_interval_baseline REPLACE the legacy pair in BaselineAggregates (appending would have moved the pinned refresh phase grid); provider reads whichever supply covers its window; legacy pair retires by a self-executing coverage condition (SupersededBaselineRelations, ~day 5 after the release). Also found+fixed: the startup baseline backfill gate read coverage through the real-time view and never fired on the creating start. unique_queries = 0 fixed (raw tail past the rollup's last day). Live evidence 2026-09-19 08:3xZ (bus 20260919T083937Z): post-install anomaly pages 11 → 24/h with 13 clean-window starts — the dented-baseline signature this fix removes (the restart zeros sat inside the hour×dow buckets). Reaches the fleet at the next install; the provider switches to the interval-honest supply once it is backfilled (BaselineWindowDays = 30 vs raw's ~30 d, so within ~a day), the legacy's DROP waits for the 35-day tier. Remains: the >24 h outage hole — read honestly (its size is the ≤2 h pre-outage tail, not the outage length); repair = widen the refresh policy's start_offset or a targeted refresh_continuous_aggregate at start — BLOCKED: ruling Q10: which?
    10 A7 Viewer rendering MERGED #3702 (f4ffafa) + MERGED #3708 (5b999e0) = the cntr_type rung, V132 / Lite v62 Both viewers shape perfmon through the stored interval via one shared DeltaSeriesShaping (/sec name proxy until the type is stored — named as a proxy); a restart's (0,0) is a line break; latch/spinlock grids render the marker as "—" with an Interval column; Lite's get_latch_stats/get_spinlock_stats publish null not 0 for it. Correction from the lane: the gap clause was ALREADY landed by #1944's cadence rule (Add.TimeSeries breaks at 3× median spacing) — the item over-stated; what was missing is the Y-side break at a normal-cadence restart. The cntr_type rung landed as #3708: perfmon_stats.cntr_type (nullable, no default, no backfill — the compressed-chunk rule), the collector writes a gauge as its level with no delta, every reader classifies by the stored type with the /sec proxy only for pre-rung rows; get_perfmon_stats/get_perfmon_trend publish cntr_type + counter_kind both SKUs. Stated, not fixed: the ten Wait Statistics Average wait time (ms) rows are PERF_AVERAGE_BULK numerators needing a base-row join this store does not do (own lane). Rule 8 of the measurement census moves from NOT CENSUS-ABLE to PINNED.
    11 A8 Rollup rate denominators MERGED #3695 (2649d33) a rollup point is rated over its bucket width (always rated, first bucket included); a raw Query Store point keeps its spacing LAG because query_store_stats stores no interval length — that residual (a quiet QS interval still halves the next raw point) needs the interval length stored beside the row: a collector change + a rung, its own decision.
    12 A10 Baseline contamination guard MERGED #3698 + MERGED #3749 (15b1a23) Mechanical half → MERGED #3698: the LAG > N heuristic applies only to pre-column (NULL-interval) rows on all three Darling arms and Lite's two wait arms; Lite BatchRequests QUALIFY retired outright; IsRealDeltaRow stays the in-memory reference (documented — the stored interval is the stronger witness at the SQL tier). DST → MERGED #3749 (Q6, ruled local clock): one shared step-function expression collection_time + (CASE WHEN collection_time < $4 THEN $5 ELSE $6 END) * INTERVAL '1' MINUTE re-keys every arm through the scaffold on both SKUs (PgTarget arms inherit; PgTarget targets have no server_properties row and stay UTC — out-of-lane), three clock params from server_properties.time_zone_id / utc_offset_minutes resolved by the new shared BaselineLocalClock, lookup via the same numbers; nothing keyed is stored, no rung, re-bucket at the next compute. Measured: neither PG/Npgsql nor DuckDB rejects a surplus Bind — the census test is the only enforcement. Item TICKED.
    13 A11 assorted SPLIT LAG-recompute of the stored interval → MERGED #3695 (viewer + Lite read sample_interval_seconds three-state through one Storage builder; the MCP reader's own raw const DarlingTrendReader.cs:414 alias → fix/3653-trend-aliases, RUNNING). datid key → resolved in #3694 as over-stated (OID reuse needs a 2³² wraparound onto the dropped OID AND the same queryid inside the 1 h gap policy; key left alone, reasoning on the PR). cpu_utilization.sample_time target-local: BLOCKED: ruling Q7: switch the collector to write UTC (existing rows stay local and readers cannot tell them apart) or add a sample_time_utc column (a rung)? utc_offset offset-not-zone: BLOCKED: ruling Q8: store the zone id (a rung; CURRENT_TIMEZONE_ID() exists only on 2022+/Azure) — yes, or leave the offset and document the DST edge? Full-edition install scripts' integer division: BLOCKED: ruling Q9: redefine the cntr_value_per_second computed column to real division in install/02, 06, 47 — a schema edit on upgrade for existing Full installs; approve?
    14 Compose MeasureCatalog Cumulative archetype MERGED #3695 every raw-tier SUM/AVG/MIN/MAX over a delta column on an interval-carrying family carries FILTER (WHERE sample_interval_seconds IS DISTINCT FROM 0); gauges/overlays/COUNT(*)/CAGG route untouched.
    15 OversizedPlanBacklog.QueryStatsFallbackSql hash-keyed BLOCKED: dependency a statement-grain (plan_handle, sql_handle, offsets) reader — none exists; the item's own precondition.
    16 10-rule measurement contract as census MERGED #3701 (f08b01c) — stays open for three named residues The contract is now a NUMBERED list in Lite.Tests/MeasurementContractCensusTests (4 and 7 keep their tests' numbers; the review's own enumeration is NOT on the record anywhere — 9 and 10 are left empty, stated, not invented; my earlier "rules 2/3/5/6/8/9" wording had no basis). Pinned tonight with positive controls and 12 planted-defect mutation checks: 1 readers NULL-not-0, 2 measured elapsed (zero cadence literals), 3 epoch → ClearServer + marker (call-site census over #3694), 6 per-second names. ROSTERED: 5 rollups interval>0 — query_stats_hourly, procedure_stats_hourly, query_stats_db_hourly still admit the (0,0) row (fix = #3698's successor shape; moves the hourly phase grid — a lane, BLOCKED: ruling Q12: approve three interval-honest hourly successors on the #3698 pattern?). 8 gauges never delta'd → PINNED by #3708 (the rung landed). The fifteen ELSE 0 rate arms → MERGED #3707 (5b50950; roster empty, nine PG trend rates now nullable on the wire, PgBaselineProvider's with_rate gains interval_sec > 0 so a NULL never enters the sample set).
    17 A15/A16 Description drift + envelope vocabulary MERGED #3696 / #3697 / #3700 / #3703 / #3706 — Q11 + one rename remain Landed: behaviour bugs (mute_analysis_finding story_path = story_path_hash + duplicate mutes; remove_server cannot remove a never-connected server), then the description lies (audit_config edition claim / PG target, ring-buffer wording, top_operators Take(10), missing_indexes.impact, AG traps, DarlingWebEndpoints.cs "percentiles", resource-semaphore ORDER BY drift). Corrections from the lane: Darling audit_config already refused PG targets (not_collected) — the pointer to get_pg_logging_audit was added; story_path cannot be NULL without a rung (column NOT NULL both SKUs) — the hash stays as a DISCLOSED placeholder when no retained finding resolves it; analysis_muted has no unique key, so a same-instant duplicate remains possible (a future rung's ON CONFLICT). Vocabulary → MERGED #3703 (13c76a5): overall_health/health_band one enum token both SKUs; limit_reached ×4 + history_capped + get_analysis_findings cap → observed truncated + *_returned through BoundPage; the 14 cut keys classified (page-cut / second-bound / note / source-side / withheld / neutral-noun / homonym / retired) and pinned; six of #3699's eight "severity spellings" were other vocabularies (rostered as homonyms with witnesses). Darling 15-second note → MERGED #3700 (0fbc83d, with the DarlingTrendReader aliases). Remains: errors-one-shape → Q11; Lite trend descriptions + shared unrated_note (both reasons, byte-identical) + latch/spinlock unknowable-row vocabulary (one shape both SKUs; Darling spinlock had NO interval key at all) → MERGED #3706 (78f60bf). Still open: the trend family's window-floor truncated → window_truncated rename (~15 consumers incl. get_query_store_top in DarlingMcpDataTools.cs, server-tabs.js, 8 instructions rows — its own lane, QUEUED); two spellings of the stored interval on the wire (sample_interval_seconds raw vs interval_seconds null-when-unknowable) — a #3703-style pass.
    18 wwwroot twins MERGED #3697 (e6b1ff5) get_pg_database_stats publishes window totals on the statement (allowance DELETED, the census now enforces it); combination_count → combinations_returned (it was already a page count, named as one now); window[] is one aggregate row per pool, NOT a series (the brief was wrong) — rendered as tables, window first; the four StampedUnderCollectionTime tools moved to captured_at on both SKUs, roster deleted.
    19 11-rule payload contract as census MERGED #3699 (3676970) — remainder is item 17's vocabulary lane + Q11 The three >= limit sites (+ get_pg_plans' unannounced cut) fixed through one McpHelpers.BoundPage; the old census missed them by ROSTER (ten A3 tools), now a sweep over every tool body on both SKUs with an executed positive control. McpPayloadContractCensusTests (23 facts) pins errors-one-shape / refuse-what-you-cannot-honor / name-is-truth as far as a census honestly can, with exact inventories: 14 truncation dialects, 8 severity spellings, TWO error wire shapes (21 PG files return a JSON envelope; ~212 tools return FormatError's bare string — a status-keyed client reads those failures as text). BLOCKED: ruling Q11: make McpHelpers.FormatError return the JSON {status:"error",…} envelope for all ~212 tools (one helper, fleet-wide wire change) — approve?
    20 Frozen-twin SqlServerFactCollector.Waits.cs divisor DECLINED as marked on the item.

    Rulings needed (Q1–Q12), each answerable in one line — Q1 A8 shape; Q2 CONFIG_CHANGED surface; Q3 recurrence discount-vs-label; Q4 blocking single-snapshot bar; Q5 LRQ opt-out knob + production read; Q6 DST bucket key; Q7 cpu_utilization.sample_time; Q8 utc_offset zone id; Q9 Full-edition install script schema edit; Q10 >24 h CAGG hole repair (widen start_offset vs targeted refresh at start); Q11 errors-one-shape wire change via FormatError; Q12 three interval-honest hourly rollup successors (moves the pinned refresh phase grid).

  2. 124 remaining items

  3. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    #4186 is merged (904d460). The three legacy query-stats rollups and their dailies no longer refresh. Their interval-honest successors take over purge and retention coverage. A stitched floor, a seam probe and the newest-first hole walk keep the raw purge held until every raw row is covered.

    Follow-ups tracked separately: #4299, #4300 and #4301.

  4. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    Closing: A6, the last open item, is delivered on dev, and every box is now ticked.

    A6, both remaining halves

    No new aggregate remains to be added, so the grid-capacity dependency in the item's text no longer applies.

  5. removed
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    client-siteOwned by the client-site agents (other laptop). Local sessions never pick these up.enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions