Monitoring pipeline rework: per-source checks, result-transform policies, check-state issues by passcod · Pull Request #362 · beyondessential/canopy

passcod · 2026-07-08T07:27:14Z

🤖 Reworks the monitoring pipeline onto the new CHK/INC/STA specs (included in this PR, along with the phased implementation plan in docs/plans/incident-pipeline-rework.md). The events/issues/incidents stack becomes: statuses (raw per-source reports) → check state (one issue row per check, carrying policy-graded results) → incidents. Each commit ships green on its own.

Specs & plan

New monitoring/checks.md (CHK), monitoring/incidents.md (INC), and public-server/statuses.md (STA), plus amendments to self-alerts.md and mcp.md. The model in brief: checks are keyed per (target, source, check) with tri-scope targeting (server / group / canopy-wide); severities are replaced by result-transform policy (a per-check ceiling, conditional rules that can regrade in any direction, and an escalates flag); incidents generalise to per-target; and every internal producer becomes a source. Supersedes the events-retention draft plan — the events table goes away instead of gaining partitioning.

Per-source status pushes

Statuses record which source pushed them: a transitionally-optional source payload field (absent ⇒ legacy transform; documented as becoming mandatory; canopy/manual reserved), history backfilled, and health issues/silences migrated from the fixed status source to alertd. Filing and recovery-by-omission are scoped to the pushing source, so two sources reporting disjoint check sets no longer flap each other's issues — the prerequisite for the Seedling integration.

Legacy pushes become the tamanu heartbeat: a push without a health array is transformed into a single always-passing tasks check from the tamanu source and flows through the normal path (state, catalog, staleness). The per-server allow_legacy_status opt-in, its UI checkbox, and the carry-forward special case are removed.

Events table and POST /events deleted

Issue state was always updated directly from each report; event rows were an append-only, unpartitioned, never-pruned audit log with three minor read surfaces. The readers, the coalescing, the Event model, the endpoints, the UI event log, and the table itself are all removed. Statuses remain the durable per-check history. The POST /events device endpoint goes too — sources report through status pushes.

Result-transform policy

healthcheck_severities becomes check_policies: keyed per (source, check), speaking one vocabulary end to end. An entry carries a ceiling (the maximum effective result), conditional rules that transform the observed result in any direction, and an escalates flag replacing the Critical severity. Ingestion grades every check through its policy; the migration converts existing severities losslessly, including rules-ladder branches.

Scoped policy: transforms can also attach to a target — a server, a group, or canopy-wide — applied after the fleet catalog (fleet → group → server, most specific last). The operator-facing silence is a scoped skipped-ceiling; both silence tables migrate in and the (source, ref) silence API/UI is unchanged. Arbitrary scoped transforms (other ceilings, scoped rule ladders) are admitted by the model but deliberately not surfaced in the UI yet.

Check-state issues

Tri-scope: issues may target a server, a group, or canopy as a whole. Self-alerts move off the nil-"meta"-server hack onto true canopy-wide rows; their direct-to-Slack path (SLACK_WEBHOOK_SELF_ALERT_URL) is retired in favour of canopy-wide incidents.
State columns: issues carry check_name, observed_result, effective_result, escalates, detail, and the degraded-streak stamps, written by every filing and backfilled for existing rows. Passing checks are kept, so "every server reporting this check" is answerable without scanning history.
Sticky broken: a broken check files on the check's own ref, retaining an open effective failure's contribution (a known failure keeps holding its incident) or counting as a warning when there is nothing to retain; policy rules can regrade brokenness.

Severities are gone

The issues.severity column, the Severity type, and the vocabulary itself are deleted. Incident membership, auto-close, grace bypass, and escalation re-notify key on (effective_result, escalates): an effective failure opens and holds incidents; an escalating failure notifies immediately. Every wire surface (private API, MCP tools and filters, self-alert views, the status snapshot) and the UI (chips, filters, icons) speaks observed/effective results. The deployed Slack workflows are untouched — the severity trigger variable is fed a result-derived urgency label.

Internal producers file checks

Reachability, per-source staleness, backup staleness/reconcile/preflight/corruption, restore verification, tailnet key expiry, MCP token expiry, self-alerts, and operator-raised manual conditions all file through one filing function under the reserved canopy/manual sources, registering their catalog entries with the policy their condition warrants.

Per-source staleness: each (server, source) that has reported is expected to keep reporting; when its last report crosses the server's down threshold, canopy files a stale/<source> check (warning ceiling). The reachability check remains the all-sources-stale arm — any report or ping refreshes it, so it fires exactly when nothing reaches canopy at all.

Read model

Health rollups (server detail, group members, MCP) and the per-check attention page read check state — worst effective result across sources — instead of scanning latest-status JSONB. The device-facing check-severities map distills ceilings and silences.

Check documentation

Each catalogued (source, check) can carry an operator-authored markdown document — by convention what the check observes, what each result means, and how to solve a failure (the editor seeds a template; nothing enforces the structure). It renders on the per-check attention page, is edited from the healthcheck settings page, and agents fetch it over MCP via get_check_documentation. Canopy's own checks ship with their documentation.

Notes for deploys

SLACK_WEBHOOK_SELF_ALERT_URL is no longer read and can be removed from ops config; straggler self-alert outbox rows drain harmlessly.
bestool's status contract-tests (in its own repo) should drop the POST /events coverage and pick up the source field.

🤖 Generated with Claude Code

Spec the reworked monitoring pipeline: per-(target, source, check) state with catalog severities replacing the events/issues aggregation layers, tri-scope targeting (server/group/canopy-wide), per-target incidents, per-source status reporting with staleness, and source-scoped responses. Self-alerts reframed as canopy-wide checks; the MCP read model loses events.

Policy becomes a transformation of check results rather than a parallel severity vocabulary: the catalog carries a ceiling plus conditional transforms (any direction) and an escalates flag, scoped per fleet, group, and server; a silence is a scoped skipped-ceiling. Issues, incidents, rollups, and the push response all speak observed and effective results.

Each catalogued (source, check) can carry markdown documentation in three sections (description, per-result meaning, solve hints), edited in the operator UI, shown with the check everywhere, and exposed over MCP so agents consult curated knowledge first.

One markdown document per check, seeded from a section template on first edit; Canopy attaches no meaning to the structure.

Phased implementation of the CHK/INC/STA specs. Supersedes the events-retention draft plan: events is deleted, not partitioned.

Statuses record which source pushed them: an optional `source` payload field (transitionally defaulting to alertd, documented as becoming mandatory; reserved names rejected), backfilling history and migrating health issues and silences from the fixed "status" source to alertd. Health-issue filing and recovery-by-omission are scoped to the pushing source, so two sources reporting disjoint check sets no longer flap each other's issues. Healthcheck silence helpers and the attention page correlate by check name across sources; legacy pushes are attributed to tamanu and internal rows to canopy.

Issue state was always updated directly from each report; event rows were a pure audit log with three read surfaces (issue event-history expander, MCP recent_events, incident event_count) and no pruning. Remove the readers, the coalescing machinery in the two save paths, the Event model, the list_events endpoint, the UI event log, and the table itself. Statuses remain the durable per-check history. Also removes the resolve_overall_health_rollups migration-replay tests: that migration inserts into events and can no longer replay against the current schema (it has long been applied everywhere).

The healthcheck_severities catalog becomes check_policies: keyed per (source, check), speaking one vocabulary end to end. An entry carries a ceiling (the maximum effective result — failed passes failures through, warning grades them down, passed records without alerting, skipped also tells the source not to run the check), conditional rules that transform the observed result in any direction, and an escalates flag replacing the Critical severity. Ingestion grades every check through its policy; issues transitionally map effective results onto severities (failed → error, escalating failed → critical, warning → warning) until they carry results themselves. The migration converts severities losslessly (critical → failed + escalates, error → failed, warning → warning, info → passed, debug → skipped), including rules-ladder branches. Also fixes healthcheckNameFromRef in the UI, which still required the retired "status" source and so had stopped linking issues to their healthcheck pages after the alertd source migration.

The server-detail checks table still matched and created silences under the retired "status" source, so silences stopped rendering (and new ones were filed under a source nothing reports) after the per-source migration. The last-status payload now carries its source and the silence popover uses it.

Issues may now be scoped to neither a server nor a group: the exactly-one scope CHECK relaxes to at-most-one, with a partial unique on (source, ref) for the global rows. Self-alerts move off the nil "meta" server onto true canopy-wide issues via a raise_global_event path (no incident evaluation — incidents stay group-keyed until they gain a canopy-wide target); the fleet issue listing excludes global rows by scope instead of by the nil id.

Issues gain the check-state columns from the CHK spec: check_name, observed_result, effective_result, and detail (the check's fields from the report, verbatim). Health-check filings stamp them — opens, closes, and the broken thread — so a row now records both sides of the policy transform; existing rows are backfilled from ref and severity. Sweeps, manual events, and device event pushes file without a stamp until they move onto the model. Exposed additively on the private issue payload; nothing consumes them yet.

A broken check no longer files a separate health-broken/<check> thread: it files on the check's one ref, retaining an open issue's contribution (its current severity, so a known failure keeps holding its incident) or warning that the check is broken when there was nothing to retain. Broken results now also flow through policy rules, so a rule can grade brokenness up to a failure or down to a pass. The migration retires existing broken-thread issues (closing incidents they alone held open) and folds their silences into the check's own.

Every check in a push now files, passing ones included: the state row records the current result and when it was last reported, completing "issues are check state". Two new timestamps keep issue semantics intact now that healthy rows exist: degraded_since tracks the current degradation streak (feeding the attention page's failing-since), and last_degraded_at never clears, letting the issue listings distinguish a recovered issue from always-healthy state (which they exclude). Degraded filings order before recoveries within a push so a failure swap joins the open incident before the outgoing failure leaves.

Incidents generalise to per-target: a server group, or canopy as a whole (server_group_id NULL, one open at a time via a partial unique and an advisory lock in place of the group-row lock). Canopy-wide issues — self-alerts — now get the full incident lifecycle: the bespoke direct-Slack path (its own outbox kinds, grace, and cancel logic) is gone, replaced by the ordinary incident open/escalate/ resolve flow labelled "Canopy" and a 3-minute default grace. The self-alert webhook env vars are no longer fed; retiring them from deploy config can follow. Incident consumers (private API, MCP, UI) carry the nullable group, with canopy-wide incidents pointing at the self-alerts view instead of a group page.

…h it file_canopy_check is the one path for canopy-determined conditions: register the (canopy, check) catalog entry with the policy the condition warrants (first sight only — operator edits stick), grade the observation through that policy (rules can predicate on the check detail and server tags), stamp the check state, and upsert at server, group, or canopy-wide scope with normal incident membership. The group/global raise paths gain check-state stamping via a shared helper. The reachability sweep is the first producer converted: operators can now tune its grading from the catalog like any other check.

Tailnet key expiry, backup staleness/never, maintenance health, the reconcile checks, restore verification, upstream preflight, backup corruption, and both self-alert conditions now file via file_canopy_check: each registers its catalog entry with the policy its condition warrants (escalating failures for key expiry, identity, object lock, and corruption — the old Critical tier), grades through operator-editable rules, and stamps check state. self_alerts::raise speaks the result vocabulary; the backup::alerts re-export module is gone (file_canopy_check is the alerting path).

The manual-event endpoint raises conditions in the check-state vocabulary: a chosen result (failed or warning; failed by default) with an optional escalates flag that seeds the condition's catalog entry, filing through file_check under the manual source — graded, stamped, and silenceable like any other check. The form swaps its severity selector for a result selector with an immediate-notify toggle. file_check gains the source field to carry manual filings (group- and canopy-wide scopes remain canopy's own).

A never-checked-in server renders setup instructions, not an issues panel, so the raise flow verifies on /incidents instead.

The health rollup reads current check state across every source — any effective failure is unhealthy, any warning or brokenness degrades, silenced checks are skipped — replacing the latest-status JSONB scan in the servers list, server detail, group members, MCP server/group listings, and the fleet summary. Canopy-determined checks (reachability, backup health, key expiry) and manual conditions now count toward the headline chip like device-reported ones. The status snapshot view keeps deriving from its own stored row: it renders a point in time, not current state. The e2e status seed helper mirrors ingestion by seeding matching check-state rows.

seedStatus now creates state rows itself, so the spec adjusts the degraded streak directly instead of hand-seeding a competing issue, and models the placeholder case as a pre-stamp inactive row.

The per-check "who's affected" page reads current check state — one row per (server, source), carrying the observed result, the check's detail, and the degraded streak as failing-since — instead of scanning every live server's latest status JSONB. Rows gain a source field; a server can appear once per source reporting the check. The now-unread reporting_check_with_servers and check_entry helpers are deleted.

A status push without a `health` array is now transformed into a single-check status from the "tamanu" source: check "tasks", always passing. It flows through the normal ingestion path (state stamping, staleness, policy) instead of the old carry-forward special case. The per-server allow_legacy_status opt-in is gone: the transform applies to every server, so the column, its UI checkbox, and create_legacy_status are all removed.

The reachability sweep grows a per-source arm: each (server, source) that has filed check state is expected to keep reporting, and when its most recent report crosses the server's down threshold canopy files a stale/<source> check (warning ceiling). The reachability check stays as the all-sources-stale arm — any report or ping refreshes it, so it fires exactly when nothing reaches canopy at all, and it still covers ping-only and never-reported servers. Source freshness reads max(last_seen) from check state rather than scanning the statuses history, so the minute-cadence sweep stays cheap.

Sources report through status pushes now — the events endpoint (and its manual-source rejection, error type included) had no remaining callers. Auth tests that used it as a convenient device-authenticated endpoint exercise the status push instead. Self-alerts flow through canopy-wide incidents since the incident rework, so the direct-to-Slack path goes too: the drainer no longer reads SLACK_WEBHOOK_SELF_ALERT_URL, the self-alert payload builders and the pending-open cancel are deleted, and straggler outbox rows from before an upgrade drain as delivered without posting.

New scoped_check_policies table: a result transform scoped to a server, a group, or canopy-wide, applied after the fleet catalog (fleet, then group, then server — the most specific scope has the last word). Both silence tables migrate in as skipped-ceiling rows and are dropped. Grading applies the scoped chain everywhere results are graded (status ingest, file_check, the attention view), so a silenced check now records its observation but grades to effective skipped at filing time — and the read-time exclusions keep covering rows graded before the silence. The (source, ref) silence API and UI are unchanged: silenced_refs becomes a facade over the scoped rows. Arbitrary scoped transforms (ceilings other than skipped, scoped rule ladders) are admitted by the model but deliberately not surfaced in the UI yet. One semantic shift: within a reporting source every silence is now a check silence — the old health/-prefix distinction only mattered for POST /events refs, which no longer exist. Reserved-source (canopy, manual) silences still stay out of the health-check ignore set.

Check state now records the escalates flag from its policy on every filing (backfilled from Critical rows). Incident membership, auto-close, open-grace bypass, and the escalation re-notify are keyed on (effective_result, escalates): an effective failure opens and holds incidents, an escalating failure notifies immediately. Rows never stamped by the check-state model fall back to the severity vocabulary, which is otherwise now write-only pending its removal from the wire.

Issue views (private API IssueData, MCP find/get issue and incident, self-alert views, the status snapshot) speak observed/effective results and the escalates flag instead of severities. The issue list filters — private API, MCP, and the incidents page — filter on effective result. The UI renders check results everywhere severities appeared: IssueRow uses the result chip, the snapshot check icons render by what policy grades an observation to, and the severity chip and vocabulary are gone. The issues.severity column is still written (unchanged values) but no longer read anywhere outside the Slack payload builder; it goes next.

The issues.severity column is gone: rows that predate the check-state model get their results backfilled from the old severity on migration, and everything — filing, incident semantics, Slack urgency, seeds, and tests — speaks results and the escalates flag. The Severity type is deleted from commons-types; the deployed Slack workflow keeps its severity trigger variable, now fed a result-derived urgency label (Critical for escalating failures, Error for failures, Warning below). Sticky-broken now retains in result space: a broken observation keeps an open effective failure filed as failed (contribution retained), and otherwise files as broken, which the rollups already count as warning.

Each catalogued (source, check) can carry a single markdown document — by convention covering what the check observes, what each result means, and how to solve a failure. Operators write it from the healthcheck settings page (the editor seeds a template of those sections; nothing enforces the structure), it renders on the per-check attention page, and agents fetch it over MCP via the new get_check_documentation tool.

Every canopy-filed check registers its catalog entry with shipped documentation on first sight (never overwriting operator edits): reachability, per-source staleness, the backup staleness / reconcile / maintenance / corruption / preflight family, restore verification, tailscale key expiry, MCP token expiry, and the Slack delivery-failure self-alert. Manual conditions carry none — the operator supplies the words.

Only alertd runs backups; the tamanu heartbeat and future sources like seedling would receive an instruction they cannot act on. Their status responses now always carry an empty backup_now list.

Every phase of docs/plans/incident-pipeline-rework.md has shipped: per-source statuses, events deleted, the check-state model and result-transform policy (fleet and scoped), per-target incidents keyed on results, internal producers filing checks, the read-model flip, per-source staleness, the legacy tamanu heartbeat, POST /events removal, the severity vocabulary retirement, and check documentation.

The aria-label the new Playwright spec selects by was missing from the editor textarea.

"Save" collided with the ceiling card's button under the spec's exact accessible-name match; "Save documentation" follows the "Save notes" pattern.

passcod · 2026-07-10T02:21:46Z

We should add a ? icon to healthchecks in the server/incident pages that pops up the rendered documentation of the healthcheck

A small ? button next to check names — on the server detail checks table and on issue rows (incidents page, incident detail, per-server issue lists) — pops up the check's rendered documentation, fetched lazily on first open. Undocumented checks keep the affordance and prompt for the missing document; both states link to the settings editor.

The popover looked up docs with a first-documented fallback across sources, which could present source B's document for source A's same-named check — actively wrong information. Every place the button appears knows the exact reporting source, so it now shows only that entry (or the write-it prompt). The attention page, which aggregates a check name across sources, returns each documented source's document and renders one panel per source, labelled when there are several.

A check's identity is the (source, check) pair; aggregating same-named checks across sources made no sense — sourceA/db and sourceB/db are unrelated checks. The page moves to /healthchecks/:source/:check, the check_attention endpoint takes and echoes the source, and its policy, documentation, and server rows all come from that one catalog entry (the per-row source field is gone — every row is the queried source). Link builders carry the source everywhere: server checks table, issue rows, and per-source links on the healthcheck settings page.

Surveyed every surface that correlated checks by bare name and scoped each to the (source, check) identity: - The snapshot rollup and its silenced-checks set now consult only the status row's own source (a status's checks all belong to one source); silenced_health_checks_for_server takes the source and the unused batch variant and silenced_refs_with_prefix are gone. - The device-facing effective check map only overlays the requested source's own silences — a silence on another source's same-named check no longer tells this source to skip it. - The server-detail checks table builds its skip-style set from the status source's silences only (the chips were already exact). - The rule-editor sample takes (source, check) so another source's same-named payload can't seed the editor's variables. - Issue::list_by_ref (the any-source ref lookup) lost its last caller with the attention rekey and is deleted.

passcod added 12 commits July 8, 2026 17:18

docs(spec): check documentation sections are template convention

caae7c5

One markdown document per check, seeded from a section template on first edit; Canopy attaches no meaning to the structure.

plan: incident pipeline rework

3609c97

Phased implementation of the CHK/INC/STA specs. Supersedes the events-retention draft plan: events is deleted, not partitioned.

passcod changed the title ~~Incident pipeline rework: per-source statuses, drop events (phases A+B)~~ Monitoring pipeline rework: per-source checks, result-transform policies, check-state issues Jul 8, 2026

passcod added 11 commits July 8, 2026 22:35

fix(e2e): assert the manual condition on the fleet issues list

c1a77c3

A never-checked-in server renders setup instructions, not an issues panel, so the raise flow verifies on /incidents instead.

fix(e2e): attention failing-since drives off seeded check state

c2cbd01

seedStatus now creates state rows itself, so the spec adjusts the degraded streak directly instead of hand-seeding a competing issue, and models the placeholder case as a pre-stamp inactive row.

passcod mentioned this pull request Jul 9, 2026

TAM-6867: Seedling x Canopy status channel — STA spec #354

Closed

passcod added 5 commits July 9, 2026 15:45

passcod force-pushed the incident-pipeline-specs branch from e0482f5 to fc7b29f Compare July 9, 2026 05:28

passcod added 6 commits July 9, 2026 17:44

feat(statuses): backup_now rides only alertd pushes

293809a

Only alertd runs backups; the tamanu heartbeat and future sources like seedling would receive an instruction they cannot act on. Their status responses now always carry an empty backup_now list.

fix(e2e): label the documentation editor for accessible-name selection

cdc7a37

The aria-label the new Playwright spec selects by was missing from the editor textarea.

fix(e2e): disambiguate the documentation save button

df37011

"Save" collided with the ceiling card's button under the spec's exact accessible-name match; "Save documentation" follows the "Save notes" pattern.

passcod added 4 commits July 14, 2026 04:16

dannash100 mentioned this pull request Jul 16, 2026

feat: TAM-6867: Canopy enrolment and status reporting channel beyondessential/seedling#116

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Monitoring pipeline rework: per-source checks, result-transform policies, check-state issues#362

Monitoring pipeline rework: per-source checks, result-transform policies, check-state issues#362
passcod wants to merge 38 commits into
mainfrom
incident-pipeline-specs

passcod commented Jul 8, 2026 •

edited

Loading

Uh oh!

passcod commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

passcod commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Specs & plan

Per-source status pushes

Events table and POST /events deleted

Result-transform policy

Check-state issues

Severities are gone

Internal producers file checks

Read model

Check documentation

Notes for deploys

Uh oh!

passcod commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

passcod commented Jul 8, 2026 •

edited

Loading