From f1503240679bbbb645a394ab1d3812a24ef95701 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Wed, 8 Jul 2026 17:15:48 +1200 Subject: [PATCH 01/38] =?UTF-8?q?docs(spec):=20check-state=20monitoring=20?= =?UTF-8?q?pipeline=20=E2=80=94=20CHK,=20INC,=20STA?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Spec the reworked monitoring pipeline: per-(target, source, check) state with catalog severities replacing the events/issues aggregation layers, tri-scope targeting (server/group/canopy-wide), per-target incidents, per-source status reporting with staleness, and source-scoped responses. Self-alerts reframed as canopy-wide checks; the MCP read model loses events. --- .workhorse/specs/monitoring/checks.md | 94 +++++++++++++++++++ .workhorse/specs/monitoring/incidents.md | 43 +++++++++ .workhorse/specs/private-server/mcp.md | 9 +- .../specs/private-server/self-alerts.md | 12 +-- .workhorse/specs/public-server/statuses.md | 42 +++++++++ 5 files changed, 188 insertions(+), 12 deletions(-) create mode 100644 .workhorse/specs/monitoring/checks.md create mode 100644 .workhorse/specs/monitoring/incidents.md create mode 100644 .workhorse/specs/public-server/statuses.md diff --git a/.workhorse/specs/monitoring/checks.md b/.workhorse/specs/monitoring/checks.md new file mode 100644 index 00000000..54df2b93 --- /dev/null +++ b/.workhorse/specs/monitoring/checks.md @@ -0,0 +1,94 @@ +--- +id: CHK +--- + +# Check state + +Canopy's monitoring is organised around checks: named conditions, each with a current result, reported by sources or determined by Canopy itself. +This spec covers the check-state model — targets, sources, results, severities, and the operator controls over them. +How device reports arrive is the status contract (see [STA](../public-server/statuses.md)); how degraded checks aggregate into incidents is the incident spec (see [INC](incidents.md)). + +## Targets + +Every check is scoped to exactly one target: a server, a server group, or Canopy as a whole. + +Server checks come from sources reporting on that server, and from Canopy's own per-server determinations such as source staleness. +Group checks are conditions Canopy determines about a group's control plane, such as backup maintenance health (see [BKJ](../jobs/backup.md)). +Canopy-wide checks are Canopy monitoring its own operation (see [SELF](../private-server/self-alerts.md)). + +## Sources + +A source is a named reporter of checks, identified by a short string. +Multiple sources may report on the same server, each concerned with part of the system, and each source's reports are independent: a report from one source says nothing about another source's checks. + +Two source names are reserved for Canopy itself: `canopy` for conditions Canopy determines on its own (staleness, reachability, backup health, key expiry, self-monitoring), and `manual` for conditions raised by operators. +Reports arriving over the device API cannot use the reserved names. + +## Results + +A check's result is one of: + +- **passed** — the condition holds. +- **warning** — the condition is degraded but not failing. +- **failed** — the condition is failing. +- **broken** — the check itself could not run; the condition is unconfirmed either way. +- **skipped** — the check deliberately did not run. + +## State + +For each (target, source, check) Canopy keeps exactly one state: the current result, the detail the source attached to the check's most recent report, when the check was first and most recently reported, and — while it is degraded — when the current degradation began. +All reported checks are kept, including passing ones, so that "every server reporting this check" is answerable without scanning history. + +A state whose result is warning or failed is an **issue**: it acquires a severity from the catalog and is eligible to contribute to incidents. +"Degraded since" is the start of the current unbroken run of degradation; a recovery ends the run, and a later degradation starts a fresh one. + +A broken result neither confirms nor clears the check's previous definite result: while broken, the state retains the severity contribution and degraded-since of its last definite result, and additionally warns that the check itself is broken. +The severity of brokenness defaults to warning and is configurable per check in the catalog. +A definite result (passed, warning, or failed) ends the broken condition and replaces the retained contribution. + +## Severity catalog + +Check severities are configured in a catalog keyed by (source, check). +A check is registered in the catalog at warning severity the first time it is reported; operators adjust from there. +Each entry carries a base severity and optionally conditional rules, evaluated against the check's own detail, the report's server-wide detail, and the server's effective tags, so the same check can be graded differently by context. + +Canopy's own checks are catalogued like any other source's, so operators control the severity of Canopy-raised conditions (reachability, staleness, backup signals) the same way as device-reported ones. + +## Reporting semantics + +A source's report for a server carries that source's complete current set of checks. +Canopy trusts the reporter: a check the source previously reported but omits from its current report has recovered, and its state records that. +Omission by one source never affects another source's checks. + +## Source staleness + +A source that has reported on a server is expected to keep reporting. +When a source's most recent report for a server is older than the server's down threshold, Canopy raises a staleness check for that (server, source) under the `canopy` source, and clears it when the source reports again. +A server all of whose sources are stale is presented as unreachable. + +## Health rollup + +A server's health is derived from its current check states across all sources: any failed check makes it unhealthy; otherwise any warning or broken check makes it degraded; otherwise it is healthy. +Passed and skipped checks, and silenced checks, do not count against a server. + +## Operator controls + +**Silences** suppress a (source, check) at server, group, or Canopy-wide scope: matching checks are still recorded and their state kept current, but they present as skipped in health rollups, raise no severity, and never contribute to incidents. +Silences record who created them and when. + +**Snoozes** suppress one state until a chosen time, after which it contributes again if still degraded. + +**Resolution** is an operator marking one state as dealt with, recording who and why. +A resolved state that is reported degraded again reopens: the resolution is cleared and the state contributes anew. + +**Notes** attach free-form operator commentary to a state. + +## Manual conditions + +Operators can raise a condition directly against a server, under the `manual` source, with a chosen check name, severity, and message. +A manual condition behaves as a reported check whose reporter is the operator: it stays active until an operator resolves it or raises it again as recovered. + +## Monitoring gate + +Server-targeted checks on a server that is not monitored are recorded and presented for visibility but do not contribute to incidents. +Group and Canopy-wide checks are not subject to any server's monitoring gate. diff --git a/.workhorse/specs/monitoring/incidents.md b/.workhorse/specs/monitoring/incidents.md new file mode 100644 index 00000000..3050bf01 --- /dev/null +++ b/.workhorse/specs/monitoring/incidents.md @@ -0,0 +1,43 @@ +--- +id: INC +--- + +# Incidents + +An incident is a span of trouble on a target: a server group, or Canopy as a whole. +It aggregates the issues active on that target over its lifetime, from when it opens until it closes or an operator resolves it. +At most one incident is open per target at a time. + +Issues and their severities are defined by the check-state model (see [CHK](checks.md)). +Issues on a server belong to the target of the server's group; issues on an ungrouped server belong to no target and cannot contribute to incidents. +Group-targeted issues belong to that group's target; Canopy-wide issues belong to the Canopy target. + +## Membership + +An incident opens when an issue at error severity or above becomes active on a target with no open incident. +While an incident is open, every active issue on its target joins it, regardless of severity, so the incident carries the full context of what was wrong during its span. + +An issue leaves the incident when it recovers, is resolved, snoozed, or silenced, when its severity drops to debug, or when its server stops being monitored. +The incident closes when its last error-or-worse issue leaves; issues below error severity never hold an incident open. + +The membership history — which issues joined and left, and when — is kept and presented as the incident's timeline. +An issue can leave and rejoin the same incident. + +Operator actions that change what counts (monitoring toggles, group membership changes, silences, severity reconfiguration) re-evaluate the affected issues' incident membership. + +## Notification + +Operators are notified over the notification channel: group incidents to the group's configured channel, Canopy-wide incidents to the operator channel. + +An incident notifies when it has stayed open past its target's grace period; an incident that closes within grace never notifies. +Whether an incident notified is recorded as its **published** flag, so flaps can be excluded from reporting. +A critical issue joining notifies immediately, bypassing any remaining grace; if the incident has already notified, the critical join escalates it with a further notification, at most once per incident. +A notified incident notifies again when it closes. + +## Resolution + +An operator can resolve an open incident, recording who and why. +Resolution cascades to the incident's open issues — each is resolved with the same attribution — and the incident closes as its members leave. +Unresolving clears the resolution record; it does not reopen the incident. + +Notes attach free-form operator commentary to an incident. diff --git a/.workhorse/specs/private-server/mcp.md b/.workhorse/specs/private-server/mcp.md index 0fb8be01..a5505cd1 100644 --- a/.workhorse/specs/private-server/mcp.md +++ b/.workhorse/specs/private-server/mcp.md @@ -78,22 +78,21 @@ When the result is truncated to its bound, the result says so, so the client doe ### Incidents and issues -An issue is a per-server (or per-group) condition raised from a known source under a stable reference, carrying a severity, a current active state, and a history of events; an incident aggregates the issues active for a group over a span of time, from when it opened until it closes or an operator resolves it. +An issue is a degraded check state (see [CHK](../monitoring/checks.md)): a condition on a server, a group, or Canopy itself, reported by a known source under a stable check name, carrying a severity and a current active state; an incident aggregates the issues active on a target over a span of time, from when it opened until it closes or an operator resolves it (see [INC](../monitoring/incidents.md)). **Find incidents** takes a look-back window (in days, defaulting to a week) and optionally one group, and returns the incidents that were open at any point within that window — those still open, plus those that closed no earlier than the window start. -Each is returned with its group, its status (open, closed, or operator-resolved), when it opened, closed, and was resolved, who resolved it and why, whether it ever escalated, how long it was open, whether it was published, and how many issues and events it covers. +Each is returned with its target, its status (open, closed, or operator-resolved), when it opened, closed, and was resolved, who resolved it and why, whether it ever escalated, how long it was open, whether it was published, and how many issues it covers. A status filter can narrow the result to only open or only resolved incidents. The window includes incidents that flapped open and shut within their group's grace period and so never surfaced to anyone. Each incident therefore carries a **published** flag — true when it actually notified operators, which happens only when it stayed open past its group's grace period or it escalated (a critical issue joined, which bypasses the grace) — and the result reports how many of the returned incidents were published. -The raw event count an incident accumulated is not a measure of its duration or severity: a high count can belong to a sub-minute flap. A summary or ranking of incidents should count published incidents rather than raw rows unless raw activity is explicitly wanted. -**Get incident** takes an incident identifier and returns the incident with the issues attached to it: each issue's severity, source, reference, message, owning server, active state, and when it joined and (if applicable) left the incident. +**Get incident** takes an incident identifier and returns the incident with the issues attached to it: each issue's severity, source, check name, message, owning server, active state, and when it joined and (if applicable) left the incident. **Find issues** returns issues across the fleet, filtered by active state, by severity, by group, by server, and by recency (issues last seen within a look-back window). -**Get issue** takes an issue identifier and returns the issue with its recent events and the incidents it is or was part of. +**Get issue** takes an issue identifier and returns the issue with the incidents it is or was part of. ## Result semantics diff --git a/.workhorse/specs/private-server/self-alerts.md b/.workhorse/specs/private-server/self-alerts.md index 2c527e6d..09067707 100644 --- a/.workhorse/specs/private-server/self-alerts.md +++ b/.workhorse/specs/private-server/self-alerts.md @@ -5,11 +5,12 @@ id: SELF # Self-alerts A self-alert is Canopy reporting a problem with its own operation, as distinct from an issue observed on a fleet member. +Self-alerts are Canopy-wide checks (see [CHK](../monitoring/checks.md)): they carry the same state, severity catalog, silences, and resolution as any other check, and they aggregate into incidents on the Canopy target (see [INC](../monitoring/incidents.md)). ## Conditions -Each self-alert condition is identified by a stable reference, and at most one alert exists per condition: repeated detections coalesce into the one alert rather than accumulating. -An alert is active while its condition holds, and recovers when the condition clears; a condition without automatic recovery stays active until an operator resolves it. +Each self-alert condition is a check with a stable name under the `canopy` source, and at most one state exists per condition: repeated detections update the one state rather than accumulating. +A condition is active while it holds, and recovers when it clears; a condition without automatic recovery stays active until an operator resolves it. The current conditions are: @@ -19,11 +20,8 @@ The current conditions are: ## Notification -An active self-alert notifies operators over the operator-notification channel directly: it does not open an incident and does not attach to any fleet group. -One notification is sent when an alert becomes active, and one when it recovers. -Alerts below critical severity wait out a short grace period before notifying, and an alert that recovers within that grace sends nothing at all; critical alerts notify immediately. -While an alert stays active, repeated detections send no further notifications. -When the notification channel is not configured for self-alerts, they are still recorded and presented, and the operator is warned that notification is off. +Self-alerts notify through the incident machinery: an error-or-worse condition opens an incident on the Canopy target, which notifies the operator channel per the incident notification rules (grace period, critical bypass, escalation, recovery notice). +When the notification channel is not configured, self-alerts are still recorded and presented, and the operator is warned that notification is off. ## Presentation diff --git a/.workhorse/specs/public-server/statuses.md b/.workhorse/specs/public-server/statuses.md new file mode 100644 index 00000000..a5528f29 --- /dev/null +++ b/.workhorse/specs/public-server/statuses.md @@ -0,0 +1,42 @@ +--- +id: STA +--- + +# Status reporting + +Devices report on their server by pushing statuses to the device API. +A status is one source's complete current picture of the server: its health checks and their results, plus server-wide metadata. +What Canopy does with the checks is the check-state model (see [CHK](../monitoring/checks.md)). + +## Push + +A status is pushed to the server it describes; the caller must present the server's enrolled device certificate, or a certificate holding the admin role. + +The payload carries: + +- **source** — the name of the reporter pushing this status. + Transitionally optional: a push without a source is attributed to `alertd`. + The field will become mandatory; new reporters must send it. + The reserved source names (see [CHK](../monitoring/checks.md), "Sources") are rejected. +- **health** — the source's complete set of checks: for each, the check's name, exactly one result (`passed`, `warning`, `failed`, `broken`, or `skipped`), and any further detail fields, which are recorded verbatim against the check. + The set may be empty, meaning the source currently has no checks — which recovers every check it previously reported. +- any further top-level fields, recorded verbatim as the status's server-wide detail. + +The server's application version is taken from the server-wide detail, falling back to the pushing client's version header. + +Per the check-state model, the push is the source's whole truth: checks omitted from it are recovered, and other sources' checks are unaffected. + +Every push is recorded in full as the server's status history. + +## Legacy pushes + +A push without a health field is a legacy report from a Tamanu server. +It is recorded with its metadata like any other status, and is treated as the source `tamanu` reporting a single check `tasks` as passed — a liveness heartbeat that participates in source staleness like any source. + +## Response + +The response to a push carries only what the pushing source needs; a source is sent nothing meant for another source, and relies on receiving nothing beyond its own concerns. + +- The effective severity of each check in the push, as resolved by the catalog, is returned to the source that pushed them. +- Whether the server should start a backup now is returned only to the source that runs backups (`alertd`). +- The server's effective tags, a server-wide fact, are returned to every source. From 7ea82999cf658744a41994e0156dcf42e2ddb83e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Wed, 8 Jul 2026 17:47:01 +1200 Subject: [PATCH 02/38] docs(spec): replace severities with result-transform policy Policy becomes a transformation of check results rather than a parallel severity vocabulary: the catalog carries a ceiling plus conditional transforms (any direction) and an escalates flag, scoped per fleet, group, and server; a silence is a scoped skipped-ceiling. Issues, incidents, rollups, and the push response all speak observed and effective results. --- .workhorse/specs/monitoring/checks.md | 64 ++++++++++++------- .workhorse/specs/monitoring/incidents.md | 14 ++-- .workhorse/specs/private-server/mcp.md | 10 +-- .../specs/private-server/self-alerts.md | 8 +-- .workhorse/specs/public-server/statuses.md | 2 +- 5 files changed, 58 insertions(+), 40 deletions(-) diff --git a/.workhorse/specs/monitoring/checks.md b/.workhorse/specs/monitoring/checks.md index 54df2b93..808dc6fc 100644 --- a/.workhorse/specs/monitoring/checks.md +++ b/.workhorse/specs/monitoring/checks.md @@ -5,7 +5,7 @@ id: CHK # Check state Canopy's monitoring is organised around checks: named conditions, each with a current result, reported by sources or determined by Canopy itself. -This spec covers the check-state model — targets, sources, results, severities, and the operator controls over them. +This spec covers the check-state model — targets, sources, results, policy, and the operator controls over them. How device reports arrive is the status contract (see [STA](../public-server/statuses.md)); how degraded checks aggregate into incidents is the incident spec (see [INC](incidents.md)). ## Targets @@ -26,33 +26,52 @@ Reports arriving over the device API cannot use the reserved names. ## Results -A check's result is one of: +A check's result is one of, in decreasing order of urgency: -- **passed** — the condition holds. -- **warning** — the condition is degraded but not failing. - **failed** — the condition is failing. +- **warning** — the condition is degraded but not failing. - **broken** — the check itself could not run; the condition is unconfirmed either way. +- **passed** — the condition holds. - **skipped** — the check deliberately did not run. -## State +Every check has two results: the **observed** result, what the source reported, and the **effective** result, what policy makes of it. +The observed result is always recorded as reported; everything Canopy acts on — issues, incidents, health rollups — follows the effective result. -For each (target, source, check) Canopy keeps exactly one state: the current result, the detail the source attached to the check's most recent report, when the check was first and most recently reported, and — while it is degraded — when the current degradation began. -All reported checks are kept, including passing ones, so that "every server reporting this check" is answerable without scanning history. +## Policy -A state whose result is warning or failed is an **issue**: it acquires a severity from the catalog and is eligible to contribute to incidents. -"Degraded since" is the start of the current unbroken run of degradation; a recovery ends the run, and a later degradation starts a fresh one. +Policy is a transformation of results: for each check it maps the observed result to the effective one. +There is one vocabulary on both sides — policy speaks in results, and what a source is told about its checks is the policy itself, not a projection of it. + +Fleet-wide policy lives in a catalog keyed by (source, check). +An entry carries: -A broken result neither confirms nor clears the check's previous definite result: while broken, the state retains the severity contribution and degraded-since of its last definite result, and additionally warns that the check itself is broken. -The severity of brokenness defaults to warning and is configurable per check in the catalog. -A definite result (passed, warning, or failed) ends the broken condition and replaces the retained contribution. +- a **ceiling** — the maximum effective result, on the urgency ordering: a ceiling of `failed` changes nothing, `warning` grades failures as warnings, `passed` means recorded but never alerting, and `skipped` additionally tells the source not to bother running the check. +- optional **rules** — conditional transforms evaluated against the check's own detail, the report's server-wide detail, and the server's effective tags; a rule can move a result in any direction, including upward: a warning graded as a failure, or a pass with a particular detail graded as a warning. +- an **escalates** flag — an effective failure of this check notifies immediately, bypassing incident grace (see [INC](incidents.md)). -## Severity catalog +A check is registered in the catalog with a ceiling of `warning` the first time it is reported; operators adjust from there. +Canopy's own checks register with the policy their condition warrants instead of the default. -Check severities are configured in a catalog keyed by (source, check). -A check is registered in the catalog at warning severity the first time it is reported; operators adjust from there. -Each entry carries a base severity and optionally conditional rules, evaluated against the check's own detail, the report's server-wide detail, and the server's effective tags, so the same check can be graded differently by context. +### Scoped policy + +Beyond the fleet catalog, a transform can be scoped to a target: per server, per group, or Canopy-wide. +Transforms apply in order — fleet catalog, then group, then server — each acting on the previous effective result, so the most specific scope has the last word. + +The operator interface presents one scoped policy: the **silence**, a scoped ceiling of `skipped`, recording who silenced and when. +A silenced check keeps recording its observed results; its effective result is skipped, so it raises nothing and counts nowhere. +The model admits arbitrary scoped transforms; surfaces beyond the silence are deliberately not offered yet. + +## State + +For each (target, source, check) Canopy keeps exactly one state: the observed and effective results, the detail the source attached to the check's most recent report, when the check was first and most recently reported, and — while it is degraded — when the current degradation began. +All reported checks are kept, including passing ones, so that "every server reporting this check" is answerable without scanning history. + +A state whose effective result is warning or failed is an **issue**, eligible to contribute to incidents. +"Degraded since" is the start of the current unbroken run of degradation; a recovery ends the run, and a later degradation starts a fresh one. -Canopy's own checks are catalogued like any other source's, so operators control the severity of Canopy-raised conditions (reachability, staleness, backup signals) the same way as device-reported ones. +An effective broken result neither confirms nor clears the check's previous definite result: while broken, the state retains the contribution and degraded-since of its last definite effective result, and the brokenness itself additionally counts as a warning. +A policy rule can grade brokenness differently — up to a failure where not being able to check is itself the failure, or down to a pass where a flaky check runner should not raise noise. +A definite effective result (passed, warning, or failed) ends the broken condition and replaces the retained contribution. ## Reporting semantics @@ -68,24 +87,23 @@ A server all of whose sources are stale is presented as unreachable. ## Health rollup -A server's health is derived from its current check states across all sources: any failed check makes it unhealthy; otherwise any warning or broken check makes it degraded; otherwise it is healthy. -Passed and skipped checks, and silenced checks, do not count against a server. +A server's health is derived from its current effective results across all sources: any failure makes it unhealthy; otherwise any warning or brokenness makes it degraded; otherwise it is healthy. +Passed and skipped checks do not count against a server. ## Operator controls -**Silences** suppress a (source, check) at server, group, or Canopy-wide scope: matching checks are still recorded and their state kept current, but they present as skipped in health rollups, raise no severity, and never contribute to incidents. -Silences record who created them and when. +**Silences** are the scoped policy described above. **Snoozes** suppress one state until a chosen time, after which it contributes again if still degraded. **Resolution** is an operator marking one state as dealt with, recording who and why. -A resolved state that is reported degraded again reopens: the resolution is cleared and the state contributes anew. +A resolved state that degrades again reopens: the resolution is cleared and the state contributes anew. **Notes** attach free-form operator commentary to a state. ## Manual conditions -Operators can raise a condition directly against a server, under the `manual` source, with a chosen check name, severity, and message. +Operators can raise a condition directly against a server, under the `manual` source, with a chosen check name, result, and message, and optionally marked as escalating. A manual condition behaves as a reported check whose reporter is the operator: it stays active until an operator resolves it or raises it again as recovered. ## Monitoring gate diff --git a/.workhorse/specs/monitoring/incidents.md b/.workhorse/specs/monitoring/incidents.md index 3050bf01..a6c8ce07 100644 --- a/.workhorse/specs/monitoring/incidents.md +++ b/.workhorse/specs/monitoring/incidents.md @@ -8,22 +8,22 @@ An incident is a span of trouble on a target: a server group, or Canopy as a who It aggregates the issues active on that target over its lifetime, from when it opens until it closes or an operator resolves it. At most one incident is open per target at a time. -Issues and their severities are defined by the check-state model (see [CHK](checks.md)). +Issues and effective results are defined by the check-state model (see [CHK](checks.md)). Issues on a server belong to the target of the server's group; issues on an ungrouped server belong to no target and cannot contribute to incidents. Group-targeted issues belong to that group's target; Canopy-wide issues belong to the Canopy target. ## Membership -An incident opens when an issue at error severity or above becomes active on a target with no open incident. -While an incident is open, every active issue on its target joins it, regardless of severity, so the incident carries the full context of what was wrong during its span. +An incident opens when a check's effective result becomes failed on a target with no open incident. +While an incident is open, every issue on its target joins it — effective warnings included — so the incident carries the full context of what was wrong during its span. -An issue leaves the incident when it recovers, is resolved, snoozed, or silenced, when its severity drops to debug, or when its server stops being monitored. -The incident closes when its last error-or-worse issue leaves; issues below error severity never hold an incident open. +An issue leaves the incident when it stops being one: its effective result recovers (to passed or skipped, whether by report or by policy), it is resolved or snoozed, or its server stops being monitored. +The incident closes when its last effective failure leaves; warnings never hold an incident open. The membership history — which issues joined and left, and when — is kept and presented as the incident's timeline. An issue can leave and rejoin the same incident. -Operator actions that change what counts (monitoring toggles, group membership changes, silences, severity reconfiguration) re-evaluate the affected issues' incident membership. +Operator actions that change what counts (monitoring toggles, group membership changes, policy and silence changes) re-evaluate the affected issues' incident membership. ## Notification @@ -31,7 +31,7 @@ Operators are notified over the notification channel: group incidents to the gro An incident notifies when it has stayed open past its target's grace period; an incident that closes within grace never notifies. Whether an incident notified is recorded as its **published** flag, so flaps can be excluded from reporting. -A critical issue joining notifies immediately, bypassing any remaining grace; if the incident has already notified, the critical join escalates it with a further notification, at most once per incident. +An escalating check's effective failure (see [CHK](checks.md), "Policy") notifies immediately, bypassing any remaining grace; if the incident has already notified, the join escalates it with a further notification, at most once per incident. A notified incident notifies again when it closes. ## Resolution diff --git a/.workhorse/specs/private-server/mcp.md b/.workhorse/specs/private-server/mcp.md index a5505cd1..2ec37fc5 100644 --- a/.workhorse/specs/private-server/mcp.md +++ b/.workhorse/specs/private-server/mcp.md @@ -74,23 +74,23 @@ When the result is truncated to its bound, the result says so, so the client doe **Fleet summary** takes no input and returns a fleet-wide overview: server counts by kind and rank, the distribution of deployed versions, a rollup of server health, the number of groups, and a rollup of backup health. -**Find backup problems** optionally narrows to one group, otherwise scans the whole fleet, and returns the current backup problems with a severity for each: server-and-type pairs whose last successful backup is overdue against its schedule, types that have never reported a backup, groups whose backup repository is in an error state, recent failed backup runs, and maintenance runs that appear stuck. +**Find backup problems** optionally narrows to one group, otherwise scans the whole fleet, and returns the current backup problems graded by urgency: server-and-type pairs whose last successful backup is overdue against its schedule, types that have never reported a backup, groups whose backup repository is in an error state, recent failed backup runs, and maintenance runs that appear stuck. ### Incidents and issues -An issue is a degraded check state (see [CHK](../monitoring/checks.md)): a condition on a server, a group, or Canopy itself, reported by a known source under a stable check name, carrying a severity and a current active state; an incident aggregates the issues active on a target over a span of time, from when it opened until it closes or an operator resolves it (see [INC](../monitoring/incidents.md)). +An issue is a degraded check state (see [CHK](../monitoring/checks.md)): a condition on a server, a group, or Canopy itself, reported by a known source under a stable check name, carrying its observed and effective results and a current active state; an incident aggregates the issues active on a target over a span of time, from when it opened until it closes or an operator resolves it (see [INC](../monitoring/incidents.md)). **Find incidents** takes a look-back window (in days, defaulting to a week) and optionally one group, and returns the incidents that were open at any point within that window — those still open, plus those that closed no earlier than the window start. Each is returned with its target, its status (open, closed, or operator-resolved), when it opened, closed, and was resolved, who resolved it and why, whether it ever escalated, how long it was open, whether it was published, and how many issues it covers. A status filter can narrow the result to only open or only resolved incidents. The window includes incidents that flapped open and shut within their group's grace period and so never surfaced to anyone. -Each incident therefore carries a **published** flag — true when it actually notified operators, which happens only when it stayed open past its group's grace period or it escalated (a critical issue joined, which bypasses the grace) — and the result reports how many of the returned incidents were published. +Each incident therefore carries a **published** flag — true when it actually notified operators, which happens only when it stayed open past its target's grace period or it escalated (an escalating check's failure joined, which bypasses the grace) — and the result reports how many of the returned incidents were published. A summary or ranking of incidents should count published incidents rather than raw rows unless raw activity is explicitly wanted. -**Get incident** takes an incident identifier and returns the incident with the issues attached to it: each issue's severity, source, check name, message, owning server, active state, and when it joined and (if applicable) left the incident. +**Get incident** takes an incident identifier and returns the incident with the issues attached to it: each issue's effective result, source, check name, message, owning server, active state, and when it joined and (if applicable) left the incident. -**Find issues** returns issues across the fleet, filtered by active state, by severity, by group, by server, and by recency (issues last seen within a look-back window). +**Find issues** returns issues across the fleet, filtered by active state, by effective result, by group, by server, and by recency (issues last seen within a look-back window). **Get issue** takes an issue identifier and returns the issue with the incidents it is or was part of. diff --git a/.workhorse/specs/private-server/self-alerts.md b/.workhorse/specs/private-server/self-alerts.md index 09067707..abdae740 100644 --- a/.workhorse/specs/private-server/self-alerts.md +++ b/.workhorse/specs/private-server/self-alerts.md @@ -5,7 +5,7 @@ id: SELF # Self-alerts A self-alert is Canopy reporting a problem with its own operation, as distinct from an issue observed on a fleet member. -Self-alerts are Canopy-wide checks (see [CHK](../monitoring/checks.md)): they carry the same state, severity catalog, silences, and resolution as any other check, and they aggregate into incidents on the Canopy target (see [INC](../monitoring/incidents.md)). +Self-alerts are Canopy-wide checks (see [CHK](../monitoring/checks.md)): they carry the same state, policy, silences, and resolution as any other check, and they aggregate into incidents on the Canopy target (see [INC](../monitoring/incidents.md)). ## Conditions @@ -14,17 +14,17 @@ A condition is active while it holds, and recovers when it clears; a condition w The current conditions are: -- Canopy's own identity for reaching backup storage is broken (critical). +- Canopy's own identity for reaching backup storage is broken (escalating). - An operator-notification delivery has permanently failed (stays until operator-resolved). - An MCP access token is within fifteen days of its expiry (see [MCP](mcp.md), "Access tokens"). ## Notification -Self-alerts notify through the incident machinery: an error-or-worse condition opens an incident on the Canopy target, which notifies the operator channel per the incident notification rules (grace period, critical bypass, escalation, recovery notice). +Self-alerts notify through the incident machinery: an effective failure opens an incident on the Canopy target, which notifies the operator channel per the incident notification rules (grace period, escalation, recovery notice). When the notification channel is not configured, self-alerts are still recorded and presented, and the operator is warned that notification is off. ## Presentation Active self-alerts are presented on their own surface in the operator UI, apart from fleet issues and incidents: a persistent notice visible from any page while any alert is active, leading to a view of the active alerts and recent recoveries. Self-alerts do not appear in the fleet issue listings, and are not presented as belonging to any server. -Each alert presents its severity, when it became active, and a description of the condition and what to do about it. +Each alert presents its effective result, when it became active, and a description of the condition and what to do about it. diff --git a/.workhorse/specs/public-server/statuses.md b/.workhorse/specs/public-server/statuses.md index a5528f29..d37f60e3 100644 --- a/.workhorse/specs/public-server/statuses.md +++ b/.workhorse/specs/public-server/statuses.md @@ -37,6 +37,6 @@ It is recorded with its metadata like any other status, and is treated as the so The response to a push carries only what the pushing source needs; a source is sent nothing meant for another source, and relies on receiving nothing beyond its own concerns. -- The effective severity of each check in the push, as resolved by the catalog, is returned to the source that pushed them. +- Each check in the push is answered with the policy applied to it (see [CHK](../monitoring/checks.md), "Policy"), so a source sees how its reports are graded and can stop running checks whose policy is `skipped`. - Whether the server should start a backup now is returned only to the source that runs backups (`alertd`). - The server's effective tags, a server-wide fact, are returned to every source. From a6b89b3a1a4ec49b9c1e149d1058704386156f98 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Wed, 8 Jul 2026 17:57:54 +1200 Subject: [PATCH 03/38] docs(spec): operator-authored check documentation Each catalogued (source, check) can carry markdown documentation in three sections (description, per-result meaning, solve hints), edited in the operator UI, shown with the check everywhere, and exposed over MCP so agents consult curated knowledge first. --- .workhorse/specs/monitoring/checks.md | 6 ++++++ .workhorse/specs/private-server/mcp.md | 3 +++ 2 files changed, 9 insertions(+) diff --git a/.workhorse/specs/monitoring/checks.md b/.workhorse/specs/monitoring/checks.md index 808dc6fc..98c32cbf 100644 --- a/.workhorse/specs/monitoring/checks.md +++ b/.workhorse/specs/monitoring/checks.md @@ -61,6 +61,12 @@ The operator interface presents one scoped policy: the **silence**, a scoped cei A silenced check keeps recording its observed results; its effective result is skipped, so it raises nothing and counts nowhere. The model admits arbitrary scoped transforms; surfaces beyond the silence are deliberately not offered yet. +## Documentation + +Each catalogued (source, check) can carry operator-authored documentation, written in markdown in three sections: a general description of what the check observes, what each result means — what makes it fail as opposed to warn — and hints for solving a failure. +Operators author and edit the documentation in the operator UI; it is presented alongside the check wherever its state is presented, and is available over the MCP interface (see [MCP](../private-server/mcp.md)) so agents work from curated knowledge about a check rather than deriving it. +Canopy's own checks ship with their documentation. + ## State For each (target, source, check) Canopy keeps exactly one state: the observed and effective results, the detail the source attached to the check's most recent report, when the check was first and most recently reported, and — while it is degraded — when the current degradation began. diff --git a/.workhorse/specs/private-server/mcp.md b/.workhorse/specs/private-server/mcp.md index 2ec37fc5..48512048 100644 --- a/.workhorse/specs/private-server/mcp.md +++ b/.workhorse/specs/private-server/mcp.md @@ -94,6 +94,9 @@ A summary or ranking of incidents should count published incidents rather than r **Get issue** takes an issue identifier and returns the issue with the incidents it is or was part of. +**Get check documentation** takes a source and check name and returns the check's operator-authored documentation (see [CHK](../monitoring/checks.md), "Documentation"): what the check observes, what each result means, and how to solve a failure. +A client investigating an issue consults this before deriving a check's meaning from other sources. + ## Result semantics A server's reported status reflects reports received within the recent-activity window; a server silent beyond that window reads as not recently seen rather than as a stale "up". From caae7c58bf0d34c6cd98485afd494f9bc401e817 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Wed, 8 Jul 2026 18:02:11 +1200 Subject: [PATCH 04/38] docs(spec): check documentation sections are template convention One markdown document per check, seeded from a section template on first edit; Canopy attaches no meaning to the structure. --- .workhorse/specs/monitoring/checks.md | 3 ++- .workhorse/specs/private-server/mcp.md | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/.workhorse/specs/monitoring/checks.md b/.workhorse/specs/monitoring/checks.md index 98c32cbf..df0dff76 100644 --- a/.workhorse/specs/monitoring/checks.md +++ b/.workhorse/specs/monitoring/checks.md @@ -63,7 +63,8 @@ The model admits arbitrary scoped transforms; surfaces beyond the silence are de ## Documentation -Each catalogued (source, check) can carry operator-authored documentation, written in markdown in three sections: a general description of what the check observes, what each result means — what makes it fail as opposed to warn — and hints for solving a failure. +Each catalogued (source, check) can carry operator-authored documentation: a single markdown document. +By convention it covers three things — a general description of what the check observes, what each result means (what makes it fail as opposed to warn), and hints for solving a failure — and the editor seeds new documentation with a template of those sections; Canopy attaches no meaning to the document's structure. Operators author and edit the documentation in the operator UI; it is presented alongside the check wherever its state is presented, and is available over the MCP interface (see [MCP](../private-server/mcp.md)) so agents work from curated knowledge about a check rather than deriving it. Canopy's own checks ship with their documentation. diff --git a/.workhorse/specs/private-server/mcp.md b/.workhorse/specs/private-server/mcp.md index 48512048..1a247fc0 100644 --- a/.workhorse/specs/private-server/mcp.md +++ b/.workhorse/specs/private-server/mcp.md @@ -94,7 +94,7 @@ A summary or ranking of incidents should count published incidents rather than r **Get issue** takes an issue identifier and returns the issue with the incidents it is or was part of. -**Get check documentation** takes a source and check name and returns the check's operator-authored documentation (see [CHK](../monitoring/checks.md), "Documentation"): what the check observes, what each result means, and how to solve a failure. +**Get check documentation** takes a source and check name and returns the check's operator-authored markdown documentation (see [CHK](../monitoring/checks.md), "Documentation"), which by convention covers what the check observes, what each result means, and how to solve a failure. A client investigating an issue consults this before deriving a check's meaning from other sources. ## Result semantics From 3609c973e7159ecaffb6aa905ce8fe0ca10c7dec Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Wed, 8 Jul 2026 18:37:07 +1200 Subject: [PATCH 05/38] plan: incident pipeline rework Phased implementation of the CHK/INC/STA specs. Supersedes the events-retention draft plan: events is deleted, not partitioned. --- docs/plans/events-retention.md | 46 ---------- docs/plans/incident-pipeline-rework.md | 118 +++++++++++++++++++++++++ 2 files changed, 118 insertions(+), 46 deletions(-) delete mode 100644 docs/plans/events-retention.md create mode 100644 docs/plans/incident-pipeline-rework.md diff --git a/docs/plans/events-retention.md b/docs/plans/events-retention.md deleted file mode 100644 index 3ee9bb6c..00000000 --- a/docs/plans/events-retention.md +++ /dev/null @@ -1,46 +0,0 @@ -# Events retention & partitioning (draft) - -**Status: draft.** `events` grows without bound — one row per issue -state-change and per contributing device push. This plan bounds it with -time-based partitioning and drop-by-partition retention. - -## Sizing - -Observed ≈ **6M rows/year**, extrapolated from the first ~1½ months of -`events` data. That's the low end of the theoretical range (worst case ≈ 100 -devices × 1 push/minute with no coalescing ≈ 50M rows/year; steady-state -coalescing cuts that roughly 10×, though a flapping issue doesn't). Re-check -the rate as the fleet and event mix grow: - -```sql -SELECT count(*) * EXTRACT(EPOCH FROM ('1 year'::interval)) - / EXTRACT(EPOCH FROM (max(created_at) - min(created_at))) -FROM events; -``` - -## Approach - -1. **Partition `events` by `created_at`, weekly** — the same pattern - `statuses` and `device_connections` already use (cron-maintained partition - manager, proven in-tree; see the weekly-partition migrations). -2. **Retention by partition drop.** Keep N weeks (e.g. 26 ≈ 6 months), drop - older partitions. The count is configurable. -3. **Indexes.** `(issue_id, created_at DESC)` within each partition; the "list - events for an issue" query wants recent rows, so partition pruning helps. - -## Non-goals - -- Don't keep daily aggregates while dropping individual rows — that's a - separate analytical concern. If long-term event-rate trends are wanted, - ship them as a materialized view that survives the partition drops. - -## Open question - -- Partition `incident_issues` too? Probably not — it grows far slower (one row - per join/leave, not per push). `incidents` likewise. - -## Effort - -Medium-small: the infrastructure exists in-repo, so it's copy-and-adapt from -the `statuses` partitioning. The retention number is an ops decision, not a -code one. diff --git a/docs/plans/incident-pipeline-rework.md b/docs/plans/incident-pipeline-rework.md new file mode 100644 index 00000000..85e480c6 --- /dev/null +++ b/docs/plans/incident-pipeline-rework.md @@ -0,0 +1,118 @@ +# Incident pipeline rework + +Implements the monitoring specs: [CHK](../../.workhorse/specs/monitoring/checks.md), +[INC](../../.workhorse/specs/monitoring/incidents.md), +[STA](../../.workhorse/specs/public-server/statuses.md), plus the amended +[SELF](../../.workhorse/specs/private-server/self-alerts.md) and +[MCP](../../.workhorse/specs/private-server/mcp.md). + +Supersedes `events-retention.md` (deleted): the `events` table goes away +instead of being partitioned. + +End state in brief: statuses gain a `source`; the events table is deleted; +issues become the current-check-state table (one row per (target, source, +check) including passing checks, observed + effective results); severities +are replaced by result-transform policy (ceiling + rules + escalates, +scoped fleet → group → server, silences = scoped skipped-ceiling); +incidents generalise to per-target (group or canopy-wide); internal +producers file through one catalog-driven function; `POST /events` is +removed; checks get operator-authored markdown documentation surfaced in +MCP. + +Each phase ships/CIs independently. Within a phase, commit per step. + +## Phase A — statuses.source and per-source scoping + +Fixes multi-source flapping; no model reshape yet. + +1. Migration: `statuses.source TEXT NOT NULL DEFAULT 'alertd'`; data + migration `issues.source 'status' → 'alertd'` and the same on + `server_silenced_refs` / `server_group_silenced_refs`. Scrub schema.rs + regen against main. +2. `StatusPayload.source: Option` — validated non-empty, reserved + names (`canopy`, `manual`) rejected; absent ⇒ `alertd`; doc comment + + openapi notes it is transitionally optional and will become mandatory. + Stored on the status row. +3. `file_health_events` files and closes under the push's source instead of + the fixed `"status"`: closes consult only open issues at (server, + push-source, `health/…`). Constant `STATUS_SOURCE` goes away. +4. Tests: two sources pushing disjoint check sets don't close each other's + issues; reserved/empty source rejected; default attribution; silence + still honoured under the new source value. Regen public-server openapi. + +## Phase B — delete the events table + +Independent of the reshape; do while A is in CI. + +1. Remove readers: `Event::list_for_issue`/`count_for_issue`, private + `/api/issues/list_events`, IssueRow event-history expander, MCP + `get_issue.recent_events`, `event_count` in `Incident::stats_for` and + `IncidentData` (UI + MCP fields, openapi regen both servers). +2. Rework the two save paths (`NewEvent::save`, `raise_group_event`) to + stop inserting event rows: drop hash/occurrences coalescing; issue + upsert behaviour unchanged. `NewEvent` stays (it's the filing input), + loses `occurred_at` plumbing where only events consumed it. +3. Migration: drop `events`. e2e seeds untouched (they never seeded + events). + +## Phase C — check-state model and policy + +The reshape. Sub-phased; each lands green. + +- **C1 schema + filing function.** Issues table gains: `check` (from + `ref`, dropping the `health/` prefix — data migration), tri-scope target + (server_id / server_group_id / neither = canopy-wide; replace the + exactly-one CHECK; migrate nil-server self-alert rows to canopy scope), + `observed_result`, `effective_result`, `detail` (latest per-check extras), + passing-check rows allowed. `healthcheck_severities` → + check catalog keyed `(source, check)`: `ceiling` (result), `rules` + (result transforms), `escalates bool`; migrate severity values + (Critical→failed+escalates, Error→failed, Warning→warning, Info→passed, + Debug→skipped). One filing function: (target, source, check, observed + result, detail) → resolve policy (fleet catalog → group → server scoped + transforms) → upsert state → incident re-evaluation. Sticky-broken + implemented here. Status ingest files every check in the push (passing + included) and recovers omitted ones per source. +- **C2 incidents per-target.** `incidents.server_group_id` → nullable + target (NULL = canopy-wide) with one-open-per-target partial indexes; + membership/auto-close/grace/published/escalation keyed on effective + results + `escalates`; canopy-target incidents notify the operator + channel; self-alert direct-Slack path deleted. +- **C3 internal producers.** Reachability, tailnet key expiry, backup + staleness/reconcile/preflight/corruption, restore verification, MCP + token expiry, self-alerts, manual events all file through the filing + function under `canopy`/`manual` with results instead of severities; + their catalog entries register with the policy they warrant. Per-(server, + source) staleness check replaces the bespoke reachability sweep; + unreachable = all sources stale. +- **C4 read model.** Health rollup + attention page + group member health + read check state (delete `reporting_check_with_servers`, latest-row + JSONB scans); silences become scoped policy rows (migrate both silence + tables; keep silence UI); UI: severity chips/filters → observed+effective + results, healthcheck severities page → policy page (ceiling, escalates; + rules stay JSON); MCP find/get issue/incident on results; seeds + e2e + updated; openapi regen. +- **C5 ingest endgame.** Legacy pushes → synthetic `("tamanu", tasks: + passed)`; `create_legacy_status` carry-forward and `allow_legacy_status` + flag removed; `POST /events` removed (public openapi regen; bestool + contract-test note for its repo); source-scoped push response (policy + per pushed check; backup_now only to `alertd`). + +## Phase D — check documentation + +`check_documentation` markdown column on the catalog; UI editor seeded +with the template (`## Description` / `## Results` with per-result +bullets / `## Solve`); shown on attention page + issue rows; MCP +`get_check_documentation` tool; canopy's own checks ship documented. + +## Cross-cutting cautions + +- `just migration NAME` for every migration; `just migrate` regenerates + schema.rs — scrub other-branch pollution. +- Public-server openapi is drift-checked too; regen after any handler + change. +- Slack payload builders read issue severity/source/ref — update in C2/C3, + not before. +- Daniel's tam-6867 branch collides on STA spec + status handler; ours + lands first, no proactive notify. +- ERRORS.md for any new problem types (e.g. reserved-source rejection). From 4390454b43cd624e36a41923aa514905298296dc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Wed, 8 Jul 2026 18:38:11 +1200 Subject: [PATCH 06/38] feat(statuses): per-source status pushes Statuses record which source pushed them: an optional `source` payload field (transitionally defaulting to alertd, documented as becoming mandatory; reserved names rejected), backfilling history and migrating health issues and silences from the fixed "status" source to alertd. Health-issue filing and recovery-by-omission are scoped to the pushing source, so two sources reporting disjoint check sets no longer flap each other's issues. Healthcheck silence helpers and the attention page correlate by check name across sources; legacy pushes are attributed to tamanu and internal rows to canopy. --- AGENTS.md | 3 +- crates/database/src/bin/seed.rs | 1 + crates/database/src/issues.rs | 20 ++ crates/database/src/schema.rs | 1 + crates/database/src/silenced_refs.rs | 31 ++- crates/database/src/statuses.rs | 6 + .../database/tests/it/check_severity_map.rs | 43 ++-- .../tests/it/silenced_health_checks.rs | 27 ++- .../database/tests/it/status_health_state.rs | 1 + crates/private-server/src/fns/statuses.rs | 23 +- crates/public-server/openapi.json | 7 + crates/public-server/src/statuses.rs | 84 ++++++-- .../tests/it/check_severities.rs | 10 +- crates/public-server/tests/it/statuses.rs | 197 ++++++++++++++---- .../down.sql | 5 + .../up.sql | 11 + 16 files changed, 344 insertions(+), 126 deletions(-) create mode 100644 migrations/2026-07-08-063717-0000_statuses_source/down.sql create mode 100644 migrations/2026-07-08-063717-0000_statuses_source/up.sql diff --git a/AGENTS.md b/AGENTS.md index 042dc6b7..de1a99b4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -126,8 +126,7 @@ just the `psql` client. On macOS there's no `/dev/shm`, so it falls back to disk `CANOPY_TEST_PG_ROLE`. To run against your **system** Postgres instead (to inspect the DB afterwards, -or where `initdb` isn't available), use `just test-system [nextest args]` — -prefix with `nice` to soften the I/O grind. +or where `initdb` isn't available), use `just test-system [nextest args]`. ## Version Control - If the working copy is a jujutsu repo (a `.jj` directory exists at the repo root), prefer `jj` commands over `git` for VCS operations (status, diff, log, commit/describe, etc.). The repo may be colocated with git, but `jj` is the source of truth for local work. diff --git a/crates/database/src/bin/seed.rs b/crates/database/src/bin/seed.rs index efc62d69..462c376b 100644 --- a/crates/database/src/bin/seed.rs +++ b/crates/database/src/bin/seed.rs @@ -864,6 +864,7 @@ async fn seed_statuses( extra, healthy, health, + source: "alertd".into(), }) .execute(conn) .await?; diff --git a/crates/database/src/issues.rs b/crates/database/src/issues.rs index 479256a0..4d2c6e01 100644 --- a/crates/database/src/issues.rs +++ b/crates/database/src/issues.rs @@ -1403,6 +1403,26 @@ impl Issue { .map_err(AppError::from) } + /// Like [`Self::list_by_source_ref`], but matching the ref under any + /// source. Healthcheck refs (`health/`) are keyed per reporting + /// source; consumers that correlate by check name alone use this. + pub async fn list_by_ref( + db: &mut AsyncPgConnection, + ref_: &str, + server_ids: &[Uuid], + ) -> Result> { + use crate::schema::issues::dsl; + if server_ids.is_empty() { + return Ok(Vec::new()); + } + dsl::issues + .select(Self::as_select()) + .filter(dsl::ref_.eq(ref_).and(dsl::server_id.eq_any(server_ids))) + .load(db) + .await + .map_err(AppError::from) + } + /// Mark an issue as operator-resolved. Triggers incident-membership /// re-evaluation (typically: leaves the incident). pub async fn resolve( diff --git a/crates/database/src/schema.rs b/crates/database/src/schema.rs index 7ec4304f..16d4c3ec 100644 --- a/crates/database/src/schema.rs +++ b/crates/database/src/schema.rs @@ -538,6 +538,7 @@ diesel::table! { device_id -> Nullable, healthy -> Bool, health -> Jsonb, + source -> Text, } } diff --git a/crates/database/src/silenced_refs.rs b/crates/database/src/silenced_refs.rs index cdf4965e..395cb21a 100644 --- a/crates/database/src/silenced_refs.rs +++ b/crates/database/src/silenced_refs.rs @@ -5,7 +5,7 @@ //! (so the issue and event rows exist), but //! [`crate::issues::re_evaluate_incident_membership`] treats them as a //! "should leave" reason, the same way it treats snoozed or unmonitored. -//! Healthcheck silences (`(status, health/)`) additionally drop the +//! Healthcheck silences (`(, health/)`) additionally drop the //! check out of the server's health rollup — see //! [`silenced_health_checks_for_servers`] and //! [`crate::statuses::Status::health_state_ignoring`]. @@ -26,13 +26,8 @@ use uuid::Uuid; use crate::issues::{reevaluate_open_issues_for_group_ref, reevaluate_open_issues_for_server_ref}; -/// The `source` the public-server status ingestion files healthcheck -/// issues under, so a silence on a healthcheck is `(status, -/// health/)`. Mirrors the public-server's `STATUS_SOURCE`. -const STATUS_SOURCE: &str = "status"; - -/// The ref prefix (with trailing separator) healthcheck issues use -/// under [`STATUS_SOURCE`]. Mirrors the public-server's `HEALTH_REF`. +/// The ref prefix (with trailing separator) healthcheck issues use, +/// whichever source reports them. Mirrors the public-server's `HEALTH_REF`. const HEALTH_REF_PREFIX: &str = "health/"; /// A silenced issue reference scoped to a single server: issues matching @@ -131,11 +126,14 @@ pub async fn is_silenced( /// `group_id` is the server's current group; pass `None` if the server is /// ungrouped. Used to build the device-facing effective check-severity map. /// May contain duplicates when a ref is silenced at both scopes. +/// +/// Matches whichever source the silence was filed under: the callers key +/// by check name only, until the health rollup becomes per-source check +/// state. pub async fn silenced_refs_with_prefix( db: &mut AsyncPgConnection, server_id: Uuid, group_id: Option, - source: &str, ref_prefix: &str, ) -> Result> { use crate::schema::{server_group_silenced_refs, server_silenced_refs}; @@ -150,7 +148,6 @@ pub async fn silenced_refs_with_prefix( .filter( server_silenced_refs::server_id .eq(server_id) - .and(server_silenced_refs::source.eq(source)) .and(server_silenced_refs::ref_.like(format!("{ref_prefix}%"))), ) .load(db) @@ -163,7 +160,6 @@ pub async fn silenced_refs_with_prefix( .filter( server_group_silenced_refs::server_group_id .eq(gid) - .and(server_group_silenced_refs::source.eq(source)) .and(server_group_silenced_refs::ref_.like(format!("{ref_prefix}%"))), ) .load(db) @@ -176,8 +172,9 @@ pub async fn silenced_refs_with_prefix( } /// Healthcheck names silenced for each of the given `(server, group)` -/// pairs, at either scope: the `` of every `(status, -/// health/)` silence entry that applies to the server. Pass each +/// pairs, at either scope: the `` of every `health/` +/// silence entry that applies to the server, whichever source it was +/// silenced under. Pass each /// server's current group id (`None` for ungrouped). Two batch queries, /// one per scope table, regardless of how many servers are asked about. /// Servers with no applicable silences are absent from the map. @@ -199,10 +196,12 @@ pub async fn silenced_health_checks_for_servers( let like_pattern = format!("{HEALTH_REF_PREFIX}%"); let server_ids: Vec = servers.iter().map(|(id, _)| *id).collect(); + // Healthcheck silences match by ref across every reporting source: the + // rollup's ignore-set is keyed by check name only, until the rollup + // itself becomes per-source check state. let server_rows: Vec<(Uuid, String)> = server_silenced_refs::table .select((server_silenced_refs::server_id, server_silenced_refs::ref_)) .filter(server_silenced_refs::server_id.eq_any(&server_ids)) - .filter(server_silenced_refs::source.eq(STATUS_SOURCE)) .filter(server_silenced_refs::ref_.like(&like_pattern)) .load(db) .await @@ -226,7 +225,6 @@ pub async fn silenced_health_checks_for_servers( server_group_silenced_refs::ref_, )) .filter(server_group_silenced_refs::server_group_id.eq_any(&group_ids)) - .filter(server_group_silenced_refs::source.eq(STATUS_SOURCE)) .filter(server_group_silenced_refs::ref_.like(&like_pattern)) .load(db) .await @@ -260,8 +258,7 @@ pub async fn silenced_health_checks_for_server( server_id: Uuid, group_id: Option, ) -> Result> { - let refs = silenced_refs_with_prefix(db, server_id, group_id, STATUS_SOURCE, HEALTH_REF_PREFIX) - .await?; + let refs = silenced_refs_with_prefix(db, server_id, group_id, HEALTH_REF_PREFIX).await?; Ok(refs .iter() .filter_map(|r| r.strip_prefix(HEALTH_REF_PREFIX)) diff --git a/crates/database/src/statuses.rs b/crates/database/src/statuses.rs index 006111c9..9c11634e 100644 --- a/crates/database/src/statuses.rs +++ b/crates/database/src/statuses.rs @@ -91,6 +91,9 @@ pub struct Status { /// `check` name and a result; any extra per-check fields are passed /// through verbatim. pub health: serde_json::Value, + /// The source that pushed this status: the named reporter (e.g. + /// `alertd`), or `canopy` for statuses generated internally. + pub source: String, } #[derive(Debug, Insertable)] @@ -104,6 +107,7 @@ pub struct NewStatus { pub extra: serde_json::Value, pub healthy: bool, pub health: serde_json::Value, + pub source: String, } impl Default for NewStatus { @@ -115,6 +119,7 @@ impl Default for NewStatus { extra: serde_json::Value::Object(Default::default()), healthy: true, health: serde_json::Value::Array(Default::default()), + source: CANOPY_SOURCE.into(), } } } @@ -162,6 +167,7 @@ impl Status { // to avoid false-positive unhealthy events from this path. healthy: true, health: serde_json::Value::Array(Default::default()), + source: CANOPY_SOURCE.into(), }) } Err(err) => { diff --git a/crates/database/tests/it/check_severity_map.rs b/crates/database/tests/it/check_severity_map.rs index c104f4cc..5be1efca 100644 --- a/crates/database/tests/it/check_severity_map.rs +++ b/crates/database/tests/it/check_severity_map.rs @@ -94,36 +94,47 @@ async fn silenced_refs_with_prefix_combines_scopes_and_filters() { let server_id = insert_server(&mut conn, Some(group_id)).await; let other_server_id = insert_server(&mut conn, None).await; - ServerSilencedRef::add(&mut conn, server_id, "status", "health/flaky", None) + ServerSilencedRef::add(&mut conn, server_id, "alertd", "health/flaky", None) .await .expect("server silence"); - ServerGroupSilencedRef::add(&mut conn, group_id, "status", "health/groupwide", None) + ServerGroupSilencedRef::add(&mut conn, group_id, "alertd", "health/groupwide", None) .await .expect("group silence"); - // None of these may leak into the result: wrong source, wrong ref - // prefix (broken issues are a separate thread), wrong server. - ServerSilencedRef::add(&mut conn, server_id, "canopy", "health/wrong-source", None) - .await - .expect("other-source silence"); - ServerSilencedRef::add(&mut conn, server_id, "status", "health-broken/flaky", None) + // A silence under any source matches: healthcheck refs are keyed + // per reporting source, and this helper correlates by check name. + ServerSilencedRef::add( + &mut conn, + server_id, + "seedling", + "health/other-source", + None, + ) + .await + .expect("other-source silence"); + // None of these may leak into the result: wrong ref prefix (broken + // issues are a separate thread), wrong server. + ServerSilencedRef::add(&mut conn, server_id, "alertd", "health-broken/flaky", None) .await .expect("broken silence"); - ServerSilencedRef::add(&mut conn, other_server_id, "status", "health/other", None) + ServerSilencedRef::add(&mut conn, other_server_id, "alertd", "health/other", None) .await .expect("other-server silence"); - let mut refs = - silenced_refs_with_prefix(&mut conn, server_id, Some(group_id), "status", "health/") - .await - .expect("refs"); + let mut refs = silenced_refs_with_prefix(&mut conn, server_id, Some(group_id), "health/") + .await + .expect("refs"); refs.sort(); - assert_eq!(refs, vec!["health/flaky", "health/groupwide"]); + assert_eq!( + refs, + vec!["health/flaky", "health/groupwide", "health/other-source"] + ); // Ungrouped lookup only sees the server-scope silences. - let refs = silenced_refs_with_prefix(&mut conn, server_id, None, "status", "health/") + let mut refs = silenced_refs_with_prefix(&mut conn, server_id, None, "health/") .await .expect("refs without group"); - assert_eq!(refs, vec!["health/flaky"]); + refs.sort(); + assert_eq!(refs, vec!["health/flaky", "health/other-source"]); }) .await } diff --git a/crates/database/tests/it/silenced_health_checks.rs b/crates/database/tests/it/silenced_health_checks.rs index 7e17bb20..37fba8dd 100644 --- a/crates/database/tests/it/silenced_health_checks.rs +++ b/crates/database/tests/it/silenced_health_checks.rs @@ -62,13 +62,13 @@ async fn resolves_server_and_group_scopes_in_batch() { let ungrouped = insert_server(&mut conn, None).await; let unsilenced = insert_server(&mut conn, Some(group)).await; - ServerSilencedRef::add(&mut conn, grouped, "status", "health/postgres", None) + ServerSilencedRef::add(&mut conn, grouped, "alertd", "health/postgres", None) .await .unwrap(); - ServerSilencedRef::add(&mut conn, ungrouped, "status", "health/disk", None) + ServerSilencedRef::add(&mut conn, ungrouped, "alertd", "health/disk", None) .await .unwrap(); - ServerGroupSilencedRef::add(&mut conn, group, "status", "health/uploads", None) + ServerGroupSilencedRef::add(&mut conn, group, "alertd", "health/uploads", None) .await .unwrap(); @@ -102,9 +102,10 @@ async fn resolves_server_and_group_scopes_in_batch() { .await } -/// Only `(status, health/)` silences are healthcheck silences: -/// other sources and non-health refs (e.g. canopy reachability) don't -/// leak into the set. +/// Only `health/` silences are healthcheck silences — whichever +/// source they were filed under, since checks are keyed per reporting +/// source. Non-health refs (e.g. canopy reachability) don't leak into +/// the set. #[tokio::test(flavor = "multi_thread")] async fn ignores_non_healthcheck_silences() { TestDb::run(async |mut conn, _url| { @@ -113,17 +114,21 @@ async fn ignores_non_healthcheck_silences() { ServerSilencedRef::add(&mut conn, server, "canopy", "reachability", None) .await .unwrap(); - ServerSilencedRef::add(&mut conn, server, "backups", "health/postgres", None) + ServerSilencedRef::add(&mut conn, server, "seedling", "health/postgres", None) .await .unwrap(); - ServerSilencedRef::add(&mut conn, server, "status", "something-else", None) + ServerSilencedRef::add(&mut conn, server, "alertd", "something-else", None) .await .unwrap(); let map = silenced_health_checks_for_servers(&mut conn, &[(server, None)]) .await .unwrap(); - assert_eq!(map.get(&server), None); + assert_eq!( + map.get(&server), + Some(&checks(&["postgres"])), + "a health/ silence counts under any source; non-health refs don't", + ); }) .await } @@ -134,7 +139,7 @@ async fn unsilencing_removes_the_check() { TestDb::run(async |mut conn, _url| { let server = insert_server(&mut conn, None).await; - ServerSilencedRef::add(&mut conn, server, "status", "health/postgres", None) + ServerSilencedRef::add(&mut conn, server, "alertd", "health/postgres", None) .await .unwrap(); assert_eq!( @@ -144,7 +149,7 @@ async fn unsilencing_removes_the_check() { checks(&["postgres"]), ); - ServerSilencedRef::remove(&mut conn, server, "status", "health/postgres") + ServerSilencedRef::remove(&mut conn, server, "alertd", "health/postgres") .await .unwrap(); assert_eq!( diff --git a/crates/database/tests/it/status_health_state.rs b/crates/database/tests/it/status_health_state.rs index eb50c62b..03f0f437 100644 --- a/crates/database/tests/it/status_health_state.rs +++ b/crates/database/tests/it/status_health_state.rs @@ -16,6 +16,7 @@ fn status(healthy: bool, health: serde_json::Value) -> Status { extra: serde_json::json!({}), healthy, health, + source: "alertd".into(), } } diff --git a/crates/private-server/src/fns/statuses.rs b/crates/private-server/src/fns/statuses.rs index 66a40a9b..86bcdaca 100644 --- a/crates/private-server/src/fns/statuses.rs +++ b/crates/private-server/src/fns/statuses.rs @@ -395,22 +395,19 @@ pub async fn check_attention( .collect(); // "Failing since" comes from the issue the public-server status - // handler files at `(status, health/)` when a check degrades: + // handler files at `(, health/)` when a check degrades: // an active issue's first_seen is exactly when the current failure // streak began (recoveries close the issue, so a re-failure starts a - // fresh one). + // fresh one). This page correlates by check name across whichever + // source reports it, so the lookup ignores the source. let server_ids: Vec = reporting.iter().map(|(s, _)| s.id).collect(); - let failing_since: HashMap = Issue::list_by_source_ref( - &mut conn, - "status", - &format!("health/{}", args.check), - &server_ids, - ) - .await? - .into_iter() - .filter(|issue| issue.active) - .filter_map(|issue| issue.server_id.map(|sid| (sid, issue.first_seen))) - .collect(); + let failing_since: HashMap = + Issue::list_by_ref(&mut conn, &format!("health/{}", args.check), &server_ids) + .await? + .into_iter() + .filter(|issue| issue.active) + .filter_map(|issue| issue.server_id.map(|sid| (sid, issue.first_seen))) + .collect(); let mut servers: Vec = reporting .into_iter() diff --git a/crates/public-server/openapi.json b/crates/public-server/openapi.json index 2bb8f05f..0f6cd72d 100644 --- a/crates/public-server/openapi.json +++ b/crates/public-server/openapi.json @@ -2058,6 +2058,13 @@ "null" ], "description": "Overall self-reported health of the server. **Absent means `true`**,\nso senders that predate this field are never treated as unhealthy by\nomission. Recorded for historical analysis and display, but **not\nconsulted for incident or severity decisions** — those are derived\nfrom the per-check results in `health`, with each check's severity\ncontrolled by an operator-managed catalog." + }, + "source": { + "type": [ + "string", + "null" + ], + "description": "The name of the source pushing this status: the reporting agent, e.g.\n`alertd`. Multiple sources may report on one server, each with its own\nset of checks; a source's push only opens and recovers its own checks.\n\n**Transitionally optional: this field will become mandatory.** A push\nwithout a `source` is attributed to `alertd`; new reporters must send\ntheir own name. Must be a non-empty string; the names `canopy` and\n`manual` are reserved for canopy itself and are rejected." } } } diff --git a/crates/public-server/src/statuses.rs b/crates/public-server/src/statuses.rs index dd522c8b..db8f0f0e 100644 --- a/crates/public-server/src/statuses.rs +++ b/crates/public-server/src/statuses.rs @@ -43,6 +43,16 @@ use crate::state::AppState; /// status data. #[derive(Debug, Deserialize, ToSchema)] pub struct StatusPayload { + /// The name of the source pushing this status: the reporting agent, e.g. + /// `alertd`. Multiple sources may report on one server, each with its own + /// set of checks; a source's push only opens and recovers its own checks. + /// + /// **Transitionally optional: this field will become mandatory.** A push + /// without a `source` is attributed to `alertd`; new reporters must send + /// their own name. Must be a non-empty string; the names `canopy` and + /// `manual` are reserved for canopy itself and are rejected. + pub source: Option, + /// Overall self-reported health of the server. **Absent means `true`**, /// so senders that predate this field are never treated as unhealthy by /// omission. Recorded for historical analysis and display, but **not @@ -110,15 +120,21 @@ pub struct HealthCheck { pub extra: serde_json::Map, } -/// `source` value for the events filed below. Distinct from -/// `canopy` (reachability sweep) so operators can tell apart "we -/// couldn't reach you" from "you told us you're sick". -const STATUS_SOURCE: &str = "status"; +/// The source a push is attributed to when it names none: the reporter +/// deployed before the `source` field existed. Also the migration value for +/// pre-source history. Transitional — the field will become mandatory. +const DEFAULT_SOURCE: &str = "alertd"; +/// The source legacy-format pushes (no `health` array) are attributed to: +/// they come from Tamanu's own direct reporting, not from alertd. +const LEGACY_SOURCE: &str = "tamanu"; +/// Source names a push may not claim: `canopy` is canopy's own +/// determinations (reachability sweep etc.), `manual` is operator-entered. +const RESERVED_SOURCES: &[&str] = &[database::statuses::CANOPY_SOURCE, "manual"]; /// Prefix for per-check refs. Each failed check is filed at -/// `(status, health/)`. +/// `(, health/)`. const HEALTH_REF: &str = "health"; /// Prefix for broken-check refs. A check reporting `result: broken` -/// files at `(status, health-broken/)` — a separate ref +/// files at `(, health-broken/)` — a separate ref /// from the check's failure issue, so a known failure can stay open /// (unconfirmed either way) while the check itself is broken, and so /// operators can silence the two independently. @@ -227,7 +243,7 @@ async fn create( }; let raw = body.map(|j| j.0).unwrap_or(serde_json::Value::Null); - let (healthy, health, extra) = split_health_from_extra(raw)?; + let (source, healthy, health, extra) = split_health_from_extra(raw)?; // The server version canopy tracks (and compares against the published // version catalog) is the Tamanu version. Prefer the payload's @@ -273,6 +289,7 @@ async fn create( extra, healthy, health, + source, } .save(conn) .await?; @@ -362,9 +379,7 @@ async fn effective_check_severities( .collect(); let health_prefix = format!("{HEALTH_REF}/"); - for r#ref in - silenced_refs_with_prefix(db, server_id, group_id, STATUS_SOURCE, &health_prefix).await? - { + for r#ref in silenced_refs_with_prefix(db, server_id, group_id, &health_prefix).await? { if let Some(check) = r#ref.strip_prefix(&health_prefix) { map.insert(check.to_string(), CheckSeverity::Skip); } @@ -398,6 +413,7 @@ async fn create_legacy_status( extra, healthy, health, + source: LEGACY_SOURCE.into(), } .save(db) .await?; @@ -485,7 +501,7 @@ async fn file_health_events( _ => "failed", }; NewEvent { - source: STATUS_SOURCE.into(), + source: status.source.clone(), r#ref: format!("{HEALTH_REF}/{check}"), severity: Some(severity), description: Some(format!("Health check '{check}' {described}")), @@ -507,7 +523,7 @@ async fn file_health_events( { let entry = find_health_entry(&status.health, check); NewEvent { - source: STATUS_SOURCE.into(), + source: status.source.clone(), r#ref: format!("{BROKEN_REF}/{check}"), severity: Some(Severity::Warning), description: Some(format!("Health check '{check}' is broken")), @@ -528,10 +544,11 @@ async fn file_health_events( // Failure closes: an open `health/` closes when the check // is now passed, skipped, or unmentioned ("trust the reporter"). // Broken does NOT close a prior failure — the check can't confirm - // the failure either way while it's broken. + // the failure either way while it's broken. Scoped to the pushing + // source: one source's push says nothing about another's checks. let health_prefix = format!("{HEALTH_REF}/"); for r#ref in - Issue::active_refs_with_prefix(conn, server_id, STATUS_SOURCE, &health_prefix).await? + Issue::active_refs_with_prefix(conn, server_id, &status.source, &health_prefix).await? { let Some(check) = r#ref.strip_prefix(&health_prefix) else { continue; @@ -549,7 +566,7 @@ async fn file_health_events( format!("Health check '{check}' recovered") }; NewEvent { - source: STATUS_SOURCE.into(), + source: status.source.clone(), r#ref, severity: Some(Severity::Info), description: None, @@ -565,7 +582,7 @@ async fn file_health_events( // the check itself is no longer broken. let broken_prefix = format!("{BROKEN_REF}/"); for r#ref in - Issue::active_refs_with_prefix(conn, server_id, STATUS_SOURCE, &broken_prefix).await? + Issue::active_refs_with_prefix(conn, server_id, &status.source, &broken_prefix).await? { let Some(check) = r#ref.strip_prefix(&broken_prefix) else { continue; @@ -574,7 +591,7 @@ async fn file_health_events( continue; } NewEvent { - source: STATUS_SOURCE.into(), + source: status.source.clone(), r#ref: format!("{BROKEN_REF}/{check}"), severity: Some(Severity::Info), description: None, @@ -639,11 +656,15 @@ fn per_check_description( (!lines.is_empty()).then(|| lines.join("\n")) } -/// Pulls the reserved `healthy` and `health` keys out of the incoming -/// status body and returns them alongside the rest of the payload +/// Pulls the reserved `source`, `healthy`, and `health` keys out of the +/// incoming status body and returns them alongside the rest of the payload /// (`extra`). Validates types per the contract: /// -/// - missing or `null` body → `healthy = true`, `health = []`, `extra = {}` +/// - missing or `null` body → `source = alertd`, `healthy = true`, +/// `health = []`, `extra = {}` +/// - `source` absent ⇒ `alertd` (transitional — the field will become +/// mandatory); present must be a non-empty string and not one of the +/// reserved names (`canopy`, `manual`) /// - `healthy` absent ⇒ `true` (legacy compat — non-negotiable, this is /// what stops every legacy server from false-positiving unhealthy on /// the day we deploy) @@ -657,7 +678,7 @@ fn per_check_description( /// verbatim. fn split_health_from_extra( raw: serde_json::Value, -) -> Result<(bool, Option, serde_json::Value)> { +) -> Result<(String, bool, Option, serde_json::Value)> { let mut obj = match raw { serde_json::Value::Null => serde_json::Map::new(), serde_json::Value::Object(m) => m, @@ -668,6 +689,24 @@ fn split_health_from_extra( } }; + let source = match obj.remove("source") { + None => DEFAULT_SOURCE.to_string(), + Some(serde_json::Value::String(s)) if !s.is_empty() => { + if RESERVED_SOURCES.iter().any(|r| s.eq_ignore_ascii_case(r)) { + return Err(AppError::BadRequest(format!( + "`source` must not be a reserved name ({})", + RESERVED_SOURCES.join(", "), + ))); + } + s + } + Some(_) => { + return Err(AppError::BadRequest( + "`source` must be a non-empty string".into(), + )); + } + }; + let healthy = match obj.remove("healthy") { None => true, Some(serde_json::Value::Bool(b)) => b, @@ -679,7 +718,7 @@ fn split_health_from_extra( // `allow_legacy_status` flag, whether to accept it (reachability-only, // carrying prior healthchecks forward) or 400 it. let Some(health_value) = obj.remove("health") else { - return Ok((healthy, None, serde_json::Value::Object(obj))); + return Ok((source, healthy, None, serde_json::Value::Object(obj))); }; let health_arr = match health_value { serde_json::Value::Array(a) => a, @@ -732,6 +771,7 @@ fn split_health_from_extra( } Ok(( + source, healthy, Some(serde_json::Value::Array(health_arr)), serde_json::Value::Object(obj), diff --git a/crates/public-server/tests/it/check_severities.rs b/crates/public-server/tests/it/check_severities.rs index 6e407f34..e905c25b 100644 --- a/crates/public-server/tests/it/check_severities.rs +++ b/crates/public-server/tests/it/check_severities.rs @@ -88,16 +88,16 @@ async fn endpoint_maps_catalog_severities_and_silences() { // Silences: flaky at server scope, groupwide at group scope; a // silence on disk_space's *broken* thread must not skip the // check itself. - ServerSilencedRef::add(&mut conn, server_id, "status", "health/flaky", None) + ServerSilencedRef::add(&mut conn, server_id, "alertd", "health/flaky", None) .await .expect("server silence"); - ServerGroupSilencedRef::add(&mut conn, group_id, "status", "health/groupwide", None) + ServerGroupSilencedRef::add(&mut conn, group_id, "alertd", "health/groupwide", None) .await .expect("group silence"); ServerSilencedRef::add( &mut conn, server_id, - "status", + "alertd", "health-broken/disk_space", None, ) @@ -134,7 +134,7 @@ async fn endpoint_works_for_ungrouped_server() { async |mut conn, cert, device_id, public, _| { let server_id = insert_server(&mut conn, Some(device_id), None).await; seed_catalog(&mut conn, "disk_space", "error", None).await; - ServerSilencedRef::add(&mut conn, server_id, "status", "health/disk_space", None) + ServerSilencedRef::add(&mut conn, server_id, "alertd", "health/disk_space", None) .await .expect("silence"); @@ -160,7 +160,7 @@ async fn status_response_carries_check_severities() { seed_catalog(&mut conn, "disk_space", "error", None).await; seed_catalog(&mut conn, "cert_expiry", "critical", None).await; - ServerSilencedRef::add(&mut conn, server_id, "status", "health/cert_expiry", None) + ServerSilencedRef::add(&mut conn, server_id, "alertd", "health/cert_expiry", None) .await .expect("silence"); diff --git a/crates/public-server/tests/it/statuses.rs b/crates/public-server/tests/it/statuses.rs index 7c1e6712..e128a2d0 100644 --- a/crates/public-server/tests/it/statuses.rs +++ b/crates/public-server/tests/it/statuses.rs @@ -843,7 +843,7 @@ async fn submit_status_legacy_allowed_carries_health_forward() { }), ) .await; - let before = fetch_issue(&mut conn, server_id, "status", "health/disk") + let before = fetch_issue(&mut conn, server_id, "alertd", "health/disk") .await .expect("per-check issue filed"); assert!(before.active); @@ -859,7 +859,7 @@ async fn submit_status_legacy_allowed_carries_health_forward() { .await; // The disk issue is untouched — not closed by the legacy push. - let after = fetch_issue(&mut conn, server_id, "status", "health/disk") + let after = fetch_issue(&mut conn, server_id, "alertd", "health/disk") .await .expect("per-check issue still present"); assert!(after.active, "legacy push must not close the failing check"); @@ -976,7 +976,7 @@ async fn submit_status_warning_check_only() { ) .await; - let per_check = fetch_issue(&mut conn, server_id, "status", "health/disk") + let per_check = fetch_issue(&mut conn, server_id, "alertd", "health/disk") .await .expect("per-check issue filed"); assert_eq!(per_check.severity, "warning"); @@ -992,7 +992,7 @@ async fn submit_status_warning_check_only() { // Rollup is retired — never filed in any case. assert!( - fetch_issue(&mut conn, server_id, "status", "health") + fetch_issue(&mut conn, server_id, "alertd", "health") .await .is_none() ); @@ -1033,14 +1033,14 @@ async fn submit_status_unhealthy_with_checks_opens_incident() { // The (status, health) rollup was retired — only per-check issues now. assert!( - fetch_issue(&mut conn, server_id, "status", "health") + fetch_issue(&mut conn, server_id, "alertd", "health") .await .is_none(), "rollup issue must not be created" ); for check in ["database", "disk"] { - let i = fetch_issue(&mut conn, server_id, "status", &format!("health/{check}")) + let i = fetch_issue(&mut conn, server_id, "alertd", &format!("health/{check}")) .await .unwrap_or_else(|| panic!("per-check issue for {check} missing")); assert_eq!(i.severity, "error", "{check}"); @@ -1048,7 +1048,7 @@ async fn submit_status_unhealthy_with_checks_opens_incident() { } // Passing check shouldn't manifest as a resolved-from-birth issue. assert!( - fetch_issue(&mut conn, server_id, "status", "health/tls") + fetch_issue(&mut conn, server_id, "alertd", "health/tls") .await .is_none(), "passing check must not create an issue" @@ -1079,7 +1079,7 @@ async fn submit_status_unhealthy_no_checks_files_nothing() { // failures to file individual issues against, so the unhealthy flag // on its own produces nothing. assert!( - fetch_issue(&mut conn, server_id, "status", "health") + fetch_issue(&mut conn, server_id, "alertd", "health") .await .is_none(), "rollup issue must not be created" @@ -1136,7 +1136,7 @@ async fn submit_status_with_all_failing_checks_silenced_opens_no_incident() { // Per-check issues exist (silence doesn't gate row creation) // but the silence prevents them from joining an incident. for check in ["database", "disk"] { - let i = fetch_issue(&mut conn, server_id, "status", &format!("health/{check}")) + let i = fetch_issue(&mut conn, server_id, "alertd", &format!("health/{check}")) .await .unwrap_or_else(|| panic!("per-check issue for {check} missing")); assert!(i.active, "{check}"); @@ -1186,13 +1186,13 @@ async fn submit_status_with_partial_silence_opens_incident_for_unsilenced() { let row = fetch_latest_health(&mut conn, server_id).await; assert!(!row.healthy); assert!( - fetch_issue(&mut conn, server_id, "status", "health") + fetch_issue(&mut conn, server_id, "alertd", "health") .await .is_none(), "rollup must not be created" ); // Disk is unsilenced and configured at Error severity → opens an incident. - let disk = fetch_issue(&mut conn, server_id, "status", "health/disk") + let disk = fetch_issue(&mut conn, server_id, "alertd", "health/disk") .await .expect("disk issue filed"); assert_eq!(disk.severity, "error"); @@ -1226,7 +1226,7 @@ async fn submit_status_per_check_severity_is_catalog_driven() { }), ) .await; - let after_first = fetch_issue(&mut conn, server_id, "status", "health/disk") + let after_first = fetch_issue(&mut conn, server_id, "alertd", "health/disk") .await .expect("per-check issue"); assert_eq!(after_first.severity, "warning"); @@ -1243,14 +1243,14 @@ async fn submit_status_per_check_severity_is_catalog_driven() { }), ) .await; - let disk = fetch_issue(&mut conn, server_id, "status", "health/disk") + let disk = fetch_issue(&mut conn, server_id, "alertd", "health/disk") .await .expect("per-check issue still present"); assert_eq!(disk.severity, "warning"); assert!(disk.active, "still failing, must stay active"); assert!( - fetch_issue(&mut conn, server_id, "status", "health") + fetch_issue(&mut conn, server_id, "alertd", "health") .await .is_none(), "rollup is retired and must not exist" @@ -1288,7 +1288,7 @@ async fn submit_status_check_recovery_explicit() { ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/db") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await .expect("per-check issue exists"); assert!(!issue.active, "explicit healthy=true closes the issue"); @@ -1327,7 +1327,7 @@ async fn submit_status_check_recovery_via_drop() { ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/db") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await .expect("per-check issue exists"); assert!(!issue.active); @@ -1432,7 +1432,7 @@ async fn submit_status_reachability_to_health_handoff() { ) .await; - let per_check = fetch_issue(&mut conn, server_id, "status", "health/db") + let per_check = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await .expect("per-check issue opened"); assert!(per_check.active); @@ -1516,11 +1516,11 @@ async fn submit_status_keeps_incident_open_when_failure_swaps() { "same incident, not a close+reopen flicker" ); - let db = fetch_issue(&mut conn, server_id, "status", "health/db") + let db = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await .expect("db issue"); assert!(!db.active, "db has recovered"); - let disk = fetch_issue(&mut conn, server_id, "status", "health/disk") + let disk = fetch_issue(&mut conn, server_id, "alertd", "health/disk") .await .expect("disk issue"); assert!(disk.active, "disk is new failure"); @@ -1617,7 +1617,7 @@ async fn submit_status_uses_catalog_severity_on_failure() { ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/tunable_check") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/tunable_check") .await .expect("per-check issue filed"); assert_eq!(issue.severity, "critical"); @@ -1694,7 +1694,7 @@ async fn submit_status_rule_on_check_extra_overrides_base() { }), ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/disk_space") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/disk_space") .await .expect("per-check issue filed"); assert_eq!(issue.severity, "critical"); @@ -1710,7 +1710,7 @@ async fn submit_status_rule_on_check_extra_overrides_base() { }), ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/disk_space") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/disk_space") .await .expect("per-check issue still present"); assert_eq!(issue.severity, "warning"); @@ -1751,7 +1751,7 @@ async fn submit_status_rule_on_bestool_version_range() { }), ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/tamanu_service") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/tamanu_service") .await .expect("per-check issue filed"); assert_eq!(issue.severity, "warning"); @@ -1768,7 +1768,7 @@ async fn submit_status_rule_on_bestool_version_range() { }), ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/tamanu_service") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/tamanu_service") .await .expect("per-check issue still present"); assert_eq!(issue.severity, "error"); @@ -1810,7 +1810,7 @@ async fn submit_status_rule_on_server_tag() { }), ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/cert_expiry") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/cert_expiry") .await .expect("per-check issue filed"); assert_eq!( @@ -1850,7 +1850,7 @@ async fn submit_status_tiered_ladder() { }), ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/cert_expiry") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/cert_expiry") .await .expect("issue"); assert_eq!(issue.severity, "error"); @@ -1866,7 +1866,7 @@ async fn submit_status_tiered_ladder() { }), ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/cert_expiry") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/cert_expiry") .await .expect("issue"); assert_eq!(issue.severity, "warning"); @@ -1953,7 +1953,7 @@ async fn submit_status_result_failed_uses_catalog_severity() { ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/db") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await .expect("per-check issue filed"); assert_eq!(issue.severity, "error"); @@ -1990,7 +1990,7 @@ async fn submit_status_result_warning_ignores_catalog_severity() { ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/db") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await .expect("per-check issue filed"); assert_eq!(issue.severity, "warning"); @@ -2034,7 +2034,7 @@ async fn submit_status_result_rule_on_check_result() { ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/db") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await .expect("per-check issue filed"); assert_eq!(issue.severity, "info", "rule overrides the fixed default"); @@ -2062,7 +2062,7 @@ async fn submit_status_result_broken_files_separate_ref_at_warning() { ) .await; - let broken = fetch_issue(&mut conn, server_id, "status", "health-broken/db") + let broken = fetch_issue(&mut conn, server_id, "alertd", "health-broken/db") .await .expect("broken issue filed"); assert_eq!(broken.severity, "warning"); @@ -2076,7 +2076,7 @@ async fn submit_status_result_broken_files_separate_ref_at_warning() { assert!(broken.message.contains("config not found")); // No failure issue, no incident. assert!( - fetch_issue(&mut conn, server_id, "status", "health/db") + fetch_issue(&mut conn, server_id, "alertd", "health/db") .await .is_none(), "broken must not file at the failure ref" @@ -2093,7 +2093,7 @@ async fn submit_status_result_broken_files_separate_ref_at_warning() { }), ) .await; - let broken = fetch_issue(&mut conn, server_id, "status", "health-broken/db") + let broken = fetch_issue(&mut conn, server_id, "alertd", "health-broken/db") .await .expect("broken issue still exists"); assert!(!broken.active); @@ -2131,11 +2131,11 @@ async fn submit_status_failed_then_broken_keeps_failure_open() { ) .await; - let failure = fetch_issue(&mut conn, server_id, "status", "health/db") + let failure = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await .expect("failure issue exists"); assert!(failure.active, "broken must not close the failure"); - let broken = fetch_issue(&mut conn, server_id, "status", "health-broken/db") + let broken = fetch_issue(&mut conn, server_id, "alertd", "health-broken/db") .await .expect("broken issue filed"); assert!(broken.active); @@ -2150,11 +2150,11 @@ async fn submit_status_failed_then_broken_keeps_failure_open() { }), ) .await; - let failure = fetch_issue(&mut conn, server_id, "status", "health/db") + let failure = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await .expect("failure issue exists"); assert!(!failure.active); - let broken = fetch_issue(&mut conn, server_id, "status", "health-broken/db") + let broken = fetch_issue(&mut conn, server_id, "alertd", "health-broken/db") .await .expect("broken issue exists"); assert!(!broken.active); @@ -2189,7 +2189,7 @@ async fn submit_status_result_skipped_closes_failure_with_message() { ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/cert") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/cert") .await .expect("per-check issue exists"); assert!(!issue.active, "skipped closes the failure"); @@ -2234,7 +2234,7 @@ async fn submit_status_result_skipped_closes_broken() { ) .await; - let broken = fetch_issue(&mut conn, server_id, "status", "health-broken/cert") + let broken = fetch_issue(&mut conn, server_id, "alertd", "health-broken/cert") .await .expect("broken issue exists"); assert!(!broken.active); @@ -2273,7 +2273,7 @@ async fn submit_status_legacy_to_result_transition() { ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/db") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await .expect("per-check issue exists"); assert!(!issue.active, "result form closes legacy-form failure"); @@ -2655,3 +2655,120 @@ async fn submit_status_versionless_when_neither_present() { ) .await } + +/// Two sources reporting disjoint check sets for the same server must not +/// close each other's issues: a push's "unmentioned means recovered" only +/// applies to the pushing source's own checks. +#[tokio::test(flavor = "multi_thread")] +async fn multi_source_pushes_do_not_flap() { + commons_tests::server::run_with_device_auth( + "server", + async |mut conn, cert, device_id, public, _| { + let server_id = insert_health_test_server(&mut conn, device_id).await; + + // Source-less push: attributed to alertd, files its failure there. + public + .post(&format!("/status/{server_id}")) + .add_header("mtls-certificate", &cert) + .json(&serde_json::json!({ "health": [{ "check": "db", "result": "failed" }] })) + .await + .assert_status_ok(); + assert!( + fetch_issue(&mut conn, server_id, "alertd", "health/db") + .await + .expect("alertd issue filed") + .active + ); + + // A second source pushes a disjoint check set: its own failure + // files under its name, and alertd's open issue is untouched. + public + .post(&format!("/status/{server_id}")) + .add_header("mtls-certificate", &cert) + .json(&serde_json::json!({ + "source": "seedling", + "health": [{ "check": "disk", "result": "failed" }], + })) + .await + .assert_status_ok(); + assert!( + fetch_issue(&mut conn, server_id, "seedling", "health/disk") + .await + .expect("seedling issue filed") + .active + ); + assert!( + fetch_issue(&mut conn, server_id, "alertd", "health/db") + .await + .expect("alertd issue still present") + .active, + "another source's push must not recover alertd's checks", + ); + + // The second source recovering only closes its own issue. + public + .post(&format!("/status/{server_id}")) + .add_header("mtls-certificate", &cert) + .json(&serde_json::json!({ "source": "seedling", "health": [] })) + .await + .assert_status_ok(); + assert!( + !fetch_issue(&mut conn, server_id, "seedling", "health/disk") + .await + .expect("seedling issue closed") + .active + ); + assert!( + fetch_issue(&mut conn, server_id, "alertd", "health/db") + .await + .expect("alertd issue survives") + .active + ); + + // Each stored status row records the source that pushed it. + #[derive(QueryableByName)] + struct SourceRow { + #[diesel(sql_type = sql_types::Text)] + source: String, + } + let sources: Vec = sql_query( + "SELECT source FROM statuses WHERE server_id = $1 ORDER BY created_at, source", + ) + .bind::(server_id) + .load(&mut conn) + .await + .expect("load status sources"); + let sources: Vec<&str> = sources.iter().map(|r| r.source.as_str()).collect(); + assert_eq!(sources, ["alertd", "seedling", "seedling"]); + }, + ) + .await +} + +/// The `source` field must be a non-empty, non-reserved string when present. +#[tokio::test(flavor = "multi_thread")] +async fn source_field_validation() { + commons_tests::server::run_with_device_auth( + "server", + async |mut conn, cert, device_id, public, _| { + let server_id = insert_health_test_server(&mut conn, device_id).await; + + for source in ["canopy", "Manual", ""] { + public + .post(&format!("/status/{server_id}")) + .add_header("mtls-certificate", &cert) + .json(&serde_json::json!({ "source": source, "health": [] })) + .await + .assert_status_bad_request(); + } + + public + .post(&format!("/status/{server_id}")) + .add_header("mtls-certificate", &cert) + .json(&serde_json::json!({ "source": 42, "health": [] })) + .await + .assert_status_bad_request(); + }, + ) + .await +} diff --git a/migrations/2026-07-08-063717-0000_statuses_source/down.sql b/migrations/2026-07-08-063717-0000_statuses_source/down.sql new file mode 100644 index 00000000..faf0c55e --- /dev/null +++ b/migrations/2026-07-08-063717-0000_statuses_source/down.sql @@ -0,0 +1,5 @@ +UPDATE server_group_silenced_refs SET source = 'status' WHERE source = 'alertd'; +UPDATE server_silenced_refs SET source = 'status' WHERE source = 'alertd'; +UPDATE issues SET source = 'status' WHERE source = 'alertd'; + +ALTER TABLE statuses DROP COLUMN source; diff --git a/migrations/2026-07-08-063717-0000_statuses_source/up.sql b/migrations/2026-07-08-063717-0000_statuses_source/up.sql new file mode 100644 index 00000000..cf54e3fb --- /dev/null +++ b/migrations/2026-07-08-063717-0000_statuses_source/up.sql @@ -0,0 +1,11 @@ +-- Every status row records the source that pushed it. Pushes that name no +-- source are attributed to 'alertd' (the transitional default), and all +-- pre-source history was pushed by alertd's doctor sweep. +ALTER TABLE statuses ADD COLUMN source TEXT NOT NULL DEFAULT 'alertd'; + +-- Health-check issues and silences were filed under the fixed source +-- 'status'; they now live under the source that reports them, which for +-- everything to date is alertd. +UPDATE issues SET source = 'alertd' WHERE source = 'status'; +UPDATE server_silenced_refs SET source = 'alertd' WHERE source = 'status'; +UPDATE server_group_silenced_refs SET source = 'alertd' WHERE source = 'status'; From c5e4589ad5c42afd3f625893276c9ac2a398dcae Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Wed, 8 Jul 2026 19:07:52 +1200 Subject: [PATCH 07/38] feat!(issues): delete the events table Issue state was always updated directly from each report; event rows were a pure audit log with three read surfaces (issue event-history expander, MCP recent_events, incident event_count) and no pruning. Remove the readers, the coalescing machinery in the two save paths, the Event model, the list_events endpoint, the UI event log, and the table itself. Statuses remain the durable per-check history. Also removes the resolve_overall_health_rollups migration-replay tests: that migration inserts into events and can no longer replay against the current schema (it has long been applied everywhere). --- crates/canopy-mcp/src/incidents.rs | 50 +--- crates/canopy-mcp/src/lib.rs | 4 +- crates/database/src/issues.rs | 210 ++-------------- crates/database/src/schema.rs | 18 -- crates/database/tests/it/incident_stats.rs | 16 +- crates/database/tests/it/main.rs | 1 - ...esolve_overall_health_rollups_migration.rs | 225 ----------------- crates/private-server/src/fns/incidents.rs | 3 - crates/private-server/src/fns/issues.rs | 94 +------ crates/private-server/tests/it/issues.rs | 73 ------ crates/private-server/tests/it/mcp.rs | 5 +- crates/public-server/openapi.json | 2 +- crates/public-server/tests/it/events.rs | 50 ++-- .../down.sql | 16 ++ .../2026-07-08-070653-0000_drop_events/up.sql | 5 + private-web/openapi.json | 229 ------------------ private-web/src/api-types.ts | 165 ------------- private-web/src/components/IncidentCard.tsx | 8 +- private-web/src/components/IssueRow.tsx | 168 +------------ private-web/src/routes/IncidentDetail.tsx | 6 - private-web/src/types.ts | 1 - 21 files changed, 74 insertions(+), 1275 deletions(-) delete mode 100644 crates/database/tests/it/resolve_overall_health_rollups_migration.rs create mode 100644 migrations/2026-07-08-070653-0000_drop_events/down.sql create mode 100644 migrations/2026-07-08-070653-0000_drop_events/up.sql diff --git a/crates/canopy-mcp/src/incidents.rs b/crates/canopy-mcp/src/incidents.rs index 5f51bf5e..5a334835 100644 --- a/crates/canopy-mcp/src/incidents.rs +++ b/crates/canopy-mcp/src/incidents.rs @@ -2,7 +2,7 @@ use commons_types::{Uuid, issue::Severity}; use database::{ - issues::{Event, Incident, Issue, IssueListFilters}, + issues::{Incident, Issue, IssueListFilters}, server_groups::ServerGroup, servers::Server, slack_outbox::SlackOutbox, @@ -88,9 +88,6 @@ struct IncidentSummary { /// How long the incident was (or has been) open, in seconds. open_duration_secs: i64, issue_count: i64, - /// Raw count of status events the incident accumulated. NOT a measure of - /// duration or severity — a high count can be a sub-minute flap. - event_count: i64, } #[derive(Serialize)] @@ -164,18 +161,6 @@ struct IssueList { issues: Vec, } -#[derive(Serialize)] -struct EventOut { - created_at: Timestamp, - occurred_at: Option, - severity: Severity, - description: Option, - message: String, - active: bool, - occurrences: i32, - last_seen: Timestamp, -} - #[derive(Serialize)] struct IncidentRefOut { incident_id: Uuid, @@ -201,7 +186,6 @@ struct IssueDetail { resolved_by: Option, resolved_reason: Option, snoozed_until: Option, - recent_events: Vec, incidents: Vec, } @@ -217,11 +201,10 @@ impl CanopyMcp { but never surfaced to anyone. An incident is `published` only if its Slack \ open notice was delivered: it stayed open past the group's grace window \ (slack_open_delay, ~3 min by default) OR it escalated (a critical issue \ - joined, which bypasses the grace). `event_count` is raw status-event churn \ - and does NOT track duration or severity — a high-event incident can be a \ - sub-minute flap. A high count dominated by unpublished short-lived rows \ - usually means a twitchy alert/health-check threshold, not a real outage. \ - `published_count` gives the surfaced subset directly." + joined, which bypasses the grace). A count dominated by unpublished \ + short-lived incidents usually means a twitchy alert/health-check \ + threshold, not a real outage. `published_count` gives the surfaced \ + subset directly." )] async fn find_incidents( &self, @@ -273,7 +256,6 @@ impl CanopyMcp { published: published.contains(&i.id), open_duration_secs: open_duration_secs(i), issue_count: s.map_or(0, |s| s.issue_count), - event_count: s.map_or(0, |s| s.event_count), } }) .collect(); @@ -356,7 +338,7 @@ impl CanopyMcp { #[tool( description = "List issues across the fleet, filtered by active state, severity, group, \ - server, and recency. Issues are the per-(server,source,ref) events that make \ + server, and recency. Issues are the per-(server,source,ref) conditions that make \ up incidents." )] async fn find_issues( @@ -401,8 +383,7 @@ impl CanopyMcp { } #[tool( - description = "Full detail for one issue: its fields, recent events, and the incidents it \ - is or was part of." + description = "Full detail for one issue: its fields and the incidents it is or was part of." )] async fn get_issue( &self, @@ -413,9 +394,6 @@ impl CanopyMcp { let Ok(issue) = Issue::get_by_id(&mut conn, id).await else { return Ok(not_found(format!("no issue with id {id}"))); }; - let events = Event::list_for_issue(&mut conn, id, 0, 20) - .await - .map_err(mcp_err)?; let inc = Incident::for_issues(&mut conn, &[id]) .await .map_err(mcp_err)?; @@ -428,19 +406,6 @@ impl CanopyMcp { None => None, }; - let recent_events = events - .iter() - .map(|e| EventOut { - created_at: e.created_at, - occurred_at: e.occurred_at, - severity: e.severity, - description: e.description.clone(), - message: e.message.clone(), - active: e.active, - occurrences: e.occurrences, - last_seen: e.last_seen, - }) - .collect(); let incidents = inc .get(&id) .into_iter() @@ -469,7 +434,6 @@ impl CanopyMcp { resolved_by: issue.resolved_by.clone(), resolved_reason: issue.resolved_reason.clone(), snoozed_until: issue.snoozed_until, - recent_events, incidents, }) } diff --git a/crates/canopy-mcp/src/lib.rs b/crates/canopy-mcp/src/lib.rs index 8a699792..55c90775 100644 --- a/crates/canopy-mcp/src/lib.rs +++ b/crates/canopy-mcp/src/lib.rs @@ -78,8 +78,8 @@ impl ServerHandler for CanopyMcp { flapping that was recorded but never surfaced. When summarizing or ranking, count \ `published` incidents (also given as `published_count`), not raw rows: an incident is \ published only if it outlived the group's grace window (~3 min default) or escalated. \ - `event_count` is raw event churn, not duration or severity — a huge count can be a \ - sub-minute flap, usually a twitchy threshold rather than a real outage." + A count dominated by unpublished short-lived incidents usually means a twitchy \ + threshold rather than a real outage." .into(), ); info.capabilities = ServerCapabilities::builder().enable_tools().build(); diff --git a/crates/database/src/issues.rs b/crates/database/src/issues.rs index 4d2c6e01..cafb5067 100644 --- a/crates/database/src/issues.rs +++ b/crates/database/src/issues.rs @@ -6,7 +6,6 @@ use diesel::prelude::*; use diesel_async::{AsyncConnection, AsyncPgConnection, RunQueryDsl}; use jiff::Timestamp; use serde::{Deserialize, Serialize}; -use sha2::{Digest, Sha256}; use uuid::Uuid; use crate::{devices::Device, server_groups::ServerGroup, servers::Server}; @@ -88,30 +87,6 @@ pub struct Issue { pub snoozed_until: Option, } -#[derive(Clone, Debug, Serialize, Deserialize, Queryable, Selectable, Associations)] -#[diesel(belongs_to(Issue))] -#[diesel(table_name = crate::schema::events)] -#[diesel(check_for_backend(diesel::pg::Pg))] -pub struct Event { - pub id: Uuid, - #[diesel(deserialize_as = jiff_diesel::Timestamp, serialize_as = jiff_diesel::Timestamp)] - pub created_at: Timestamp, - #[diesel(deserialize_as = jiff_diesel::NullableTimestamp, serialize_as = jiff_diesel::NullableTimestamp)] - pub occurred_at: Option, - pub issue_id: Uuid, - #[diesel(deserialize_as = String, serialize_as = String)] - pub severity: Severity, - /// Single-line title/subject. See [`NewEvent::description`]. - pub description: Option, - /// Body / long detail. See [`NewEvent::message`]. - pub message: String, - pub active: bool, - pub hash: Vec, - pub occurrences: i32, - #[diesel(deserialize_as = jiff_diesel::Timestamp, serialize_as = jiff_diesel::Timestamp)] - pub last_seen: Timestamp, -} - #[derive(Clone, Debug, Serialize, Deserialize, Queryable, Selectable, Associations)] #[diesel(belongs_to(ServerGroup, foreign_key = server_group_id))] #[diesel(table_name = crate::schema::incidents)] @@ -151,9 +126,8 @@ pub struct IncidentIssue { /// A single occurrence to report against an issue, sent either by a device /// or by an operator. If an issue with the same `source` and `ref` is -/// already open, this occurrence is folded into it (bumping its last-seen -/// time, and coalescing into the latest recorded event when the content is -/// identical); otherwise a new issue is created. +/// already open, this occurrence is folded into it (updating its state and +/// bumping its last-seen time); otherwise a new issue is created. #[derive(Debug, Clone, Deserialize, utoipa::ToSchema)] #[serde(rename_all = "camelCase")] pub struct NewEvent { @@ -207,35 +181,18 @@ pub struct IssueListFilters { pub since: Option, } -fn hash_event( - severity: Severity, - active: bool, - message: &str, - description: Option<&str>, -) -> Vec { - let mut h = Sha256::new(); - h.update(severity.to_string().as_bytes()); - h.update([0]); - h.update([u8::from(active)]); - h.update([0]); - h.update(message.as_bytes()); - h.update([0]); - h.update(description.unwrap_or("").as_bytes()); - h.finalize().to_vec() -} - impl NewEvent { /// Persist this event push: /// 1. find-or-create the issue keyed by (server_id, source, ref), - /// 2. append the event or coalesce into the latest matching one, - /// 3. if the server is in a group: (re)evaluate incident contribution. + /// updating its state from this report, + /// 2. if the server is in a group: (re)evaluate incident contribution. /// /// `server_id` is the server the issue is attached to: derived from the /// device for public submissions, supplied by the operator for manual. /// `device_id` is `None` for manual events. /// /// Issues from an **ungrouped** server are still recorded — the issue - /// and event rows go in just like any other push — but the incident + /// row goes in just like any other push — but the incident /// flow is skipped (`server_group_id` would have nowhere to point). /// When the operator later assigns the server to a group, /// [`Server::assign_to_group`] runs `reevaluate_open_issues_for_server` @@ -247,7 +204,7 @@ impl NewEvent { server_id: Uuid, device_id: Option, ) -> Result { - use crate::schema::{events, issues}; + use crate::schema::issues; // description is the single-line title; reject multi-line input // up front so the UI never has to fight with a multi-line @@ -266,7 +223,6 @@ impl NewEvent { let now = Timestamp::now(); let effective_time = self.occurred_at.unwrap_or(now); let description = self.description.as_deref(); - let hash = hash_event(severity, active, &self.message, description); // Look up the server's group up-front; needed by the incident-open // path. If `None`, we still record the issue/event but skip @@ -357,46 +313,7 @@ impl NewEvent { .await? }; - // 2. coalesce into latest event or insert new. - let latest_event: Option = events::table - .select(Event::as_select()) - .filter(events::issue_id.eq(issue.id)) - .order(events::created_at.desc()) - .first(conn) - .await - .optional()?; - - let coalesce = matches!(&latest_event, Some(e) if e.hash == hash); - if let (true, Some(latest)) = (coalesce, latest_event) { - let new_last = if effective_time > latest.last_seen { - effective_time - } else { - latest.last_seen - }; - diesel::update(events::table.filter(events::id.eq(latest.id))) - .set(( - events::occurrences.eq(latest.occurrences + 1), - events::last_seen.eq(jiff_diesel::Timestamp::from(new_last)), - )) - .execute(conn) - .await?; - } else { - diesel::insert_into(events::table) - .values(( - events::issue_id.eq(issue.id), - events::occurred_at.eq(self.occurred_at.map(jiff_diesel::Timestamp::from)), - events::severity.eq(severity), - events::description.eq(description), - events::message.eq(&self.message), - events::active.eq(active), - events::hash.eq(&hash), - events::last_seen.eq(jiff_diesel::Timestamp::from(effective_time)), - )) - .execute(conn) - .await?; - } - - // 3. (re-)evaluate incident contribution against the new issue state. + // 2. (re-)evaluate incident contribution against the new issue state. // `by = None`: this came from a device push, not an operator action. // Skipped when the server is ungrouped: incidents are group-keyed. if let Some(gid) = server_group_id { @@ -418,8 +335,8 @@ impl NewEvent { /// Mirrors [`NewEvent::save`] but keys the issue on /// `(server_group_id, source, ref)` instead of `(server_id, …)`: /// 1. find-or-create the group-scoped issue (server_id = NULL), -/// 2. append the event or coalesce into the latest matching one, -/// 3. run incident membership evaluation with `monitored = true` so the +/// updating its state from this report, +/// 2. run incident membership evaluation with `monitored = true` so the /// incident opens/pages regardless of any member server's monitored flag. /// /// Recovery is the same `(source, ref)` with `active = false` at a lower @@ -427,8 +344,7 @@ impl NewEvent { /// identical lifecycle to the per-server path. /// /// `description` is an optional single-line headline (rejected if multi-line); -/// `message` is the body. Both feed the dedup hash, so a repeated identical -/// alert coalesces into one event with a bumped occurrence count. +/// `message` is the body. pub async fn raise_group_event( conn: &mut AsyncPgConnection, group_id: Uuid, @@ -438,7 +354,7 @@ pub async fn raise_group_event( message: &str, active: bool, ) -> Result { - use crate::schema::{events, issues}; + use crate::schema::issues; if let Some(d) = description && d.contains('\n') @@ -450,7 +366,6 @@ pub async fn raise_group_event( let source = crate::statuses::CANOPY_SOURCE; let now = Timestamp::now(); - let hash = hash_event(severity, active, message, description); conn.transaction::<_, AppError, _>(async |conn| { // 1. find-or-create the group-scoped issue. @@ -524,45 +439,7 @@ pub async fn raise_group_event( .await? }; - // 2. coalesce into latest event or insert new. - let latest_event: Option = events::table - .select(Event::as_select()) - .filter(events::issue_id.eq(issue.id)) - .order(events::created_at.desc()) - .first(conn) - .await - .optional()?; - - let coalesce = matches!(&latest_event, Some(e) if e.hash == hash); - if let (true, Some(latest)) = (coalesce, latest_event) { - let new_last = if now > latest.last_seen { - now - } else { - latest.last_seen - }; - diesel::update(events::table.filter(events::id.eq(latest.id))) - .set(( - events::occurrences.eq(latest.occurrences + 1), - events::last_seen.eq(jiff_diesel::Timestamp::from(new_last)), - )) - .execute(conn) - .await?; - } else { - diesel::insert_into(events::table) - .values(( - events::issue_id.eq(issue.id), - events::severity.eq(severity), - events::description.eq(description), - events::message.eq(message), - events::active.eq(active), - events::hash.eq(&hash), - events::last_seen.eq(jiff_diesel::Timestamp::from(now)), - )) - .execute(conn) - .await?; - } - - // 3. group-aware incident evaluation — monitored = true unconditionally. + // 2. group-aware incident evaluation — monitored = true unconditionally. re_evaluate_incident_membership(conn, &issue, group_id, true, now, None).await?; Ok(issue) @@ -1537,7 +1414,6 @@ pub struct IssueIncidentRef { #[derive(Debug, Clone, Copy, Default)] pub struct IncidentStats { pub issue_count: i64, - pub event_count: i64, /// `incident_notes` for this incident + `issue_notes` across all linked issues. pub note_count: i64, } @@ -1549,18 +1425,18 @@ impl Incident { /// `incident_issues` is keyed on `(incident_id, issue_id, joined_at)`, /// so an issue that leaves and rejoins the same incident produces /// multiple rows for the same pair. Counts must dedupe on the pair - /// before joining to `events` or `issue_notes`, otherwise every event - /// or note gets multiplied by the rejoin count. + /// before joining to `issue_notes`, otherwise every note gets + /// multiplied by the rejoin count. /// /// Strategy: pull the distinct `(incident_id, issue_id)` pairs first, - /// then run the per-issue event/note counts and the direct - /// incident_notes count concurrently. Takes the pool (`&Db`) so the - /// parallel futures don't fight over one mutable conn handle. + /// then run the per-issue note counts and the direct incident_notes + /// count concurrently. Takes the pool (`&Db`) so the parallel futures + /// don't fight over one mutable conn handle. pub async fn stats_for( pool: &crate::Db, incident_ids: &[Uuid], ) -> Result> { - use crate::schema::{events, incident_issues, incident_notes, issue_notes}; + use crate::schema::{incident_issues, incident_notes, issue_notes}; use diesel::dsl::count_star; use std::collections::{HashMap, HashSet}; @@ -1602,19 +1478,6 @@ impl Incident { .into_iter() .collect(); - let f_events = async { - if unique_issue_ids.is_empty() { - return Result::>::Ok(Vec::new()); - } - let mut c = pool.get().await?; - events::table - .group_by(events::issue_id) - .select((events::issue_id, count_star())) - .filter(events::issue_id.eq_any(&unique_issue_ids)) - .load::<(Uuid, i64)>(&mut c) - .await - .map_err(AppError::from) - }; let f_inotes = async { let mut c = pool.get().await?; incident_notes::table @@ -1638,16 +1501,13 @@ impl Incident { .await .map_err(AppError::from) }; - let (event_rows, inote_rows, jnote_rows) = - futures::try_join!(f_events, f_inotes, f_jnotes)?; + let (inote_rows, jnote_rows) = futures::try_join!(f_inotes, f_jnotes)?; - let events_per_issue: HashMap = event_rows.into_iter().collect(); let notes_per_issue: HashMap = jnote_rows.into_iter().collect(); for (incident_id, issues) in &issues_by_incident { let entry = out.entry(*incident_id).or_default(); for issue_id in issues { - entry.event_count += events_per_issue.get(issue_id).copied().unwrap_or(0); entry.note_count += notes_per_issue.get(issue_id).copied().unwrap_or(0); } } @@ -1659,38 +1519,6 @@ impl Incident { } } -impl Event { - pub async fn list_for_issue( - db: &mut AsyncPgConnection, - issue_id: Uuid, - offset: i64, - limit: i64, - ) -> Result> { - use crate::schema::events::dsl; - - dsl::events - .select(Self::as_select()) - .filter(dsl::issue_id.eq(issue_id)) - .order(dsl::created_at.desc()) - .offset(offset) - .limit(limit) - .load(db) - .await - .map_err(AppError::from) - } - - pub async fn count_for_issue(db: &mut AsyncPgConnection, issue_id: Uuid) -> Result { - use crate::schema::events::dsl; - - dsl::events - .filter(dsl::issue_id.eq(issue_id)) - .count() - .get_result(db) - .await - .map_err(AppError::from) - } -} - impl Incident { /// Incidents are owned by the server's group, so a caller asking for a /// specific server's incidents really wants the group's. Look up the diff --git a/crates/database/src/schema.rs b/crates/database/src/schema.rs index 16d4c3ec..5f334c9e 100644 --- a/crates/database/src/schema.rs +++ b/crates/database/src/schema.rs @@ -223,22 +223,6 @@ diesel::table! { } } -diesel::table! { - events (id) { - id -> Uuid, - created_at -> Timestamptz, - occurred_at -> Nullable, - issue_id -> Uuid, - severity -> Text, - description -> Nullable, - message -> Text, - active -> Bool, - hash -> Bytea, - occurrences -> Int4, - last_seen -> Timestamptz, - } -} - diesel::table! { healthcheck_severities (check_name) { check_name -> Text, @@ -604,7 +588,6 @@ diesel::joinable!(device_connections -> devices (device_id)); diesel::joinable!(device_keys -> devices (device_id)); diesel::joinable!(device_server_associations -> devices (device_id)); diesel::joinable!(device_server_associations -> servers (server_id)); -diesel::joinable!(events -> issues (issue_id)); diesel::joinable!(incident_issues -> incidents (incident_id)); diesel::joinable!(incident_issues -> issues (issue_id)); diesel::joinable!(incident_notes -> incidents (incident_id)); @@ -649,7 +632,6 @@ diesel::allow_tables_to_appear_in_same_query!( device_keys, device_server_associations, devices, - events, healthcheck_severities, incident_issues, incident_notes, diff --git a/crates/database/tests/it/incident_stats.rs b/crates/database/tests/it/incident_stats.rs index be9f6677..c11d5098 100644 --- a/crates/database/tests/it/incident_stats.rs +++ b/crates/database/tests/it/incident_stats.rs @@ -2,8 +2,8 @@ //! counting. `incident_issues` is keyed on `(incident_id, issue_id, //! joined_at)`, so an issue that leaves and rejoins the same incident //! produces multiple rows for the same pair. Without dedup, the issue -//! count inflates by the number of rejoins, and the event/note counts -//! get multiplied by the same factor. +//! count inflates by the number of rejoins, and the note counts get +//! multiplied by the same factor. use database::issues::Incident; use diesel_async::SimpleAsyncConnection as _; @@ -17,8 +17,6 @@ async fn stats_for_dedupes_repeat_join_rows() { let device_id = Uuid::new_v4(); let issue_id = Uuid::new_v4(); let incident_id = Uuid::new_v4(); - let event_a = Uuid::new_v4(); - let event_b = Uuid::new_v4(); let issue_note = Uuid::new_v4(); let incident_note = Uuid::new_v4(); @@ -33,11 +31,6 @@ async fn stats_for_dedupes_repeat_join_rows() { ('{issue_id}', '{server_id}', '{device_id}', 'test', 'r', 'error', 'm', true, NOW(), NOW()); \ INSERT INTO incidents (id, server_group_id, opened_at) \ VALUES ('{incident_id}', '{group_id}', NOW()); \ - INSERT INTO events \ - (id, issue_id, severity, message, active, hash, occurrences, last_seen) \ - VALUES \ - ('{event_a}', '{issue_id}', 'error', 'one', true, '\\x01'::bytea, 7, NOW()), \ - ('{event_b}', '{issue_id}', 'error', 'two', true, '\\x02'::bytea, 1, NOW()); \ INSERT INTO issue_notes (id, issue_id, author, body) \ VALUES ('{issue_note}', '{issue_id}', 'op', 'jn'); \ INSERT INTO incident_notes (id, incident_id, author, body) \ @@ -59,10 +52,6 @@ async fn stats_for_dedupes_repeat_join_rows() { .expect("stats_for"); let s = stats.get(&incident_id).expect("stats present"); assert_eq!(s.issue_count, 1, "one distinct issue despite 5 join rows"); - assert_eq!( - s.event_count, 2, - "two event rows, not multiplied by 5 join rows" - ); assert_eq!( s.note_count, 2, "one incident_note + one issue_note, not 1 + 5" @@ -88,7 +77,6 @@ async fn stats_for_handles_missing_and_empty_inputs() { .expect("phantom"); let s = stats.get(&phantom).expect("phantom entry"); assert_eq!(s.issue_count, 0); - assert_eq!(s.event_count, 0); assert_eq!(s.note_count, 0); }) .await diff --git a/crates/database/tests/it/main.rs b/crates/database/tests/it/main.rs index 241e0647..ee63d455 100644 --- a/crates/database/tests/it/main.rs +++ b/crates/database/tests/it/main.rs @@ -17,7 +17,6 @@ mod incident_stats; mod mcp_tokens; mod reachability_sweep; mod recovery_vault; -mod resolve_overall_health_rollups_migration; mod restore; mod self_alerts; mod server_enrollment; diff --git a/crates/database/tests/it/resolve_overall_health_rollups_migration.rs b/crates/database/tests/it/resolve_overall_health_rollups_migration.rs deleted file mode 100644 index f3e46376..00000000 --- a/crates/database/tests/it/resolve_overall_health_rollups_migration.rs +++ /dev/null @@ -1,225 +0,0 @@ -//! The `2026-05-28-100000-0000_resolve_overall_health_rollups` migration -//! retires the `(status, health)` rollup issue category — see -//! `docs/plans/healthcheck-severity-catalog.md`. This test seeds the -//! pre-deprecation state (active rollup issue, incident contributed to -//! only by that issue, pending Slack `incident_open`) and replays the -//! migration's SQL to verify the full cleanup cascade. - -use diesel::sql_types; -use diesel_async::{RunQueryDsl, SimpleAsyncConnection as _}; -use uuid::Uuid; - -const MIGRATION_UP: &str = include_str!( - "../../../../migrations/2026-05-28-100000-0000_resolve_overall_health_rollups/up.sql" -); - -#[tokio::test(flavor = "multi_thread")] -async fn migration_retires_active_rollup_and_orphan_incident() { - commons_tests::db::TestDb::run(async |mut conn, _| { - let group_id = Uuid::new_v4(); - let server_id = Uuid::new_v4(); - let device_id = Uuid::new_v4(); - let rollup_issue_id = Uuid::new_v4(); - let incident_id = Uuid::new_v4(); - let outbox_id = Uuid::new_v4(); - - conn.batch_execute(&format!( - "INSERT INTO devices (id, role) VALUES ('{device_id}', 'server'); \ - INSERT INTO server_groups (id, name) VALUES ('{group_id}', 'g'); \ - INSERT INTO servers (id, host, kind, device_id, group_id) VALUES \ - ('{server_id}', 'https://example.com', 'central', '{device_id}', '{group_id}'); \ - INSERT INTO issues \ - (id, server_id, device_id, source, ref, severity, message, active, first_seen, last_seen) \ - VALUES \ - ('{rollup_issue_id}', '{server_id}', '{device_id}', 'status', 'health', 'error', \ - 'Server reports unhealthy', true, NOW() - interval '1 hour', NOW() - interval '5 min'); \ - INSERT INTO incidents (id, server_group_id, opened_at) \ - VALUES ('{incident_id}', '{group_id}', NOW() - interval '1 hour'); \ - INSERT INTO incident_issues (incident_id, issue_id, joined_at) VALUES \ - ('{incident_id}', '{rollup_issue_id}', NOW() - interval '1 hour'); \ - INSERT INTO slack_outbox (id, kind, incident_id, issue_id, payload, deliver_after) \ - VALUES \ - ('{outbox_id}', 'incident_open', '{incident_id}', '{rollup_issue_id}', \ - '{{}}'::jsonb, NOW() + interval '5 min');" - )) - .await - .expect("seed"); - - // Replay the migration's up.sql. - conn.batch_execute(MIGRATION_UP) - .await - .expect("replay migration up.sql"); - - // 1. The rollup issue is deactivated and human-resolved. - let row: IssueState = diesel::sql_query( - "SELECT active, resolved_at IS NOT NULL AS resolved, resolved_by, resolved_reason \ - FROM issues WHERE id = $1", - ) - .bind::(rollup_issue_id) - .get_result(&mut conn) - .await - .expect("rollup issue row still exists"); - assert!(!row.active, "rollup issue must be deactivated"); - assert!(row.resolved, "resolved_at must be set"); - assert_eq!( - row.resolved_by.as_deref(), - Some("migration:2026-05-28-resolve_overall_health_rollups") - ); - assert_eq!(row.resolved_reason.as_deref(), Some("expected")); - - // 2. A close event with the synthetic hash + retirement message - // has been appended to the rollup issue. - let evt: EventState = diesel::sql_query( - "SELECT severity, active, message FROM events WHERE issue_id = $1 ORDER BY created_at DESC LIMIT 1", - ) - .bind::(rollup_issue_id) - .get_result(&mut conn) - .await - .expect("close event was appended"); - assert_eq!(evt.severity, "info"); - assert!(!evt.active, "close event has active=false"); - assert_eq!( - evt.message, - "Overall-health roll-up retired; per-check issues remain." - ); - - // 3. The incident_issues link is marked left. - let link: LinkState = diesel::sql_query( - "SELECT left_at IS NOT NULL AS departed FROM incident_issues \ - WHERE incident_id = $1 AND issue_id = $2", - ) - .bind::(incident_id) - .bind::(rollup_issue_id) - .get_result(&mut conn) - .await - .expect("link row"); - assert!( - link.departed, - "incident_issues.left_at must be set after migration" - ); - - // 4. The orphan incident (no other live contributors) is closed. - let inc: IncidentState = diesel::sql_query( - "SELECT closed_at IS NOT NULL AS closed FROM incidents WHERE id = $1", - ) - .bind::(incident_id) - .get_result(&mut conn) - .await - .expect("incident row"); - assert!(inc.closed, "orphan incident must be closed"); - - // 5. Pending Slack incident_open is cancelled (gave_up_at set, - // last_error explains why). - let outbox: OutboxState = diesel::sql_query( - "SELECT gave_up_at IS NOT NULL AS given_up, last_error \ - FROM slack_outbox WHERE id = $1", - ) - .bind::(outbox_id) - .get_result(&mut conn) - .await - .expect("outbox row"); - assert!(outbox.given_up, "pending Slack open must be given up"); - assert!( - outbox - .last_error - .as_deref() - .is_some_and(|m| m.contains("overall-health rollup retirement migration")), - "last_error explains the cancellation: {:?}", - outbox.last_error - ); - }) - .await -} - -#[tokio::test(flavor = "multi_thread")] -async fn migration_leaves_incident_with_other_live_contributors_open() { - commons_tests::db::TestDb::run(async |mut conn, _| { - let group_id = Uuid::new_v4(); - let server_id = Uuid::new_v4(); - let device_id = Uuid::new_v4(); - let rollup_issue_id = Uuid::new_v4(); - let per_check_issue_id = Uuid::new_v4(); - let incident_id = Uuid::new_v4(); - - // Rollup + a separate per-check issue both contributing to the - // same incident. The per-check issue keeps the incident alive. - conn.batch_execute(&format!( - "INSERT INTO devices (id, role) VALUES ('{device_id}', 'server'); \ - INSERT INTO server_groups (id, name) VALUES ('{group_id}', 'g'); \ - INSERT INTO servers (id, host, kind, device_id, group_id) VALUES \ - ('{server_id}', 'https://example.com', 'central', '{device_id}', '{group_id}'); \ - INSERT INTO issues \ - (id, server_id, device_id, source, ref, severity, message, active, first_seen, last_seen) VALUES \ - ('{rollup_issue_id}', '{server_id}', '{device_id}', 'status', 'health', \ - 'error', 'roll', true, NOW(), NOW()), \ - ('{per_check_issue_id}', '{server_id}', '{device_id}', 'status', 'health/db', \ - 'error', 'db down', true, NOW(), NOW()); \ - INSERT INTO incidents (id, server_group_id, opened_at) \ - VALUES ('{incident_id}', '{group_id}', NOW()); \ - INSERT INTO incident_issues (incident_id, issue_id, joined_at) VALUES \ - ('{incident_id}', '{rollup_issue_id}', NOW()), \ - ('{incident_id}', '{per_check_issue_id}', NOW());" - )) - .await - .expect("seed"); - - conn.batch_execute(MIGRATION_UP) - .await - .expect("replay migration up.sql"); - - let inc: IncidentState = diesel::sql_query( - "SELECT closed_at IS NOT NULL AS closed FROM incidents WHERE id = $1", - ) - .bind::(incident_id) - .get_result(&mut conn) - .await - .expect("incident row"); - assert!( - !inc.closed, - "incident must stay open while another live contributor exists" - ); - }) - .await -} - -#[derive(diesel::QueryableByName)] -struct IssueState { - #[diesel(sql_type = sql_types::Bool)] - active: bool, - #[diesel(sql_type = sql_types::Bool)] - resolved: bool, - #[diesel(sql_type = sql_types::Nullable)] - resolved_by: Option, - #[diesel(sql_type = sql_types::Nullable)] - resolved_reason: Option, -} - -#[derive(diesel::QueryableByName)] -struct EventState { - #[diesel(sql_type = sql_types::Text)] - severity: String, - #[diesel(sql_type = sql_types::Bool)] - active: bool, - #[diesel(sql_type = sql_types::Text)] - message: String, -} - -#[derive(diesel::QueryableByName)] -struct LinkState { - #[diesel(sql_type = sql_types::Bool)] - departed: bool, -} - -#[derive(diesel::QueryableByName)] -struct IncidentState { - #[diesel(sql_type = sql_types::Bool)] - closed: bool, -} - -#[derive(diesel::QueryableByName)] -struct OutboxState { - #[diesel(sql_type = sql_types::Bool)] - given_up: bool, - #[diesel(sql_type = sql_types::Nullable)] - last_error: Option, -} diff --git a/crates/private-server/src/fns/incidents.rs b/crates/private-server/src/fns/incidents.rs index 15080089..bb3c945e 100644 --- a/crates/private-server/src/fns/incidents.rs +++ b/crates/private-server/src/fns/incidents.rs @@ -52,8 +52,6 @@ pub struct IncidentData { pub resolved_reason: Option, /// Number of distinct issues that have ever contributed to the incident. pub issue_count: i64, - /// Total number of events across all contributing issues. - pub event_count: i64, /// Combined count of notes on the incident itself plus notes on all its /// contributing issues. pub note_count: i64, @@ -91,7 +89,6 @@ impl IncidentData { resolved_by_pic: res_pic, resolved_reason: i.resolved_reason, issue_count: stats.issue_count, - event_count: stats.event_count, note_count: stats.note_count, notification_held_until, created_at: i.created_at, diff --git a/crates/private-server/src/fns/issues.rs b/crates/private-server/src/fns/issues.rs index faa85a1b..432bb986 100644 --- a/crates/private-server/src/fns/issues.rs +++ b/crates/private-server/src/fns/issues.rs @@ -8,7 +8,7 @@ use commons_types::{ issue::{ResolvedReason, Severity}, }; use database::issues::{ - Event, Incident, Issue, IssueFilter, IssueIncidentRef, IssueListFilters, NewEvent, + Incident, Issue, IssueFilter, IssueIncidentRef, IssueListFilters, NewEvent, }; use database::notes::IssueNote; use database::servers::Server; @@ -17,7 +17,6 @@ use jiff::Timestamp; use serde::{Deserialize, Serialize}; use utoipa::ToSchema; -use crate::fns::Page; use crate::state::AppState; const DEFAULT_LIMIT: i64 = 100; @@ -261,57 +260,11 @@ pub(crate) async fn enrich_issue( )) } -/// A single recorded occurrence of an issue's underlying condition — one -/// push from a device or a manually submitted event. -#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)] -pub struct EventData { - /// Unique identifier for this event. - pub id: Uuid, - /// Id of the issue this event belongs to. - pub issue_id: Uuid, - /// When this event was recorded on the server. - pub created_at: Timestamp, - /// When the underlying condition actually occurred, if reported - /// separately from the time it was recorded. - pub occurred_at: Option, - /// Severity reported for this event. - pub severity: Severity, - /// Short headline for this event, if one was given. - pub description: Option, - /// Human-readable message describing this event. - pub message: String, - /// Whether the underlying condition was active as of this event. - pub active: bool, - /// Number of times this same condition has repeated and been coalesced - /// into this event rather than creating a new one. - pub occurrences: i32, - /// When this condition was last seen recurring. - pub last_seen: Timestamp, -} - -impl From for EventData { - fn from(e: Event) -> Self { - Self { - id: e.id, - issue_id: e.issue_id, - created_at: e.created_at, - occurred_at: e.occurred_at, - severity: e.severity, - description: e.description, - message: e.message, - active: e.active, - occurrences: e.occurrences, - last_seen: e.last_seen, - } - } -} - pub fn routes() -> OpenApiRouter { OpenApiRouter::new() .routes(routes!(list)) .routes(routes!(list_for_device)) .routes(routes!(list_for_server)) - .routes(routes!(list_events)) .routes(routes!(submit_manual_event)) .routes(routes!(resolve)) .routes(routes!(unresolve)) @@ -476,51 +429,6 @@ pub async fn list_for_server( Ok(Json(enrich_issues(&mut conn, issues).await?)) } -/// Selects an issue and a page of its events. -#[derive(Deserialize, ToSchema)] -pub struct ListEventsArgs { - /// Id of the issue whose events to list. - pub issue_id: Uuid, - /// Number of events to skip, for pagination. Defaults to 0. - #[serde(default)] - pub offset: Option, - /// Maximum number of events to return. Defaults to 100 when omitted. - #[serde(default)] - pub limit: Option, -} - -/// List the events recorded against a specific issue, most recent first. -/// -/// Returns a page of events along with the total number of events recorded -/// for the issue, for pagination with `offset`/`limit`. -#[utoipa::path( - post, - path = "/list_events", - tag = "issues", - security(("tailscale-user" = [])), - request_body = ListEventsArgs, - responses( - (status = 200, body = Page), - ), -)] -pub async fn list_events( - State(state): State, - _user: TailscaleUser, - Json(args): Json, -) -> Result>> { - let mut conn = state.db_read.get().await?; - let total = Event::count_for_issue(&mut conn, args.issue_id).await? as u64; - let events = Event::list_for_issue( - &mut conn, - args.issue_id, - args.offset.unwrap_or(0), - args.limit.unwrap_or(DEFAULT_LIMIT), - ) - .await?; - let items = events.into_iter().map(EventData::from).collect(); - Ok(Json(Page { items, total })) -} - /// A manually entered event to record against a server. #[derive(Deserialize, ToSchema)] #[serde(rename_all = "camelCase")] diff --git a/crates/private-server/tests/it/issues.rs b/crates/private-server/tests/it/issues.rs index 5197bc46..c16076e5 100644 --- a/crates/private-server/tests/it/issues.rs +++ b/crates/private-server/tests/it/issues.rs @@ -474,79 +474,6 @@ async fn severity_downgrade_keeps_issue_in_incident() { .await; } -#[tokio::test(flavor = "multi_thread")] -async fn list_events_returns_event_log() { - commons_tests::server::run(async |mut conn, _public, private| { - let server_id = Uuid::new_v4(); - conn.batch_execute(&format!( - "INSERT INTO servers (id, host, kind) VALUES \ - ('{server_id}', 'https://example.com', 'central');" - )) - .await - .expect("seed"); - - // Three distinct events. - for (sev, msg) in [("error", "a"), ("error", "b"), ("warning", "b")] { - private - .post("/api/issues/submit_manual_event") - .json(&serde_json::json!({ - "serverId": server_id, - "ref": "x", - "severity": sev, - "message": msg, - })) - .await - .assert_status_ok(); - } - - // Find the issue. - let issues = private - .post("/api/issues/list_for_server") - .json(&serde_json::json!({ "server_id": server_id })) - .await; - issues.assert_status_ok(); - let items: Vec = issues.json(); - assert_eq!(items.len(), 1); - let issue_id = items[0].get("id").unwrap().as_str().unwrap(); - - let events = private - .post("/api/issues/list_events") - .json(&serde_json::json!({ "issue_id": issue_id })) - .await; - events.assert_status_ok(); - let page: serde_json::Value = events.json(); - let items = page.get("items").and_then(|v| v.as_array()).unwrap(); - assert_eq!(items.len(), 3, "each distinct push is its own event row"); - assert_eq!(page.get("total").and_then(|v| v.as_u64()), Some(3)); - - // Pagination: limit=2 returns 2 items but total still reflects 3. - let page1 = private - .post("/api/issues/list_events") - .json(&serde_json::json!({ "issue_id": issue_id, "offset": 0, "limit": 2 })) - .await; - page1.assert_status_ok(); - let page1: serde_json::Value = page1.json(); - assert_eq!( - page1.get("items").and_then(|v| v.as_array()).unwrap().len(), - 2 - ); - assert_eq!(page1.get("total").and_then(|v| v.as_u64()), Some(3)); - - let page2 = private - .post("/api/issues/list_events") - .json(&serde_json::json!({ "issue_id": issue_id, "offset": 2, "limit": 2 })) - .await; - page2.assert_status_ok(); - let page2: serde_json::Value = page2.json(); - assert_eq!( - page2.get("items").and_then(|v| v.as_array()).unwrap().len(), - 1 - ); - assert_eq!(page2.get("total").and_then(|v| v.as_u64()), Some(3)); - }) - .await; -} - async fn open_issue( conn: &mut database::diesel_async::AsyncPgConnection, private: &commons_tests::axum_test::TestServer, diff --git a/crates/private-server/tests/it/mcp.rs b/crates/private-server/tests/it/mcp.rs index e02f9457..e4813574 100644 --- a/crates/private-server/tests/it/mcp.rs +++ b/crates/private-server/tests/it/mcp.rs @@ -349,8 +349,6 @@ async fn seed_incidents(conn: &mut impl SimpleAsyncConnection) { INSERT INTO issues (id, created_at, updated_at, server_id, source, ref, severity, description, message, active, first_seen, last_seen) VALUES \ ('{ISSUE1}', NOW(), NOW(), '{ISRV}', 'test', 'r1', 'error', 'Disk full', 'disk usage 98%', true, NOW() - interval '2 days', NOW() - interval '1 hour'), \ ('{ISSUE2}', NOW(), NOW(), '{ISRV}', 'test', 'r2', 'warning', NULL, 'slow query', false, NOW() - interval '10 days', NOW() - interval '9 days'); \ - INSERT INTO events (id, created_at, issue_id, severity, message, active, hash, occurrences, last_seen) VALUES \ - (gen_random_uuid(), NOW() - interval '1 hour', '{ISSUE1}', 'error', 'disk usage 98%', true, '\\x00'::bytea, 3, NOW() - interval '1 hour'); \ INSERT INTO incidents (id, created_at, updated_at, server_group_id, opened_at, closed_at) VALUES \ ('{INC_OPEN}', NOW(), NOW(), '{IGROUP}', NOW() - interval '2 days', NULL), \ ('{INC_CLOSED}', NOW(), NOW(), '{IGROUP}', NOW() - interval '5 days', NOW() - interval '3 days'); \ @@ -472,14 +470,13 @@ async fn issues_filter_and_detail() { let w = warnings["issues"].as_array().unwrap(); assert!(!w.is_empty() && w.iter().all(|i| i["severity"] == "warning")); - // Detail: events + the incident it belongs to. + // Detail: the issue's fields + the incident it belongs to. let detail = call_tool!( private, "get_issue", serde_json::json!({ "issue_id": ISSUE1 }) ); assert_eq!(detail["severity"], "error"); - assert!(!detail["recent_events"].as_array().unwrap().is_empty()); let inc_ids: Vec = detail["incidents"] .as_array() .unwrap() diff --git a/crates/public-server/openapi.json b/crates/public-server/openapi.json index 0f6cd72d..5f2dac26 100644 --- a/crates/public-server/openapi.json +++ b/crates/public-server/openapi.json @@ -1710,7 +1710,7 @@ }, "NewEvent": { "type": "object", - "description": "A single occurrence to report against an issue, sent either by a device\nor by an operator. If an issue with the same `source` and `ref` is\nalready open, this occurrence is folded into it (bumping its last-seen\ntime, and coalescing into the latest recorded event when the content is\nidentical); otherwise a new issue is created.", + "description": "A single occurrence to report against an issue, sent either by a device\nor by an operator. If an issue with the same `source` and `ref` is\nalready open, this occurrence is folded into it (updating its state and\nbumping its last-seen time); otherwise a new issue is created.", "required": [ "source", "ref", diff --git a/crates/public-server/tests/it/events.rs b/crates/public-server/tests/it/events.rs index 6cc71c99..bd7fbbf9 100644 --- a/crates/public-server/tests/it/events.rs +++ b/crates/public-server/tests/it/events.rs @@ -150,33 +150,20 @@ async fn submit_event_dedups_by_ref() { assert_eq!(row.server_id, server_id); assert_eq!(row.severity, "critical"); assert_eq!(row.message, "less than 1% free"); - - #[derive(QueryableByName)] - struct Counts { - #[diesel(sql_type = sql_types::BigInt)] - event_count: i64, - } - let counts: Counts = - sql_query("SELECT COUNT(*) AS event_count FROM events WHERE issue_id = $1") - .bind::(Uuid::parse_str(&first_id).unwrap()) - .get_result(&mut conn) - .await - .expect("count events"); - assert_eq!(counts.event_count, 2, "different content → two event rows"); }, ) .await } #[tokio::test(flavor = "multi_thread")] -async fn submit_event_coalesces_identical_pushes() { +async fn submit_event_repeated_pushes_fold_into_one_issue() { commons_tests::server::run_with_device_auth( "server", async |mut conn, cert, device_id, public, _| { provision_server(&mut conn, device_id).await; - // Three identical pushes. - let mut issue_id = String::new(); + // Three identical pushes fold into the one issue. + let mut ids = Vec::new(); for _ in 0..3 { let r = public .post("/events") @@ -188,27 +175,20 @@ async fn submit_event_coalesces_identical_pushes() { })) .await; r.assert_status_ok(); - issue_id = r.json::().get("id").unwrap().as_str().unwrap().to_string(); + ids.push( + r.json::() + .get("id") + .unwrap() + .as_str() + .unwrap() + .to_string(), + ); } - #[derive(QueryableByName)] - struct Counts { - #[diesel(sql_type = sql_types::BigInt)] - event_count: i64, - #[diesel(sql_type = sql_types::Integer)] - latest_occurrences: i32, - } - let counts: Counts = sql_query( - "SELECT \ - (SELECT COUNT(*) FROM events WHERE issue_id = $1) AS event_count, \ - (SELECT occurrences FROM events WHERE issue_id = $1 ORDER BY created_at DESC LIMIT 1) AS latest_occurrences", - ) - .bind::(Uuid::parse_str(&issue_id).unwrap()) - .get_result(&mut conn) - .await - .expect("count"); - assert_eq!(counts.event_count, 1, "identical pushes collapse into one row"); - assert_eq!(counts.latest_occurrences, 3); + ids.dedup(); + assert_eq!(ids.len(), 1, "repeat pushes fold into one issue"); + let row = issue_by_id(&mut conn, Uuid::parse_str(&ids[0]).unwrap()).await; + assert_eq!(row.message, "same content"); }, ) .await diff --git a/migrations/2026-07-08-070653-0000_drop_events/down.sql b/migrations/2026-07-08-070653-0000_drop_events/down.sql new file mode 100644 index 00000000..ebc7083f --- /dev/null +++ b/migrations/2026-07-08-070653-0000_drop_events/down.sql @@ -0,0 +1,16 @@ +-- Recreates the table shape only; the audit history is not recoverable. +CREATE TABLE events ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(), + occurred_at TIMESTAMP WITH TIME ZONE, + issue_id UUID NOT NULL REFERENCES issues (id) ON DELETE CASCADE ON UPDATE CASCADE, + severity TEXT NOT NULL, + description TEXT, + message TEXT NOT NULL, + active BOOLEAN NOT NULL, + hash BYTEA NOT NULL, + occurrences INTEGER NOT NULL DEFAULT 1, + last_seen TIMESTAMP WITH TIME ZONE NOT NULL +); + +CREATE INDEX events_issue_created ON events (issue_id, created_at DESC); diff --git a/migrations/2026-07-08-070653-0000_drop_events/up.sql b/migrations/2026-07-08-070653-0000_drop_events/up.sql new file mode 100644 index 00000000..5f51372f --- /dev/null +++ b/migrations/2026-07-08-070653-0000_drop_events/up.sql @@ -0,0 +1,5 @@ +-- Events were an append-only audit log of issue state changes; nothing +-- derived state from them, and issue state itself is the durable record. +-- Statuses hold the per-check history. Unpartitioned and unpruned, the +-- table only grew; it goes away rather than gaining a retention scheme. +DROP TABLE events; diff --git a/private-web/openapi.json b/private-web/openapi.json index cdaf8e62..056db5cc 100644 --- a/private-web/openapi.json +++ b/private-web/openapi.json @@ -3263,43 +3263,6 @@ ] } }, - "/api/issues/list_events": { - "post": { - "tags": [ - "issues" - ], - "summary": "List the events recorded against a specific issue, most recent first.", - "description": "Returns a page of events along with the total number of events recorded\nfor the issue, for pagination with `offset`/`limit`.", - "operationId": "list_events", - "requestBody": { - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/ListEventsArgs" - } - } - }, - "required": true - }, - "responses": { - "200": { - "description": "", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Page_EventData" - } - } - } - } - }, - "security": [ - { - "tailscale-user": [] - } - ] - } - }, "/api/issues/list_for_device": { "post": { "tags": [ @@ -7541,74 +7504,6 @@ } } }, - "EventData": { - "type": "object", - "description": "A single recorded occurrence of an issue's underlying condition — one\npush from a device or a manually submitted event.", - "required": [ - "id", - "issue_id", - "created_at", - "severity", - "message", - "active", - "occurrences", - "last_seen" - ], - "properties": { - "active": { - "type": "boolean", - "description": "Whether the underlying condition was active as of this event." - }, - "created_at": { - "type": "string", - "format": "date-time", - "description": "When this event was recorded on the server." - }, - "description": { - "type": [ - "string", - "null" - ], - "description": "Short headline for this event, if one was given." - }, - "id": { - "type": "string", - "format": "uuid", - "description": "Unique identifier for this event." - }, - "issue_id": { - "type": "string", - "format": "uuid", - "description": "Id of the issue this event belongs to." - }, - "last_seen": { - "type": "string", - "format": "date-time", - "description": "When this condition was last seen recurring." - }, - "message": { - "type": "string", - "description": "Human-readable message describing this event." - }, - "occurred_at": { - "type": [ - "string", - "null" - ], - "format": "date-time", - "description": "When the underlying condition actually occurred, if reported\nseparately from the time it was recorded." - }, - "occurrences": { - "type": "integer", - "format": "int32", - "description": "Number of times this same condition has repeated and been coalesced\ninto this event rather than creating a new one." - }, - "severity": { - "$ref": "#/components/schemas/Severity", - "description": "Severity reported for this event." - } - } - }, "ExecuteArgs": { "type": "object", "description": "Request body for running a query in the SQL playground.", @@ -8128,7 +8023,6 @@ "server_group_name", "opened_at", "issue_count", - "event_count", "note_count", "created_at", "updated_at" @@ -8147,11 +8041,6 @@ "format": "date-time", "description": "When the incident record was created." }, - "event_count": { - "type": "integer", - "format": "int64", - "description": "Total number of events across all contributing issues." - }, "id": { "type": "string", "format": "uuid", @@ -8933,36 +8822,6 @@ } } }, - "ListEventsArgs": { - "type": "object", - "description": "Selects an issue and a page of its events.", - "required": [ - "issue_id" - ], - "properties": { - "issue_id": { - "type": "string", - "format": "uuid", - "description": "Id of the issue whose events to list." - }, - "limit": { - "type": [ - "integer", - "null" - ], - "format": "int64", - "description": "Maximum number of events to return. Defaults to 100 when omitted." - }, - "offset": { - "type": [ - "integer", - "null" - ], - "format": "int64", - "description": "Number of events to skip, for pagination. Defaults to 0." - } - } - }, "ListForDeviceArgs": { "type": "object", "description": "Filters for listing the issues raised by one device.", @@ -9351,94 +9210,6 @@ } } }, - "Page_EventData": { - "type": "object", - "description": "A single page of a paginated list response.", - "required": [ - "items", - "total" - ], - "properties": { - "items": { - "type": "array", - "items": { - "type": "object", - "description": "A single recorded occurrence of an issue's underlying condition — one\npush from a device or a manually submitted event.", - "required": [ - "id", - "issue_id", - "created_at", - "severity", - "message", - "active", - "occurrences", - "last_seen" - ], - "properties": { - "active": { - "type": "boolean", - "description": "Whether the underlying condition was active as of this event." - }, - "created_at": { - "type": "string", - "format": "date-time", - "description": "When this event was recorded on the server." - }, - "description": { - "type": [ - "string", - "null" - ], - "description": "Short headline for this event, if one was given." - }, - "id": { - "type": "string", - "format": "uuid", - "description": "Unique identifier for this event." - }, - "issue_id": { - "type": "string", - "format": "uuid", - "description": "Id of the issue this event belongs to." - }, - "last_seen": { - "type": "string", - "format": "date-time", - "description": "When this condition was last seen recurring." - }, - "message": { - "type": "string", - "description": "Human-readable message describing this event." - }, - "occurred_at": { - "type": [ - "string", - "null" - ], - "format": "date-time", - "description": "When the underlying condition actually occurred, if reported\nseparately from the time it was recorded." - }, - "occurrences": { - "type": "integer", - "format": "int32", - "description": "Number of times this same condition has repeated and been coalesced\ninto this event rather than creating a new one." - }, - "severity": { - "$ref": "#/components/schemas/Severity", - "description": "Severity reported for this event." - } - } - }, - "description": "The items in this page." - }, - "total": { - "type": "integer", - "format": "int64", - "description": "The total number of items across all pages, not just this one — use\nthis to render page counts without a separate request.", - "minimum": 0 - } - } - }, "Page_ServerInfo": { "type": "object", "description": "A single page of a paginated list response.", diff --git a/private-web/src/api-types.ts b/private-web/src/api-types.ts index ab26d004..141f846e 100644 --- a/private-web/src/api-types.ts +++ b/private-web/src/api-types.ts @@ -1632,27 +1632,6 @@ export interface paths { patch?: never; trace?: never; }; - "/api/issues/list_events": { - parameters: { - query?: never; - header?: never; - path?: never; - cookie?: never; - }; - get?: never; - put?: never; - /** - * List the events recorded against a specific issue, most recent first. - * @description Returns a page of events along with the total number of events recorded - * for the issue, for pagination with `offset`/`limit`. - */ - post: operations["list_events"]; - delete?: never; - options?: never; - head?: never; - patch?: never; - trace?: never; - }; "/api/issues/list_for_device": { parameters: { query?: never; @@ -4052,52 +4031,6 @@ export interface components { */ ticket: string; }; - /** - * @description A single recorded occurrence of an issue's underlying condition — one - * push from a device or a manually submitted event. - */ - EventData: { - /** @description Whether the underlying condition was active as of this event. */ - active: boolean; - /** - * Format: date-time - * @description When this event was recorded on the server. - */ - created_at: string; - /** @description Short headline for this event, if one was given. */ - description?: string | null; - /** - * Format: uuid - * @description Unique identifier for this event. - */ - id: string; - /** - * Format: uuid - * @description Id of the issue this event belongs to. - */ - issue_id: string; - /** - * Format: date-time - * @description When this condition was last seen recurring. - */ - last_seen: string; - /** @description Human-readable message describing this event. */ - message: string; - /** - * Format: date-time - * @description When the underlying condition actually occurred, if reported - * separately from the time it was recorded. - */ - occurred_at?: string | null; - /** - * Format: int32 - * @description Number of times this same condition has repeated and been coalesced - * into this event rather than creating a new one. - */ - occurrences: number; - /** @description Severity reported for this event. */ - severity: components["schemas"]["Severity"]; - }; /** @description Request body for running a query in the SQL playground. */ ExecuteArgs: { /** @description The query to execute. */ @@ -4472,11 +4405,6 @@ export interface components { * @description When the incident record was created. */ created_at: string; - /** - * Format: int64 - * @description Total number of events across all contributing issues. - */ - event_count: number; /** * Format: uuid * @description Unique identifier of the incident. @@ -5000,24 +4928,6 @@ export interface components { */ limit?: number | null; }; - /** @description Selects an issue and a page of its events. */ - ListEventsArgs: { - /** - * Format: uuid - * @description Id of the issue whose events to list. - */ - issue_id: string; - /** - * Format: int64 - * @description Maximum number of events to return. Defaults to 100 when omitted. - */ - limit?: number | null; - /** - * Format: int64 - * @description Number of events to skip, for pagination. Defaults to 0. - */ - offset?: number | null; - }; /** @description Filters for listing the issues raised by one device. */ ListForDeviceArgs: { /** @@ -5252,58 +5162,6 @@ export interface components { total: number; }; /** @description A single page of a paginated list response. */ - Page_EventData: { - /** @description The items in this page. */ - items: { - /** @description Whether the underlying condition was active as of this event. */ - active: boolean; - /** - * Format: date-time - * @description When this event was recorded on the server. - */ - created_at: string; - /** @description Short headline for this event, if one was given. */ - description?: string | null; - /** - * Format: uuid - * @description Unique identifier for this event. - */ - id: string; - /** - * Format: uuid - * @description Id of the issue this event belongs to. - */ - issue_id: string; - /** - * Format: date-time - * @description When this condition was last seen recurring. - */ - last_seen: string; - /** @description Human-readable message describing this event. */ - message: string; - /** - * Format: date-time - * @description When the underlying condition actually occurred, if reported - * separately from the time it was recorded. - */ - occurred_at?: string | null; - /** - * Format: int32 - * @description Number of times this same condition has repeated and been coalesced - * into this event rather than creating a new one. - */ - occurrences: number; - /** @description Severity reported for this event. */ - severity: components["schemas"]["Severity"]; - }[]; - /** - * Format: int64 - * @description The total number of items across all pages, not just this one — use - * this to render page counts without a separate request. - */ - total: number; - }; - /** @description A single page of a paginated list response. */ Page_ServerInfo: { /** @description The items in this page. */ items: { @@ -9683,29 +9541,6 @@ export interface operations { }; }; }; - list_events: { - parameters: { - query?: never; - header?: never; - path?: never; - cookie?: never; - }; - requestBody: { - content: { - "application/json": components["schemas"]["ListEventsArgs"]; - }; - }; - responses: { - 200: { - headers: { - [name: string]: unknown; - }; - content: { - "application/json": components["schemas"]["Page_EventData"]; - }; - }; - }; - }; list_for_device: { parameters: { query?: never; diff --git a/private-web/src/components/IncidentCard.tsx b/private-web/src/components/IncidentCard.tsx index 923d96e5..1f703b44 100644 --- a/private-web/src/components/IncidentCard.tsx +++ b/private-web/src/components/IncidentCard.tsx @@ -1,7 +1,6 @@ import { Box, Stack, Tooltip, Typography } from "@mui/material"; import BugReportIcon from "@mui/icons-material/BugReport"; import NotesIcon from "@mui/icons-material/StickyNote2"; -import TimelineIcon from "@mui/icons-material/Timeline"; import { Link as RouterLink } from "react-router-dom"; import TimeAgo from "./TimeAgo"; import { useIsNotificationHeld } from "../hooks/useIsNotificationHeld"; @@ -62,12 +61,7 @@ export default function IncidentCard({ incident }: { incident: IncidentData }) { value={incident.issue_count ?? 0} noun="issue" /> - } - value={incident.event_count ?? 0} - noun="event" - /> - } value={incident.note_count ?? 0} noun="note" diff --git a/private-web/src/components/IssueRow.tsx b/private-web/src/components/IssueRow.tsx index 55bdfa73..b9512abd 100644 --- a/private-web/src/components/IssueRow.tsx +++ b/private-web/src/components/IssueRow.tsx @@ -5,7 +5,6 @@ import { IconButton, Link as MuiLink, MenuItem, - Pagination, Stack, TextField, Typography, @@ -17,7 +16,7 @@ import NotificationsOffOutlinedIcon from "@mui/icons-material/NotificationsOffOu import SnoozeIcon from "@mui/icons-material/Snooze"; import { Fragment, useState } from "react"; import { Link as RouterLink } from "react-router-dom"; -import { useApi, useApiAction } from "../api"; +import { useApiAction } from "../api"; import { useIsAdmin } from "../hooks/useIsAdmin"; import { humanDuration } from "../lib/humanDuration"; import MessageView from "./MessageView"; @@ -50,8 +49,8 @@ function headline(issue: IssueData): string { } /** One issue. Expanded by default: shows a provenance line (source, ref, - * incident links), the message body, action buttons and a two-column - * Events / Notes panel. When collapsed, only the header row is visible — + * incident links), the message body, action buttons and the notes panel. + * When collapsed, only the header row is visible — * even the action buttons are hidden. Header layout: * `[toggle] [server] [headline] [resolver-avatar?] [snapshot] [severity] [time]`. * The headline is struck through when the issue is inactive or resolved. @@ -304,18 +303,7 @@ function Body({ onChanged={onChanged} /> )} - - + ); } - -const COLLAPSED_EVENT_COUNT = 3; -const EXPANDED_PAGE_SIZE = 10; - -function EventLog({ - issueId, - serverId, -}: { - issueId: string; - serverId?: string; -}) { - const [expanded, setExpanded] = useState(false); - const [page, setPage] = useState(0); - const limit = expanded ? EXPANDED_PAGE_SIZE : COLLAPSED_EVENT_COUNT; - const offset = expanded ? page * EXPANDED_PAGE_SIZE : 0; - const result = useApi( - "issues", - "list_events", - { issue_id: issueId, offset, limit }, - [issueId, offset, limit], - ); - const [snapshotOpen, setSnapshotOpen] = useState>( - () => new Set(), - ); - const toggleSnapshot = (eventId: string) => - setSnapshotOpen((prev) => { - const next = new Set(prev); - if (next.has(eventId)) next.delete(eventId); - else next.add(eventId); - return next; - }); - const closeSnapshot = (eventId: string) => - setSnapshotOpen((prev) => { - if (!prev.has(eventId)) return prev; - const next = new Set(prev); - next.delete(eventId); - return next; - }); - - if (result.status === "loading" || result.status === "idle") - return ( - - Loading… - - ); - if (result.status === "error") - return {result.error.message}; - const { items, total } = result.data; - if (total === 0) - return ( - - No events recorded. - - ); - - const pageCount = Math.max(1, Math.ceil(total / EXPANDED_PAGE_SIZE)); - const hiddenCount = Math.max(0, total - COLLAPSED_EVENT_COUNT); - - return ( - - {items.map((e) => { - const at = e.occurred_at ?? e.created_at; - const open = snapshotOpen.has(e.id); - return ( - - - - {e.occurrences > 1 && ( - - ×{e.occurrences} - - )} - - toggleSnapshot(e.id)} - tooltip="Status snapshot at this event" - /> - - - - - - - - {open && serverId && ( - closeSnapshot(e.id)} - /> - )} - - ); - })} - {!expanded && hiddenCount > 0 && ( - - )} - {expanded && ( - - - {pageCount > 1 && ( - setPage(p - 1)} - /> - )} - - )} - - ); -} diff --git a/private-web/src/routes/IncidentDetail.tsx b/private-web/src/routes/IncidentDetail.tsx index f1b9a79e..d9accd2b 100644 --- a/private-web/src/routes/IncidentDetail.tsx +++ b/private-web/src/routes/IncidentDetail.tsx @@ -18,7 +18,6 @@ import BugReportIcon from "@mui/icons-material/BugReport"; import CheckCircleOutlinedIcon from "@mui/icons-material/CheckCircleOutlined"; import NotesIcon from "@mui/icons-material/StickyNote2"; import RefreshIcon from "@mui/icons-material/Refresh"; -import TimelineIcon from "@mui/icons-material/Timeline"; import { useState } from "react"; import { Link as RouterLink, useParams } from "react-router-dom"; import { useApi, useApiAction } from "../api"; @@ -303,11 +302,6 @@ function Header({ value={incident.issue_count ?? 0} noun="issue" /> - } - value={incident.event_count ?? 0} - noun="event" - /> } value={incident.note_count ?? 0} diff --git a/private-web/src/types.ts b/private-web/src/types.ts index b3f01520..1bb59fb6 100644 --- a/private-web/src/types.ts +++ b/private-web/src/types.ts @@ -125,7 +125,6 @@ export type HealthcheckSampleResponse = Solidify; export type IssueIncidentLink = Solidify; -export type EventData = Solidify; export type IncidentData = Solidify; export type IncidentIssueData = Solidify; export type IncidentWithIssues = Solidify; From 5f8dc90515957baf5efa5117d3fc462b0c2a25bf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Wed, 8 Jul 2026 19:30:00 +1200 Subject: [PATCH 08/38] feat!(policy): check catalog becomes result-transform policies MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The healthcheck_severities catalog becomes check_policies: keyed per (source, check), speaking one vocabulary end to end. An entry carries a ceiling (the maximum effective result — failed passes failures through, warning grades them down, passed records without alerting, skipped also tells the source not to run the check), conditional rules that transform the observed result in any direction, and an escalates flag replacing the Critical severity. Ingestion grades every check through its policy; issues transitionally map effective results onto severities (failed → error, escalating failed → critical, warning → warning) until they carry results themselves. The migration converts severities losslessly (critical → failed + escalates, error → failed, warning → warning, info → passed, debug → skipped), including rules-ladder branches. Also fixes healthcheckNameFromRef in the UI, which still required the retired "status" source and so had stopped linking issues to their healthcheck pages after the alertd source migration. --- crates/commons-types/src/status.rs | 54 +++ crates/database/src/bin/seed.rs | 48 ++- ...hcheck_severities.rs => check_policies.rs} | 338 +++++++++++------- crates/database/src/lib.rs | 2 +- crates/database/src/schema.rs | 30 +- crates/database/tests/it/check_policies.rs | 229 ++++++++++++ ...everity_rules.rs => check_policy_rules.rs} | 206 ++++++----- .../database/tests/it/check_severity_map.rs | 69 ++-- .../tests/it/healthcheck_severities.rs | 128 ------- crates/database/tests/it/main.rs | 4 +- crates/private-server/src/fns/healthchecks.rs | 154 ++++---- crates/private-server/src/fns/statuses.rs | 52 ++- .../private-server/tests/it/healthchecks.rs | 81 +++-- .../tests/it/private_statuses.rs | 47 +-- crates/public-server/openapi.json | 2 +- crates/public-server/src/statuses.rs | 128 ++++--- .../tests/it/check_severities.rs | 34 +- crates/public-server/tests/it/statuses.rs | 78 ++-- .../down.sql | 56 +++ .../up.sql | 82 +++++ private-web/e2e/healthcheck-attention.spec.ts | 16 +- private-web/e2e/seed.ts | 44 ++- private-web/openapi.json | 217 ++++++----- private-web/src/api-types.ts | 246 +++++++------ .../src/components/CheckResultChip.tsx | 28 ++ private-web/src/lib/healthcheck-rule-eval.ts | 2 +- .../src/routes/HealthcheckAttention.tsx | 37 +- private-web/src/routes/HealthcheckDetail.tsx | 167 +++++---- private-web/src/routes/Healthchecks.tsx | 95 +++-- private-web/src/types.ts | 42 ++- 30 files changed, 1725 insertions(+), 991 deletions(-) rename crates/database/src/{healthcheck_severities.rs => check_policies.rs} (59%) create mode 100644 crates/database/tests/it/check_policies.rs rename crates/database/tests/it/{healthcheck_severity_rules.rs => check_policy_rules.rs} (66%) delete mode 100644 crates/database/tests/it/healthcheck_severities.rs create mode 100644 migrations/2026-07-08-072903-0000_check_policies/down.sql create mode 100644 migrations/2026-07-08-072903-0000_check_policies/up.sql create mode 100644 private-web/src/components/CheckResultChip.tsx diff --git a/crates/commons-types/src/status.rs b/crates/commons-types/src/status.rs index a2417fb3..4288c387 100644 --- a/crates/commons-types/src/status.rs +++ b/crates/commons-types/src/status.rs @@ -30,6 +30,20 @@ impl From for CheckSeverity { } } +impl From for CheckSeverity { + /// Distill a policy ceiling into the device-facing vocabulary: a + /// `failed` ceiling means failures count as failures, `warning` and + /// `broken` cap below that, and `passed`/`skipped` mean the check + /// never alerts (so the device may skip it). + fn from(ceiling: CheckResult) -> Self { + match ceiling { + CheckResult::Failed => Self::Fail, + CheckResult::Warning | CheckResult::Broken => Self::Warn, + CheckResult::Passed | CheckResult::Skipped => Self::Skip, + } + } +} + impl Display for CheckSeverity { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { match self { @@ -109,6 +123,32 @@ impl CheckResult { } } +impl CheckResult { + /// Position on the urgency ordering used by policy ceilings and + /// display sorting: failed > warning > broken > passed > skipped + /// (lower rank = more urgent). + pub fn urgency_rank(self) -> u8 { + match self { + CheckResult::Failed => 0, + CheckResult::Warning => 1, + CheckResult::Broken => 2, + CheckResult::Passed => 3, + CheckResult::Skipped => 4, + } + } + + /// Cap this result at `ceiling` on the urgency ordering: a result + /// more urgent than the ceiling grades down to it; anything at or + /// below the ceiling passes through unchanged. + pub fn capped_at(self, ceiling: CheckResult) -> CheckResult { + if self.urgency_rank() < ceiling.urgency_rank() { + ceiling + } else { + self + } + } +} + impl Display for CheckResult { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { match self { @@ -121,6 +161,20 @@ impl Display for CheckResult { } } +impl TryFrom for CheckResult { + type Error = String; + + fn try_from(value: String) -> Result { + value.parse() + } +} + +impl From for String { + fn from(result: CheckResult) -> Self { + result.to_string() + } +} + impl std::str::FromStr for CheckResult { type Err = String; diff --git a/crates/database/src/bin/seed.rs b/crates/database/src/bin/seed.rs index 462c376b..52dcc30c 100644 --- a/crates/database/src/bin/seed.rs +++ b/crates/database/src/bin/seed.rs @@ -18,13 +18,14 @@ use commons_types::{ geo::GeoPoint, issue::{ResolvedReason, Severity}, server::{TagMap, kind::ServerKind, rank::ServerRank}, + status::CheckResult, version::{VersionStatus, VersionStr}, }; use database::{ Device, DeviceKey, admins::Admin, + check_policies::CheckPolicy, devices::NewDeviceConnection, - healthcheck_severities::HealthcheckSeverity, issues::{Incident, Issue, NewEvent}, notes::{IncidentNote, IssueNote}, pg_duration::PgDuration, @@ -59,7 +60,6 @@ const TRUNCATE_TABLES: &[&str] = &[ "issue_notes", "incident_issues", "incidents", - "events", "issues", "server_silenced_refs", "server_group_silenced_refs", @@ -70,7 +70,7 @@ const TRUNCATE_TABLES: &[&str] = &[ "server_groups", "version_known_issues", "versions", - "healthcheck_severities", + "check_policies", "admins", ]; @@ -323,49 +323,62 @@ async fn seed_healthchecks(conn: &mut AsyncPgConnection, admins: &[String]) -> R "certificate_expiry", "backup_freshness", ] { - HealthcheckSeverity::upsert_default(conn, check).await?; + CheckPolicy::upsert_default(conn, "alertd", check).await?; } let reviewer = &admins[0]; - HealthcheckSeverity::update( + CheckPolicy::update( conn, + "alertd", "database_connectivity", - Severity::Critical, + CheckResult::Failed, + true, Some("DB down means the server is effectively offline."), reviewer, ) .await?; - HealthcheckSeverity::update( + CheckPolicy::update( conn, + "alertd", "disk_space", - Severity::Error, + CheckResult::Failed, + false, Some("Page when disk is critically low."), reviewer, ) .await?; - HealthcheckSeverity::update(conn, "backup_freshness", Severity::Info, None, reviewer).await?; + CheckPolicy::update( + conn, + "alertd", + "backup_freshness", + CheckResult::Passed, + false, + None, + reviewer, + ) + .await?; - // A conditional rules ladder: escalate sync_lag based on how far behind it is. - use database::healthcheck_severities::{Condition, IfLadder, Var}; + // A conditional rules ladder: grade sync_lag by how far behind it is. + use database::check_policies::{Condition, IfLadder, Var}; let ladder = IfLadder { branches: vec![ ( Condition::Gt( "check.lag_seconds".parse::().expect("var parses"), - serde_json::json!(3600), + serde_json::json!(600), ), - Severity::Critical, + CheckResult::Failed, ), ( Condition::Gt( "check.lag_seconds".parse::().expect("var parses"), - serde_json::json!(600), + serde_json::json!(60), ), - Severity::Error, + CheckResult::Warning, ), ], }; - HealthcheckSeverity::update_rules(conn, "sync_lag", Some(&ladder), reviewer).await?; + CheckPolicy::update_rules(conn, "alertd", "sync_lag", Some(&ladder), reviewer).await?; Ok(()) } @@ -1166,7 +1179,7 @@ async fn report(conn: &mut AsyncPgConnection) -> Result<()> { "admins", "versions", "version_known_issues", - "healthcheck_severities", + "check_policies", "devices", "device_keys", "device_connections", @@ -1175,7 +1188,6 @@ async fn report(conn: &mut AsyncPgConnection) -> Result<()> { "server_enrollment_tokens", "statuses", "issues", - "events", "incidents", "incident_issues", "issue_notes", diff --git a/crates/database/src/healthcheck_severities.rs b/crates/database/src/check_policies.rs similarity index 59% rename from crates/database/src/healthcheck_severities.rs rename to crates/database/src/check_policies.rs index 00ef41cd..dbb1df40 100644 --- a/crates/database/src/healthcheck_severities.rs +++ b/crates/database/src/check_policies.rs @@ -1,14 +1,23 @@ -//! Operator-owned catalog of healthcheck names → the severity to file -//! their failures at. +//! Operator-owned catalog of check policies: per (source, check), how an +//! observed result transforms into the effective result Canopy acts on. +//! +//! An entry carries a **ceiling** (the maximum effective result on the +//! urgency ordering — `failed` changes nothing, `warning` grades failures +//! as warnings, `passed` means recorded but never alerting, `skipped` +//! additionally tells the source not to bother running the check), +//! optional conditional **rules** (transforms in any direction, evaluated +//! against the check's detail, the report's server-wide detail, and the +//! server's tags), and an **escalates** flag (an effective failure of +//! this check notifies immediately, bypassing incident grace). //! //! Ingestion (in the public-server status handler) calls -//! [`HealthcheckSeverity::upsert_default`] for every check name seen on -//! a push, then [`HealthcheckSeverity::severity_for`] when filing a -//! failing per-check issue. Operators read and edit the catalog via the -//! private-server `/api/healthchecks` endpoints. +//! [`CheckPolicy::upsert_default`] for every check seen on a push, then +//! [`CheckPolicy::apply`] to grade each observed result. Operators read +//! and edit the catalog via the private-server `/api/healthchecks` +//! endpoints. use commons_errors::{AppError, Result}; -use commons_types::{issue::Severity, status::CheckResult}; +use commons_types::status::CheckResult; use diesel::prelude::*; use diesel_async::{AsyncPgConnection, RunQueryDsl}; use jiff::Timestamp; @@ -16,21 +25,38 @@ use serde::{Deserialize, Deserializer, Serialize, Serializer, de}; use serde_json::Value as JsonValue; use std::collections::{BTreeMap, HashMap}; -/// The severity policy for one named healthcheck. An entry is created -/// automatically the first time a check with this name is reported, using -/// a default policy; operators then review and adjust how that check's -/// failures should be classified going forward. +/// The policy for one (source, check). An entry is created automatically +/// the first time a source reports a check with this name, at the default +/// ceiling; operators then review and adjust how that check's results are +/// graded going forward. #[derive(Clone, Debug, Serialize, Deserialize, Queryable, Selectable, utoipa::ToSchema)] -#[diesel(table_name = crate::schema::healthcheck_severities)] +#[diesel(table_name = crate::schema::check_policies)] #[diesel(check_for_backend(diesel::pg::Pg))] -pub struct HealthcheckSeverity { - /// The healthcheck's name, as reported in status pushes. Uniquely - /// identifies this policy. +pub struct CheckPolicy { + /// The source that reports this check. Together with `check_name`, + /// uniquely identifies this policy. + pub source: String, + /// The check's name, as reported in status pushes. pub check_name: String, - /// The severity assigned to a failure of this check when no conditional - /// rule (see `rules`) overrides it. + /// The maximum effective result for this check when no conditional + /// rule (see `rules`) overrides it: an observed result more urgent + /// than the ceiling grades down to it. #[diesel(deserialize_as = String, serialize_as = String)] - pub severity: Severity, + #[schema(value_type = String)] + pub ceiling: CheckResult, + /// Whether an effective failure of this check notifies immediately, + /// bypassing the incident grace period. + pub escalates: bool, + /// Optional conditional rules that can grade a result differently — + /// in any direction — depending on the details of the check, the + /// surrounding status report, or the server's tags. `None` means no + /// conditional rules are configured and the ceiling always applies. + /// When present, the rules are evaluated in order and the first + /// matching one wins; if none match, the ceiling applies. + #[schema(value_type = Option)] + pub rules: Option, + /// Free-form operator notes about this check. + pub notes: Option, /// When this check was first observed and this policy entry was created. #[diesel(deserialize_as = jiff_diesel::Timestamp, serialize_as = jiff_diesel::Timestamp)] pub first_seen: Timestamp, @@ -41,32 +67,36 @@ pub struct HealthcheckSeverity { /// The operator who last reviewed this policy. `None` if it has never /// been reviewed. pub reviewed_by: Option, - /// Free-form operator notes about this check. - pub notes: Option, /// When this policy was last modified. #[diesel(deserialize_as = jiff_diesel::Timestamp, serialize_as = jiff_diesel::Timestamp)] pub updated_at: Timestamp, - /// Optional conditional rules that can assign a different severity - /// depending on the details of the failing check, the surrounding status - /// report, or the server's tags. `None` means no conditional rules are - /// configured and `severity` always applies. When present, the rules are - /// evaluated in order and the first matching one wins; if none match, - /// `severity` is used as the fallback. - #[schema(value_type = Option)] - pub rules: Option, } -impl HealthcheckSeverity { - /// Insert a row for `check_name` with default values (severity = - /// warning, reviewed_at = NULL) if and only if no row exists yet. - /// Idempotent: safe to call on every status push for every check - /// seen, including healthy ones. Concurrent pushes are serialised - /// by Postgres via `ON CONFLICT DO NOTHING`. - pub async fn upsert_default(db: &mut AsyncPgConnection, check_name: &str) -> Result<()> { - use crate::schema::healthcheck_severities::dsl; - diesel::insert_into(dsl::healthcheck_severities) - .values(dsl::check_name.eq(check_name)) - .on_conflict(dsl::check_name) +/// The outcome of applying a check's policy to an observed result. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct GradedResult { + /// The effective result after the policy transform. + pub effective: CheckResult, + /// Whether this check's effective failures escalate (notify + /// immediately, bypassing incident grace). + pub escalates: bool, +} + +impl CheckPolicy { + /// Insert a row for `(source, check_name)` with default values + /// (ceiling = warning) if and only if no row exists yet. Idempotent: + /// safe to call on every status push for every check seen, including + /// healthy ones. Concurrent pushes are serialised by Postgres via + /// `ON CONFLICT DO NOTHING`. + pub async fn upsert_default( + db: &mut AsyncPgConnection, + source: &str, + check_name: &str, + ) -> Result<()> { + use crate::schema::check_policies::dsl; + diesel::insert_into(dsl::check_policies) + .values((dsl::source.eq(source), dsl::check_name.eq(check_name))) + .on_conflict((dsl::source, dsl::check_name)) .do_nothing() .execute(db) .await @@ -74,57 +104,60 @@ impl HealthcheckSeverity { Ok(()) } - /// Look up the effective severity for a `check_name` reporting - /// `result` (warning or failed — the only kinds that file at the - /// catalog severity), given the supplied evaluation context. If - /// the row has a `rules` ladder it's evaluated against the context - /// first; the first matching branch's severity wins. Otherwise (no - /// ladder, no matching branch, or malformed JSON) the fallback - /// depends on the result kind: warning-result checks land at fixed - /// [`Severity::Warning`]; failed checks use the row's base - /// `severity` column. The catalog column is thus "the severity of - /// this check's failures" — warnings only deviate from Warning via - /// an explicit rule (which can condition on `check.result`). + /// Apply the `(source, check_name)` policy to an `observed` result: + /// if the entry has a `rules` ladder and a branch matches the + /// supplied evaluation context, that branch's result wins (any + /// direction — rules can upgrade as well as downgrade); otherwise the + /// observed result is capped at the entry's ceiling. /// - /// Falls back to `Severity::Warning` if no row exists yet — in - /// practice the status handler upserts before reading, so this - /// branch only covers the genuine race / programmer-error case. - pub async fn severity_for( + /// Falls back to the default policy (ceiling = warning, no + /// escalation) if no row exists yet — in practice the status handler + /// upserts before reading, so this branch only covers the genuine + /// race / programmer-error case. + pub async fn apply( db: &mut AsyncPgConnection, + source: &str, check_name: &str, - result: CheckResult, + observed: CheckResult, ctx: &EvaluationContext<'_>, - ) -> Result { - use crate::schema::healthcheck_severities::dsl; - let row: Option<(String, Option)> = dsl::healthcheck_severities - .select((dsl::severity, dsl::rules)) - .filter(dsl::check_name.eq(check_name)) + ) -> Result { + use crate::schema::check_policies::dsl; + let row: Option<(String, bool, Option)> = dsl::check_policies + .select((dsl::ceiling, dsl::escalates, dsl::rules)) + .filter(dsl::source.eq(source).and(dsl::check_name.eq(check_name))) .first(db) .await .optional()?; - let Some((sev_str, rules_json)) = row else { - return Ok(Severity::Warning); + let Some((ceiling_str, escalates, rules_json)) = row else { + return Ok(GradedResult { + effective: observed.capped_at(CheckResult::Warning), + escalates: false, + }); }; - let base = sev_str.parse().unwrap_or(Severity::Warning); + let ceiling = ceiling_str.parse().unwrap_or(CheckResult::Warning); if let Some(rules) = rules_json { match serde_json::from_value::(rules) { Ok(ladder) => { - if let Some(s) = ladder.evaluate(ctx) { - return Ok(s); + if let Some(result) = ladder.evaluate(ctx) { + return Ok(GradedResult { + effective: result, + escalates, + }); } } Err(err) => { tracing::warn!( + source, check_name, ?err, - "failed to parse healthcheck severity rules; falling back to base" + "failed to parse check policy rules; falling back to ceiling" ); } } } - Ok(match result { - CheckResult::Warning => Severity::Warning, - _ => base, + Ok(GradedResult { + effective: observed.capped_at(ceiling), + escalates, }) } @@ -133,113 +166,142 @@ impl HealthcheckSeverity { /// rules also counts as a review for the catalog row. pub async fn update_rules( db: &mut AsyncPgConnection, + source: &str, check_name: &str, rules: Option<&IfLadder>, by: &str, ) -> Result { - use crate::schema::healthcheck_severities::dsl; + use crate::schema::check_policies::dsl; let now = Timestamp::now(); let rules_json: Option = rules.map(|l| serde_json::to_value(l).expect("IfLadder always serialises")); - diesel::update(dsl::healthcheck_severities.filter(dsl::check_name.eq(check_name))) - .set(( - dsl::rules.eq(rules_json), - dsl::reviewed_at.eq(jiff_diesel::Timestamp::from(now)), - dsl::reviewed_by.eq(by), - )) - .returning(Self::as_select()) - .get_result(db) - .await - .map_err(AppError::from) + diesel::update( + dsl::check_policies.filter(dsl::source.eq(source).and(dsl::check_name.eq(check_name))), + ) + .set(( + dsl::rules.eq(rules_json), + dsl::reviewed_at.eq(jiff_diesel::Timestamp::from(now)), + dsl::reviewed_by.eq(by), + )) + .returning(Self::as_select()) + .get_result(db) + .await + .map_err(AppError::from) } - /// The whole catalog as `check_name → base severity`, for building the - /// device-facing effective check-severity map. Deliberately reads only - /// the static `severity` column: conditional `rules` ladders are - /// expressions evaluated per push against the report's contents, so they - /// can't be resolved ahead of time and are ignored here. An unparseable - /// severity falls back to [`Severity::Warning`], same as `severity_for`. - pub async fn base_severity_map( + /// One source's catalog as `check_name → ceiling`, for building the + /// device-facing effective check map. Deliberately reads only the + /// static `ceiling` column: conditional `rules` ladders are + /// expressions evaluated per push against the report's contents, so + /// they can't be resolved ahead of time and are ignored here. An + /// unparseable ceiling falls back to warning, same as [`Self::apply`]. + pub async fn ceiling_map_for_source( db: &mut AsyncPgConnection, - ) -> Result> { - use crate::schema::healthcheck_severities::dsl; - let rows: Vec<(String, String)> = dsl::healthcheck_severities - .select((dsl::check_name, dsl::severity)) + source: &str, + ) -> Result> { + use crate::schema::check_policies::dsl; + let rows: Vec<(String, String)> = dsl::check_policies + .select((dsl::check_name, dsl::ceiling)) + .filter(dsl::source.eq(source)) .load(db) .await .map_err(AppError::from)?; Ok(rows .into_iter() - .map(|(name, sev)| (name, sev.parse().unwrap_or(Severity::Warning))) + .map(|(name, ceiling)| (name, ceiling.parse().unwrap_or(CheckResult::Warning))) .collect()) } pub async fn list(db: &mut AsyncPgConnection) -> Result> { - use crate::schema::healthcheck_severities::dsl; - dsl::healthcheck_severities + use crate::schema::check_policies::dsl; + dsl::check_policies .select(Self::as_select()) - .order(dsl::check_name.asc()) + .order((dsl::source.asc(), dsl::check_name.asc())) .load(db) .await .map_err(AppError::from) } - /// The catalog row for a single check name, or `None` if no server has - /// ever reported it (so ingestion has never upserted a row). - pub async fn get(db: &mut AsyncPgConnection, check_name: &str) -> Result> { - use crate::schema::healthcheck_severities::dsl; - dsl::healthcheck_severities + /// The catalog row for a single (source, check), or `None` if that + /// source has never reported it (so ingestion has never upserted a + /// row). + pub async fn get( + db: &mut AsyncPgConnection, + source: &str, + check_name: &str, + ) -> Result> { + use crate::schema::check_policies::dsl; + dsl::check_policies .select(Self::as_select()) - .filter(dsl::check_name.eq(check_name)) + .filter(dsl::source.eq(source).and(dsl::check_name.eq(check_name))) .first(db) .await .optional() .map_err(AppError::from) } - /// Update the severity (and optionally notes) for a check, stamping - /// `reviewed_at = NOW()` and `reviewed_by = by`. Even a no-op save - /// (same severity) marks the row reviewed — operators can ack - /// a check without changing it. + /// The catalog rows for a check name across every source that + /// reports it. Pages that correlate by check name alone (the + /// per-check attention view) use this. + pub async fn get_by_name(db: &mut AsyncPgConnection, check_name: &str) -> Result> { + use crate::schema::check_policies::dsl; + dsl::check_policies + .select(Self::as_select()) + .filter(dsl::check_name.eq(check_name)) + .order(dsl::source.asc()) + .load(db) + .await + .map_err(AppError::from) + } + + /// Update the ceiling, escalation flag, and optionally notes for a + /// check, stamping `reviewed_at = NOW()` and `reviewed_by = by`. Even + /// a no-op save marks the row reviewed — operators can ack a check + /// without changing it. pub async fn update( db: &mut AsyncPgConnection, + source: &str, check_name: &str, - severity: Severity, + ceiling: CheckResult, + escalates: bool, notes: Option<&str>, by: &str, ) -> Result { - use crate::schema::healthcheck_severities::dsl; + use crate::schema::check_policies::dsl; let now = Timestamp::now(); - diesel::update(dsl::healthcheck_severities.filter(dsl::check_name.eq(check_name))) - .set(( - dsl::severity.eq(severity), - dsl::notes.eq(notes), - dsl::reviewed_at.eq(jiff_diesel::Timestamp::from(now)), - dsl::reviewed_by.eq(by), - )) - .returning(Self::as_select()) - .get_result(db) - .await - .map_err(AppError::from) + diesel::update( + dsl::check_policies.filter(dsl::source.eq(source).and(dsl::check_name.eq(check_name))), + ) + .set(( + dsl::ceiling.eq(ceiling.to_string()), + dsl::escalates.eq(escalates), + dsl::notes.eq(notes), + dsl::reviewed_at.eq(jiff_diesel::Timestamp::from(now)), + dsl::reviewed_by.eq(by), + )) + .returning(Self::as_select()) + .get_result(db) + .await + .map_err(AppError::from) } } // ── Rule model: JsonLogic-encoded if-ladder ──────────────────────────────── -/// A JsonLogic `if`-ladder evaluating to a Severity string. +/// A JsonLogic `if`-ladder evaluating to a [`CheckResult`] string. /// -/// Wire shape: `{"if": [c1, s1, c2, s2, …, cN, sN]}` — even-length +/// Wire shape: `{"if": [c1, r1, c2, r2, …, cN, rN]}` — even-length /// argument list, every odd-index entry is a [`Condition`], every -/// even-index entry is a Severity literal. No trailing else: when no +/// even-index entry is a result literal. No trailing else: when no /// branch matches, [`Self::evaluate`] returns `None` and the calling -/// `severity_for` falls through to the row's `severity` column. +/// [`CheckPolicy::apply`] falls through to the entry's ceiling. /// /// Empty ladders are forbidden by the deserialiser; the API layer /// normalises them to `None` (clearing the `rules` column) before /// hitting the database. #[derive(Clone, Debug, PartialEq)] pub struct IfLadder { - pub branches: Vec<(Condition, Severity)>, + pub branches: Vec<(Condition, CheckResult)>, } /// One predicate inside an [`IfLadder`] branch. Maps 1:1 with a single @@ -277,8 +339,8 @@ pub struct EvaluationContext<'a> { /// Top-level status extras (`statuses.extra`). Bestool sends /// `bestoolVersion`, `tamanuVersion`, `uptimeSecs`, etc. pub status_extra: &'a serde_json::Map, - /// The failing check's own fields (the `health[i]` object minus - /// `check` and `healthy`). + /// The check's own fields (the `health[i]` object minus `check` and + /// `healthy`). pub check_extra: &'a serde_json::Map, /// Server's resolved tag map (merged server + group). Each value is /// already wrapped as `JsonValue::String` for uniform comparison. @@ -286,13 +348,13 @@ pub struct EvaluationContext<'a> { } impl IfLadder { - /// Returns the first matching branch's severity, or `None` if no - /// branch matches. The caller falls back to the catalog's base - /// severity in that case. - pub fn evaluate(&self, ctx: &EvaluationContext) -> Option { + /// Returns the first matching branch's result, or `None` if no + /// branch matches. The caller falls back to the entry's ceiling in + /// that case. + pub fn evaluate(&self, ctx: &EvaluationContext) -> Option { self.branches .iter() - .find_map(|(c, s)| c.matches(ctx).then_some(*s)) + .find_map(|(c, r)| c.matches(ctx).then_some(*r)) } } @@ -499,9 +561,9 @@ impl<'de> Deserialize<'de> for Condition { impl Serialize for IfLadder { fn serialize(&self, ser: S) -> std::result::Result { let mut args: Vec = Vec::with_capacity(self.branches.len() * 2); - for (c, s) in &self.branches { + for (c, r) in &self.branches { args.push(serde_json::to_value(c).map_err(serde::ser::Error::custom)?); - args.push(JsonValue::String(s.to_string())); + args.push(JsonValue::String(r.to_string())); } serde_json::json!({ "if": args }).serialize(ser) } @@ -534,7 +596,7 @@ impl<'de> Deserialize<'de> for IfLadder { } if args.len() % 2 != 0 { return Err(de::Error::custom( - "'if' args must have even length (alternating condition, severity); \ + "'if' args must have even length (alternating condition, result); \ a trailing else is not allowed", )); } @@ -542,13 +604,13 @@ impl<'de> Deserialize<'de> for IfLadder { for chunk in args.chunks_exact(2) { let cond: Condition = serde_json::from_value(chunk[0].clone()).map_err(de::Error::custom)?; - let sev_str = chunk[1].as_str().ok_or_else(|| { - de::Error::custom("each odd-indexed 'if' arg must be a severity string") + let result_str = chunk[1].as_str().ok_or_else(|| { + de::Error::custom("each odd-indexed 'if' arg must be a result string") })?; - let sev: Severity = sev_str + let result: CheckResult = result_str .parse() - .map_err(|_| de::Error::custom(format!("invalid severity '{sev_str}'")))?; - branches.push((cond, sev)); + .map_err(|_| de::Error::custom(format!("invalid result '{result_str}'")))?; + branches.push((cond, result)); } Ok(IfLadder { branches }) } diff --git a/crates/database/src/lib.rs b/crates/database/src/lib.rs index a31e4f36..0c18b840 100644 --- a/crates/database/src/lib.rs +++ b/crates/database/src/lib.rs @@ -10,9 +10,9 @@ pub mod artifacts; pub mod backup; pub mod backups; pub mod bestool_snippets; +pub mod check_policies; pub mod chrome_releases; pub mod devices; -pub mod healthcheck_severities; pub mod issues; pub mod mcp_tokens; pub mod notes; diff --git a/crates/database/src/schema.rs b/crates/database/src/schema.rs index 5f334c9e..1c102bc4 100644 --- a/crates/database/src/schema.rs +++ b/crates/database/src/schema.rs @@ -169,6 +169,21 @@ diesel::table! { } } +diesel::table! { + check_policies (source, check_name) { + source -> Text, + check_name -> Text, + ceiling -> Text, + escalates -> Bool, + rules -> Nullable, + notes -> Nullable, + first_seen -> Timestamptz, + reviewed_at -> Nullable, + reviewed_by -> Nullable, + updated_at -> Timestamptz, + } +} + diesel::table! { chrome_releases (version) { version -> Text, @@ -223,19 +238,6 @@ diesel::table! { } } -diesel::table! { - healthcheck_severities (check_name) { - check_name -> Text, - severity -> Text, - first_seen -> Timestamptz, - reviewed_at -> Nullable, - reviewed_by -> Nullable, - notes -> Nullable, - updated_at -> Timestamptz, - rules -> Nullable, - } -} - diesel::table! { incident_issues (incident_id, issue_id, joined_at) { incident_id -> Uuid, @@ -627,12 +629,12 @@ diesel::allow_tables_to_appear_in_same_query!( backup_runs, backup_type_defaults, bestool_snippets, + check_policies, chrome_releases, device_connections, device_keys, device_server_associations, devices, - healthcheck_severities, incident_issues, incident_notes, incidents, diff --git a/crates/database/tests/it/check_policies.rs b/crates/database/tests/it/check_policies.rs new file mode 100644 index 00000000..f2c68a4e --- /dev/null +++ b/crates/database/tests/it/check_policies.rs @@ -0,0 +1,229 @@ +//! `CheckPolicy` model: catalog of per-(source, check) result policies, +//! maintained by the public-server status ingestion path and edited +//! through the private-server `/api/healthchecks` endpoints. + +use commons_types::status::CheckResult; +use database::check_policies::CheckPolicy; + +#[tokio::test(flavor = "multi_thread")] +async fn upsert_default_inserts_then_is_idempotent() { + commons_tests::db::TestDb::run(async |mut conn, _| { + CheckPolicy::upsert_default(&mut conn, "alertd", "disk_space") + .await + .expect("first upsert"); + CheckPolicy::upsert_default(&mut conn, "alertd", "disk_space") + .await + .expect("second upsert is no-op"); + + // Both upserts should land at the schema default, pending review. + let rows = CheckPolicy::list(&mut conn).await.expect("list"); + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].source, "alertd"); + assert_eq!(rows[0].check_name, "disk_space"); + assert_eq!(rows[0].ceiling, CheckResult::Warning); + assert!(!rows[0].escalates); + assert!(rows[0].reviewed_at.is_none(), "pending review by default"); + assert!(rows[0].reviewed_by.is_none()); + }) + .await +} + +#[tokio::test(flavor = "multi_thread")] +async fn same_check_name_is_distinct_per_source() { + commons_tests::db::TestDb::run(async |mut conn, _| { + CheckPolicy::upsert_default(&mut conn, "alertd", "disk_space") + .await + .expect("alertd upsert"); + CheckPolicy::upsert_default(&mut conn, "seedling", "disk_space") + .await + .expect("seedling upsert"); + + CheckPolicy::update( + &mut conn, + "alertd", + "disk_space", + CheckResult::Failed, + false, + None, + "alice", + ) + .await + .expect("update alertd entry"); + + let seedling = CheckPolicy::get(&mut conn, "seedling", "disk_space") + .await + .expect("get") + .expect("seedling row exists"); + assert_eq!( + seedling.ceiling, + CheckResult::Warning, + "the other source's entry is untouched", + ); + }) + .await +} + +#[tokio::test(flavor = "multi_thread")] +async fn upsert_default_does_not_overwrite_operator_policy() { + commons_tests::db::TestDb::run(async |mut conn, _| { + CheckPolicy::upsert_default(&mut conn, "alertd", "disk_space") + .await + .expect("seed"); + CheckPolicy::update( + &mut conn, + "alertd", + "disk_space", + CheckResult::Failed, + true, + None, + "alice", + ) + .await + .expect("operator update"); + + // A subsequent push from ingestion must not revert the row. + CheckPolicy::upsert_default(&mut conn, "alertd", "disk_space") + .await + .expect("upsert again"); + + let rows = CheckPolicy::list(&mut conn).await.expect("list"); + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].ceiling, CheckResult::Failed); + assert!(rows[0].escalates); + assert!(rows[0].reviewed_at.is_some()); + assert_eq!(rows[0].reviewed_by.as_deref(), Some("alice")); + }) + .await +} + +#[tokio::test(flavor = "multi_thread")] +async fn apply_caps_at_ceiling_or_defaults_to_warning() { + commons_tests::db::TestDb::run(async |mut conn, _| { + let empty_map = serde_json::Map::new(); + let empty_tags = std::collections::HashMap::new(); + let ctx = database::check_policies::EvaluationContext { + status_extra: &empty_map, + check_extra: &empty_map, + tags: &empty_tags, + }; + // Unknown check → default policy (programmer-error path; + // ingestion upserts before reading in production). + let unknown = CheckPolicy::apply(&mut conn, "alertd", "ghost", CheckResult::Failed, &ctx) + .await + .expect("apply"); + assert_eq!(unknown.effective, CheckResult::Warning); + assert!(!unknown.escalates); + + CheckPolicy::upsert_default(&mut conn, "alertd", "cert_expiry") + .await + .expect("seed"); + CheckPolicy::update( + &mut conn, + "alertd", + "cert_expiry", + CheckResult::Failed, + true, + None, + "bob", + ) + .await + .expect("update"); + + // A failed ceiling passes failures through, carrying the flag. + let failed = CheckPolicy::apply( + &mut conn, + "alertd", + "cert_expiry", + CheckResult::Failed, + &ctx, + ) + .await + .expect("apply"); + assert_eq!(failed.effective, CheckResult::Failed); + assert!(failed.escalates); + + // A warning observation is already below the ceiling: unchanged. + let warned = CheckPolicy::apply( + &mut conn, + "alertd", + "cert_expiry", + CheckResult::Warning, + &ctx, + ) + .await + .expect("apply"); + assert_eq!(warned.effective, CheckResult::Warning); + + // A passed ceiling means the check never alerts. + CheckPolicy::update( + &mut conn, + "alertd", + "cert_expiry", + CheckResult::Passed, + false, + None, + "bob", + ) + .await + .expect("downgrade"); + let ignored = CheckPolicy::apply( + &mut conn, + "alertd", + "cert_expiry", + CheckResult::Failed, + &ctx, + ) + .await + .expect("apply"); + assert_eq!(ignored.effective, CheckResult::Passed); + }) + .await +} + +#[tokio::test(flavor = "multi_thread")] +async fn update_stamps_review_metadata_even_on_no_op_save() { + commons_tests::db::TestDb::run(async |mut conn, _| { + CheckPolicy::upsert_default(&mut conn, "alertd", "noisy_check") + .await + .expect("seed"); + + // "Mark reviewed without changing the policy": pass the current + // (default) values. reviewed_at + reviewed_by must still be set. + let updated = CheckPolicy::update( + &mut conn, + "alertd", + "noisy_check", + CheckResult::Warning, + false, + None, + "carol", + ) + .await + .expect("update"); + assert_eq!(updated.ceiling, CheckResult::Warning); + assert!(updated.reviewed_at.is_some()); + assert_eq!(updated.reviewed_by.as_deref(), Some("carol")); + }) + .await +} + +#[tokio::test(flavor = "multi_thread")] +async fn list_orders_by_source_then_check_name() { + commons_tests::db::TestDb::run(async |mut conn, _| { + for (source, name) in [("alertd", "zeta"), ("alertd", "alpha"), ("seedling", "mu")] { + CheckPolicy::upsert_default(&mut conn, source, name) + .await + .expect("seed"); + } + let rows = CheckPolicy::list(&mut conn).await.expect("list"); + let keys: Vec<(&str, &str)> = rows + .iter() + .map(|r| (r.source.as_str(), r.check_name.as_str())) + .collect(); + assert_eq!( + keys, + vec![("alertd", "alpha"), ("alertd", "zeta"), ("seedling", "mu"),] + ); + }) + .await +} diff --git a/crates/database/tests/it/healthcheck_severity_rules.rs b/crates/database/tests/it/check_policy_rules.rs similarity index 66% rename from crates/database/tests/it/healthcheck_severity_rules.rs rename to crates/database/tests/it/check_policy_rules.rs index 6c6ac215..89d5c44b 100644 --- a/crates/database/tests/it/healthcheck_severity_rules.rs +++ b/crates/database/tests/it/check_policy_rules.rs @@ -1,10 +1,10 @@ -//! v2 conditional-rule model: JsonLogic if-ladder serde + evaluator + -//! `severity_for` integration. The model is the single source of truth -//! for "given a status push, what severity does this check fail at?" -//! See `docs/plans/healthcheck-severity-rules-v2.md`. +//! Conditional-rule model: JsonLogic if-ladder serde + evaluator + +//! `CheckPolicy::apply` integration. The model is the single source of +//! truth for "given a status push, what effective result does this check +//! grade to?" -use commons_types::{issue::Severity, status::CheckResult}; -use database::healthcheck_severities::{EvaluationContext, HealthcheckSeverity, IfLadder}; +use commons_types::status::CheckResult; +use database::check_policies::{CheckPolicy, EvaluationContext, IfLadder}; use serde_json::json; use std::collections::HashMap; @@ -25,7 +25,7 @@ fn empty_ctx<'a>( #[test] fn deserialise_accepts_all_supported_ops() { for op in ["==", "!=", "<", "<=", ">", ">="] { - let json = json!({"if": [{op: [{"var": "check.x"}, 10]}, "error"]}); + let json = json!({"if": [{op: [{"var": "check.x"}, 10]}, "failed"]}); serde_json::from_value::(json).unwrap_or_else(|e| panic!("op {op} failed: {e}")); } let in_range = json!({"if": [ @@ -38,12 +38,12 @@ fn deserialise_accepts_all_supported_ops() { #[test] fn deserialise_rejects_composition_and_unknown_ops() { let cases: &[(serde_json::Value, &str)] = &[ - (json!({"if": [{"and": [true, true]}, "error"]}), "and"), - (json!({"if": [{"or": [true, true]}, "error"]}), "or"), - (json!({"if": [{"!": [true]}, "error"]}), "not"), - (json!({"if": [{"if": [true, true]}, "error"]}), "nested if"), + (json!({"if": [{"and": [true, true]}, "failed"]}), "and"), + (json!({"if": [{"or": [true, true]}, "failed"]}), "or"), + (json!({"if": [{"!": [true]}, "failed"]}), "not"), + (json!({"if": [{"if": [true, true]}, "failed"]}), "nested if"), ( - json!({"if": [{"foo": [{"var": "check.x"}, 1]}, "error"]}), + json!({"if": [{"foo": [{"var": "check.x"}, 1]}, "failed"]}), "unknown", ), ]; @@ -68,15 +68,15 @@ fn deserialise_rejects_malformed_shapes() { ), (json!({"not_if": []}), "wrong top-level op"), ( - json!({"if": [{"==": [{"var": "check.x"}, 1]}, "not_a_severity"]}), - "bad severity", + json!({"if": [{"==": [{"var": "check.x"}, 1]}, "not_a_result"]}), + "bad result", ), ( - json!({"if": [{"==": [{"var": "BAD.field"}, 1]}, "error"]}), + json!({"if": [{"==": [{"var": "BAD.field"}, 1]}, "failed"]}), "unknown var namespace", ), ( - json!({"if": [{"==": [{"var": "check..bad"}, 1]}, "error"]}), + json!({"if": [{"==": [{"var": "check..bad"}, 1]}, "failed"]}), "empty field segment", ), ]; @@ -91,8 +91,8 @@ fn deserialise_rejects_malformed_shapes() { fn round_trip_preserves_shape() { let original = json!({"if": [ {"in_range": [{"var": "status.bestoolVersion"}, ">=2.4.0 <2.5.4"]}, "warning", - {">": [{"var": "check.used_pct"}, 95]}, "critical", - {"==": [{"var": "tag.environment"}, "prod"]}, "error" + {">": [{"var": "check.used_pct"}, 95]}, "failed", + {"==": [{"var": "tag.environment"}, "prod"]}, "passed" ]}); let ladder: IfLadder = serde_json::from_value(original.clone()).expect("parse"); let back = serde_json::to_value(&ladder).expect("serialise"); @@ -115,7 +115,7 @@ fn in_range_matches_inside_misses_outside_and_falses_on_non_semver() { status.insert("bestoolVersion".into(), json!("2.4.7")); assert_eq!( ladder.evaluate(&empty_ctx(&check_extra, &status, &tags)), - Some(Severity::Warning), + Some(CheckResult::Warning), ); status.insert("bestoolVersion".into(), json!("2.5.4")); @@ -142,7 +142,7 @@ fn in_range_matches_inside_misses_outside_and_falses_on_non_semver() { #[test] fn numeric_ops_coerce_string_of_digits() { let ladder: IfLadder = serde_json::from_value(json!({"if": [ - {">": [{"var": "check.value"}, 95]}, "critical" + {">": [{"var": "check.value"}, 95]}, "failed" ]})) .unwrap(); let status = serde_json::Map::new(); @@ -153,14 +153,14 @@ fn numeric_ops_coerce_string_of_digits() { check.insert("value".into(), json!(97)); assert_eq!( ladder.evaluate(&empty_ctx(&check, &status, &tags)), - Some(Severity::Critical), + Some(CheckResult::Failed), ); // String of digits > 95 (bestool sometimes does this) → matches. check.insert("value".into(), json!("100")); assert_eq!( ladder.evaluate(&empty_ctx(&check, &status, &tags)), - Some(Severity::Critical), + Some(CheckResult::Failed), ); // Non-numeric string → false. @@ -189,7 +189,7 @@ fn eq_neq_handle_bool_string_numeric() { ]; for (lhs, rhs, want_eq, label) in cases { let ladder: IfLadder = serde_json::from_value(json!({"if": [ - {"==": [{"var": "check.value"}, rhs.clone()]}, "error" + {"==": [{"var": "check.value"}, rhs.clone()]}, "failed" ]})) .unwrap(); let mut check = serde_json::Map::new(); @@ -206,7 +206,7 @@ fn eq_neq_handle_bool_string_numeric() { #[test] fn tag_namespace_resolves_against_tag_map() { let ladder: IfLadder = serde_json::from_value(json!({"if": [ - {"==": [{"var": "tag.environment"}, "prod"]}, "error" + {"==": [{"var": "tag.environment"}, "prod"]}, "failed" ]})) .unwrap(); let check = serde_json::Map::new(); @@ -219,7 +219,7 @@ fn tag_namespace_resolves_against_tag_map() { ); assert_eq!( ladder.evaluate(&empty_ctx(&check, &status, &tags)), - Some(Severity::Error), + Some(CheckResult::Failed), ); tags.insert( @@ -235,7 +235,7 @@ fn tag_namespace_resolves_against_tag_map() { #[test] fn first_match_wins() { let ladder: IfLadder = serde_json::from_value(json!({"if": [ - {"<": [{"var": "check.days_remaining"}, 7]}, "error", + {"<": [{"var": "check.days_remaining"}, 7]}, "failed", {"<": [{"var": "check.days_remaining"}, 30]}, "warning" ]})) .unwrap(); @@ -246,23 +246,23 @@ fn first_match_wins() { check.insert("days_remaining".into(), json!(3)); assert_eq!( ladder.evaluate(&empty_ctx(&check, &status, &tags)), - Some(Severity::Error), + Some(CheckResult::Failed), ); check.insert("days_remaining".into(), json!(15)); assert_eq!( ladder.evaluate(&empty_ctx(&check, &status, &tags)), - Some(Severity::Warning), + Some(CheckResult::Warning), ); check.insert("days_remaining".into(), json!(60)); assert_eq!(ladder.evaluate(&empty_ctx(&check, &status, &tags)), None); } -// ── severity_for integration ───────────────────────────────────────────── +// ── CheckPolicy::apply integration ─────────────────────────────────────── #[tokio::test(flavor = "multi_thread")] -async fn severity_for_uses_rules_or_falls_back_to_base() { +async fn apply_uses_rules_or_falls_back_to_ceiling() { commons_tests::db::TestDb::run(async |mut conn, _| { - HealthcheckSeverity::upsert_default(&mut conn, "tamanu_service") + CheckPolicy::upsert_default(&mut conn, "alertd", "tamanu_service") .await .expect("seed"); @@ -270,68 +270,87 @@ async fn severity_for_uses_rules_or_falls_back_to_base() { {"in_range": [{"var": "status.bestoolVersion"}, ">=2.4.0 <2.5.4"]}, "warning" ]})) .unwrap(); - HealthcheckSeverity::update_rules(&mut conn, "tamanu_service", Some(&ladder), "ops") + CheckPolicy::update_rules(&mut conn, "alertd", "tamanu_service", Some(&ladder), "ops") .await .expect("save rules"); - // Bestool inside the range → ladder fires → Warning. + // Bestool inside the range → ladder fires → warning. let mut status = serde_json::Map::new(); status.insert("bestoolVersion".into(), json!("2.4.7")); let check = serde_json::Map::new(); let tags = HashMap::new(); - let sev = HealthcheckSeverity::severity_for( + let graded = CheckPolicy::apply( &mut conn, + "alertd", "tamanu_service", CheckResult::Failed, &empty_ctx(&check, &status, &tags), ) .await - .expect("severity_for"); - assert_eq!(sev, Severity::Warning); + .expect("apply"); + assert_eq!(graded.effective, CheckResult::Warning); - // Bestool outside the range → falls back to the base (Warning by default). + // Bestool outside the range → capped at the default warning ceiling. status.insert("bestoolVersion".into(), json!("2.6.0")); - let sev = HealthcheckSeverity::severity_for( + let graded = CheckPolicy::apply( &mut conn, + "alertd", "tamanu_service", CheckResult::Failed, &empty_ctx(&check, &status, &tags), ) .await - .expect("severity_for"); - assert_eq!(sev, Severity::Warning); + .expect("apply"); + assert_eq!(graded.effective, CheckResult::Warning); - // Bump the base to Error and re-test fallback path. - HealthcheckSeverity::update(&mut conn, "tamanu_service", Severity::Error, None, "ops") - .await - .expect("bump base"); + // Lift the ceiling to failed and re-test the fallback path. + CheckPolicy::update( + &mut conn, + "alertd", + "tamanu_service", + CheckResult::Failed, + false, + None, + "ops", + ) + .await + .expect("lift ceiling"); status.insert("bestoolVersion".into(), json!("2.6.0")); - let sev = HealthcheckSeverity::severity_for( + let graded = CheckPolicy::apply( &mut conn, + "alertd", "tamanu_service", CheckResult::Failed, &empty_ctx(&check, &status, &tags), ) .await - .expect("severity_for"); + .expect("apply"); assert_eq!( - sev, - Severity::Error, - "falls back to base when no branch matches" + graded.effective, + CheckResult::Failed, + "falls back to the ceiling cap when no branch matches" ); }) .await } #[tokio::test(flavor = "multi_thread")] -async fn severity_for_warning_result_rules_then_fixed_warning() { +async fn apply_rules_transform_in_both_directions() { commons_tests::db::TestDb::run(async |mut conn, _| { - HealthcheckSeverity::upsert_default(&mut conn, "queue_depth") + CheckPolicy::upsert_default(&mut conn, "alertd", "queue_depth") .await .expect("seed"); - HealthcheckSeverity::update(&mut conn, "queue_depth", Severity::Critical, None, "ops") - .await - .expect("bump base"); + CheckPolicy::update( + &mut conn, + "alertd", + "queue_depth", + CheckResult::Failed, + true, + None, + "ops", + ) + .await + .expect("lift ceiling"); let status = serde_json::Map::new(); let tags = HashMap::new(); @@ -339,66 +358,91 @@ async fn severity_for_warning_result_rules_then_fixed_warning() { let mut check = serde_json::Map::new(); check.insert("result".into(), json!("warning")); - // No rules: warning-result checks ignore the catalog column and - // land at fixed Warning. - let sev = HealthcheckSeverity::severity_for( + // No rules: a warning observation is below the ceiling — unchanged. + let graded = CheckPolicy::apply( &mut conn, + "alertd", "queue_depth", CheckResult::Warning, &empty_ctx(&check, &status, &tags), ) .await - .expect("severity_for"); - assert_eq!(sev, Severity::Warning); + .expect("apply"); + assert_eq!(graded.effective, CheckResult::Warning); + assert!(graded.escalates, "the flag rides along whatever the result"); - // A rule conditioned on check.result overrides the fixed default. + // A rule conditioned on check.result grades warnings down to passed. let ladder: IfLadder = serde_json::from_value(json!({"if": [ - {"==": [{"var": "check.result"}, "warning"]}, "info" + {"==": [{"var": "check.result"}, "warning"]}, "passed" ]})) .unwrap(); - HealthcheckSeverity::update_rules(&mut conn, "queue_depth", Some(&ladder), "ops") + CheckPolicy::update_rules(&mut conn, "alertd", "queue_depth", Some(&ladder), "ops") .await .expect("save rules"); - let sev = HealthcheckSeverity::severity_for( + let graded = CheckPolicy::apply( &mut conn, + "alertd", "queue_depth", CheckResult::Warning, &empty_ctx(&check, &status, &tags), ) .await - .expect("severity_for"); - assert_eq!(sev, Severity::Info); + .expect("apply"); + assert_eq!(graded.effective, CheckResult::Passed); // The same rule doesn't fire for a failed check, which keeps - // using the catalog base. + // using the ceiling. let mut failed_check = serde_json::Map::new(); failed_check.insert("result".into(), json!("failed")); - let sev = HealthcheckSeverity::severity_for( + let graded = CheckPolicy::apply( &mut conn, + "alertd", "queue_depth", CheckResult::Failed, &empty_ctx(&failed_check, &status, &tags), ) .await - .expect("severity_for"); - assert_eq!(sev, Severity::Critical); + .expect("apply"); + assert_eq!(graded.effective, CheckResult::Failed); + assert!(graded.escalates); + + // And a rule can upgrade: a passed observation graded to warning. + let ladder: IfLadder = serde_json::from_value(json!({"if": [ + {"==": [{"var": "check.result"}, "passed"]}, "warning" + ]})) + .unwrap(); + CheckPolicy::update_rules(&mut conn, "alertd", "queue_depth", Some(&ladder), "ops") + .await + .expect("save rules"); + let mut passed_check = serde_json::Map::new(); + passed_check.insert("result".into(), json!("passed")); + let graded = CheckPolicy::apply( + &mut conn, + "alertd", + "queue_depth", + CheckResult::Passed, + &empty_ctx(&passed_check, &status, &tags), + ) + .await + .expect("apply"); + assert_eq!(graded.effective, CheckResult::Warning); }) .await } #[tokio::test(flavor = "multi_thread")] -async fn severity_for_falls_back_when_rules_are_malformed() { +async fn apply_falls_back_when_rules_are_malformed() { use diesel::sql_query; use diesel::sql_types; use diesel_async::RunQueryDsl; commons_tests::db::TestDb::run(async |mut conn, _| { - HealthcheckSeverity::upsert_default(&mut conn, "tamanu_service") + CheckPolicy::upsert_default(&mut conn, "alertd", "tamanu_service") .await .expect("seed"); // Manually inject garbage into the rules column. sql_query( - "UPDATE healthcheck_severities SET rules = $1::jsonb WHERE check_name = 'tamanu_service'", + "UPDATE check_policies SET rules = $1::jsonb WHERE check_name = 'tamanu_service'", ) .bind::(r#"{"and": [true, true]}"#) .execute(&mut conn) @@ -408,16 +452,18 @@ async fn severity_for_falls_back_when_rules_are_malformed() { let check = serde_json::Map::new(); let status = serde_json::Map::new(); let tags = HashMap::new(); - let sev = HealthcheckSeverity::severity_for( + let graded = CheckPolicy::apply( &mut conn, + "alertd", "tamanu_service", CheckResult::Failed, &empty_ctx(&check, &status, &tags), ) .await - .expect("severity_for"); - // Catalog default is Warning; malformed rules don't crash, just defer. - assert_eq!(sev, Severity::Warning); + .expect("apply"); + // Catalog default ceiling is warning; malformed rules don't crash, + // just defer. + assert_eq!(graded.effective, CheckResult::Warning); }) .await } @@ -425,20 +471,20 @@ async fn severity_for_falls_back_when_rules_are_malformed() { #[tokio::test(flavor = "multi_thread")] async fn update_rules_clears_column_when_passed_none() { commons_tests::db::TestDb::run(async |mut conn, _| { - HealthcheckSeverity::upsert_default(&mut conn, "disk_space") + CheckPolicy::upsert_default(&mut conn, "alertd", "disk_space") .await .expect("seed"); let ladder: IfLadder = serde_json::from_value(json!({"if": [ - {">": [{"var": "check.used_pct"}, 95]}, "critical" + {">": [{"var": "check.used_pct"}, 95]}, "failed" ]})) .unwrap(); let saved = - HealthcheckSeverity::update_rules(&mut conn, "disk_space", Some(&ladder), "ops") + CheckPolicy::update_rules(&mut conn, "alertd", "disk_space", Some(&ladder), "ops") .await .expect("save"); assert!(saved.rules.is_some(), "rules should be populated"); - let cleared = HealthcheckSeverity::update_rules(&mut conn, "disk_space", None, "ops") + let cleared = CheckPolicy::update_rules(&mut conn, "alertd", "disk_space", None, "ops") .await .expect("clear"); assert!(cleared.rules.is_none(), "rules should be cleared by None"); diff --git a/crates/database/tests/it/check_severity_map.rs b/crates/database/tests/it/check_severity_map.rs index 5be1efca..2dfcd491 100644 --- a/crates/database/tests/it/check_severity_map.rs +++ b/crates/database/tests/it/check_severity_map.rs @@ -1,10 +1,10 @@ -//! Queries backing the device-facing effective check-severity map: -//! `HealthcheckSeverity::base_severity_map` (static catalog severities, +//! Queries backing the device-facing effective check map: +//! `CheckPolicy::ceiling_map_for_source` (static policy ceilings, //! ignoring conditional rules) and `silenced_refs::silenced_refs_with_prefix` -//! (server- plus group-scope silences under a source/ref prefix). +//! (server- plus group-scope silences under a ref prefix). -use commons_types::issue::Severity; -use database::healthcheck_severities::{HealthcheckSeverity, IfLadder}; +use commons_types::status::CheckResult; +use database::check_policies::{CheckPolicy, IfLadder}; use database::silenced_refs::{ ServerGroupSilencedRef, ServerSilencedRef, silenced_refs_with_prefix, }; @@ -38,51 +38,70 @@ async fn insert_server(conn: &mut diesel_async::AsyncPgConnection, group_id: Opt } #[tokio::test(flavor = "multi_thread")] -async fn base_severity_map_returns_static_severities() { +async fn ceiling_map_returns_static_ceilings_for_one_source() { commons_tests::db::TestDb::run(async |mut conn, _| { for check in ["disk_space", "cert_expiry", "chatty"] { - HealthcheckSeverity::upsert_default(&mut conn, check) + CheckPolicy::upsert_default(&mut conn, "alertd", check) .await .expect("seed"); } - HealthcheckSeverity::update(&mut conn, "disk_space", Severity::Error, None, "alice") + CheckPolicy::upsert_default(&mut conn, "seedling", "other_source_check") .await - .expect("update disk_space"); - HealthcheckSeverity::update(&mut conn, "chatty", Severity::Info, None, "alice") - .await - .expect("update chatty"); + .expect("seed other source"); + CheckPolicy::update( + &mut conn, + "alertd", + "disk_space", + CheckResult::Failed, + false, + None, + "alice", + ) + .await + .expect("update disk_space"); + CheckPolicy::update( + &mut conn, + "alertd", + "chatty", + CheckResult::Passed, + false, + None, + "alice", + ) + .await + .expect("update chatty"); - let map = HealthcheckSeverity::base_severity_map(&mut conn) + let map = CheckPolicy::ceiling_map_for_source(&mut conn, "alertd") .await .expect("map"); - assert_eq!(map.len(), 3); - assert_eq!(map.get("disk_space"), Some(&Severity::Error)); - assert_eq!(map.get("cert_expiry"), Some(&Severity::Warning)); - assert_eq!(map.get("chatty"), Some(&Severity::Info)); + assert_eq!(map.len(), 3, "only the requested source's checks"); + assert_eq!(map.get("disk_space"), Some(&CheckResult::Failed)); + assert_eq!(map.get("cert_expiry"), Some(&CheckResult::Warning)); + assert_eq!(map.get("chatty"), Some(&CheckResult::Passed)); }) .await } #[tokio::test(flavor = "multi_thread")] -async fn base_severity_map_ignores_conditional_rules() { +async fn ceiling_map_ignores_conditional_rules() { commons_tests::db::TestDb::run(async |mut conn, _| { - HealthcheckSeverity::upsert_default(&mut conn, "ruled") + CheckPolicy::upsert_default(&mut conn, "alertd", "ruled") .await .expect("seed"); let ladder: IfLadder = serde_json::from_value(json!({"if": [ - {"==": [{"var": "check.result"}, "failed"]}, "critical", + {"==": [{"var": "check.result"}, "failed"]}, "failed", ]})) .expect("parse ladder"); - HealthcheckSeverity::update_rules(&mut conn, "ruled", Some(&ladder), "alice") + CheckPolicy::update_rules(&mut conn, "alertd", "ruled", Some(&ladder), "alice") .await .expect("set rules"); - // The expression could raise a failure to critical at push time, but - // the static map must only reflect the base severity column. - let map = HealthcheckSeverity::base_severity_map(&mut conn) + // The expression could grade a failure through at push time, but + // the static map must only reflect the ceiling column. + let map = CheckPolicy::ceiling_map_for_source(&mut conn, "alertd") .await .expect("map"); - assert_eq!(map.get("ruled"), Some(&Severity::Warning)); + assert_eq!(map.get("ruled"), Some(&CheckResult::Warning)); }) .await } diff --git a/crates/database/tests/it/healthcheck_severities.rs b/crates/database/tests/it/healthcheck_severities.rs deleted file mode 100644 index bf04a25a..00000000 --- a/crates/database/tests/it/healthcheck_severities.rs +++ /dev/null @@ -1,128 +0,0 @@ -//! `HealthcheckSeverity` model: catalog of healthcheck names → severity, -//! maintained by the public-server status ingestion path and edited -//! through the private-server `/api/healthchecks` endpoints. - -use commons_types::{issue::Severity, status::CheckResult}; -use database::healthcheck_severities::HealthcheckSeverity; - -#[tokio::test(flavor = "multi_thread")] -async fn upsert_default_inserts_then_is_idempotent() { - commons_tests::db::TestDb::run(async |mut conn, _| { - HealthcheckSeverity::upsert_default(&mut conn, "disk_space") - .await - .expect("first upsert"); - HealthcheckSeverity::upsert_default(&mut conn, "disk_space") - .await - .expect("second upsert is no-op"); - - // Both upserts should land at the schema default, pending review. - let rows = HealthcheckSeverity::list(&mut conn).await.expect("list"); - assert_eq!(rows.len(), 1); - assert_eq!(rows[0].check_name, "disk_space"); - assert_eq!(rows[0].severity, Severity::Warning); - assert!(rows[0].reviewed_at.is_none(), "pending review by default"); - assert!(rows[0].reviewed_by.is_none()); - }) - .await -} - -#[tokio::test(flavor = "multi_thread")] -async fn upsert_default_does_not_overwrite_operator_severity() { - commons_tests::db::TestDb::run(async |mut conn, _| { - HealthcheckSeverity::upsert_default(&mut conn, "disk_space") - .await - .expect("seed"); - HealthcheckSeverity::update(&mut conn, "disk_space", Severity::Error, None, "alice") - .await - .expect("operator update"); - - // A subsequent push from ingestion must not revert the row. - HealthcheckSeverity::upsert_default(&mut conn, "disk_space") - .await - .expect("upsert again"); - - let rows = HealthcheckSeverity::list(&mut conn).await.expect("list"); - assert_eq!(rows.len(), 1); - assert_eq!(rows[0].severity, Severity::Error); - assert!(rows[0].reviewed_at.is_some()); - assert_eq!(rows[0].reviewed_by.as_deref(), Some("alice")); - }) - .await -} - -#[tokio::test(flavor = "multi_thread")] -async fn severity_for_returns_catalog_value_or_warning_default() { - commons_tests::db::TestDb::run(async |mut conn, _| { - let empty_map = serde_json::Map::new(); - let empty_tags = std::collections::HashMap::new(); - let ctx = database::healthcheck_severities::EvaluationContext { - status_extra: &empty_map, - check_extra: &empty_map, - tags: &empty_tags, - }; - // Unknown check → fallback (programmer-error path; ingestion - // upserts before reading in production). - let unknown = - HealthcheckSeverity::severity_for(&mut conn, "ghost", CheckResult::Failed, &ctx) - .await - .expect("lookup"); - assert_eq!(unknown, Severity::Warning); - - HealthcheckSeverity::upsert_default(&mut conn, "cert_expiry") - .await - .expect("seed"); - HealthcheckSeverity::update(&mut conn, "cert_expiry", Severity::Critical, None, "bob") - .await - .expect("update"); - - let known = - HealthcheckSeverity::severity_for(&mut conn, "cert_expiry", CheckResult::Failed, &ctx) - .await - .expect("lookup"); - assert_eq!(known, Severity::Critical); - - // A warning-result check ignores the catalog column and lands - // at fixed Warning when no rule matches. - let warned = - HealthcheckSeverity::severity_for(&mut conn, "cert_expiry", CheckResult::Warning, &ctx) - .await - .expect("lookup"); - assert_eq!(warned, Severity::Warning); - }) - .await -} - -#[tokio::test(flavor = "multi_thread")] -async fn update_stamps_review_metadata_even_on_no_op_save() { - commons_tests::db::TestDb::run(async |mut conn, _| { - HealthcheckSeverity::upsert_default(&mut conn, "noisy_check") - .await - .expect("seed"); - - // "Mark reviewed without changing severity": pass the current - // (default) value. reviewed_at + reviewed_by must still be set. - let updated = - HealthcheckSeverity::update(&mut conn, "noisy_check", Severity::Warning, None, "carol") - .await - .expect("update"); - assert_eq!(updated.severity, Severity::Warning); - assert!(updated.reviewed_at.is_some()); - assert_eq!(updated.reviewed_by.as_deref(), Some("carol")); - }) - .await -} - -#[tokio::test(flavor = "multi_thread")] -async fn list_orders_by_check_name() { - commons_tests::db::TestDb::run(async |mut conn, _| { - for name in ["zeta", "alpha", "mu"] { - HealthcheckSeverity::upsert_default(&mut conn, name) - .await - .expect("seed"); - } - let rows = HealthcheckSeverity::list(&mut conn).await.expect("list"); - let names: Vec<&str> = rows.iter().map(|r| r.check_name.as_str()).collect(); - assert_eq!(names, vec!["alpha", "mu", "zeta"]); - }) - .await -} diff --git a/crates/database/tests/it/main.rs b/crates/database/tests/it/main.rs index ee63d455..6dd6b331 100644 --- a/crates/database/tests/it/main.rs +++ b/crates/database/tests/it/main.rs @@ -6,10 +6,10 @@ mod backfill_registered_at_migration; mod backup_detection; mod backups; +mod check_policies; +mod check_policy_rules; mod check_severity_map; mod event_validation; -mod healthcheck_severities; -mod healthcheck_severity_rules; mod incident_close_severity; mod incident_get_with_issues; mod incident_severity_semantics; diff --git a/crates/private-server/src/fns/healthchecks.rs b/crates/private-server/src/fns/healthchecks.rs index 1362e3c3..b41662e0 100644 --- a/crates/private-server/src/fns/healthchecks.rs +++ b/crates/private-server/src/fns/healthchecks.rs @@ -1,5 +1,5 @@ -//! Operator-owned catalog of healthcheck names → severity. Read and -//! edit endpoints for the catalog page at /healthchecks. Ingestion +//! Operator-owned catalog of check policies per (source, check). Read +//! and edit endpoints for the catalog page at /healthchecks. Ingestion //! (in the public-server status handler) maintains the rows; this //! module exposes them to admins. @@ -8,8 +8,8 @@ use axum::extract::State; use canopy_utoipa_axum::{router::OpenApiRouter, routes}; use commons_errors::{AppError, ProblemDetailsSchema, Result}; use commons_servers::tailscale_auth::TailscaleAdmin; -use commons_types::issue::Severity; -use database::healthcheck_severities::{HealthcheckSeverity, IfLadder}; +use commons_types::status::CheckResult; +use database::check_policies::{CheckPolicy, IfLadder}; use database::servers::Server; use database::statuses::Status; use jiff::Timestamp; @@ -29,16 +29,25 @@ pub fn routes() -> OpenApiRouter { .routes(routes!(tag_keys)) } -/// A named healthcheck's alerting policy: the base severity assigned to -/// its failures, plus optional conditional rules that can override that -/// severity based on the details of a given failure. +/// One (source, check)'s policy: the ceiling capping its effective +/// result, the escalation flag, plus optional conditional rules that can +/// grade a given report differently. #[derive(Debug, Clone, Serialize, ToSchema)] -pub struct HealthcheckSeverityData { +pub struct CheckPolicyData { + /// The source that reports this check. + pub source: String, /// The healthcheck's name, exactly as reported by monitored servers. pub check_name: String, - /// The severity assigned to a failure of this check when no - /// conditional rule (see `rules`) overrides it. - pub severity: Severity, + /// The maximum effective result for this check when no conditional + /// rule (see `rules`) overrides it: an observed result more urgent + /// than the ceiling grades down to it. One of `failed`, `warning`, + /// `passed`, or `skipped` (`skipped` also tells the reporting source + /// it may stop running the check). + #[schema(value_type = String)] + pub ceiling: CheckResult, + /// Whether an effective failure of this check notifies immediately, + /// bypassing the incident grace period. + pub escalates: bool, /// When this check was first reported and this policy entry was /// created. pub first_seen: Timestamp, @@ -54,19 +63,19 @@ pub struct HealthcheckSeverityData { pub updated_at: Timestamp, /// `true` if no operator has reviewed this policy yet. pub pending_review: bool, - /// Conditional rules that can assign a different severity than - /// `severity`, depending on the failing check's own fields, the - /// surrounding status report, or the reporting server's tags. `null` - /// means no conditional rules are configured, and `severity` always - /// applies. + /// Conditional rules that can grade a report to a different result + /// than the ceiling would — in any direction — depending on the + /// check's own fields, the surrounding status report, or the + /// reporting server's tags. `null` means no conditional rules are + /// configured, and the ceiling always applies. /// /// When present, this is a single-key object shaped like - /// `{"if": [condition_1, severity_1, condition_2, severity_2, ...]}`. - /// Conditions are tried in order, and the severity paired with the - /// first matching condition is used; if none match, the base - /// `severity` is used instead. There's no explicit "else" branch — - /// the fallback to `severity` plays that role — so the array must - /// have an even number of entries and at least one pair. + /// `{"if": [condition_1, result_1, condition_2, result_2, ...]}`. + /// Conditions are tried in order, and the result paired with the + /// first matching condition is used; if none match, the observed + /// result capped at the ceiling is used instead. There's no explicit + /// "else" branch — the ceiling fallback plays that role — so the + /// array must have an even number of entries and at least one pair. /// /// Each condition is a single-key object naming a comparison /// operator — one of `==`, `!=`, `<`, `<=`, `>`, `>=`, or `in_range` @@ -83,7 +92,7 @@ pub struct HealthcheckSeverityData { /// is treated the same as `null` (no conditional rules). #[schema(value_type = Option)] pub rules: Option, - /// Number of condition/severity branches in `rules`; `0` when + /// Number of condition/result branches in `rules`; `0` when /// `rules` is `null` or couldn't be parsed. Lets a caller tell /// whether conditional rules exist without parsing `rules` itself. pub rule_count: u32, @@ -96,13 +105,15 @@ fn rule_count(rules: &Option) -> u32 { .unwrap_or(0) } -impl From for HealthcheckSeverityData { - fn from(h: HealthcheckSeverity) -> Self { +impl From for CheckPolicyData { + fn from(h: CheckPolicy) -> Self { let pending_review = h.reviewed_at.is_none(); let rule_count = rule_count(&h.rules); Self { + source: h.source, check_name: h.check_name, - severity: h.severity, + ceiling: h.ceiling, + escalates: h.escalates, first_seen: h.first_seen, reviewed_at: h.reviewed_at, reviewed_by: h.reviewed_by, @@ -115,11 +126,11 @@ impl From for HealthcheckSeverityData { } } -/// List the healthcheck severity catalog. +/// List the check policy catalog. /// -/// Returns every known healthcheck name together with its current -/// severity policy, ordered by name. An entry exists for every check name -/// any server has ever reported; new checks are added automatically the +/// Returns every known (source, check) together with its current policy, +/// ordered by source then name. An entry exists for every check any +/// source has ever reported; new checks are added automatically the /// first time they're seen, with a default policy pending review. #[utoipa::path( post, @@ -128,7 +139,7 @@ impl From for HealthcheckSeverityData { tag = "healthchecks", security(("tailscale-admin" = [])), responses( - (status = 200, description = "Catalog rows ordered by check_name.", body = Vec), + (status = 200, description = "Catalog rows ordered by source then check_name.", body = Vec), (status = 401, body = ProblemDetailsSchema), (status = 403, body = ProblemDetailsSchema), ), @@ -136,22 +147,30 @@ impl From for HealthcheckSeverityData { pub async fn list( State(state): State, _admin: TailscaleAdmin, -) -> Result>> { +) -> Result>> { let mut conn = state.db.get().await?; - let rows = HealthcheckSeverity::list(&mut conn).await?; + let rows = CheckPolicy::list(&mut conn).await?; Ok(Json(rows.into_iter().map(Into::into).collect())) } -/// Request body for updating a healthcheck's base severity policy. +/// Request body for updating a check's base policy. #[derive(Deserialize, ToSchema)] pub struct HealthcheckUpdateArgs { + /// The source whose check to update. + pub source: String, /// The healthcheck name to update; must already exist in the /// catalog. pub check_name: String, - /// The severity to assign to this check's failures when no - /// conditional rule overrides it. - pub severity: Severity, - /// Operator notes to store alongside the new severity. Omitting this + /// The ceiling to apply to this check's observed results when no + /// conditional rule overrides it: one of `failed`, `warning`, + /// `passed`, or `skipped`. + #[schema(value_type = String)] + pub ceiling: CheckResult, + /// Whether an effective failure of this check should notify + /// immediately, bypassing the incident grace period. + #[serde(default)] + pub escalates: bool, + /// Operator notes to store alongside the new policy. Omitting this /// or sending `null` clears any existing notes — there's no way to /// leave them unchanged implicitly, so resend the current value to /// keep it. @@ -159,12 +178,12 @@ pub struct HealthcheckUpdateArgs { pub notes: Option, } -/// Update a healthcheck's severity policy. +/// Update a check's policy. /// -/// Sets the base severity (and optionally notes) for the given check, and -/// marks it as reviewed by the caller. Saving with the same severity as -/// before still counts as a review, so an operator can acknowledge a -/// check without changing its policy. +/// Sets the ceiling and escalation flag (and optionally notes) for the +/// given (source, check), and marks it as reviewed by the caller. Saving +/// with the same values as before still counts as a review, so an +/// operator can acknowledge a check without changing its policy. #[utoipa::path( post, path = "/update", @@ -173,7 +192,7 @@ pub struct HealthcheckUpdateArgs { security(("tailscale-admin" = [])), request_body = HealthcheckUpdateArgs, responses( - (status = 200, description = "Updated catalog row.", body = HealthcheckSeverityData), + (status = 200, description = "Updated catalog row.", body = CheckPolicyData), (status = 401, body = ProblemDetailsSchema), (status = 403, body = ProblemDetailsSchema), ), @@ -182,12 +201,22 @@ pub async fn update( State(state): State, admin: TailscaleAdmin, Json(args): Json, -) -> Result> { +) -> Result> { + if !matches!( + args.ceiling, + CheckResult::Failed | CheckResult::Warning | CheckResult::Passed | CheckResult::Skipped + ) { + return Err(AppError::BadRequest( + "ceiling must be one of failed, warning, passed, skipped".into(), + )); + } let mut conn = state.db.get().await?; - let row = HealthcheckSeverity::update( + let row = CheckPolicy::update( &mut conn, + &args.source, &args.check_name, - args.severity, + args.ceiling, + args.escalates, args.notes.as_deref(), &admin.0.login, ) @@ -195,28 +224,30 @@ pub async fn update( Ok(Json(row.into())) } -/// Request body for replacing a healthcheck's conditional severity rules. +/// Request body for replacing a check's conditional rules. #[derive(Deserialize, ToSchema)] pub struct UpdateRulesArgs { + /// The source whose check's rules to replace. + pub source: String, /// The healthcheck name whose rules to replace; must already exist /// in the catalog. pub check_name: String, /// The new conditional rules to store, or `null` to remove all - /// conditional rules and rely solely on the base severity. Same - /// shape as the `rules` field returned when listing checks. A ladder - /// with no condition/severity pairs is treated the same as `null`. + /// conditional rules and rely solely on the ceiling. Same shape as + /// the `rules` field returned when listing checks. A ladder with no + /// condition/result pairs is treated the same as `null`. #[schema(value_type = Option)] #[serde(default)] pub rules: Option, } -/// Replace a healthcheck's conditional severity rules. +/// Replace a check's conditional rules. /// -/// Stores a new set of conditional rules for the given check (or removes -/// them, if `rules` is `null`), and marks the check as reviewed by the -/// caller. Returns 400 if `rules` doesn't parse as a valid ladder — for -/// example an unknown comparison operator, a malformed variable -/// reference, or an odd number of entries. +/// Stores a new set of conditional rules for the given (source, check) +/// (or removes them, if `rules` is `null`), and marks the check as +/// reviewed by the caller. Returns 400 if `rules` doesn't parse as a +/// valid ladder — for example an unknown comparison operator, a +/// malformed variable reference, or an odd number of entries. #[utoipa::path( post, path = "/update_rules", @@ -225,7 +256,7 @@ pub struct UpdateRulesArgs { security(("tailscale-admin" = [])), request_body = UpdateRulesArgs, responses( - (status = 200, description = "Updated catalog row.", body = HealthcheckSeverityData), + (status = 200, description = "Updated catalog row.", body = CheckPolicyData), (status = 400, body = ProblemDetailsSchema), (status = 401, body = ProblemDetailsSchema), (status = 403, body = ProblemDetailsSchema), @@ -235,7 +266,7 @@ pub async fn update_rules( State(state): State, admin: TailscaleAdmin, Json(args): Json, -) -> Result> { +) -> Result> { let ladder: Option = match args.rules { None | Some(JsonValue::Null) => None, Some(v) => { @@ -251,8 +282,9 @@ pub async fn update_rules( } }; let mut conn = state.db.get().await?; - let row = HealthcheckSeverity::update_rules( + let row = CheckPolicy::update_rules( &mut conn, + &args.source, &args.check_name, ladder.as_ref(), &admin.0.login, @@ -346,7 +378,7 @@ pub async fn sample( // by name; we don't require a failing result here so we still // surface the check's typical shape even on a passing push). Strip // the reserved fields and inject the normalised `result` so the UI - // sees exactly what the ingestion path passes to severity_for — + // sees exactly what the ingestion path passes to the policy — // including a `check.result` value on legacy (`healthy: bool`) // pushes. let check_extra = status diff --git a/crates/private-server/src/fns/statuses.rs b/crates/private-server/src/fns/statuses.rs index 86bcdaca..d8dae579 100644 --- a/crates/private-server/src/fns/statuses.rs +++ b/crates/private-server/src/fns/statuses.rs @@ -6,7 +6,6 @@ use canopy_utoipa_axum::{router::OpenApiRouter, routes}; use commons_errors::{ProblemDetailsSchema, Result}; use commons_servers::tailscale_auth::TailscaleUser; use commons_types::{ - issue::Severity, server::{ cards::{FacilityServerStatus, ServerGroupCard}, rank::ServerRank, @@ -15,7 +14,7 @@ use commons_types::{ version::VersionStr, }; use database::{ - devices::DeviceConnection, healthcheck_severities::HealthcheckSeverity, issues::Issue, + check_policies::CheckPolicy, devices::DeviceConnection, issues::Issue, server_groups::ServerGroup, servers::Server, statuses::Status, tailscale_users::TailscaleUser as CachedTailscaleUser, versions::Version, }; @@ -321,7 +320,7 @@ pub struct CheckAttentionArgs { pub check: String, } -/// Response for [`check_attention`]: the queried check's catalog severity +/// Response for [`check_attention`]: the queried check's catalog policy /// (if it has one yet) and every live server whose latest status reports /// it, failing or healthy. #[derive(Debug, Clone, Serialize, Deserialize, ToSchema)] @@ -329,9 +328,14 @@ pub struct CheckAttentionData { /// The check name that was queried, echoed back so the page can /// render its heading without re-decoding the request. pub check: String, - /// The catalog's configured base severity for this check, or `None` - /// if no server has ever reported it yet (so it has no catalog row). - pub severity: Option, + /// The most urgent configured policy ceiling for this check across + /// the sources that report it, or `None` if no server has ever + /// reported it yet (so it has no catalog row). + #[schema(value_type = Option)] + pub ceiling: Option, + /// Whether any source's policy for this check escalates its + /// effective failures. + pub escalates: bool, /// Every live server whose latest status reports this check, at any /// result, ordered as a TODO list: failed, warning, broken, passed, /// skipped (most urgent first), then by group name then server name. @@ -433,13 +437,17 @@ pub async fn check_attention( .then_with(|| a.server_name.cmp(&b.server_name)) }); - let severity = HealthcheckSeverity::get(&mut conn, &args.check) - .await? - .map(|row| row.severity); + let policies = CheckPolicy::get_by_name(&mut conn, &args.check).await?; + let ceiling = policies + .iter() + .map(|row| row.ceiling) + .min_by_key(|c| c.urgency_rank()); + let escalates = policies.iter().any(|row| row.escalates); Ok(Json(CheckAttentionData { check: args.check, - severity, + ceiling, + escalates, servers, })) } @@ -652,20 +660,20 @@ pub(crate) async fn enrich_operators( Ok(()) } -/// For every warning/failed check on `status`, resolve the catalog + -/// rules severity given the snapshot's actual extras and the server's -/// resolved tag map. Mirrors the public-server ingestion path -/// (`file_health_events`) so the UI displays what *would* be filed. -/// Broken checks aren't included — they file at a fixed Warning and -/// the UI renders them from the result directly; passed/skipped file -/// nothing. +/// For every warning/failed check on `status`, resolve the policy + +/// rules grading given the snapshot's actual extras and the server's +/// resolved tag map, expressed as the severity ingestion would file. +/// Mirrors the public-server ingestion path (`file_health_events`) so +/// the UI displays what *would* be filed. Broken checks aren't included +/// — they file at a fixed Warning and the UI renders them from the +/// result directly. async fn compute_check_severities( conn: &mut database::diesel_async::AsyncPgConnection, server: &Server, status: &Status, ) -> commons_errors::Result> { use commons_types::status::CheckResult; - use database::healthcheck_severities::{EvaluationContext, HealthcheckSeverity}; + use database::check_policies::{CheckPolicy, EvaluationContext}; let Some(arr) = status.health.as_array() else { return Ok(Default::default()); @@ -720,7 +728,13 @@ async fn compute_check_severities( check_extra: &check_extra, tags: &tags, }; - let sev = HealthcheckSeverity::severity_for(conn, &name, result, &ctx).await?; + let graded = CheckPolicy::apply(conn, &status.source, &name, result, &ctx).await?; + let sev = match graded.effective { + CheckResult::Failed if graded.escalates => commons_types::issue::Severity::Critical, + CheckResult::Failed => commons_types::issue::Severity::Error, + CheckResult::Warning | CheckResult::Broken => commons_types::issue::Severity::Warning, + CheckResult::Passed | CheckResult::Skipped => continue, + }; out.insert(name, sev); } Ok(out) diff --git a/crates/private-server/tests/it/healthchecks.rs b/crates/private-server/tests/it/healthchecks.rs index 75e90162..7fe9e18f 100644 --- a/crates/private-server/tests/it/healthchecks.rs +++ b/crates/private-server/tests/it/healthchecks.rs @@ -7,10 +7,10 @@ use serde_json::json; async fn list_returns_catalog_rows_with_pending_review_flag() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name, severity) VALUES \ - ('disk_space', 'warning'), \ - ('reviewed_check', 'error'); \ - UPDATE healthcheck_severities \ + "INSERT INTO check_policies (source, check_name, ceiling) VALUES \ + ('alertd', 'disk_space', 'warning'), \ + ('alertd', 'reviewed_check', 'failed'); \ + UPDATE check_policies \ SET reviewed_at = NOW(), reviewed_by = 'alice@example.com' \ WHERE check_name = 'reviewed_check';", ) @@ -26,14 +26,15 @@ async fn list_returns_catalog_rows_with_pending_review_flag() { let body: Vec = response.json(); assert_eq!(body.len(), 2); - // Ordered by check_name. + // Ordered by source then check_name. + assert_eq!(body[0]["source"], "alertd"); assert_eq!(body[0]["check_name"], "disk_space"); - assert_eq!(body[0]["severity"], "warning"); + assert_eq!(body[0]["ceiling"], "warning"); assert_eq!(body[0]["pending_review"], true); assert!(body[0]["reviewed_at"].is_null()); assert_eq!(body[1]["check_name"], "reviewed_check"); - assert_eq!(body[1]["severity"], "error"); + assert_eq!(body[1]["ceiling"], "failed"); assert_eq!(body[1]["pending_review"], false); assert_eq!(body[1]["reviewed_by"], "alice@example.com"); }) @@ -41,10 +42,10 @@ async fn list_returns_catalog_rows_with_pending_review_flag() { } #[tokio::test(flavor = "multi_thread")] -async fn update_changes_severity_and_stamps_review_metadata() { +async fn update_changes_policy_and_stamps_review_metadata() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name) VALUES ('disk_space');", + "INSERT INTO check_policies (source, check_name) VALUES ('alertd', 'disk_space');", ) .await .unwrap(); @@ -52,15 +53,18 @@ async fn update_changes_severity_and_stamps_review_metadata() { let response = private .post("/api/healthchecks/update") .json(&json!({ + "source": "alertd", "check_name": "disk_space", - "severity": "error" + "ceiling": "failed", + "escalates": true })) .await; response.assert_status_ok(); let body: serde_json::Value = response.json(); assert_eq!(body["check_name"], "disk_space"); - assert_eq!(body["severity"], "error"); + assert_eq!(body["ceiling"], "failed"); + assert_eq!(body["escalates"], true); assert_eq!(body["pending_review"], false); assert!( !body["reviewed_at"].is_null(), @@ -73,10 +77,10 @@ async fn update_changes_severity_and_stamps_review_metadata() { } #[tokio::test(flavor = "multi_thread")] -async fn update_rejects_unknown_severity() { +async fn update_rejects_unknown_ceiling() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name) VALUES ('disk_space');", + "INSERT INTO check_policies (source, check_name) VALUES ('alertd', 'disk_space');", ) .await .unwrap(); @@ -84,8 +88,9 @@ async fn update_rejects_unknown_severity() { let response = private .post("/api/healthchecks/update") .json(&json!({ + "source": "alertd", "check_name": "disk_space", - "severity": "extremely_critical" + "ceiling": "extremely_critical" })) .await; response.assert_status_not_ok(); @@ -94,20 +99,21 @@ async fn update_rejects_unknown_severity() { } #[tokio::test(flavor = "multi_thread")] -async fn update_marks_reviewed_even_when_severity_unchanged() { +async fn update_marks_reviewed_even_when_policy_unchanged() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name) VALUES ('noisy_check');", + "INSERT INTO check_policies (source, check_name) VALUES ('alertd', 'noisy_check');", ) .await .unwrap(); - // Pass the same default severity to ack without changing. + // Pass the same default policy to ack without changing. let response = private .post("/api/healthchecks/update") .json(&json!({ + "source": "alertd", "check_name": "noisy_check", - "severity": "warning" + "ceiling": "warning" })) .await; response.assert_status_ok(); @@ -126,11 +132,11 @@ async fn update_marks_reviewed_even_when_severity_unchanged() { async fn list_returns_rules_and_rule_count() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name, rules) VALUES \ - ('no_rules', NULL), \ - ('one_rule', '{\"if\": [{\"==\": [{\"var\": \"check.x\"}, 1]}, \"error\"]}'::jsonb), \ - ('two_rules', '{\"if\": [{\"==\": [{\"var\": \"check.x\"}, 1]}, \"error\", {\"==\": [{\"var\": \"check.x\"}, 2]}, \"warning\"]}'::jsonb), \ - ('garbage_rules', '{\"and\": [true]}'::jsonb);", + "INSERT INTO check_policies (source, check_name, rules) VALUES \ + ('alertd', 'no_rules', NULL), \ + ('alertd', 'one_rule', '{\"if\": [{\"==\": [{\"var\": \"check.x\"}, 1]}, \"failed\"]}'::jsonb), \ + ('alertd', 'two_rules', '{\"if\": [{\"==\": [{\"var\": \"check.x\"}, 1]}, \"failed\", {\"==\": [{\"var\": \"check.x\"}, 2]}, \"warning\"]}'::jsonb), \ + ('alertd', 'garbage_rules', '{\"and\": [true]}'::jsonb);", ) .await .unwrap(); @@ -165,7 +171,7 @@ async fn list_returns_rules_and_rule_count() { async fn update_rules_accepts_valid_ladder() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name) VALUES ('disk_space');", + "INSERT INTO check_policies (source, check_name) VALUES ('alertd', 'disk_space');", ) .await .unwrap(); @@ -173,9 +179,10 @@ async fn update_rules_accepts_valid_ladder() { let response = private .post("/api/healthchecks/update_rules") .json(&json!({ + "source": "alertd", "check_name": "disk_space", "rules": {"if": [ - {">": [{"var": "check.used_pct"}, 95]}, "critical" + {">": [{"var": "check.used_pct"}, 95]}, "failed" ]} })) .await; @@ -193,15 +200,15 @@ async fn update_rules_accepts_valid_ladder() { async fn update_rules_with_null_clears_the_column() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name, rules) VALUES \ - ('disk_space', '{\"if\": [{\"==\": [{\"var\": \"check.x\"}, 1]}, \"error\"]}'::jsonb);", + "INSERT INTO check_policies (source, check_name, rules) VALUES \ + ('alertd', 'disk_space', '{\"if\": [{\"==\": [{\"var\": \"check.x\"}, 1]}, \"failed\"]}'::jsonb);", ) .await .unwrap(); let response = private .post("/api/healthchecks/update_rules") - .json(&json!({"check_name": "disk_space", "rules": null})) + .json(&json!({"source": "alertd", "check_name": "disk_space", "rules": null})) .await; response.assert_status_ok(); let body: serde_json::Value = response.json(); @@ -215,7 +222,7 @@ async fn update_rules_with_null_clears_the_column() { async fn update_rules_normalises_empty_ladder_to_null() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name) VALUES ('disk_space');", + "INSERT INTO check_policies (source, check_name) VALUES ('alertd', 'disk_space');", ) .await .unwrap(); @@ -223,7 +230,7 @@ async fn update_rules_normalises_empty_ladder_to_null() { // normalise it to null at write time. let response = private .post("/api/healthchecks/update_rules") - .json(&json!({"check_name": "disk_space", "rules": {"if": []}})) + .json(&json!({"source": "alertd", "check_name": "disk_space", "rules": {"if": []}})) .await; // Either the API rejects an empty ladder OR normalises it. Both // land at `rule_count == 0` and a null rules column. @@ -242,20 +249,20 @@ async fn update_rules_normalises_empty_ladder_to_null() { async fn update_rules_rejects_malformed_shapes() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name) VALUES ('disk_space');", + "INSERT INTO check_policies (source, check_name) VALUES ('alertd', 'disk_space');", ) .await .unwrap(); let cases: &[(serde_json::Value, &str)] = &[ (json!({"and": [true]}), "AND composition"), - (json!({"if": [{"if": [true, true]}, "error"]}), "nested if"), + (json!({"if": [{"if": [true, true]}, "failed"]}), "nested if"), ( - json!({"if": [{"==": [{"var": "BAD.x"}, 1]}, "error"]}), + json!({"if": [{"==": [{"var": "BAD.x"}, 1]}, "failed"]}), "unknown var namespace", ), ( - json!({"if": [{"==": [{"var": "check.x"}, 1]}, "not_a_severity"]}), - "bad severity", + json!({"if": [{"==": [{"var": "check.x"}, 1]}, "not_a_result"]}), + "bad result", ), ( json!({"if": [{"in_range": [{"var": "status.v"}, "not-a-range"]}, "warning"]}), @@ -265,7 +272,7 @@ async fn update_rules_rejects_malformed_shapes() { for (rules, label) in cases { let response = private .post("/api/healthchecks/update_rules") - .json(&json!({"check_name": "disk_space", "rules": rules})) + .json(&json!({"source": "alertd", "check_name": "disk_space", "rules": rules})) .await; assert!( !response.status_code().is_success(), @@ -282,7 +289,7 @@ async fn update_rules_rejects_malformed_shapes() { async fn sample_returns_null_when_no_server_has_reported_the_check() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name) VALUES ('uncharted_check');", + "INSERT INTO check_policies (source, check_name) VALUES ('alertd', 'uncharted_check');", ) .await .unwrap(); diff --git a/crates/private-server/tests/it/private_statuses.rs b/crates/private-server/tests/it/private_statuses.rs index 679df671..30bc6d69 100644 --- a/crates/private-server/tests/it/private_statuses.rs +++ b/crates/private-server/tests/it/private_statuses.rs @@ -1046,7 +1046,7 @@ struct CheckAttentionServer { #[derive(Debug, Deserialize)] struct CheckAttentionResponse { check: String, - severity: Option, + ceiling: Option, servers: Vec, } @@ -1060,7 +1060,7 @@ async fn check_attention_empty_database() { r.assert_status_ok(); let data: CheckAttentionResponse = r.json(); assert_eq!(data.check, "postgres"); - assert_eq!(data.severity, None); + assert_eq!(data.ceiling, None); assert!(data.servers.is_empty()); }) .await @@ -1099,7 +1099,7 @@ async fn check_attention_lists_servers_reporting_that_check_ordered_failed_first let data: CheckAttentionResponse = r.json(); assert_eq!(data.check, "postgres"); - assert_eq!(data.severity, None, "no catalog row was ever created"); + assert_eq!(data.ceiling, None, "no catalog row was ever created"); assert_eq!( data.servers.len(), 3, @@ -1241,10 +1241,10 @@ async fn check_attention_excludes_ungrouped_and_archived_servers() { } #[tokio::test(flavor = "multi_thread")] -async fn check_attention_returns_catalog_severity_and_ignores_non_matching_check() { +async fn check_attention_returns_catalog_policy_and_ignores_non_matching_check() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name, severity) VALUES ('postgres', 'error'); + "INSERT INTO check_policies (source, check_name, ceiling) VALUES ('alertd', 'postgres', 'failed'); INSERT INTO servers (id, name, host, rank, kind) VALUES ('11111111-1111-1111-1111-111111111111', 'Failing Server', 'https://failing.example.com', 'production', 'central'); INSERT INTO statuses (server_id, created_at, healthy, health) VALUES @@ -1261,7 +1261,7 @@ async fn check_attention_returns_catalog_severity_and_ignores_non_matching_check .await; r.assert_status_ok(); let data: CheckAttentionResponse = r.json(); - assert_eq!(data.severity, Some("error".to_string())); + assert_eq!(data.ceiling, Some("failed".to_string())); assert_eq!(data.servers.len(), 1); // A different, never-reported check name: no servers, but the @@ -1273,7 +1273,7 @@ async fn check_attention_returns_catalog_severity_and_ignores_non_matching_check r.assert_status_ok(); let data: CheckAttentionResponse = r.json(); assert_eq!(data.check, "unrelated_check"); - assert_eq!(data.severity, None); + assert_eq!(data.ceiling, None); assert!(data.servers.is_empty()); }) .await @@ -1282,18 +1282,18 @@ async fn check_attention_returns_catalog_severity_and_ignores_non_matching_check #[tokio::test(flavor = "multi_thread")] async fn snapshot_surfaces_per_check_severity() { commons_tests::server::run(async |mut conn, _, private| { - // Seed catalog: catalog_only stays at the default warning; - // elevated has its base severity bumped to error so the snapshot - // returns the operator-set value, not the legacy heuristic; - // version_gated has a rules ladder firing on a specific - // status.bestoolVersion. + // Seed catalog: catalog_only stays at the default warning + // ceiling; elevated has its ceiling lifted to failed so the + // snapshot returns the operator-set grading; version_gated has a + // rules ladder firing on a specific status.bestoolVersion, and + // escalates. conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name, severity) VALUES \ - ('catalog_only', 'warning'), \ - ('elevated', 'error'), \ - ('version_gated', 'warning'); \ - UPDATE healthcheck_severities \ - SET rules = '{\"if\":[{\"in_range\":[{\"var\":\"status.bestoolVersion\"},\">=1.0.0 <2.0.0\"]},\"critical\"]}'::jsonb \ + "INSERT INTO check_policies (source, check_name, ceiling, escalates) VALUES \ + ('alertd', 'catalog_only', 'warning', FALSE), \ + ('alertd', 'elevated', 'failed', FALSE), \ + ('alertd', 'version_gated', 'warning', TRUE); \ + UPDATE check_policies \ + SET rules = '{\"if\":[{\"in_range\":[{\"var\":\"status.bestoolVersion\"},\">=1.0.0 <2.0.0\"]},\"failed\"]}'::jsonb \ WHERE check_name = 'version_gated';", ) .await @@ -1325,7 +1325,8 @@ async fn snapshot_surfaces_per_check_severity() { assert_eq!(severities["catalog_only"], "warning"); assert_eq!(severities["elevated"], "error"); // version_gated's rule fires because bestoolVersion 1.5.0 is in - // the >=1.0.0 <2.0.0 range — Critical wins over the base. + // the >=1.0.0 <2.0.0 range — graded failed, and the entry + // escalates, so it presents as critical. assert_eq!(severities["version_gated"], "critical"); // Passing checks must not appear in the map. assert!( @@ -1342,9 +1343,9 @@ async fn snapshot_check_severities_cover_result_form() { // `elevated` has its catalog base bumped to error: a failed // result uses it, a warning result ignores it (fixed Warning). conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name, severity) VALUES \ - ('elevated', 'error'), \ - ('degraded', 'error');", + "INSERT INTO check_policies (source, check_name, ceiling) VALUES \ + ('alertd', 'elevated', 'failed'), \ + ('alertd', 'degraded', 'failed');", ) .await .unwrap(); @@ -1376,7 +1377,7 @@ async fn snapshot_check_severities_cover_result_form() { assert_eq!(severities["elevated"], "error"); assert_eq!( severities["degraded"], "warning", - "warning result lands at fixed Warning regardless of catalog base" + "a warning observation is already below the ceiling" ); // Broken/skipped/passed don't go through the rules engine. for check in ["busted", "absent", "fine"] { diff --git a/crates/public-server/openapi.json b/crates/public-server/openapi.json index 5f2dac26..c021df55 100644 --- a/crates/public-server/openapi.json +++ b/crates/public-server/openapi.json @@ -818,7 +818,7 @@ "statuses" ], "summary": "Fetch the effective healthcheck severity mapping for a server.", - "description": "Returns, for every healthcheck canopy knows about, how a failure of that\ncheck is classified for this server: `skip` (the check is silenced for\nthis server — at server or group scope — or its severity is below\nwarning), `warn` (warning), or `fail` (error or critical). Keys are check\nnames as reported in `health[].check` on status pushes. Only the static\nseverity baseline is reflected; operator-defined conditional rules are\nevaluated per push and not included here. The same mapping also rides\nalong every status-push response as `check_severities`.\n\nThe calling device must be the one enrolled for this exact server (or\nhold the admin role).", + "description": "Returns, for every healthcheck the `alertd` source reports, how that\ncheck is handled for this server: `skip` (the check is silenced for\nthis server — at server or group scope — or its policy ceiling means it\nnever alerts), `warn` (graded at most a warning), or `fail` (failures\ncount as failures). Keys are check names as reported in\n`health[].check` on status pushes. Only the static policy ceiling is\nreflected; operator-defined conditional rules are evaluated per push\nand not included here. The same mapping also rides along every\nstatus-push response as `check_severities`, scoped to the pushing\nsource.\n\nThe calling device must be the one enrolled for this exact server (or\nhold the admin role).", "operationId": "check_severities", "parameters": [ { diff --git a/crates/public-server/src/statuses.rs b/crates/public-server/src/statuses.rs index db8f0f0e..b80ef70f 100644 --- a/crates/public-server/src/statuses.rs +++ b/crates/public-server/src/statuses.rs @@ -20,9 +20,9 @@ use commons_types::{ }; use database::{ Db, + check_policies::{CheckPolicy, EvaluationContext, GradedResult}, devices::Device, diesel_async::{AsyncConnection, AsyncPgConnection}, - healthcheck_severities::{EvaluationContext, HealthcheckSeverity}, issues::{Issue, NewEvent}, servers::Server, silenced_refs::silenced_refs_with_prefix, @@ -260,7 +260,7 @@ async fn create( } create_legacy_status(&mut db, server_id, id, extra, version).await?; let check_severities = - effective_check_severities(&mut db, server_id, server.group_id).await?; + effective_check_severities(&mut db, server_id, server.group_id, &source).await?; let tags = crate::tags::effective_tags_for_server(&mut db, &server).await?; return Ok(Json(StatusResponse { backup_now, @@ -289,7 +289,7 @@ async fn create( extra, healthy, health, - source, + source: source.clone(), } .save(conn) .await?; @@ -302,7 +302,8 @@ async fn create( // Computed after the transaction so checks first seen on this very push // (upserted into the catalog above) are already in the map. - let check_severities = effective_check_severities(&mut db, server_id, server.group_id).await?; + let check_severities = + effective_check_severities(&mut db, server_id, server.group_id, &source).await?; let effective_tags = crate::tags::effective_tags_for_server(&mut db, &server).await?; Ok(Json(StatusResponse { @@ -314,14 +315,16 @@ async fn create( /// Fetch the effective healthcheck severity mapping for a server. /// -/// Returns, for every healthcheck canopy knows about, how a failure of that -/// check is classified for this server: `skip` (the check is silenced for -/// this server — at server or group scope — or its severity is below -/// warning), `warn` (warning), or `fail` (error or critical). Keys are check -/// names as reported in `health[].check` on status pushes. Only the static -/// severity baseline is reflected; operator-defined conditional rules are -/// evaluated per push and not included here. The same mapping also rides -/// along every status-push response as `check_severities`. +/// Returns, for every healthcheck the `alertd` source reports, how that +/// check is handled for this server: `skip` (the check is silenced for +/// this server — at server or group scope — or its policy ceiling means it +/// never alerts), `warn` (graded at most a warning), or `fail` (failures +/// count as failures). Keys are check names as reported in +/// `health[].check` on status pushes. Only the static policy ceiling is +/// reflected; operator-defined conditional rules are evaluated per push +/// and not included here. The same mapping also rides along every +/// status-push response as `check_severities`, scoped to the pushing +/// source. /// /// The calling device must be the one enrolled for this exact server (or /// hold the admin role). @@ -356,26 +359,28 @@ async fn check_severities( )); } - let map = effective_check_severities(&mut db, server_id, server.group_id).await?; + let map = + effective_check_severities(&mut db, server_id, server.group_id, DEFAULT_SOURCE).await?; Ok(Json(map)) } -/// Build the effective per-check severity map for a server: every check in -/// the operator catalog mapped from its static base severity (error and -/// above → `fail`, warning → `warn`, below warning → `skip`), then any check -/// silenced for this server (at server or group scope) forced to `skip`. -/// Conditional severity rules are deliberately not consulted — they depend -/// on each push's contents, so only the static baseline can be mapped ahead -/// of time. +/// Build the effective per-check map for a server and source: every check +/// in the source's catalog mapped from its static policy ceiling (`failed` +/// → `fail`, `warning`/`broken` → `warn`, `passed`/`skipped` → `skip`), +/// then any check silenced for this server (at server or group scope) +/// forced to `skip`. Conditional rules are deliberately not consulted — +/// they depend on each push's contents, so only the static ceiling can be +/// mapped ahead of time. async fn effective_check_severities( db: &mut AsyncPgConnection, server_id: Uuid, group_id: Option, + source: &str, ) -> Result> { - let mut map: BTreeMap = HealthcheckSeverity::base_severity_map(db) + let mut map: BTreeMap = CheckPolicy::ceiling_map_for_source(db, source) .await? .into_iter() - .map(|(name, severity)| (name, severity.into())) + .map(|(name, ceiling)| (name, ceiling.into())) .collect(); let health_prefix = format!("{HEALTH_REF}/"); @@ -443,14 +448,19 @@ fn resolve_version(extra: &serde_json::Value, header: Option) -> Opt /// (`result: skipped` — precondition not met) file nothing and close /// both refs. /// -/// Severity for each warning/failed check comes from the -/// operator-owned `healthcheck_severities` catalog (see -/// [`HealthcheckSeverity::severity_for`] for the rules/fallback -/// contract). Every check seen on a push — whatever its result — -/// upserts a default catalog row so new checks are visible to -/// operators immediately at the default Warning severity. -/// `status.healthy` is intentionally not consulted: the catalog is -/// canopy's single source of truth for per-check severity. +/// Each check's effective result comes from applying the operator-owned +/// `check_policies` catalog entry for `(source, check)` (see +/// [`CheckPolicy::apply`] for the rules/ceiling contract) to the +/// observed result. Every check seen on a push — whatever its result — +/// upserts a default catalog row so new checks are visible to operators +/// immediately at the default warning ceiling. `status.healthy` is +/// intentionally not consulted: the catalog is canopy's single source +/// of truth for per-check grading. +/// +/// Until issues themselves carry results, the effective result maps to +/// the issue severity: failed → error (critical when the policy +/// escalates), warning and broken → warning; passed and skipped file +/// nothing and close prior issues. async fn file_health_events( conn: &mut AsyncPgConnection, server_id: Uuid, @@ -462,22 +472,24 @@ async fn file_health_events( let occurred_at = Some(status.created_at); // Upsert a catalog row for every check name seen on this push, - // whatever its result. New checks land at default Warning; - // operators can review and adjust from the /healthchecks page. + // whatever its result. New checks land at the default warning + // ceiling; operators can review and adjust from the /healthchecks + // page. for check_name in curr_check_results.keys() { - HealthcheckSeverity::upsert_default(conn, check_name).await?; + CheckPolicy::upsert_default(conn, &status.source, check_name).await?; } // Status-level extras are shared across every per-check evaluation. let empty_map = serde_json::Map::new(); let status_extra = status.extra.as_object().unwrap_or(&empty_map); - // Per-check opens: warning and failed file on the same ref (one - // thread per check; the filed severity is what differs). - for (check, result) in curr_check_results - .iter() - .filter(|(_, r)| matches!(r, CheckResult::Warning | CheckResult::Failed)) - { + // Grade every check in the push through its policy. Observed broken + // stays on its own thread below and is not graded here. + let mut effective: BTreeMap<&String, GradedResult> = BTreeMap::new(); + for (check, result) in &curr_check_results { + if matches!(result, CheckResult::Broken) { + continue; + } let entry = find_health_entry(&status.health, check); // Strip the reserved `check` / `healthy` keys, and replace any // wire-form `result` with the normalised value so rules see a @@ -495,8 +507,21 @@ async fn file_health_events( check_extra: &check_extra, tags, }; - let severity = HealthcheckSeverity::severity_for(conn, check, *result, &ctx).await?; - let described = match result { + let graded = CheckPolicy::apply(conn, &status.source, check, *result, &ctx).await?; + effective.insert(check, graded); + } + + // Per-check opens: effective warning and failed file on the same ref + // (one thread per check; the filed severity is what differs). + for (check, graded) in &effective { + let severity = match graded.effective { + CheckResult::Failed if graded.escalates => Severity::Critical, + CheckResult::Failed => Severity::Error, + CheckResult::Warning | CheckResult::Broken => Severity::Warning, + CheckResult::Passed | CheckResult::Skipped => continue, + }; + let entry = find_health_entry(&status.health, check); + let described = match graded.effective { CheckResult::Warning => "warned", _ => "failed", }; @@ -541,11 +566,12 @@ async fn file_health_events( // failure open), so the previous push alone can't tell us what // needs closing. - // Failure closes: an open `health/` closes when the check - // is now passed, skipped, or unmentioned ("trust the reporter"). - // Broken does NOT close a prior failure — the check can't confirm - // the failure either way while it's broken. Scoped to the pushing - // source: one source's push says nothing about another's checks. + // Failure closes: an open `health/` closes when the check's + // effective result is now passed or skipped, or the check is + // unmentioned ("trust the reporter"). Observed broken does NOT close + // a prior failure — the check can't confirm the failure either way + // while it's broken. Scoped to the pushing source: one source's push + // says nothing about another's checks. let health_prefix = format!("{HEALTH_REF}/"); for r#ref in Issue::active_refs_with_prefix(conn, server_id, &status.source, &health_prefix).await? @@ -554,10 +580,12 @@ async fn file_health_events( continue; }; let curr = curr_check_results.get(check); - if matches!( - curr, - Some(CheckResult::Warning | CheckResult::Failed | CheckResult::Broken) - ) { + let keeps_open = matches!(curr, Some(CheckResult::Broken)) + || matches!( + effective.get(&check.to_string()).map(|g| g.effective), + Some(CheckResult::Warning | CheckResult::Failed | CheckResult::Broken) + ); + if keeps_open { continue; } let message = if matches!(curr, Some(CheckResult::Skipped)) { diff --git a/crates/public-server/tests/it/check_severities.rs b/crates/public-server/tests/it/check_severities.rs index e905c25b..b6398f36 100644 --- a/crates/public-server/tests/it/check_severities.rs +++ b/crates/public-server/tests/it/check_severities.rs @@ -37,22 +37,22 @@ async fn insert_server( server_id } -/// Seed a catalog row at a given severity, optionally with a conditional +/// Seed a policy row at a given ceiling, optionally with a conditional /// rules ladder (which the mapping must ignore). async fn seed_catalog( conn: &mut diesel_async::AsyncPgConnection, check_name: &str, - severity: &str, + ceiling: &str, rules: Option, ) { sql_query( - "INSERT INTO healthcheck_severities (check_name, severity, rules, reviewed_at, reviewed_by) \ - VALUES ($1, $2, $3, NOW(), 'test') \ - ON CONFLICT (check_name) DO UPDATE \ - SET severity = EXCLUDED.severity, rules = EXCLUDED.rules", + "INSERT INTO check_policies (source, check_name, ceiling, rules, reviewed_at, reviewed_by) \ + VALUES ('alertd', $1, $2, $3, NOW(), 'test') \ + ON CONFLICT (source, check_name) DO UPDATE \ + SET ceiling = EXCLUDED.ceiling, rules = EXCLUDED.rules", ) .bind::(check_name) - .bind::(severity) + .bind::(ceiling) .bind::, _>(rules) .execute(conn) .await @@ -67,20 +67,20 @@ async fn endpoint_maps_catalog_severities_and_silences() { let group_id = insert_group(&mut conn).await; let server_id = insert_server(&mut conn, Some(device_id), Some(group_id)).await; - seed_catalog(&mut conn, "disk_space", "error", None).await; + seed_catalog(&mut conn, "disk_space", "failed", None).await; seed_catalog(&mut conn, "cert_expiry", "warning", None).await; - seed_catalog(&mut conn, "chatty", "info", None).await; - seed_catalog(&mut conn, "verbose", "debug", None).await; - seed_catalog(&mut conn, "flaky", "critical", None).await; - seed_catalog(&mut conn, "groupwide", "error", None).await; - // A conditional ladder that would escalate to critical at push + seed_catalog(&mut conn, "chatty", "passed", None).await; + seed_catalog(&mut conn, "verbose", "skipped", None).await; + seed_catalog(&mut conn, "flaky", "failed", None).await; + seed_catalog(&mut conn, "groupwide", "failed", None).await; + // A conditional ladder that would grade up to failed at push // time must not leak into the static mapping. seed_catalog( &mut conn, "ruled", "warning", Some(serde_json::json!({"if": [ - {"==": [{"var": "check.result"}, "failed"]}, "critical", + {"==": [{"var": "check.result"}, "failed"]}, "failed", ]})), ) .await; @@ -133,7 +133,7 @@ async fn endpoint_works_for_ungrouped_server() { "server", async |mut conn, cert, device_id, public, _| { let server_id = insert_server(&mut conn, Some(device_id), None).await; - seed_catalog(&mut conn, "disk_space", "error", None).await; + seed_catalog(&mut conn, "disk_space", "failed", None).await; ServerSilencedRef::add(&mut conn, server_id, "alertd", "health/disk_space", None) .await .expect("silence"); @@ -158,8 +158,8 @@ async fn status_response_carries_check_severities() { let group_id = insert_group(&mut conn).await; let server_id = insert_server(&mut conn, Some(device_id), Some(group_id)).await; - seed_catalog(&mut conn, "disk_space", "error", None).await; - seed_catalog(&mut conn, "cert_expiry", "critical", None).await; + seed_catalog(&mut conn, "disk_space", "failed", None).await; + seed_catalog(&mut conn, "cert_expiry", "failed", None).await; ServerSilencedRef::add(&mut conn, server_id, "alertd", "health/cert_expiry", None) .await .expect("silence"); diff --git a/crates/public-server/tests/it/statuses.rs b/crates/public-server/tests/it/statuses.rs index e128a2d0..477826fe 100644 --- a/crates/public-server/tests/it/statuses.rs +++ b/crates/public-server/tests/it/statuses.rs @@ -81,29 +81,40 @@ struct IncidentRow { id: Uuid, } -/// Pre-seed (or update) a catalog row so a check's failure severity is -/// known up-front. v1 ingestion would otherwise auto-insert at the -/// default Warning, which only opens an incident when one already -/// exists. Tests that want to exercise Error-class behaviour seed -/// here. +/// Pre-seed (or update) a policy row so a check's grading is known +/// up-front, expressed in the old severity vocabulary the tests speak: +/// critical → failed + escalates, error → failed, warning → warning, +/// info → passed, debug → skipped. Ingestion would otherwise +/// auto-insert at the default warning ceiling, which only opens an +/// incident when one already exists. async fn set_check_severity( conn: &mut diesel_async::AsyncPgConnection, check_name: &str, severity: &str, ) { + let (ceiling, escalates) = match severity { + "critical" => ("failed", true), + "error" => ("failed", false), + "warning" => ("warning", false), + "info" => ("passed", false), + "debug" => ("skipped", false), + other => panic!("unknown severity {other}"), + }; sql_query( - "INSERT INTO healthcheck_severities (check_name, severity, reviewed_at, reviewed_by) \ - VALUES ($1, $2, NOW(), 'test') \ - ON CONFLICT (check_name) DO UPDATE \ - SET severity = EXCLUDED.severity, \ + "INSERT INTO check_policies (source, check_name, ceiling, escalates, reviewed_at, reviewed_by) \ + VALUES ('alertd', $1, $2, $3, NOW(), 'test') \ + ON CONFLICT (source, check_name) DO UPDATE \ + SET ceiling = EXCLUDED.ceiling, \ + escalates = EXCLUDED.escalates, \ reviewed_at = EXCLUDED.reviewed_at, \ reviewed_by = EXCLUDED.reviewed_by", ) .bind::(check_name) - .bind::(severity) + .bind::(ceiling) + .bind::(escalates) .execute(conn) .await - .expect("seed catalog severity"); + .expect("seed catalog policy"); } async fn fetch_open_incident( @@ -1532,7 +1543,9 @@ async fn submit_status_keeps_incident_open_when_failure_swaps() { #[derive(QueryableByName, Debug)] struct CatalogRow { #[diesel(sql_type = sql_types::Text)] - severity: String, + source: String, + #[diesel(sql_type = sql_types::Text)] + ceiling: String, #[diesel(sql_type = sql_types::Bool)] pending_review: bool, } @@ -1542,8 +1555,8 @@ async fn fetch_catalog( check_name: &str, ) -> Option { sql_query( - "SELECT severity, reviewed_at IS NULL AS pending_review \ - FROM healthcheck_severities WHERE check_name = $1", + "SELECT source, ceiling, reviewed_at IS NULL AS pending_review \ + FROM check_policies WHERE check_name = $1", ) .bind::(check_name) .get_result(conn) @@ -1582,13 +1595,15 @@ async fn submit_status_seeds_catalog_for_new_checks() { let failing = fetch_catalog(&mut conn, "brand_new_check") .await .expect("failing check seeded in catalog"); - assert_eq!(failing.severity, "warning"); + assert_eq!(failing.source, "alertd"); + assert_eq!(failing.ceiling, "warning"); assert!(failing.pending_review); let passing = fetch_catalog(&mut conn, "passing_check") .await .expect("passing check seeded in catalog"); - assert_eq!(passing.severity, "warning"); + assert_eq!(passing.source, "alertd"); + assert_eq!(passing.ceiling, "warning"); assert!(passing.pending_review); }, ) @@ -1638,14 +1653,14 @@ async fn set_check_rules( ) { // Ensure the catalog row exists. sql_query( - "INSERT INTO healthcheck_severities (check_name) VALUES ($1) \ - ON CONFLICT (check_name) DO NOTHING", + "INSERT INTO check_policies (source, check_name) VALUES ('alertd', $1) \ + ON CONFLICT (source, check_name) DO NOTHING", ) .bind::(check_name) .execute(conn) .await .expect("ensure catalog row"); - sql_query("UPDATE healthcheck_severities SET rules = $1::jsonb WHERE check_name = $2") + sql_query("UPDATE check_policies SET rules = $1::jsonb WHERE check_name = $2") .bind::(rules.to_string()) .bind::(check_name) .execute(conn) @@ -1679,7 +1694,7 @@ async fn submit_status_rule_on_check_extra_overrides_base() { &mut conn, "disk_space", serde_json::json!({"if": [ - {">": [{"var": "check.used_pct"}, 90]}, "critical" + {">": [{"var": "check.used_pct"}, 90]}, "failed" ]}), ) .await; @@ -1697,7 +1712,7 @@ async fn submit_status_rule_on_check_extra_overrides_base() { let issue = fetch_issue(&mut conn, server_id, "alertd", "health/disk_space") .await .expect("per-check issue filed"); - assert_eq!(issue.severity, "critical"); + assert_eq!(issue.severity, "error"); // Below-threshold push falls back to base (default warning). post_status( @@ -1795,7 +1810,7 @@ async fn submit_status_rule_on_server_tag() { &mut conn, "cert_expiry", serde_json::json!({"if": [ - {"==": [{"var": "tag.environment"}, "prod"]}, "error" + {"==": [{"var": "tag.environment"}, "prod"]}, "failed" ]}), ) .await; @@ -1833,7 +1848,7 @@ async fn submit_status_tiered_ladder() { &mut conn, "cert_expiry", serde_json::json!({"if": [ - {"<": [{"var": "check.days_remaining"}, 7]}, "error", + {"<": [{"var": "check.days_remaining"}, 7]}, "failed", {"<": [{"var": "check.days_remaining"}, 30]}, "warning" ]}), ) @@ -2008,7 +2023,8 @@ async fn submit_status_result_warning_ignores_catalog_severity() { } /// Custom rules can condition on the normalised `check.result` — and -/// they win over both the fixed-Warning default and the catalog base. +/// they win over both the observed result and the ceiling. A warning +/// graded down to passed files nothing at all. #[tokio::test(flavor = "multi_thread")] async fn submit_status_result_rule_on_check_result() { commons_tests::server::run_with_device_auth( @@ -2019,7 +2035,7 @@ async fn submit_status_result_rule_on_check_result() { &mut conn, "db", serde_json::json!({"if": [ - {"==": [{"var": "check.result"}, "warning"]}, "info" + {"==": [{"var": "check.result"}, "warning"]}, "passed" ]}), ) .await; @@ -2034,10 +2050,12 @@ async fn submit_status_result_rule_on_check_result() { ) .await; - let issue = fetch_issue(&mut conn, server_id, "alertd", "health/db") - .await - .expect("per-check issue filed"); - assert_eq!(issue.severity, "info", "rule overrides the fixed default"); + assert!( + fetch_issue(&mut conn, server_id, "alertd", "health/db") + .await + .is_none(), + "a warning graded to passed files nothing", + ); }, ) .await @@ -2312,7 +2330,7 @@ async fn submit_status_result_all_kinds_upsert_catalog() { check_name: String, } let rows: Vec = - sql_query("SELECT check_name FROM healthcheck_severities ORDER BY check_name") + sql_query("SELECT check_name FROM check_policies ORDER BY check_name") .get_results(&mut conn) .await .expect("list catalog"); diff --git a/migrations/2026-07-08-072903-0000_check_policies/down.sql b/migrations/2026-07-08-072903-0000_check_policies/down.sql new file mode 100644 index 00000000..7edae493 --- /dev/null +++ b/migrations/2026-07-08-072903-0000_check_policies/down.sql @@ -0,0 +1,56 @@ +-- Best-effort reversal: ceilings map back to base severities (escalating +-- failures to critical), rules branches map back to severity strings, and +-- per-source entries collapse to one row per check name (alertd wins, +-- otherwise an arbitrary source's entry). +CREATE TABLE healthcheck_severities ( + check_name TEXT PRIMARY KEY, + severity TEXT NOT NULL DEFAULT 'warning', + first_seen TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(), + reviewed_at TIMESTAMP WITH TIME ZONE, + reviewed_by TEXT, + notes TEXT, + updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(), + rules JSONB +); + +INSERT INTO healthcheck_severities + (check_name, severity, first_seen, reviewed_at, reviewed_by, notes, updated_at, rules) +SELECT DISTINCT ON (check_name) + check_name, + CASE ceiling + WHEN 'failed' THEN CASE WHEN escalates THEN 'critical' ELSE 'error' END + WHEN 'warning' THEN 'warning' + WHEN 'passed' THEN 'info' + WHEN 'skipped' THEN 'debug' + ELSE 'warning' + END, + first_seen, + reviewed_at, + reviewed_by, + notes, + updated_at, + ( + SELECT jsonb_build_object( + 'if', + jsonb_agg( + CASE + WHEN ord % 2 = 1 THEN elem + ELSE to_jsonb( + CASE elem #>> '{}' + WHEN 'failed' THEN 'error' + WHEN 'warning' THEN 'warning' + WHEN 'passed' THEN 'info' + WHEN 'skipped' THEN 'debug' + ELSE 'warning' + END + ) + END + ORDER BY ord + ) + ) + FROM jsonb_array_elements(rules -> 'if') WITH ORDINALITY AS t(elem, ord) + ) +FROM check_policies +ORDER BY check_name, (source = 'alertd') DESC; + +DROP TABLE check_policies; diff --git a/migrations/2026-07-08-072903-0000_check_policies/up.sql b/migrations/2026-07-08-072903-0000_check_policies/up.sql new file mode 100644 index 00000000..28958b8a --- /dev/null +++ b/migrations/2026-07-08-072903-0000_check_policies/up.sql @@ -0,0 +1,82 @@ +-- The severity catalog becomes the check-policy catalog: keyed per +-- (source, check) now that checks are keyed per reporting source, and +-- speaking result transforms instead of severities. The ceiling is the +-- maximum effective result (failed = no cap, warning = failures grade as +-- warnings, passed = recorded but never alerting, skipped = the source +-- may stop running the check); escalates marks checks whose effective +-- failure notifies immediately, bypassing incident grace — the residue +-- of the Critical severity. +CREATE TABLE check_policies ( + source TEXT NOT NULL, + check_name TEXT NOT NULL, + ceiling TEXT NOT NULL DEFAULT 'warning', + escalates BOOLEAN NOT NULL DEFAULT FALSE, + rules JSONB, + notes TEXT, + first_seen TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(), + reviewed_at TIMESTAMP WITH TIME ZONE, + reviewed_by TEXT, + updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(), + PRIMARY KEY (source, check_name) +); + +-- Existing catalog rows were all reported by alertd. Severity → policy: +-- Critical → failed + escalates, Error → failed, Warning → warning, +-- Info → passed, Debug → skipped. An entry whose rules ladder could +-- output critical also escalates (escalation is per-entry now, not +-- per-branch). +INSERT INTO check_policies + (source, check_name, ceiling, escalates, rules, notes, first_seen, reviewed_at, reviewed_by, updated_at) +SELECT + 'alertd', + check_name, + CASE severity + WHEN 'critical' THEN 'failed' + WHEN 'error' THEN 'failed' + WHEN 'warning' THEN 'warning' + WHEN 'info' THEN 'passed' + WHEN 'debug' THEN 'skipped' + ELSE 'warning' + END, + severity = 'critical' OR EXISTS ( + SELECT 1 + FROM jsonb_array_elements(rules -> 'if') WITH ORDINALITY AS t(elem, ord) + WHERE ord % 2 = 0 AND elem #>> '{}' = 'critical' + ), + rules, + notes, + first_seen, + reviewed_at, + reviewed_by, + updated_at +FROM healthcheck_severities; + +-- Rules ladders output results now: rewrite each branch's severity +-- string (the even-ordinal elements of the "if" array) to the result it +-- maps to. +UPDATE check_policies +SET rules = ( + SELECT jsonb_build_object( + 'if', + jsonb_agg( + CASE + WHEN ord % 2 = 1 THEN elem + ELSE to_jsonb( + CASE elem #>> '{}' + WHEN 'critical' THEN 'failed' + WHEN 'error' THEN 'failed' + WHEN 'warning' THEN 'warning' + WHEN 'info' THEN 'passed' + WHEN 'debug' THEN 'skipped' + ELSE 'warning' + END + ) + END + ORDER BY ord + ) + ) + FROM jsonb_array_elements(rules -> 'if') WITH ORDINALITY AS t(elem, ord) +) +WHERE rules IS NOT NULL; + +DROP TABLE healthcheck_severities; diff --git a/private-web/e2e/healthcheck-attention.spec.ts b/private-web/e2e/healthcheck-attention.spec.ts index 4d0fcf86..73b3b614 100644 --- a/private-web/e2e/healthcheck-attention.spec.ts +++ b/private-web/e2e/healthcheck-attention.spec.ts @@ -1,7 +1,7 @@ import { expect, test } from "./test-fixtures"; import { resetSeededTables, - seedHealthcheckSeverity, + seedCheckPolicy, seedIssue, seedServer, seedServerGroup, @@ -238,7 +238,7 @@ test.describe("healthcheck attention page", () => { await expect(page.getByText("—")).toBeVisible(); }); - test("shows the catalog severity when one is configured", async ({ + test("shows the catalog ceiling when one is configured", async ({ page, sql, }) => { @@ -247,13 +247,19 @@ test.describe("healthcheck attention page", () => { serverId: server.id, health: [{ check: "disk_space", result: "failed", free_pct: 2 }], }); - await seedHealthcheckSeverity(sql, { + await seedCheckPolicy(sql, { checkName: "disk_space", - severity: "critical", + ceiling: "failed", + escalates: true, }); await page.goto("/healthchecks/disk_space"); - await expect(page.getByText("critical", { exact: true })).toBeVisible(); + // The heading chip renders the configured ceiling, plus the + // escalates marker. + await expect( + page.getByRole("heading", { name: "disk_space" }), + ).toBeVisible(); + await expect(page.getByText("escalates", { exact: true })).toBeVisible(); }); test("URL-encodes check names with special characters", async ({ diff --git a/private-web/e2e/seed.ts b/private-web/e2e/seed.ts index fc20f829..84863ab2 100644 --- a/private-web/e2e/seed.ts +++ b/private-web/e2e/seed.ts @@ -47,7 +47,7 @@ function randomLabel(prefix: string): string { * statement with CASCADE. */ export async function resetSeededTables(sql: Sql): Promise { await sql.query( - "TRUNCATE statuses, issues, device_keys, servers, server_groups, devices, versions, tailscale_users, healthcheck_severities, server_group_backup_config, server_group_backup_schedule, server_backup_capabilities, backup_requests, backup_runs, backup_repo_stats, backup_maintenance_runs, backup_credential_issuances, restore_replicas, restore_consumer_capabilities, backup_restore_checks, recovery_vault_writes RESTART IDENTITY CASCADE", + "TRUNCATE statuses, issues, device_keys, servers, server_groups, devices, versions, tailscale_users, check_policies, server_group_backup_config, server_group_backup_schedule, server_backup_capabilities, backup_requests, backup_runs, backup_repo_stats, backup_maintenance_runs, backup_credential_issuances, restore_replicas, restore_consumer_capabilities, backup_restore_checks, recovery_vault_writes RESTART IDENTITY CASCADE", ); // The truncate takes the migration-seeded nil "Canopy" server with it; // self-alerts attach to that row, so put it back. @@ -237,34 +237,44 @@ export async function seedStatus( return { id, createdAt: String(rows[0]!.created_at) }; } -export interface SeededHealthcheckSeverity { +export interface SeededCheckPolicy { + source: string; checkName: string; } -/** Catalog row for a healthcheck name, as ingestion would have upserted - * (plus an operator-set severity). Upserts so tests can call it without +/** Policy row for a (source, check), as ingestion would have upserted + * (plus an operator-set ceiling). Upserts so tests can call it without * worrying whether ingestion already created a default row. */ -export async function seedHealthcheckSeverity( +export async function seedCheckPolicy( sql: Sql, opts: { checkName: string; - severity?: string; + source?: string; + ceiling?: string; + escalates?: boolean; notes?: string | null; }, -): Promise { +): Promise { + const source = opts.source ?? "alertd"; await sql.query( - `INSERT INTO healthcheck_severities (check_name, severity, notes) - VALUES ($1, $2, $3) - ON CONFLICT (check_name) - DO UPDATE SET severity = EXCLUDED.severity, notes = EXCLUDED.notes`, - [opts.checkName, opts.severity ?? "warning", opts.notes ?? null], + `INSERT INTO check_policies (source, check_name, ceiling, escalates, notes) + VALUES ($1, $2, $3, $4, $5) + ON CONFLICT (source, check_name) + DO UPDATE SET ceiling = EXCLUDED.ceiling, escalates = EXCLUDED.escalates, notes = EXCLUDED.notes`, + [ + source, + opts.checkName, + opts.ceiling ?? "warning", + opts.escalates ?? false, + opts.notes ?? null, + ], ); - return { checkName: opts.checkName }; + return { source, checkName: opts.checkName }; } /** Server-scope silence for a `(source, ref)` pair, as the UI's * silence button would create. For a healthcheck, pass - * `ref: "health/"` (source defaults to "status"). */ + * `ref: "health/"` (source defaults to "alertd"). */ export async function seedServerSilencedRef( sql: Sql, opts: { @@ -278,7 +288,7 @@ export async function seedServerSilencedRef( `INSERT INTO server_silenced_refs (server_id, source, ref, created_by) VALUES ($1, $2, $3, $4) ON CONFLICT DO NOTHING`, - [opts.serverId, opts.source ?? "status", opts.ref, opts.createdBy ?? null], + [opts.serverId, opts.source ?? "alertd", opts.ref, opts.createdBy ?? null], ); } @@ -297,7 +307,7 @@ export async function seedGroupSilencedRef( `INSERT INTO server_group_silenced_refs (server_group_id, source, ref, created_by) VALUES ($1, $2, $3, $4) ON CONFLICT DO NOTHING`, - [opts.groupId, opts.source ?? "status", opts.ref, opts.createdBy ?? null], + [opts.groupId, opts.source ?? "alertd", opts.ref, opts.createdBy ?? null], ); } @@ -361,7 +371,7 @@ export async function seedIssue( opts.serverId ?? null, opts.serverGroupId ?? null, opts.deviceId ?? null, - opts.source ?? "status", + opts.source ?? "alertd", opts.ref ?? "health", opts.severity ?? "error", opts.message ?? "Issue message", diff --git a/private-web/openapi.json b/private-web/openapi.json index 056db5cc..d6b469eb 100644 --- a/private-web/openapi.json +++ b/private-web/openapi.json @@ -2512,18 +2512,18 @@ "tags": [ "healthchecks" ], - "summary": "List the healthcheck severity catalog.", - "description": "Returns every known healthcheck name together with its current\nseverity policy, ordered by name. An entry exists for every check name\nany server has ever reported; new checks are added automatically the\nfirst time they're seen, with a default policy pending review.", + "summary": "List the check policy catalog.", + "description": "Returns every known (source, check) together with its current policy,\nordered by source then name. An entry exists for every check any\nsource has ever reported; new checks are added automatically the\nfirst time they're seen, with a default policy pending review.", "operationId": "healthcheck_list", "responses": { "200": { - "description": "Catalog rows ordered by check_name.", + "description": "Catalog rows ordered by source then check_name.", "content": { "application/json": { "schema": { "type": "array", "items": { - "$ref": "#/components/schemas/HealthcheckSeverityData" + "$ref": "#/components/schemas/CheckPolicyData" } } } @@ -2669,8 +2669,8 @@ "tags": [ "healthchecks" ], - "summary": "Update a healthcheck's severity policy.", - "description": "Sets the base severity (and optionally notes) for the given check, and\nmarks it as reviewed by the caller. Saving with the same severity as\nbefore still counts as a review, so an operator can acknowledge a\ncheck without changing its policy.", + "summary": "Update a check's policy.", + "description": "Sets the ceiling and escalation flag (and optionally notes) for the\ngiven (source, check), and marks it as reviewed by the caller. Saving\nwith the same values as before still counts as a review, so an\noperator can acknowledge a check without changing its policy.", "operationId": "healthcheck_update", "requestBody": { "content": { @@ -2688,7 +2688,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/HealthcheckSeverityData" + "$ref": "#/components/schemas/CheckPolicyData" } } } @@ -2726,8 +2726,8 @@ "tags": [ "healthchecks" ], - "summary": "Replace a healthcheck's conditional severity rules.", - "description": "Stores a new set of conditional rules for the given check (or removes\nthem, if `rules` is `null`), and marks the check as reviewed by the\ncaller. Returns 400 if `rules` doesn't parse as a valid ladder — for\nexample an unknown comparison operator, a malformed variable\nreference, or an odd number of entries.", + "summary": "Replace a check's conditional rules.", + "description": "Stores a new set of conditional rules for the given (source, check)\n(or removes them, if `rules` is `null`), and marks the check as\nreviewed by the caller. Returns 400 if `rules` doesn't parse as a\nvalid ladder — for example an unknown comparison operator, a\nmalformed variable reference, or an odd number of entries.", "operationId": "healthcheck_update_rules", "requestBody": { "content": { @@ -2745,7 +2745,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/HealthcheckSeverityData" + "$ref": "#/components/schemas/CheckPolicyData" } } } @@ -6860,33 +6860,34 @@ }, "CheckAttentionData": { "type": "object", - "description": "Response for [`check_attention`]: the queried check's catalog severity\n(if it has one yet) and every live server whose latest status reports\nit, failing or healthy.", + "description": "Response for [`check_attention`]: the queried check's catalog policy\n(if it has one yet) and every live server whose latest status reports\nit, failing or healthy.", "required": [ "check", + "escalates", "servers" ], "properties": { + "ceiling": { + "type": [ + "string", + "null" + ], + "description": "The most urgent configured policy ceiling for this check across\nthe sources that report it, or `None` if no server has ever\nreported it yet (so it has no catalog row)." + }, "check": { "type": "string", "description": "The check name that was queried, echoed back so the page can\nrender its heading without re-decoding the request." }, + "escalates": { + "type": "boolean", + "description": "Whether any source's policy for this check escalates its\neffective failures." + }, "servers": { "type": "array", "items": { "$ref": "#/components/schemas/CheckAttentionServerData" }, "description": "Every live server whose latest status reports this check, at any\nresult, ordered as a TODO list: failed, warning, broken, passed,\nskipped (most urgent first), then by group name then server name.\nThe client filters out the passed/skipped tail unless the \"show\nhealthy\" toggle is on." - }, - "severity": { - "oneOf": [ - { - "type": "null" - }, - { - "$ref": "#/components/schemas/Severity", - "description": "The catalog's configured base severity for this check, or `None`\nif no server has ever reported it yet (so it has no catalog row)." - } - ] } } }, @@ -6947,6 +6948,83 @@ } } }, + "CheckPolicyData": { + "type": "object", + "description": "One (source, check)'s policy: the ceiling capping its effective\nresult, the escalation flag, plus optional conditional rules that can\ngrade a given report differently.", + "required": [ + "source", + "check_name", + "ceiling", + "escalates", + "first_seen", + "updated_at", + "pending_review", + "rule_count" + ], + "properties": { + "ceiling": { + "type": "string", + "description": "The maximum effective result for this check when no conditional\nrule (see `rules`) overrides it: an observed result more urgent\nthan the ceiling grades down to it. One of `failed`, `warning`,\n`passed`, or `skipped` (`skipped` also tells the reporting source\nit may stop running the check)." + }, + "check_name": { + "type": "string", + "description": "The healthcheck's name, exactly as reported by monitored servers." + }, + "escalates": { + "type": "boolean", + "description": "Whether an effective failure of this check notifies immediately,\nbypassing the incident grace period." + }, + "first_seen": { + "type": "string", + "format": "date-time", + "description": "When this check was first reported and this policy entry was\ncreated." + }, + "notes": { + "type": [ + "string", + "null" + ], + "description": "Free-form operator notes about this check." + }, + "pending_review": { + "type": "boolean", + "description": "`true` if no operator has reviewed this policy yet." + }, + "reviewed_at": { + "type": [ + "string", + "null" + ], + "format": "date-time", + "description": "When an operator last reviewed or updated this policy. `null` if\nit has never been reviewed." + }, + "reviewed_by": { + "type": [ + "string", + "null" + ], + "description": "The operator who last reviewed this policy. `null` if it has\nnever been reviewed." + }, + "rule_count": { + "type": "integer", + "format": "int32", + "description": "Number of condition/result branches in `rules`; `0` when\n`rules` is `null` or couldn't be parsed. Lets a caller tell\nwhether conditional rules exist without parsing `rules` itself.", + "minimum": 0 + }, + "rules": { + "description": "Conditional rules that can grade a report to a different result\nthan the ceiling would — in any direction — depending on the\ncheck's own fields, the surrounding status report, or the\nreporting server's tags. `null` means no conditional rules are\nconfigured, and the ceiling always applies.\n\nWhen present, this is a single-key object shaped like\n`{\"if\": [condition_1, result_1, condition_2, result_2, ...]}`.\nConditions are tried in order, and the result paired with the\nfirst matching condition is used; if none match, the observed\nresult capped at the ceiling is used instead. There's no explicit\n\"else\" branch — the ceiling fallback plays that role — so the\narray must have an even number of entries and at least one pair.\n\nEach condition is a single-key object naming a comparison\noperator — one of `==`, `!=`, `<`, `<=`, `>`, `>=`, or `in_range`\n— whose value is a two-element array: a variable reference and a\nvalue to compare it against. A variable reference has the shape\n`{\"var\": \".\"}`, where `` is one of\n`check` (a field on the failing check itself), `status` (a\ntop-level field on the status report that contained it), or\n`tag` (a tag on the reporting server, merged with its group's\ntags). `in_range` compares a version-like string against a\nsemantic version range (e.g. `\">=1.2.0 <2.0.0\"`). If the named\nvariable isn't present in the data being evaluated, the condition\ndoesn't match. A `rules` value that doesn't parse into this shape\nis treated the same as `null` (no conditional rules)." + }, + "source": { + "type": "string", + "description": "The source that reports this check." + }, + "updated_at": { + "type": "string", + "format": "date-time", + "description": "When this policy was last modified." + } + } + }, "CheckResult": { "type": "string", "description": "Outcome of a single health check reported in a server's status update.\n\nOlder reports may send a plain pass/fail flag instead of one of these\noutcomes; when that happens a passing flag is treated as `passed` and a\nfailing flag as `failed`.", @@ -7845,95 +7923,37 @@ } } }, - "HealthcheckSeverityData": { + "HealthcheckUpdateArgs": { "type": "object", - "description": "A named healthcheck's alerting policy: the base severity assigned to\nits failures, plus optional conditional rules that can override that\nseverity based on the details of a given failure.", + "description": "Request body for updating a check's base policy.", "required": [ + "source", "check_name", - "severity", - "first_seen", - "updated_at", - "pending_review", - "rule_count" + "ceiling" ], "properties": { - "check_name": { - "type": "string", - "description": "The healthcheck's name, exactly as reported by monitored servers." - }, - "first_seen": { + "ceiling": { "type": "string", - "format": "date-time", - "description": "When this check was first reported and this policy entry was\ncreated." - }, - "notes": { - "type": [ - "string", - "null" - ], - "description": "Free-form operator notes about this check." - }, - "pending_review": { - "type": "boolean", - "description": "`true` if no operator has reviewed this policy yet." - }, - "reviewed_at": { - "type": [ - "string", - "null" - ], - "format": "date-time", - "description": "When an operator last reviewed or updated this policy. `null` if\nit has never been reviewed." - }, - "reviewed_by": { - "type": [ - "string", - "null" - ], - "description": "The operator who last reviewed this policy. `null` if it has\nnever been reviewed." - }, - "rule_count": { - "type": "integer", - "format": "int32", - "description": "Number of condition/severity branches in `rules`; `0` when\n`rules` is `null` or couldn't be parsed. Lets a caller tell\nwhether conditional rules exist without parsing `rules` itself.", - "minimum": 0 - }, - "rules": { - "description": "Conditional rules that can assign a different severity than\n`severity`, depending on the failing check's own fields, the\nsurrounding status report, or the reporting server's tags. `null`\nmeans no conditional rules are configured, and `severity` always\napplies.\n\nWhen present, this is a single-key object shaped like\n`{\"if\": [condition_1, severity_1, condition_2, severity_2, ...]}`.\nConditions are tried in order, and the severity paired with the\nfirst matching condition is used; if none match, the base\n`severity` is used instead. There's no explicit \"else\" branch —\nthe fallback to `severity` plays that role — so the array must\nhave an even number of entries and at least one pair.\n\nEach condition is a single-key object naming a comparison\noperator — one of `==`, `!=`, `<`, `<=`, `>`, `>=`, or `in_range`\n— whose value is a two-element array: a variable reference and a\nvalue to compare it against. A variable reference has the shape\n`{\"var\": \".\"}`, where `` is one of\n`check` (a field on the failing check itself), `status` (a\ntop-level field on the status report that contained it), or\n`tag` (a tag on the reporting server, merged with its group's\ntags). `in_range` compares a version-like string against a\nsemantic version range (e.g. `\">=1.2.0 <2.0.0\"`). If the named\nvariable isn't present in the data being evaluated, the condition\ndoesn't match. A `rules` value that doesn't parse into this shape\nis treated the same as `null` (no conditional rules)." - }, - "severity": { - "$ref": "#/components/schemas/Severity", - "description": "The severity assigned to a failure of this check when no\nconditional rule (see `rules`) overrides it." + "description": "The ceiling to apply to this check's observed results when no\nconditional rule overrides it: one of `failed`, `warning`,\n`passed`, or `skipped`." }, - "updated_at": { - "type": "string", - "format": "date-time", - "description": "When this policy was last modified." - } - } - }, - "HealthcheckUpdateArgs": { - "type": "object", - "description": "Request body for updating a healthcheck's base severity policy.", - "required": [ - "check_name", - "severity" - ], - "properties": { "check_name": { "type": "string", "description": "The healthcheck name to update; must already exist in the\ncatalog." }, + "escalates": { + "type": "boolean", + "description": "Whether an effective failure of this check should notify\nimmediately, bypassing the incident grace period." + }, "notes": { "type": [ "string", "null" ], - "description": "Operator notes to store alongside the new severity. Omitting this\nor sending `null` clears any existing notes — there's no way to\nleave them unchanged implicitly, so resend the current value to\nkeep it." + "description": "Operator notes to store alongside the new policy. Omitting this\nor sending `null` clears any existing notes — there's no way to\nleave them unchanged implicitly, so resend the current value to\nkeep it." }, - "severity": { - "$ref": "#/components/schemas/Severity", - "description": "The severity to assign to this check's failures when no\nconditional rule overrides it." + "source": { + "type": "string", + "description": "The source whose check to update." } } }, @@ -12441,8 +12461,9 @@ }, "UpdateRulesArgs": { "type": "object", - "description": "Request body for replacing a healthcheck's conditional severity rules.", + "description": "Request body for replacing a check's conditional rules.", "required": [ + "source", "check_name" ], "properties": { @@ -12451,7 +12472,11 @@ "description": "The healthcheck name whose rules to replace; must already exist\nin the catalog." }, "rules": { - "description": "The new conditional rules to store, or `null` to remove all\nconditional rules and rely solely on the base severity. Same\nshape as the `rules` field returned when listing checks. A ladder\nwith no condition/severity pairs is treated the same as `null`." + "description": "The new conditional rules to store, or `null` to remove all\nconditional rules and rely solely on the ceiling. Same shape as\nthe `rules` field returned when listing checks. A ladder with no\ncondition/result pairs is treated the same as `null`." + }, + "source": { + "type": "string", + "description": "The source whose check's rules to replace." } } }, diff --git a/private-web/src/api-types.ts b/private-web/src/api-types.ts index 141f846e..77269e7f 100644 --- a/private-web/src/api-types.ts +++ b/private-web/src/api-types.ts @@ -1265,10 +1265,10 @@ export interface paths { get?: never; put?: never; /** - * List the healthcheck severity catalog. - * @description Returns every known healthcheck name together with its current - * severity policy, ordered by name. An entry exists for every check name - * any server has ever reported; new checks are added automatically the + * List the check policy catalog. + * @description Returns every known (source, check) together with its current policy, + * ordered by source then name. An entry exists for every check any + * source has ever reported; new checks are added automatically the * first time they're seen, with a default policy pending review. */ post: operations["healthcheck_list"]; @@ -1337,11 +1337,11 @@ export interface paths { get?: never; put?: never; /** - * Update a healthcheck's severity policy. - * @description Sets the base severity (and optionally notes) for the given check, and - * marks it as reviewed by the caller. Saving with the same severity as - * before still counts as a review, so an operator can acknowledge a - * check without changing its policy. + * Update a check's policy. + * @description Sets the ceiling and escalation flag (and optionally notes) for the + * given (source, check), and marks it as reviewed by the caller. Saving + * with the same values as before still counts as a review, so an + * operator can acknowledge a check without changing its policy. */ post: operations["healthcheck_update"]; delete?: never; @@ -1360,12 +1360,12 @@ export interface paths { get?: never; put?: never; /** - * Replace a healthcheck's conditional severity rules. - * @description Stores a new set of conditional rules for the given check (or removes - * them, if `rules` is `null`), and marks the check as reviewed by the - * caller. Returns 400 if `rules` doesn't parse as a valid ladder — for - * example an unknown comparison operator, a malformed variable - * reference, or an odd number of entries. + * Replace a check's conditional rules. + * @description Stores a new set of conditional rules for the given (source, check) + * (or removes them, if `rules` is `null`), and marks the check as + * reviewed by the caller. Returns 400 if `rules` doesn't parse as a + * valid ladder — for example an unknown comparison operator, a + * malformed variable reference, or an odd number of entries. */ post: operations["healthcheck_update_rules"]; delete?: never; @@ -3644,16 +3644,27 @@ export interface components { check: string; }; /** - * @description Response for [`check_attention`]: the queried check's catalog severity + * @description Response for [`check_attention`]: the queried check's catalog policy * (if it has one yet) and every live server whose latest status reports * it, failing or healthy. */ CheckAttentionData: { + /** + * @description The most urgent configured policy ceiling for this check across + * the sources that report it, or `None` if no server has ever + * reported it yet (so it has no catalog row). + */ + ceiling?: string | null; /** * @description The check name that was queried, echoed back so the page can * render its heading without re-decoding the request. */ check: string; + /** + * @description Whether any source's policy for this check escalates its + * effective failures. + */ + escalates: boolean; /** * @description Every live server whose latest status reports this check, at any * result, ordered as a TODO list: failed, warning, broken, passed, @@ -3662,7 +3673,6 @@ export interface components { * healthy" toggle is on. */ servers: components["schemas"]["CheckAttentionServerData"][]; - severity?: null | components["schemas"]["Severity"]; }; /** * @description One server whose latest status reports [`CheckAttentionData::check`], @@ -3712,6 +3722,93 @@ export interface components { */ status_created_at: string; }; + /** + * @description One (source, check)'s policy: the ceiling capping its effective + * result, the escalation flag, plus optional conditional rules that can + * grade a given report differently. + */ + CheckPolicyData: { + /** + * @description The maximum effective result for this check when no conditional + * rule (see `rules`) overrides it: an observed result more urgent + * than the ceiling grades down to it. One of `failed`, `warning`, + * `passed`, or `skipped` (`skipped` also tells the reporting source + * it may stop running the check). + */ + ceiling: string; + /** @description The healthcheck's name, exactly as reported by monitored servers. */ + check_name: string; + /** + * @description Whether an effective failure of this check notifies immediately, + * bypassing the incident grace period. + */ + escalates: boolean; + /** + * Format: date-time + * @description When this check was first reported and this policy entry was + * created. + */ + first_seen: string; + /** @description Free-form operator notes about this check. */ + notes?: string | null; + /** @description `true` if no operator has reviewed this policy yet. */ + pending_review: boolean; + /** + * Format: date-time + * @description When an operator last reviewed or updated this policy. `null` if + * it has never been reviewed. + */ + reviewed_at?: string | null; + /** + * @description The operator who last reviewed this policy. `null` if it has + * never been reviewed. + */ + reviewed_by?: string | null; + /** + * Format: int32 + * @description Number of condition/result branches in `rules`; `0` when + * `rules` is `null` or couldn't be parsed. Lets a caller tell + * whether conditional rules exist without parsing `rules` itself. + */ + rule_count: number; + /** + * @description Conditional rules that can grade a report to a different result + * than the ceiling would — in any direction — depending on the + * check's own fields, the surrounding status report, or the + * reporting server's tags. `null` means no conditional rules are + * configured, and the ceiling always applies. + * + * When present, this is a single-key object shaped like + * `{"if": [condition_1, result_1, condition_2, result_2, ...]}`. + * Conditions are tried in order, and the result paired with the + * first matching condition is used; if none match, the observed + * result capped at the ceiling is used instead. There's no explicit + * "else" branch — the ceiling fallback plays that role — so the + * array must have an even number of entries and at least one pair. + * + * Each condition is a single-key object naming a comparison + * operator — one of `==`, `!=`, `<`, `<=`, `>`, `>=`, or `in_range` + * — whose value is a two-element array: a variable reference and a + * value to compare it against. A variable reference has the shape + * `{"var": "."}`, where `` is one of + * `check` (a field on the failing check itself), `status` (a + * top-level field on the status report that contained it), or + * `tag` (a tag on the reporting server, merged with its group's + * tags). `in_range` compares a version-like string against a + * semantic version range (e.g. `">=1.2.0 <2.0.0"`). If the named + * variable isn't present in the data being evaluated, the condition + * doesn't match. A `rules` value that doesn't parse into this shape + * is treated the same as `null` (no conditional rules). + */ + rules?: unknown; + /** @description The source that reports this check. */ + source: string; + /** + * Format: date-time + * @description When this policy was last modified. + */ + updated_at: string; + }; /** * @description Outcome of a single health check reported in a server's status update. * @@ -4239,102 +4336,33 @@ export interface components { check_name: string; sample?: null | components["schemas"]["HealthcheckSample"]; }; - /** - * @description A named healthcheck's alerting policy: the base severity assigned to - * its failures, plus optional conditional rules that can override that - * severity based on the details of a given failure. - */ - HealthcheckSeverityData: { - /** @description The healthcheck's name, exactly as reported by monitored servers. */ - check_name: string; - /** - * Format: date-time - * @description When this check was first reported and this policy entry was - * created. - */ - first_seen: string; - /** @description Free-form operator notes about this check. */ - notes?: string | null; - /** @description `true` if no operator has reviewed this policy yet. */ - pending_review: boolean; - /** - * Format: date-time - * @description When an operator last reviewed or updated this policy. `null` if - * it has never been reviewed. - */ - reviewed_at?: string | null; - /** - * @description The operator who last reviewed this policy. `null` if it has - * never been reviewed. - */ - reviewed_by?: string | null; - /** - * Format: int32 - * @description Number of condition/severity branches in `rules`; `0` when - * `rules` is `null` or couldn't be parsed. Lets a caller tell - * whether conditional rules exist without parsing `rules` itself. - */ - rule_count: number; - /** - * @description Conditional rules that can assign a different severity than - * `severity`, depending on the failing check's own fields, the - * surrounding status report, or the reporting server's tags. `null` - * means no conditional rules are configured, and `severity` always - * applies. - * - * When present, this is a single-key object shaped like - * `{"if": [condition_1, severity_1, condition_2, severity_2, ...]}`. - * Conditions are tried in order, and the severity paired with the - * first matching condition is used; if none match, the base - * `severity` is used instead. There's no explicit "else" branch — - * the fallback to `severity` plays that role — so the array must - * have an even number of entries and at least one pair. - * - * Each condition is a single-key object naming a comparison - * operator — one of `==`, `!=`, `<`, `<=`, `>`, `>=`, or `in_range` - * — whose value is a two-element array: a variable reference and a - * value to compare it against. A variable reference has the shape - * `{"var": "."}`, where `` is one of - * `check` (a field on the failing check itself), `status` (a - * top-level field on the status report that contained it), or - * `tag` (a tag on the reporting server, merged with its group's - * tags). `in_range` compares a version-like string against a - * semantic version range (e.g. `">=1.2.0 <2.0.0"`). If the named - * variable isn't present in the data being evaluated, the condition - * doesn't match. A `rules` value that doesn't parse into this shape - * is treated the same as `null` (no conditional rules). - */ - rules?: unknown; - /** - * @description The severity assigned to a failure of this check when no - * conditional rule (see `rules`) overrides it. - */ - severity: components["schemas"]["Severity"]; + /** @description Request body for updating a check's base policy. */ + HealthcheckUpdateArgs: { /** - * Format: date-time - * @description When this policy was last modified. + * @description The ceiling to apply to this check's observed results when no + * conditional rule overrides it: one of `failed`, `warning`, + * `passed`, or `skipped`. */ - updated_at: string; - }; - /** @description Request body for updating a healthcheck's base severity policy. */ - HealthcheckUpdateArgs: { + ceiling: string; /** * @description The healthcheck name to update; must already exist in the * catalog. */ check_name: string; /** - * @description Operator notes to store alongside the new severity. Omitting this + * @description Whether an effective failure of this check should notify + * immediately, bypassing the incident grace period. + */ + escalates?: boolean; + /** + * @description Operator notes to store alongside the new policy. Omitting this * or sending `null` clears any existing notes — there's no way to * leave them unchanged implicitly, so resend the current value to * keep it. */ notes?: string | null; - /** - * @description The severity to assign to this check's failures when no - * conditional rule overrides it. - */ - severity: components["schemas"]["Severity"]; + /** @description The source whose check to update. */ + source: string; }; /** @description Pagination parameters for browsing the shared query history. */ HistoryArgs: { @@ -7150,7 +7178,7 @@ export interface components { /** @description New display name for the key, or null to clear it. */ name?: string | null; }; - /** @description Request body for replacing a healthcheck's conditional severity rules. */ + /** @description Request body for replacing a check's conditional rules. */ UpdateRulesArgs: { /** * @description The healthcheck name whose rules to replace; must already exist @@ -7159,11 +7187,13 @@ export interface components { check_name: string; /** * @description The new conditional rules to store, or `null` to remove all - * conditional rules and rely solely on the base severity. Same - * shape as the `rules` field returned when listing checks. A ladder - * with no condition/severity pairs is treated the same as `null`. + * conditional rules and rely solely on the ceiling. Same shape as + * the `rules` field returned when listing checks. A ladder with no + * condition/result pairs is treated the same as `null`. */ rules?: unknown; + /** @description The source whose check's rules to replace. */ + source: string; }; /** @description Identifies a version and the publication status to set on it. */ UpdateStatusArgs: { @@ -9054,13 +9084,13 @@ export interface operations { }; requestBody?: never; responses: { - /** @description Catalog rows ordered by check_name. */ + /** @description Catalog rows ordered by source then check_name. */ 200: { headers: { [name: string]: unknown; }; content: { - "application/json": components["schemas"]["HealthcheckSeverityData"][]; + "application/json": components["schemas"]["CheckPolicyData"][]; }; }; 401: { @@ -9176,7 +9206,7 @@ export interface operations { [name: string]: unknown; }; content: { - "application/json": components["schemas"]["HealthcheckSeverityData"]; + "application/json": components["schemas"]["CheckPolicyData"]; }; }; 401: { @@ -9216,7 +9246,7 @@ export interface operations { [name: string]: unknown; }; content: { - "application/json": components["schemas"]["HealthcheckSeverityData"]; + "application/json": components["schemas"]["CheckPolicyData"]; }; }; 400: { diff --git a/private-web/src/components/CheckResultChip.tsx b/private-web/src/components/CheckResultChip.tsx new file mode 100644 index 00000000..e071a888 --- /dev/null +++ b/private-web/src/components/CheckResultChip.tsx @@ -0,0 +1,28 @@ +import { Chip, Tooltip } from "@mui/material"; +import { CHECK_RESULT_INTENT, type CheckResult } from "../types"; + +type Color = "error" | "warning" | "success" | "default"; + +/// Broken reads as a warning since it says nothing about the system +/// under test, just the check itself; passed/skipped read calm. +const COLOR: Record = { + failed: "error", + warning: "warning", + broken: "warning", + passed: "success", + skipped: "default", +}; + +export default function CheckResultChip({ + result, + variant, +}: { + result: CheckResult; + variant?: "filled" | "outlined"; +}) { + return ( + + + + ); +} diff --git a/private-web/src/lib/healthcheck-rule-eval.ts b/private-web/src/lib/healthcheck-rule-eval.ts index 643fe5a3..df2ce4f7 100644 --- a/private-web/src/lib/healthcheck-rule-eval.ts +++ b/private-web/src/lib/healthcheck-rule-eval.ts @@ -1,5 +1,5 @@ // Client-side mirror of the Rust IfLadder evaluator (see -// crates/database/src/healthcheck_severities.rs). Used by the rule +// crates/database/src/check_policies.rs). Used by the rule // editor to preview whether a candidate rule would match a sampled // status push. // diff --git a/private-web/src/routes/HealthcheckAttention.tsx b/private-web/src/routes/HealthcheckAttention.tsx index 709ef01d..0338a4c2 100644 --- a/private-web/src/routes/HealthcheckAttention.tsx +++ b/private-web/src/routes/HealthcheckAttention.tsx @@ -23,27 +23,13 @@ import { useState } from "react"; import { Link as RouterLink, useParams } from "react-router-dom"; import { useApi } from "../api"; import CheckExtrasList, { checkEntryExtras } from "../components/CheckExtras"; +import CheckResultChip from "../components/CheckResultChip"; import ServerNameWithGroup from "../components/ServerNameWithGroup"; -import SeverityChip from "../components/SeverityChip"; import TimeAgo from "../components/TimeAgo"; import { usePageTitle } from "../hooks/usePageTitle"; import { useReloadInterval } from "../hooks/useReloadInterval"; import { type CheckAttentionServerData, type CheckResult } from "../types"; -/// MUI chip colour per check result. Broken reads as a warning since it -/// says nothing about the system under test, just the check itself; -/// passed/skipped (visible behind the "show healthy" toggle) read calm. -const CHECK_CHIP_COLOR: Record< - CheckResult, - "error" | "warning" | "success" | "default" -> = { - failed: "error", - warning: "warning", - broken: "warning", - passed: "success", - skipped: "default", -}; - const HEALTHY_RESULTS: readonly string[] = ["passed", "skipped"]; /// Dedicated page for a single healthcheck: every live server whose @@ -75,8 +61,17 @@ export default function HealthcheckAttention() { {check} - {result.status === "ok" && result.data.severity && ( - + {result.status === "ok" && result.data.ceiling && ( + + )} + {result.status === "ok" && result.data.escalates && ( + )} @@ -85,7 +80,7 @@ export default function HealthcheckAttention() { component={RouterLink} to={`/settings/healthchecks/${encodeURIComponent(check ?? "")}`} > - Configure severity / rules + Configure ceiling / rules @@ -202,11 +197,9 @@ function AttentionRow({ server }: { server: CheckAttentionServerData }) { /> - diff --git a/private-web/src/routes/HealthcheckDetail.tsx b/private-web/src/routes/HealthcheckDetail.tsx index cad6f367..e747043f 100644 --- a/private-web/src/routes/HealthcheckDetail.tsx +++ b/private-web/src/routes/HealthcheckDetail.tsx @@ -8,6 +8,7 @@ import { DialogActions, DialogContent, DialogTitle, + FormControlLabel, IconButton, InputAdornment, LinearProgress, @@ -15,6 +16,7 @@ import { Paper, Select, Stack, + Switch, Table, TableBody, TableCell, @@ -33,23 +35,26 @@ import WarningAmberIcon from "@mui/icons-material/WarningAmber"; import { useMemo, useState } from "react"; import { Link as RouterLink, useParams } from "react-router-dom"; import { ApiError, useApi, useApiAction } from "../api"; -import SeverityChip from "../components/SeverityChip"; +import CheckResultChip from "../components/CheckResultChip"; import TimeAgo from "../components/TimeAgo"; import { useIsAdmin } from "../hooks/useIsAdmin"; import { usePageTitle } from "../hooks/usePageTitle"; import { evaluate as evalCondition } from "../lib/healthcheck-rule-eval"; import { - SEVERITIES, - SEVERITY_INTENT, + CEILINGS, + CEILING_INTENT, + CHECK_RESULT_INTENT, + CHECK_RESULT_ORDER, healthcheckPath, + type Ceiling, + type CheckPolicyData, + type CheckResult, type HealthcheckSample, - type HealthcheckSeverityData, - type Severity, } from "../types"; // ── Constrained JsonLogic shape ─────────────────────────────────────────── // -// The Rust deserialiser only accepts {if: [c1, s1, c2, s2, …]} ladders +// The Rust deserialiser only accepts {if: [c1, r1, c2, r2, …]} ladders // where each condition is {: [{var: ""}, ]}. The // helpers below parse the raw JSON returned by the API into the typed // shape this page edits, and serialize it back for /update_rules. @@ -71,7 +76,7 @@ interface Branch { varPath: string; op: RuleOp; value: unknown; - severity: Severity; + result: CheckResult; } function parseRules(raw: unknown): { branches: Branch[]; error: string | null } { @@ -90,20 +95,23 @@ function parseRules(raw: unknown): { branches: Branch[]; error: string | null } const branches: Branch[] = []; for (let i = 0; i < args.length; i += 2) { const condRaw = args[i]; - const sevRaw = args[i + 1]; + const resultRaw = args[i + 1]; const cond = parseCondition(condRaw); if (cond.error) return { branches: [], error: `branch ${i / 2}: ${cond.error}` }; - if (typeof sevRaw !== "string" || !(SEVERITIES as readonly string[]).includes(sevRaw)) { + if ( + typeof resultRaw !== "string" || + !(CHECK_RESULT_ORDER as readonly string[]).includes(resultRaw) + ) { return { branches: [], - error: `branch ${i / 2}: invalid severity '${String(sevRaw)}'`, + error: `branch ${i / 2}: invalid result '${String(resultRaw)}'`, }; } branches.push({ varPath: cond.varPath, op: cond.op, value: cond.value, - severity: sevRaw as Severity, + result: resultRaw as CheckResult, }); } return { branches, error: null }; @@ -149,7 +157,7 @@ function serializeRules(branches: Branch[]): unknown | null { const args: unknown[] = []; for (const b of branches) { args.push({ [b.op]: [{ var: b.varPath }, b.value] }); - args.push(b.severity); + args.push(b.result); } return { if: args }; } @@ -185,8 +193,8 @@ export default function HealthcheckDetail() { const isAdmin = useIsAdmin() === true; const list = useApi("healthchecks", "list"); - const row: HealthcheckSeverityData | undefined = - list.status === "ok" ? list.data.find((r) => r.check_name === checkName) : undefined; + const rows: CheckPolicyData[] = + list.status === "ok" ? list.data.filter((r) => r.check_name === checkName) : []; return ( @@ -208,24 +216,31 @@ export default function HealthcheckDetail() { ) : list.status === "error" ? ( {list.error.message} - ) : row == null ? ( + ) : rows.length === 0 ? ( No healthcheck named {checkName} in the catalog yet — it'll appear here once a server reports it. ) : ( - <> - - - - - + rows.map((row) => ( + + {rows.length > 1 && ( + + source: {row.source} + + )} + + + + + + )) )} ); } -function RowMetadata({ row }: { row: HealthcheckSeverityData }) { +function RowMetadata({ row }: { row: CheckPolicyData }) { return ( void; }) { const update = useApiAction("healthchecks", "update"); - const [severity, setSeverity] = useState(row.severity); + const [ceiling, setCeiling] = useState(row.ceiling as Ceiling); + const [escalates, setEscalates] = useState(row.escalates); const save = async () => { try { - await update.call({ check_name: row.check_name, severity, notes: row.notes }); + await update.call({ + source: row.source, + check_name: row.check_name, + ceiling, + escalates, + notes: row.notes, + }); onChanged(); } catch { - setSeverity(row.severity); + setCeiling(row.ceiling as Ceiling); + setEscalates(row.escalates); } }; return ( - Base severity + Ceiling - Used when this check fails and no rule below matches. + The maximum effective result for this check when no rule below + matches: anything more urgent grades down to it. {canEdit ? ( ) : ( - + )} + setEscalates(e.target.checked)} + disabled={!canEdit || update.pending} + /> + } + label="Escalates: an effective failure notifies immediately, bypassing the incident grace period" + slotProps={{ typography: { variant: "caption" } }} + /> {canEdit && (