diff --git a/.workhorse/specs/monitoring/checks.md b/.workhorse/specs/monitoring/checks.md new file mode 100644 index 00000000..df0dff76 --- /dev/null +++ b/.workhorse/specs/monitoring/checks.md @@ -0,0 +1,119 @@ +--- +id: CHK +--- + +# Check state + +Canopy's monitoring is organised around checks: named conditions, each with a current result, reported by sources or determined by Canopy itself. +This spec covers the check-state model — targets, sources, results, policy, and the operator controls over them. +How device reports arrive is the status contract (see [STA](../public-server/statuses.md)); how degraded checks aggregate into incidents is the incident spec (see [INC](incidents.md)). + +## Targets + +Every check is scoped to exactly one target: a server, a server group, or Canopy as a whole. + +Server checks come from sources reporting on that server, and from Canopy's own per-server determinations such as source staleness. +Group checks are conditions Canopy determines about a group's control plane, such as backup maintenance health (see [BKJ](../jobs/backup.md)). +Canopy-wide checks are Canopy monitoring its own operation (see [SELF](../private-server/self-alerts.md)). + +## Sources + +A source is a named reporter of checks, identified by a short string. +Multiple sources may report on the same server, each concerned with part of the system, and each source's reports are independent: a report from one source says nothing about another source's checks. + +Two source names are reserved for Canopy itself: `canopy` for conditions Canopy determines on its own (staleness, reachability, backup health, key expiry, self-monitoring), and `manual` for conditions raised by operators. +Reports arriving over the device API cannot use the reserved names. + +## Results + +A check's result is one of, in decreasing order of urgency: + +- **failed** — the condition is failing. +- **warning** — the condition is degraded but not failing. +- **broken** — the check itself could not run; the condition is unconfirmed either way. +- **passed** — the condition holds. +- **skipped** — the check deliberately did not run. + +Every check has two results: the **observed** result, what the source reported, and the **effective** result, what policy makes of it. +The observed result is always recorded as reported; everything Canopy acts on — issues, incidents, health rollups — follows the effective result. + +## Policy + +Policy is a transformation of results: for each check it maps the observed result to the effective one. +There is one vocabulary on both sides — policy speaks in results, and what a source is told about its checks is the policy itself, not a projection of it. + +Fleet-wide policy lives in a catalog keyed by (source, check). +An entry carries: + +- a **ceiling** — the maximum effective result, on the urgency ordering: a ceiling of `failed` changes nothing, `warning` grades failures as warnings, `passed` means recorded but never alerting, and `skipped` additionally tells the source not to bother running the check. +- optional **rules** — conditional transforms evaluated against the check's own detail, the report's server-wide detail, and the server's effective tags; a rule can move a result in any direction, including upward: a warning graded as a failure, or a pass with a particular detail graded as a warning. +- an **escalates** flag — an effective failure of this check notifies immediately, bypassing incident grace (see [INC](incidents.md)). + +A check is registered in the catalog with a ceiling of `warning` the first time it is reported; operators adjust from there. +Canopy's own checks register with the policy their condition warrants instead of the default. + +### Scoped policy + +Beyond the fleet catalog, a transform can be scoped to a target: per server, per group, or Canopy-wide. +Transforms apply in order — fleet catalog, then group, then server — each acting on the previous effective result, so the most specific scope has the last word. + +The operator interface presents one scoped policy: the **silence**, a scoped ceiling of `skipped`, recording who silenced and when. +A silenced check keeps recording its observed results; its effective result is skipped, so it raises nothing and counts nowhere. +The model admits arbitrary scoped transforms; surfaces beyond the silence are deliberately not offered yet. + +## Documentation + +Each catalogued (source, check) can carry operator-authored documentation: a single markdown document. +By convention it covers three things — a general description of what the check observes, what each result means (what makes it fail as opposed to warn), and hints for solving a failure — and the editor seeds new documentation with a template of those sections; Canopy attaches no meaning to the document's structure. +Operators author and edit the documentation in the operator UI; it is presented alongside the check wherever its state is presented, and is available over the MCP interface (see [MCP](../private-server/mcp.md)) so agents work from curated knowledge about a check rather than deriving it. +Canopy's own checks ship with their documentation. + +## State + +For each (target, source, check) Canopy keeps exactly one state: the observed and effective results, the detail the source attached to the check's most recent report, when the check was first and most recently reported, and — while it is degraded — when the current degradation began. +All reported checks are kept, including passing ones, so that "every server reporting this check" is answerable without scanning history. + +A state whose effective result is warning or failed is an **issue**, eligible to contribute to incidents. +"Degraded since" is the start of the current unbroken run of degradation; a recovery ends the run, and a later degradation starts a fresh one. + +An effective broken result neither confirms nor clears the check's previous definite result: while broken, the state retains the contribution and degraded-since of its last definite effective result, and the brokenness itself additionally counts as a warning. +A policy rule can grade brokenness differently — up to a failure where not being able to check is itself the failure, or down to a pass where a flaky check runner should not raise noise. +A definite effective result (passed, warning, or failed) ends the broken condition and replaces the retained contribution. + +## Reporting semantics + +A source's report for a server carries that source's complete current set of checks. +Canopy trusts the reporter: a check the source previously reported but omits from its current report has recovered, and its state records that. +Omission by one source never affects another source's checks. + +## Source staleness + +A source that has reported on a server is expected to keep reporting. +When a source's most recent report for a server is older than the server's down threshold, Canopy raises a staleness check for that (server, source) under the `canopy` source, and clears it when the source reports again. +A server all of whose sources are stale is presented as unreachable. + +## Health rollup + +A server's health is derived from its current effective results across all sources: any failure makes it unhealthy; otherwise any warning or brokenness makes it degraded; otherwise it is healthy. +Passed and skipped checks do not count against a server. + +## Operator controls + +**Silences** are the scoped policy described above. + +**Snoozes** suppress one state until a chosen time, after which it contributes again if still degraded. + +**Resolution** is an operator marking one state as dealt with, recording who and why. +A resolved state that degrades again reopens: the resolution is cleared and the state contributes anew. + +**Notes** attach free-form operator commentary to a state. + +## Manual conditions + +Operators can raise a condition directly against a server, under the `manual` source, with a chosen check name, result, and message, and optionally marked as escalating. +A manual condition behaves as a reported check whose reporter is the operator: it stays active until an operator resolves it or raises it again as recovered. + +## Monitoring gate + +Server-targeted checks on a server that is not monitored are recorded and presented for visibility but do not contribute to incidents. +Group and Canopy-wide checks are not subject to any server's monitoring gate. diff --git a/.workhorse/specs/monitoring/incidents.md b/.workhorse/specs/monitoring/incidents.md new file mode 100644 index 00000000..a6c8ce07 --- /dev/null +++ b/.workhorse/specs/monitoring/incidents.md @@ -0,0 +1,43 @@ +--- +id: INC +--- + +# Incidents + +An incident is a span of trouble on a target: a server group, or Canopy as a whole. +It aggregates the issues active on that target over its lifetime, from when it opens until it closes or an operator resolves it. +At most one incident is open per target at a time. + +Issues and effective results are defined by the check-state model (see [CHK](checks.md)). +Issues on a server belong to the target of the server's group; issues on an ungrouped server belong to no target and cannot contribute to incidents. +Group-targeted issues belong to that group's target; Canopy-wide issues belong to the Canopy target. + +## Membership + +An incident opens when a check's effective result becomes failed on a target with no open incident. +While an incident is open, every issue on its target joins it — effective warnings included — so the incident carries the full context of what was wrong during its span. + +An issue leaves the incident when it stops being one: its effective result recovers (to passed or skipped, whether by report or by policy), it is resolved or snoozed, or its server stops being monitored. +The incident closes when its last effective failure leaves; warnings never hold an incident open. + +The membership history — which issues joined and left, and when — is kept and presented as the incident's timeline. +An issue can leave and rejoin the same incident. + +Operator actions that change what counts (monitoring toggles, group membership changes, policy and silence changes) re-evaluate the affected issues' incident membership. + +## Notification + +Operators are notified over the notification channel: group incidents to the group's configured channel, Canopy-wide incidents to the operator channel. + +An incident notifies when it has stayed open past its target's grace period; an incident that closes within grace never notifies. +Whether an incident notified is recorded as its **published** flag, so flaps can be excluded from reporting. +An escalating check's effective failure (see [CHK](checks.md), "Policy") notifies immediately, bypassing any remaining grace; if the incident has already notified, the join escalates it with a further notification, at most once per incident. +A notified incident notifies again when it closes. + +## Resolution + +An operator can resolve an open incident, recording who and why. +Resolution cascades to the incident's open issues — each is resolved with the same attribution — and the incident closes as its members leave. +Unresolving clears the resolution record; it does not reopen the incident. + +Notes attach free-form operator commentary to an incident. diff --git a/.workhorse/specs/private-server/mcp.md b/.workhorse/specs/private-server/mcp.md index 0fb8be01..1a247fc0 100644 --- a/.workhorse/specs/private-server/mcp.md +++ b/.workhorse/specs/private-server/mcp.md @@ -74,26 +74,28 @@ When the result is truncated to its bound, the result says so, so the client doe **Fleet summary** takes no input and returns a fleet-wide overview: server counts by kind and rank, the distribution of deployed versions, a rollup of server health, the number of groups, and a rollup of backup health. -**Find backup problems** optionally narrows to one group, otherwise scans the whole fleet, and returns the current backup problems with a severity for each: server-and-type pairs whose last successful backup is overdue against its schedule, types that have never reported a backup, groups whose backup repository is in an error state, recent failed backup runs, and maintenance runs that appear stuck. +**Find backup problems** optionally narrows to one group, otherwise scans the whole fleet, and returns the current backup problems graded by urgency: server-and-type pairs whose last successful backup is overdue against its schedule, types that have never reported a backup, groups whose backup repository is in an error state, recent failed backup runs, and maintenance runs that appear stuck. ### Incidents and issues -An issue is a per-server (or per-group) condition raised from a known source under a stable reference, carrying a severity, a current active state, and a history of events; an incident aggregates the issues active for a group over a span of time, from when it opened until it closes or an operator resolves it. +An issue is a degraded check state (see [CHK](../monitoring/checks.md)): a condition on a server, a group, or Canopy itself, reported by a known source under a stable check name, carrying its observed and effective results and a current active state; an incident aggregates the issues active on a target over a span of time, from when it opened until it closes or an operator resolves it (see [INC](../monitoring/incidents.md)). **Find incidents** takes a look-back window (in days, defaulting to a week) and optionally one group, and returns the incidents that were open at any point within that window — those still open, plus those that closed no earlier than the window start. -Each is returned with its group, its status (open, closed, or operator-resolved), when it opened, closed, and was resolved, who resolved it and why, whether it ever escalated, how long it was open, whether it was published, and how many issues and events it covers. +Each is returned with its target, its status (open, closed, or operator-resolved), when it opened, closed, and was resolved, who resolved it and why, whether it ever escalated, how long it was open, whether it was published, and how many issues it covers. A status filter can narrow the result to only open or only resolved incidents. The window includes incidents that flapped open and shut within their group's grace period and so never surfaced to anyone. -Each incident therefore carries a **published** flag — true when it actually notified operators, which happens only when it stayed open past its group's grace period or it escalated (a critical issue joined, which bypasses the grace) — and the result reports how many of the returned incidents were published. -The raw event count an incident accumulated is not a measure of its duration or severity: a high count can belong to a sub-minute flap. +Each incident therefore carries a **published** flag — true when it actually notified operators, which happens only when it stayed open past its target's grace period or it escalated (an escalating check's failure joined, which bypasses the grace) — and the result reports how many of the returned incidents were published. A summary or ranking of incidents should count published incidents rather than raw rows unless raw activity is explicitly wanted. -**Get incident** takes an incident identifier and returns the incident with the issues attached to it: each issue's severity, source, reference, message, owning server, active state, and when it joined and (if applicable) left the incident. +**Get incident** takes an incident identifier and returns the incident with the issues attached to it: each issue's effective result, source, check name, message, owning server, active state, and when it joined and (if applicable) left the incident. -**Find issues** returns issues across the fleet, filtered by active state, by severity, by group, by server, and by recency (issues last seen within a look-back window). +**Find issues** returns issues across the fleet, filtered by active state, by effective result, by group, by server, and by recency (issues last seen within a look-back window). -**Get issue** takes an issue identifier and returns the issue with its recent events and the incidents it is or was part of. +**Get issue** takes an issue identifier and returns the issue with the incidents it is or was part of. + +**Get check documentation** takes a source and check name and returns the check's operator-authored markdown documentation (see [CHK](../monitoring/checks.md), "Documentation"), which by convention covers what the check observes, what each result means, and how to solve a failure. +A client investigating an issue consults this before deriving a check's meaning from other sources. ## Result semantics diff --git a/.workhorse/specs/private-server/self-alerts.md b/.workhorse/specs/private-server/self-alerts.md index 2c527e6d..abdae740 100644 --- a/.workhorse/specs/private-server/self-alerts.md +++ b/.workhorse/specs/private-server/self-alerts.md @@ -5,28 +5,26 @@ id: SELF # Self-alerts A self-alert is Canopy reporting a problem with its own operation, as distinct from an issue observed on a fleet member. +Self-alerts are Canopy-wide checks (see [CHK](../monitoring/checks.md)): they carry the same state, policy, silences, and resolution as any other check, and they aggregate into incidents on the Canopy target (see [INC](../monitoring/incidents.md)). ## Conditions -Each self-alert condition is identified by a stable reference, and at most one alert exists per condition: repeated detections coalesce into the one alert rather than accumulating. -An alert is active while its condition holds, and recovers when the condition clears; a condition without automatic recovery stays active until an operator resolves it. +Each self-alert condition is a check with a stable name under the `canopy` source, and at most one state exists per condition: repeated detections update the one state rather than accumulating. +A condition is active while it holds, and recovers when it clears; a condition without automatic recovery stays active until an operator resolves it. The current conditions are: -- Canopy's own identity for reaching backup storage is broken (critical). +- Canopy's own identity for reaching backup storage is broken (escalating). - An operator-notification delivery has permanently failed (stays until operator-resolved). - An MCP access token is within fifteen days of its expiry (see [MCP](mcp.md), "Access tokens"). ## Notification -An active self-alert notifies operators over the operator-notification channel directly: it does not open an incident and does not attach to any fleet group. -One notification is sent when an alert becomes active, and one when it recovers. -Alerts below critical severity wait out a short grace period before notifying, and an alert that recovers within that grace sends nothing at all; critical alerts notify immediately. -While an alert stays active, repeated detections send no further notifications. -When the notification channel is not configured for self-alerts, they are still recorded and presented, and the operator is warned that notification is off. +Self-alerts notify through the incident machinery: an effective failure opens an incident on the Canopy target, which notifies the operator channel per the incident notification rules (grace period, escalation, recovery notice). +When the notification channel is not configured, self-alerts are still recorded and presented, and the operator is warned that notification is off. ## Presentation Active self-alerts are presented on their own surface in the operator UI, apart from fleet issues and incidents: a persistent notice visible from any page while any alert is active, leading to a view of the active alerts and recent recoveries. Self-alerts do not appear in the fleet issue listings, and are not presented as belonging to any server. -Each alert presents its severity, when it became active, and a description of the condition and what to do about it. +Each alert presents its effective result, when it became active, and a description of the condition and what to do about it. diff --git a/.workhorse/specs/public-server/statuses.md b/.workhorse/specs/public-server/statuses.md new file mode 100644 index 00000000..d37f60e3 --- /dev/null +++ b/.workhorse/specs/public-server/statuses.md @@ -0,0 +1,42 @@ +--- +id: STA +--- + +# Status reporting + +Devices report on their server by pushing statuses to the device API. +A status is one source's complete current picture of the server: its health checks and their results, plus server-wide metadata. +What Canopy does with the checks is the check-state model (see [CHK](../monitoring/checks.md)). + +## Push + +A status is pushed to the server it describes; the caller must present the server's enrolled device certificate, or a certificate holding the admin role. + +The payload carries: + +- **source** — the name of the reporter pushing this status. + Transitionally optional: a push without a source is attributed to `alertd`. + The field will become mandatory; new reporters must send it. + The reserved source names (see [CHK](../monitoring/checks.md), "Sources") are rejected. +- **health** — the source's complete set of checks: for each, the check's name, exactly one result (`passed`, `warning`, `failed`, `broken`, or `skipped`), and any further detail fields, which are recorded verbatim against the check. + The set may be empty, meaning the source currently has no checks — which recovers every check it previously reported. +- any further top-level fields, recorded verbatim as the status's server-wide detail. + +The server's application version is taken from the server-wide detail, falling back to the pushing client's version header. + +Per the check-state model, the push is the source's whole truth: checks omitted from it are recovered, and other sources' checks are unaffected. + +Every push is recorded in full as the server's status history. + +## Legacy pushes + +A push without a health field is a legacy report from a Tamanu server. +It is recorded with its metadata like any other status, and is treated as the source `tamanu` reporting a single check `tasks` as passed — a liveness heartbeat that participates in source staleness like any source. + +## Response + +The response to a push carries only what the pushing source needs; a source is sent nothing meant for another source, and relies on receiving nothing beyond its own concerns. + +- Each check in the push is answered with the policy applied to it (see [CHK](../monitoring/checks.md), "Policy"), so a source sees how its reports are graded and can stop running checks whose policy is `skipped`. +- Whether the server should start a backup now is returned only to the source that runs backups (`alertd`). +- The server's effective tags, a server-wide fact, are returned to every source. diff --git a/AGENTS.md b/AGENTS.md index 042dc6b7..de1a99b4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -126,8 +126,7 @@ just the `psql` client. On macOS there's no `/dev/shm`, so it falls back to disk `CANOPY_TEST_PG_ROLE`. To run against your **system** Postgres instead (to inspect the DB afterwards, -or where `initdb` isn't available), use `just test-system [nextest args]` — -prefix with `nice` to soften the I/O grind. +or where `initdb` isn't available), use `just test-system [nextest args]`. ## Version Control - If the working copy is a jujutsu repo (a `.jj` directory exists at the repo root), prefer `jj` commands over `git` for VCS operations (status, diff, log, commit/describe, etc.). The repo may be colocated with git, but `jj` is the source of truth for local work. diff --git a/ERRORS.md b/ERRORS.md index b20489df..8d41e67e 100644 --- a/ERRORS.md +++ b/ERRORS.md @@ -64,14 +64,9 @@ Issued when authentication fails for unspecified reasons. ## Device has no server -Issued when a device tries to submit an event but is not registered against -any server. Devices must be linked to a server (`servers.device_id`) before -they can report issues. - -## Source manual forbidden - -Issued when an event submission to the public API uses `source = "manual"`. -That source is reserved for operator-submitted events via the private API. +Issued when a device calls an endpoint that acts on "its" server but is not +registered against any server. Devices must be linked to a server +(`servers.device_id`) first. ## Auth: tailnet directory unavailable diff --git a/crates/canopy-mcp/src/fleet.rs b/crates/canopy-mcp/src/fleet.rs index 1db85f18..f7502a46 100644 --- a/crates/canopy-mcp/src/fleet.rs +++ b/crates/canopy-mcp/src/fleet.rs @@ -108,6 +108,11 @@ impl CanopyMcp { .await .map_err(mcp_err)?; let st_by: HashMap = statuses.iter().map(|s| (s.server_id, s)).collect(); + let server_groups: Vec<(Uuid, Option)> = + servers.iter().map(|s| (s.id, s.group_id)).collect(); + let state_health = database::issues::health_from_check_state(&mut conn, &server_groups) + .await + .map_err(mcp_err)?; let mut counts = Counts::default(); let mut health = HealthRollup::default(); @@ -120,7 +125,7 @@ impl CanopyMcp { let st = st_by.get(&s.id).copied(); match st.map_or(ShortStatus::Gone, |s| s.short_status()) { ShortStatus::Down | ShortStatus::Gone => health.unreachable += 1, - _ => match st.map_or(HealthState::default(), |s| s.health_state()) { + _ => match state_health.get(&s.id).copied().unwrap_or_default() { HealthState::Healthy => health.healthy += 1, HealthState::Warning => health.warning += 1, HealthState::Unhealthy => health.unhealthy += 1, diff --git a/crates/canopy-mcp/src/groups.rs b/crates/canopy-mcp/src/groups.rs index 763a7c59..97723325 100644 --- a/crates/canopy-mcp/src/groups.rs +++ b/crates/canopy-mcp/src/groups.rs @@ -164,11 +164,9 @@ impl CanopyMcp { let st_by: HashMap = statuses.iter().map(|s| (s.server_id, s)).collect(); let member_groups: Vec<(Uuid, Option)> = members.iter().map(|s| (s.id, s.group_id)).collect(); - let silenced = - database::silenced_refs::silenced_health_checks_for_servers(&mut conn, &member_groups) - .await - .map_err(mcp_err)?; - let no_silences = std::collections::BTreeSet::new(); + let health = database::issues::health_from_check_state(&mut conn, &member_groups) + .await + .map_err(mcp_err)?; let member_summaries = members .iter() .map(|s| { @@ -176,7 +174,7 @@ impl CanopyMcp { s, st_by.get(&s.id).copied(), Some(group.name.clone()), - silenced.get(&s.id).unwrap_or(&no_silences), + health.get(&s.id).copied().unwrap_or_default(), ) }) .collect(); diff --git a/crates/canopy-mcp/src/incidents.rs b/crates/canopy-mcp/src/incidents.rs index 5f51bf5e..56a7992a 100644 --- a/crates/canopy-mcp/src/incidents.rs +++ b/crates/canopy-mcp/src/incidents.rs @@ -1,8 +1,8 @@ //! `find_incidents` / `get_incident` / `find_issues` / `get_issue` tools. -use commons_types::{Uuid, issue::Severity}; +use commons_types::{Uuid, status::CheckResult}; use database::{ - issues::{Event, Incident, Issue, IssueListFilters}, + issues::{Incident, Issue, IssueListFilters}, server_groups::ServerGroup, servers::Server, slack_outbox::SlackOutbox, @@ -48,8 +48,9 @@ pub struct IncidentIdArgs { pub struct FindIssuesArgs { /// Only currently-active, unresolved issues. Default true. pub active_only: Option, - /// Filter to these severities: `critical`, `error`, `warning`, `info`, `debug`. - pub severities: Option>, + /// Filter to issues whose latest effective result is one of these: + /// `failed`, `warning`, `broken`, `passed`, `skipped`. + pub results: Option>, /// Restrict to issues whose server is in this group's id. pub group_id: Option, /// Restrict to one server's id. @@ -60,6 +61,25 @@ pub struct FindIssuesArgs { pub limit: Option, } +#[derive(Debug, Deserialize, JsonSchema)] +pub struct CheckDocArgs { + /// The source that reports the check (e.g. `alertd`, `canopy`). + pub source: String, + /// The check's name. + pub check_name: String, +} + +#[derive(Serialize)] +struct CheckDocOut { + source: String, + check_name: String, + ceiling: CheckResult, + escalates: bool, + /// Operator-authored markdown, or `null` if nobody has documented + /// this check yet. + documentation: Option, +} + #[derive(Debug, Deserialize, JsonSchema)] pub struct IssueIdArgs { /// The issue's id. @@ -69,7 +89,9 @@ pub struct IssueIdArgs { #[derive(Serialize)] struct IncidentSummary { id: Uuid, - group_id: Uuid, + /// The server group the incident targets, or `null` for a canopy-wide + /// incident (aggregating canopy's self-alerts). + group_id: Option, group_name: Option, /// `open` (not closed), `resolved` (operator-resolved), or `closed`. status: &'static str, @@ -88,9 +110,6 @@ struct IncidentSummary { /// How long the incident was (or has been) open, in seconds. open_duration_secs: i64, issue_count: i64, - /// Raw count of status events the incident accumulated. NOT a measure of - /// duration or severity — a high count can be a sub-minute flap. - event_count: i64, } #[derive(Serialize)] @@ -105,7 +124,13 @@ struct IncidentList { #[derive(Serialize)] struct IncidentIssueOut { issue_id: Uuid, - severity: Severity, + /// What the source reported on the latest filing, before policy. + observed_result: Option, + /// What policy made of it — the result canopy acts on. + effective_result: Option, + /// Whether the check's policy escalates (an effective failure + /// notifies immediately, bypassing incident grace). + escalates: bool, source: String, r#ref: String, description: Option, @@ -123,7 +148,9 @@ struct IncidentIssueOut { #[derive(Serialize)] struct IncidentDetail { id: Uuid, - group_id: Uuid, + /// The server group the incident targets, or `null` for a canopy-wide + /// incident (aggregating canopy's self-alerts). + group_id: Option, group_name: Option, status: &'static str, opened_at: Timestamp, @@ -148,7 +175,9 @@ struct IssueSummary { group_id: Option, source: String, r#ref: String, - severity: Severity, + observed_result: Option, + effective_result: Option, + escalates: bool, description: Option, message: String, active: bool, @@ -164,18 +193,6 @@ struct IssueList { issues: Vec, } -#[derive(Serialize)] -struct EventOut { - created_at: Timestamp, - occurred_at: Option, - severity: Severity, - description: Option, - message: String, - active: bool, - occurrences: i32, - last_seen: Timestamp, -} - #[derive(Serialize)] struct IncidentRefOut { incident_id: Uuid, @@ -191,7 +208,9 @@ struct IssueDetail { group_id: Option, source: String, r#ref: String, - severity: Severity, + observed_result: Option, + effective_result: Option, + escalates: bool, description: Option, message: String, active: bool, @@ -201,7 +220,6 @@ struct IssueDetail { resolved_by: Option, resolved_reason: Option, snoozed_until: Option, - recent_events: Vec, incidents: Vec, } @@ -217,11 +235,10 @@ impl CanopyMcp { but never surfaced to anyone. An incident is `published` only if its Slack \ open notice was delivered: it stayed open past the group's grace window \ (slack_open_delay, ~3 min by default) OR it escalated (a critical issue \ - joined, which bypasses the grace). `event_count` is raw status-event churn \ - and does NOT track duration or severity — a high-event incident can be a \ - sub-minute flap. A high count dominated by unpublished short-lived rows \ - usually means a twitchy alert/health-check threshold, not a real outage. \ - `published_count` gives the surfaced subset directly." + joined, which bypasses the grace). A count dominated by unpublished \ + short-lived incidents usually means a twitchy alert/health-check \ + threshold, not a real outage. `published_count` gives the surfaced \ + subset directly." )] async fn find_incidents( &self, @@ -246,7 +263,7 @@ impl CanopyMcp { let group_names = group_names( &mut conn, - &unique(incidents.iter().map(|i| i.server_group_id)), + &unique(incidents.iter().filter_map(|i| i.server_group_id)), ) .await?; let ids: Vec = incidents.iter().map(|i| i.id).collect(); @@ -262,7 +279,9 @@ impl CanopyMcp { IncidentSummary { id: i.id, group_id: i.server_group_id, - group_name: group_names.get(&i.server_group_id).cloned(), + group_name: i + .server_group_id + .and_then(|gid| group_names.get(&gid).cloned()), status: incident_status(i), opened_at: i.opened_at, closed_at: i.closed_at, @@ -273,7 +292,6 @@ impl CanopyMcp { published: published.contains(&i.id), open_duration_secs: open_duration_secs(i), issue_count: s.map_or(0, |s| s.issue_count), - event_count: s.map_or(0, |s| s.event_count), } }) .collect(); @@ -299,9 +317,10 @@ impl CanopyMcp { let Ok((incident, rows)) = Incident::get_with_issues(&mut conn, id).await else { return Ok(not_found(format!("no incident with id {id}"))); }; - let group = ServerGroup::get_by_id(&mut conn, incident.server_group_id) - .await - .ok(); + let group = match incident.server_group_id { + Some(gid) => ServerGroup::get_by_id(&mut conn, gid).await.ok(), + None => None, + }; let published = SlackOutbox::delivered_open_ids(&mut conn, &[incident.id]) .await .map_err(mcp_err)? @@ -317,7 +336,9 @@ impl CanopyMcp { .iter() .map(|(link, iss)| IncidentIssueOut { issue_id: iss.id, - severity: iss.severity, + observed_result: iss.observed_result, + effective_result: iss.effective_result, + escalates: iss.escalates, source: iss.source.clone(), r#ref: iss.r#ref.clone(), description: iss.description.clone(), @@ -355,16 +376,16 @@ impl CanopyMcp { } #[tool( - description = "List issues across the fleet, filtered by active state, severity, group, \ - server, and recency. Issues are the per-(server,source,ref) events that make \ - up incidents." + description = "List issues across the fleet, filtered by active state, effective result, \ + group, server, and recency. Issues are the per-(server,source,check) conditions \ + that make up incidents." )] async fn find_issues( &self, Parameters(args): Parameters, ) -> Result { let mut conn = self.conn().await?; - let severities = parse_severities(&args.severities)?; + let results = parse_results(&args.results)?; let group = parse_opt_uuid(&args.group_id, "group_id")?; let server = parse_opt_uuid(&args.server_id, "server_id")?; let since = args.since_days.map(since_from_days); @@ -374,7 +395,7 @@ impl CanopyMcp { &mut conn, IssueListFilters { active_only: args.active_only.unwrap_or(true), - severities, + results, server_group_id: group, since, }, @@ -401,8 +422,7 @@ impl CanopyMcp { } #[tool( - description = "Full detail for one issue: its fields, recent events, and the incidents it \ - is or was part of." + description = "Full detail for one issue: its fields and the incidents it is or was part of." )] async fn get_issue( &self, @@ -413,9 +433,6 @@ impl CanopyMcp { let Ok(issue) = Issue::get_by_id(&mut conn, id).await else { return Ok(not_found(format!("no issue with id {id}"))); }; - let events = Event::list_for_issue(&mut conn, id, 0, 20) - .await - .map_err(mcp_err)?; let inc = Incident::for_issues(&mut conn, &[id]) .await .map_err(mcp_err)?; @@ -428,19 +445,6 @@ impl CanopyMcp { None => None, }; - let recent_events = events - .iter() - .map(|e| EventOut { - created_at: e.created_at, - occurred_at: e.occurred_at, - severity: e.severity, - description: e.description.clone(), - message: e.message.clone(), - active: e.active, - occurrences: e.occurrences, - last_seen: e.last_seen, - }) - .collect(); let incidents = inc .get(&id) .into_iter() @@ -459,7 +463,9 @@ impl CanopyMcp { group_id: issue.server_group_id, source: issue.source.clone(), r#ref: issue.r#ref.clone(), - severity: issue.severity, + observed_result: issue.observed_result, + effective_result: issue.effective_result, + escalates: issue.escalates, description: issue.description.clone(), message: issue.message.clone(), active: issue.active, @@ -469,10 +475,39 @@ impl CanopyMcp { resolved_by: issue.resolved_by.clone(), resolved_reason: issue.resolved_reason.clone(), snoozed_until: issue.snoozed_until, - recent_events, incidents, }) } + + #[tool( + description = "Get the operator-authored documentation for a (source, check): what the \ + check observes, what each result means, and hints for solving a failure. \ + Prefer this curated knowledge over inferring what a check does from its \ + name. Also returns the check's current policy (ceiling, escalates)." + )] + async fn get_check_documentation( + &self, + Parameters(args): Parameters, + ) -> Result { + use database::check_policies::CheckPolicy; + let mut conn = self.conn().await?; + let Some(policy) = CheckPolicy::get(&mut conn, &args.source, &args.check_name) + .await + .map_err(mcp_err)? + else { + return Ok(not_found(format!( + "no catalog entry for ({}, {}) — that source has never reported that check", + args.source, args.check_name + ))); + }; + ok_json(&CheckDocOut { + source: policy.source, + check_name: policy.check_name, + ceiling: policy.ceiling, + escalates: policy.escalates, + documentation: policy.documentation, + }) + } } /// How long the incident was (or has been) open, in seconds. @@ -491,14 +526,16 @@ fn incident_status(i: &Incident) -> &'static str { } } -fn parse_severities(v: &Option>) -> Result>, McpError> { +fn parse_results(v: &Option>) -> Result>, McpError> { match v { Some(list) if !list.is_empty() => { let mut out = Vec::with_capacity(list.len()); for s in list { - out.push(s.parse::().map_err(|_| { - McpError::invalid_params(format!("invalid severity: {s}"), None) - })?); + out.push( + s.parse::().map_err(|_| { + McpError::invalid_params(format!("invalid result: {s}"), None) + })?, + ); } Ok(Some(out)) } @@ -520,7 +557,9 @@ fn issue_summary( group_id: i.server_group_id, source: i.source.clone(), r#ref: i.r#ref.clone(), - severity: i.severity, + observed_result: i.observed_result, + effective_result: i.effective_result, + escalates: i.escalates, description: i.description.clone(), message: i.message.clone(), active: i.active, diff --git a/crates/canopy-mcp/src/lib.rs b/crates/canopy-mcp/src/lib.rs index 8a699792..55c90775 100644 --- a/crates/canopy-mcp/src/lib.rs +++ b/crates/canopy-mcp/src/lib.rs @@ -78,8 +78,8 @@ impl ServerHandler for CanopyMcp { flapping that was recorded but never surfaced. When summarizing or ranking, count \ `published` incidents (also given as `published_count`), not raw rows: an incident is \ published only if it outlived the group's grace window (~3 min default) or escalated. \ - `event_count` is raw event churn, not duration or severity — a huge count can be a \ - sub-minute flap, usually a twitchy threshold rather than a real outage." + A count dominated by unpublished short-lived incidents usually means a twitchy \ + threshold rather than a real outage." .into(), ); info.capabilities = ServerCapabilities::builder().enable_tools().build(); diff --git a/crates/canopy-mcp/src/servers.rs b/crates/canopy-mcp/src/servers.rs index 362201bb..05d8d8f5 100644 --- a/crates/canopy-mcp/src/servers.rs +++ b/crates/canopy-mcp/src/servers.rs @@ -1,7 +1,7 @@ //! `find_servers` / `get_server` tools, and the [`ServerSummary`] shape //! reused by the groups module for member listings. -use std::collections::{BTreeSet, HashMap}; +use std::collections::HashMap; use commons_types::{ Uuid, @@ -169,11 +169,9 @@ impl CanopyMcp { .map_err(mcp_err)?; let server_groups: Vec<(Uuid, Option)> = servers.iter().map(|s| (s.id, s.group_id)).collect(); - let silenced = - database::silenced_refs::silenced_health_checks_for_servers(&mut conn, &server_groups) - .await - .map_err(mcp_err)?; - let no_silences = BTreeSet::new(); + let health = database::issues::health_from_check_state(&mut conn, &server_groups) + .await + .map_err(mcp_err)?; let summaries: Vec = servers .iter() @@ -182,7 +180,7 @@ impl CanopyMcp { s, st_by.get(&s.id).copied(), group_names.get(&s.id).cloned().flatten(), - silenced.get(&s.id).unwrap_or(&no_silences), + health.get(&s.id).copied().unwrap_or_default(), ) }) .collect(); @@ -243,20 +241,21 @@ impl CanopyMcp { }); } - // Operator-silenced healthchecks don't count toward the health - // rollup; `checks` still carries their raw results verbatim. - let silenced_checks = database::silenced_refs::silenced_health_checks_for_server( - &mut conn, - server.id, - server.group_id, - ) - .await - .map_err(mcp_err)?; + // Health rolls up current check state across every source + // (silenced checks skipped in the rollup); `checks` still carries + // the latest push's raw results verbatim. + let health = + database::issues::health_from_check_state(&mut conn, &[(server.id, server.group_id)]) + .await + .map_err(mcp_err)? + .get(&server.id) + .copied() + .unwrap_or_default(); let latest_status = latest.as_ref().map(|s| StatusOut { reported_at: s.created_at, version: version.clone(), - health: s.health_state_ignoring(&silenced_checks), + health, healthy: s.healthy, reachability: s.short_status(), platform: s.platform(), @@ -281,9 +280,7 @@ impl CanopyMcp { reachability: latest .as_ref() .map_or(ShortStatus::Gone, |s| s.short_status()), - health: latest.as_ref().map_or(HealthState::default(), |s| { - s.health_state_ignoring(&silenced_checks) - }), + health, latest_status, backups, }) @@ -291,13 +288,13 @@ impl CanopyMcp { } /// Shared with the groups module, for member listings on [`crate::groups::GroupDetail`]. -/// `silenced_checks` is the server's silenced healthcheck set (server + -/// group scope) — those checks don't count toward `health`. +/// `health` is the server's check-state rollup (silenced checks already +/// skipped). pub(crate) fn summarize( s: &Server, st: Option<&Status>, group_name: Option, - silenced_checks: &BTreeSet, + health: HealthState, ) -> ServerSummary { ServerSummary { id: s.id, @@ -312,9 +309,7 @@ pub(crate) fn summarize( last_seen: st.map(|s| s.created_at), version: st.and_then(|s| s.version.clone()), reachability: st.map_or(ShortStatus::Gone, |s| s.short_status()), - health: st.map_or(HealthState::default(), |s| { - s.health_state_ignoring(silenced_checks) - }), + health, } } diff --git a/crates/commons-errors/src/lib.rs b/crates/commons-errors/src/lib.rs index e80db2b1..7a836df7 100644 --- a/crates/commons-errors/src/lib.rs +++ b/crates/commons-errors/src/lib.rs @@ -93,9 +93,6 @@ pub enum AppError { #[error("device is not registered against any server")] DeviceHasNoServer, - #[error("source 'manual' is reserved for operator submissions")] - SourceManualForbidden, - #[error("tailnet directory is unavailable")] AuthTailnetDirectoryUnavailable, @@ -207,7 +204,6 @@ impl AppError { Self::AuthInsufficientPermissions { .. } => StatusCode::FORBIDDEN, Self::AuthFailed { .. } => StatusCode::UNAUTHORIZED, Self::DeviceHasNoServer => StatusCode::PRECONDITION_FAILED, - Self::SourceManualForbidden => StatusCode::BAD_REQUEST, Self::AuthTailnetDirectoryUnavailable => StatusCode::SERVICE_UNAVAILABLE, Self::AuthTailnetNodeNotPermitted => StatusCode::FORBIDDEN, Self::TaggedDeviceNotAllowed => StatusCode::FORBIDDEN, @@ -259,7 +255,6 @@ impl AppError { Self::AuthInsufficientPermissions { .. } => "auth-insufficient-permissions", Self::AuthFailed { .. } => "auth-failed", Self::DeviceHasNoServer => "device-has-no-server", - Self::SourceManualForbidden => "source-manual-forbidden", Self::AuthTailnetDirectoryUnavailable => "auth-tailnet-directory-unavailable", Self::AuthTailnetNodeNotPermitted => "auth-tailnet-node-not-permitted", diff --git a/crates/commons-servers/src/tailnet_sweeps.rs b/crates/commons-servers/src/tailnet_sweeps.rs index e9e44f3d..82703bb9 100644 --- a/crates/commons-servers/src/tailnet_sweeps.rs +++ b/crates/commons-servers/src/tailnet_sweeps.rs @@ -3,13 +3,12 @@ //! persisted device rows and files / closes the relevant issues. use commons_errors::Result; -use commons_types::{Uuid, issue::Severity}; +use commons_types::{Uuid, status::CheckResult}; use database::{ devices::Device, - issues::{Issue, NewEvent}, + issues::{CheckFiling, FilingScope, Issue, file_check}, }; use diesel_async::AsyncPgConnection; -use jiff::Timestamp; use crate::tailnet_directory::TailnetDirectory; @@ -20,9 +19,23 @@ pub const TAILSCALE_SOURCE: &str = "canopy"; /// `(server, device)` pair. pub const KEY_EXPIRY_REF: &str = "tailscale-key-expiry"; +pub const KEY_EXPIRY_DOC: &str = "## Description + +The server's Tailscale node key is nearing expiry (and key expiry isn't disabled for the node). When it lapses, the device drops off the tailnet and canopy loses its management path. + +## Results + +- **fail** — the node key expires within the alert lead. Escalates: an expired key severs connectivity. Recovers when the key is rotated or expiry is disabled. + +## Solve + +Re-authenticate the node (`tailscale up`) or disable key expiry for it in the Tailscale admin console."; + /// Sweep every tailnet-attached device that's wired to at least one -/// server, and file (or close) a Critical issue per `(server, device)` -/// pair based on the node's `keyExpiryDisabled`. +/// server, and file (or close) the key-expiry check per `(server, +/// device)` pair based on the node's `keyExpiryDisabled`. The check +/// registers as an escalating failure — losing contact with a node is +/// stop-the-world — but operators can regrade it from the catalog. /// /// Tailnet-attached devices with no server are intentionally skipped: /// the tailnet hosts plenty of nodes that aren't canopy-managed @@ -51,7 +64,6 @@ pub async fn sweep_key_expiry( .filter_map(|i| i.server_id.map(|sid| (sid, i))) .collect(); - let now = Timestamp::now(); let mut filed = 0usize; for (device, server_id, node_id) in &pairs { let entry = snapshot.get(node_id); @@ -65,39 +77,52 @@ pub async fn sweep_key_expiry( continue; }; - let event = match (entry.key_expiry_disabled, existing_issue) { + let (observed, title, message) = match (entry.key_expiry_disabled, existing_issue) { // Healthy state, no issue: nothing to do. (true, None) => continue, // Healthy state, issue already inactive: nothing to do. (true, Some(issue)) if !issue.active => continue, // Healthy state, but an active issue exists: close it. - (true, Some(_)) => NewEvent { - source: TAILSCALE_SOURCE.into(), - r#ref: KEY_EXPIRY_REF.into(), - severity: Some(Severity::Info), - description: None, - message: format!("Tailscale key expiry disabled for {}", entry.node_name,), - active: Some(false), - occurred_at: Some(now), - }, - // Unhealthy state: file or refresh a critical issue. - (false, _) => NewEvent { - source: TAILSCALE_SOURCE.into(), - r#ref: KEY_EXPIRY_REF.into(), - severity: Some(Severity::Critical), - description: Some(format!("Tailscale key will expire for {}", entry.node_name,)), - message: format!( + (true, Some(_)) => ( + CheckResult::Passed, + None, + format!("Tailscale key expiry disabled for {}", entry.node_name), + ), + // Unhealthy state: file or refresh the failure. + (false, _) => ( + CheckResult::Failed, + Some(format!("Tailscale key will expire for {}", entry.node_name)), + format!( "Tailnet node {} ({}) has key expiry enabled. When the \ node's key expires, it will drop off the tailnet and \ canopy will lose contact.", entry.node_name, entry.node_id, ), - active: Some(true), - occurred_at: Some(now), - }, + ), }; - event.save(db, *server_id, Some(device.id)).await?; + file_check( + db, + CheckFiling { + source: TAILSCALE_SOURCE, + scope: FilingScope::Server { + server_id: *server_id, + device_id: Some(device.id), + }, + check: KEY_EXPIRY_REF, + observed, + title: title.as_deref(), + message: &message, + detail: Some(serde_json::json!({ + "node_id": entry.node_id, + "node_name": entry.node_name, + })), + default_ceiling: CheckResult::Failed, + default_escalates: true, + documentation: Some(KEY_EXPIRY_DOC), + }, + ) + .await?; filed += 1; } diff --git a/crates/commons-types/src/issue.rs b/crates/commons-types/src/issue.rs index 2da55534..140844f0 100644 --- a/crates/commons-types/src/issue.rs +++ b/crates/commons-types/src/issue.rs @@ -7,130 +7,6 @@ use diesel::{ }; use serde::{Deserialize, Serialize}; -/// Canopy's severity vocabulary, narrowed from RFC 5424 to a five-level -/// set with operator semantics: -/// -/// - `Debug`: never participates in incidents (filed for the audit -/// trail / per-server view only). -/// - `Info`, `Warning`: join an open incident for context but don't -/// open one or keep one open on their own. -/// - `Error`: opens an incident; sits in the per-group `slack_open_delay` -/// holding window before the Slack notification ships. -/// - `Critical`: opens an incident and bypasses the holding window — -/// the Slack notification is enqueued for immediate delivery. -/// -/// Stored as text in Postgres; validated as this enum at the API layer. -/// Default is `Error` (the most common severity for a deliberately -/// filed event). The legacy syslog severities `emergency` / `alert` / -/// `notice` have been retired — see the -/// `2026-05-29-000000-0000_restrict_severities` migration; the -/// `FromStr` impl still accepts them as aliases for forward-compat with -/// any device that hasn't been updated. -#[derive( - Debug, - Clone, - Copy, - Default, - PartialEq, - Eq, - Serialize, - Deserialize, - AsExpression, - utoipa::ToSchema, -)] -#[diesel(sql_type = Text)] -#[serde(rename_all = "lowercase")] -pub enum Severity { - Critical, - #[default] - Error, - Warning, - Info, - Debug, -} - -impl Severity { - /// Severities that may open an incident on their own and may keep one - /// open. Lower-severity issues (warning and below) join an already-open - /// incident for context but don't hold it open — see - /// `database::issues::re_evaluate_incident_membership`. - pub const OPENS_INCIDENT: &'static [Severity] = &[Self::Critical, Self::Error]; - - /// Issues at or above this severity open incidents. - pub fn opens_incident(self) -> bool { - Self::OPENS_INCIDENT.contains(&self) - } -} - -#[derive(Debug, Clone, Copy, thiserror::Error)] -#[error("invalid severity; expected one of: critical, error, warning, info, debug")] -pub struct SeverityFromStringError; - -impl std::str::FromStr for Severity { - type Err = SeverityFromStringError; - - fn from_str(s: &str) -> Result { - match s.to_ascii_lowercase().as_ref() { - // Retired severities still parse — emergency/alert collapse into - // critical and notice collapses into info, matching the migration. - "emergency" | "emerg" | "panic" | "alert" | "critical" | "crit" => Ok(Self::Critical), - "error" | "err" => Ok(Self::Error), - "warning" | "warn" => Ok(Self::Warning), - "notice" | "info" | "informational" => Ok(Self::Info), - "debug" => Ok(Self::Debug), - _ => Err(SeverityFromStringError), - } - } -} - -impl TryFrom for Severity { - type Error = SeverityFromStringError; - - fn try_from(value: String) -> Result { - value.parse() - } -} - -impl std::fmt::Display for Severity { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - let s = match self { - Self::Critical => "critical", - Self::Error => "error", - Self::Warning => "warning", - Self::Info => "info", - Self::Debug => "debug", - }; - write!(f, "{}", s) - } -} - -impl From for String { - fn from(s: Severity) -> Self { - s.to_string() - } -} - -impl FromSql for Severity -where - DB: Backend, - String: FromSql, -{ - fn from_sql(bytes: DB::RawValue<'_>) -> deserialize::Result { - let s = String::from_sql(bytes)?; - Ok(Severity::try_from(s)?) - } -} - -impl ToSql for Severity -where - String: ToSql, -{ - fn to_sql<'b>(&'b self, out: &mut Output<'b, '_, diesel::pg::Pg>) -> serialize::Result { - let v = String::from(*self); - >::to_sql(&v, &mut out.reborrow()) - } -} - /// Reason an issue or incident was resolved by an operator. /// /// Stored as text in Postgres, validated as this enum at the API layer. diff --git a/crates/commons-types/src/status.rs b/crates/commons-types/src/status.rs index a2417fb3..703fc5ea 100644 --- a/crates/commons-types/src/status.rs +++ b/crates/commons-types/src/status.rs @@ -2,17 +2,14 @@ use std::fmt::Display; use serde::{Deserialize, Serialize}; -use crate::issue::Severity; - /// How a server should treat one of its healthchecks, distilled from -/// canopy's operator-side configuration (the severity catalog and the -/// silence lists) into a three-level device-facing vocabulary. +/// canopy's operator-side configuration (the policy catalog and the +/// silences) into a three-level device-facing vocabulary. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize, utoipa::ToSchema)] #[serde(rename_all = "lowercase")] pub enum CheckSeverity { /// This check's failures are ignored on the canopy side — it is - /// silenced for this server, or its severity is classified below - /// warning. + /// silenced for this server, or its policy grades it below warning. Skip, /// This check's failures are treated as warnings. Warn, @@ -20,12 +17,16 @@ pub enum CheckSeverity { Fail, } -impl From for CheckSeverity { - fn from(severity: Severity) -> Self { - match severity { - Severity::Critical | Severity::Error => Self::Fail, - Severity::Warning => Self::Warn, - Severity::Info | Severity::Debug => Self::Skip, +impl From for CheckSeverity { + /// Distill a policy ceiling into the device-facing vocabulary: a + /// `failed` ceiling means failures count as failures, `warning` and + /// `broken` cap below that, and `passed`/`skipped` mean the check + /// never alerts (so the device may skip it). + fn from(ceiling: CheckResult) -> Self { + match ceiling { + CheckResult::Failed => Self::Fail, + CheckResult::Warning | CheckResult::Broken => Self::Warn, + CheckResult::Passed | CheckResult::Skipped => Self::Skip, } } } @@ -109,6 +110,32 @@ impl CheckResult { } } +impl CheckResult { + /// Position on the urgency ordering used by policy ceilings and + /// display sorting: failed > warning > broken > passed > skipped + /// (lower rank = more urgent). + pub fn urgency_rank(self) -> u8 { + match self { + CheckResult::Failed => 0, + CheckResult::Warning => 1, + CheckResult::Broken => 2, + CheckResult::Passed => 3, + CheckResult::Skipped => 4, + } + } + + /// Cap this result at `ceiling` on the urgency ordering: a result + /// more urgent than the ceiling grades down to it; anything at or + /// below the ceiling passes through unchanged. + pub fn capped_at(self, ceiling: CheckResult) -> CheckResult { + if self.urgency_rank() < ceiling.urgency_rank() { + ceiling + } else { + self + } + } +} + impl Display for CheckResult { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { match self { @@ -121,6 +148,20 @@ impl Display for CheckResult { } } +impl TryFrom for CheckResult { + type Error = String; + + fn try_from(value: String) -> Result { + value.parse() + } +} + +impl From for String { + fn from(result: CheckResult) -> Self { + result.to_string() + } +} + impl std::str::FromStr for CheckResult { type Err = String; diff --git a/crates/database/src/backup.rs b/crates/database/src/backup.rs index 563caf93..03dcdebc 100644 --- a/crates/database/src/backup.rs +++ b/crates/database/src/backup.rs @@ -1,19 +1,18 @@ //! Backup-credentials detection / alerting (DB-driven half of the control //! plane). The persistent models live in [`crate::backups`]; this module owns -//! the periodic *logic*: staleness, report-vs-inventory reconciliation, and -//! the shared group-level alerting entrypoint. +//! the periodic *logic*: staleness and report-vs-inventory reconciliation. //! //! - [`refs`] — the stable `(source, ref)` alert keys (a contract). -//! - [`alerts`] — [`alerts::raise_group_event`], the single group-scoped -//! incident entrypoint (bypasses per-server `is_monitored`). //! - [`staleness`] — staleness scan over reported runs + maintenance staleness. //! - [`reconcile`] — reconcile device reports against repo inventory. //! -//! The preflight (AWS-touching) lives in the `jobs` crate binary, not here — -//! the `database` crate must not gain an AWS dependency — but its *alerting* -//! reuses [`alerts::raise_group_event`]. +//! Alerting goes through [`crate::issues::file_check`], which grades +//! each observation through the operator's check policy and bypasses the +//! per-server `is_monitored` gate for group-scoped filings. The preflight +//! (AWS-touching) lives in the `jobs` crate binary, not here — the +//! `database` crate must not gain an AWS dependency — but its alerting uses +//! the same path. -pub mod alerts; pub mod reconcile; pub mod refs; pub mod staleness; diff --git a/crates/database/src/backup/alerts.rs b/crates/database/src/backup/alerts.rs deleted file mode 100644 index 105b3f5c..00000000 --- a/crates/database/src/backup/alerts.rs +++ /dev/null @@ -1,17 +0,0 @@ -//! Group-level alerting entrypoint for the backup-credentials system. -//! -//! [`raise_group_event`] is the single place that opens (or recovers) a -//! group-scoped incident, bypassing the per-server `is_monitored` gate. It is -//! consumed by: -//! - this crate's staleness/reconcile scan ([`crate::backup::staleness`], -//! [`crate::backup::reconcile`]), -//! - the **inspection Job** component for [`refs::CORRUPTION`], -//! - **PGRO ingest** (later) for [`refs::RESTORE_VERIFICATION`]. -//! -//! Owning it here means exactly one code path knows how to raise a group-level -//! issue without re-inheriting the monitored gate. - -pub use crate::backup::refs; -// Re-exported from `issues` (which owns the private incident plumbing) so all -// callers reach it via the backup module: `database::backup::alerts::raise_group_event`. -pub use crate::issues::raise_group_event; diff --git a/crates/database/src/backup/reconcile.rs b/crates/database/src/backup/reconcile.rs index c23c531d..68d0f4d7 100644 --- a/crates/database/src/backup/reconcile.rs +++ b/crates/database/src/backup/reconcile.rs @@ -30,15 +30,15 @@ use std::collections::HashMap; use commons_errors::Result; -use commons_types::{backup::BackupType, issue::Severity}; +use commons_types::{backup::BackupType, status::CheckResult}; use diesel::prelude::*; use diesel_async::{AsyncPgConnection, RunQueryDsl}; use jiff::{SignedDuration, Timestamp}; use uuid::Uuid; use crate::{ - backup::{alerts::raise_group_event, refs, staleness::ScanRow}, - issues::NewEvent, + backup::{refs, staleness::ScanRow}, + issues::{CheckFiling, FilingScope, file_check}, servers::Server, }; @@ -105,17 +105,26 @@ pub async fn sweep(db: &mut AsyncPgConnection, rows: &[ScanRow]) -> Result { - raise_group_event( + file_check( db, - row.group_id, - &missing_ref, - Severity::Error, - None, - &format!( - "Server {} reported a successful {} backup but no matching repo snapshot landed", - row.server_id, row.r#type, - ), - true, + CheckFiling { + source: crate::statuses::CANOPY_SOURCE, + scope: FilingScope::Group(row.group_id), + check: &missing_ref, + observed: CheckResult::Failed, + title: None, + message: &format!( + "Server {} reported a successful {} backup but no matching repo snapshot landed", + row.server_id, row.r#type, + ), + detail: Some(serde_json::json!({ + "type": row.r#type.to_string(), + "server_id": row.server_id, + })), + default_ceiling: CheckResult::Failed, + default_escalates: false, + documentation: Some(refs::RECONCILE_MISSING_DOC), + }, ) .await?; filed += 1; @@ -123,47 +132,63 @@ pub async fn sweep(db: &mut AsyncPgConnection, rows: &[ScanRow]) -> Result { if open_group_active(db, row.group_id, &missing_ref).await? { - raise_group_event( + file_check( db, - row.group_id, - &missing_ref, - Severity::Info, - None, - &format!( - "Server {} backup report and repo snapshot agree again", - row.server_id - ), - false, + CheckFiling { + source: crate::statuses::CANOPY_SOURCE, + scope: FilingScope::Group(row.group_id), + check: &missing_ref, + observed: CheckResult::Passed, + title: None, + message: &format!( + "Server {} backup report and repo snapshot agree again", + row.server_id + ), + detail: None, + default_ceiling: CheckResult::Failed, + default_escalates: false, + documentation: Some(refs::RECONCILE_MISSING_DOC), + }, ) .await?; filed += 1; } - filed += clear_report_gap(db, row, &gap_ref, now).await?; + filed += clear_report_gap(db, row, &gap_ref).await?; } // Snapshot landed but no recent report → the reporting path is - // broken. Per-server Warning (non-paging). + // broken. Per-server warning (non-paging). (false, true) => { let server = Server::get_by_id(db, row.server_id).await?; - NewEvent { - source: refs::CANOPY_SOURCE.into(), - r#ref: gap_ref, - severity: Some(Severity::Warning), - description: None, - message: format!( - "A fresh {} repo snapshot exists for {} but no backup run was reported", - row.r#type, server.id, - ), - active: Some(true), - occurred_at: Some(now), - } - .save(db, row.server_id, row.device_id) + file_check( + db, + CheckFiling { + source: crate::statuses::CANOPY_SOURCE, + scope: FilingScope::Server { + server_id: row.server_id, + device_id: row.device_id, + }, + check: &gap_ref, + observed: CheckResult::Warning, + title: None, + message: &format!( + "A fresh {} repo snapshot exists for {} but no backup run was reported", + row.r#type, server.id, + ), + detail: Some(serde_json::json!({ + "type": row.r#type.to_string(), + })), + default_ceiling: CheckResult::Warning, + default_escalates: false, + documentation: Some(refs::RECONCILE_REPORT_GAP_DOC), + }, + ) .await?; filed += 1; } // Neither fresh → the staleness scan owns it; clear stale reconcile // alerts. (false, false) => { - filed += clear_report_gap(db, row, &gap_ref, now).await?; + filed += clear_report_gap(db, row, &gap_ref).await?; } // (true, false) but the repo inventory is stale: skip the missing // verdict. @@ -176,25 +201,37 @@ pub async fn sweep(db: &mut AsyncPgConnection, rows: &[ScanRow]) -> Result { let server = Server::get_by_id(db, row.server_id).await?; - NewEvent { - source: refs::CANOPY_SOURCE.into(), - r#ref: size_ref, - severity: Some(Severity::Warning), - description: None, - message: format!( - "Server {} reported a {} snapshot size of {reported} bytes but the repo holds {observed}", - server.id, row.r#type, - ), - active: Some(true), - occurred_at: Some(now), - } - .save(db, row.server_id, row.device_id) + file_check( + db, + CheckFiling { + source: crate::statuses::CANOPY_SOURCE, + scope: FilingScope::Server { + server_id: row.server_id, + device_id: row.device_id, + }, + check: &size_ref, + observed: CheckResult::Warning, + title: None, + message: &format!( + "Server {} reported a {} snapshot size of {reported} bytes but the repo holds {observed}", + server.id, row.r#type, + ), + detail: Some(serde_json::json!({ + "type": row.r#type.to_string(), + "reported_bytes": reported, + "observed_bytes": observed, + })), + default_ceiling: CheckResult::Warning, + default_escalates: false, + documentation: Some(refs::RECONCILE_SIZE_MISMATCH_DOC), + }, + ) .await?; filed += 1; } // Agree, or no comparable run → clear any open mismatch. _ => { - filed += clear_size_mismatch(db, row, &size_ref, now).await?; + filed += clear_size_mismatch(db, row, &size_ref).await?; } } } @@ -207,24 +244,31 @@ async fn clear_size_mismatch( db: &mut AsyncPgConnection, row: &ScanRow, size_ref: &str, - now: Timestamp, ) -> Result { if !crate::backup::staleness::open_server_issue_active(db, row.server_id, size_ref).await? { return Ok(0); } - NewEvent { - source: refs::CANOPY_SOURCE.into(), - r#ref: size_ref.to_string(), - severity: Some(Severity::Info), - description: None, - message: format!( - "Reported and repo {} snapshot sizes for {} agree again", - row.r#type, row.server_id - ), - active: Some(false), - occurred_at: Some(now), - } - .save(db, row.server_id, row.device_id) + file_check( + db, + CheckFiling { + source: crate::statuses::CANOPY_SOURCE, + scope: FilingScope::Server { + server_id: row.server_id, + device_id: row.device_id, + }, + check: size_ref, + observed: CheckResult::Passed, + title: None, + message: &format!( + "Reported and repo {} snapshot sizes for {} agree again", + row.r#type, row.server_id + ), + detail: None, + default_ceiling: CheckResult::Warning, + default_escalates: false, + documentation: Some(refs::RECONCILE_SIZE_MISMATCH_DOC), + }, + ) .await?; Ok(1) } @@ -235,24 +279,31 @@ async fn clear_report_gap( db: &mut AsyncPgConnection, row: &ScanRow, gap_ref: &str, - now: Timestamp, ) -> Result { if !crate::backup::staleness::open_server_issue_active(db, row.server_id, gap_ref).await? { return Ok(0); } - NewEvent { - source: refs::CANOPY_SOURCE.into(), - r#ref: gap_ref.to_string(), - severity: Some(Severity::Info), - description: None, - message: format!( - "Backup reporting for {} ({}) recovered", - row.server_id, row.r#type - ), - active: Some(false), - occurred_at: Some(now), - } - .save(db, row.server_id, row.device_id) + file_check( + db, + CheckFiling { + source: crate::statuses::CANOPY_SOURCE, + scope: FilingScope::Server { + server_id: row.server_id, + device_id: row.device_id, + }, + check: gap_ref, + observed: CheckResult::Passed, + title: None, + message: &format!( + "Backup reporting for {} ({}) recovered", + row.server_id, row.r#type + ), + detail: None, + default_ceiling: CheckResult::Warning, + default_escalates: false, + documentation: Some(refs::RECONCILE_REPORT_GAP_DOC), + }, + ) .await?; Ok(1) } diff --git a/crates/database/src/backup/refs.rs b/crates/database/src/backup/refs.rs index 85bf1d84..1366b65c 100644 --- a/crates/database/src/backup/refs.rs +++ b/crates/database/src/backup/refs.rs @@ -49,8 +49,8 @@ pub const MAINTENANCE_ERROR: &str = "backup-maintenance-error"; pub const RECONCILE_MISSING: &str = "backup-reconcile-missing"; /// Repo corruption / poisoning detected by the inspection Job. Group-scoped, -/// `Critical`. Raised by the inspection-Job component via -/// [`crate::backup::alerts::raise_group_event`]; this constant is the contract. +/// registers as an escalating failure. Raised by the inspection-Job component +/// via [`crate::issues::file_check`]; this constant is the contract. pub const CORRUPTION: &str = "backup-corruption"; /// Canopy's own `sts:GetCallerIdentity` failed — the shared IRSA identity is @@ -71,3 +71,149 @@ pub const PREFLIGHT_OBJECT_LOCK: &str = "preflight-object-lock"; /// restorability check. Group-scoped, `Error`. Routed through the same /// group-level helper. pub const RESTORE_VERIFICATION: &str = "restore-verification"; + +// --- shipped documentation (seeded into the catalog on first filing) --- + +pub const STALENESS_DOC: &str = "## Description + +The server has backed this type up successfully before, but not recently: the latest success is older than twice the expected interval for the type. + +## Results + +- **fail** — no successful run within 2\u{d7} the expected interval; recovers on the next successful run. + +## Solve + +Check the device's bestool logs for failed or stuck runs, confirm the backup schedule is enabled for the group, and run the backup manually (`bestool canopy backup`) to see the error first-hand. Credential or bucket problems usually show as separate preflight checks."; + +pub const NEVER_DOC: &str = "## Description + +The server is expected to back this type up but has never reported a single successful run, and it has been enrolled long enough that one should have happened. + +## Results + +- **fail** — no success ever, past the enrolment/schedule grace; recovers on the first successful run. + +## Solve + +Confirm bestool is installed and enrolled on the server, the backup type is enabled for its group, and run `bestool canopy backup` manually to surface the error."; + +pub const RECONCILE_REPORT_GAP_DOC: &str = "## Description + +A fresh snapshot for this server exists in the repository, but no recent run report reached canopy — the backup works, the reporting path doesn't. + +## Results + +- **warn** — snapshot fresh, report missing; recovers when a run report arrives. + +## Solve + +Check the device's connectivity to canopy and its bestool logs for failed report submissions. The data is safe; only visibility is degraded."; + +pub const RECONCILE_SIZE_MISMATCH_DOC: &str = "## Description + +The device reported a snapshot size that disagrees with the size the same snapshot occupies in the repository. + +## Results + +- **warn** — sizes disagree (compared only when both are known and non-zero). + +## Solve + +Usually a reporting bug or a partially-uploaded snapshot. Compare the run report against `kopia snapshot list` for the source, and re-run the backup if the repo-side size looks truncated."; + +pub const MAINTENANCE_STALE_DOC: &str = "## Description + +The group's repository maintenance (compaction, blob GC) hasn't succeeded within its cadence. + +## Results + +- **fail** — last successful maintenance run is older than the cadence threshold; recovers on the next success. + +## Solve + +Check the maintenance job's logs for the group. A repository that misses maintenance grows unbounded but stays fully restorable."; + +pub const MAINTENANCE_ERROR_DOC: &str = "## Description + +The group's most recently finished repository maintenance run failed (distinct from maintenance being absent). + +## Results + +- **fail** — latest finished run errored; clears when a newer run finishes successfully. + +## Solve + +Read the run's error in the group's backups panel. Common causes: credential expiry mid-run and repository lock contention."; + +pub const RECONCILE_MISSING_DOC: &str = "## Description + +A run reported success but no matching snapshot landed in the repository — the device's report and the repo disagree. + +## Results + +- **fail** — reported snapshot absent from the repo. + +## Solve + +Treat the backup as not having happened. Check the device's kopia logs for upload failures after the snapshot was cut, and re-run the backup."; + +pub const CORRUPTION_DOC: &str = "## Description + +The repository inspection job detected corruption or poisoning in the group's backup repository. + +## Results + +- **fail** — inspection found corrupt or unreadable data. Escalates: this is a threat to restorability itself. + +## Solve + +Do not run maintenance (it may GC evidence). Inspect the repository with kopia directly, identify the damaged blobs and affected snapshots, and restore repository health from object-lock history if needed."; + +pub const PREFLIGHT_IDENTITY_DOC: &str = "## Description + +Canopy's own AWS identity (`sts:GetCallerIdentity` under the shared IRSA role) failed — no backup credential can be minted for anyone. + +## Results + +- **fail** — the identity call errors. Escalates: every backup and restore across the fleet is blocked. + +## Solve + +Check the canopy deployment's IRSA annotation and the AWS-side trust policy; recent cluster or account changes are the usual cause."; + +pub const PREFLIGHT_ASSUME_DOC: &str = "## Description + +Cross-account `AssumeRole` (or the read-only no-op S3 probe) failed for this group's backup or restore leg. + +## Results + +- **fail** — the group's role can't be assumed or can't reach its bucket. + +## Solve + +Check the group's target role ARN and its trust policy against canopy's identity, and the bucket policy on the target account."; + +pub const PREFLIGHT_OBJECT_LOCK_DOC: &str = "## Description + +The group's backup bucket has missing or weakened Object-Lock protection (mode absent, or retention under 30 days). + +## Results + +- **fail** — protection below the floor. Escalates: backups are no longer ransomware-resistant. + +## Solve + +Restore the bucket's Object-Lock configuration to GOVERNANCE mode with at least 30 days retention. Investigate who changed it — this setting should never weaken."; + +pub const RESTORE_VERIFICATION_DOC: &str = "## Description + +The managed restore replica for this (server, type, intent) reported a failed or stale restorability check. + +## Results + +- **fail** — the replica couldn't restore or verify the latest snapshot. + +## Solve + +Check the restore consumer's report detail: restore errors point at the snapshot or credentials, staleness at the consumer itself."; diff --git a/crates/database/src/backup/staleness.rs b/crates/database/src/backup/staleness.rs index c23a2be7..d3bde3e9 100644 --- a/crates/database/src/backup/staleness.rs +++ b/crates/database/src/backup/staleness.rs @@ -14,7 +14,7 @@ use std::collections::HashMap; use commons_errors::Result; use commons_types::{ backup::{BackupType, RunOutcome}, - issue::Severity, + status::CheckResult, }; use diesel::prelude::*; use diesel_async::{AsyncPgConnection, RunQueryDsl}; @@ -22,8 +22,8 @@ use jiff::{SignedDuration, Span, SpanRelativeTo, SpanRound, Timestamp, Unit}; use uuid::Uuid; use crate::{ - backup::{alerts::raise_group_event, refs}, - issues::NewEvent, + backup::refs, + issues::{CheckFiling, FilingScope, file_check}, servers::Server, }; @@ -236,40 +236,60 @@ pub async fn sweep(db: &mut AsyncPgConnection, rows: &[ScanRow]) -> Result { let grace = row.grace(); - NewEvent { - source: refs::CANOPY_SOURCE.into(), - r#ref: staleness_ref, - severity: Some(Severity::Error), - description: None, - message: format!( - "Server {label} has no successful {} backup newer than {} (last success {})", - row.r#type, - fmt_dur(grace), - row.last_success_at - .map(|t| t.to_string()) - .unwrap_or_else(|| "never".into()), - ), - active: Some(true), - occurred_at: Some(now), - } - .save(db, row.server_id, row.device_id) + file_check( + db, + CheckFiling { + source: crate::statuses::CANOPY_SOURCE, + scope: FilingScope::Server { + server_id: row.server_id, + device_id: row.device_id, + }, + check: &staleness_ref, + observed: CheckResult::Failed, + title: None, + message: &format!( + "Server {label} has no successful {} backup newer than {} (last success {})", + row.r#type, + fmt_dur(grace), + row.last_success_at + .map(|t| t.to_string()) + .unwrap_or_else(|| "never".into()), + ), + detail: Some(serde_json::json!({ + "type": row.r#type.to_string(), + "grace_secs": grace.as_secs(), + "last_success_at": row.last_success_at.map(|t| t.to_string()), + })), + default_ceiling: CheckResult::Failed, + default_escalates: false, + documentation: Some(refs::STALENESS_DOC), + }, + ) .await?; filed += 1; } StalenessVerdict::Recovered => { - NewEvent { - source: refs::CANOPY_SOURCE.into(), - r#ref: staleness_ref, - severity: Some(Severity::Info), - description: None, - message: format!( - "Server {label} reported a successful {} backup again", - row.r#type - ), - active: Some(false), - occurred_at: Some(now), - } - .save(db, row.server_id, row.device_id) + file_check( + db, + CheckFiling { + source: crate::statuses::CANOPY_SOURCE, + scope: FilingScope::Server { + server_id: row.server_id, + device_id: row.device_id, + }, + check: &staleness_ref, + observed: CheckResult::Passed, + title: None, + message: &format!( + "Server {label} reported a successful {} backup again", + row.r#type + ), + detail: None, + default_ceiling: CheckResult::Failed, + default_escalates: false, + documentation: Some(refs::STALENESS_DOC), + }, + ) .await?; filed += 1; } @@ -278,24 +298,36 @@ pub async fn sweep(db: &mut AsyncPgConnection, rows: &[ScanRow]) -> Result Result Result let was_active = open_group_issue_active(db, group_id, refs::MAINTENANCE_STALE).await?; if stale { - raise_group_event( + file_check( db, - group_id, - refs::MAINTENANCE_STALE, - Severity::Error, - None, - &format!( - "No successful repo maintenance for {} (last {})", - fmt_dur(MAINTENANCE_STALE_AFTER), - latest_success - .map(|t| Timestamp::from(t).to_string()) - .unwrap_or_else(|| "never".into()), - ), - true, + CheckFiling { + source: crate::statuses::CANOPY_SOURCE, + scope: FilingScope::Group(group_id), + check: refs::MAINTENANCE_STALE, + observed: CheckResult::Failed, + title: None, + message: &format!( + "No successful repo maintenance for {} (last {})", + fmt_dur(MAINTENANCE_STALE_AFTER), + latest_success + .map(|t| Timestamp::from(t).to_string()) + .unwrap_or_else(|| "never".into()), + ), + detail: Some(serde_json::json!({ + "threshold_secs": MAINTENANCE_STALE_AFTER.as_secs(), + "last_success_at": latest_success.map(|t| Timestamp::from(t).to_string()), + })), + default_ceiling: CheckResult::Failed, + default_escalates: false, + documentation: Some(refs::MAINTENANCE_STALE_DOC), + }, ) .await?; filed += 1; } else if !stale && was_active { - raise_group_event( + file_check( db, - group_id, - refs::MAINTENANCE_STALE, - Severity::Info, - None, - "Repo maintenance completed successfully again", - false, + CheckFiling { + source: crate::statuses::CANOPY_SOURCE, + scope: FilingScope::Group(group_id), + check: refs::MAINTENANCE_STALE, + observed: CheckResult::Passed, + title: None, + message: "Repo maintenance completed successfully again", + detail: None, + default_ceiling: CheckResult::Failed, + default_escalates: false, + documentation: Some(refs::MAINTENANCE_STALE_DOC), + }, ) .await?; filed += 1; @@ -404,18 +459,27 @@ async fn sweep_maintenance(db: &mut AsyncPgConnection, now: Timestamp) -> Result let err_active = open_group_issue_active(db, group_id, refs::MAINTENANCE_ERROR).await?; match latest_completed { Some(run) if run.outcome == Some(RunOutcome::Failure) => { - raise_group_event( + file_check( db, - group_id, - refs::MAINTENANCE_ERROR, - Severity::Error, - None, - &format!( - "Repo maintenance ({}) failed: {}", - run.kind, - run.error.as_deref().unwrap_or("(no detail reported)"), - ), - true, + CheckFiling { + source: crate::statuses::CANOPY_SOURCE, + scope: FilingScope::Group(group_id), + check: refs::MAINTENANCE_ERROR, + observed: CheckResult::Failed, + title: None, + message: &format!( + "Repo maintenance ({}) failed: {}", + run.kind, + run.error.as_deref().unwrap_or("(no detail reported)"), + ), + detail: Some(serde_json::json!({ + "kind": run.kind, + "error": run.error, + })), + default_ceiling: CheckResult::Failed, + default_escalates: false, + documentation: Some(refs::MAINTENANCE_ERROR_DOC), + }, ) .await?; filed += 1; @@ -423,14 +487,20 @@ async fn sweep_maintenance(db: &mut AsyncPgConnection, now: Timestamp) -> Result // Most recent finished run succeeded (or there is none): clear any // open failure issue. _ if err_active => { - raise_group_event( + file_check( db, - group_id, - refs::MAINTENANCE_ERROR, - Severity::Info, - None, - "Repo maintenance completed successfully again", - false, + CheckFiling { + source: crate::statuses::CANOPY_SOURCE, + scope: FilingScope::Group(group_id), + check: refs::MAINTENANCE_ERROR, + observed: CheckResult::Passed, + title: None, + message: "Repo maintenance completed successfully again", + detail: None, + default_ceiling: CheckResult::Failed, + default_escalates: false, + documentation: Some(refs::MAINTENANCE_ERROR_DOC), + }, ) .await?; filed += 1; diff --git a/crates/database/src/bin/seed.rs b/crates/database/src/bin/seed.rs index efc62d69..cfacec5c 100644 --- a/crates/database/src/bin/seed.rs +++ b/crates/database/src/bin/seed.rs @@ -16,15 +16,16 @@ use commons_errors::{AppError, Result}; use commons_types::{ device::DeviceRole, geo::GeoPoint, - issue::{ResolvedReason, Severity}, + issue::ResolvedReason, server::{TagMap, kind::ServerKind, rank::ServerRank}, + status::CheckResult, version::{VersionStatus, VersionStr}, }; use database::{ Device, DeviceKey, admins::Admin, + check_policies::CheckPolicy, devices::NewDeviceConnection, - healthcheck_severities::HealthcheckSeverity, issues::{Incident, Issue, NewEvent}, notes::{IncidentNote, IssueNote}, pg_duration::PgDuration, @@ -59,10 +60,8 @@ const TRUNCATE_TABLES: &[&str] = &[ "issue_notes", "incident_issues", "incidents", - "events", "issues", - "server_silenced_refs", - "server_group_silenced_refs", + "scoped_check_policies", "device_server_associations", "device_connections", "device_keys", @@ -70,7 +69,7 @@ const TRUNCATE_TABLES: &[&str] = &[ "server_groups", "version_known_issues", "versions", - "healthcheck_severities", + "check_policies", "admins", ]; @@ -323,49 +322,62 @@ async fn seed_healthchecks(conn: &mut AsyncPgConnection, admins: &[String]) -> R "certificate_expiry", "backup_freshness", ] { - HealthcheckSeverity::upsert_default(conn, check).await?; + CheckPolicy::upsert_default(conn, "alertd", check).await?; } let reviewer = &admins[0]; - HealthcheckSeverity::update( + CheckPolicy::update( conn, + "alertd", "database_connectivity", - Severity::Critical, + CheckResult::Failed, + true, Some("DB down means the server is effectively offline."), reviewer, ) .await?; - HealthcheckSeverity::update( + CheckPolicy::update( conn, + "alertd", "disk_space", - Severity::Error, + CheckResult::Failed, + false, Some("Page when disk is critically low."), reviewer, ) .await?; - HealthcheckSeverity::update(conn, "backup_freshness", Severity::Info, None, reviewer).await?; + CheckPolicy::update( + conn, + "alertd", + "backup_freshness", + CheckResult::Passed, + false, + None, + reviewer, + ) + .await?; - // A conditional rules ladder: escalate sync_lag based on how far behind it is. - use database::healthcheck_severities::{Condition, IfLadder, Var}; + // A conditional rules ladder: grade sync_lag by how far behind it is. + use database::check_policies::{Condition, IfLadder, Var}; let ladder = IfLadder { branches: vec![ ( Condition::Gt( "check.lag_seconds".parse::().expect("var parses"), - serde_json::json!(3600), + serde_json::json!(600), ), - Severity::Critical, + CheckResult::Failed, ), ( Condition::Gt( "check.lag_seconds".parse::().expect("var parses"), - serde_json::json!(600), + serde_json::json!(60), ), - Severity::Error, + CheckResult::Warning, ), ], }; - HealthcheckSeverity::update_rules(conn, "sync_lag", Some(&ladder), reviewer).await?; + CheckPolicy::update_rules(conn, "alertd", "sync_lag", Some(&ladder), reviewer).await?; Ok(()) } @@ -603,7 +615,6 @@ async fn seed_servers( cloud: None, geolocation: None, is_monitored: true, - allow_legacy_status: false, alert_when_down_for: TEN_MINUTES, notes: String::new(), tags: TagMap::default(), @@ -864,6 +875,7 @@ async fn seed_statuses( extra, healthy, health, + source: "alertd".into(), }) .execute(conn) .await?; @@ -967,21 +979,32 @@ async fn seed_issues_and_incidents( server_id: Uuid, source: &str, r#ref: &str, - severity: Severity, + result: CheckResult, + escalates: bool, description: &str, message: &str, occurred_at: Timestamp, ) -> Result { + let stamp = database::issues::CheckStateStamp { + check: r#ref.to_string(), + observed: result, + effective: result, + escalates, + detail: None, + }; + let active = matches!( + result, + CheckResult::Failed | CheckResult::Warning | CheckResult::Broken + ); NewEvent { source: source.to_string(), r#ref: r#ref.to_string(), - severity: Some(severity), description: Some(description.to_string()), message: message.to_string(), - active: Some(true), + active: Some(active), occurred_at: Some(occurred_at), } - .save(conn, server_id, None) + .save_with_state(conn, server_id, None, Some(&stamp)) .await } @@ -991,7 +1014,8 @@ async fn seed_issues_and_incidents( servers.unhealthy_facility, "healthcheck", "database_connectivity", - Severity::Critical, + CheckResult::Failed, + true, "Database connection refused", "The server cannot reach its PostgreSQL instance (connection refused). \ Sync and API requests are failing.", @@ -1007,7 +1031,8 @@ async fn seed_issues_and_incidents( servers.unhealthy_facility, "healthcheck", "database_connectivity", - Severity::Critical, + CheckResult::Failed, + true, "Database connection refused", "The server cannot reach its PostgreSQL instance (connection refused). \ Sync and API requests are failing.", @@ -1023,7 +1048,8 @@ async fn seed_issues_and_incidents( servers.healthy_central, "healthcheck", "certificate_expiry", - Severity::Warning, + CheckResult::Warning, + false, "TLS certificate expiring soon", "The TLS certificate expires in 9 days.", now - SignedDuration::from_mins(30), @@ -1037,7 +1063,8 @@ async fn seed_issues_and_incidents( servers.warning_facility, "healthcheck", "sync_lag", - Severity::Error, + CheckResult::Failed, + false, "Sync lag exceeds threshold", "Sync lag has been above 30 minutes for the last hour.", now - SignedDuration::from_mins(60), @@ -1051,7 +1078,8 @@ async fn seed_issues_and_incidents( servers.warning_facility, "healthcheck", "disk_space", - Severity::Error, + CheckResult::Failed, + false, "Disk almost full", "Disk usage crossed 95% earlier; has since recovered.", now - SignedDuration::from_hours(6), @@ -1059,26 +1087,28 @@ async fn seed_issues_and_incidents( .await?; Issue::resolve(conn, transient.id, &admins[1], ResolvedReason::Fixed).await?; - // An info-level issue on the ungrouped server (recorded, no incident). + // A warning on the ungrouped server (recorded, no incident). push( conn, servers.ungrouped, "app", "slow-query", - Severity::Info, + CheckResult::Warning, + false, "Slow query detected", "A report query took 12s; investigate indexing.", now - SignedDuration::from_hours(2), ) .await?; - // A debug-level issue (never participates in incidents). + // A skipped-graded condition (recorded, never participates). push( conn, servers.demo_server, "app", "debug-trace", - Severity::Debug, + CheckResult::Skipped, + false, "Verbose trace captured", "Captured a debug trace during a demo session.", now - SignedDuration::from_hours(1), @@ -1165,7 +1195,7 @@ async fn report(conn: &mut AsyncPgConnection) -> Result<()> { "admins", "versions", "version_known_issues", - "healthcheck_severities", + "check_policies", "devices", "device_keys", "device_connections", @@ -1174,14 +1204,12 @@ async fn report(conn: &mut AsyncPgConnection) -> Result<()> { "server_enrollment_tokens", "statuses", "issues", - "events", "incidents", "incident_issues", "issue_notes", "incident_notes", "slack_outbox", - "server_silenced_refs", - "server_group_silenced_refs", + "scoped_check_policies", ]; eprintln!("seed: row counts"); diff --git a/crates/database/src/check_policies.rs b/crates/database/src/check_policies.rs new file mode 100644 index 00000000..ea142b67 --- /dev/null +++ b/crates/database/src/check_policies.rs @@ -0,0 +1,930 @@ +//! Operator-owned catalog of check policies: per (source, check), how an +//! observed result transforms into the effective result Canopy acts on. +//! +//! An entry carries a **ceiling** (the maximum effective result on the +//! urgency ordering — `failed` changes nothing, `warning` grades failures +//! as warnings, `passed` means recorded but never alerting, `skipped` +//! additionally tells the source not to bother running the check), +//! optional conditional **rules** (transforms in any direction, evaluated +//! against the check's detail, the report's server-wide detail, and the +//! server's tags), and an **escalates** flag (an effective failure of +//! this check notifies immediately, bypassing incident grace). +//! +//! Ingestion (in the public-server status handler) calls +//! [`CheckPolicy::upsert_default`] for every check seen on a push, then +//! [`CheckPolicy::apply`] to grade each observed result. Operators read +//! and edit the catalog via the private-server `/api/healthchecks` +//! endpoints. + +use commons_errors::{AppError, Result}; +use commons_types::status::CheckResult; +use diesel::prelude::*; +use diesel_async::{AsyncPgConnection, RunQueryDsl}; +use jiff::Timestamp; +use serde::{Deserialize, Deserializer, Serialize, Serializer, de}; +use serde_json::Value as JsonValue; +use std::collections::{BTreeMap, HashMap}; +use uuid::Uuid; + +/// The policy for one (source, check). An entry is created automatically +/// the first time a source reports a check with this name, at the default +/// ceiling; operators then review and adjust how that check's results are +/// graded going forward. +#[derive(Clone, Debug, Serialize, Deserialize, Queryable, Selectable, utoipa::ToSchema)] +#[diesel(table_name = crate::schema::check_policies)] +#[diesel(check_for_backend(diesel::pg::Pg))] +pub struct CheckPolicy { + /// The source that reports this check. Together with `check_name`, + /// uniquely identifies this policy. + pub source: String, + /// The check's name, as reported in status pushes. + pub check_name: String, + /// The maximum effective result for this check when no conditional + /// rule (see `rules`) overrides it: an observed result more urgent + /// than the ceiling grades down to it. + #[diesel(deserialize_as = String, serialize_as = String)] + #[schema(value_type = String)] + pub ceiling: CheckResult, + /// Whether an effective failure of this check notifies immediately, + /// bypassing the incident grace period. + pub escalates: bool, + /// Optional conditional rules that can grade a result differently — + /// in any direction — depending on the details of the check, the + /// surrounding status report, or the server's tags. `None` means no + /// conditional rules are configured and the ceiling always applies. + /// When present, the rules are evaluated in order and the first + /// matching one wins; if none match, the ceiling applies. + #[schema(value_type = Option)] + pub rules: Option, + /// Free-form operator notes about this check. + pub notes: Option, + /// When this check was first observed and this policy entry was created. + #[diesel(deserialize_as = jiff_diesel::Timestamp, serialize_as = jiff_diesel::Timestamp)] + pub first_seen: Timestamp, + /// When an operator last reviewed or edited this policy. `None` if it + /// has never been reviewed. + #[diesel(deserialize_as = jiff_diesel::NullableTimestamp, serialize_as = jiff_diesel::NullableTimestamp)] + pub reviewed_at: Option, + /// The operator who last reviewed this policy. `None` if it has never + /// been reviewed. + pub reviewed_by: Option, + /// When this policy was last modified. + #[diesel(deserialize_as = jiff_diesel::Timestamp, serialize_as = jiff_diesel::Timestamp)] + pub updated_at: Timestamp, + /// Operator-authored documentation for this check: a single markdown + /// document, presented wherever the check's state is presented and + /// over MCP. By convention it covers what the check observes, what + /// each result means, and hints for solving a failure; canopy + /// attaches no meaning to its structure. + pub documentation: Option, +} + +/// The outcome of applying a check's policy to an observed result. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct GradedResult { + /// The effective result after the policy transform. + pub effective: CheckResult, + /// Whether this check's effective failures escalate (notify + /// immediately, bypassing incident grace). + pub escalates: bool, +} + +impl CheckPolicy { + /// Insert a row for `(source, check_name)` with default values + /// (ceiling = warning) if and only if no row exists yet. Idempotent: + /// safe to call on every status push for every check seen, including + /// healthy ones. Concurrent pushes are serialised by Postgres via + /// `ON CONFLICT DO NOTHING`. + pub async fn upsert_default( + db: &mut AsyncPgConnection, + source: &str, + check_name: &str, + ) -> Result<()> { + use crate::schema::check_policies::dsl; + diesel::insert_into(dsl::check_policies) + .values((dsl::source.eq(source), dsl::check_name.eq(check_name))) + .on_conflict((dsl::source, dsl::check_name)) + .do_nothing() + .execute(db) + .await + .map_err(AppError::from)?; + Ok(()) + } + + /// Insert a row for `(source, check_name)` with the given policy — + /// and, for canopy's own checks, shipped documentation — if and only + /// if no row exists yet. Canopy's own checks register with the policy + /// their condition warrants instead of the default warning ceiling; + /// operator edits stick (this never overwrites). + pub async fn register( + db: &mut AsyncPgConnection, + source: &str, + check_name: &str, + ceiling: CheckResult, + escalates: bool, + documentation: Option<&str>, + ) -> Result<()> { + use crate::schema::check_policies::dsl; + diesel::insert_into(dsl::check_policies) + .values(( + dsl::source.eq(source), + dsl::check_name.eq(check_name), + dsl::ceiling.eq(ceiling.to_string()), + dsl::escalates.eq(escalates), + dsl::documentation.eq(documentation), + )) + .on_conflict((dsl::source, dsl::check_name)) + .do_nothing() + .execute(db) + .await + .map_err(AppError::from)?; + Ok(()) + } + + /// Apply the `(source, check_name)` policy to an `observed` result: + /// if the entry has a `rules` ladder and a branch matches the + /// supplied evaluation context, that branch's result wins (any + /// direction — rules can upgrade as well as downgrade); otherwise the + /// observed result is capped at the entry's ceiling. + /// + /// Falls back to the default policy (ceiling = warning, no + /// escalation) if no row exists yet — in practice the status handler + /// upserts before reading, so this branch only covers the genuine + /// race / programmer-error case. + pub async fn apply( + db: &mut AsyncPgConnection, + source: &str, + check_name: &str, + observed: CheckResult, + ctx: &EvaluationContext<'_>, + ) -> Result { + use crate::schema::check_policies::dsl; + let row: Option<(String, bool, Option)> = dsl::check_policies + .select((dsl::ceiling, dsl::escalates, dsl::rules)) + .filter(dsl::source.eq(source).and(dsl::check_name.eq(check_name))) + .first(db) + .await + .optional()?; + let Some((ceiling_str, escalates, rules_json)) = row else { + return Ok(GradedResult { + effective: observed.capped_at(CheckResult::Warning), + escalates: false, + }); + }; + let ceiling = ceiling_str.parse().unwrap_or(CheckResult::Warning); + if let Some(rules) = rules_json { + match serde_json::from_value::(rules) { + Ok(ladder) => { + if let Some(result) = ladder.evaluate(ctx) { + return Ok(GradedResult { + effective: result, + escalates, + }); + } + } + Err(err) => { + tracing::warn!( + source, + check_name, + ?err, + "failed to parse check policy rules; falling back to ceiling" + ); + } + } + } + Ok(GradedResult { + effective: observed.capped_at(ceiling), + escalates, + }) + } + + /// [`Self::apply`], then the scoped transforms that cover the filing's + /// target: fleet catalog, then group, then server — each acting on + /// the previous effective result, so the most specific scope has the + /// last word. A canopy-wide filing (no server, no group) chains the + /// canopy-wide scoped transform instead. + /// + /// A scoped silence is a skipped ceiling in this chain: whatever the + /// fleet grading said, the effective result lands at skipped. + pub async fn apply_scoped( + db: &mut AsyncPgConnection, + source: &str, + check_name: &str, + observed: CheckResult, + ctx: &EvaluationContext<'_>, + server_id: Option, + group_id: Option, + ) -> Result { + let fleet = Self::apply(db, source, check_name, observed, ctx).await?; + let scoped = + ScopedCheckPolicy::chain_for(db, source, check_name, server_id, group_id).await?; + let mut effective = fleet.effective; + for transform in &scoped { + effective = transform.transform(effective, ctx); + } + Ok(GradedResult { + effective, + escalates: fleet.escalates, + }) + } + + /// Replace the documentation for a check (or clear it with `None`). + /// Doesn't stamp the review columns — documenting a check is not the + /// same as reviewing its policy. + pub async fn update_documentation( + db: &mut AsyncPgConnection, + source: &str, + check_name: &str, + documentation: Option<&str>, + ) -> Result { + use crate::schema::check_policies::dsl; + diesel::update( + dsl::check_policies.filter(dsl::source.eq(source).and(dsl::check_name.eq(check_name))), + ) + .set(dsl::documentation.eq(documentation)) + .returning(Self::as_select()) + .get_result(db) + .await + .map_err(AppError::from) + } + + /// Replace the conditional-rules ladder for a check (or clear it + /// with `None`). Stamps `reviewed_at` / `reviewed_by`, so editing + /// rules also counts as a review for the catalog row. + pub async fn update_rules( + db: &mut AsyncPgConnection, + source: &str, + check_name: &str, + rules: Option<&IfLadder>, + by: &str, + ) -> Result { + use crate::schema::check_policies::dsl; + let now = Timestamp::now(); + let rules_json: Option = + rules.map(|l| serde_json::to_value(l).expect("IfLadder always serialises")); + diesel::update( + dsl::check_policies.filter(dsl::source.eq(source).and(dsl::check_name.eq(check_name))), + ) + .set(( + dsl::rules.eq(rules_json), + dsl::reviewed_at.eq(jiff_diesel::Timestamp::from(now)), + dsl::reviewed_by.eq(by), + )) + .returning(Self::as_select()) + .get_result(db) + .await + .map_err(AppError::from) + } + + /// One source's catalog as `check_name → ceiling`, for building the + /// device-facing effective check map. Deliberately reads only the + /// static `ceiling` column: conditional `rules` ladders are + /// expressions evaluated per push against the report's contents, so + /// they can't be resolved ahead of time and are ignored here. An + /// unparseable ceiling falls back to warning, same as [`Self::apply`]. + pub async fn ceiling_map_for_source( + db: &mut AsyncPgConnection, + source: &str, + ) -> Result> { + use crate::schema::check_policies::dsl; + let rows: Vec<(String, String)> = dsl::check_policies + .select((dsl::check_name, dsl::ceiling)) + .filter(dsl::source.eq(source)) + .load(db) + .await + .map_err(AppError::from)?; + Ok(rows + .into_iter() + .map(|(name, ceiling)| (name, ceiling.parse().unwrap_or(CheckResult::Warning))) + .collect()) + } + + pub async fn list(db: &mut AsyncPgConnection) -> Result> { + use crate::schema::check_policies::dsl; + dsl::check_policies + .select(Self::as_select()) + .order((dsl::source.asc(), dsl::check_name.asc())) + .load(db) + .await + .map_err(AppError::from) + } + + /// The catalog row for a single (source, check), or `None` if that + /// source has never reported it (so ingestion has never upserted a + /// row). + pub async fn get( + db: &mut AsyncPgConnection, + source: &str, + check_name: &str, + ) -> Result> { + use crate::schema::check_policies::dsl; + dsl::check_policies + .select(Self::as_select()) + .filter(dsl::source.eq(source).and(dsl::check_name.eq(check_name))) + .first(db) + .await + .optional() + .map_err(AppError::from) + } + + /// Update the ceiling, escalation flag, and optionally notes for a + /// check, stamping `reviewed_at = NOW()` and `reviewed_by = by`. Even + /// a no-op save marks the row reviewed — operators can ack a check + /// without changing it. + pub async fn update( + db: &mut AsyncPgConnection, + source: &str, + check_name: &str, + ceiling: CheckResult, + escalates: bool, + notes: Option<&str>, + by: &str, + ) -> Result { + use crate::schema::check_policies::dsl; + let now = Timestamp::now(); + diesel::update( + dsl::check_policies.filter(dsl::source.eq(source).and(dsl::check_name.eq(check_name))), + ) + .set(( + dsl::ceiling.eq(ceiling.to_string()), + dsl::escalates.eq(escalates), + dsl::notes.eq(notes), + dsl::reviewed_at.eq(jiff_diesel::Timestamp::from(now)), + dsl::reviewed_by.eq(by), + )) + .returning(Self::as_select()) + .get_result(db) + .await + .map_err(AppError::from) + } +} + +/// The scope a scoped transform (or silence) attaches to: a server, a +/// server group, or canopy-wide — mirroring check targets. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum PolicyScope { + Server(Uuid), + Group(Uuid), + Global, +} + +/// A result transform scoped to one target, applied after the fleet +/// catalog: fleet, then group, then server, each acting on the previous +/// effective result. Either side may be present — a ceiling, a rules +/// ladder, or both (rules first, then the ceiling caps their outcome). +/// +/// The operator-facing **silence** is a scoped ceiling of `skipped`: +/// the check keeps recording observed results but its effective result +/// is skipped, so it raises nothing and counts nowhere. Arbitrary +/// scoped transforms are admitted here; the UI only offers silences. +#[derive(Clone, Debug, Serialize, Deserialize, Queryable, Selectable, utoipa::ToSchema)] +#[diesel(table_name = crate::schema::scoped_check_policies)] +#[diesel(check_for_backend(diesel::pg::Pg))] +pub struct ScopedCheckPolicy { + /// Unique identifier of this scoped transform. + pub id: Uuid, + /// When this transform was created. + #[diesel(deserialize_as = jiff_diesel::Timestamp, serialize_as = jiff_diesel::Timestamp)] + pub created_at: Timestamp, + /// When this transform was last modified. + #[diesel(deserialize_as = jiff_diesel::Timestamp, serialize_as = jiff_diesel::Timestamp)] + pub updated_at: Timestamp, + /// The source whose check this transform applies to. + pub source: String, + /// The check this transform applies to. + pub check_name: String, + /// Set for a server-scoped transform. + pub server_id: Option, + /// Set for a group-scoped transform. Both `server_id` and + /// `server_group_id` unset means canopy-wide scope. + pub server_group_id: Option, + /// Scoped ceiling: caps the effective result arriving from the + /// previous transform in the chain. `skipped` is the silence. + pub ceiling: Option, + /// Scoped conditional rules, same shape as the fleet catalog's. + #[schema(value_type = Option)] + pub rules: Option, + /// The operator who created this transform. `None` if not recorded. + pub created_by: Option, +} + +impl ScopedCheckPolicy { + fn scope_filter(scope: PolicyScope) -> (Option, Option) { + match scope { + PolicyScope::Server(id) => (Some(id), None), + PolicyScope::Group(id) => (None, Some(id)), + PolicyScope::Global => (None, None), + } + } + + /// The transform at exactly this (scope, source, check), if any. + pub async fn get( + db: &mut AsyncPgConnection, + scope: PolicyScope, + source: &str, + check_name: &str, + ) -> Result> { + use crate::schema::scoped_check_policies::dsl; + let (server, group) = Self::scope_filter(scope); + dsl::scoped_check_policies + .select(Self::as_select()) + .filter( + dsl::source + .eq(source) + .and(dsl::check_name.eq(check_name)) + .and(dsl::server_id.is_not_distinct_from(server)) + .and(dsl::server_group_id.is_not_distinct_from(group)), + ) + .first(db) + .await + .optional() + .map_err(AppError::from) + } + + /// Upsert a silence: a skipped ceiling at this scope. An existing + /// transform at the same (scope, source, check) keeps its rules; its + /// ceiling becomes skipped. Idempotent. + pub async fn silence( + db: &mut AsyncPgConnection, + scope: PolicyScope, + source: &str, + check_name: &str, + created_by: Option<&str>, + ) -> Result { + use crate::schema::scoped_check_policies::dsl; + let (server, group) = Self::scope_filter(scope); + if let Some(existing) = Self::get(db, scope, source, check_name).await? { + return diesel::update(dsl::scoped_check_policies.filter(dsl::id.eq(existing.id))) + .set(( + dsl::ceiling.eq(CheckResult::Skipped.to_string()), + dsl::updated_at.eq(jiff_diesel::Timestamp::from(Timestamp::now())), + )) + .returning(Self::as_select()) + .get_result(db) + .await + .map_err(AppError::from); + } + diesel::insert_into(dsl::scoped_check_policies) + .values(( + dsl::source.eq(source), + dsl::check_name.eq(check_name), + dsl::server_id.eq(server), + dsl::server_group_id.eq(group), + dsl::ceiling.eq(CheckResult::Skipped.to_string()), + dsl::created_by.eq(created_by), + )) + .returning(Self::as_select()) + .get_result(db) + .await + .map_err(AppError::from) + } + + /// Remove a silence at this scope: the row is deleted when the + /// silence was all it carried, or just the skipped ceiling is lifted + /// when scoped rules remain. A no-op if nothing is silenced there. + pub async fn unsilence( + db: &mut AsyncPgConnection, + scope: PolicyScope, + source: &str, + check_name: &str, + ) -> Result<()> { + use crate::schema::scoped_check_policies::dsl; + let Some(existing) = Self::get(db, scope, source, check_name).await? else { + return Ok(()); + }; + if existing.ceiling.as_deref() != Some("skipped") { + return Ok(()); + } + if existing.rules.is_some() { + diesel::update(dsl::scoped_check_policies.filter(dsl::id.eq(existing.id))) + .set(( + dsl::ceiling.eq(None::), + dsl::updated_at.eq(jiff_diesel::Timestamp::from(Timestamp::now())), + )) + .execute(db) + .await + .map_err(AppError::from)?; + } else { + diesel::delete(dsl::scoped_check_policies.filter(dsl::id.eq(existing.id))) + .execute(db) + .await + .map_err(AppError::from)?; + } + Ok(()) + } + + /// All silences (skipped-ceiling transforms) at one scope, newest + /// first. + pub async fn list_silences( + db: &mut AsyncPgConnection, + scope: PolicyScope, + ) -> Result> { + use crate::schema::scoped_check_policies::dsl; + let (server, group) = Self::scope_filter(scope); + dsl::scoped_check_policies + .select(Self::as_select()) + .filter( + dsl::server_id + .is_not_distinct_from(server) + .and(dsl::server_group_id.is_not_distinct_from(group)) + .and(dsl::ceiling.eq(CheckResult::Skipped.to_string())), + ) + .order(dsl::created_at.desc()) + .load(db) + .await + .map_err(AppError::from) + } + + /// The scoped transforms that apply to a filing, in application + /// order. A server filing chains group then server; a group filing + /// its group row; a canopy-wide filing the global row. + pub async fn chain_for( + db: &mut AsyncPgConnection, + source: &str, + check_name: &str, + server_id: Option, + group_id: Option, + ) -> Result> { + use crate::schema::scoped_check_policies::dsl; + let mut query = dsl::scoped_check_policies + .select(Self::as_select()) + .filter(dsl::source.eq(source).and(dsl::check_name.eq(check_name))) + .into_boxed(); + query = match (server_id, group_id) { + (None, None) => { + query.filter(dsl::server_id.is_null().and(dsl::server_group_id.is_null())) + } + (server, group) => query.filter( + dsl::server_id + .is_not_distinct_from(server) + .and(dsl::server_id.is_not_null()) + .or(dsl::server_group_id + .is_not_distinct_from(group) + .and(dsl::server_group_id.is_not_null())), + ), + }; + let mut rows: Vec = query.load(db).await.map_err(AppError::from)?; + // Group scope applies before server scope: the most specific + // transform has the last word. + rows.sort_by_key(|r| r.server_id.is_some()); + Ok(rows) + } + + /// Apply this transform to the effective result arriving from the + /// previous step in the chain: rules first (a matching branch's + /// result replaces the input), then the ceiling caps the outcome. + pub fn transform(&self, input: CheckResult, ctx: &EvaluationContext<'_>) -> CheckResult { + let mut result = input; + if let Some(rules) = &self.rules { + match serde_json::from_value::(rules.clone()) { + Ok(ladder) => { + if let Some(matched) = ladder.evaluate(ctx) { + result = matched; + } + } + Err(err) => { + tracing::warn!( + source = self.source, + check_name = self.check_name, + ?err, + "failed to parse scoped policy rules; ignoring them" + ); + } + } + } + if let Some(ceiling) = self.ceiling.as_deref().and_then(|c| c.parse().ok()) { + result = result.capped_at(ceiling); + } + result + } +} + +// ── Rule model: JsonLogic-encoded if-ladder ──────────────────────────────── + +/// A JsonLogic `if`-ladder evaluating to a [`CheckResult`] string. +/// +/// Wire shape: `{"if": [c1, r1, c2, r2, …, cN, rN]}` — even-length +/// argument list, every odd-index entry is a [`Condition`], every +/// even-index entry is a result literal. No trailing else: when no +/// branch matches, [`Self::evaluate`] returns `None` and the calling +/// [`CheckPolicy::apply`] falls through to the entry's ceiling. +/// +/// Empty ladders are forbidden by the deserialiser; the API layer +/// normalises them to `None` (clearing the `rules` column) before +/// hitting the database. +#[derive(Clone, Debug, PartialEq)] +pub struct IfLadder { + pub branches: Vec<(Condition, CheckResult)>, +} + +/// One predicate inside an [`IfLadder`] branch. Maps 1:1 with a single +/// JsonLogic operator; composition (`and`/`or`/`!`, nested `if`) is +/// rejected at deserialise time. +#[derive(Clone, Debug, PartialEq)] +pub enum Condition { + Eq(Var, JsonValue), + Neq(Var, JsonValue), + Lt(Var, JsonValue), + Lte(Var, JsonValue), + Gt(Var, JsonValue), + Gte(Var, JsonValue), + /// `value` is a `node_semver::Range`-parseable string. + InRange(Var, String), +} + +/// Dotted path into the [`EvaluationContext`]: `kind.field`. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Var { + pub kind: VarKind, + pub field: String, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum VarKind { + Check, + Status, + Tag, +} + +/// Inputs available to a rule at evaluation time. Built once per status +/// push and shared across all rule evaluations for that push. +pub struct EvaluationContext<'a> { + /// Top-level status extras (`statuses.extra`). Bestool sends + /// `bestoolVersion`, `tamanuVersion`, `uptimeSecs`, etc. + pub status_extra: &'a serde_json::Map, + /// The check's own fields (the `health[i]` object minus `check` and + /// `healthy`). + pub check_extra: &'a serde_json::Map, + /// Server's resolved tag map (merged server + group). Each value is + /// already wrapped as `JsonValue::String` for uniform comparison. + pub tags: &'a HashMap, +} + +impl IfLadder { + /// Returns the first matching branch's result, or `None` if no + /// branch matches. The caller falls back to the entry's ceiling in + /// that case. + pub fn evaluate(&self, ctx: &EvaluationContext) -> Option { + self.branches + .iter() + .find_map(|(c, r)| c.matches(ctx).then_some(*r)) + } +} + +impl Var { + pub fn resolve<'a>(&self, ctx: &'a EvaluationContext<'a>) -> Option<&'a JsonValue> { + match self.kind { + VarKind::Check => ctx.check_extra.get(&self.field), + VarKind::Status => ctx.status_extra.get(&self.field), + VarKind::Tag => ctx.tags.get(&self.field), + } + } +} + +impl Condition { + /// Borrow the variable and right-hand operand without cloning. + fn var(&self) -> &Var { + match self { + Self::Eq(v, _) + | Self::Neq(v, _) + | Self::Lt(v, _) + | Self::Lte(v, _) + | Self::Gt(v, _) + | Self::Gte(v, _) + | Self::InRange(v, _) => v, + } + } + + pub fn matches(&self, ctx: &EvaluationContext) -> bool { + let Some(lhs) = self.var().resolve(ctx) else { + return false; + }; + match self { + Self::Eq(_, rhs) => json_equal(lhs, rhs), + Self::Neq(_, rhs) => !json_equal(lhs, rhs), + Self::Lt(_, rhs) => json_compare(lhs, rhs, |a, b| a < b).unwrap_or(false), + Self::Lte(_, rhs) => json_compare(lhs, rhs, |a, b| a <= b).unwrap_or(false), + Self::Gt(_, rhs) => json_compare(lhs, rhs, |a, b| a > b).unwrap_or(false), + Self::Gte(_, rhs) => json_compare(lhs, rhs, |a, b| a >= b).unwrap_or(false), + Self::InRange(_, range_str) => { + let Some(lhs_str) = lhs.as_str() else { + return false; + }; + let Ok(version) = lhs_str.parse::() else { + return false; + }; + let Ok(range) = range_str.parse::() else { + return false; + }; + range.satisfies(&version) + } + } + } +} + +/// Bestool sometimes sends what look like numeric values as JSON strings +/// (e.g. `"currentSyncTick": "21608625"`). Both `==`/`!=` and the numeric +/// comparisons treat such strings numerically when both sides are +/// numeric-coercible. Strict JSON equality is the first check, so +/// `"abc" == "abc"` still matches; the coercion only kicks in when at +/// least one side is a number. +fn to_f64(v: &JsonValue) -> Option { + match v { + JsonValue::Number(n) => n.as_f64(), + JsonValue::String(s) => s.parse::().ok(), + _ => None, + } +} + +fn json_equal(a: &JsonValue, b: &JsonValue) -> bool { + if a == b { + return true; + } + if let (Some(an), Some(bn)) = (to_f64(a), to_f64(b)) { + return an == bn; + } + false +} + +fn json_compare bool>(a: &JsonValue, b: &JsonValue, f: F) -> Option { + let an = to_f64(a)?; + let bn = to_f64(b)?; + Some(f(an, bn)) +} + +// ── Serde: typed <-> JsonLogic ───────────────────────────────────────────── + +impl std::str::FromStr for Var { + type Err = String; + fn from_str(s: &str) -> std::result::Result { + let (kind_str, field) = s + .split_once('.') + .ok_or_else(|| format!("var path '{s}' is missing a '.' segment"))?; + let kind = match kind_str { + "check" => VarKind::Check, + "status" => VarKind::Status, + "tag" => VarKind::Tag, + other => { + return Err(format!( + "unknown var namespace '{other}' (expected check, status, or tag)" + )); + } + }; + if field.is_empty() { + return Err(format!("var path '{s}' has an empty field name")); + } + if !field.chars().all(|c| c.is_ascii_alphanumeric() || c == '_') { + return Err(format!( + "var field '{field}' must be ASCII alphanumeric or underscore" + )); + } + Ok(Var { + kind, + field: field.to_string(), + }) + } +} + +impl std::fmt::Display for Var { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + let kind = match self.kind { + VarKind::Check => "check", + VarKind::Status => "status", + VarKind::Tag => "tag", + }; + write!(f, "{kind}.{}", self.field) + } +} + +impl Serialize for Condition { + fn serialize(&self, ser: S) -> std::result::Result { + let (op, rhs): (&str, JsonValue) = match self { + Self::Eq(_, v) => ("==", v.clone()), + Self::Neq(_, v) => ("!=", v.clone()), + Self::Lt(_, v) => ("<", v.clone()), + Self::Lte(_, v) => ("<=", v.clone()), + Self::Gt(_, v) => (">", v.clone()), + Self::Gte(_, v) => (">=", v.clone()), + Self::InRange(_, r) => ("in_range", JsonValue::String(r.clone())), + }; + let var_obj = serde_json::json!({ "var": self.var().to_string() }); + let value = serde_json::json!({ op: [var_obj, rhs] }); + value.serialize(ser) + } +} + +impl<'de> Deserialize<'de> for Condition { + fn deserialize>(de: D) -> std::result::Result { + let value = JsonValue::deserialize(de)?; + let obj = value + .as_object() + .ok_or_else(|| de::Error::custom("condition must be a JSON object"))?; + if obj.len() != 1 { + return Err(de::Error::custom( + "condition object must have exactly one key (the JsonLogic op)", + )); + } + let (op, args) = obj.iter().next().expect("len == 1"); + let args = args + .as_array() + .ok_or_else(|| de::Error::custom(format!("'{op}' args must be an array")))?; + if args.len() != 2 { + return Err(de::Error::custom(format!( + "'{op}' must have exactly 2 args (got {})", + args.len() + ))); + } + let var_obj = args[0] + .as_object() + .ok_or_else(|| de::Error::custom("first arg must be a {\"var\": ...} object"))?; + if var_obj.len() != 1 { + return Err(de::Error::custom( + "first arg must be {\"var\": \"\"} with no extras", + )); + } + let var_str = var_obj + .get("var") + .and_then(|v| v.as_str()) + .ok_or_else(|| de::Error::custom("first arg's 'var' must be a string"))?; + let var: Var = var_str.parse().map_err(de::Error::custom)?; + let rhs = args[1].clone(); + match op.as_str() { + "==" => Ok(Self::Eq(var, rhs)), + "!=" => Ok(Self::Neq(var, rhs)), + "<" => Ok(Self::Lt(var, rhs)), + "<=" => Ok(Self::Lte(var, rhs)), + ">" => Ok(Self::Gt(var, rhs)), + ">=" => Ok(Self::Gte(var, rhs)), + "in_range" => { + let range = rhs + .as_str() + .ok_or_else(|| de::Error::custom("'in_range' value must be a string"))?; + range.parse::().map_err(|e| { + de::Error::custom(format!("invalid semver range '{range}': {e}")) + })?; + Ok(Self::InRange(var, range.to_string())) + } + other => Err(de::Error::custom(format!( + "unsupported op '{other}' (allowed: ==, !=, <, <=, >, >=, in_range)" + ))), + } + } +} + +impl Serialize for IfLadder { + fn serialize(&self, ser: S) -> std::result::Result { + let mut args: Vec = Vec::with_capacity(self.branches.len() * 2); + for (c, r) in &self.branches { + args.push(serde_json::to_value(c).map_err(serde::ser::Error::custom)?); + args.push(JsonValue::String(r.to_string())); + } + serde_json::json!({ "if": args }).serialize(ser) + } +} + +impl<'de> Deserialize<'de> for IfLadder { + fn deserialize>(de: D) -> std::result::Result { + let value = JsonValue::deserialize(de)?; + let obj = value + .as_object() + .ok_or_else(|| de::Error::custom("rules must be a JSON object"))?; + if obj.len() != 1 { + return Err(de::Error::custom( + "rules must be a single-key {\"if\": …} object", + )); + } + let (key, args) = obj.iter().next().expect("len == 1"); + if key != "if" { + return Err(de::Error::custom(format!( + "rules op must be 'if' (got '{key}')" + ))); + } + let args = args + .as_array() + .ok_or_else(|| de::Error::custom("'if' args must be an array"))?; + if args.is_empty() { + return Err(de::Error::custom( + "'if' args must be non-empty; clear `rules` to remove the ladder instead", + )); + } + if args.len() % 2 != 0 { + return Err(de::Error::custom( + "'if' args must have even length (alternating condition, result); \ + a trailing else is not allowed", + )); + } + let mut branches = Vec::with_capacity(args.len() / 2); + for chunk in args.chunks_exact(2) { + let cond: Condition = + serde_json::from_value(chunk[0].clone()).map_err(de::Error::custom)?; + let result_str = chunk[1].as_str().ok_or_else(|| { + de::Error::custom("each odd-indexed 'if' arg must be a result string") + })?; + let result: CheckResult = result_str + .parse() + .map_err(|_| de::Error::custom(format!("invalid result '{result_str}'")))?; + branches.push((cond, result)); + } + Ok(IfLadder { branches }) + } +} diff --git a/crates/database/src/healthcheck_severities.rs b/crates/database/src/healthcheck_severities.rs deleted file mode 100644 index 00ef41cd..00000000 --- a/crates/database/src/healthcheck_severities.rs +++ /dev/null @@ -1,555 +0,0 @@ -//! Operator-owned catalog of healthcheck names → the severity to file -//! their failures at. -//! -//! Ingestion (in the public-server status handler) calls -//! [`HealthcheckSeverity::upsert_default`] for every check name seen on -//! a push, then [`HealthcheckSeverity::severity_for`] when filing a -//! failing per-check issue. Operators read and edit the catalog via the -//! private-server `/api/healthchecks` endpoints. - -use commons_errors::{AppError, Result}; -use commons_types::{issue::Severity, status::CheckResult}; -use diesel::prelude::*; -use diesel_async::{AsyncPgConnection, RunQueryDsl}; -use jiff::Timestamp; -use serde::{Deserialize, Deserializer, Serialize, Serializer, de}; -use serde_json::Value as JsonValue; -use std::collections::{BTreeMap, HashMap}; - -/// The severity policy for one named healthcheck. An entry is created -/// automatically the first time a check with this name is reported, using -/// a default policy; operators then review and adjust how that check's -/// failures should be classified going forward. -#[derive(Clone, Debug, Serialize, Deserialize, Queryable, Selectable, utoipa::ToSchema)] -#[diesel(table_name = crate::schema::healthcheck_severities)] -#[diesel(check_for_backend(diesel::pg::Pg))] -pub struct HealthcheckSeverity { - /// The healthcheck's name, as reported in status pushes. Uniquely - /// identifies this policy. - pub check_name: String, - /// The severity assigned to a failure of this check when no conditional - /// rule (see `rules`) overrides it. - #[diesel(deserialize_as = String, serialize_as = String)] - pub severity: Severity, - /// When this check was first observed and this policy entry was created. - #[diesel(deserialize_as = jiff_diesel::Timestamp, serialize_as = jiff_diesel::Timestamp)] - pub first_seen: Timestamp, - /// When an operator last reviewed or edited this policy. `None` if it - /// has never been reviewed. - #[diesel(deserialize_as = jiff_diesel::NullableTimestamp, serialize_as = jiff_diesel::NullableTimestamp)] - pub reviewed_at: Option, - /// The operator who last reviewed this policy. `None` if it has never - /// been reviewed. - pub reviewed_by: Option, - /// Free-form operator notes about this check. - pub notes: Option, - /// When this policy was last modified. - #[diesel(deserialize_as = jiff_diesel::Timestamp, serialize_as = jiff_diesel::Timestamp)] - pub updated_at: Timestamp, - /// Optional conditional rules that can assign a different severity - /// depending on the details of the failing check, the surrounding status - /// report, or the server's tags. `None` means no conditional rules are - /// configured and `severity` always applies. When present, the rules are - /// evaluated in order and the first matching one wins; if none match, - /// `severity` is used as the fallback. - #[schema(value_type = Option)] - pub rules: Option, -} - -impl HealthcheckSeverity { - /// Insert a row for `check_name` with default values (severity = - /// warning, reviewed_at = NULL) if and only if no row exists yet. - /// Idempotent: safe to call on every status push for every check - /// seen, including healthy ones. Concurrent pushes are serialised - /// by Postgres via `ON CONFLICT DO NOTHING`. - pub async fn upsert_default(db: &mut AsyncPgConnection, check_name: &str) -> Result<()> { - use crate::schema::healthcheck_severities::dsl; - diesel::insert_into(dsl::healthcheck_severities) - .values(dsl::check_name.eq(check_name)) - .on_conflict(dsl::check_name) - .do_nothing() - .execute(db) - .await - .map_err(AppError::from)?; - Ok(()) - } - - /// Look up the effective severity for a `check_name` reporting - /// `result` (warning or failed — the only kinds that file at the - /// catalog severity), given the supplied evaluation context. If - /// the row has a `rules` ladder it's evaluated against the context - /// first; the first matching branch's severity wins. Otherwise (no - /// ladder, no matching branch, or malformed JSON) the fallback - /// depends on the result kind: warning-result checks land at fixed - /// [`Severity::Warning`]; failed checks use the row's base - /// `severity` column. The catalog column is thus "the severity of - /// this check's failures" — warnings only deviate from Warning via - /// an explicit rule (which can condition on `check.result`). - /// - /// Falls back to `Severity::Warning` if no row exists yet — in - /// practice the status handler upserts before reading, so this - /// branch only covers the genuine race / programmer-error case. - pub async fn severity_for( - db: &mut AsyncPgConnection, - check_name: &str, - result: CheckResult, - ctx: &EvaluationContext<'_>, - ) -> Result { - use crate::schema::healthcheck_severities::dsl; - let row: Option<(String, Option)> = dsl::healthcheck_severities - .select((dsl::severity, dsl::rules)) - .filter(dsl::check_name.eq(check_name)) - .first(db) - .await - .optional()?; - let Some((sev_str, rules_json)) = row else { - return Ok(Severity::Warning); - }; - let base = sev_str.parse().unwrap_or(Severity::Warning); - if let Some(rules) = rules_json { - match serde_json::from_value::(rules) { - Ok(ladder) => { - if let Some(s) = ladder.evaluate(ctx) { - return Ok(s); - } - } - Err(err) => { - tracing::warn!( - check_name, - ?err, - "failed to parse healthcheck severity rules; falling back to base" - ); - } - } - } - Ok(match result { - CheckResult::Warning => Severity::Warning, - _ => base, - }) - } - - /// Replace the conditional-rules ladder for a check (or clear it - /// with `None`). Stamps `reviewed_at` / `reviewed_by`, so editing - /// rules also counts as a review for the catalog row. - pub async fn update_rules( - db: &mut AsyncPgConnection, - check_name: &str, - rules: Option<&IfLadder>, - by: &str, - ) -> Result { - use crate::schema::healthcheck_severities::dsl; - let now = Timestamp::now(); - let rules_json: Option = - rules.map(|l| serde_json::to_value(l).expect("IfLadder always serialises")); - diesel::update(dsl::healthcheck_severities.filter(dsl::check_name.eq(check_name))) - .set(( - dsl::rules.eq(rules_json), - dsl::reviewed_at.eq(jiff_diesel::Timestamp::from(now)), - dsl::reviewed_by.eq(by), - )) - .returning(Self::as_select()) - .get_result(db) - .await - .map_err(AppError::from) - } - - /// The whole catalog as `check_name → base severity`, for building the - /// device-facing effective check-severity map. Deliberately reads only - /// the static `severity` column: conditional `rules` ladders are - /// expressions evaluated per push against the report's contents, so they - /// can't be resolved ahead of time and are ignored here. An unparseable - /// severity falls back to [`Severity::Warning`], same as `severity_for`. - pub async fn base_severity_map( - db: &mut AsyncPgConnection, - ) -> Result> { - use crate::schema::healthcheck_severities::dsl; - let rows: Vec<(String, String)> = dsl::healthcheck_severities - .select((dsl::check_name, dsl::severity)) - .load(db) - .await - .map_err(AppError::from)?; - Ok(rows - .into_iter() - .map(|(name, sev)| (name, sev.parse().unwrap_or(Severity::Warning))) - .collect()) - } - - pub async fn list(db: &mut AsyncPgConnection) -> Result> { - use crate::schema::healthcheck_severities::dsl; - dsl::healthcheck_severities - .select(Self::as_select()) - .order(dsl::check_name.asc()) - .load(db) - .await - .map_err(AppError::from) - } - - /// The catalog row for a single check name, or `None` if no server has - /// ever reported it (so ingestion has never upserted a row). - pub async fn get(db: &mut AsyncPgConnection, check_name: &str) -> Result> { - use crate::schema::healthcheck_severities::dsl; - dsl::healthcheck_severities - .select(Self::as_select()) - .filter(dsl::check_name.eq(check_name)) - .first(db) - .await - .optional() - .map_err(AppError::from) - } - - /// Update the severity (and optionally notes) for a check, stamping - /// `reviewed_at = NOW()` and `reviewed_by = by`. Even a no-op save - /// (same severity) marks the row reviewed — operators can ack - /// a check without changing it. - pub async fn update( - db: &mut AsyncPgConnection, - check_name: &str, - severity: Severity, - notes: Option<&str>, - by: &str, - ) -> Result { - use crate::schema::healthcheck_severities::dsl; - let now = Timestamp::now(); - diesel::update(dsl::healthcheck_severities.filter(dsl::check_name.eq(check_name))) - .set(( - dsl::severity.eq(severity), - dsl::notes.eq(notes), - dsl::reviewed_at.eq(jiff_diesel::Timestamp::from(now)), - dsl::reviewed_by.eq(by), - )) - .returning(Self::as_select()) - .get_result(db) - .await - .map_err(AppError::from) - } -} - -// ── Rule model: JsonLogic-encoded if-ladder ──────────────────────────────── - -/// A JsonLogic `if`-ladder evaluating to a Severity string. -/// -/// Wire shape: `{"if": [c1, s1, c2, s2, …, cN, sN]}` — even-length -/// argument list, every odd-index entry is a [`Condition`], every -/// even-index entry is a Severity literal. No trailing else: when no -/// branch matches, [`Self::evaluate`] returns `None` and the calling -/// `severity_for` falls through to the row's `severity` column. -/// -/// Empty ladders are forbidden by the deserialiser; the API layer -/// normalises them to `None` (clearing the `rules` column) before -/// hitting the database. -#[derive(Clone, Debug, PartialEq)] -pub struct IfLadder { - pub branches: Vec<(Condition, Severity)>, -} - -/// One predicate inside an [`IfLadder`] branch. Maps 1:1 with a single -/// JsonLogic operator; composition (`and`/`or`/`!`, nested `if`) is -/// rejected at deserialise time. -#[derive(Clone, Debug, PartialEq)] -pub enum Condition { - Eq(Var, JsonValue), - Neq(Var, JsonValue), - Lt(Var, JsonValue), - Lte(Var, JsonValue), - Gt(Var, JsonValue), - Gte(Var, JsonValue), - /// `value` is a `node_semver::Range`-parseable string. - InRange(Var, String), -} - -/// Dotted path into the [`EvaluationContext`]: `kind.field`. -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct Var { - pub kind: VarKind, - pub field: String, -} - -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub enum VarKind { - Check, - Status, - Tag, -} - -/// Inputs available to a rule at evaluation time. Built once per status -/// push and shared across all rule evaluations for that push. -pub struct EvaluationContext<'a> { - /// Top-level status extras (`statuses.extra`). Bestool sends - /// `bestoolVersion`, `tamanuVersion`, `uptimeSecs`, etc. - pub status_extra: &'a serde_json::Map, - /// The failing check's own fields (the `health[i]` object minus - /// `check` and `healthy`). - pub check_extra: &'a serde_json::Map, - /// Server's resolved tag map (merged server + group). Each value is - /// already wrapped as `JsonValue::String` for uniform comparison. - pub tags: &'a HashMap, -} - -impl IfLadder { - /// Returns the first matching branch's severity, or `None` if no - /// branch matches. The caller falls back to the catalog's base - /// severity in that case. - pub fn evaluate(&self, ctx: &EvaluationContext) -> Option { - self.branches - .iter() - .find_map(|(c, s)| c.matches(ctx).then_some(*s)) - } -} - -impl Var { - pub fn resolve<'a>(&self, ctx: &'a EvaluationContext<'a>) -> Option<&'a JsonValue> { - match self.kind { - VarKind::Check => ctx.check_extra.get(&self.field), - VarKind::Status => ctx.status_extra.get(&self.field), - VarKind::Tag => ctx.tags.get(&self.field), - } - } -} - -impl Condition { - /// Borrow the variable and right-hand operand without cloning. - fn var(&self) -> &Var { - match self { - Self::Eq(v, _) - | Self::Neq(v, _) - | Self::Lt(v, _) - | Self::Lte(v, _) - | Self::Gt(v, _) - | Self::Gte(v, _) - | Self::InRange(v, _) => v, - } - } - - pub fn matches(&self, ctx: &EvaluationContext) -> bool { - let Some(lhs) = self.var().resolve(ctx) else { - return false; - }; - match self { - Self::Eq(_, rhs) => json_equal(lhs, rhs), - Self::Neq(_, rhs) => !json_equal(lhs, rhs), - Self::Lt(_, rhs) => json_compare(lhs, rhs, |a, b| a < b).unwrap_or(false), - Self::Lte(_, rhs) => json_compare(lhs, rhs, |a, b| a <= b).unwrap_or(false), - Self::Gt(_, rhs) => json_compare(lhs, rhs, |a, b| a > b).unwrap_or(false), - Self::Gte(_, rhs) => json_compare(lhs, rhs, |a, b| a >= b).unwrap_or(false), - Self::InRange(_, range_str) => { - let Some(lhs_str) = lhs.as_str() else { - return false; - }; - let Ok(version) = lhs_str.parse::() else { - return false; - }; - let Ok(range) = range_str.parse::() else { - return false; - }; - range.satisfies(&version) - } - } - } -} - -/// Bestool sometimes sends what look like numeric values as JSON strings -/// (e.g. `"currentSyncTick": "21608625"`). Both `==`/`!=` and the numeric -/// comparisons treat such strings numerically when both sides are -/// numeric-coercible. Strict JSON equality is the first check, so -/// `"abc" == "abc"` still matches; the coercion only kicks in when at -/// least one side is a number. -fn to_f64(v: &JsonValue) -> Option { - match v { - JsonValue::Number(n) => n.as_f64(), - JsonValue::String(s) => s.parse::().ok(), - _ => None, - } -} - -fn json_equal(a: &JsonValue, b: &JsonValue) -> bool { - if a == b { - return true; - } - if let (Some(an), Some(bn)) = (to_f64(a), to_f64(b)) { - return an == bn; - } - false -} - -fn json_compare bool>(a: &JsonValue, b: &JsonValue, f: F) -> Option { - let an = to_f64(a)?; - let bn = to_f64(b)?; - Some(f(an, bn)) -} - -// ── Serde: typed <-> JsonLogic ───────────────────────────────────────────── - -impl std::str::FromStr for Var { - type Err = String; - fn from_str(s: &str) -> std::result::Result { - let (kind_str, field) = s - .split_once('.') - .ok_or_else(|| format!("var path '{s}' is missing a '.' segment"))?; - let kind = match kind_str { - "check" => VarKind::Check, - "status" => VarKind::Status, - "tag" => VarKind::Tag, - other => { - return Err(format!( - "unknown var namespace '{other}' (expected check, status, or tag)" - )); - } - }; - if field.is_empty() { - return Err(format!("var path '{s}' has an empty field name")); - } - if !field.chars().all(|c| c.is_ascii_alphanumeric() || c == '_') { - return Err(format!( - "var field '{field}' must be ASCII alphanumeric or underscore" - )); - } - Ok(Var { - kind, - field: field.to_string(), - }) - } -} - -impl std::fmt::Display for Var { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - let kind = match self.kind { - VarKind::Check => "check", - VarKind::Status => "status", - VarKind::Tag => "tag", - }; - write!(f, "{kind}.{}", self.field) - } -} - -impl Serialize for Condition { - fn serialize(&self, ser: S) -> std::result::Result { - let (op, rhs): (&str, JsonValue) = match self { - Self::Eq(_, v) => ("==", v.clone()), - Self::Neq(_, v) => ("!=", v.clone()), - Self::Lt(_, v) => ("<", v.clone()), - Self::Lte(_, v) => ("<=", v.clone()), - Self::Gt(_, v) => (">", v.clone()), - Self::Gte(_, v) => (">=", v.clone()), - Self::InRange(_, r) => ("in_range", JsonValue::String(r.clone())), - }; - let var_obj = serde_json::json!({ "var": self.var().to_string() }); - let value = serde_json::json!({ op: [var_obj, rhs] }); - value.serialize(ser) - } -} - -impl<'de> Deserialize<'de> for Condition { - fn deserialize>(de: D) -> std::result::Result { - let value = JsonValue::deserialize(de)?; - let obj = value - .as_object() - .ok_or_else(|| de::Error::custom("condition must be a JSON object"))?; - if obj.len() != 1 { - return Err(de::Error::custom( - "condition object must have exactly one key (the JsonLogic op)", - )); - } - let (op, args) = obj.iter().next().expect("len == 1"); - let args = args - .as_array() - .ok_or_else(|| de::Error::custom(format!("'{op}' args must be an array")))?; - if args.len() != 2 { - return Err(de::Error::custom(format!( - "'{op}' must have exactly 2 args (got {})", - args.len() - ))); - } - let var_obj = args[0] - .as_object() - .ok_or_else(|| de::Error::custom("first arg must be a {\"var\": ...} object"))?; - if var_obj.len() != 1 { - return Err(de::Error::custom( - "first arg must be {\"var\": \"\"} with no extras", - )); - } - let var_str = var_obj - .get("var") - .and_then(|v| v.as_str()) - .ok_or_else(|| de::Error::custom("first arg's 'var' must be a string"))?; - let var: Var = var_str.parse().map_err(de::Error::custom)?; - let rhs = args[1].clone(); - match op.as_str() { - "==" => Ok(Self::Eq(var, rhs)), - "!=" => Ok(Self::Neq(var, rhs)), - "<" => Ok(Self::Lt(var, rhs)), - "<=" => Ok(Self::Lte(var, rhs)), - ">" => Ok(Self::Gt(var, rhs)), - ">=" => Ok(Self::Gte(var, rhs)), - "in_range" => { - let range = rhs - .as_str() - .ok_or_else(|| de::Error::custom("'in_range' value must be a string"))?; - range.parse::().map_err(|e| { - de::Error::custom(format!("invalid semver range '{range}': {e}")) - })?; - Ok(Self::InRange(var, range.to_string())) - } - other => Err(de::Error::custom(format!( - "unsupported op '{other}' (allowed: ==, !=, <, <=, >, >=, in_range)" - ))), - } - } -} - -impl Serialize for IfLadder { - fn serialize(&self, ser: S) -> std::result::Result { - let mut args: Vec = Vec::with_capacity(self.branches.len() * 2); - for (c, s) in &self.branches { - args.push(serde_json::to_value(c).map_err(serde::ser::Error::custom)?); - args.push(JsonValue::String(s.to_string())); - } - serde_json::json!({ "if": args }).serialize(ser) - } -} - -impl<'de> Deserialize<'de> for IfLadder { - fn deserialize>(de: D) -> std::result::Result { - let value = JsonValue::deserialize(de)?; - let obj = value - .as_object() - .ok_or_else(|| de::Error::custom("rules must be a JSON object"))?; - if obj.len() != 1 { - return Err(de::Error::custom( - "rules must be a single-key {\"if\": …} object", - )); - } - let (key, args) = obj.iter().next().expect("len == 1"); - if key != "if" { - return Err(de::Error::custom(format!( - "rules op must be 'if' (got '{key}')" - ))); - } - let args = args - .as_array() - .ok_or_else(|| de::Error::custom("'if' args must be an array"))?; - if args.is_empty() { - return Err(de::Error::custom( - "'if' args must be non-empty; clear `rules` to remove the ladder instead", - )); - } - if args.len() % 2 != 0 { - return Err(de::Error::custom( - "'if' args must have even length (alternating condition, severity); \ - a trailing else is not allowed", - )); - } - let mut branches = Vec::with_capacity(args.len() / 2); - for chunk in args.chunks_exact(2) { - let cond: Condition = - serde_json::from_value(chunk[0].clone()).map_err(de::Error::custom)?; - let sev_str = chunk[1].as_str().ok_or_else(|| { - de::Error::custom("each odd-indexed 'if' arg must be a severity string") - })?; - let sev: Severity = sev_str - .parse() - .map_err(|_| de::Error::custom(format!("invalid severity '{sev_str}'")))?; - branches.push((cond, sev)); - } - Ok(IfLadder { branches }) - } -} diff --git a/crates/database/src/issues.rs b/crates/database/src/issues.rs index 479256a0..c0e4b571 100644 --- a/crates/database/src/issues.rs +++ b/crates/database/src/issues.rs @@ -1,12 +1,11 @@ //! Issues, events, incidents. use commons_errors::{AppError, Result}; -use commons_types::issue::Severity; +use commons_types::status::CheckResult; use diesel::prelude::*; use diesel_async::{AsyncConnection, AsyncPgConnection, RunQueryDsl}; -use jiff::Timestamp; +use jiff::{SignedDuration, Timestamp}; use serde::{Deserialize, Serialize}; -use sha2::{Digest, Sha256}; use uuid::Uuid; use crate::{devices::Device, server_groups::ServerGroup, servers::Server}; @@ -55,9 +54,6 @@ pub struct Issue { #[diesel(column_name = "ref_")] #[serde(rename = "ref")] pub r#ref: String, - /// The issue's current severity. - #[diesel(deserialize_as = String, serialize_as = String)] - pub severity: Severity, /// A short, single-line title for the issue, shown as its headline in /// the UI and in Slack notifications. `None` if no title was given. pub description: Option, @@ -86,30 +82,73 @@ pub struct Issue { /// excluded from incidents and notifications even while still active. #[diesel(deserialize_as = jiff_diesel::NullableTimestamp, serialize_as = jiff_diesel::NullableTimestamp)] pub snoozed_until: Option, + /// The check this issue tracks, for issues that are check state (the + /// ref minus its namespace prefix). `None` for issues that predate the + /// check-state model or that no filing has stamped yet. + pub check_name: Option, + /// The result the source reported on the latest filing, before policy. + #[diesel(deserialize_as = MaybeCheckResult, serialize_as = Option)] + pub observed_result: Option, + /// What policy made of the latest observed result. This is the result + /// canopy acts on; the transitional `severity` is derived from it. + #[diesel(deserialize_as = MaybeCheckResult, serialize_as = Option)] + pub effective_result: Option, + /// The check's own fields from the latest report, verbatim (minus the + /// reserved keys), for display alongside the state. + pub detail: Option, + /// When the current degradation streak began, for state that is + /// currently degraded (effective warning/failed/broken). `None` while + /// healthy; a recovery clears it and a later degradation starts a + /// fresh streak. + #[diesel(deserialize_as = jiff_diesel::NullableTimestamp, serialize_as = jiff_diesel::NullableTimestamp)] + pub degraded_since: Option, + /// When this state last filed degraded. Never cleared: distinguishes + /// a recovered issue (worth listing) from always-healthy check state. + #[diesel(deserialize_as = jiff_diesel::NullableTimestamp, serialize_as = jiff_diesel::NullableTimestamp)] + pub last_degraded_at: Option, + /// Whether this check's policy escalates: an effective failure + /// notifies immediately, bypassing incident grace. Stamped from the + /// catalog on every filing. + pub escalates: bool, } -#[derive(Clone, Debug, Serialize, Deserialize, Queryable, Selectable, Associations)] -#[diesel(belongs_to(Issue))] -#[diesel(table_name = crate::schema::events)] -#[diesel(check_for_backend(diesel::pg::Pg))] -pub struct Event { - pub id: Uuid, - #[diesel(deserialize_as = jiff_diesel::Timestamp, serialize_as = jiff_diesel::Timestamp)] - pub created_at: Timestamp, - #[diesel(deserialize_as = jiff_diesel::NullableTimestamp, serialize_as = jiff_diesel::NullableTimestamp)] - pub occurred_at: Option, - pub issue_id: Uuid, - #[diesel(deserialize_as = String, serialize_as = String)] - pub severity: Severity, - /// Single-line title/subject. See [`NewEvent::description`]. - pub description: Option, - /// Body / long detail. See [`NewEvent::message`]. - pub message: String, - pub active: bool, - pub hash: Vec, - pub occurrences: i32, - #[diesel(deserialize_as = jiff_diesel::Timestamp, serialize_as = jiff_diesel::Timestamp)] - pub last_seen: Timestamp, +impl Issue { + /// Does this state open an incident on its own? An effective failure + /// does; anything less joins an already-open incident but doesn't + /// create (or hold open) one. + pub fn opens_incident(&self) -> bool { + self.effective_result == Some(CheckResult::Failed) + } + + /// Is this state an escalating failure right now — one whose + /// notification bypasses the incident grace period? + pub fn escalates_now(&self) -> bool { + self.opens_incident() && self.escalates + } +} + +/// Diesel helper: a nullable text column read as an optional +/// [`CheckResult`], treating unparseable text as `None` rather than +/// failing the whole row load. +pub struct MaybeCheckResult(Option); + +impl From for Option { + fn from(v: MaybeCheckResult) -> Self { + v.0 + } +} + +impl + diesel::deserialize::Queryable< + diesel::sql_types::Nullable, + diesel::pg::Pg, + > for MaybeCheckResult +{ + type Row = Option; + + fn build(row: Option) -> diesel::deserialize::Result { + Ok(Self(row.and_then(|s| s.parse().ok()))) + } } #[derive(Clone, Debug, Serialize, Deserialize, Queryable, Selectable, Associations)] @@ -122,7 +161,9 @@ pub struct Incident { pub created_at: Timestamp, #[diesel(deserialize_as = jiff_diesel::Timestamp, serialize_as = jiff_diesel::Timestamp)] pub updated_at: Timestamp, - pub server_group_id: Uuid, + /// The server group this incident targets, or `None` for a canopy-wide + /// incident (aggregating canopy-wide issues — self-alerts). + pub server_group_id: Option, #[diesel(deserialize_as = jiff_diesel::Timestamp, serialize_as = jiff_diesel::Timestamp)] pub opened_at: Timestamp, #[diesel(deserialize_as = jiff_diesel::NullableTimestamp, serialize_as = jiff_diesel::NullableTimestamp)] @@ -151,9 +192,8 @@ pub struct IncidentIssue { /// A single occurrence to report against an issue, sent either by a device /// or by an operator. If an issue with the same `source` and `ref` is -/// already open, this occurrence is folded into it (bumping its last-seen -/// time, and coalescing into the latest recorded event when the content is -/// identical); otherwise a new issue is created. +/// already open, this occurrence is folded into it (updating its state and +/// bumping its last-seen time); otherwise a new issue is created. #[derive(Debug, Clone, Deserialize, utoipa::ToSchema)] #[serde(rename_all = "camelCase")] pub struct NewEvent { @@ -165,9 +205,6 @@ pub struct NewEvent { /// Required — mint a UUID if deduplication isn't needed. #[serde(rename = "ref")] pub r#ref: String, - /// Severity of this event. Defaults to a standard severity if omitted. - #[serde(default)] - pub severity: Option, /// A short, single-line title for this event, shown as the issue's /// headline in the UI and as the subject of any Slack notification. /// Must not contain newlines. Use `message` for the full body text. @@ -200,42 +237,88 @@ pub enum IssueFilter { #[derive(Debug, Clone, Default)] pub struct IssueListFilters { pub active_only: bool, - pub severities: Option>, + /// Restrict to issues whose latest effective result is one of these. + pub results: Option>, /// Restrict to issues whose server belongs to this group. pub server_group_id: Option, /// When `Some`, restrict to issues last seen at or after this time. pub since: Option, } -fn hash_event( - severity: Severity, - active: bool, - message: &str, - description: Option<&str>, -) -> Vec { - let mut h = Sha256::new(); - h.update(severity.to_string().as_bytes()); - h.update([0]); - h.update([u8::from(active)]); - h.update([0]); - h.update(message.as_bytes()); - h.update([0]); - h.update(description.unwrap_or("").as_bytes()); - h.finalize().to_vec() +/// The check-state stamp accompanying a filing that is a check result: +/// which check, both sides of the policy transform, and the check's own +/// detail from the report. Filings that aren't check results (device +/// event pushes, producers not yet on the check-state model) carry no +/// stamp and leave the state columns null. +#[derive(Debug, Clone)] +pub struct CheckStateStamp { + pub check: String, + pub observed: CheckResult, + pub effective: CheckResult, + /// Whether the check's policy escalates (see [`Issue::escalates`]). + pub escalates: bool, + pub detail: Option, +} + +/// Write a filing's check-state stamp onto its issue row, maintaining the +/// degraded-streak timestamps: `degraded_since` holds while degraded +/// (starting a fresh streak on the healthy → degraded transition, using +/// `prior` — the row's value before this filing), clears on recovery; +/// `last_degraded_at` never clears. Returns the updated row. +async fn stamp_check_state( + conn: &mut AsyncPgConnection, + issue_id: Uuid, + prior: Option<(Option, Option)>, + stamp: &CheckStateStamp, + at: Timestamp, +) -> Result { + use crate::schema::issues; + + let degraded = matches!( + stamp.effective, + CheckResult::Warning | CheckResult::Failed | CheckResult::Broken + ); + let (prior_degraded_since, prior_last_degraded) = prior.unwrap_or((None, None)); + let degraded_since = if degraded { + Some(jiff_diesel::Timestamp::from( + prior_degraded_since.unwrap_or(at), + )) + } else { + None + }; + let last_degraded_at = if degraded { + Some(jiff_diesel::Timestamp::from(at)) + } else { + prior_last_degraded.map(jiff_diesel::Timestamp::from) + }; + diesel::update(issues::table.filter(issues::id.eq(issue_id))) + .set(( + issues::check_name.eq(&stamp.check), + issues::observed_result.eq(stamp.observed.to_string()), + issues::effective_result.eq(stamp.effective.to_string()), + issues::escalates.eq(stamp.escalates), + issues::detail.eq(&stamp.detail), + issues::degraded_since.eq(degraded_since), + issues::last_degraded_at.eq(last_degraded_at), + )) + .returning(Issue::as_select()) + .get_result(conn) + .await + .map_err(AppError::from) } impl NewEvent { /// Persist this event push: /// 1. find-or-create the issue keyed by (server_id, source, ref), - /// 2. append the event or coalesce into the latest matching one, - /// 3. if the server is in a group: (re)evaluate incident contribution. + /// updating its state from this report, + /// 2. if the server is in a group: (re)evaluate incident contribution. /// /// `server_id` is the server the issue is attached to: derived from the /// device for public submissions, supplied by the operator for manual. /// `device_id` is `None` for manual events. /// /// Issues from an **ungrouped** server are still recorded — the issue - /// and event rows go in just like any other push — but the incident + /// row goes in just like any other push — but the incident /// flow is skipped (`server_group_id` would have nowhere to point). /// When the operator later assigns the server to a group, /// [`Server::assign_to_group`] runs `reevaluate_open_issues_for_server` @@ -247,7 +330,20 @@ impl NewEvent { server_id: Uuid, device_id: Option, ) -> Result { - use crate::schema::{events, issues}; + self.save_with_state(db, server_id, device_id, None).await + } + + /// [`Self::save`], stamping the issue's check-state columns from a + /// check result. Status ingestion passes the stamp; everything else + /// files via [`Self::save`]. + pub async fn save_with_state( + self, + db: &mut AsyncPgConnection, + server_id: Uuid, + device_id: Option, + state: Option<&CheckStateStamp>, + ) -> Result { + use crate::schema::issues; // description is the single-line title; reject multi-line input // up front so the UI never has to fight with a multi-line @@ -261,12 +357,10 @@ impl NewEvent { )); } - let severity = self.severity.unwrap_or_default(); let active = self.active.unwrap_or(true); let now = Timestamp::now(); let effective_time = self.occurred_at.unwrap_or(now); let description = self.description.as_deref(); - let hash = hash_event(severity, active, &self.message, description); // Look up the server's group up-front; needed by the incident-open // path. If `None`, we still record the issue/event but skip @@ -304,10 +398,19 @@ impl NewEvent { // operator-resolved issue clears the resolved_* fields (issue is back // in unresolved state). let clear_resolved = active && existing.resolved_at.is_some(); + if let Some(stamp) = state { + stamp_check_state( + conn, + existing.id, + Some((existing.degraded_since, existing.last_degraded_at)), + stamp, + effective_time, + ) + .await?; + } let issue = diesel::update(issues::table.filter(issues::id.eq(existing.id))) .set(( issues::device_id.eq(device_id), - issues::severity.eq(severity), issues::description.eq(description), issues::message.eq(&self.message), issues::active.eq(active), @@ -339,13 +442,12 @@ impl NewEvent { .await?; issue } else { - diesel::insert_into(issues::table) + let inserted: Issue = diesel::insert_into(issues::table) .values(( issues::server_id.eq(server_id), issues::device_id.eq(device_id), issues::source.eq(&self.source), issues::ref_.eq(&self.r#ref), - issues::severity.eq(severity), issues::description.eq(description), issues::message.eq(&self.message), issues::active.eq(active), @@ -354,54 +456,28 @@ impl NewEvent { )) .returning(Issue::as_select()) .get_result(conn) - .await? - }; - - // 2. coalesce into latest event or insert new. - let latest_event: Option = events::table - .select(Event::as_select()) - .filter(events::issue_id.eq(issue.id)) - .order(events::created_at.desc()) - .first(conn) - .await - .optional()?; - - let coalesce = matches!(&latest_event, Some(e) if e.hash == hash); - if let (true, Some(latest)) = (coalesce, latest_event) { - let new_last = if effective_time > latest.last_seen { - effective_time - } else { - latest.last_seen - }; - diesel::update(events::table.filter(events::id.eq(latest.id))) - .set(( - events::occurrences.eq(latest.occurrences + 1), - events::last_seen.eq(jiff_diesel::Timestamp::from(new_last)), - )) - .execute(conn) .await?; - } else { - diesel::insert_into(events::table) - .values(( - events::issue_id.eq(issue.id), - events::occurred_at.eq(self.occurred_at.map(jiff_diesel::Timestamp::from)), - events::severity.eq(severity), - events::description.eq(description), - events::message.eq(&self.message), - events::active.eq(active), - events::hash.eq(&hash), - events::last_seen.eq(jiff_diesel::Timestamp::from(effective_time)), - )) - .execute(conn) - .await?; - } + match state { + Some(stamp) => { + stamp_check_state(conn, inserted.id, None, stamp, effective_time).await? + } + None => inserted, + } + }; - // 3. (re-)evaluate incident contribution against the new issue state. + // 2. (re-)evaluate incident contribution against the new issue state. // `by = None`: this came from a device push, not an operator action. // Skipped when the server is ungrouped: incidents are group-keyed. if let Some(gid) = server_group_id { - re_evaluate_incident_membership(conn, &issue, gid, monitored, effective_time, None) - .await?; + re_evaluate_incident_membership( + conn, + &issue, + IncidentTarget::Group(gid), + monitored, + effective_time, + None, + ) + .await?; } Ok(issue) @@ -418,8 +494,8 @@ impl NewEvent { /// Mirrors [`NewEvent::save`] but keys the issue on /// `(server_group_id, source, ref)` instead of `(server_id, …)`: /// 1. find-or-create the group-scoped issue (server_id = NULL), -/// 2. append the event or coalesce into the latest matching one, -/// 3. run incident membership evaluation with `monitored = true` so the +/// updating its state from this report, +/// 2. run incident membership evaluation with `monitored = true` so the /// incident opens/pages regardless of any member server's monitored flag. /// /// Recovery is the same `(source, ref)` with `active = false` at a lower @@ -427,18 +503,29 @@ impl NewEvent { /// identical lifecycle to the per-server path. /// /// `description` is an optional single-line headline (rejected if multi-line); -/// `message` is the body. Both feed the dedup hash, so a repeated identical -/// alert coalesces into one event with a bumped occurrence count. +/// `message` is the body. pub async fn raise_group_event( conn: &mut AsyncPgConnection, group_id: Uuid, r#ref: &str, - severity: Severity, description: Option<&str>, message: &str, active: bool, ) -> Result { - use crate::schema::{events, issues}; + raise_group_event_with_state(conn, group_id, r#ref, description, message, active, None).await +} + +/// [`raise_group_event`], stamping the issue's check-state columns. +pub async fn raise_group_event_with_state( + conn: &mut AsyncPgConnection, + group_id: Uuid, + r#ref: &str, + description: Option<&str>, + message: &str, + active: bool, + state: Option<&CheckStateStamp>, +) -> Result { + use crate::schema::issues; if let Some(d) = description && d.contains('\n') @@ -450,7 +537,6 @@ pub async fn raise_group_event( let source = crate::statuses::CANOPY_SOURCE; let now = Timestamp::now(); - let hash = hash_event(severity, active, message, description); conn.transaction::<_, AppError, _>(async |conn| { // 1. find-or-create the group-scoped issue. @@ -467,6 +553,9 @@ pub async fn raise_group_event( .await .optional()?; + let prior_state = existing + .as_ref() + .map(|e| (e.degraded_since, e.last_degraded_at)); let issue: Issue = if let Some(existing) = existing { let new_last_seen = if now > existing.last_seen { now @@ -476,7 +565,6 @@ pub async fn raise_group_event( let clear_resolved = active && existing.resolved_at.is_some(); diesel::update(issues::table.filter(issues::id.eq(existing.id))) .set(( - issues::severity.eq(severity), issues::description.eq(description), issues::message.eq(message), issues::active.eq(active), @@ -512,7 +600,6 @@ pub async fn raise_group_event( issues::server_group_id.eq(group_id), issues::source.eq(source), issues::ref_.eq(r#ref), - issues::severity.eq(severity), issues::description.eq(description), issues::message.eq(message), issues::active.eq(active), @@ -524,76 +611,497 @@ pub async fn raise_group_event( .await? }; - // 2. coalesce into latest event or insert new. - let latest_event: Option = events::table - .select(Event::as_select()) - .filter(events::issue_id.eq(issue.id)) - .order(events::created_at.desc()) + let issue = match state { + Some(stamp) => stamp_check_state(conn, issue.id, prior_state, stamp, now).await?, + None => issue, + }; + + // 2. group-aware incident evaluation — monitored = true unconditionally. + re_evaluate_incident_membership( + conn, + &issue, + IncidentTarget::Group(group_id), + true, + now, + None, + ) + .await?; + + Ok(issue) + }) + .await +} + +/// Open (or recover) a **canopy-wide** issue: one scoped to neither a +/// server nor a group, keyed `(source = canopy, ref)` under the global +/// partial unique index. This is the state store for canopy monitoring +/// its own operation (self-alerts). +/// +/// Mirrors [`raise_group_event`]'s find-or-create; incident evaluation +/// runs against the global target, so canopy-wide issues get the full +/// incident lifecycle (grace, escalation, resolution) like any other. +pub async fn raise_global_event( + conn: &mut AsyncPgConnection, + r#ref: &str, + description: Option<&str>, + message: &str, + active: bool, +) -> Result { + raise_global_event_with_state(conn, r#ref, description, message, active, None).await +} + +/// [`raise_global_event`], stamping the issue's check-state columns. +pub async fn raise_global_event_with_state( + conn: &mut AsyncPgConnection, + r#ref: &str, + description: Option<&str>, + message: &str, + active: bool, + state: Option<&CheckStateStamp>, +) -> Result { + use crate::schema::issues; + + if let Some(d) = description + && d.contains('\n') + { + return Err(AppError::BadRequest( + "description must be a single line (no newlines); use `message` for body text".into(), + )); + } + + let source = crate::statuses::CANOPY_SOURCE; + let now = Timestamp::now(); + + conn.transaction::<_, AppError, _>(async |conn| { + let existing: Option = issues::table + .select(Issue::as_select()) + .filter( + issues::server_id + .is_null() + .and(issues::server_group_id.is_null()) + .and(issues::source.eq(source)) + .and(issues::ref_.eq(r#ref)), + ) + .for_update() .first(conn) .await .optional()?; - let coalesce = matches!(&latest_event, Some(e) if e.hash == hash); - if let (true, Some(latest)) = (coalesce, latest_event) { - let new_last = if now > latest.last_seen { + let prior_state = existing + .as_ref() + .map(|e| (e.degraded_since, e.last_degraded_at)); + let issue: Issue = if let Some(existing) = existing { + let new_last_seen = if now > existing.last_seen { now } else { - latest.last_seen + existing.last_seen }; - diesel::update(events::table.filter(events::id.eq(latest.id))) + let clear_resolved = active && existing.resolved_at.is_some(); + diesel::update(issues::table.filter(issues::id.eq(existing.id))) .set(( - events::occurrences.eq(latest.occurrences + 1), - events::last_seen.eq(jiff_diesel::Timestamp::from(new_last)), + issues::description.eq(description), + issues::message.eq(message), + issues::active.eq(active), + issues::last_seen.eq(jiff_diesel::Timestamp::from(new_last_seen)), + issues::resolved_at.eq(diesel::dsl::sql::< + diesel::sql_types::Nullable, + >(if clear_resolved { + "NULL" + } else { + "issues.resolved_at" + })), + issues::resolved_by.eq(diesel::dsl::sql::< + diesel::sql_types::Nullable, + >(if clear_resolved { + "NULL" + } else { + "issues.resolved_by" + })), + issues::resolved_reason.eq(diesel::dsl::sql::< + diesel::sql_types::Nullable, + >(if clear_resolved { + "NULL" + } else { + "issues.resolved_reason" + })), )) - .execute(conn) - .await?; + .returning(Issue::as_select()) + .get_result(conn) + .await? } else { - diesel::insert_into(events::table) + diesel::insert_into(issues::table) .values(( - events::issue_id.eq(issue.id), - events::severity.eq(severity), - events::description.eq(description), - events::message.eq(message), - events::active.eq(active), - events::hash.eq(&hash), - events::last_seen.eq(jiff_diesel::Timestamp::from(now)), + issues::source.eq(source), + issues::ref_.eq(r#ref), + issues::description.eq(description), + issues::message.eq(message), + issues::active.eq(active), + issues::first_seen.eq(jiff_diesel::Timestamp::from(now)), + issues::last_seen.eq(jiff_diesel::Timestamp::from(now)), )) - .execute(conn) - .await?; - } + .returning(Issue::as_select()) + .get_result(conn) + .await? + }; - // 3. group-aware incident evaluation — monitored = true unconditionally. - re_evaluate_incident_membership(conn, &issue, group_id, true, now, None).await?; + let issue = match state { + Some(stamp) => stamp_check_state(conn, issue.id, prior_state, stamp, now).await?, + None => issue, + }; + + // Global incident evaluation — always monitored. + re_evaluate_incident_membership(conn, &issue, IncidentTarget::Global, true, now, None) + .await?; Ok(issue) }) .await } +/// Where a canopy-determined check's state attaches. +#[derive(Debug, Clone, Copy)] +pub enum FilingScope { + Server { + server_id: Uuid, + device_id: Option, + }, + Group(Uuid), + Global, +} + +/// The source operator-raised manual conditions file under. +pub const MANUAL_SOURCE: &str = "manual"; + +/// One canopy-determined or operator-raised check result to file: +/// reachability, backup health, key expiry, self-monitoring, manual +/// conditions, and the like. +#[derive(Debug, Clone)] +pub struct CheckFiling<'a> { + /// The reserved source this filing belongs to: [`MANUAL_SOURCE`] for + /// operator-raised conditions (server scope only), `canopy` for + /// canopy's own determinations. + pub source: &'a str, + pub scope: FilingScope, + /// The check's stable name (doubles as the issue ref under the + /// source): a contract with stored silences. + pub check: &'a str, + /// What was observed this pass. Policy grades it from there. + pub observed: CheckResult, + /// Single-line headline for degraded filings. + pub title: Option<&'a str>, + pub message: &'a str, + /// The check's own fields, available to policy rules as `check.*` + /// and displayed alongside the state. + pub detail: Option, + /// The policy this check registers with on first sight — the ceiling + /// and escalation its condition warrants. Operator edits stick; + /// these only seed the catalog row. + pub default_ceiling: CheckResult, + pub default_escalates: bool, + /// The documentation canopy's own checks ship with, seeded into the + /// catalog on first sight (never overwriting operator edits). See + /// the CHK spec's Documentation section for the convention. + pub documentation: Option<&'a str>, +} + +/// File one canopy-determined or operator-raised check result: register +/// its catalog entry (first sight only), grade the observation through +/// the operator's policy, and upsert the check state at the right scope +/// — driving incident membership exactly like a device-reported check. +/// +/// Group- and canopy-wide scopes are canopy's own (their raise paths +/// file under the `canopy` source); manual conditions are server-scoped. +/// +/// Until issues themselves carry results, the effective result maps to +/// the issue severity the same way status ingestion does: failed → +/// error (critical when the policy escalates), warning/broken → +/// warning; passed and skipped record healthy state and close. +pub async fn file_check(conn: &mut AsyncPgConnection, filing: CheckFiling<'_>) -> Result { + use crate::check_policies::{CheckPolicy, EvaluationContext}; + + let source = filing.source; + debug_assert!( + matches!(filing.scope, FilingScope::Server { .. }) + || source == crate::statuses::CANOPY_SOURCE, + "group- and canopy-wide filings are canopy's own", + ); + CheckPolicy::register( + conn, + source, + filing.check, + filing.default_ceiling, + filing.default_escalates, + filing.documentation, + ) + .await?; + + // Rule-evaluation context: the check's detail (with the normalised + // result injected, mirroring status ingestion), no report-wide + // extras, and the server's tags where there is a server. Filings + // whose observation policy shouldn't touch (an operator explicitly + // raising a manual condition) still flow through so the catalog row + // and stamps exist, but manual entries register at a failed ceiling + // so the operator's chosen result passes through ungraded by default. + let mut check_extra = filing + .detail + .as_ref() + .and_then(|d| d.as_object().cloned()) + .unwrap_or_default(); + check_extra.insert( + "result".into(), + serde_json::Value::String(filing.observed.to_string()), + ); + let status_extra = serde_json::Map::new(); + let (tags, scope_server, scope_group): ( + std::collections::HashMap, + Option, + Option, + ) = match filing.scope { + FilingScope::Server { server_id, .. } => { + let server = Server::get_by_id(conn, server_id).await?; + let group_id = server.group_id; + let tags = server + .tags_merged_with_group(conn) + .await? + .0 + .into_iter() + .map(|(k, v)| (k, serde_json::Value::String(v))) + .collect(); + (tags, Some(server_id), group_id) + } + FilingScope::Group(group_id) => (Default::default(), None, Some(group_id)), + FilingScope::Global => (Default::default(), None, None), + }; + let ctx = EvaluationContext { + status_extra: &status_extra, + check_extra: &check_extra, + tags: &tags, + }; + let graded = CheckPolicy::apply_scoped( + conn, + source, + filing.check, + filing.observed, + &ctx, + scope_server, + scope_group, + ) + .await?; + + let active = matches!( + graded.effective, + CheckResult::Failed | CheckResult::Warning | CheckResult::Broken + ); + let description = if active { filing.title } else { None }; + let stamp = CheckStateStamp { + check: filing.check.to_string(), + observed: filing.observed, + effective: graded.effective, + escalates: graded.escalates, + detail: filing.detail.clone(), + }; + + match filing.scope { + FilingScope::Server { + server_id, + device_id, + } => { + NewEvent { + source: source.to_string(), + r#ref: filing.check.to_string(), + description: description.map(str::to_string), + message: filing.message.to_string(), + active: Some(active), + occurred_at: None, + } + .save_with_state(conn, server_id, device_id, Some(&stamp)) + .await + } + FilingScope::Group(gid) => { + raise_group_event_with_state( + conn, + gid, + filing.check, + description, + filing.message, + active, + Some(&stamp), + ) + .await + } + FilingScope::Global => { + raise_global_event_with_state( + conn, + filing.check, + description, + filing.message, + active, + Some(&stamp), + ) + .await + } + } +} + +/// Per-server health rollup from current check state: the worst +/// effective result across every source's checks on the server — any +/// failure ⇒ unhealthy, otherwise any warning or brokenness ⇒ warning, +/// otherwise healthy. Silenced checks are skipped. Servers with no +/// check state are absent from the map (callers default them healthy, +/// matching the "no signal" semantics). +pub async fn health_from_check_state( + conn: &mut AsyncPgConnection, + servers: &[(Uuid, Option)], +) -> Result> { + use crate::schema::{issues, scoped_check_policies}; + use commons_types::status::HealthState; + use std::collections::{HashMap, HashSet}; + + let mut out: HashMap = HashMap::new(); + if servers.is_empty() { + return Ok(out); + } + let server_ids: Vec = servers.iter().map(|(id, _)| *id).collect(); + let group_of: HashMap> = servers.iter().copied().collect(); + + let rows: Vec<(Option, String, Option, Option)> = issues::table + .select(( + issues::server_id, + issues::source, + issues::check_name, + issues::effective_result, + )) + .filter(issues::server_id.eq_any(&server_ids)) + .filter(issues::check_name.is_not_null()) + .filter(issues::effective_result.is_not_null()) + .load(conn) + .await?; + + let group_ids: Vec = group_of.values().filter_map(|g| *g).collect(); + let silence_rows: Vec<(Option, Option, String, String)> = + scoped_check_policies::table + .select(( + scoped_check_policies::server_id, + scoped_check_policies::server_group_id, + scoped_check_policies::source, + scoped_check_policies::check_name, + )) + .filter(scoped_check_policies::ceiling.eq("skipped")) + .filter( + scoped_check_policies::server_id + .eq_any(&server_ids) + .or(scoped_check_policies::server_group_id.eq_any(&group_ids)), + ) + .load(conn) + .await?; + let mut server_silences: HashSet<(Uuid, String, String)> = HashSet::new(); + let mut group_silences: HashSet<(Uuid, String, String)> = HashSet::new(); + for (server_id, group_id, source, check) in silence_rows { + if let Some(sid) = server_id { + server_silences.insert((sid, source, check)); + } else if let Some(gid) = group_id { + group_silences.insert((gid, source, check)); + } + } + + for (server_id, source, check_name, effective) in rows { + let Some(server_id) = server_id else { + continue; + }; + let Some(check_name) = check_name else { + continue; + }; + let key = (server_id, source, check_name); + if server_silences.contains(&key) { + continue; + } + if let Some(Some(gid)) = group_of.get(&server_id) + && group_silences.contains(&(*gid, key.1.clone(), key.2.clone())) + { + continue; + } + let contribution = match effective.as_deref().and_then(|e| e.parse().ok()) { + Some(CheckResult::Failed) => HealthState::Unhealthy, + Some(CheckResult::Warning | CheckResult::Broken) => HealthState::Warning, + _ => continue, + }; + let entry = out.entry(server_id).or_insert(HealthState::Healthy); + if contribution == HealthState::Unhealthy || *entry == HealthState::Healthy { + *entry = contribution; + } + } + + Ok(out) +} + +impl Issue { + /// Server-scoped check state for one (source, check). The per-check + /// attention page's data source: rows carry the observed/effective + /// results, the check's detail, and the degraded-streak timestamps. + /// A check's identity is the pair — a same-named check from another + /// source is a different check. + pub async fn check_state_for_check( + conn: &mut AsyncPgConnection, + source: &str, + check_name: &str, + ) -> Result> { + use crate::schema::issues::dsl; + + dsl::issues + .select(Issue::as_select()) + .filter(dsl::source.eq(source)) + .filter(dsl::check_name.eq(check_name)) + .filter(dsl::server_id.is_not_null()) + .filter(dsl::observed_result.is_not_null()) + .load(conn) + .await + .map_err(AppError::from) + } +} + +/// The canopy-wide issue at `(canopy, ref)`, if it has ever been raised. +pub async fn get_global_issue(conn: &mut AsyncPgConnection, r#ref: &str) -> Result> { + use crate::schema::issues::dsl; + + dsl::issues + .select(Issue::as_select()) + .filter( + dsl::server_id + .is_null() + .and(dsl::server_group_id.is_null()) + .and(dsl::source.eq(crate::statuses::CANOPY_SOURCE)) + .and(dsl::ref_.eq(r#ref)), + ) + .first(conn) + .await + .optional() + .map_err(AppError::from) +} + /// Compute whether the issue *should* currently be contributing to an /// open incident, and apply join/leave accordingly. The rules: /// /// - **Leave**: `!active || resolved || snoozed || silenced || !monitored`. -/// Severity downgrade alone does *not* remove an issue — once +/// A result downgrade alone does *not* remove an issue — once /// contributing, it stays until it's actually gone or explicitly /// suppressed. Flipping the server to unmonitored *does* remove it: the /// operator has said they're not watching this server, so its issues -/// stop counting. The same applies if `(source, ref)` is on the server -/// or group silence list — see [`crate::silenced_refs`]. +/// stop counting. The same applies if the check is silenced at server +/// or group scope — see [`crate::silenced_refs`]. /// - **Join**: not leaving, AND one of: -/// - severity ≥ floor (`error`), so this issue is high-priority enough to -/// create a new incident on its own; or -/// - the group already has an open incident — then any active issue, -/// even low-severity ones, joins it. The threshold only governs -/// incident *creation*; once an incident is in progress everything else -/// piles in for context. -/// - **Close**: the incident auto-closes when the last contributor at -/// severity ≥ error leaves. Low-severity contributors that joined -/// because the group had an open incident stay attached (so the audit -/// trail and Slack thread retain them) but **do not** hold the incident -/// open by themselves. Without this asymmetry, a check that's stuck -/// firing at warning could keep an incident open indefinitely after the -/// higher-severity contributor that opened it has long since gone away. +/// - the state opens incidents on its own — an effective failure (see +/// [`Issue::opens_incident`]); or +/// - the target already has an open incident — then any active issue, +/// warnings included, joins it. The threshold only governs incident +/// *creation*; once an incident is in progress everything else piles +/// in for context. +/// - **Close**: the incident auto-closes when the last effective-failure +/// contributor leaves. Lesser contributors that joined because the +/// target had an open incident stay attached (so the audit trail and +/// Slack thread retain them) but **do not** hold the incident open by +/// themselves. Without this asymmetry, a check that's stuck firing at +/// warning could keep an incident open indefinitely after the failure +/// that opened it has long since gone away. /// /// `monitored` reflects the server's `is_monitored()` at call time. When /// `false`, the issue is treated as a "leave": this is what makes @@ -609,7 +1117,7 @@ pub async fn raise_group_event( async fn re_evaluate_incident_membership( conn: &mut AsyncPgConnection, issue: &Issue, - server_group_id: Uuid, + target: IncidentTarget, monitored: bool, transition_time: Timestamp, by: Option<&str>, @@ -620,39 +1128,30 @@ async fn re_evaluate_incident_membership( let snoozed = issue.snoozed_until.map_or(false, |t| t > Timestamp::now()); // A group-scoped issue (server_id = None) can only be silenced at the // group level; pass the nil server so only the group list is consulted. + // Canopy-wide issues have no silence scope (yet). let silenced = crate::silenced_refs::is_silenced( conn, issue.server_id.unwrap_or(Uuid::nil()), - Some(server_group_id), + target.group_id(), &issue.source, &issue.r#ref, ) .await?; - let group_open = group_has_open_incident(conn, server_group_id).await?; - - // Debug-severity issues are intentionally invisible to the incident - // workflow: they don't join, and any that are currently attached - // (because their catalog/rule severity dropped to Debug after they - // joined) get treated as a leave on next re-evaluation. - let is_debug = issue.severity == Severity::Debug; - let should_leave = is_debug - || !issue.active - || issue.resolved_at.is_some() - || snoozed - || silenced - || !monitored; - let should_join = !is_debug - && monitored + let target_open = target_has_open_incident(conn, target).await?; + + let should_leave = + !issue.active || issue.resolved_at.is_some() || snoozed || silenced || !monitored; + let should_join = monitored && !silenced && issue.active && issue.resolved_at.is_none() && !snoozed - && (issue.severity.opens_incident() || group_open); + && (issue.opens_incident() || target_open); match (was_in, should_join, should_leave) { (false, true, _) => { let (incident_id, newly_opened) = - find_or_open_incident(conn, server_group_id, transition_time).await?; + find_or_open_incident(conn, target, transition_time).await?; diesel::insert_into(incident_issues::table) .values(( incident_issues::incident_id.eq(incident_id), @@ -662,9 +1161,9 @@ async fn re_evaluate_incident_membership( .execute(conn) .await?; if newly_opened { - enqueue_slack_open(conn, incident_id, server_group_id, issue).await?; - } else if issue.severity == Severity::Critical { - // Two sub-cases when a Critical joins an existing incident: + enqueue_slack_open(conn, incident_id, target, issue).await?; + } else if issue.escalates_now() { + // Two sub-cases when an escalating failure joins an existing incident: // - The original open is still pending in the outbox → // accelerate so the "incident opened" message lands // immediately. No second message: the open hasn't been @@ -687,7 +1186,7 @@ async fn re_evaluate_incident_membership( .await .optional()?; if escalated.is_some() { - enqueue_slack_open(conn, incident_id, server_group_id, issue).await?; + enqueue_slack_open(conn, incident_id, target, issue).await?; } } } @@ -733,14 +1232,10 @@ async fn re_evaluate_incident_membership( .execute(conn) .await?; - // Only count contributors that are *currently* at a severity - // that opens an incident. Low-severity issues stay attached - // for context but don't hold the incident open on their own; - // see the function doc-comment for the rationale. - let opens_incident_severities: Vec = Severity::OPENS_INCIDENT - .iter() - .map(|s| s.to_string()) - .collect(); + // Only count contributors that *currently* open an incident + // (effective failures). Lesser contributors stay attached for + // context but don't hold the incident open on their own; see + // the function doc-comment for the rationale. use crate::schema::issues; let remaining_open: i64 = incident_issues::table .inner_join(issues::table.on(issues::id.eq(incident_issues::issue_id))) @@ -748,17 +1243,17 @@ async fn re_evaluate_incident_membership( incident_issues::incident_id .eq(open_link.incident_id) .and(incident_issues::left_at.is_null()) - .and(issues::severity.eq_any(&opens_incident_severities)), + .and(issues::effective_result.eq("failed")), ) .count() .get_result(conn) .await?; if remaining_open == 0 { // Filter on `closed_at IS NULL` so that when a stranded - // low-severity contributor eventually leaves an already- - // closed incident (because the severity-filter close above - // already retired it), we skip both the no-op update and - // the double Slack resolve. + // lesser contributor eventually leaves an already-closed + // incident (because the failure-filter close above already + // retired it), we skip both the no-op update and the + // double Slack resolve. let closed: Option = diesel::update( incidents::table .filter(incidents::id.eq(open_link.incident_id)) @@ -779,8 +1274,8 @@ async fn re_evaluate_incident_membership( Ok(()) } -/// Resolve the `(server_group_id, monitored)` pair an issue should be -/// re-evaluated against, handling both server-scoped and group-scoped issues. +/// Resolve the `(target, monitored)` pair an issue should be re-evaluated +/// against, handling all three scopes. /// /// - Server-scoped (`server_id = Some`): look the server up; its `group_id` /// and `is_monitored` drive the evaluation. `None` group → ungrouped, no @@ -788,17 +1283,20 @@ async fn re_evaluate_incident_membership( /// - Group-scoped (`server_group_id = Some`): use the group directly and force /// `monitored = true` so the per-server gate never silences a control-plane /// issue. -async fn issue_group_and_monitored( +/// - Canopy-wide (neither): the global target, always monitored. +async fn issue_target_and_monitored( conn: &mut AsyncPgConnection, issue: &Issue, -) -> Result> { +) -> Result> { match (issue.server_id, issue.server_group_id) { - (_, Some(gid)) => Ok(Some((gid, true))), + (_, Some(gid)) => Ok(Some((IncidentTarget::Group(gid), true))), (Some(sid), None) => { let server = Server::get_by_id(conn, sid).await?; - Ok(server.group_id.map(|gid| (gid, server.is_monitored))) + Ok(server + .group_id + .map(|gid| (IncidentTarget::Group(gid), server.is_monitored))) } - (None, None) => Ok(None), + (None, None) => Ok(Some((IncidentTarget::Global, true))), } } @@ -832,7 +1330,15 @@ pub async fn reevaluate_open_issues_for_server( let now = Timestamp::now(); for issue in open_issues { - re_evaluate_incident_membership(db, &issue, gid, monitored, now, None).await?; + re_evaluate_incident_membership( + db, + &issue, + IncidentTarget::Group(gid), + monitored, + now, + None, + ) + .await?; } Ok(()) } @@ -867,7 +1373,15 @@ pub async fn reevaluate_open_issues_for_server_ref( let now = Timestamp::now(); for issue in open_issues { - re_evaluate_incident_membership(db, &issue, gid, monitored, now, None).await?; + re_evaluate_incident_membership( + db, + &issue, + IncidentTarget::Group(gid), + monitored, + now, + None, + ) + .await?; } Ok(()) } @@ -919,7 +1433,15 @@ pub async fn reevaluate_open_issues_for_group_ref( Some(sid) => monitored_by_server.get(&sid).copied().unwrap_or(true), None => true, }; - re_evaluate_incident_membership(db, &issue, server_group_id, monitored, now, None).await?; + re_evaluate_incident_membership( + db, + &issue, + IncidentTarget::Group(server_group_id), + monitored, + now, + None, + ) + .await?; } Ok(()) } @@ -973,7 +1495,15 @@ pub async fn reconcile_open_incidents(db: &mut AsyncPgConnection) -> Result<(usi // Group-scoped issue: resolve its group directly, bypass the // per-server is_monitored gate (monitored = true). (None, Some(gid)) => { - re_evaluate_incident_membership(conn, &issue, gid, true, now, None).await?; + re_evaluate_incident_membership( + conn, + &issue, + IncidentTarget::Group(gid), + true, + now, + None, + ) + .await?; evaluated += 1; } // Server-scoped issue: look up the server and its group. @@ -988,7 +1518,7 @@ pub async fn reconcile_open_incidents(db: &mut AsyncPgConnection) -> Result<(usi re_evaluate_incident_membership( conn, &issue, - gid, + IncidentTarget::Group(gid), server.is_monitored, now, None, @@ -996,7 +1526,19 @@ pub async fn reconcile_open_incidents(db: &mut AsyncPgConnection) -> Result<(usi .await?; evaluated += 1; } - (None, None) => continue, + // Canopy-wide issue: the global target, always monitored. + (None, None) => { + re_evaluate_incident_membership( + conn, + &issue, + IncidentTarget::Global, + true, + now, + None, + ) + .await?; + evaluated += 1; + } } } Ok((by_id.len(), evaluated)) @@ -1018,18 +1560,45 @@ async fn is_issue_in_open_incident(db: &mut AsyncPgConnection, issue_id: Uuid) - Ok(count > 0) } -async fn group_has_open_incident( +/// What an issue's incident contribution attaches to: its server's group, +/// or canopy as a whole for canopy-wide issues (self-alerts). Issues on +/// ungrouped servers have no target and no incident path. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum IncidentTarget { + Group(Uuid), + Global, +} + +impl IncidentTarget { + /// The `incidents.server_group_id` value for this target. + fn group_id(self) -> Option { + match self { + Self::Group(gid) => Some(gid), + Self::Global => None, + } + } + + /// The target an existing incident row belongs to. + pub fn of_incident(incident: &Incident) -> Self { + match incident.server_group_id { + Some(gid) => Self::Group(gid), + None => Self::Global, + } + } +} + +async fn target_has_open_incident( db: &mut AsyncPgConnection, - server_group_id: Uuid, + target: IncidentTarget, ) -> Result { use crate::schema::incidents::dsl; - let count: i64 = dsl::incidents - .filter(dsl::server_group_id.eq(server_group_id)) - .filter(dsl::closed_at.is_null()) - .count() - .get_result(db) - .await?; + let mut q = dsl::incidents.filter(dsl::closed_at.is_null()).into_boxed(); + q = match target { + IncidentTarget::Group(gid) => q.filter(dsl::server_group_id.eq(gid)), + IncidentTarget::Global => q.filter(dsl::server_group_id.is_null()), + }; + let count: i64 = q.count().get_result(db).await?; Ok(count > 0) } @@ -1038,32 +1607,46 @@ async fn group_has_open_incident( /// whether a Slack `incident_open` outbox row should be enqueued (re-joining /// an existing incident shouldn't re-notify). /// -/// Two parallel event pushes against the same group must not each insert a -/// fresh incident row. We take a `FOR UPDATE` lock on the group's row -/// up-front so the find-or-create pair serializes per-group. A unique -/// partial index on `incidents (server_group_id) WHERE closed_at IS NULL` -/// is the belt-and-braces backstop at the DB layer. +/// Two parallel event pushes against the same target must not each insert +/// a fresh incident row. For a group target we take a `FOR UPDATE` lock on +/// the group's row up-front so the find-or-create pair serializes +/// per-group; the global target has no row to lock, so it serializes on a +/// transaction-scoped advisory lock instead. The unique partial indexes on +/// open incidents are the belt-and-braces backstop at the DB layer. async fn find_or_open_incident( db: &mut AsyncPgConnection, - server_group_id: Uuid, + target: IncidentTarget, opened_at: Timestamp, ) -> Result<(Uuid, bool)> { use crate::schema::{incidents, server_groups}; - let _group_lock: Uuid = server_groups::table - .select(server_groups::id) - .filter(server_groups::id.eq(server_group_id)) - .for_update() - .first(db) - .await?; + match target { + IncidentTarget::Group(gid) => { + let _group_lock: Uuid = server_groups::table + .select(server_groups::id) + .filter(server_groups::id.eq(gid)) + .for_update() + .first(db) + .await?; + } + IncidentTarget::Global => { + // Arbitrary constant, stable across releases: the one global + // incident slot. + diesel::sql_query("SELECT pg_advisory_xact_lock(818_723_001)") + .execute(db) + .await?; + } + } - let open: Option = incidents::table + let mut q = incidents::table .select(Incident::as_select()) - .filter( - incidents::server_group_id - .eq(server_group_id) - .and(incidents::closed_at.is_null()), - ) + .filter(incidents::closed_at.is_null()) + .into_boxed(); + q = match target { + IncidentTarget::Group(gid) => q.filter(incidents::server_group_id.eq(gid)), + IncidentTarget::Global => q.filter(incidents::server_group_id.is_null()), + }; + let open: Option = q .order(incidents::opened_at.desc()) .first(db) .await @@ -1074,7 +1657,7 @@ async fn find_or_open_incident( let new_incident: Incident = diesel::insert_into(incidents::table) .values(( - incidents::server_group_id.eq(server_group_id), + incidents::server_group_id.eq(target.group_id()), incidents::opened_at.eq(jiff_diesel::Timestamp::from(opened_at)), )) .returning(Incident::as_select()) @@ -1083,35 +1666,55 @@ async fn find_or_open_incident( Ok((new_incident.id, true)) } +/// Flap grace for canopy-wide incidents, mirroring the per-group +/// `slack_open_delay` default (the global target has no config row). +const GLOBAL_OPEN_GRACE: SignedDuration = SignedDuration::from_secs(3 * 60); + async fn enqueue_slack_open( conn: &mut AsyncPgConnection, incident_id: Uuid, - server_group_id: Uuid, + target: IncidentTarget, issue: &Issue, ) -> Result<()> { - let group = ServerGroup::get_by_id(conn, server_group_id).await?; - let server = match issue.server_id { - Some(sid) => Some(Server::get_by_id(conn, sid).await?), - None => None, + let (label, open_delay) = match target { + IncidentTarget::Group(gid) => { + let group = ServerGroup::get_by_id(conn, gid).await?; + let server = match issue.server_id { + Some(sid) => Some(Server::get_by_id(conn, sid).await?), + None => None, + }; + ( + format_group_label(&group, server.as_ref()), + group.slack_open_delay.0, + ) + } + IncidentTarget::Global => ("Canopy".to_string(), GLOBAL_OPEN_GRACE), + }; + // The deployed Slack workflow's trigger declares a `severity` + // variable; feed it the result-derived urgency label. + let urgency = if issue.escalates_now() { + "Critical" + } else if issue.opens_incident() { + "Error" + } else { + "Warning" }; - let label = format_group_label(&group, server.as_ref()); let payload = crate::slack_outbox::vars::incident_open( &label, - issue.severity, + urgency, &issue.source, &issue.r#ref, &issue.message, ); - // Normally the row sits in the outbox for the group's - // `slack_open_delay` before the drainer can ship it — that's the - // flap-suppression window, so a transient open/close pair never - // reaches Slack. Critical bypasses this: it's the - // "stop-the-world" tier where operators have signalled they don't - // want any delay. - let deliver_after = if issue.severity == Severity::Critical { + // Normally the row sits in the outbox for the target's open delay + // before the drainer can ship it — that's the flap-suppression + // window, so a transient open/close pair never reaches Slack. + // An escalating failure bypasses this: the operator has marked the + // check's policy as escalating — they don't want any delay. + let deliver_after = if issue.escalates_now() { Timestamp::now() } else { - Timestamp::now() + group.slack_open_delay.0 + Timestamp::now() + open_delay }; crate::slack_outbox::SlackOutbox::enqueue( conn, @@ -1184,8 +1787,13 @@ async fn enqueue_slack_resolve_inner( if cancelled > 0 { return Ok(()); } - let group = ServerGroup::get_by_id(conn, incident.server_group_id).await?; - let label = format_group_label(&group, None); + let label = match incident.server_group_id { + Some(gid) => { + let group = ServerGroup::get_by_id(conn, gid).await?; + format_group_label(&group, None) + } + None => "Canopy".to_string(), + }; let payload = crate::slack_outbox::vars::incident_resolve(&label, by); crate::slack_outbox::SlackOutbox::enqueue( conn, @@ -1227,6 +1835,9 @@ impl Issue { let mut q = dsl::issues .select(Self::as_select()) .filter(dsl::device_id.eq(device_id)) + // Healthy check state (rows that never degraded) isn't an + // issue; it has its own read surfaces. + .filter(dsl::active.eq(true).or(dsl::last_degraded_at.is_not_null())) .into_boxed(); if filter == IssueFilter::ActiveOnly { q = q @@ -1251,6 +1862,9 @@ impl Issue { let mut q = dsl::issues .select(Self::as_select()) .filter(dsl::server_id.eq(server_id)) + // Healthy check state (rows that never degraded) isn't an + // issue; it has its own read surfaces. + .filter(dsl::active.eq(true).or(dsl::last_degraded_at.is_not_null())) .into_boxed(); if filter == IssueFilter::ActiveOnly { q = q @@ -1269,7 +1883,8 @@ impl Issue { /// - `active_only`: when true, only `active = true` *and* unresolved /// issues — operator-resolved items don't count even if the source /// keeps pushing them. - /// - `severities`: when `Some` and non-empty, restrict to those. + /// - `results`: when `Some` and non-empty, restrict to issues whose + /// latest effective result is one of those. /// - `server_group_id`: when `Some`, restrict to issues whose server is /// in that group. A direct `IN (SELECT id FROM servers WHERE group_id = $)`. pub async fn list( @@ -1282,18 +1897,26 @@ impl Issue { let mut q = dsl::issues .select(Self::as_select()) - // Self-alerts (the nil "Canopy" server's issues) have their own - // surface (`crate::self_alerts`); they are not fleet issues. - .filter(dsl::server_id.is_distinct_from(Uuid::nil())) + // Canopy-wide issues (self-alerts: neither server- nor + // group-scoped) have their own surface (`crate::self_alerts`); + // they are not fleet issues. + .filter( + dsl::server_id + .is_not_null() + .or(dsl::server_group_id.is_not_null()), + ) + // Healthy check state (rows that never degraded) isn't an + // issue; it has its own read surfaces. + .filter(dsl::active.eq(true).or(dsl::last_degraded_at.is_not_null())) .into_boxed(); if filters.active_only { q = q .filter(dsl::active.eq(true)) .filter(dsl::resolved_at.is_null()); } - if let Some(sevs) = filters.severities.as_ref().filter(|v| !v.is_empty()) { - let strs: Vec = sevs.iter().map(|s| s.to_string()).collect(); - q = q.filter(dsl::severity.eq_any(strs)); + if let Some(results) = filters.results.as_ref().filter(|v| !v.is_empty()) { + let strs: Vec = results.iter().map(|r| r.to_string()).collect(); + q = q.filter(dsl::effective_result.eq_any(strs)); } if let Some(gid) = filters.server_group_id { let server_ids: Vec = servers::table @@ -1403,6 +2026,72 @@ impl Issue { .map_err(AppError::from) } + /// Like [`Self::list_by_source_ref`], but matching any of several refs + /// at once. Used by the staleness sweep, whose per-source check refs + /// are only known at runtime. + pub async fn list_by_source_refs( + db: &mut AsyncPgConnection, + source: &str, + refs: &[String], + server_ids: &[Uuid], + ) -> Result> { + use crate::schema::issues::dsl; + if server_ids.is_empty() || refs.is_empty() { + return Ok(Vec::new()); + } + dsl::issues + .select(Self::as_select()) + .filter( + dsl::source + .eq(source) + .and(dsl::ref_.eq_any(refs)) + .and(dsl::server_id.eq_any(server_ids)), + ) + .load(db) + .await + .map_err(AppError::from) + } + + /// Most recent check-state stamp per (server, reporting source): every + /// report a source pushes re-stamps `last_seen` on the state rows of + /// the checks it mentions, so the max per source is when that source + /// last reported — maintained incrementally by ingestion, with no scan + /// of the statuses history. The reserved sources are excluded (canopy + /// and manual filings aren't reports), as are rows never stamped by + /// the check-state model. + pub async fn source_freshness( + db: &mut AsyncPgConnection, + server_ids: &[Uuid], + ) -> Result> { + use crate::schema::issues::dsl; + if server_ids.is_empty() { + return Ok(Vec::new()); + } + let rows: Vec<(Uuid, String, jiff_diesel::Timestamp)> = dsl::issues + .filter( + dsl::server_id + .eq_any(server_ids) + .and( + dsl::source + .ne_all([crate::statuses::CANOPY_SOURCE, crate::issues::MANUAL_SOURCE]), + ) + .and(dsl::check_name.is_not_null()), + ) + .group_by((dsl::server_id, dsl::source)) + .select(( + dsl::server_id.assume_not_null(), + dsl::source, + diesel::dsl::max(dsl::last_seen).assume_not_null(), + )) + .load(db) + .await + .map_err(AppError::from)?; + Ok(rows + .into_iter() + .map(|(server, source, seen)| (server, source, seen.into())) + .collect()) + } + /// Mark an issue as operator-resolved. Triggers incident-membership /// re-evaluation (typically: leaves the incident). pub async fn resolve( @@ -1424,8 +2113,8 @@ impl Issue { .returning(Self::as_select()) .get_result(conn) .await?; - if let Some((gid, monitored)) = issue_group_and_monitored(conn, &issue).await? { - re_evaluate_incident_membership(conn, &issue, gid, monitored, now, Some(by)) + if let Some((target, monitored)) = issue_target_and_monitored(conn, &issue).await? { + re_evaluate_incident_membership(conn, &issue, target, monitored, now, Some(by)) .await?; } Ok(issue) @@ -1448,8 +2137,8 @@ impl Issue { .get_result(conn) .await?; // Unresolve rejoins; cascade close path doesn't fire here. - if let Some((gid, monitored)) = issue_group_and_monitored(conn, &issue).await? { - re_evaluate_incident_membership(conn, &issue, gid, monitored, now, None).await?; + if let Some((target, monitored)) = issue_target_and_monitored(conn, &issue).await? { + re_evaluate_incident_membership(conn, &issue, target, monitored, now, None).await?; } Ok(issue) }) @@ -1477,8 +2166,8 @@ impl Issue { .returning(Self::as_select()) .get_result(conn) .await?; - if let Some((gid, monitored)) = issue_group_and_monitored(conn, &issue).await? { - re_evaluate_incident_membership(conn, &issue, gid, monitored, now, None).await?; + if let Some((target, monitored)) = issue_target_and_monitored(conn, &issue).await? { + re_evaluate_incident_membership(conn, &issue, target, monitored, now, None).await?; } Ok(issue) }) @@ -1495,8 +2184,8 @@ impl Issue { .returning(Self::as_select()) .get_result(conn) .await?; - if let Some((gid, monitored)) = issue_group_and_monitored(conn, &issue).await? { - re_evaluate_incident_membership(conn, &issue, gid, monitored, now, None).await?; + if let Some((target, monitored)) = issue_target_and_monitored(conn, &issue).await? { + re_evaluate_incident_membership(conn, &issue, target, monitored, now, None).await?; } Ok(issue) }) @@ -1517,7 +2206,6 @@ pub struct IssueIncidentRef { #[derive(Debug, Clone, Copy, Default)] pub struct IncidentStats { pub issue_count: i64, - pub event_count: i64, /// `incident_notes` for this incident + `issue_notes` across all linked issues. pub note_count: i64, } @@ -1529,18 +2217,18 @@ impl Incident { /// `incident_issues` is keyed on `(incident_id, issue_id, joined_at)`, /// so an issue that leaves and rejoins the same incident produces /// multiple rows for the same pair. Counts must dedupe on the pair - /// before joining to `events` or `issue_notes`, otherwise every event - /// or note gets multiplied by the rejoin count. + /// before joining to `issue_notes`, otherwise every note gets + /// multiplied by the rejoin count. /// /// Strategy: pull the distinct `(incident_id, issue_id)` pairs first, - /// then run the per-issue event/note counts and the direct - /// incident_notes count concurrently. Takes the pool (`&Db`) so the - /// parallel futures don't fight over one mutable conn handle. + /// then run the per-issue note counts and the direct incident_notes + /// count concurrently. Takes the pool (`&Db`) so the parallel futures + /// don't fight over one mutable conn handle. pub async fn stats_for( pool: &crate::Db, incident_ids: &[Uuid], ) -> Result> { - use crate::schema::{events, incident_issues, incident_notes, issue_notes}; + use crate::schema::{incident_issues, incident_notes, issue_notes}; use diesel::dsl::count_star; use std::collections::{HashMap, HashSet}; @@ -1582,19 +2270,6 @@ impl Incident { .into_iter() .collect(); - let f_events = async { - if unique_issue_ids.is_empty() { - return Result::>::Ok(Vec::new()); - } - let mut c = pool.get().await?; - events::table - .group_by(events::issue_id) - .select((events::issue_id, count_star())) - .filter(events::issue_id.eq_any(&unique_issue_ids)) - .load::<(Uuid, i64)>(&mut c) - .await - .map_err(AppError::from) - }; let f_inotes = async { let mut c = pool.get().await?; incident_notes::table @@ -1618,16 +2293,13 @@ impl Incident { .await .map_err(AppError::from) }; - let (event_rows, inote_rows, jnote_rows) = - futures::try_join!(f_events, f_inotes, f_jnotes)?; + let (inote_rows, jnote_rows) = futures::try_join!(f_inotes, f_jnotes)?; - let events_per_issue: HashMap = event_rows.into_iter().collect(); let notes_per_issue: HashMap = jnote_rows.into_iter().collect(); for (incident_id, issues) in &issues_by_incident { let entry = out.entry(*incident_id).or_default(); for issue_id in issues { - entry.event_count += events_per_issue.get(issue_id).copied().unwrap_or(0); entry.note_count += notes_per_issue.get(issue_id).copied().unwrap_or(0); } } @@ -1639,38 +2311,6 @@ impl Incident { } } -impl Event { - pub async fn list_for_issue( - db: &mut AsyncPgConnection, - issue_id: Uuid, - offset: i64, - limit: i64, - ) -> Result> { - use crate::schema::events::dsl; - - dsl::events - .select(Self::as_select()) - .filter(dsl::issue_id.eq(issue_id)) - .order(dsl::created_at.desc()) - .offset(offset) - .limit(limit) - .load(db) - .await - .map_err(AppError::from) - } - - pub async fn count_for_issue(db: &mut AsyncPgConnection, issue_id: Uuid) -> Result { - use crate::schema::events::dsl; - - dsl::events - .filter(dsl::issue_id.eq(issue_id)) - .count() - .get_result(db) - .await - .map_err(AppError::from) - } -} - impl Incident { /// Incidents are owned by the server's group, so a caller asking for a /// specific server's incidents really wants the group's. Look up the @@ -1868,7 +2508,7 @@ impl Incident { .filter(incidents::id.eq(incident_id)) .first(conn) .await?; - let gid = incident_loaded.server_group_id; + let target = IncidentTarget::of_incident(&incident_loaded); let open_issue_ids: Vec = incident_issues::table .select(incident_issues::issue_id) @@ -1908,7 +2548,7 @@ impl Incident { Some(sid) => Server::get_by_id(conn, sid).await?.is_monitored, None => true, }; - re_evaluate_incident_membership(conn, &issue, gid, monitored, now, Some(by)) + re_evaluate_incident_membership(conn, &issue, target, monitored, now, Some(by)) .await?; } diff --git a/crates/database/src/lib.rs b/crates/database/src/lib.rs index a31e4f36..0c18b840 100644 --- a/crates/database/src/lib.rs +++ b/crates/database/src/lib.rs @@ -10,9 +10,9 @@ pub mod artifacts; pub mod backup; pub mod backups; pub mod bestool_snippets; +pub mod check_policies; pub mod chrome_releases; pub mod devices; -pub mod healthcheck_severities; pub mod issues; pub mod mcp_tokens; pub mod notes; diff --git a/crates/database/src/mcp_tokens.rs b/crates/database/src/mcp_tokens.rs index 2ec2f7fc..95b567da 100644 --- a/crates/database/src/mcp_tokens.rs +++ b/crates/database/src/mcp_tokens.rs @@ -193,12 +193,24 @@ impl McpToken { /// refs, this is a contract: silences and Slack messages reference it. pub const EXPIRY_REF: &str = "mcp-token-expiry"; +pub const EXPIRY_DOC: &str = "## Description + +One or more MCP access tokens (bearer tokens agents use against the public /mcp endpoint) are inside the rotation lead window of their fixed one-year expiry. + +## Results + +- **fail** — an un-revoked token expires soon (or has expired); recovers once every such token is rotated or revoked. + +## Solve + +Mint a replacement token in Settings, update the agent using the old one, then revoke it. The message lists each expiring token by name and minter."; + /// Rotation alert, run from the monitor loop: a [self-alert](crate::self_alerts) /// — one coalescing record, one notification per raise/recovery, never one per /// group. Active while any un-revoked token is inside [`EXPIRY_ALERT_LEAD`] of /// its expiry; recovers when none are. Idle sweeps write nothing. pub async fn sweep_token_expiry(db: &mut AsyncPgConnection) -> Result { - use commons_types::issue::Severity; + use commons_types::status::CheckResult; let expiring = McpToken::expiring_soon(db).await?; @@ -236,7 +248,10 @@ pub async fn sweep_token_expiry(db: &mut AsyncPgConnection) -> Result { crate::self_alerts::raise( db, EXPIRY_REF, - Severity::Error, + CheckResult::Failed, + CheckResult::Failed, + false, + Some(EXPIRY_DOC), "MCP access token nearing expiry", &message, ) diff --git a/crates/database/src/restore.rs b/crates/database/src/restore.rs index 829cb1a7..efe48a81 100644 --- a/crates/database/src/restore.rs +++ b/crates/database/src/restore.rs @@ -8,7 +8,7 @@ use std::collections::HashMap; use commons_errors::{AppError, Result}; use commons_types::backup::{BackupType, IntentDescriptor, RestoreIntent, RunOutcome, semantics}; -use commons_types::issue::Severity; +use commons_types::status::CheckResult; use diesel::{ prelude::*, result::{DatabaseErrorKind, Error as DieselError}, @@ -18,9 +18,9 @@ use jiff::Timestamp; use serde::Serialize; use uuid::Uuid; -use crate::backup::alerts::raise_group_event; use crate::backup::refs; use crate::backups::BackupRun; +use crate::issues::{CheckFiling, FilingScope, file_check}; use crate::pg_duration::PgDuration; /// An operator's declaration that a restore consumer should maintain a @@ -452,16 +452,22 @@ async fn recover_old_scope_alerts( for sid in servers { let r#ref = restore_verification_ref(sid, old_type, old_intent); - raise_group_event( + file_check( db, - old_group_id, - &r#ref, - Severity::Info, - None, - &format!( - "Restore verification no longer tracked at this scope: {old_type} / {old_intent} for server {sid}" - ), - false, + CheckFiling { + source: crate::statuses::CANOPY_SOURCE, + scope: FilingScope::Group(old_group_id), + check: &r#ref, + observed: CheckResult::Passed, + title: None, + message: &format!( + "Restore verification no longer tracked at this scope: {old_type} / {old_intent} for server {sid}" + ), + detail: None, + default_ceiling: CheckResult::Failed, + default_escalates: false, + documentation: Some(refs::RESTORE_VERIFICATION_DOC), + }, ) .await?; } @@ -591,32 +597,51 @@ impl BackupRestoreCheck { if let Some(sid) = server_id { let r#ref = restore_verification_ref(sid, &r#type, &intent); if healthy { - raise_group_event( + file_check( db, - group_id, - &r#ref, - Severity::Info, - None, - &format!("Restore verification healthy: {type} / {intent} for server {sid}"), - false, + CheckFiling { + source: crate::statuses::CANOPY_SOURCE, + scope: FilingScope::Group(group_id), + check: &r#ref, + observed: CheckResult::Passed, + title: None, + message: &format!( + "Restore verification healthy: {type} / {intent} for server {sid}" + ), + detail: None, + default_ceiling: CheckResult::Failed, + default_escalates: false, + documentation: Some(refs::RESTORE_VERIFICATION_DOC), + }, ) .await?; } else { - let detail = + let error_detail = error.unwrap_or_else(|| "restored database did not come up healthy".into()); let snap = snapshot_id + .clone() .map(|s| format!(" (snapshot {s})")) .unwrap_or_default(); - raise_group_event( + file_check( db, - group_id, - &r#ref, - Severity::Error, - Some("restore verification failed"), - &format!( - "Restore verification failed: {type} / {intent} for server {sid}{snap}: {detail}" - ), - true, + CheckFiling { + source: crate::statuses::CANOPY_SOURCE, + scope: FilingScope::Group(group_id), + check: &r#ref, + observed: CheckResult::Failed, + title: Some("restore verification failed"), + message: &format!( + "Restore verification failed: {type} / {intent} for server {sid}{snap}: {error_detail}" + ), + detail: Some(serde_json::json!({ + "type": r#type.to_string(), + "intent": intent.to_string(), + "snapshot_id": snapshot_id, + })), + default_ceiling: CheckResult::Failed, + default_escalates: false, + documentation: Some(refs::RESTORE_VERIFICATION_DOC), + }, ) .await?; } @@ -846,14 +871,24 @@ pub async fn sweep_overdue(db: &mut AsyncPgConnection) -> Result { d.r#type, d.intent ) }; - raise_group_event( + file_check( db, - d.group_id, - &r#ref, - Severity::Error, - Some("restore verification overdue"), - &message, - true, + CheckFiling { + source: crate::statuses::CANOPY_SOURCE, + scope: FilingScope::Group(d.group_id), + check: &r#ref, + observed: CheckResult::Failed, + title: Some("restore verification overdue"), + message: &message, + detail: Some(serde_json::json!({ + "type": d.r#type.to_string(), + "intent": d.intent.to_string(), + "latest_snapshot_unverified": once, + })), + default_ceiling: CheckResult::Failed, + default_escalates: false, + documentation: Some(refs::RESTORE_VERIFICATION_DOC), + }, ) .await?; filed += 1; diff --git a/crates/database/src/schema.rs b/crates/database/src/schema.rs index 7ec4304f..1334d4ab 100644 --- a/crates/database/src/schema.rs +++ b/crates/database/src/schema.rs @@ -169,6 +169,22 @@ diesel::table! { } } +diesel::table! { + check_policies (source, check_name) { + source -> Text, + check_name -> Text, + ceiling -> Text, + escalates -> Bool, + rules -> Nullable, + notes -> Nullable, + first_seen -> Timestamptz, + reviewed_at -> Nullable, + reviewed_by -> Nullable, + updated_at -> Timestamptz, + documentation -> Nullable, + } +} + diesel::table! { chrome_releases (version) { version -> Text, @@ -223,35 +239,6 @@ diesel::table! { } } -diesel::table! { - events (id) { - id -> Uuid, - created_at -> Timestamptz, - occurred_at -> Nullable, - issue_id -> Uuid, - severity -> Text, - description -> Nullable, - message -> Text, - active -> Bool, - hash -> Bytea, - occurrences -> Int4, - last_seen -> Timestamptz, - } -} - -diesel::table! { - healthcheck_severities (check_name) { - check_name -> Text, - severity -> Text, - first_seen -> Timestamptz, - reviewed_at -> Nullable, - reviewed_by -> Nullable, - notes -> Nullable, - updated_at -> Timestamptz, - rules -> Nullable, - } -} - diesel::table! { incident_issues (incident_id, issue_id, joined_at) { incident_id -> Uuid, @@ -281,7 +268,7 @@ diesel::table! { resolved_at -> Nullable, resolved_by -> Nullable, resolved_reason -> Nullable, - server_group_id -> Uuid, + server_group_id -> Nullable, escalated_at -> Nullable, } } @@ -306,7 +293,6 @@ diesel::table! { source -> Text, #[sql_name = "ref"] ref_ -> Text, - severity -> Text, description -> Nullable, message -> Text, active -> Bool, @@ -317,6 +303,13 @@ diesel::table! { resolved_reason -> Nullable, snoozed_until -> Nullable, server_group_id -> Nullable, + check_name -> Nullable, + observed_result -> Nullable, + effective_result -> Nullable, + detail -> Nullable, + degraded_since -> Nullable, + last_degraded_at -> Nullable, + escalates -> Bool, } } @@ -371,6 +364,21 @@ diesel::table! { } } +diesel::table! { + scoped_check_policies (id) { + id -> Uuid, + created_at -> Timestamptz, + updated_at -> Timestamptz, + source -> Text, + check_name -> Text, + server_id -> Nullable, + server_group_id -> Nullable, + ceiling -> Nullable, + rules -> Nullable, + created_by -> Nullable, + } +} + diesel::table! { server_backup_capabilities (server_id, type_) { server_id -> Uuid, @@ -438,17 +446,6 @@ diesel::table! { } } -diesel::table! { - server_group_silenced_refs (server_group_id, source, ref_) { - server_group_id -> Uuid, - source -> Text, - #[sql_name = "ref"] - ref_ -> Text, - created_at -> Timestamptz, - created_by -> Nullable, - } -} - diesel::table! { server_groups (id) { id -> Uuid, @@ -464,17 +461,6 @@ diesel::table! { } } -diesel::table! { - server_silenced_refs (server_id, source, ref_) { - server_id -> Uuid, - source -> Text, - #[sql_name = "ref"] - ref_ -> Text, - created_at -> Timestamptz, - created_by -> Nullable, - } -} - diesel::table! { servers (id) { id -> Uuid, @@ -495,7 +481,6 @@ diesel::table! { is_monitored -> Bool, deleted_at -> Nullable, registered_at -> Nullable, - allow_legacy_status -> Bool, restore_allowed_until -> Nullable, restore_allowed_by -> Nullable, } @@ -538,6 +523,7 @@ diesel::table! { device_id -> Nullable, healthy -> Bool, health -> Jsonb, + source -> Text, } } @@ -603,7 +589,6 @@ diesel::joinable!(device_connections -> devices (device_id)); diesel::joinable!(device_keys -> devices (device_id)); diesel::joinable!(device_server_associations -> devices (device_id)); diesel::joinable!(device_server_associations -> servers (server_id)); -diesel::joinable!(events -> issues (issue_id)); diesel::joinable!(incident_issues -> incidents (incident_id)); diesel::joinable!(incident_issues -> issues (issue_id)); diesel::joinable!(incident_notes -> incidents (incident_id)); @@ -616,13 +601,13 @@ diesel::joinable!(restore_consumer_capabilities -> devices (consumer_device_id)) diesel::joinable!(restore_replicas -> devices (consumer_device_id)); diesel::joinable!(restore_replicas -> server_groups (group_id)); diesel::joinable!(restore_replicas -> servers (server_id)); +diesel::joinable!(scoped_check_policies -> server_groups (server_group_id)); +diesel::joinable!(scoped_check_policies -> servers (server_id)); diesel::joinable!(server_backup_capabilities -> servers (server_id)); diesel::joinable!(server_enrollment_challenges -> servers (server_id)); diesel::joinable!(server_enrollment_tokens -> servers (server_id)); diesel::joinable!(server_group_backup_config -> server_groups (group_id)); diesel::joinable!(server_group_backup_schedule -> server_groups (group_id)); -diesel::joinable!(server_group_silenced_refs -> server_groups (server_group_id)); -diesel::joinable!(server_silenced_refs -> servers (server_id)); diesel::joinable!(servers -> devices (device_id)); diesel::joinable!(slack_outbox -> incident_notes (note_id)); diesel::joinable!(slack_outbox -> incidents (incident_id)); @@ -643,13 +628,12 @@ diesel::allow_tables_to_appear_in_same_query!( backup_runs, backup_type_defaults, bestool_snippets, + check_policies, chrome_releases, device_connections, device_keys, device_server_associations, devices, - events, - healthcheck_severities, incident_issues, incident_notes, incidents, @@ -659,14 +643,13 @@ diesel::allow_tables_to_appear_in_same_query!( recovery_vault_writes, restore_consumer_capabilities, restore_replicas, + scoped_check_policies, server_backup_capabilities, server_enrollment_challenges, server_enrollment_tokens, server_group_backup_config, server_group_backup_schedule, - server_group_silenced_refs, server_groups, - server_silenced_refs, servers, slack_outbox, sql_playground_history, diff --git a/crates/database/src/self_alerts.rs b/crates/database/src/self_alerts.rs index 60fdcfc2..6ab25656 100644 --- a/crates/database/src/self_alerts.rs +++ b/crates/database/src/self_alerts.rs @@ -2,87 +2,74 @@ //! //! Spec: `.workhorse/specs/private-server/self-alerts.md` (id `SELF`). //! -//! Each condition is one coalescing issue on the nil "Canopy" server — -//! which is never grouped, so these never touch the per-group incident -//! flow. Notification goes straight to the Slack outbox instead -//! ([`crate::slack_outbox::KIND_SELF_ALERT_OPEN`] / -//! [`KIND_SELF_ALERT_RESOLVE`](crate::slack_outbox::KIND_SELF_ALERT_RESOLVE)), -//! with the same flap grace incidents get: sub-Critical raises wait out -//! [`GRACE`] before shipping, and a recovery inside that window cancels the -//! pending open so Slack hears nothing at all. +//! Each condition is one coalescing canopy-wide issue — scoped to +//! neither a server nor a group. Notification rides the incident +//! machinery: an error-or-worse condition opens an incident on the +//! canopy-wide target, with the same grace, escalation, and recovery +//! rules as any group incident (see +//! [`crate::issues::raise_global_event`]). use commons_errors::Result; -use commons_types::issue::Severity; +use commons_types::status::CheckResult; use diesel_async::AsyncPgConnection; -use jiff::{SignedDuration, Timestamp}; -use uuid::Uuid; -use crate::issues::{Issue, NewEvent}; -use crate::slack_outbox::{KIND_SELF_ALERT_OPEN, KIND_SELF_ALERT_RESOLVE, SlackOutbox, vars}; -use crate::statuses::CANOPY_SOURCE; +use crate::issues::{CheckFiling, FilingScope, Issue, file_check, get_global_issue}; /// An operator-notification delivery permanently failed (the drainer gave up /// on an outbox row). No automatic recovery: stays until operator-resolved. pub const SLACK_DELIVERY_FAILURE_REF: &str = "slack-delivery-failure"; -/// Flap grace for sub-Critical self-alerts, mirroring the per-group -/// `slack_open_delay` default. -pub const GRACE: SignedDuration = SignedDuration::from_secs(3 * 60); +pub const SLACK_DELIVERY_FAILURE_DOC: &str = "## Description -/// Raise (or re-affirm) a self-alert. Files the coalescing event against the -/// nil server; on the not-alerting → alerting transition — including a -/// re-raise of an issue an operator resolved while the condition still held — -/// enqueues the Slack notification. Repeated raises while alerting change -/// nothing Slack-side. +The Slack outbox drainer gave up delivering a notification after exhausting its retries — operators may be missing incident notices. + +## Results + +- **fail** — at least one outbox row was abandoned; recovers when a later delivery succeeds. + +## Solve + +Check the abandoned row's last error and response in the slack_outbox table, and the webhook URLs in the drainer's configuration. Slack workflow-trigger changes are the usual cause."; + +/// Raise (or re-affirm) a self-alert: file the coalescing canopy-wide +/// check with the given observation, registering its catalog entry with +/// the policy the condition warrants (first sight only — operator edits +/// stick). The incident machinery handles notification — a fresh +/// effective failure opens (or joins) the canopy-wide incident, and +/// repeated raises while alerting change nothing Slack-side. +#[allow(clippy::too_many_arguments)] pub async fn raise( conn: &mut AsyncPgConnection, r#ref: &str, - severity: Severity, + observed: CheckResult, + default_ceiling: CheckResult, + default_escalates: bool, + documentation: Option<&str>, title: &str, message: &str, ) -> Result { - let was_alerting = current(conn, r#ref) - .await? - .map(|i| i.active && i.resolved_at.is_none()) - .unwrap_or(false); - - let issue = NewEvent { - source: CANOPY_SOURCE.into(), - r#ref: r#ref.into(), - severity: Some(severity), - description: Some(title.into()), - message: message.into(), - active: Some(true), - occurred_at: None, - } - .save(conn, Uuid::nil(), None) - .await?; - - if !was_alerting { - let deliver_after = if severity == Severity::Critical { - Timestamp::now() - } else { - Timestamp::now() + GRACE - }; - SlackOutbox::enqueue( - conn, - KIND_SELF_ALERT_OPEN, - None, - Some(issue.id), - None, - vars::self_alert_open(severity, r#ref, title, message), - deliver_after, - ) - .await?; - } - Ok(issue) + file_check( + conn, + CheckFiling { + source: crate::statuses::CANOPY_SOURCE, + scope: FilingScope::Global, + check: r#ref, + observed, + title: Some(title), + message, + detail: None, + default_ceiling, + default_escalates, + documentation, + }, + ) + .await } -/// Recover a self-alert. Writes nothing when the alert isn't active. On the -/// active → inactive transition, cancels a still-pending open (a flap inside -/// [`GRACE`] makes no Slack noise) or enqueues the recovery notification — -/// unless an operator had already resolved the issue, in which case Slack -/// stays quiet too. +/// Recover a self-alert. Writes nothing when the alert isn't active. On +/// the active → inactive transition the issue leaves the canopy-wide +/// incident, which closes (and notifies, or cancels a still-pending open) +/// once its last contributor is gone. pub async fn recover( conn: &mut AsyncPgConnection, r#ref: &str, @@ -94,55 +81,35 @@ pub async fn recover( if !existing.active { return Ok(None); } - let was_alerting = existing.resolved_at.is_none(); - let issue = NewEvent { - source: CANOPY_SOURCE.into(), - r#ref: r#ref.into(), - severity: Some(Severity::Info), - description: None, - message: message.into(), - active: Some(false), - occurred_at: None, - } - .save(conn, Uuid::nil(), None) + // The catalog entry exists (the active issue implies a prior raise + // registered it), so the defaults here are inert. + let issue = file_check( + conn, + CheckFiling { + source: crate::statuses::CANOPY_SOURCE, + scope: FilingScope::Global, + check: r#ref, + observed: CheckResult::Passed, + title: None, + message, + detail: None, + default_ceiling: CheckResult::Warning, + default_escalates: false, + documentation: None, + }, + ) .await?; - - if was_alerting { - let cancelled = SlackOutbox::cancel_pending_self_alert_open( - conn, - issue.id, - "cancelled: self-alert recovered before the open had been delivered to Slack", - ) - .await?; - if cancelled == 0 { - SlackOutbox::enqueue( - conn, - KIND_SELF_ALERT_RESOLVE, - None, - Some(issue.id), - None, - vars::self_alert_resolve(r#ref, message), - Timestamp::now(), - ) - .await?; - } - } Ok(Some(issue)) } /// The one issue for this condition, if it has ever been raised. pub async fn current(conn: &mut AsyncPgConnection, r#ref: &str) -> Result> { - Ok( - Issue::list_by_source_ref(conn, CANOPY_SOURCE, r#ref, &[Uuid::nil()]) - .await? - .into_iter() - .next(), - ) + get_global_issue(conn, r#ref).await } -/// All self-alert issues (the nil server's), newest activity first. The -/// operator UI's alerts view; the fleet issue listings exclude these. +/// All self-alert issues (the canopy-wide ones), newest activity first. +/// The operator UI's alerts view; the fleet issue listings exclude these. pub async fn list(conn: &mut AsyncPgConnection, limit: i64) -> Result> { use crate::schema::issues::dsl; use diesel::prelude::*; @@ -150,7 +117,7 @@ pub async fn list(conn: &mut AsyncPgConnection, limit: i64) -> Result dsl::issues .select(Issue::as_select()) - .filter(dsl::server_id.eq(Some(Uuid::nil()))) + .filter(dsl::server_id.is_null().and(dsl::server_group_id.is_null())) .order(dsl::last_seen.desc()) .limit(limit) .load(conn) diff --git a/crates/database/src/servers.rs b/crates/database/src/servers.rs index 7f86c47e..22bf9314 100644 --- a/crates/database/src/servers.rs +++ b/crates/database/src/servers.rs @@ -90,15 +90,6 @@ pub struct Server { /// `alert_when_down_for` threshold is preserved while unmonitored, so /// turning monitoring back on doesn't lose the configured value. pub is_monitored: bool, - /// Opt-in to accepting the older status-report format that doesn't - /// include a healthchecks array. Off by default, in which case a push in - /// the old format is rejected. When `true`, an old-format push is - /// accepted but only refreshes reachability — it carries the server's - /// last known healthchecks forward instead of clearing them, so a server - /// that's switching between old and new reporters doesn't flap its - /// health issues. Turn this off once the server reports in the current - /// format. - pub allow_legacy_status: bool, /// How long, in seconds, a server's status may go without an update /// before it's considered unreachable and an issue is filed. Increase /// it for servers with flaky connectivity; decrease it for critical @@ -825,7 +816,6 @@ fn test_server_serialization() { cloud: None, geolocation: None, is_monitored: true, - allow_legacy_status: false, alert_when_down_for: TEN_MINUTES, notes: String::new(), tags: TagMap::default(), @@ -847,7 +837,6 @@ fn test_server_serialization() { "device_id": "00000000-0000-0000-0000-000000000000", "public_name": "Test Server", "is_monitored": true, - "allow_legacy_status": false, "alert_when_down_for": 600, "notes": "", "tags": {} @@ -890,7 +879,6 @@ impl From for Server { cloud: None, geolocation: None, is_monitored: true, - allow_legacy_status: false, alert_when_down_for: TEN_MINUTES, notes: String::new(), tags: TagMap::default(), @@ -937,8 +925,6 @@ pub struct PartialServer { pub geolocation: Option>, /// New monitored state for the server. pub is_monitored: Option, - /// New legacy-status-format opt-in for the server. - pub allow_legacy_status: Option, /// New downtime threshold for the server, in seconds. #[schema(value_type = Option)] #[diesel(serialize_as = PgDuration)] diff --git a/crates/database/src/silenced_refs.rs b/crates/database/src/silenced_refs.rs index cdf4965e..c016b99c 100644 --- a/crates/database/src/silenced_refs.rs +++ b/crates/database/src/silenced_refs.rs @@ -1,57 +1,66 @@ -//! Operator-managed silence list for issue refs. +//! Operator-managed silences, stored as scoped check policies. //! -//! A silenced `(source, ref)` tuple at server or group scope tells the -//! incident workflow to ignore the matching issues — they still record -//! (so the issue and event rows exist), but -//! [`crate::issues::re_evaluate_incident_membership`] treats them as a -//! "should leave" reason, the same way it treats snoozed or unmonitored. -//! Healthcheck silences (`(status, health/)`) additionally drop the -//! check out of the server's health rollup — see -//! [`silenced_health_checks_for_servers`] and -//! [`crate::statuses::Status::health_state_ignoring`]. +//! A silence is a scoped transform with a `skipped` ceiling (see +//! [`crate::check_policies::ScopedCheckPolicy`]): the matching check +//! keeps recording its observed results, but its effective result is +//! skipped, so it raises nothing and counts nowhere — +//! [`crate::issues::re_evaluate_incident_membership`] treats it as a +//! "should leave" reason, and the health rollups drop it. //! -//! Two sibling tables (`server_silenced_refs`, `server_group_silenced_refs`) -//! keep referential integrity tight without nullable FKs. A given issue is -//! silenced if either applies (server-scope wins for the server itself, -//! group-scope catches the whole group). +//! This module keeps the historical (source, ref) surface the private +//! API and UI speak: refs carry the `health/` namespace prefix for +//! source-reported checks, while the scoped-policy storage is keyed by +//! bare check name. The mapping is applied on the way in and out. -use std::collections::{BTreeSet, HashMap}; +use std::collections::BTreeSet; -use commons_errors::{AppError, Result}; +use commons_errors::Result; use diesel::prelude::*; use diesel_async::{AsyncPgConnection, RunQueryDsl}; use jiff::Timestamp; use serde::{Deserialize, Serialize}; use uuid::Uuid; -use crate::issues::{reevaluate_open_issues_for_group_ref, reevaluate_open_issues_for_server_ref}; +use crate::check_policies::{PolicyScope, ScopedCheckPolicy}; +use crate::issues::{ + MANUAL_SOURCE, reevaluate_open_issues_for_group_ref, reevaluate_open_issues_for_server_ref, +}; +use crate::statuses::CANOPY_SOURCE; -/// The `source` the public-server status ingestion files healthcheck -/// issues under, so a silence on a healthcheck is `(status, -/// health/)`. Mirrors the public-server's `STATUS_SOURCE`. -const STATUS_SOURCE: &str = "status"; - -/// The ref prefix (with trailing separator) healthcheck issues use -/// under [`STATUS_SOURCE`]. Mirrors the public-server's `HEALTH_REF`. +/// The ref prefix (with trailing separator) healthcheck issues use, +/// whichever source reports them. Mirrors the public-server's `HEALTH_REF`. const HEALTH_REF_PREFIX: &str = "health/"; +/// The check name a silence ref maps to: refs of source-reported checks +/// carry the `health/` namespace prefix, canopy/manual refs are already +/// bare check names. +fn ref_to_check(r#ref: &str) -> &str { + r#ref.strip_prefix(HEALTH_REF_PREFIX).unwrap_or(r#ref) +} + +/// The ref a silenced check name presents as: reserved sources file at +/// bare refs, everything else under the `health/` namespace. +fn check_to_ref(source: &str, check: &str) -> String { + if source == CANOPY_SOURCE || source == MANUAL_SOURCE { + check.to_string() + } else { + format!("{HEALTH_REF_PREFIX}{check}") + } +} + /// A silenced issue reference scoped to a single server: issues matching /// this `(source, ref)` on this server are still recorded, but are excluded /// from incidents and notifications. -#[derive(Debug, Clone, Serialize, Deserialize, Queryable, Selectable, utoipa::ToSchema)] -#[diesel(table_name = crate::schema::server_silenced_refs)] -#[diesel(check_for_backend(diesel::pg::Pg))] +#[derive(Debug, Clone, Serialize, Deserialize, utoipa::ToSchema)] pub struct ServerSilencedRef { /// The server this silence applies to. pub server_id: Uuid, /// The issue source this silence matches. pub source: String, /// The issue reference this silence matches. - #[diesel(column_name = ref_)] #[serde(rename = "ref")] pub r#ref: String, /// When this silence was created. - #[diesel(deserialize_as = jiff_diesel::Timestamp, serialize_as = jiff_diesel::Timestamp)] pub created_at: Timestamp, /// The operator who created this silence. `None` if not recorded. pub created_by: Option, @@ -61,29 +70,25 @@ pub struct ServerSilencedRef { /// matching this `(source, ref)` on any server in the group (or raised /// directly against the group) are still recorded, but are excluded from /// incidents and notifications. -#[derive(Debug, Clone, Serialize, Deserialize, Queryable, Selectable, utoipa::ToSchema)] -#[diesel(table_name = crate::schema::server_group_silenced_refs)] -#[diesel(check_for_backend(diesel::pg::Pg))] +#[derive(Debug, Clone, Serialize, Deserialize, utoipa::ToSchema)] pub struct ServerGroupSilencedRef { /// The server group this silence applies to. pub server_group_id: Uuid, /// The issue source this silence matches. pub source: String, /// The issue reference this silence matches. - #[diesel(column_name = ref_)] #[serde(rename = "ref")] pub r#ref: String, /// When this silence was created. - #[diesel(deserialize_as = jiff_diesel::Timestamp, serialize_as = jiff_diesel::Timestamp)] pub created_at: Timestamp, /// The operator who created this silence. `None` if not recorded. pub created_by: Option, } -/// Does either the server-scope or group-scope silence list contain -/// `(source, ref)` for this server? `group_id` is the server's current -/// group; pass `None` if the server is ungrouped (and so can't be silenced -/// at group scope). +/// Is a silence in force for `(source, ref)` on this server, at either +/// server or group scope? `group_id` is the server's current group; pass +/// `None` if the server is ungrouped (and so can't be silenced at group +/// scope). pub async fn is_silenced( db: &mut AsyncPgConnection, server_id: Uuid, @@ -91,189 +96,67 @@ pub async fn is_silenced( source: &str, r#ref: &str, ) -> Result { - use crate::schema::{server_group_silenced_refs, server_silenced_refs}; - - let server_hit: i64 = server_silenced_refs::table - .filter( - server_silenced_refs::server_id - .eq(server_id) - .and(server_silenced_refs::source.eq(source)) - .and(server_silenced_refs::ref_.eq(r#ref)), - ) - .count() - .get_result(db) - .await - .map_err(AppError::from)?; - if server_hit > 0 { + let check = ref_to_check(r#ref); + let is_silence = + |p: Option| p.is_some_and(|p| p.ceiling.as_deref() == Some("skipped")); + if is_silence(ScopedCheckPolicy::get(db, PolicyScope::Server(server_id), source, check).await?) + { return Ok(true); } - let Some(gid) = group_id else { return Ok(false); }; - - let group_hit: i64 = server_group_silenced_refs::table - .filter( - server_group_silenced_refs::server_group_id - .eq(gid) - .and(server_group_silenced_refs::source.eq(source)) - .and(server_group_silenced_refs::ref_.eq(r#ref)), - ) - .count() - .get_result(db) - .await - .map_err(AppError::from)?; - Ok(group_hit > 0) + Ok(is_silence( + ScopedCheckPolicy::get(db, PolicyScope::Group(gid), source, check).await?, + )) } -/// All refs silenced for this server under `source` and starting with -/// `ref_prefix`, combining the server's own silence list with its group's. -/// `group_id` is the server's current group; pass `None` if the server is -/// ungrouped. Used to build the device-facing effective check-severity map. -/// May contain duplicates when a ref is silenced at both scopes. -pub async fn silenced_refs_with_prefix( +/// Check names silenced for a server under one reporting source, at +/// either server or group scope. `group_id` is the server's current +/// group; pass `None` if ungrouped. A check's identity is the (source, +/// check) pair, so a silence on another source's same-named check never +/// applies. +/// +/// This feeds [`crate::statuses::Status::health_state_ignoring`] (a +/// status row's checks all belong to the row's one source) and the +/// device-facing effective check map: a silenced check keeps recording +/// results but is presented as skipped and doesn't count toward the +/// server's health rollup. +pub async fn silenced_health_checks_for_server( db: &mut AsyncPgConnection, server_id: Uuid, group_id: Option, source: &str, - ref_prefix: &str, -) -> Result> { - use crate::schema::{server_group_silenced_refs, server_silenced_refs}; - - debug_assert!( - !ref_prefix.contains(['%', '_', '\\']), - "prefix is used in a LIKE pattern and must not contain wildcards" - ); +) -> Result> { + use crate::schema::scoped_check_policies::dsl; - let mut refs: Vec = server_silenced_refs::table - .select(server_silenced_refs::ref_) + let rows: Vec = dsl::scoped_check_policies + .select(dsl::check_name) + .filter(dsl::ceiling.eq("skipped")) + .filter(dsl::source.eq(source)) .filter( - server_silenced_refs::server_id - .eq(server_id) - .and(server_silenced_refs::source.eq(source)) - .and(server_silenced_refs::ref_.like(format!("{ref_prefix}%"))), + dsl::server_id.eq(server_id).or(dsl::server_group_id + .is_not_distinct_from(group_id) + .and(dsl::server_group_id.is_not_null())), ) .load(db) - .await - .map_err(AppError::from)?; - - if let Some(gid) = group_id { - let group_refs: Vec = server_group_silenced_refs::table - .select(server_group_silenced_refs::ref_) - .filter( - server_group_silenced_refs::server_group_id - .eq(gid) - .and(server_group_silenced_refs::source.eq(source)) - .and(server_group_silenced_refs::ref_.like(format!("{ref_prefix}%"))), - ) - .load(db) - .await - .map_err(AppError::from)?; - refs.extend(group_refs); - } - - Ok(refs) -} - -/// Healthcheck names silenced for each of the given `(server, group)` -/// pairs, at either scope: the `` of every `(status, -/// health/)` silence entry that applies to the server. Pass each -/// server's current group id (`None` for ungrouped). Two batch queries, -/// one per scope table, regardless of how many servers are asked about. -/// Servers with no applicable silences are absent from the map. -/// -/// This feeds [`crate::statuses::Status::health_state_ignoring`]: a -/// silenced check keeps recording results but is presented as skipped -/// and doesn't count toward the server's health rollup. -pub async fn silenced_health_checks_for_servers( - db: &mut AsyncPgConnection, - servers: &[(Uuid, Option)], -) -> Result>> { - use crate::schema::{server_group_silenced_refs, server_silenced_refs}; - - let mut out: HashMap> = HashMap::new(); - if servers.is_empty() { - return Ok(out); - } - - let like_pattern = format!("{HEALTH_REF_PREFIX}%"); - - let server_ids: Vec = servers.iter().map(|(id, _)| *id).collect(); - let server_rows: Vec<(Uuid, String)> = server_silenced_refs::table - .select((server_silenced_refs::server_id, server_silenced_refs::ref_)) - .filter(server_silenced_refs::server_id.eq_any(&server_ids)) - .filter(server_silenced_refs::source.eq(STATUS_SOURCE)) - .filter(server_silenced_refs::ref_.like(&like_pattern)) - .load(db) - .await - .map_err(AppError::from)?; - for (server_id, r#ref) in server_rows { - if let Some(check) = r#ref.strip_prefix(HEALTH_REF_PREFIX) { - out.entry(server_id).or_default().insert(check.to_string()); - } - } - - let group_ids: Vec = servers - .iter() - .filter_map(|(_, group)| *group) - .collect::>() - .into_iter() - .collect(); - if !group_ids.is_empty() { - let group_rows: Vec<(Uuid, String)> = server_group_silenced_refs::table - .select(( - server_group_silenced_refs::server_group_id, - server_group_silenced_refs::ref_, - )) - .filter(server_group_silenced_refs::server_group_id.eq_any(&group_ids)) - .filter(server_group_silenced_refs::source.eq(STATUS_SOURCE)) - .filter(server_group_silenced_refs::ref_.like(&like_pattern)) - .load(db) - .await - .map_err(AppError::from)?; - let mut by_group: HashMap> = HashMap::new(); - for (group_id, r#ref) in group_rows { - if let Some(check) = r#ref.strip_prefix(HEALTH_REF_PREFIX) { - by_group - .entry(group_id) - .or_default() - .push(check.to_string()); - } - } - for (server_id, group_id) in servers { - if let Some(checks) = group_id.as_ref().and_then(|g| by_group.get(g)) { - out.entry(*server_id) - .or_default() - .extend(checks.iter().cloned()); - } - } - } - - Ok(out) -} - -/// Single-server variant of [`silenced_health_checks_for_servers`], -/// via [`silenced_refs_with_prefix`]. `group_id` is the server's -/// current group; pass `None` if ungrouped. -pub async fn silenced_health_checks_for_server( - db: &mut AsyncPgConnection, - server_id: Uuid, - group_id: Option, -) -> Result> { - let refs = silenced_refs_with_prefix(db, server_id, group_id, STATUS_SOURCE, HEALTH_REF_PREFIX) .await?; - Ok(refs - .iter() - .filter_map(|r| r.strip_prefix(HEALTH_REF_PREFIX)) - .map(String::from) - .collect()) + Ok(rows.into_iter().collect()) } impl ServerSilencedRef { + fn from_policy(p: ScopedCheckPolicy) -> Option { + Some(Self { + server_id: p.server_id?, + r#ref: check_to_ref(&p.source, &p.check_name), + source: p.source, + created_at: p.created_at, + created_by: p.created_by, + }) + } + /// Add a server-scoped silence and re-evaluate any currently-open - /// matching issues so they leave their incident. Idempotent: a - /// duplicate (`server_id`, `source`, `ref`) is a no-op (the - /// existing row's metadata is preserved). + /// matching issues so they leave their incident. Idempotent. pub async fn add( db: &mut AsyncPgConnection, server_id: Uuid, @@ -281,30 +164,16 @@ impl ServerSilencedRef { r#ref: &str, created_by: Option<&str>, ) -> Result { - use crate::schema::server_silenced_refs; - - let row: Self = diesel::insert_into(server_silenced_refs::table) - .values(( - server_silenced_refs::server_id.eq(server_id), - server_silenced_refs::source.eq(source), - server_silenced_refs::ref_.eq(r#ref), - server_silenced_refs::created_by.eq(created_by), - )) - .on_conflict(( - server_silenced_refs::server_id, - server_silenced_refs::source, - server_silenced_refs::ref_, - )) - .do_update() - // no-op update so we can RETURNING the existing row - .set(server_silenced_refs::server_id.eq(server_id)) - .returning(Self::as_select()) - .get_result(db) - .await - .map_err(AppError::from)?; - + let policy = ScopedCheckPolicy::silence( + db, + PolicyScope::Server(server_id), + source, + ref_to_check(r#ref), + created_by, + ) + .await?; reevaluate_open_issues_for_server_ref(db, server_id, source, r#ref).await?; - Ok(row) + Ok(Self::from_policy(policy).expect("server-scoped silence has a server_id")) } /// Remove a server-scoped silence and re-evaluate any currently-open @@ -315,37 +184,39 @@ impl ServerSilencedRef { source: &str, r#ref: &str, ) -> Result<()> { - use crate::schema::server_silenced_refs; - - diesel::delete( - server_silenced_refs::table.filter( - server_silenced_refs::server_id - .eq(server_id) - .and(server_silenced_refs::source.eq(source)) - .and(server_silenced_refs::ref_.eq(r#ref)), - ), + ScopedCheckPolicy::unsilence( + db, + PolicyScope::Server(server_id), + source, + ref_to_check(r#ref), ) - .execute(db) - .await - .map_err(AppError::from)?; - + .await?; reevaluate_open_issues_for_server_ref(db, server_id, source, r#ref).await?; Ok(()) } pub async fn list_for_server(db: &mut AsyncPgConnection, server_id: Uuid) -> Result> { - use crate::schema::server_silenced_refs::dsl; - dsl::server_silenced_refs - .select(Self::as_select()) - .filter(dsl::server_id.eq(server_id)) - .order(dsl::created_at.desc()) - .load(db) - .await - .map_err(AppError::from) + Ok( + ScopedCheckPolicy::list_silences(db, PolicyScope::Server(server_id)) + .await? + .into_iter() + .filter_map(Self::from_policy) + .collect(), + ) } } impl ServerGroupSilencedRef { + fn from_policy(p: ScopedCheckPolicy) -> Option { + Some(Self { + server_group_id: p.server_group_id?, + r#ref: check_to_ref(&p.source, &p.check_name), + source: p.source, + created_at: p.created_at, + created_by: p.created_by, + }) + } + pub async fn add( db: &mut AsyncPgConnection, server_group_id: Uuid, @@ -353,29 +224,16 @@ impl ServerGroupSilencedRef { r#ref: &str, created_by: Option<&str>, ) -> Result { - use crate::schema::server_group_silenced_refs; - - let row: Self = diesel::insert_into(server_group_silenced_refs::table) - .values(( - server_group_silenced_refs::server_group_id.eq(server_group_id), - server_group_silenced_refs::source.eq(source), - server_group_silenced_refs::ref_.eq(r#ref), - server_group_silenced_refs::created_by.eq(created_by), - )) - .on_conflict(( - server_group_silenced_refs::server_group_id, - server_group_silenced_refs::source, - server_group_silenced_refs::ref_, - )) - .do_update() - .set(server_group_silenced_refs::server_group_id.eq(server_group_id)) - .returning(Self::as_select()) - .get_result(db) - .await - .map_err(AppError::from)?; - + let policy = ScopedCheckPolicy::silence( + db, + PolicyScope::Group(server_group_id), + source, + ref_to_check(r#ref), + created_by, + ) + .await?; reevaluate_open_issues_for_group_ref(db, server_group_id, source, r#ref).await?; - Ok(row) + Ok(Self::from_policy(policy).expect("group-scoped silence has a server_group_id")) } pub async fn remove( @@ -384,20 +242,13 @@ impl ServerGroupSilencedRef { source: &str, r#ref: &str, ) -> Result<()> { - use crate::schema::server_group_silenced_refs; - - diesel::delete( - server_group_silenced_refs::table.filter( - server_group_silenced_refs::server_group_id - .eq(server_group_id) - .and(server_group_silenced_refs::source.eq(source)) - .and(server_group_silenced_refs::ref_.eq(r#ref)), - ), + ScopedCheckPolicy::unsilence( + db, + PolicyScope::Group(server_group_id), + source, + ref_to_check(r#ref), ) - .execute(db) - .await - .map_err(AppError::from)?; - + .await?; reevaluate_open_issues_for_group_ref(db, server_group_id, source, r#ref).await?; Ok(()) } @@ -406,13 +257,12 @@ impl ServerGroupSilencedRef { db: &mut AsyncPgConnection, server_group_id: Uuid, ) -> Result> { - use crate::schema::server_group_silenced_refs::dsl; - dsl::server_group_silenced_refs - .select(Self::as_select()) - .filter(dsl::server_group_id.eq(server_group_id)) - .order(dsl::created_at.desc()) - .load(db) - .await - .map_err(AppError::from) + Ok( + ScopedCheckPolicy::list_silences(db, PolicyScope::Group(server_group_id)) + .await? + .into_iter() + .filter_map(Self::from_policy) + .collect(), + ) } } diff --git a/crates/database/src/slack_outbox/mod.rs b/crates/database/src/slack_outbox/mod.rs index a4e4c88b..fd759c1f 100644 --- a/crates/database/src/slack_outbox/mod.rs +++ b/crates/database/src/slack_outbox/mod.rs @@ -36,11 +36,11 @@ pub mod vars; pub const KIND_INCIDENT_OPEN: &str = "incident_open"; /// Incident resolved — Phase A: top-level; Phase B: reply in the incident thread. pub const KIND_INCIDENT_RESOLVE: &str = "incident_resolve"; -/// Self-alert became active ([`crate::self_alerts`]). No incident; `issue_id` -/// carries the nil-server issue. +/// Legacy: direct self-alert notices, from before self-alerts flowed +/// through canopy-wide incidents. Nothing enqueues these anymore; the +/// constants remain so the drainer can drain straggler rows harmlessly. pub const KIND_SELF_ALERT_OPEN: &str = "self_alert_open"; -/// Self-alert recovered. Routed to the same webhook as the open; the payload's -/// `state` field tells the workflow which is which. +/// Legacy: see [`KIND_SELF_ALERT_OPEN`]. pub const KIND_SELF_ALERT_RESOLVE: &str = "self_alert_resolve"; #[derive(Clone, Debug, Serialize, Deserialize, Queryable, Selectable)] @@ -51,7 +51,7 @@ pub struct SlackOutbox { #[diesel(deserialize_as = jiff_diesel::Timestamp, serialize_as = jiff_diesel::Timestamp)] pub created_at: Timestamp, pub kind: String, - /// `None` for self-alert rows, which have no incident. + /// `None` for legacy self-alert rows, which had no incident. pub incident_id: Option, pub issue_id: Option, pub note_id: Option, @@ -143,34 +143,6 @@ impl SlackOutbox { Ok(rows) } - /// Self-alert twin of [`cancel_pending_open`](Self::cancel_pending_open), - /// keyed on the nil-server issue instead of an incident. Same contract: a - /// non-zero return means the open never reached Slack and the caller - /// should skip the resolve enqueue. - pub async fn cancel_pending_self_alert_open( - db: &mut AsyncPgConnection, - issue_id: Uuid, - reason: &str, - ) -> Result { - use crate::schema::slack_outbox::dsl; - let now = Timestamp::now(); - let rows = diesel::update( - dsl::slack_outbox - .filter(dsl::issue_id.eq(Some(issue_id))) - .filter(dsl::kind.eq(KIND_SELF_ALERT_OPEN)) - .filter(dsl::delivered_at.is_null()) - .filter(dsl::gave_up_at.is_null()), - ) - .set(( - dsl::gave_up_at.eq(jiff_diesel::Timestamp::from(now)), - dsl::last_error.eq(reason), - )) - .execute(db) - .await - .map_err(AppError::from)?; - Ok(rows) - } - /// For each of `incident_ids`, return the `deliver_after` of its /// pending `incident_open` row — i.e. an incident that has been opened /// in the database but whose Slack notice hasn't been sent yet (and diff --git a/crates/database/src/slack_outbox/vars.rs b/crates/database/src/slack_outbox/vars.rs index 10993538..5af8bbf0 100644 --- a/crates/database/src/slack_outbox/vars.rs +++ b/crates/database/src/slack_outbox/vars.rs @@ -6,15 +6,12 @@ //! POST a flat JSON object keyed by the declared variable names — no //! `blocks`, no `text`. //! -//! Three workflows: +//! Two workflows: //! - `incident_open`: variables `server`, `severity`, `source_ref`, `message`, `link` //! - `incident_resolve`: variables `server`, `by`, `link`. `by` is either the //! resolving operator's login, or — when the incident retired on its own — //! "the healthcheck recovering" (it describes the event, not an actor: the //! check went healthy again, which does not imply nobody intervened). -//! - self-alert (both `self_alert_open` and `self_alert_resolve` kinds): -//! variables `state`, `severity`, `source_ref`, `title`, `message`, `link` — -//! one workflow serves raise and recovery, told apart by `state`. //! //! Note: `link` is **not** rendered here. It's pure config (incident_id is //! already on the row, and `PRIVATE_URL` is operator-set), so the drainer @@ -23,55 +20,25 @@ //! (public-server, private-server, reachability job) doesn't have to know //! the operator-facing URL. -use commons_types::issue::Severity; use serde_json::{Value, json}; -fn title_case(s: &str) -> String { - let mut chars = s.chars(); - match chars.next() { - Some(c) => c.to_uppercase().chain(chars).collect(), - None => String::new(), - } -} - +/// `urgency` is the result-derived label for the workflow's `severity` +/// variable: Critical (escalating failure), Error (failure), Warning. pub fn incident_open( server_label: &str, - severity: Severity, + urgency: &str, source: &str, issue_ref: &str, message: &str, ) -> Value { json!({ "server": server_label, - "severity": title_case(&severity.to_string()), + "severity": urgency, "source_ref": format!("{source}/{issue_ref}"), "message": message, }) } -/// Self-alert raise. `title` is the issue's single-line description. -pub fn self_alert_open(severity: Severity, issue_ref: &str, title: &str, message: &str) -> Value { - json!({ - "state": "alert", - "severity": title_case(&severity.to_string()), - "source_ref": format!("canopy/{issue_ref}"), - "title": title, - "message": message, - }) -} - -/// Self-alert recovery. Same variable set as the raise so one Workflow -/// Builder workflow serves both. -pub fn self_alert_resolve(issue_ref: &str, message: &str) -> Value { - json!({ - "state": "recovered", - "severity": "Info", - "source_ref": format!("canopy/{issue_ref}"), - "title": "recovered", - "message": message, - }) -} - pub fn incident_resolve(server_label: &str, by: Option<&str>) -> Value { json!({ "server": server_label, diff --git a/crates/database/src/statuses.rs b/crates/database/src/statuses.rs index 006111c9..a68643d1 100644 --- a/crates/database/src/statuses.rs +++ b/crates/database/src/statuses.rs @@ -5,7 +5,6 @@ use std::{ use commons_errors::{AppError, Result}; use commons_types::{ - issue::Severity, server::rank::ServerRank, status::{CheckResult, HealthState, ShortStatus}, version::VersionStr, @@ -19,7 +18,7 @@ use serde::{Deserialize, Serialize}; use tracing::{debug, info, warn}; use uuid::Uuid; -use crate::issues::{Issue, NewEvent}; +use crate::issues::Issue; use crate::servers::Server; /// Source value canopy uses when it files reachability issues on behalf of a @@ -31,6 +30,35 @@ pub const CANOPY_SOURCE: &str = "canopy"; /// same issue row. pub const REACHABILITY_REF: &str = "reachability"; +/// Ref prefix for the per-(server, source) staleness checks canopy files +/// when a reporting source stops reporting: `stale/`, e.g. +/// `stale/alertd`. A contract with stored silences and policy rows. +pub const STALE_REF_PREFIX: &str = "stale/"; + +pub const REACHABILITY_DOC: &str = "## Description + +Nothing is reaching canopy about this server: no source has reported and no ping has succeeded within the server's down threshold. This is the all-sources-stale signal — the server is presented as unreachable. + +## Results + +- **fail** — no status from any source within the threshold (or ever); recovers as soon as anything reports. + +## Solve + +Check whether the server itself is down, its network/VPN path to canopy, and whether its reporting agents are running. The per-source `stale/` checks narrow down which reporter went quiet first."; + +pub const STALE_DOC: &str = "## Description + +A source that has been reporting on this server has gone quiet: its most recent report is older than the server's down threshold, while other paths may still reach canopy. + +## Results + +- **warn** — the source's last report crossed the threshold; recovers when it reports again. + +## Solve + +Check that the reporting agent for this source is running on the server and can reach canopy. If the source was deliberately decommissioned, silence this check for the server."; + fn server_label(s: &Server) -> String { s.name .clone() @@ -91,6 +119,9 @@ pub struct Status { /// `check` name and a result; any extra per-check fields are passed /// through verbatim. pub health: serde_json::Value, + /// The source that pushed this status: the named reporter (e.g. + /// `alertd`), or `canopy` for statuses generated internally. + pub source: String, } #[derive(Debug, Insertable)] @@ -104,6 +135,7 @@ pub struct NewStatus { pub extra: serde_json::Value, pub healthy: bool, pub health: serde_json::Value, + pub source: String, } impl Default for NewStatus { @@ -115,6 +147,7 @@ impl Default for NewStatus { extra: serde_json::Value::Object(Default::default()), healthy: true, health: serde_json::Value::Array(Default::default()), + source: CANOPY_SOURCE.into(), } } } @@ -162,6 +195,7 @@ impl Status { // to avoid false-positive unhealthy events from this path. healthy: true, health: serde_json::Value::Array(Default::default()), + source: CANOPY_SOURCE.into(), }) } Err(err) => { @@ -215,18 +249,26 @@ impl Status { Ok(()) } - /// Sweep every server's most recent status. For each monitored server - /// (`is_monitored = true`) whose freshness has crossed the per-server - /// `alert_when_down_for` threshold, file (or close) a canopy-sourced - /// issue keyed by [`REACHABILITY_REF`]. + /// Sweep every monitored server (`is_monitored = true`) for staleness + /// against its per-server `alert_when_down_for` threshold, on two + /// levels: /// - /// Most servers report by pushing their own status to the public-server - /// (so their `device_id` is non-null); the pingtask only handles legacy - /// foreign servers without a registered device. Both paths feed the same - /// `statuses` table, so this sweep doesn't care which one is in play. + /// - **Reachability**: the server's most recent status row, from any + /// source. Every source's report (and every pingtask probe) lands a + /// status row, so this goes stale exactly when *all* of the server's + /// sources are stale — the server is unreachable. Filed (or closed) + /// as a canopy-sourced check keyed by [`REACHABILITY_REF`]. This arm + /// also covers servers that have never reported at all, and legacy + /// foreign servers without a registered device that only the + /// pingtask reaches. + /// - **Per-source staleness**: a source that has reported checks on a + /// server is expected to keep reporting. When its most recent report + /// is older than the threshold, file a `stale/` check (see + /// [`STALE_REF_PREFIX`]) — catching one reporter going quiet while + /// others keep the server reachable. /// /// Returns the number of events filed in this pass. - pub async fn sweep_reachability(db: &mut AsyncPgConnection) -> Result { + pub async fn sweep_staleness(db: &mut AsyncPgConnection) -> Result { let servers = Server::get_all(db, 0, None).await?; let monitored: Vec<&Server> = servers .iter() @@ -264,24 +306,16 @@ impl Status { }; let existing = issue_map.get(&server.id).copied(); - let event = match (down, existing) { + let (observed, message) = match (down, existing) { (false, None) => continue, (false, Some(issue)) if !issue.active => continue, - (false, Some(_)) => NewEvent { - source: CANOPY_SOURCE.into(), - r#ref: REACHABILITY_REF.into(), - severity: Some(Severity::Info), - description: None, - message: format!("Server {} is reachable again", server_label(server)), - active: Some(false), - occurred_at: Some(now), - }, - (true, _) => NewEvent { - source: CANOPY_SOURCE.into(), - r#ref: REACHABILITY_REF.into(), - severity: Some(Severity::Error), - description: None, - message: match elapsed { + (false, Some(_)) => ( + CheckResult::Passed, + format!("Server {} is reachable again", server_label(server)), + ), + (true, _) => ( + CheckResult::Failed, + match elapsed { Some(e) => format!( "Server {} has not reported for {} (threshold {})", server_label(server), @@ -294,23 +328,142 @@ impl Status { format_secs(threshold.as_secs()), ), }, - active: Some(true), - occurred_at: Some(now), + ), + }; + crate::issues::file_check( + db, + crate::issues::CheckFiling { + source: CANOPY_SOURCE, + scope: crate::issues::FilingScope::Server { + server_id: server.id, + device_id: None, + }, + check: REACHABILITY_REF, + observed, + title: Some("Server unreachable"), + message: &message, + detail: Some(serde_json::json!({ + "elapsed_secs": elapsed.map(|e| e.as_secs()), + "threshold_secs": threshold.as_secs(), + })), + default_ceiling: CheckResult::Failed, + default_escalates: false, + documentation: Some(REACHABILITY_DOC), }, + ) + .await?; + filed += 1; + } + + filed += Self::sweep_source_staleness(db, &monitored, now).await?; + Ok(filed) + } + + /// The per-source arm of [`Self::sweep_staleness`]: file (or close) a + /// `stale/` check for each (monitored server, reporting source) + /// whose most recent report has crossed the server's threshold. + /// + /// Freshness comes from check state, not the statuses history: every + /// report re-stamps `last_seen` on the state rows of the checks it + /// mentions, so [`Issue::source_freshness`] is both the set of sources + /// expected to report and when each last did. Registered at a warning + /// ceiling — one quiet reporter degrades a server, full unreachability + /// is the reachability arm's failure. + async fn sweep_source_staleness( + db: &mut AsyncPgConnection, + monitored: &[&Server], + now: Timestamp, + ) -> Result { + let server_map: std::collections::HashMap = + monitored.iter().map(|s| (s.id, *s)).collect(); + let server_ids: Vec = monitored.iter().map(|s| s.id).collect(); + let freshness = Issue::source_freshness(db, &server_ids).await?; + if freshness.is_empty() { + return Ok(0); + } + + let refs: Vec = freshness + .iter() + .map(|(_, source, _)| format!("{STALE_REF_PREFIX}{source}")) + .collect::>() + .into_iter() + .collect(); + let existing = Issue::list_by_source_refs(db, CANOPY_SOURCE, &refs, &server_ids).await?; + let existing_map: std::collections::HashMap<(Uuid, &str), &Issue> = existing + .iter() + .filter_map(|i| i.server_id.map(|sid| ((sid, i.r#ref.as_str()), i))) + .collect(); + + let mut filed = 0usize; + for (server_id, source, last_seen) in &freshness { + let Some(server) = server_map.get(server_id) else { + continue; }; - event.save(db, server.id, None).await?; + let threshold = server.alert_when_down_for.0; + let elapsed = now.duration_since(*last_seen).abs(); + let stale = elapsed >= threshold; + let check = format!("{STALE_REF_PREFIX}{source}"); + let existing = existing_map.get(&(*server_id, check.as_str())).copied(); + + let (observed, message) = match (stale, existing) { + (false, None) => continue, + (false, Some(issue)) if !issue.active => continue, + (false, Some(_)) => ( + CheckResult::Passed, + format!( + "Source {source} on server {} is reporting again", + server_label(server), + ), + ), + (true, _) => ( + CheckResult::Failed, + format!( + "Source {source} on server {} has not reported for {} (threshold {})", + server_label(server), + format_secs(elapsed.as_secs()), + format_secs(threshold.as_secs()), + ), + ), + }; + crate::issues::file_check( + db, + crate::issues::CheckFiling { + source: CANOPY_SOURCE, + scope: crate::issues::FilingScope::Server { + server_id: *server_id, + device_id: None, + }, + check: &check, + observed, + title: Some("Source stopped reporting"), + message: &message, + detail: Some(serde_json::json!({ + "source": source, + "last_reported": last_seen.to_string(), + "elapsed_secs": elapsed.as_secs(), + "threshold_secs": threshold.as_secs(), + })), + default_ceiling: CheckResult::Warning, + default_escalates: false, + documentation: Some(STALE_DOC), + }, + ) + .await?; filed += 1; } Ok(filed) } - /// Most recent status row (across all servers) whose `health` array - /// contains an entry for `check_name`. Used by the rule-editor UI to - /// surface a realistic sample of the variables an operator can - /// predicate on (the check's extras, the status-level extras, and - /// the server's tags resolved up the group). + /// Most recent status row (across all servers) pushed by `source` + /// whose `health` array contains an entry for `check_name`. Used by + /// the rule-editor UI to surface a realistic sample of the variables + /// an operator can predicate on (the check's extras, the status-level + /// extras, and the server's tags resolved up the group). Scoped to + /// the check's own source — another source's same-named check may + /// carry entirely different fields. pub async fn latest_for_check_name( db: &mut AsyncPgConnection, + source: &str, check_name: &str, ) -> Result> { use diesel::sql_types::{Text, Uuid as DUuid}; @@ -327,9 +480,11 @@ impl Status { // express cleanly), then load the typed Status row by id. let picked: Option = sql_query( "SELECT id FROM statuses \ - WHERE health @> jsonb_build_array(jsonb_build_object('check', $1::text)) \ + WHERE source = $1 \ + AND health @> jsonb_build_array(jsonb_build_object('check', $2::text)) \ ORDER BY created_at DESC LIMIT 1", ) + .bind::(source) .bind::(check_name) .get_result(db) .await @@ -448,76 +603,6 @@ impl Status { query.load::(db).await.map_err(AppError::from) } - /// Live (non-deleted, non-canopy-own) servers whose latest status - /// reports `check_name` at **any** result, paired with that status — - /// the data backing the per-healthcheck "who's affected" page (which - /// shows the failing servers by default and the healthy ones behind a - /// toggle). Mirrors the live-server scoping in - /// [`crate::servers::Server::get_all`] / the status-board queries: - /// archived servers and canopy's own row (`id = Uuid::nil()`) never - /// appear. - /// - /// The check-name filter stays on the Rust side rather than folding - /// into the SQL: reproducing [`CheckResult::from_entry`]'s legacy - /// `healthy: bool` fallback as a jsonb predicate would just duplicate - /// logic that already lives in [`Self::check_entry`], and the - /// per-server latest-status set is small. - pub async fn reporting_check_with_servers( - db: &mut AsyncPgConnection, - check_name: &str, - ) -> Result> { - use crate::schema::servers::dsl as servers_dsl; - - let servers: Vec = servers_dsl::servers - .select(Server::as_select()) - .filter(servers_dsl::id.ne(Uuid::nil())) - .filter(servers_dsl::deleted_at.is_null()) - .load(db) - .await - .map_err(AppError::from)?; - let server_ids: Vec = servers.iter().map(|s| s.id).collect(); - - let mut status_map: std::collections::HashMap = - Self::latest_for_servers(db, &server_ids) - .await? - .into_iter() - .map(|s| (s.server_id, s)) - .collect(); - - Ok(servers - .into_iter() - .filter_map(|s| { - let status = status_map.remove(&s.id)?; - status - .check_entry(check_name) - .is_some() - .then_some((s, status)) - }) - .collect()) - } - - /// This status row's `health[]` entry for `check_name`: the normalised - /// result plus the entry's full JSON object (including the reserved - /// `check`/`healthy`/`result` keys, so callers can ship it to a UI - /// that renders it the same way the server-detail checks table does). - /// `None` when the row doesn't report that check, or the entry is - /// malformed (no readable result — same rule as every other `health[]` - /// reader). - pub fn check_entry( - &self, - check_name: &str, - ) -> Option<(CheckResult, serde_json::Map)> { - let arr = self.health.as_array()?; - arr.iter().find_map(|entry| { - let obj = entry.as_object()?; - if obj.get("check")?.as_str()? != check_name { - return None; - } - let result = CheckResult::from_entry(obj)?; - Some((result, obj.clone())) - }) - } - pub async fn production_versions(db: &mut AsyncPgConnection) -> Result> { use crate::schema::servers::dsl as servers_dsl; diff --git a/crates/database/tests/it/backup_detection.rs b/crates/database/tests/it/backup_detection.rs index d17279f2..c809e4da 100644 --- a/crates/database/tests/it/backup_detection.rs +++ b/crates/database/tests/it/backup_detection.rs @@ -15,7 +15,7 @@ use commons_tests::db::TestDb; use commons_types::backup::BackupType; -use commons_types::issue::Severity; +use commons_types::status::CheckResult; use database::backup::refs; use database::backup::staleness::{ScanRow, StalenessVerdict}; use database::diesel_async::AsyncPgConnection; @@ -209,8 +209,8 @@ async fn insert_backup_success_aged( /// if any. #[derive(QueryableByName, Debug)] struct IssueRow { - #[diesel(sql_type = sql_types::Text)] - severity: String, + #[diesel(sql_type = sql_types::Nullable)] + effective_result: Option, #[diesel(sql_type = sql_types::Bool)] active: bool, } @@ -221,7 +221,7 @@ async fn server_issue( r#ref: &str, ) -> Option { sql_query( - "SELECT severity, active FROM issues \ + "SELECT effective_result, active FROM issues \ WHERE server_id = $1 AND source = $2 AND \"ref\" = $3", ) .bind::(server_id) @@ -238,7 +238,7 @@ async fn group_issue( r#ref: &str, ) -> Option { sql_query( - "SELECT severity, active FROM issues \ + "SELECT effective_result, active FROM issues \ WHERE server_group_id = $1 AND source = $2 AND \"ref\" = $3", ) .bind::(group_id) @@ -440,7 +440,7 @@ async fn sweep_files_staleness_for_monitored_server_with_old_success() { let issue = server_issue(&mut conn, server_id, &sref) .await .expect("staleness issue filed"); - assert_eq!(issue.severity, Severity::Error.to_string()); + assert_eq!(issue.effective_result.as_deref(), Some("failed")); assert!(issue.active, "staleness issue is active"); // Error opens an incident; monitored server contributes. assert_eq!( @@ -478,7 +478,7 @@ async fn sweep_files_never_for_server_that_never_succeeded() { .expect("never issue filed"); // Never-reported is a warning (so first-time setup doesn't open an // incident); a *missed* backup is the error that pages. - assert_eq!(issue.severity, Severity::Warning.to_string()); + assert_eq!(issue.effective_result.as_deref(), Some("warning")); assert!(issue.active); // Staleness ref must NOT be filed when there's never been a success. assert!( @@ -584,9 +584,9 @@ async fn reconcile_files_report_gap_when_snapshot_fresh_but_no_report() { .await .expect("report-gap issue filed"); assert_eq!( - issue.severity, - Severity::Warning.to_string(), - "report-gap is Warning (non-paging)", + issue.effective_result.as_deref(), + Some("warning"), + "report-gap is a warning (non-paging)", ); assert!(issue.active); // Warning never opens an incident on its own. @@ -649,7 +649,7 @@ async fn reconcile_files_missing_when_report_fresh_but_no_snapshot() { let issue = group_issue(&mut conn, group_id, &mref) .await .expect("reconcile-missing issue filed (group-scoped)"); - assert_eq!(issue.severity, Severity::Error.to_string()); + assert_eq!(issue.effective_result.as_deref(), Some("failed")); assert!(issue.active); // Group-level Error opens an incident. assert_eq!( @@ -727,7 +727,7 @@ async fn reconcile_clears_report_gap_when_report_and_snapshot_agree() { !issue.active, "report-gap cleared once report and snapshot agree" ); - assert_eq!(issue.severity, Severity::Info.to_string()); + assert_eq!(issue.effective_result.as_deref(), Some("passed")); }) .await; } @@ -776,9 +776,9 @@ async fn reconcile_files_size_mismatch_when_reported_size_differs_from_repo() { .await .expect("size-mismatch issue filed"); assert_eq!( - issue.severity, - Severity::Warning.to_string(), - "size-mismatch is Warning (non-paging)", + issue.effective_result.as_deref(), + Some("warning"), + "size-mismatch is a warning (non-paging)", ); assert!(issue.active); assert_eq!( @@ -868,7 +868,7 @@ async fn reconcile_clears_size_mismatch_when_latest_sizes_agree() { !issue.active, "size-mismatch cleared once latest sizes agree" ); - assert_eq!(issue.severity, Severity::Info.to_string()); + assert_eq!(issue.effective_result.as_deref(), Some("passed")); }) .await; } @@ -885,14 +885,21 @@ async fn group_event_pages_even_when_all_members_unmonitored() { let _s1 = insert_server(&mut conn, group_id, false).await; let _s2 = insert_server(&mut conn, group_id, false).await; - let issue = database::issues::raise_group_event( + let stamp = database::issues::CheckStateStamp { + check: refs::CORRUPTION.into(), + observed: CheckResult::Failed, + effective: CheckResult::Failed, + escalates: true, + detail: None, + }; + let issue = database::issues::raise_group_event_with_state( &mut conn, group_id, refs::CORRUPTION, - Severity::Critical, None, "repo corruption detected", true, + Some(&stamp), ) .await .expect("raise group event"); @@ -912,16 +919,23 @@ async fn group_event_pages_even_when_all_members_unmonitored() { .expect("list incidents"); assert_eq!(open.len(), 1, "exactly one open incident on the group"); - // Recovery: same (source, ref) with active=false at lower severity - // leaves the incident and closes it. - database::issues::raise_group_event( + // Recovery: same (source, ref) with active=false and a passed + // result leaves the incident and closes it. + let stamp = database::issues::CheckStateStamp { + check: refs::CORRUPTION.into(), + observed: CheckResult::Passed, + effective: CheckResult::Passed, + escalates: true, + detail: None, + }; + database::issues::raise_group_event_with_state( &mut conn, group_id, refs::CORRUPTION, - Severity::Info, None, "repo corruption cleared", false, + Some(&stamp), ) .await .expect("clear group event"); @@ -974,7 +988,7 @@ async fn sweep_files_maintenance_error_when_latest_run_failed_then_clears_on_suc let issue = group_issue(&mut conn, group_id, refs::MAINTENANCE_ERROR) .await .expect("maintenance-error issue filed"); - assert_eq!(issue.severity, Severity::Error.to_string()); + assert_eq!(issue.effective_result.as_deref(), Some("failed")); assert!(issue.active, "failure issue is active"); assert_eq!( group_issue_open_links(&mut conn, group_id, refs::MAINTENANCE_ERROR).await, diff --git a/crates/database/tests/it/check_policies.rs b/crates/database/tests/it/check_policies.rs new file mode 100644 index 00000000..f2c68a4e --- /dev/null +++ b/crates/database/tests/it/check_policies.rs @@ -0,0 +1,229 @@ +//! `CheckPolicy` model: catalog of per-(source, check) result policies, +//! maintained by the public-server status ingestion path and edited +//! through the private-server `/api/healthchecks` endpoints. + +use commons_types::status::CheckResult; +use database::check_policies::CheckPolicy; + +#[tokio::test(flavor = "multi_thread")] +async fn upsert_default_inserts_then_is_idempotent() { + commons_tests::db::TestDb::run(async |mut conn, _| { + CheckPolicy::upsert_default(&mut conn, "alertd", "disk_space") + .await + .expect("first upsert"); + CheckPolicy::upsert_default(&mut conn, "alertd", "disk_space") + .await + .expect("second upsert is no-op"); + + // Both upserts should land at the schema default, pending review. + let rows = CheckPolicy::list(&mut conn).await.expect("list"); + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].source, "alertd"); + assert_eq!(rows[0].check_name, "disk_space"); + assert_eq!(rows[0].ceiling, CheckResult::Warning); + assert!(!rows[0].escalates); + assert!(rows[0].reviewed_at.is_none(), "pending review by default"); + assert!(rows[0].reviewed_by.is_none()); + }) + .await +} + +#[tokio::test(flavor = "multi_thread")] +async fn same_check_name_is_distinct_per_source() { + commons_tests::db::TestDb::run(async |mut conn, _| { + CheckPolicy::upsert_default(&mut conn, "alertd", "disk_space") + .await + .expect("alertd upsert"); + CheckPolicy::upsert_default(&mut conn, "seedling", "disk_space") + .await + .expect("seedling upsert"); + + CheckPolicy::update( + &mut conn, + "alertd", + "disk_space", + CheckResult::Failed, + false, + None, + "alice", + ) + .await + .expect("update alertd entry"); + + let seedling = CheckPolicy::get(&mut conn, "seedling", "disk_space") + .await + .expect("get") + .expect("seedling row exists"); + assert_eq!( + seedling.ceiling, + CheckResult::Warning, + "the other source's entry is untouched", + ); + }) + .await +} + +#[tokio::test(flavor = "multi_thread")] +async fn upsert_default_does_not_overwrite_operator_policy() { + commons_tests::db::TestDb::run(async |mut conn, _| { + CheckPolicy::upsert_default(&mut conn, "alertd", "disk_space") + .await + .expect("seed"); + CheckPolicy::update( + &mut conn, + "alertd", + "disk_space", + CheckResult::Failed, + true, + None, + "alice", + ) + .await + .expect("operator update"); + + // A subsequent push from ingestion must not revert the row. + CheckPolicy::upsert_default(&mut conn, "alertd", "disk_space") + .await + .expect("upsert again"); + + let rows = CheckPolicy::list(&mut conn).await.expect("list"); + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].ceiling, CheckResult::Failed); + assert!(rows[0].escalates); + assert!(rows[0].reviewed_at.is_some()); + assert_eq!(rows[0].reviewed_by.as_deref(), Some("alice")); + }) + .await +} + +#[tokio::test(flavor = "multi_thread")] +async fn apply_caps_at_ceiling_or_defaults_to_warning() { + commons_tests::db::TestDb::run(async |mut conn, _| { + let empty_map = serde_json::Map::new(); + let empty_tags = std::collections::HashMap::new(); + let ctx = database::check_policies::EvaluationContext { + status_extra: &empty_map, + check_extra: &empty_map, + tags: &empty_tags, + }; + // Unknown check → default policy (programmer-error path; + // ingestion upserts before reading in production). + let unknown = CheckPolicy::apply(&mut conn, "alertd", "ghost", CheckResult::Failed, &ctx) + .await + .expect("apply"); + assert_eq!(unknown.effective, CheckResult::Warning); + assert!(!unknown.escalates); + + CheckPolicy::upsert_default(&mut conn, "alertd", "cert_expiry") + .await + .expect("seed"); + CheckPolicy::update( + &mut conn, + "alertd", + "cert_expiry", + CheckResult::Failed, + true, + None, + "bob", + ) + .await + .expect("update"); + + // A failed ceiling passes failures through, carrying the flag. + let failed = CheckPolicy::apply( + &mut conn, + "alertd", + "cert_expiry", + CheckResult::Failed, + &ctx, + ) + .await + .expect("apply"); + assert_eq!(failed.effective, CheckResult::Failed); + assert!(failed.escalates); + + // A warning observation is already below the ceiling: unchanged. + let warned = CheckPolicy::apply( + &mut conn, + "alertd", + "cert_expiry", + CheckResult::Warning, + &ctx, + ) + .await + .expect("apply"); + assert_eq!(warned.effective, CheckResult::Warning); + + // A passed ceiling means the check never alerts. + CheckPolicy::update( + &mut conn, + "alertd", + "cert_expiry", + CheckResult::Passed, + false, + None, + "bob", + ) + .await + .expect("downgrade"); + let ignored = CheckPolicy::apply( + &mut conn, + "alertd", + "cert_expiry", + CheckResult::Failed, + &ctx, + ) + .await + .expect("apply"); + assert_eq!(ignored.effective, CheckResult::Passed); + }) + .await +} + +#[tokio::test(flavor = "multi_thread")] +async fn update_stamps_review_metadata_even_on_no_op_save() { + commons_tests::db::TestDb::run(async |mut conn, _| { + CheckPolicy::upsert_default(&mut conn, "alertd", "noisy_check") + .await + .expect("seed"); + + // "Mark reviewed without changing the policy": pass the current + // (default) values. reviewed_at + reviewed_by must still be set. + let updated = CheckPolicy::update( + &mut conn, + "alertd", + "noisy_check", + CheckResult::Warning, + false, + None, + "carol", + ) + .await + .expect("update"); + assert_eq!(updated.ceiling, CheckResult::Warning); + assert!(updated.reviewed_at.is_some()); + assert_eq!(updated.reviewed_by.as_deref(), Some("carol")); + }) + .await +} + +#[tokio::test(flavor = "multi_thread")] +async fn list_orders_by_source_then_check_name() { + commons_tests::db::TestDb::run(async |mut conn, _| { + for (source, name) in [("alertd", "zeta"), ("alertd", "alpha"), ("seedling", "mu")] { + CheckPolicy::upsert_default(&mut conn, source, name) + .await + .expect("seed"); + } + let rows = CheckPolicy::list(&mut conn).await.expect("list"); + let keys: Vec<(&str, &str)> = rows + .iter() + .map(|r| (r.source.as_str(), r.check_name.as_str())) + .collect(); + assert_eq!( + keys, + vec![("alertd", "alpha"), ("alertd", "zeta"), ("seedling", "mu"),] + ); + }) + .await +} diff --git a/crates/database/tests/it/healthcheck_severity_rules.rs b/crates/database/tests/it/check_policy_rules.rs similarity index 66% rename from crates/database/tests/it/healthcheck_severity_rules.rs rename to crates/database/tests/it/check_policy_rules.rs index 6c6ac215..89d5c44b 100644 --- a/crates/database/tests/it/healthcheck_severity_rules.rs +++ b/crates/database/tests/it/check_policy_rules.rs @@ -1,10 +1,10 @@ -//! v2 conditional-rule model: JsonLogic if-ladder serde + evaluator + -//! `severity_for` integration. The model is the single source of truth -//! for "given a status push, what severity does this check fail at?" -//! See `docs/plans/healthcheck-severity-rules-v2.md`. +//! Conditional-rule model: JsonLogic if-ladder serde + evaluator + +//! `CheckPolicy::apply` integration. The model is the single source of +//! truth for "given a status push, what effective result does this check +//! grade to?" -use commons_types::{issue::Severity, status::CheckResult}; -use database::healthcheck_severities::{EvaluationContext, HealthcheckSeverity, IfLadder}; +use commons_types::status::CheckResult; +use database::check_policies::{CheckPolicy, EvaluationContext, IfLadder}; use serde_json::json; use std::collections::HashMap; @@ -25,7 +25,7 @@ fn empty_ctx<'a>( #[test] fn deserialise_accepts_all_supported_ops() { for op in ["==", "!=", "<", "<=", ">", ">="] { - let json = json!({"if": [{op: [{"var": "check.x"}, 10]}, "error"]}); + let json = json!({"if": [{op: [{"var": "check.x"}, 10]}, "failed"]}); serde_json::from_value::(json).unwrap_or_else(|e| panic!("op {op} failed: {e}")); } let in_range = json!({"if": [ @@ -38,12 +38,12 @@ fn deserialise_accepts_all_supported_ops() { #[test] fn deserialise_rejects_composition_and_unknown_ops() { let cases: &[(serde_json::Value, &str)] = &[ - (json!({"if": [{"and": [true, true]}, "error"]}), "and"), - (json!({"if": [{"or": [true, true]}, "error"]}), "or"), - (json!({"if": [{"!": [true]}, "error"]}), "not"), - (json!({"if": [{"if": [true, true]}, "error"]}), "nested if"), + (json!({"if": [{"and": [true, true]}, "failed"]}), "and"), + (json!({"if": [{"or": [true, true]}, "failed"]}), "or"), + (json!({"if": [{"!": [true]}, "failed"]}), "not"), + (json!({"if": [{"if": [true, true]}, "failed"]}), "nested if"), ( - json!({"if": [{"foo": [{"var": "check.x"}, 1]}, "error"]}), + json!({"if": [{"foo": [{"var": "check.x"}, 1]}, "failed"]}), "unknown", ), ]; @@ -68,15 +68,15 @@ fn deserialise_rejects_malformed_shapes() { ), (json!({"not_if": []}), "wrong top-level op"), ( - json!({"if": [{"==": [{"var": "check.x"}, 1]}, "not_a_severity"]}), - "bad severity", + json!({"if": [{"==": [{"var": "check.x"}, 1]}, "not_a_result"]}), + "bad result", ), ( - json!({"if": [{"==": [{"var": "BAD.field"}, 1]}, "error"]}), + json!({"if": [{"==": [{"var": "BAD.field"}, 1]}, "failed"]}), "unknown var namespace", ), ( - json!({"if": [{"==": [{"var": "check..bad"}, 1]}, "error"]}), + json!({"if": [{"==": [{"var": "check..bad"}, 1]}, "failed"]}), "empty field segment", ), ]; @@ -91,8 +91,8 @@ fn deserialise_rejects_malformed_shapes() { fn round_trip_preserves_shape() { let original = json!({"if": [ {"in_range": [{"var": "status.bestoolVersion"}, ">=2.4.0 <2.5.4"]}, "warning", - {">": [{"var": "check.used_pct"}, 95]}, "critical", - {"==": [{"var": "tag.environment"}, "prod"]}, "error" + {">": [{"var": "check.used_pct"}, 95]}, "failed", + {"==": [{"var": "tag.environment"}, "prod"]}, "passed" ]}); let ladder: IfLadder = serde_json::from_value(original.clone()).expect("parse"); let back = serde_json::to_value(&ladder).expect("serialise"); @@ -115,7 +115,7 @@ fn in_range_matches_inside_misses_outside_and_falses_on_non_semver() { status.insert("bestoolVersion".into(), json!("2.4.7")); assert_eq!( ladder.evaluate(&empty_ctx(&check_extra, &status, &tags)), - Some(Severity::Warning), + Some(CheckResult::Warning), ); status.insert("bestoolVersion".into(), json!("2.5.4")); @@ -142,7 +142,7 @@ fn in_range_matches_inside_misses_outside_and_falses_on_non_semver() { #[test] fn numeric_ops_coerce_string_of_digits() { let ladder: IfLadder = serde_json::from_value(json!({"if": [ - {">": [{"var": "check.value"}, 95]}, "critical" + {">": [{"var": "check.value"}, 95]}, "failed" ]})) .unwrap(); let status = serde_json::Map::new(); @@ -153,14 +153,14 @@ fn numeric_ops_coerce_string_of_digits() { check.insert("value".into(), json!(97)); assert_eq!( ladder.evaluate(&empty_ctx(&check, &status, &tags)), - Some(Severity::Critical), + Some(CheckResult::Failed), ); // String of digits > 95 (bestool sometimes does this) → matches. check.insert("value".into(), json!("100")); assert_eq!( ladder.evaluate(&empty_ctx(&check, &status, &tags)), - Some(Severity::Critical), + Some(CheckResult::Failed), ); // Non-numeric string → false. @@ -189,7 +189,7 @@ fn eq_neq_handle_bool_string_numeric() { ]; for (lhs, rhs, want_eq, label) in cases { let ladder: IfLadder = serde_json::from_value(json!({"if": [ - {"==": [{"var": "check.value"}, rhs.clone()]}, "error" + {"==": [{"var": "check.value"}, rhs.clone()]}, "failed" ]})) .unwrap(); let mut check = serde_json::Map::new(); @@ -206,7 +206,7 @@ fn eq_neq_handle_bool_string_numeric() { #[test] fn tag_namespace_resolves_against_tag_map() { let ladder: IfLadder = serde_json::from_value(json!({"if": [ - {"==": [{"var": "tag.environment"}, "prod"]}, "error" + {"==": [{"var": "tag.environment"}, "prod"]}, "failed" ]})) .unwrap(); let check = serde_json::Map::new(); @@ -219,7 +219,7 @@ fn tag_namespace_resolves_against_tag_map() { ); assert_eq!( ladder.evaluate(&empty_ctx(&check, &status, &tags)), - Some(Severity::Error), + Some(CheckResult::Failed), ); tags.insert( @@ -235,7 +235,7 @@ fn tag_namespace_resolves_against_tag_map() { #[test] fn first_match_wins() { let ladder: IfLadder = serde_json::from_value(json!({"if": [ - {"<": [{"var": "check.days_remaining"}, 7]}, "error", + {"<": [{"var": "check.days_remaining"}, 7]}, "failed", {"<": [{"var": "check.days_remaining"}, 30]}, "warning" ]})) .unwrap(); @@ -246,23 +246,23 @@ fn first_match_wins() { check.insert("days_remaining".into(), json!(3)); assert_eq!( ladder.evaluate(&empty_ctx(&check, &status, &tags)), - Some(Severity::Error), + Some(CheckResult::Failed), ); check.insert("days_remaining".into(), json!(15)); assert_eq!( ladder.evaluate(&empty_ctx(&check, &status, &tags)), - Some(Severity::Warning), + Some(CheckResult::Warning), ); check.insert("days_remaining".into(), json!(60)); assert_eq!(ladder.evaluate(&empty_ctx(&check, &status, &tags)), None); } -// ── severity_for integration ───────────────────────────────────────────── +// ── CheckPolicy::apply integration ─────────────────────────────────────── #[tokio::test(flavor = "multi_thread")] -async fn severity_for_uses_rules_or_falls_back_to_base() { +async fn apply_uses_rules_or_falls_back_to_ceiling() { commons_tests::db::TestDb::run(async |mut conn, _| { - HealthcheckSeverity::upsert_default(&mut conn, "tamanu_service") + CheckPolicy::upsert_default(&mut conn, "alertd", "tamanu_service") .await .expect("seed"); @@ -270,68 +270,87 @@ async fn severity_for_uses_rules_or_falls_back_to_base() { {"in_range": [{"var": "status.bestoolVersion"}, ">=2.4.0 <2.5.4"]}, "warning" ]})) .unwrap(); - HealthcheckSeverity::update_rules(&mut conn, "tamanu_service", Some(&ladder), "ops") + CheckPolicy::update_rules(&mut conn, "alertd", "tamanu_service", Some(&ladder), "ops") .await .expect("save rules"); - // Bestool inside the range → ladder fires → Warning. + // Bestool inside the range → ladder fires → warning. let mut status = serde_json::Map::new(); status.insert("bestoolVersion".into(), json!("2.4.7")); let check = serde_json::Map::new(); let tags = HashMap::new(); - let sev = HealthcheckSeverity::severity_for( + let graded = CheckPolicy::apply( &mut conn, + "alertd", "tamanu_service", CheckResult::Failed, &empty_ctx(&check, &status, &tags), ) .await - .expect("severity_for"); - assert_eq!(sev, Severity::Warning); + .expect("apply"); + assert_eq!(graded.effective, CheckResult::Warning); - // Bestool outside the range → falls back to the base (Warning by default). + // Bestool outside the range → capped at the default warning ceiling. status.insert("bestoolVersion".into(), json!("2.6.0")); - let sev = HealthcheckSeverity::severity_for( + let graded = CheckPolicy::apply( &mut conn, + "alertd", "tamanu_service", CheckResult::Failed, &empty_ctx(&check, &status, &tags), ) .await - .expect("severity_for"); - assert_eq!(sev, Severity::Warning); + .expect("apply"); + assert_eq!(graded.effective, CheckResult::Warning); - // Bump the base to Error and re-test fallback path. - HealthcheckSeverity::update(&mut conn, "tamanu_service", Severity::Error, None, "ops") - .await - .expect("bump base"); + // Lift the ceiling to failed and re-test the fallback path. + CheckPolicy::update( + &mut conn, + "alertd", + "tamanu_service", + CheckResult::Failed, + false, + None, + "ops", + ) + .await + .expect("lift ceiling"); status.insert("bestoolVersion".into(), json!("2.6.0")); - let sev = HealthcheckSeverity::severity_for( + let graded = CheckPolicy::apply( &mut conn, + "alertd", "tamanu_service", CheckResult::Failed, &empty_ctx(&check, &status, &tags), ) .await - .expect("severity_for"); + .expect("apply"); assert_eq!( - sev, - Severity::Error, - "falls back to base when no branch matches" + graded.effective, + CheckResult::Failed, + "falls back to the ceiling cap when no branch matches" ); }) .await } #[tokio::test(flavor = "multi_thread")] -async fn severity_for_warning_result_rules_then_fixed_warning() { +async fn apply_rules_transform_in_both_directions() { commons_tests::db::TestDb::run(async |mut conn, _| { - HealthcheckSeverity::upsert_default(&mut conn, "queue_depth") + CheckPolicy::upsert_default(&mut conn, "alertd", "queue_depth") .await .expect("seed"); - HealthcheckSeverity::update(&mut conn, "queue_depth", Severity::Critical, None, "ops") - .await - .expect("bump base"); + CheckPolicy::update( + &mut conn, + "alertd", + "queue_depth", + CheckResult::Failed, + true, + None, + "ops", + ) + .await + .expect("lift ceiling"); let status = serde_json::Map::new(); let tags = HashMap::new(); @@ -339,66 +358,91 @@ async fn severity_for_warning_result_rules_then_fixed_warning() { let mut check = serde_json::Map::new(); check.insert("result".into(), json!("warning")); - // No rules: warning-result checks ignore the catalog column and - // land at fixed Warning. - let sev = HealthcheckSeverity::severity_for( + // No rules: a warning observation is below the ceiling — unchanged. + let graded = CheckPolicy::apply( &mut conn, + "alertd", "queue_depth", CheckResult::Warning, &empty_ctx(&check, &status, &tags), ) .await - .expect("severity_for"); - assert_eq!(sev, Severity::Warning); + .expect("apply"); + assert_eq!(graded.effective, CheckResult::Warning); + assert!(graded.escalates, "the flag rides along whatever the result"); - // A rule conditioned on check.result overrides the fixed default. + // A rule conditioned on check.result grades warnings down to passed. let ladder: IfLadder = serde_json::from_value(json!({"if": [ - {"==": [{"var": "check.result"}, "warning"]}, "info" + {"==": [{"var": "check.result"}, "warning"]}, "passed" ]})) .unwrap(); - HealthcheckSeverity::update_rules(&mut conn, "queue_depth", Some(&ladder), "ops") + CheckPolicy::update_rules(&mut conn, "alertd", "queue_depth", Some(&ladder), "ops") .await .expect("save rules"); - let sev = HealthcheckSeverity::severity_for( + let graded = CheckPolicy::apply( &mut conn, + "alertd", "queue_depth", CheckResult::Warning, &empty_ctx(&check, &status, &tags), ) .await - .expect("severity_for"); - assert_eq!(sev, Severity::Info); + .expect("apply"); + assert_eq!(graded.effective, CheckResult::Passed); // The same rule doesn't fire for a failed check, which keeps - // using the catalog base. + // using the ceiling. let mut failed_check = serde_json::Map::new(); failed_check.insert("result".into(), json!("failed")); - let sev = HealthcheckSeverity::severity_for( + let graded = CheckPolicy::apply( &mut conn, + "alertd", "queue_depth", CheckResult::Failed, &empty_ctx(&failed_check, &status, &tags), ) .await - .expect("severity_for"); - assert_eq!(sev, Severity::Critical); + .expect("apply"); + assert_eq!(graded.effective, CheckResult::Failed); + assert!(graded.escalates); + + // And a rule can upgrade: a passed observation graded to warning. + let ladder: IfLadder = serde_json::from_value(json!({"if": [ + {"==": [{"var": "check.result"}, "passed"]}, "warning" + ]})) + .unwrap(); + CheckPolicy::update_rules(&mut conn, "alertd", "queue_depth", Some(&ladder), "ops") + .await + .expect("save rules"); + let mut passed_check = serde_json::Map::new(); + passed_check.insert("result".into(), json!("passed")); + let graded = CheckPolicy::apply( + &mut conn, + "alertd", + "queue_depth", + CheckResult::Passed, + &empty_ctx(&passed_check, &status, &tags), + ) + .await + .expect("apply"); + assert_eq!(graded.effective, CheckResult::Warning); }) .await } #[tokio::test(flavor = "multi_thread")] -async fn severity_for_falls_back_when_rules_are_malformed() { +async fn apply_falls_back_when_rules_are_malformed() { use diesel::sql_query; use diesel::sql_types; use diesel_async::RunQueryDsl; commons_tests::db::TestDb::run(async |mut conn, _| { - HealthcheckSeverity::upsert_default(&mut conn, "tamanu_service") + CheckPolicy::upsert_default(&mut conn, "alertd", "tamanu_service") .await .expect("seed"); // Manually inject garbage into the rules column. sql_query( - "UPDATE healthcheck_severities SET rules = $1::jsonb WHERE check_name = 'tamanu_service'", + "UPDATE check_policies SET rules = $1::jsonb WHERE check_name = 'tamanu_service'", ) .bind::(r#"{"and": [true, true]}"#) .execute(&mut conn) @@ -408,16 +452,18 @@ async fn severity_for_falls_back_when_rules_are_malformed() { let check = serde_json::Map::new(); let status = serde_json::Map::new(); let tags = HashMap::new(); - let sev = HealthcheckSeverity::severity_for( + let graded = CheckPolicy::apply( &mut conn, + "alertd", "tamanu_service", CheckResult::Failed, &empty_ctx(&check, &status, &tags), ) .await - .expect("severity_for"); - // Catalog default is Warning; malformed rules don't crash, just defer. - assert_eq!(sev, Severity::Warning); + .expect("apply"); + // Catalog default ceiling is warning; malformed rules don't crash, + // just defer. + assert_eq!(graded.effective, CheckResult::Warning); }) .await } @@ -425,20 +471,20 @@ async fn severity_for_falls_back_when_rules_are_malformed() { #[tokio::test(flavor = "multi_thread")] async fn update_rules_clears_column_when_passed_none() { commons_tests::db::TestDb::run(async |mut conn, _| { - HealthcheckSeverity::upsert_default(&mut conn, "disk_space") + CheckPolicy::upsert_default(&mut conn, "alertd", "disk_space") .await .expect("seed"); let ladder: IfLadder = serde_json::from_value(json!({"if": [ - {">": [{"var": "check.used_pct"}, 95]}, "critical" + {">": [{"var": "check.used_pct"}, 95]}, "failed" ]})) .unwrap(); let saved = - HealthcheckSeverity::update_rules(&mut conn, "disk_space", Some(&ladder), "ops") + CheckPolicy::update_rules(&mut conn, "alertd", "disk_space", Some(&ladder), "ops") .await .expect("save"); assert!(saved.rules.is_some(), "rules should be populated"); - let cleared = HealthcheckSeverity::update_rules(&mut conn, "disk_space", None, "ops") + let cleared = CheckPolicy::update_rules(&mut conn, "alertd", "disk_space", None, "ops") .await .expect("clear"); assert!(cleared.rules.is_none(), "rules should be cleared by None"); diff --git a/crates/database/tests/it/check_severity_map.rs b/crates/database/tests/it/check_severity_map.rs index c104f4cc..52b7355f 100644 --- a/crates/database/tests/it/check_severity_map.rs +++ b/crates/database/tests/it/check_severity_map.rs @@ -1,12 +1,13 @@ -//! Queries backing the device-facing effective check-severity map: -//! `HealthcheckSeverity::base_severity_map` (static catalog severities, -//! ignoring conditional rules) and `silenced_refs::silenced_refs_with_prefix` -//! (server- plus group-scope silences under a source/ref prefix). +//! Queries backing the device-facing effective check map: +//! `CheckPolicy::ceiling_map_for_source` (static policy ceilings, +//! ignoring conditional rules) and +//! `silenced_refs::silenced_health_checks_for_server` (server- plus +//! group-scope silences under one reporting source). -use commons_types::issue::Severity; -use database::healthcheck_severities::{HealthcheckSeverity, IfLadder}; +use commons_types::status::CheckResult; +use database::check_policies::{CheckPolicy, IfLadder}; use database::silenced_refs::{ - ServerGroupSilencedRef, ServerSilencedRef, silenced_refs_with_prefix, + ServerGroupSilencedRef, ServerSilencedRef, silenced_health_checks_for_server, }; use diesel::{sql_query, sql_types}; use diesel_async::RunQueryDsl; @@ -38,92 +39,120 @@ async fn insert_server(conn: &mut diesel_async::AsyncPgConnection, group_id: Opt } #[tokio::test(flavor = "multi_thread")] -async fn base_severity_map_returns_static_severities() { +async fn ceiling_map_returns_static_ceilings_for_one_source() { commons_tests::db::TestDb::run(async |mut conn, _| { for check in ["disk_space", "cert_expiry", "chatty"] { - HealthcheckSeverity::upsert_default(&mut conn, check) + CheckPolicy::upsert_default(&mut conn, "alertd", check) .await .expect("seed"); } - HealthcheckSeverity::update(&mut conn, "disk_space", Severity::Error, None, "alice") + CheckPolicy::upsert_default(&mut conn, "seedling", "other_source_check") .await - .expect("update disk_space"); - HealthcheckSeverity::update(&mut conn, "chatty", Severity::Info, None, "alice") - .await - .expect("update chatty"); + .expect("seed other source"); + CheckPolicy::update( + &mut conn, + "alertd", + "disk_space", + CheckResult::Failed, + false, + None, + "alice", + ) + .await + .expect("update disk_space"); + CheckPolicy::update( + &mut conn, + "alertd", + "chatty", + CheckResult::Passed, + false, + None, + "alice", + ) + .await + .expect("update chatty"); - let map = HealthcheckSeverity::base_severity_map(&mut conn) + let map = CheckPolicy::ceiling_map_for_source(&mut conn, "alertd") .await .expect("map"); - assert_eq!(map.len(), 3); - assert_eq!(map.get("disk_space"), Some(&Severity::Error)); - assert_eq!(map.get("cert_expiry"), Some(&Severity::Warning)); - assert_eq!(map.get("chatty"), Some(&Severity::Info)); + assert_eq!(map.len(), 3, "only the requested source's checks"); + assert_eq!(map.get("disk_space"), Some(&CheckResult::Failed)); + assert_eq!(map.get("cert_expiry"), Some(&CheckResult::Warning)); + assert_eq!(map.get("chatty"), Some(&CheckResult::Passed)); }) .await } #[tokio::test(flavor = "multi_thread")] -async fn base_severity_map_ignores_conditional_rules() { +async fn ceiling_map_ignores_conditional_rules() { commons_tests::db::TestDb::run(async |mut conn, _| { - HealthcheckSeverity::upsert_default(&mut conn, "ruled") + CheckPolicy::upsert_default(&mut conn, "alertd", "ruled") .await .expect("seed"); let ladder: IfLadder = serde_json::from_value(json!({"if": [ - {"==": [{"var": "check.result"}, "failed"]}, "critical", + {"==": [{"var": "check.result"}, "failed"]}, "failed", ]})) .expect("parse ladder"); - HealthcheckSeverity::update_rules(&mut conn, "ruled", Some(&ladder), "alice") + CheckPolicy::update_rules(&mut conn, "alertd", "ruled", Some(&ladder), "alice") .await .expect("set rules"); - // The expression could raise a failure to critical at push time, but - // the static map must only reflect the base severity column. - let map = HealthcheckSeverity::base_severity_map(&mut conn) + // The expression could grade a failure through at push time, but + // the static map must only reflect the ceiling column. + let map = CheckPolicy::ceiling_map_for_source(&mut conn, "alertd") .await .expect("map"); - assert_eq!(map.get("ruled"), Some(&Severity::Warning)); + assert_eq!(map.get("ruled"), Some(&CheckResult::Warning)); }) .await } #[tokio::test(flavor = "multi_thread")] -async fn silenced_refs_with_prefix_combines_scopes_and_filters() { +async fn silenced_checks_combine_scopes_and_stay_per_source() { commons_tests::db::TestDb::run(async |mut conn, _| { let group_id = insert_group(&mut conn).await; let server_id = insert_server(&mut conn, Some(group_id)).await; let other_server_id = insert_server(&mut conn, None).await; - ServerSilencedRef::add(&mut conn, server_id, "status", "health/flaky", None) + ServerSilencedRef::add(&mut conn, server_id, "alertd", "health/flaky", None) .await .expect("server silence"); - ServerGroupSilencedRef::add(&mut conn, group_id, "status", "health/groupwide", None) + ServerGroupSilencedRef::add(&mut conn, group_id, "alertd", "health/groupwide", None) .await .expect("group silence"); - // None of these may leak into the result: wrong source, wrong ref - // prefix (broken issues are a separate thread), wrong server. - ServerSilencedRef::add(&mut conn, server_id, "canopy", "health/wrong-source", None) - .await - .expect("other-source silence"); - ServerSilencedRef::add(&mut conn, server_id, "status", "health-broken/flaky", None) + // None of these may leak into alertd's set: a check's identity is + // the (source, check) pair, so another source's silence never + // applies; nor do canopy's own silences or other servers'. + ServerSilencedRef::add( + &mut conn, + server_id, + "seedling", + "health/other-source", + None, + ) + .await + .expect("other-source silence"); + ServerSilencedRef::add(&mut conn, server_id, "canopy", "reachability", None) .await - .expect("broken silence"); - ServerSilencedRef::add(&mut conn, other_server_id, "status", "health/other", None) + .expect("canopy silence"); + ServerSilencedRef::add(&mut conn, other_server_id, "alertd", "health/other", None) .await .expect("other-server silence"); - let mut refs = - silenced_refs_with_prefix(&mut conn, server_id, Some(group_id), "status", "health/") + let checks = + silenced_health_checks_for_server(&mut conn, server_id, Some(group_id), "alertd") .await - .expect("refs"); - refs.sort(); - assert_eq!(refs, vec!["health/flaky", "health/groupwide"]); + .expect("checks"); + assert_eq!( + checks.into_iter().collect::>(), + vec!["flaky", "groupwide"] + ); // Ungrouped lookup only sees the server-scope silences. - let refs = silenced_refs_with_prefix(&mut conn, server_id, None, "status", "health/") + let checks = silenced_health_checks_for_server(&mut conn, server_id, None, "alertd") .await - .expect("refs without group"); - assert_eq!(refs, vec!["health/flaky"]); + .expect("checks without group"); + assert_eq!(checks.into_iter().collect::>(), vec!["flaky"]); }) .await } diff --git a/crates/database/tests/it/event_validation.rs b/crates/database/tests/it/event_validation.rs index d5d1ecfc..29208124 100644 --- a/crates/database/tests/it/event_validation.rs +++ b/crates/database/tests/it/event_validation.rs @@ -2,7 +2,6 @@ //! single-line title, multi-line content belongs in `message`. use commons_errors::AppError; -use commons_types::issue::Severity; use database::issues::NewEvent; use uuid::Uuid; @@ -12,7 +11,6 @@ async fn save_rejects_newline_in_description() { let event = NewEvent { source: "test".into(), r#ref: "ref-newline".into(), - severity: Some(Severity::Error), description: Some("line one\nline two".into()), message: "body".into(), active: Some(true), @@ -36,7 +34,6 @@ async fn save_accepts_single_line_description() { let event = NewEvent { source: "test".into(), r#ref: "ref-singleline".into(), - severity: Some(Severity::Error), description: Some("a perfectly fine subject line".into()), message: "body".into(), active: Some(true), diff --git a/crates/database/tests/it/healthcheck_severities.rs b/crates/database/tests/it/healthcheck_severities.rs deleted file mode 100644 index bf04a25a..00000000 --- a/crates/database/tests/it/healthcheck_severities.rs +++ /dev/null @@ -1,128 +0,0 @@ -//! `HealthcheckSeverity` model: catalog of healthcheck names → severity, -//! maintained by the public-server status ingestion path and edited -//! through the private-server `/api/healthchecks` endpoints. - -use commons_types::{issue::Severity, status::CheckResult}; -use database::healthcheck_severities::HealthcheckSeverity; - -#[tokio::test(flavor = "multi_thread")] -async fn upsert_default_inserts_then_is_idempotent() { - commons_tests::db::TestDb::run(async |mut conn, _| { - HealthcheckSeverity::upsert_default(&mut conn, "disk_space") - .await - .expect("first upsert"); - HealthcheckSeverity::upsert_default(&mut conn, "disk_space") - .await - .expect("second upsert is no-op"); - - // Both upserts should land at the schema default, pending review. - let rows = HealthcheckSeverity::list(&mut conn).await.expect("list"); - assert_eq!(rows.len(), 1); - assert_eq!(rows[0].check_name, "disk_space"); - assert_eq!(rows[0].severity, Severity::Warning); - assert!(rows[0].reviewed_at.is_none(), "pending review by default"); - assert!(rows[0].reviewed_by.is_none()); - }) - .await -} - -#[tokio::test(flavor = "multi_thread")] -async fn upsert_default_does_not_overwrite_operator_severity() { - commons_tests::db::TestDb::run(async |mut conn, _| { - HealthcheckSeverity::upsert_default(&mut conn, "disk_space") - .await - .expect("seed"); - HealthcheckSeverity::update(&mut conn, "disk_space", Severity::Error, None, "alice") - .await - .expect("operator update"); - - // A subsequent push from ingestion must not revert the row. - HealthcheckSeverity::upsert_default(&mut conn, "disk_space") - .await - .expect("upsert again"); - - let rows = HealthcheckSeverity::list(&mut conn).await.expect("list"); - assert_eq!(rows.len(), 1); - assert_eq!(rows[0].severity, Severity::Error); - assert!(rows[0].reviewed_at.is_some()); - assert_eq!(rows[0].reviewed_by.as_deref(), Some("alice")); - }) - .await -} - -#[tokio::test(flavor = "multi_thread")] -async fn severity_for_returns_catalog_value_or_warning_default() { - commons_tests::db::TestDb::run(async |mut conn, _| { - let empty_map = serde_json::Map::new(); - let empty_tags = std::collections::HashMap::new(); - let ctx = database::healthcheck_severities::EvaluationContext { - status_extra: &empty_map, - check_extra: &empty_map, - tags: &empty_tags, - }; - // Unknown check → fallback (programmer-error path; ingestion - // upserts before reading in production). - let unknown = - HealthcheckSeverity::severity_for(&mut conn, "ghost", CheckResult::Failed, &ctx) - .await - .expect("lookup"); - assert_eq!(unknown, Severity::Warning); - - HealthcheckSeverity::upsert_default(&mut conn, "cert_expiry") - .await - .expect("seed"); - HealthcheckSeverity::update(&mut conn, "cert_expiry", Severity::Critical, None, "bob") - .await - .expect("update"); - - let known = - HealthcheckSeverity::severity_for(&mut conn, "cert_expiry", CheckResult::Failed, &ctx) - .await - .expect("lookup"); - assert_eq!(known, Severity::Critical); - - // A warning-result check ignores the catalog column and lands - // at fixed Warning when no rule matches. - let warned = - HealthcheckSeverity::severity_for(&mut conn, "cert_expiry", CheckResult::Warning, &ctx) - .await - .expect("lookup"); - assert_eq!(warned, Severity::Warning); - }) - .await -} - -#[tokio::test(flavor = "multi_thread")] -async fn update_stamps_review_metadata_even_on_no_op_save() { - commons_tests::db::TestDb::run(async |mut conn, _| { - HealthcheckSeverity::upsert_default(&mut conn, "noisy_check") - .await - .expect("seed"); - - // "Mark reviewed without changing severity": pass the current - // (default) value. reviewed_at + reviewed_by must still be set. - let updated = - HealthcheckSeverity::update(&mut conn, "noisy_check", Severity::Warning, None, "carol") - .await - .expect("update"); - assert_eq!(updated.severity, Severity::Warning); - assert!(updated.reviewed_at.is_some()); - assert_eq!(updated.reviewed_by.as_deref(), Some("carol")); - }) - .await -} - -#[tokio::test(flavor = "multi_thread")] -async fn list_orders_by_check_name() { - commons_tests::db::TestDb::run(async |mut conn, _| { - for name in ["zeta", "alpha", "mu"] { - HealthcheckSeverity::upsert_default(&mut conn, name) - .await - .expect("seed"); - } - let rows = HealthcheckSeverity::list(&mut conn).await.expect("list"); - let names: Vec<&str> = rows.iter().map(|r| r.check_name.as_str()).collect(); - assert_eq!(names, vec!["alpha", "mu", "zeta"]); - }) - .await -} diff --git a/crates/database/tests/it/incident_close_severity.rs b/crates/database/tests/it/incident_close_result.rs similarity index 87% rename from crates/database/tests/it/incident_close_severity.rs rename to crates/database/tests/it/incident_close_result.rs index 6f04515b..b10f6e38 100644 --- a/crates/database/tests/it/incident_close_severity.rs +++ b/crates/database/tests/it/incident_close_result.rs @@ -1,10 +1,10 @@ //! The auto-close path in `re_evaluate_incident_membership` counts only -//! severity ≥ error contributors when deciding whether an incident still -//! has reason to stay open. Lower-severity issues that joined while the +//! effective-failure contributors when deciding whether an incident +//! still has reason to stay open. Lesser issues that joined while the //! incident was open stay attached (audit trail / Slack context) but //! don't hold the incident open by themselves. -use commons_types::issue::Severity; +use commons_types::status::CheckResult; use database::issues::{Incident, NewEvent}; use database::slack_outbox::{KIND_INCIDENT_OPEN, KIND_INCIDENT_RESOLVE, SlackOutbox}; use diesel::{QueryableByName, sql_query, sql_types}; @@ -36,20 +36,30 @@ async fn save_event( conn: &mut diesel_async::AsyncPgConnection, server_id: Uuid, r#ref: &str, - severity: Severity, - active: bool, + result: CheckResult, + escalates: bool, message: &str, ) { + let active = matches!( + result, + CheckResult::Failed | CheckResult::Warning | CheckResult::Broken + ); + let stamp = database::issues::CheckStateStamp { + check: r#ref.into(), + observed: result, + effective: result, + escalates, + detail: None, + }; NewEvent { source: "test".into(), r#ref: r#ref.into(), - severity: Some(severity), description: None, message: message.into(), active: Some(active), occurred_at: None, } - .save(conn, server_id, None) + .save_with_state(conn, server_id, None, Some(&stamp)) .await .expect("save event"); } @@ -109,8 +119,8 @@ async fn warning_does_not_hold_incident_open_after_error_resolves() { &mut conn, server_id, "error-ref", - Severity::Error, - true, + CheckResult::Failed, + false, "boom", ) .await; @@ -130,8 +140,8 @@ async fn warning_does_not_hold_incident_open_after_error_resolves() { &mut conn, server_id, "warning-ref", - Severity::Warning, - true, + CheckResult::Warning, + false, "noise", ) .await; @@ -150,7 +160,7 @@ async fn warning_does_not_hold_incident_open_after_error_resolves() { &mut conn, server_id, "error-ref", - Severity::Info, + CheckResult::Passed, false, "recovered", ) @@ -183,15 +193,31 @@ async fn incident_stays_open_while_a_second_error_contributor_is_alive() { commons_tests::db::TestDb::run(async |mut conn, _| { let server_id = insert_grouped_server(&mut conn, "http://two-errors.invalid/").await; - save_event(&mut conn, server_id, "error-a", Severity::Error, true, "a").await; - save_event(&mut conn, server_id, "error-b", Severity::Error, true, "b").await; + save_event( + &mut conn, + server_id, + "error-a", + CheckResult::Failed, + false, + "a", + ) + .await; + save_event( + &mut conn, + server_id, + "error-b", + CheckResult::Failed, + false, + "b", + ) + .await; // Resolve only one. save_event( &mut conn, server_id, "error-a", - Severity::Info, + CheckResult::Passed, false, "recovered", ) @@ -221,8 +247,8 @@ async fn stranded_warning_resolve_does_not_re_enqueue_slack() { &mut conn, server_id, "error-ref", - Severity::Error, - true, + CheckResult::Failed, + false, "boom", ) .await; @@ -237,8 +263,8 @@ async fn stranded_warning_resolve_does_not_re_enqueue_slack() { &mut conn, server_id, "warning-ref", - Severity::Warning, - true, + CheckResult::Warning, + false, "noise", ) .await; @@ -248,7 +274,7 @@ async fn stranded_warning_resolve_does_not_re_enqueue_slack() { &mut conn, server_id, "error-ref", - Severity::Info, + CheckResult::Passed, false, "recovered", ) @@ -261,7 +287,7 @@ async fn stranded_warning_resolve_does_not_re_enqueue_slack() { &mut conn, server_id, "warning-ref", - Severity::Info, + CheckResult::Passed, false, "settled", ) diff --git a/crates/database/tests/it/incident_get_with_issues.rs b/crates/database/tests/it/incident_get_with_issues.rs index fa344958..83251dce 100644 --- a/crates/database/tests/it/incident_get_with_issues.rs +++ b/crates/database/tests/it/incident_get_with_issues.rs @@ -25,10 +25,10 @@ async fn get_with_issues_dedupes_repeat_join_rows() { INSERT INTO servers (id, host, kind, device_id, group_id) VALUES \ ('{server_id}', 'https://example.com', 'central', '{device_id}', '{group_id}'); \ INSERT INTO issues \ - (id, server_id, device_id, source, ref, severity, message, active, first_seen, last_seen) \ + (id, server_id, device_id, source, ref, check_name, observed_result, effective_result, message, active, first_seen, last_seen) \ VALUES \ - ('{issue_a}', '{server_id}', '{device_id}', 'test', 'a', 'error', 'm', true, NOW(), NOW()), \ - ('{issue_b}', '{server_id}', '{device_id}', 'test', 'b', 'error', 'm', true, NOW(), NOW()); \ + ('{issue_a}', '{server_id}', '{device_id}', 'test', 'a', 'a', 'failed', 'failed', 'm', true, NOW(), NOW()), \ + ('{issue_b}', '{server_id}', '{device_id}', 'test', 'b', 'b', 'failed', 'failed', 'm', true, NOW(), NOW()); \ INSERT INTO incidents (id, server_group_id, opened_at) \ VALUES ('{incident_id}', '{group_id}', NOW()); \ INSERT INTO incident_issues (incident_id, issue_id, joined_at, left_at) VALUES \ diff --git a/crates/database/tests/it/incident_severity_semantics.rs b/crates/database/tests/it/incident_result_semantics.rs similarity index 90% rename from crates/database/tests/it/incident_severity_semantics.rs rename to crates/database/tests/it/incident_result_semantics.rs index 1a85d7e4..b9fcfe5d 100644 --- a/crates/database/tests/it/incident_severity_semantics.rs +++ b/crates/database/tests/it/incident_result_semantics.rs @@ -1,13 +1,14 @@ -//! Severity semantics for the incident workflow: +//! Result semantics for the incident workflow: //! -//! - **Debug** never participates in incidents — neither joining a new -//! one nor staying attached if a contributor's severity drops to -//! Debug after the fact. -//! - **Critical** opens the incident (or joins it) without sitting in -//! the per-group `slack_open_delay` holding window: the outbox row's -//! `deliver_after` is pulled forward to NOW(). +//! - A **skipped** effective result never participates in incidents — +//! neither joining a new one nor staying attached once a contributor +//! grades to skipped after the fact. +//! - An **escalating failure** opens the incident (or joins it) without +//! sitting in the per-group `slack_open_delay` holding window: the +//! outbox row's `deliver_after` is pulled forward to NOW(). -use commons_types::issue::{ResolvedReason, Severity}; +use commons_types::issue::ResolvedReason; +use commons_types::status::CheckResult; use database::issues::{Incident, NewEvent}; use database::slack_outbox::{KIND_INCIDENT_OPEN, SlackOutbox}; use diesel::{QueryableByName, sql_query, sql_types}; @@ -39,20 +40,30 @@ async fn save_event( conn: &mut diesel_async::AsyncPgConnection, server_id: Uuid, r#ref: &str, - severity: Severity, - active: bool, + result: CheckResult, + escalates: bool, message: &str, ) { + let active = matches!( + result, + CheckResult::Failed | CheckResult::Warning | CheckResult::Broken + ); + let stamp = database::issues::CheckStateStamp { + check: r#ref.into(), + observed: result, + effective: result, + escalates, + detail: None, + }; NewEvent { source: "test".into(), r#ref: r#ref.into(), - severity: Some(severity), description: None, message: message.into(), active: Some(active), occurred_at: None, } - .save(conn, server_id, None) + .save_with_state(conn, server_id, None, Some(&stamp)) .await .expect("save event"); } @@ -111,8 +122,8 @@ async fn debug_issue_does_not_join_open_incidents() { &mut conn, server_id, "real-error", - Severity::Error, - true, + CheckResult::Failed, + false, "boom", ) .await; @@ -128,8 +139,8 @@ async fn debug_issue_does_not_join_open_incidents() { &mut conn, server_id, "debug-noise", - Severity::Debug, - true, + CheckResult::Skipped, + false, "low signal", ) .await; @@ -155,8 +166,8 @@ async fn issue_downgraded_to_debug_leaves_incident_on_next_evaluation() { &mut conn, server_id, "main-error", - Severity::Error, - true, + CheckResult::Failed, + false, "boom", ) .await; @@ -164,8 +175,8 @@ async fn issue_downgraded_to_debug_leaves_incident_on_next_evaluation() { &mut conn, server_id, "noisy", - Severity::Warning, - true, + CheckResult::Warning, + false, "noise", ) .await; @@ -176,8 +187,8 @@ async fn issue_downgraded_to_debug_leaves_incident_on_next_evaluation() { &mut conn, server_id, "noisy", - Severity::Debug, - true, + CheckResult::Skipped, + false, "demoted", ) .await; @@ -200,7 +211,7 @@ async fn critical_open_sets_deliver_after_to_now() { &mut conn, server_id, "crit", - Severity::Critical, + CheckResult::Failed, true, "red alert", ) @@ -229,8 +240,8 @@ async fn non_critical_open_still_honours_holding_window() { &mut conn, server_id, "boom", - Severity::Error, - true, + CheckResult::Failed, + false, "less urgent", ) .await; @@ -261,8 +272,8 @@ async fn critical_joining_existing_open_accelerates_pending_delivery() { &mut conn, server_id, "warmup", - Severity::Error, - true, + CheckResult::Failed, + false, "boom", ) .await; @@ -284,7 +295,7 @@ async fn critical_joining_existing_open_accelerates_pending_delivery() { &mut conn, server_id, "crit", - Severity::Critical, + CheckResult::Failed, true, "red alert", ) @@ -315,8 +326,8 @@ async fn critical_join_after_delivered_open_fires_escalation_open() { &mut conn, server_id, "warmup", - Severity::Error, - true, + CheckResult::Failed, + false, "boom", ) .await; @@ -341,7 +352,7 @@ async fn critical_join_after_delivered_open_fires_escalation_open() { &mut conn, server_id, "crit", - Severity::Critical, + CheckResult::Failed, true, "red alert", ) @@ -405,8 +416,8 @@ async fn repeated_critical_joins_do_not_re_fire_escalation() { &mut conn, server_id, "warmup", - Severity::Error, - true, + CheckResult::Failed, + false, "boom", ) .await; @@ -426,7 +437,7 @@ async fn repeated_critical_joins_do_not_re_fire_escalation() { &mut conn, server_id, "crit-1", - Severity::Critical, + CheckResult::Failed, true, "red alert", ) @@ -436,7 +447,7 @@ async fn repeated_critical_joins_do_not_re_fire_escalation() { &mut conn, server_id, "crit-2", - Severity::Critical, + CheckResult::Failed, true, "also red", ) @@ -475,8 +486,8 @@ async fn debug_filing_still_records_the_issue_row() { &mut conn, server_id, "logspam", - Severity::Debug, - true, + CheckResult::Skipped, + false, "verbose", ) .await; diff --git a/crates/database/tests/it/incident_stats.rs b/crates/database/tests/it/incident_stats.rs index be9f6677..567430d2 100644 --- a/crates/database/tests/it/incident_stats.rs +++ b/crates/database/tests/it/incident_stats.rs @@ -2,8 +2,8 @@ //! counting. `incident_issues` is keyed on `(incident_id, issue_id, //! joined_at)`, so an issue that leaves and rejoins the same incident //! produces multiple rows for the same pair. Without dedup, the issue -//! count inflates by the number of rejoins, and the event/note counts -//! get multiplied by the same factor. +//! count inflates by the number of rejoins, and the note counts get +//! multiplied by the same factor. use database::issues::Incident; use diesel_async::SimpleAsyncConnection as _; @@ -17,8 +17,6 @@ async fn stats_for_dedupes_repeat_join_rows() { let device_id = Uuid::new_v4(); let issue_id = Uuid::new_v4(); let incident_id = Uuid::new_v4(); - let event_a = Uuid::new_v4(); - let event_b = Uuid::new_v4(); let issue_note = Uuid::new_v4(); let incident_note = Uuid::new_v4(); @@ -28,16 +26,11 @@ async fn stats_for_dedupes_repeat_join_rows() { INSERT INTO servers (id, host, kind, device_id, group_id) VALUES \ ('{server_id}', 'https://example.com', 'central', '{device_id}', '{group_id}'); \ INSERT INTO issues \ - (id, server_id, device_id, source, ref, severity, message, active, first_seen, last_seen) \ + (id, server_id, device_id, source, ref, check_name, observed_result, effective_result, message, active, first_seen, last_seen) \ VALUES \ - ('{issue_id}', '{server_id}', '{device_id}', 'test', 'r', 'error', 'm', true, NOW(), NOW()); \ + ('{issue_id}', '{server_id}', '{device_id}', 'test', 'r', 'r', 'failed', 'failed', 'm', true, NOW(), NOW()); \ INSERT INTO incidents (id, server_group_id, opened_at) \ VALUES ('{incident_id}', '{group_id}', NOW()); \ - INSERT INTO events \ - (id, issue_id, severity, message, active, hash, occurrences, last_seen) \ - VALUES \ - ('{event_a}', '{issue_id}', 'error', 'one', true, '\\x01'::bytea, 7, NOW()), \ - ('{event_b}', '{issue_id}', 'error', 'two', true, '\\x02'::bytea, 1, NOW()); \ INSERT INTO issue_notes (id, issue_id, author, body) \ VALUES ('{issue_note}', '{issue_id}', 'op', 'jn'); \ INSERT INTO incident_notes (id, incident_id, author, body) \ @@ -59,10 +52,6 @@ async fn stats_for_dedupes_repeat_join_rows() { .expect("stats_for"); let s = stats.get(&incident_id).expect("stats present"); assert_eq!(s.issue_count, 1, "one distinct issue despite 5 join rows"); - assert_eq!( - s.event_count, 2, - "two event rows, not multiplied by 5 join rows" - ); assert_eq!( s.note_count, 2, "one incident_note + one issue_note, not 1 + 5" @@ -88,7 +77,6 @@ async fn stats_for_handles_missing_and_empty_inputs() { .expect("phantom"); let s = stats.get(&phantom).expect("phantom entry"); assert_eq!(s.issue_count, 0); - assert_eq!(s.event_count, 0); assert_eq!(s.note_count, 0); }) .await diff --git a/crates/database/tests/it/main.rs b/crates/database/tests/it/main.rs index 241e0647..12a38ca5 100644 --- a/crates/database/tests/it/main.rs +++ b/crates/database/tests/it/main.rs @@ -6,19 +6,19 @@ mod backfill_registered_at_migration; mod backup_detection; mod backups; +mod check_policies; +mod check_policy_rules; mod check_severity_map; mod event_validation; -mod healthcheck_severities; -mod healthcheck_severity_rules; -mod incident_close_severity; +mod incident_close_result; mod incident_get_with_issues; -mod incident_severity_semantics; +mod incident_result_semantics; mod incident_stats; mod mcp_tokens; mod reachability_sweep; mod recovery_vault; -mod resolve_overall_health_rollups_migration; mod restore; +mod scoped_check_policies; mod self_alerts; mod server_enrollment; mod server_group_archival; diff --git a/crates/database/tests/it/mcp_tokens.rs b/crates/database/tests/it/mcp_tokens.rs index c27c25ad..5bc51298 100644 --- a/crates/database/tests/it/mcp_tokens.rs +++ b/crates/database/tests/it/mcp_tokens.rs @@ -169,7 +169,8 @@ async fn expiry_sweep_files_one_self_alert_and_recovers() { panic!("exactly one self-alert, got: {alerts:?}"); }; assert!(issue.active); - assert_eq!(issue.server_id, Some(Uuid::nil())); + assert_eq!(issue.server_id, None); + assert_eq!(issue.server_group_id, None); assert!(issue.server_group_id.is_none()); assert!(issue.message.contains("claude"), "{}", issue.message); assert!(issue.message.contains("in 10 days"), "{}", issue.message); diff --git a/crates/database/tests/it/reachability_sweep.rs b/crates/database/tests/it/reachability_sweep.rs index e39e8778..0e67a130 100644 --- a/crates/database/tests/it/reachability_sweep.rs +++ b/crates/database/tests/it/reachability_sweep.rs @@ -1,7 +1,7 @@ -use commons_types::issue::Severity; +use commons_types::status::CheckResult; use database::{ issues::Issue, - statuses::{CANOPY_SOURCE, REACHABILITY_REF, Status}, + statuses::{CANOPY_SOURCE, REACHABILITY_REF, STALE_REF_PREFIX, Status}, }; use diesel::{QueryableByName, sql_query, sql_types}; use diesel_async::RunQueryDsl; @@ -71,16 +71,62 @@ async fn issue_for(conn: &mut diesel_async::AsyncPgConnection, server_id: Uuid) .next() } +/// Seed check state as if `source` reported `check` on the server +/// `minutes_ago` — what ingestion stamps on every report, and what the +/// per-source staleness arm reads freshness from. +async fn insert_check_state( + conn: &mut diesel_async::AsyncPgConnection, + server_id: Uuid, + source: &str, + check: &str, + minutes_ago: i32, +) { + sql_query( + r#" + INSERT INTO issues + (server_id, source, ref, check_name, observed_result, effective_result, + message, active, first_seen, last_seen) + VALUES ($1, $2, 'health/' || $3, $3, 'passed', 'passed', + 'seeded', false, + NOW() - ($4 || ' minutes')::INTERVAL, NOW() - ($4 || ' minutes')::INTERVAL) + "#, + ) + .bind::(server_id) + .bind::(source) + .bind::(check) + .bind::(minutes_ago.to_string()) + .execute(conn) + .await + .expect("insert check state"); +} + +async fn stale_issue_for( + conn: &mut diesel_async::AsyncPgConnection, + server_id: Uuid, + source: &str, +) -> Option { + Issue::list_by_source_ref( + conn, + CANOPY_SOURCE, + &format!("{STALE_REF_PREFIX}{source}"), + &[server_id], + ) + .await + .expect("list stale issues") + .into_iter() + .next() +} + #[tokio::test(flavor = "multi_thread")] async fn sweep_files_error_when_threshold_crossed() { commons_tests::db::TestDb::run(async |mut conn, _| { // 10-min threshold, status 15 minutes old → cross. let id = insert_server(&mut conn, "http://down.invalid/", 600).await; insert_status_at(&mut conn, id, 15).await; - let filed = Status::sweep_reachability(&mut conn).await.expect("sweep"); + let filed = Status::sweep_staleness(&mut conn).await.expect("sweep"); assert_eq!(filed, 1); let issue = issue_for(&mut conn, id).await.expect("issue exists"); - assert_eq!(issue.severity, Severity::Error); + assert_eq!(issue.effective_result, Some(CheckResult::Failed)); assert!(issue.active); }) .await @@ -92,7 +138,7 @@ async fn sweep_skips_when_below_threshold() { // 10-min threshold, status 5 minutes old → still fresh. let id = insert_server(&mut conn, "http://fresh.invalid/", 600).await; insert_status_at(&mut conn, id, 5).await; - let filed = Status::sweep_reachability(&mut conn).await.expect("sweep"); + let filed = Status::sweep_staleness(&mut conn).await.expect("sweep"); assert_eq!(filed, 0); assert!(issue_for(&mut conn, id).await.is_none()); }) @@ -104,10 +150,10 @@ async fn sweep_files_when_no_status_ever() { commons_tests::db::TestDb::run(async |mut conn, _| { // No status row at all → infinite downtime → always crosses threshold. let id = insert_server(&mut conn, "http://gone.invalid/", 600).await; - let filed = Status::sweep_reachability(&mut conn).await.expect("sweep"); + let filed = Status::sweep_staleness(&mut conn).await.expect("sweep"); assert_eq!(filed, 1); let issue = issue_for(&mut conn, id).await.expect("issue exists"); - assert_eq!(issue.severity, Severity::Error); + assert_eq!(issue.effective_result, Some(CheckResult::Failed)); assert!(issue.active); assert!( issue.message.contains("has never reported"), @@ -128,7 +174,7 @@ async fn sweep_skips_unmonitored_server() { // resumes with the chosen value. let id = insert_server_full(&mut conn, "http://silenced.invalid/", 600, false).await; insert_status_at(&mut conn, id, 120).await; - let filed = Status::sweep_reachability(&mut conn).await.expect("sweep"); + let filed = Status::sweep_staleness(&mut conn).await.expect("sweep"); assert_eq!(filed, 0); assert!(issue_for(&mut conn, id).await.is_none()); }) @@ -140,7 +186,7 @@ async fn sweep_skips_up_server() { commons_tests::db::TestDb::run(async |mut conn, _| { let id = insert_server(&mut conn, "http://up.invalid/", 600).await; insert_status_at(&mut conn, id, 0).await; - let filed = Status::sweep_reachability(&mut conn).await.expect("sweep"); + let filed = Status::sweep_staleness(&mut conn).await.expect("sweep"); assert_eq!(filed, 0); assert!(issue_for(&mut conn, id).await.is_none()); }) @@ -157,7 +203,7 @@ async fn sweep_uses_per_server_threshold() { let critical = insert_server(&mut conn, "http://critical.invalid/", 60).await; insert_status_at(&mut conn, critical, 15).await; - let filed = Status::sweep_reachability(&mut conn).await.expect("sweep"); + let filed = Status::sweep_staleness(&mut conn).await.expect("sweep"); assert_eq!(filed, 1); assert!(issue_for(&mut conn, flappy).await.is_none()); assert!(issue_for(&mut conn, critical).await.is_some()); @@ -213,19 +259,135 @@ async fn check_constraint_forbids_non_positive_duration() { .await } +#[tokio::test(flavor = "multi_thread")] +async fn sweep_files_stale_source_at_warning() { + commons_tests::db::TestDb::run(async |mut conn, _| { + // The server itself is reachable (fresh status row), but alertd's + // check state was last stamped 15 minutes ago against a 10-min + // threshold → the source has gone quiet. + let id = insert_server(&mut conn, "http://quiet-source.invalid/", 600).await; + insert_status_at(&mut conn, id, 0).await; + insert_check_state(&mut conn, id, "alertd", "db", 15).await; + + let filed = Status::sweep_staleness(&mut conn).await.expect("sweep"); + assert_eq!(filed, 1); + assert!( + issue_for(&mut conn, id).await.is_none(), + "reachability must not fire while a fresh status row exists" + ); + let stale = stale_issue_for(&mut conn, id, "alertd") + .await + .expect("stale issue exists"); + assert!(stale.active); + assert_eq!( + stale.observed_result, + Some(commons_types::status::CheckResult::Failed) + ); + assert_eq!( + stale.effective_result, + Some(commons_types::status::CheckResult::Warning), + "stale/ registers at a warning ceiling" + ); + assert!(stale.message.contains("has not reported for")); + }) + .await +} + +#[tokio::test(flavor = "multi_thread")] +async fn sweep_skips_fresh_source() { + commons_tests::db::TestDb::run(async |mut conn, _| { + let id = insert_server(&mut conn, "http://fresh-source.invalid/", 600).await; + insert_status_at(&mut conn, id, 0).await; + insert_check_state(&mut conn, id, "alertd", "db", 5).await; + + let filed = Status::sweep_staleness(&mut conn).await.expect("sweep"); + assert_eq!(filed, 0); + assert!(stale_issue_for(&mut conn, id, "alertd").await.is_none()); + }) + .await +} + +#[tokio::test(flavor = "multi_thread")] +async fn sweep_flags_only_the_stale_source() { + commons_tests::db::TestDb::run(async |mut conn, _| { + let id = insert_server(&mut conn, "http://mixed-sources.invalid/", 600).await; + insert_status_at(&mut conn, id, 0).await; + insert_check_state(&mut conn, id, "alertd", "db", 45).await; + insert_check_state(&mut conn, id, "tamanu", "tasks", 2).await; + + let filed = Status::sweep_staleness(&mut conn).await.expect("sweep"); + assert_eq!(filed, 1); + assert!( + stale_issue_for(&mut conn, id, "alertd") + .await + .unwrap() + .active + ); + assert!(stale_issue_for(&mut conn, id, "tamanu").await.is_none()); + }) + .await +} + +#[tokio::test(flavor = "multi_thread")] +async fn sweep_closes_stale_source_when_it_reports_again() { + commons_tests::db::TestDb::run(async |mut conn, _| { + let id = insert_server(&mut conn, "http://returning-source.invalid/", 600).await; + insert_status_at(&mut conn, id, 0).await; + insert_check_state(&mut conn, id, "alertd", "db", 45).await; + Status::sweep_staleness(&mut conn) + .await + .expect("first sweep"); + assert!( + stale_issue_for(&mut conn, id, "alertd") + .await + .unwrap() + .active + ); + + // The source reports again: ingestion re-stamps its check state. + sql_query("UPDATE issues SET last_seen = NOW() WHERE server_id = $1 AND source = 'alertd'") + .bind::(id) + .execute(&mut conn) + .await + .expect("restamp state"); + Status::sweep_staleness(&mut conn) + .await + .expect("second sweep"); + let stale = stale_issue_for(&mut conn, id, "alertd").await.unwrap(); + assert!(!stale.active); + assert!(stale.message.contains("is reporting again")); + }) + .await +} + +#[tokio::test(flavor = "multi_thread")] +async fn sweep_skips_unmonitored_server_sources() { + commons_tests::db::TestDb::run(async |mut conn, _| { + let id = + insert_server_full(&mut conn, "http://quiet-unmonitored.invalid/", 600, false).await; + insert_status_at(&mut conn, id, 0).await; + insert_check_state(&mut conn, id, "alertd", "db", 120).await; + + let filed = Status::sweep_staleness(&mut conn).await.expect("sweep"); + assert_eq!(filed, 0); + assert!(stale_issue_for(&mut conn, id, "alertd").await.is_none()); + }) + .await +} + #[tokio::test(flavor = "multi_thread")] async fn sweep_closes_issue_when_server_returns() { commons_tests::db::TestDb::run(async |mut conn, _| { let id = insert_server(&mut conn, "http://recover.invalid/", 600).await; insert_status_at(&mut conn, id, 45).await; - Status::sweep_reachability(&mut conn) + Status::sweep_staleness(&mut conn) .await .expect("first sweep"); assert!(issue_for(&mut conn, id).await.unwrap().active); // New status row at "now" → fresh again, sweep should close the issue. insert_status_at(&mut conn, id, 0).await; - Status::sweep_reachability(&mut conn) + Status::sweep_staleness(&mut conn) .await .expect("second sweep"); assert!(!issue_for(&mut conn, id).await.unwrap().active); diff --git a/crates/database/tests/it/resolve_overall_health_rollups_migration.rs b/crates/database/tests/it/resolve_overall_health_rollups_migration.rs deleted file mode 100644 index f3e46376..00000000 --- a/crates/database/tests/it/resolve_overall_health_rollups_migration.rs +++ /dev/null @@ -1,225 +0,0 @@ -//! The `2026-05-28-100000-0000_resolve_overall_health_rollups` migration -//! retires the `(status, health)` rollup issue category — see -//! `docs/plans/healthcheck-severity-catalog.md`. This test seeds the -//! pre-deprecation state (active rollup issue, incident contributed to -//! only by that issue, pending Slack `incident_open`) and replays the -//! migration's SQL to verify the full cleanup cascade. - -use diesel::sql_types; -use diesel_async::{RunQueryDsl, SimpleAsyncConnection as _}; -use uuid::Uuid; - -const MIGRATION_UP: &str = include_str!( - "../../../../migrations/2026-05-28-100000-0000_resolve_overall_health_rollups/up.sql" -); - -#[tokio::test(flavor = "multi_thread")] -async fn migration_retires_active_rollup_and_orphan_incident() { - commons_tests::db::TestDb::run(async |mut conn, _| { - let group_id = Uuid::new_v4(); - let server_id = Uuid::new_v4(); - let device_id = Uuid::new_v4(); - let rollup_issue_id = Uuid::new_v4(); - let incident_id = Uuid::new_v4(); - let outbox_id = Uuid::new_v4(); - - conn.batch_execute(&format!( - "INSERT INTO devices (id, role) VALUES ('{device_id}', 'server'); \ - INSERT INTO server_groups (id, name) VALUES ('{group_id}', 'g'); \ - INSERT INTO servers (id, host, kind, device_id, group_id) VALUES \ - ('{server_id}', 'https://example.com', 'central', '{device_id}', '{group_id}'); \ - INSERT INTO issues \ - (id, server_id, device_id, source, ref, severity, message, active, first_seen, last_seen) \ - VALUES \ - ('{rollup_issue_id}', '{server_id}', '{device_id}', 'status', 'health', 'error', \ - 'Server reports unhealthy', true, NOW() - interval '1 hour', NOW() - interval '5 min'); \ - INSERT INTO incidents (id, server_group_id, opened_at) \ - VALUES ('{incident_id}', '{group_id}', NOW() - interval '1 hour'); \ - INSERT INTO incident_issues (incident_id, issue_id, joined_at) VALUES \ - ('{incident_id}', '{rollup_issue_id}', NOW() - interval '1 hour'); \ - INSERT INTO slack_outbox (id, kind, incident_id, issue_id, payload, deliver_after) \ - VALUES \ - ('{outbox_id}', 'incident_open', '{incident_id}', '{rollup_issue_id}', \ - '{{}}'::jsonb, NOW() + interval '5 min');" - )) - .await - .expect("seed"); - - // Replay the migration's up.sql. - conn.batch_execute(MIGRATION_UP) - .await - .expect("replay migration up.sql"); - - // 1. The rollup issue is deactivated and human-resolved. - let row: IssueState = diesel::sql_query( - "SELECT active, resolved_at IS NOT NULL AS resolved, resolved_by, resolved_reason \ - FROM issues WHERE id = $1", - ) - .bind::(rollup_issue_id) - .get_result(&mut conn) - .await - .expect("rollup issue row still exists"); - assert!(!row.active, "rollup issue must be deactivated"); - assert!(row.resolved, "resolved_at must be set"); - assert_eq!( - row.resolved_by.as_deref(), - Some("migration:2026-05-28-resolve_overall_health_rollups") - ); - assert_eq!(row.resolved_reason.as_deref(), Some("expected")); - - // 2. A close event with the synthetic hash + retirement message - // has been appended to the rollup issue. - let evt: EventState = diesel::sql_query( - "SELECT severity, active, message FROM events WHERE issue_id = $1 ORDER BY created_at DESC LIMIT 1", - ) - .bind::(rollup_issue_id) - .get_result(&mut conn) - .await - .expect("close event was appended"); - assert_eq!(evt.severity, "info"); - assert!(!evt.active, "close event has active=false"); - assert_eq!( - evt.message, - "Overall-health roll-up retired; per-check issues remain." - ); - - // 3. The incident_issues link is marked left. - let link: LinkState = diesel::sql_query( - "SELECT left_at IS NOT NULL AS departed FROM incident_issues \ - WHERE incident_id = $1 AND issue_id = $2", - ) - .bind::(incident_id) - .bind::(rollup_issue_id) - .get_result(&mut conn) - .await - .expect("link row"); - assert!( - link.departed, - "incident_issues.left_at must be set after migration" - ); - - // 4. The orphan incident (no other live contributors) is closed. - let inc: IncidentState = diesel::sql_query( - "SELECT closed_at IS NOT NULL AS closed FROM incidents WHERE id = $1", - ) - .bind::(incident_id) - .get_result(&mut conn) - .await - .expect("incident row"); - assert!(inc.closed, "orphan incident must be closed"); - - // 5. Pending Slack incident_open is cancelled (gave_up_at set, - // last_error explains why). - let outbox: OutboxState = diesel::sql_query( - "SELECT gave_up_at IS NOT NULL AS given_up, last_error \ - FROM slack_outbox WHERE id = $1", - ) - .bind::(outbox_id) - .get_result(&mut conn) - .await - .expect("outbox row"); - assert!(outbox.given_up, "pending Slack open must be given up"); - assert!( - outbox - .last_error - .as_deref() - .is_some_and(|m| m.contains("overall-health rollup retirement migration")), - "last_error explains the cancellation: {:?}", - outbox.last_error - ); - }) - .await -} - -#[tokio::test(flavor = "multi_thread")] -async fn migration_leaves_incident_with_other_live_contributors_open() { - commons_tests::db::TestDb::run(async |mut conn, _| { - let group_id = Uuid::new_v4(); - let server_id = Uuid::new_v4(); - let device_id = Uuid::new_v4(); - let rollup_issue_id = Uuid::new_v4(); - let per_check_issue_id = Uuid::new_v4(); - let incident_id = Uuid::new_v4(); - - // Rollup + a separate per-check issue both contributing to the - // same incident. The per-check issue keeps the incident alive. - conn.batch_execute(&format!( - "INSERT INTO devices (id, role) VALUES ('{device_id}', 'server'); \ - INSERT INTO server_groups (id, name) VALUES ('{group_id}', 'g'); \ - INSERT INTO servers (id, host, kind, device_id, group_id) VALUES \ - ('{server_id}', 'https://example.com', 'central', '{device_id}', '{group_id}'); \ - INSERT INTO issues \ - (id, server_id, device_id, source, ref, severity, message, active, first_seen, last_seen) VALUES \ - ('{rollup_issue_id}', '{server_id}', '{device_id}', 'status', 'health', \ - 'error', 'roll', true, NOW(), NOW()), \ - ('{per_check_issue_id}', '{server_id}', '{device_id}', 'status', 'health/db', \ - 'error', 'db down', true, NOW(), NOW()); \ - INSERT INTO incidents (id, server_group_id, opened_at) \ - VALUES ('{incident_id}', '{group_id}', NOW()); \ - INSERT INTO incident_issues (incident_id, issue_id, joined_at) VALUES \ - ('{incident_id}', '{rollup_issue_id}', NOW()), \ - ('{incident_id}', '{per_check_issue_id}', NOW());" - )) - .await - .expect("seed"); - - conn.batch_execute(MIGRATION_UP) - .await - .expect("replay migration up.sql"); - - let inc: IncidentState = diesel::sql_query( - "SELECT closed_at IS NOT NULL AS closed FROM incidents WHERE id = $1", - ) - .bind::(incident_id) - .get_result(&mut conn) - .await - .expect("incident row"); - assert!( - !inc.closed, - "incident must stay open while another live contributor exists" - ); - }) - .await -} - -#[derive(diesel::QueryableByName)] -struct IssueState { - #[diesel(sql_type = sql_types::Bool)] - active: bool, - #[diesel(sql_type = sql_types::Bool)] - resolved: bool, - #[diesel(sql_type = sql_types::Nullable)] - resolved_by: Option, - #[diesel(sql_type = sql_types::Nullable)] - resolved_reason: Option, -} - -#[derive(diesel::QueryableByName)] -struct EventState { - #[diesel(sql_type = sql_types::Text)] - severity: String, - #[diesel(sql_type = sql_types::Bool)] - active: bool, - #[diesel(sql_type = sql_types::Text)] - message: String, -} - -#[derive(diesel::QueryableByName)] -struct LinkState { - #[diesel(sql_type = sql_types::Bool)] - departed: bool, -} - -#[derive(diesel::QueryableByName)] -struct IncidentState { - #[diesel(sql_type = sql_types::Bool)] - closed: bool, -} - -#[derive(diesel::QueryableByName)] -struct OutboxState { - #[diesel(sql_type = sql_types::Bool)] - given_up: bool, - #[diesel(sql_type = sql_types::Nullable)] - last_error: Option, -} diff --git a/crates/database/tests/it/scoped_check_policies.rs b/crates/database/tests/it/scoped_check_policies.rs new file mode 100644 index 00000000..5b1b98e0 --- /dev/null +++ b/crates/database/tests/it/scoped_check_policies.rs @@ -0,0 +1,280 @@ +//! Scoped check policies: transforms applied after the fleet catalog — +//! fleet, then group, then server — with the operator silence as a +//! scoped skipped-ceiling. Grade-time behaviour is exercised through +//! `file_check`; the silence CRUD through the `silenced_refs` facade. + +use commons_types::status::CheckResult; +use database::{ + check_policies::{CheckPolicy, EvaluationContext, PolicyScope, ScopedCheckPolicy}, + issues::{CheckFiling, FilingScope, Issue, file_check}, + silenced_refs::ServerSilencedRef, + statuses::CANOPY_SOURCE, +}; +use diesel::{QueryableByName, sql_query, sql_types}; +use diesel_async::RunQueryDsl; +use uuid::Uuid; + +#[derive(QueryableByName)] +struct RowId { + #[diesel(sql_type = sql_types::Uuid)] + id: Uuid, +} + +async fn insert_group(conn: &mut diesel_async::AsyncPgConnection) -> Uuid { + let row: RowId = sql_query("INSERT INTO server_groups (name) VALUES ('g') RETURNING id") + .get_result(conn) + .await + .expect("insert group"); + row.id +} + +async fn insert_server(conn: &mut diesel_async::AsyncPgConnection, group_id: Option) -> Uuid { + let row: RowId = sql_query( + "INSERT INTO servers (host, group_id) VALUES ('http://scoped.invalid/', $1) RETURNING id", + ) + .bind::, _>(group_id) + .get_result(conn) + .await + .expect("insert server"); + row.id +} + +fn filing(server_id: Uuid, check: &str, observed: CheckResult) -> CheckFiling<'_> { + CheckFiling { + source: CANOPY_SOURCE, + scope: FilingScope::Server { + server_id, + device_id: None, + }, + check, + observed, + title: Some("Scoped test"), + message: "scoped test filing", + detail: None, + default_ceiling: CheckResult::Failed, + default_escalates: false, + documentation: None, + } +} + +async fn state_for( + conn: &mut diesel_async::AsyncPgConnection, + server_id: Uuid, + check: &str, +) -> Issue { + Issue::list_by_source_ref(conn, CANOPY_SOURCE, check, &[server_id]) + .await + .expect("list") + .into_iter() + .next() + .expect("state row filed") +} + +#[tokio::test(flavor = "multi_thread")] +async fn server_silence_grades_filings_to_skipped() { + commons_tests::db::TestDb::run(async |mut conn, _| { + let server_id = insert_server(&mut conn, None).await; + ServerSilencedRef::add(&mut conn, server_id, CANOPY_SOURCE, "noisy", None) + .await + .expect("silence"); + + file_check(&mut conn, filing(server_id, "noisy", CheckResult::Failed)) + .await + .expect("file"); + + let state = state_for(&mut conn, server_id, "noisy").await; + assert_eq!(state.observed_result, Some(CheckResult::Failed)); + assert_eq!( + state.effective_result, + Some(CheckResult::Skipped), + "a silenced check records its observation but grades to skipped" + ); + assert!(!state.active); + }) + .await +} + +#[tokio::test(flavor = "multi_thread")] +async fn group_silence_covers_member_servers() { + commons_tests::db::TestDb::run(async |mut conn, _| { + let group_id = insert_group(&mut conn).await; + let server_id = insert_server(&mut conn, Some(group_id)).await; + ScopedCheckPolicy::silence( + &mut conn, + PolicyScope::Group(group_id), + CANOPY_SOURCE, + "noisy", + Some("op"), + ) + .await + .expect("group silence"); + + file_check(&mut conn, filing(server_id, "noisy", CheckResult::Failed)) + .await + .expect("file"); + let state = state_for(&mut conn, server_id, "noisy").await; + assert_eq!(state.effective_result, Some(CheckResult::Skipped)); + + // Lifting the group silence restores normal grading on the next + // filing. + ScopedCheckPolicy::unsilence( + &mut conn, + PolicyScope::Group(group_id), + CANOPY_SOURCE, + "noisy", + ) + .await + .expect("unsilence"); + file_check(&mut conn, filing(server_id, "noisy", CheckResult::Failed)) + .await + .expect("file again"); + let state = state_for(&mut conn, server_id, "noisy").await; + assert_eq!(state.effective_result, Some(CheckResult::Failed)); + assert!(state.active); + }) + .await +} + +#[tokio::test(flavor = "multi_thread")] +async fn server_scoped_rule_can_upgrade_past_the_fleet_ceiling() { + commons_tests::db::TestDb::run(async |mut conn, _| { + let server_id = insert_server(&mut conn, None).await; + + // Fleet catalog grades this check down to warning... + CheckPolicy::register( + &mut conn, + CANOPY_SOURCE, + "tiered", + CheckResult::Warning, + false, + None, + ) + .await + .expect("register"); + + // ...but a server-scoped rule upgrades observed failures back to + // failed for this one server (the backend admits arbitrary scoped + // transforms; only silences are surfaced in the UI). + sql_query( + r#" + INSERT INTO scoped_check_policies (source, check_name, server_id, rules) + VALUES ($1, 'tiered', $2, + '{"if": [{"==": [{"var": "check.result"}, "failed"]}, "failed"]}'::jsonb) + "#, + ) + .bind::(CANOPY_SOURCE) + .bind::(server_id) + .execute(&mut conn) + .await + .expect("scoped rule"); + + let ctx = EvaluationContext { + status_extra: &serde_json::Map::new(), + check_extra: &serde_json::Map::from_iter([( + "result".to_string(), + serde_json::Value::String("failed".into()), + )]), + tags: &Default::default(), + }; + let graded = CheckPolicy::apply_scoped( + &mut conn, + CANOPY_SOURCE, + "tiered", + CheckResult::Failed, + &ctx, + Some(server_id), + None, + ) + .await + .expect("apply"); + assert_eq!( + graded.effective, + CheckResult::Failed, + "the server-scoped rule has the last word over the fleet ceiling" + ); + + // A different server without the scoped rule keeps the fleet grade. + let other = insert_server(&mut conn, None).await; + let graded = CheckPolicy::apply_scoped( + &mut conn, + CANOPY_SOURCE, + "tiered", + CheckResult::Failed, + &ctx, + Some(other), + None, + ) + .await + .expect("apply other"); + assert_eq!(graded.effective, CheckResult::Warning); + }) + .await +} + +#[tokio::test(flavor = "multi_thread")] +async fn silence_on_a_scoped_rule_row_keeps_the_rules() { + commons_tests::db::TestDb::run(async |mut conn, _| { + let server_id = insert_server(&mut conn, None).await; + sql_query( + r#" + INSERT INTO scoped_check_policies (source, check_name, server_id, rules) + VALUES ('alertd', 'ruled', $1, + '{"if": [{"==": [{"var": "check.result"}, "failed"]}, "failed"]}'::jsonb) + "#, + ) + .bind::(server_id) + .execute(&mut conn) + .await + .expect("scoped rule"); + + ScopedCheckPolicy::silence( + &mut conn, + PolicyScope::Server(server_id), + "alertd", + "ruled", + Some("op"), + ) + .await + .expect("silence"); + let row = + ScopedCheckPolicy::get(&mut conn, PolicyScope::Server(server_id), "alertd", "ruled") + .await + .expect("get") + .expect("row exists"); + assert_eq!(row.ceiling.as_deref(), Some("skipped")); + assert!(row.rules.is_some(), "silencing keeps the scoped rules"); + + // Unsilencing lifts the ceiling but keeps the rules row. + ScopedCheckPolicy::unsilence(&mut conn, PolicyScope::Server(server_id), "alertd", "ruled") + .await + .expect("unsilence"); + let row = + ScopedCheckPolicy::get(&mut conn, PolicyScope::Server(server_id), "alertd", "ruled") + .await + .expect("get") + .expect("row still exists"); + assert_eq!(row.ceiling, None); + assert!(row.rules.is_some()); + + // A plain silence row deletes outright on unsilence. + ScopedCheckPolicy::silence( + &mut conn, + PolicyScope::Server(server_id), + "alertd", + "plain", + None, + ) + .await + .expect("plain silence"); + ScopedCheckPolicy::unsilence(&mut conn, PolicyScope::Server(server_id), "alertd", "plain") + .await + .expect("plain unsilence"); + assert!( + ScopedCheckPolicy::get(&mut conn, PolicyScope::Server(server_id), "alertd", "plain") + .await + .expect("get") + .is_none() + ); + }) + .await +} diff --git a/crates/database/tests/it/self_alerts.rs b/crates/database/tests/it/self_alerts.rs index 9354993d..094bdb55 100644 --- a/crates/database/tests/it/self_alerts.rs +++ b/crates/database/tests/it/self_alerts.rs @@ -1,19 +1,41 @@ -//! Self-alert lifecycle: a raise files one coalescing nil-server issue and -//! enqueues exactly one Slack open on the not-alerting → alerting transition -//! (with flap grace below Critical, immediate at Critical); recovery inside +//! Self-alert lifecycle: a raise files one coalescing canopy-wide issue +//! which opens a canopy-wide incident, enqueuing exactly one Slack open on +//! the not-alerting → alerting transition (with flap grace for +//! non-escalating checks, immediate for escalating ones); recovery inside //! the grace cancels the open and sends nothing; recovery after delivery //! enqueues the resolve; idle recovers write nothing. -use commons_types::issue::Severity; +use commons_types::status::CheckResult; use database::self_alerts; -use database::slack_outbox::{KIND_SELF_ALERT_OPEN, KIND_SELF_ALERT_RESOLVE, SlackOutbox}; +use database::slack_outbox::{KIND_INCIDENT_OPEN, KIND_INCIDENT_RESOLVE, SlackOutbox}; use diesel::prelude::*; use diesel_async::RunQueryDsl; use jiff::Timestamp; -use uuid::Uuid; const REF: &str = "test-self-alert"; +/// Raise REF as a failure; `escalates` picks the registered policy tier +/// (escalating ≙ the old Critical, immediate notify; otherwise graced). +/// Each test runs on a fresh database, so the first raise registers the +/// catalog entry at the given tier. +async fn raise_with( + conn: &mut diesel_async::AsyncPgConnection, + escalates: bool, +) -> database::issues::Issue { + self_alerts::raise( + conn, + REF, + CheckResult::Failed, + CheckResult::Failed, + escalates, + None, + "title", + "body", + ) + .await + .expect("raise") +} + async fn outbox_rows(conn: &mut diesel_async::AsyncPgConnection) -> Vec { use database::schema::slack_outbox::dsl; dsl::slack_outbox @@ -36,30 +58,31 @@ async fn raise_enqueues_once_and_flap_recovery_is_silent() { ); assert!(outbox_rows(&mut conn).await.is_empty()); - // First raise: issue + one open row, delayed by the grace (Error). - let issue = self_alerts::raise(&mut conn, REF, Severity::Error, "title", "body") - .await - .expect("raise"); - assert_eq!(issue.server_id, Some(Uuid::nil())); + // First raise: issue + canopy-wide incident + one open row, delayed + // by the grace (non-escalating). + let issue = raise_with(&mut conn, false).await; + assert_eq!(issue.server_id, None); + assert_eq!(issue.server_group_id, None); assert!(issue.active); let rows = outbox_rows(&mut conn).await; let [open] = rows.as_slice() else { panic!("exactly one outbox row, got {rows:?}"); }; - assert_eq!(open.kind, KIND_SELF_ALERT_OPEN); - assert_eq!(open.incident_id, None); + assert_eq!(open.kind, KIND_INCIDENT_OPEN); + assert!( + open.incident_id.is_some(), + "the raise opened a canopy-wide incident" + ); assert_eq!(open.issue_id, Some(issue.id)); assert!( open.deliver_after > Timestamp::now(), - "sub-Critical opens wait out the grace" + "non-escalating opens wait out the grace" ); - assert_eq!(open.payload["state"], "alert"); + assert_eq!(open.payload["server"], "Canopy"); assert_eq!(open.payload["source_ref"], format!("canopy/{REF}")); // Re-raise while alerting: no new outbox row. - self_alerts::raise(&mut conn, REF, Severity::Error, "title", "body") - .await - .expect("re-raise"); + raise_with(&mut conn, false).await; assert_eq!(outbox_rows(&mut conn).await.len(), 1); // Recover inside the grace: open cancelled, no resolve enqueued. @@ -72,25 +95,21 @@ async fn raise_enqueues_once_and_flap_recovery_is_silent() { assert!(rows[0].gave_up_at.is_some(), "pending open cancelled"); // Raise again: a fresh transition, a fresh open. - self_alerts::raise(&mut conn, REF, Severity::Error, "title", "body") - .await - .expect("raise again"); + raise_with(&mut conn, false).await; assert_eq!(outbox_rows(&mut conn).await.len(), 2); }) .await } #[tokio::test(flavor = "multi_thread")] -async fn critical_ships_immediately_and_recovery_after_delivery_resolves() { +async fn escalating_ships_immediately_and_recovery_after_delivery_resolves() { commons_tests::db::TestDb::run(async |mut conn, _url| { - self_alerts::raise(&mut conn, REF, Severity::Critical, "title", "body") - .await - .expect("raise"); + raise_with(&mut conn, true).await; let rows = outbox_rows(&mut conn).await; assert_eq!(rows.len(), 1); assert!( rows[0].deliver_after <= Timestamp::now(), - "critical skips the grace" + "an escalating check skips the grace" ); // Simulate the drainer shipping the open, then recover: a resolve @@ -105,8 +124,8 @@ async fn critical_ships_immediately_and_recovery_after_delivery_resolves() { let rows = outbox_rows(&mut conn).await; assert_eq!(rows.len(), 2); let resolve = &rows[1]; - assert_eq!(resolve.kind, KIND_SELF_ALERT_RESOLVE); - assert_eq!(resolve.payload["state"], "recovered"); + assert_eq!(resolve.kind, KIND_INCIDENT_RESOLVE); + assert_eq!(resolve.payload["server"], "Canopy"); assert!(resolve.deliver_after <= Timestamp::now()); // Recovering again is a no-op. @@ -126,9 +145,7 @@ async fn operator_resolved_but_persisting_condition_re_notifies() { commons_tests::db::TestDb::run(async |mut conn, _url| { use database::issues::Issue; - let issue = self_alerts::raise(&mut conn, REF, Severity::Critical, "title", "body") - .await - .expect("raise"); + let issue = raise_with(&mut conn, true).await; assert_eq!(outbox_rows(&mut conn).await.len(), 1); // An operator resolves it, but the condition still holds: the next @@ -142,9 +159,7 @@ async fn operator_resolved_but_persisting_condition_re_notifies() { ) .await .expect("operator resolve"); - self_alerts::raise(&mut conn, REF, Severity::Critical, "title", "body") - .await - .expect("re-raise"); + raise_with(&mut conn, true).await; assert_eq!(outbox_rows(&mut conn).await.len(), 2); }) .await @@ -155,16 +170,14 @@ async fn self_alert_issues_are_excluded_from_the_fleet_listing() { commons_tests::db::TestDb::run(async |mut conn, _url| { use database::issues::Issue; - self_alerts::raise(&mut conn, REF, Severity::Error, "title", "body") - .await - .expect("raise"); + raise_with(&mut conn, false).await; let fleet = Issue::list(&mut conn, Default::default(), 100) .await .expect("list"); assert!( fleet.is_empty(), - "nil-server issues must not appear in the fleet listing: {fleet:?}" + "canopy-wide issues must not appear in the fleet listing: {fleet:?}" ); let alerts = self_alerts::list(&mut conn, 50).await.expect("alerts"); assert_eq!(alerts.len(), 1); diff --git a/crates/database/tests/it/server_enrollment.rs b/crates/database/tests/it/server_enrollment.rs index dcd613ac..76fded5a 100644 --- a/crates/database/tests/it/server_enrollment.rs +++ b/crates/database/tests/it/server_enrollment.rs @@ -21,7 +21,6 @@ fn new_server(host: &str) -> Server { cloud: None, geolocation: None, is_monitored: true, - allow_legacy_status: false, alert_when_down_for: PgDuration(SignedDuration::from_secs(600)), notes: String::new(), tags: TagMap::default(), diff --git a/crates/database/tests/it/server_restore_window.rs b/crates/database/tests/it/server_restore_window.rs index c6f20e1c..83b57860 100644 --- a/crates/database/tests/it/server_restore_window.rs +++ b/crates/database/tests/it/server_restore_window.rs @@ -18,7 +18,6 @@ fn new_server() -> Server { cloud: None, geolocation: None, is_monitored: true, - allow_legacy_status: false, alert_when_down_for: PgDuration(SignedDuration::from_secs(600)), notes: String::new(), tags: TagMap::default(), diff --git a/crates/database/tests/it/silenced_health_checks.rs b/crates/database/tests/it/silenced_health_checks.rs index 7e17bb20..74bcfce4 100644 --- a/crates/database/tests/it/silenced_health_checks.rs +++ b/crates/database/tests/it/silenced_health_checks.rs @@ -1,15 +1,15 @@ -//! `silenced_refs::silenced_health_checks_for_server(s)`: resolving the -//! set of healthcheck names silenced for a server from its `(status, -//! health/)` silence entries, at server and group scope, in one -//! batch. This set feeds `Status::health_state_ignoring` so silenced -//! checks don't count toward the health rollup. +//! `silenced_refs::silenced_health_checks_for_server`: resolving the set +//! of healthcheck names silenced for a server under one reporting +//! source, at server and group scope. This set feeds +//! `Status::health_state_ignoring` so silenced checks don't count toward +//! the health rollup — scoped to the status row's own source, since a +//! check's identity is the (source, check) pair. use std::collections::BTreeSet; use commons_tests::db::TestDb; use database::silenced_refs::{ ServerGroupSilencedRef, ServerSilencedRef, silenced_health_checks_for_server, - silenced_health_checks_for_servers, }; use diesel::{sql_query, sql_types}; use diesel_async::RunQueryDsl; @@ -55,75 +55,80 @@ fn checks(names: &[&str]) -> BTreeSet { } #[tokio::test(flavor = "multi_thread")] -async fn resolves_server_and_group_scopes_in_batch() { +async fn combines_server_and_group_scopes() { TestDb::run(async |mut conn, _url| { let group = insert_group(&mut conn, "g").await; let grouped = insert_server(&mut conn, Some(group)).await; let ungrouped = insert_server(&mut conn, None).await; let unsilenced = insert_server(&mut conn, Some(group)).await; - ServerSilencedRef::add(&mut conn, grouped, "status", "health/postgres", None) + ServerSilencedRef::add(&mut conn, grouped, "alertd", "health/postgres", None) .await .unwrap(); - ServerSilencedRef::add(&mut conn, ungrouped, "status", "health/disk", None) + ServerSilencedRef::add(&mut conn, ungrouped, "alertd", "health/disk", None) .await .unwrap(); - ServerGroupSilencedRef::add(&mut conn, group, "status", "health/uploads", None) + ServerGroupSilencedRef::add(&mut conn, group, "alertd", "health/uploads", None) .await .unwrap(); - let map = silenced_health_checks_for_servers( - &mut conn, - &[ - (grouped, Some(group)), - (ungrouped, None), - (unsilenced, Some(group)), - ], - ) - .await - .unwrap(); - // Server scope and group scope combine for the grouped server. - assert_eq!(map.get(&grouped), Some(&checks(&["postgres", "uploads"]))); + assert_eq!( + silenced_health_checks_for_server(&mut conn, grouped, Some(group), "alertd") + .await + .unwrap(), + checks(&["postgres", "uploads"]), + ); // The ungrouped server only sees its own silences. - assert_eq!(map.get(&ungrouped), Some(&checks(&["disk"]))); + assert_eq!( + silenced_health_checks_for_server(&mut conn, ungrouped, None, "alertd") + .await + .unwrap(), + checks(&["disk"]), + ); // A group member with no server-scope silence still inherits the // group's. - assert_eq!(map.get(&unsilenced), Some(&checks(&["uploads"]))); - - // The single-server convenience agrees. assert_eq!( - silenced_health_checks_for_server(&mut conn, grouped, Some(group)) + silenced_health_checks_for_server(&mut conn, unsilenced, Some(group), "alertd") .await .unwrap(), - checks(&["postgres", "uploads"]), + checks(&["uploads"]), ); }) .await } -/// Only `(status, health/)` silences are healthcheck silences: -/// other sources and non-health refs (e.g. canopy reachability) don't -/// leak into the set. +/// A check's identity is the (source, check) pair: another source's +/// silence on a same-named check never applies, and neither do canopy's +/// own or manual silences. #[tokio::test(flavor = "multi_thread")] -async fn ignores_non_healthcheck_silences() { +async fn scoped_to_the_reporting_source() { TestDb::run(async |mut conn, _url| { let server = insert_server(&mut conn, None).await; ServerSilencedRef::add(&mut conn, server, "canopy", "reachability", None) .await .unwrap(); - ServerSilencedRef::add(&mut conn, server, "backups", "health/postgres", None) + ServerSilencedRef::add(&mut conn, server, "seedling", "health/postgres", None) .await .unwrap(); - ServerSilencedRef::add(&mut conn, server, "status", "something-else", None) + ServerSilencedRef::add(&mut conn, server, "alertd", "health/disk", None) .await .unwrap(); - let map = silenced_health_checks_for_servers(&mut conn, &[(server, None)]) - .await - .unwrap(); - assert_eq!(map.get(&server), None); + assert_eq!( + silenced_health_checks_for_server(&mut conn, server, None, "alertd") + .await + .unwrap(), + checks(&["disk"]), + "only alertd's own silence applies to alertd's checks", + ); + assert_eq!( + silenced_health_checks_for_server(&mut conn, server, None, "seedling") + .await + .unwrap(), + checks(&["postgres"]), + ); }) .await } @@ -134,21 +139,21 @@ async fn unsilencing_removes_the_check() { TestDb::run(async |mut conn, _url| { let server = insert_server(&mut conn, None).await; - ServerSilencedRef::add(&mut conn, server, "status", "health/postgres", None) + ServerSilencedRef::add(&mut conn, server, "alertd", "health/postgres", None) .await .unwrap(); assert_eq!( - silenced_health_checks_for_server(&mut conn, server, None) + silenced_health_checks_for_server(&mut conn, server, None, "alertd") .await .unwrap(), checks(&["postgres"]), ); - ServerSilencedRef::remove(&mut conn, server, "status", "health/postgres") + ServerSilencedRef::remove(&mut conn, server, "alertd", "health/postgres") .await .unwrap(); assert_eq!( - silenced_health_checks_for_server(&mut conn, server, None) + silenced_health_checks_for_server(&mut conn, server, None, "alertd") .await .unwrap(), BTreeSet::new(), diff --git a/crates/database/tests/it/slack_outbox_enqueue.rs b/crates/database/tests/it/slack_outbox_enqueue.rs index ce2ae576..a8300f9c 100644 --- a/crates/database/tests/it/slack_outbox_enqueue.rs +++ b/crates/database/tests/it/slack_outbox_enqueue.rs @@ -5,7 +5,7 @@ //! incident transitions, an outbox row exists with the expected `kind` and //! a non-empty `payload`. -use commons_types::issue::{ResolvedReason, Severity}; +use commons_types::{issue::ResolvedReason, status::CheckResult}; use database::{ issues::{Incident, Issue, NewEvent}, slack_outbox::{KIND_INCIDENT_OPEN, KIND_INCIDENT_RESOLVE, SlackOutbox}, @@ -118,16 +118,25 @@ async fn mark_open_delivered(conn: &mut diesel_async::AsyncPgConnection, inciden async fn opening_incident_enqueues_slack_open_row() { commons_tests::db::TestDb::run(async |mut conn, _| { let server_id = insert_server(&mut conn, "http://open.invalid/").await; + let stamp = database::issues::CheckStateStamp { + check: "ref-1".into(), + observed: CheckResult::Failed, + effective: CheckResult::Failed, + escalates: false, + detail: None, + }; let event = NewEvent { source: "test".into(), r#ref: "ref-1".into(), - severity: Some(Severity::Error), description: None, message: "boom".into(), active: Some(true), occurred_at: None, }; - let issue = event.save(&mut conn, server_id, None).await.expect("save"); + let issue = event + .save_with_state(&mut conn, server_id, None, Some(&stamp)) + .await + .expect("save"); // The save call should have opened an incident *and* enqueued an // `incident_open` outbox row. @@ -186,16 +195,25 @@ async fn resolving_incident_after_open_delivered_enqueues_resolve_row() { // yet; everything past that point is the historical behaviour. commons_tests::db::TestDb::run(async |mut conn, _| { let server_id = insert_server(&mut conn, "http://resolve.invalid/").await; + let stamp = database::issues::CheckStateStamp { + check: "ref-2".into(), + observed: CheckResult::Failed, + effective: CheckResult::Failed, + escalates: false, + detail: None, + }; let event = NewEvent { source: "test".into(), r#ref: "ref-2".into(), - severity: Some(Severity::Error), description: None, message: "boom".into(), active: Some(true), occurred_at: None, }; - event.save(&mut conn, server_id, None).await.expect("save"); + event + .save_with_state(&mut conn, server_id, None, Some(&stamp)) + .await + .expect("save"); let incident = Incident::list_for_server(&mut conn, server_id, false, 10) .await .expect("list incidents") @@ -232,16 +250,25 @@ async fn resolving_before_open_ships_cancels_open_and_skips_resolve() { // (given-up, with a reason in `last_error`) for the audit trail. commons_tests::db::TestDb::run(async |mut conn, _| { let server_id = insert_server(&mut conn, "http://flap.invalid/").await; + let stamp = database::issues::CheckStateStamp { + check: "ref-flap".into(), + observed: CheckResult::Failed, + effective: CheckResult::Failed, + escalates: false, + detail: None, + }; let event = NewEvent { source: "test".into(), r#ref: "ref-flap".into(), - severity: Some(Severity::Error), description: None, message: "boom".into(), active: Some(true), occurred_at: None, }; - event.save(&mut conn, server_id, None).await.expect("save"); + event + .save_with_state(&mut conn, server_id, None, Some(&stamp)) + .await + .expect("save"); let incident = Incident::list_for_server(&mut conn, server_id, false, 10) .await .expect("list incidents") @@ -303,16 +330,25 @@ async fn cascade_close_via_issue_resolve_attributes_to_operator() { // `None` through the cascade path and lost the attribution. commons_tests::db::TestDb::run(async |mut conn, _| { let server_id = insert_server(&mut conn, "http://attribute.invalid/").await; + let stamp = database::issues::CheckStateStamp { + check: "ref-only".into(), + observed: CheckResult::Failed, + effective: CheckResult::Failed, + escalates: false, + detail: None, + }; let event = NewEvent { source: "test".into(), r#ref: "ref-only".into(), - severity: Some(Severity::Error), description: None, message: "boom".into(), active: Some(true), occurred_at: None, }; - let issue = event.save(&mut conn, server_id, None).await.expect("save"); + let issue = event + .save_with_state(&mut conn, server_id, None, Some(&stamp)) + .await + .expect("save"); let incident = Incident::list_for_server(&mut conn, server_id, false, 10) .await .expect("list incidents") @@ -355,17 +391,23 @@ async fn nil_server_events_do_not_open_incidents() { // group-level incident — which means the Slack drainer can never // loop back into itself by re-firing on a canopy-self failure. commons_tests::db::TestDb::run(async |mut conn, _| { + let stamp = database::issues::CheckStateStamp { + check: "slack-delivery-failure".into(), + observed: CheckResult::Failed, + effective: CheckResult::Failed, + escalates: false, + detail: None, + }; let event = NewEvent { source: "canopy".into(), r#ref: "slack-delivery-failure".into(), - severity: Some(Severity::Error), description: None, message: "boom".into(), active: Some(true), occurred_at: None, }; event - .save(&mut conn, Uuid::nil(), None) + .save_with_state(&mut conn, Uuid::nil(), None, Some(&stamp)) .await .expect("save"); @@ -384,16 +426,25 @@ async fn nil_server_events_do_not_open_incidents() { async fn mark_given_up_removes_row_from_claim_pending() { commons_tests::db::TestDb::run(async |mut conn, _| { let server_id = insert_server(&mut conn, "http://giveup.invalid/").await; + let stamp = database::issues::CheckStateStamp { + check: "ref-g".into(), + observed: CheckResult::Failed, + effective: CheckResult::Failed, + escalates: false, + detail: None, + }; let event = NewEvent { source: "test".into(), r#ref: "ref-g".into(), - severity: Some(Severity::Error), description: None, message: "boom".into(), active: Some(true), occurred_at: None, }; - event.save(&mut conn, server_id, None).await.expect("save"); + event + .save_with_state(&mut conn, server_id, None, Some(&stamp)) + .await + .expect("save"); // `event.save` enqueues an open whose deliver_after sits // `slack_open_delay` in the future. Drop it back to the past so // claim_pending will return the row immediately. @@ -428,16 +479,25 @@ async fn claim_pending_skips_rows_whose_deliver_after_is_in_the_future() { // claimable. This is the core flap-suppression mechanism. commons_tests::db::TestDb::run(async |mut conn, _| { let server_id = insert_server(&mut conn, "http://delayed.invalid/").await; + let stamp = database::issues::CheckStateStamp { + check: "ref-d".into(), + observed: CheckResult::Failed, + effective: CheckResult::Failed, + escalates: false, + detail: None, + }; let event = NewEvent { source: "test".into(), r#ref: "ref-d".into(), - severity: Some(Severity::Error), description: None, message: "boom".into(), active: Some(true), occurred_at: None, }; - event.save(&mut conn, server_id, None).await.expect("save"); + event + .save_with_state(&mut conn, server_id, None, Some(&stamp)) + .await + .expect("save"); let pending_before = SlackOutbox::claim_pending(&mut conn, 10) .await .expect("claim"); @@ -470,16 +530,25 @@ async fn open_delay_honours_per_group_slack_open_delay() { commons_tests::db::TestDb::run(async |mut conn, _| { let server_id = insert_server_with_delay(&mut conn, "http://nowait.invalid/", Some(0)).await; + let stamp = database::issues::CheckStateStamp { + check: "ref-zero".into(), + observed: CheckResult::Failed, + effective: CheckResult::Failed, + escalates: false, + detail: None, + }; let event = NewEvent { source: "test".into(), r#ref: "ref-zero".into(), - severity: Some(Severity::Error), description: None, message: "boom".into(), active: Some(true), occurred_at: None, }; - event.save(&mut conn, server_id, None).await.expect("save"); + event + .save_with_state(&mut conn, server_id, None, Some(&stamp)) + .await + .expect("save"); let claimable = SlackOutbox::claim_pending(&mut conn, 10) .await @@ -509,17 +578,23 @@ async fn pending_opens_until_filters_to_undelivered_in_window() { commons_tests::db::TestDb::run(async |mut conn, _| { // Held incident: default-delay group → open sits 3 min in the future. let held_server = insert_server(&mut conn, "http://held.invalid/").await; + let stamp = database::issues::CheckStateStamp { + check: "ref-held".into(), + observed: CheckResult::Failed, + effective: CheckResult::Failed, + escalates: false, + detail: None, + }; let event = NewEvent { source: "test".into(), r#ref: "ref-held".into(), - severity: Some(Severity::Error), description: None, message: "boom".into(), active: Some(true), occurred_at: None, }; event - .save(&mut conn, held_server, None) + .save_with_state(&mut conn, held_server, None, Some(&stamp)) .await .expect("save held"); let held_incident = Incident::list_for_server(&mut conn, held_server, false, 10) @@ -529,17 +604,23 @@ async fn pending_opens_until_filters_to_undelivered_in_window() { // Delivered incident: open already shipped → not held any more. let delivered_server = insert_server(&mut conn, "http://delivered.invalid/").await; + let stamp = database::issues::CheckStateStamp { + check: "ref-delivered".into(), + observed: CheckResult::Failed, + effective: CheckResult::Failed, + escalates: false, + detail: None, + }; let event = NewEvent { source: "test".into(), r#ref: "ref-delivered".into(), - severity: Some(Severity::Error), description: None, message: "boom".into(), active: Some(true), occurred_at: None, }; event - .save(&mut conn, delivered_server, None) + .save_with_state(&mut conn, delivered_server, None, Some(&stamp)) .await .expect("save delivered"); let delivered_incident = Incident::list_for_server(&mut conn, delivered_server, false, 10) @@ -578,17 +659,23 @@ async fn expire_deliver_after(conn: &mut diesel_async::AsyncPgConnection) { async fn rejoining_open_incident_does_not_re_enqueue_open() { commons_tests::db::TestDb::run(async |mut conn, _| { let server_id = insert_server(&mut conn, "http://rejoin.invalid/").await; + let stamp_a = database::issues::CheckStateStamp { + check: "ref-a".into(), + observed: CheckResult::Failed, + effective: CheckResult::Failed, + escalates: false, + detail: None, + }; let event_a = NewEvent { source: "test".into(), r#ref: "ref-a".into(), - severity: Some(Severity::Error), description: None, message: "first".into(), active: Some(true), occurred_at: None, }; event_a - .save(&mut conn, server_id, None) + .save_with_state(&mut conn, server_id, None, Some(&stamp_a)) .await .expect("save a"); let incident = Incident::list_for_server(&mut conn, server_id, false, 10) @@ -599,17 +686,23 @@ async fn rejoining_open_incident_does_not_re_enqueue_open() { .expect("incident"); // Second active issue, same server: joins the existing incident. + let stamp_b = database::issues::CheckStateStamp { + check: "ref-b".into(), + observed: CheckResult::Failed, + effective: CheckResult::Failed, + escalates: false, + detail: None, + }; let event_b = NewEvent { source: "test".into(), r#ref: "ref-b".into(), - severity: Some(Severity::Error), description: None, message: "second".into(), active: Some(true), occurred_at: None, }; event_b - .save(&mut conn, server_id, None) + .save_with_state(&mut conn, server_id, None, Some(&stamp_b)) .await .expect("save b"); diff --git a/crates/database/tests/it/status_health_state.rs b/crates/database/tests/it/status_health_state.rs index eb50c62b..03f0f437 100644 --- a/crates/database/tests/it/status_health_state.rs +++ b/crates/database/tests/it/status_health_state.rs @@ -16,6 +16,7 @@ fn status(healthy: bool, health: serde_json::Value) -> Status { extra: serde_json::json!({}), healthy, health, + source: "alertd".into(), } } diff --git a/crates/database/tests/it/tag_reserved_prefix.rs b/crates/database/tests/it/tag_reserved_prefix.rs index f7bdef01..114115bc 100644 --- a/crates/database/tests/it/tag_reserved_prefix.rs +++ b/crates/database/tests/it/tag_reserved_prefix.rs @@ -34,7 +34,6 @@ fn new_server(host: &str) -> Server { cloud: None, geolocation: None, is_monitored: true, - allow_legacy_status: false, alert_when_down_for: PgDuration(SignedDuration::from_secs(600)), notes: String::new(), tags: TagMap::default(), @@ -80,7 +79,6 @@ async fn server_update_rejects_reserved_tag_keys() { cloud: None, geolocation: None, is_monitored: None, - allow_legacy_status: None, alert_when_down_for: None, notes: None, tags: Some(reserved_tags()), diff --git a/crates/jobs/src/backup/complete.rs b/crates/jobs/src/backup/complete.rs index 7315176e..94ec4313 100644 --- a/crates/jobs/src/backup/complete.rs +++ b/crates/jobs/src/backup/complete.rs @@ -8,7 +8,7 @@ use std::str::FromStr; -use commons_types::{backup::BackupType, issue::Severity}; +use commons_types::{backup::BackupType, status::CheckResult}; use database::{BackupConfigStatus, BackupMaintenanceRun, RunOutcome, ServerGroupBackupConfig}; use diesel_async::AsyncPgConnection; use jiff::Timestamp; @@ -56,19 +56,15 @@ pub(crate) async fn complete_init( Ok(()) } -/// The CORRUPTION alert (severity, single-line description, active) implied by -/// an inspect result's `verify_ok`. `verify_ok:false` → a Critical, active -/// alert; `verify_ok:true` → an Info recovery (active false). Pure so the -/// outcome→decision mapping is unit-testable. -pub(crate) fn corruption_decision(verify_ok: bool) -> (Severity, Option<&'static str>, bool) { +/// The CORRUPTION check observation (result, single-line title) implied +/// by an inspect result's `verify_ok`. `verify_ok:false` → an observed +/// failure; `verify_ok:true` → a recovery. Pure so the outcome→decision +/// mapping is unit-testable. +pub(crate) fn corruption_decision(verify_ok: bool) -> (CheckResult, Option<&'static str>) { if verify_ok { - (Severity::Info, None, false) + (CheckResult::Passed, None) } else { - ( - Severity::Critical, - Some("backup repository verify failed"), - true, - ) + (CheckResult::Failed, Some("backup repository verify failed")) } } @@ -116,15 +112,21 @@ pub(crate) async fn complete_inspect( .await .map_err(|err| err.to_string())?; - let (severity, description, active) = corruption_decision(outcome.verify_ok); - database::backup::alerts::raise_group_event( + let (observed, title) = corruption_decision(outcome.verify_ok); + database::issues::file_check( db, - group_id, - database::backup::alerts::refs::CORRUPTION, - severity, - description, - "kopia snapshot verify", - active, + database::issues::CheckFiling { + source: database::statuses::CANOPY_SOURCE, + scope: database::issues::FilingScope::Group(group_id), + check: database::backup::refs::CORRUPTION, + observed, + title, + message: "kopia snapshot verify", + detail: None, + default_ceiling: CheckResult::Failed, + default_escalates: true, + documentation: Some(database::backup::refs::CORRUPTION_DOC), + }, ) .await .map_err(|err| err.to_string())?; @@ -139,13 +141,9 @@ mod tests { fn corruption_decision_maps_verify_ok() { assert_eq!( corruption_decision(false), - ( - Severity::Critical, - Some("backup repository verify failed"), - true - ) + (CheckResult::Failed, Some("backup repository verify failed"),) ); - assert_eq!(corruption_decision(true), (Severity::Info, None, false)); + assert_eq!(corruption_decision(true), (CheckResult::Passed, None)); } mod db { @@ -314,22 +312,25 @@ mod tests { #[derive(diesel::QueryableByName)] struct CorruptionRow { - #[diesel(sql_type = sql_types::Text)] - severity: String, + #[diesel(sql_type = sql_types::Nullable)] + effective_result: Option, + #[diesel(sql_type = sql_types::Bool)] + escalates: bool, #[diesel(sql_type = sql_types::Bool)] active: bool, } let rows: Vec = sql_query( - "SELECT severity, active FROM issues \ + "SELECT effective_result, escalates, active FROM issues \ WHERE server_group_id = $1 AND \"ref\" = $2", ) .bind::(group_id) - .bind::(database::backup::alerts::refs::CORRUPTION) + .bind::(database::backup::refs::CORRUPTION) .get_results(&mut conn) .await .expect("query corruption issues"); assert_eq!(rows.len(), 1, "exactly one corruption issue"); - assert_eq!(rows[0].severity, "critical"); + assert_eq!(rows[0].effective_result.as_deref(), Some("failed")); + assert!(rows[0].escalates); assert!(rows[0].active, "corruption issue is active"); }) .await; diff --git a/crates/jobs/src/backup/preflight.rs b/crates/jobs/src/backup/preflight.rs index 0fab234a..1bb925ae 100644 --- a/crates/jobs/src/backup/preflight.rs +++ b/crates/jobs/src/backup/preflight.rs @@ -12,7 +12,7 @@ //! (issuance) path is validated at onboarding by private-server's probe, not //! here; later drift surfaces as real device backups failing. //! -//! Alerts go through the group-level path (`database::backup::alerts`), which +//! Alerts go through `database::issues::file_check`, which //! bypasses per-server `is_monitored`. AWS calls are not unit-tested here (no //! live AWS in CI); the pure logic — the Object-Lock assertion — is. @@ -22,10 +22,11 @@ use aws_sdk_s3::types::ObjectLockRetentionMode; use aws_sdk_sts::error::{ProvideErrorMetadata, SdkError}; use aws_sdk_sts::operation::RequestId; use commons_servers::backup_jobs::slot_deadline_due; -use commons_types::issue::Severity; +use commons_types::status::CheckResult; use database::{ BackupConfigStatus, ServerGroupBackupConfig, - backup::alerts::{raise_group_event, refs}, + backup::refs, + issues::{CheckFiling, FilingScope, file_check}, }; use jiff::Timestamp; use tokio::{ @@ -58,7 +59,8 @@ fn validate_object_lock( } } -/// Raise the shared-identity self-alert (Critical → notifies immediately). +/// Raise the shared-identity self-alert (registers as an escalating +/// failure → notifies immediately). async fn file_identity_alert( db: &mut diesel_async::AsyncPgConnection, msg: &str, @@ -66,7 +68,10 @@ async fn file_identity_alert( database::self_alerts::raise( db, refs::PREFLIGHT_IDENTITY, - Severity::Critical, + CheckResult::Failed, + CheckResult::Failed, + true, + Some(refs::PREFLIGHT_IDENTITY_DOC), "Canopy IRSA identity broken", msg, ) @@ -89,14 +94,20 @@ async fn recover_identity_alert( Issue::active_group_ids_by_source_ref(db, refs::CANOPY_SOURCE, refs::PREFLIGHT_IDENTITY) .await? { - raise_group_event( + file_check( db, - group_id, - refs::PREFLIGHT_IDENTITY, - Severity::Info, - None, - "caller identity ok", - false, + CheckFiling { + source: database::statuses::CANOPY_SOURCE, + scope: FilingScope::Group(group_id), + check: refs::PREFLIGHT_IDENTITY, + observed: CheckResult::Passed, + title: None, + message: "caller identity ok", + detail: None, + default_ceiling: CheckResult::Failed, + default_escalates: true, + documentation: Some(refs::PREFLIGHT_IDENTITY_DOC), + }, ) .await?; } @@ -230,54 +241,78 @@ async fn deep_check_group( ) { match run_deep_check(aws, cfg).await { Ok(()) => { - let _ = raise_group_event( + let _ = file_check( db, - cfg.group_id, - refs::PREFLIGHT_ASSUME, - Severity::Info, - None, - "maintenance-role access ok", - false, + CheckFiling { + source: database::statuses::CANOPY_SOURCE, + scope: FilingScope::Group(cfg.group_id), + check: refs::PREFLIGHT_ASSUME, + observed: CheckResult::Passed, + title: None, + message: "maintenance-role access ok", + detail: None, + default_ceiling: CheckResult::Failed, + default_escalates: false, + documentation: Some(refs::PREFLIGHT_ASSUME_DOC), + }, ) .await; } Err(msg) => { error!(group = %cfg.group_id, "preflight assume failed: {msg}"); - let _ = raise_group_event( + let _ = file_check( db, - cfg.group_id, - refs::PREFLIGHT_ASSUME, - Severity::Error, - Some("backup maintenance-role access failed"), - &msg, - true, + CheckFiling { + source: database::statuses::CANOPY_SOURCE, + scope: FilingScope::Group(cfg.group_id), + check: refs::PREFLIGHT_ASSUME, + observed: CheckResult::Failed, + title: Some("backup maintenance-role access failed"), + message: &msg, + detail: None, + default_ceiling: CheckResult::Failed, + default_escalates: false, + documentation: Some(refs::PREFLIGHT_ASSUME_DOC), + }, ) .await; } } match check_object_lock(aws, cfg).await { Ok(()) => { - let _ = raise_group_event( + let _ = file_check( db, - cfg.group_id, - refs::PREFLIGHT_OBJECT_LOCK, - Severity::Info, - None, - "object lock ok", - false, + CheckFiling { + source: database::statuses::CANOPY_SOURCE, + scope: FilingScope::Group(cfg.group_id), + check: refs::PREFLIGHT_OBJECT_LOCK, + observed: CheckResult::Passed, + title: None, + message: "object lock ok", + detail: None, + default_ceiling: CheckResult::Failed, + default_escalates: true, + documentation: Some(refs::PREFLIGHT_OBJECT_LOCK_DOC), + }, ) .await; } Err(msg) => { error!(group = %cfg.group_id, "object-lock check failed: {msg}"); - let _ = raise_group_event( + let _ = file_check( db, - cfg.group_id, - refs::PREFLIGHT_OBJECT_LOCK, - Severity::Critical, - Some("bucket Object-Lock missing or weakened"), - &msg, - true, + CheckFiling { + source: database::statuses::CANOPY_SOURCE, + scope: FilingScope::Group(cfg.group_id), + check: refs::PREFLIGHT_OBJECT_LOCK, + observed: CheckResult::Failed, + title: Some("bucket Object-Lock missing or weakened"), + message: &msg, + detail: None, + default_ceiling: CheckResult::Failed, + default_escalates: true, + documentation: Some(refs::PREFLIGHT_OBJECT_LOCK_DOC), + }, ) .await; } @@ -318,7 +353,7 @@ pub fn spawn() -> JoinHandle<()> { }; // Shared check (every tick): IRSA identity resolves. Filed ONCE - // against the nil/meta server — a broken shared identity is one + // as a canopy-wide self-alert — a broken shared identity is one // fact about canopy, not one per group, and paging N groups at // once for it helps nobody. let identity_ok = aws.sts.get_caller_identity().send().await; @@ -377,7 +412,7 @@ mod tests { #[tokio::test(flavor = "multi_thread")] async fn identity_alert_files_once_and_recovers_legacy_fanout() { - use database::issues::Issue; + use database::issues::{Issue, raise_group_event}; use diesel_async::SimpleAsyncConnection as _; use uuid::Uuid; @@ -394,7 +429,6 @@ mod tests { &mut conn, group, refs::PREFLIGHT_IDENTITY, - Severity::Critical, Some("Canopy IRSA identity broken"), "legacy fan-out alert", true, @@ -402,35 +436,26 @@ mod tests { .await .expect("seed legacy alert"); - // Filing twice coalesces into the one nil/meta-server issue. + // Filing twice coalesces into the one canopy-wide issue. file_identity_alert(&mut conn, "sts:GetCallerIdentity failed: boom") .await .expect("file"); file_identity_alert(&mut conn, "sts:GetCallerIdentity failed: boom") .await .expect("file again"); - let nil_issues = Issue::list_by_source_ref( - &mut conn, - refs::CANOPY_SOURCE, - refs::PREFLIGHT_IDENTITY, - &[Uuid::nil()], - ) - .await - .expect("list"); - assert_eq!(nil_issues.len(), 1, "one coalescing meta-server issue"); - assert!(nil_issues[0].active); + let global = database::issues::get_global_issue(&mut conn, refs::PREFLIGHT_IDENTITY) + .await + .expect("get") + .expect("one coalescing canopy-wide issue"); + assert!(global.active); - // Recovery clears the meta-server issue AND the legacy one. + // Recovery clears the canopy-wide issue AND the legacy one. recover_identity_alert(&mut conn).await.expect("recover"); - let nil_issues = Issue::list_by_source_ref( - &mut conn, - refs::CANOPY_SOURCE, - refs::PREFLIGHT_IDENTITY, - &[Uuid::nil()], - ) - .await - .expect("list"); - assert!(!nil_issues[0].active, "meta-server alert recovered"); + let global = database::issues::get_global_issue(&mut conn, refs::PREFLIGHT_IDENTITY) + .await + .expect("get") + .expect("issue still exists"); + assert!(!global.active, "canopy-wide alert recovered"); assert!( Issue::active_group_ids_by_source_ref( &mut conn, diff --git a/crates/jobs/src/bin/monitor.rs b/crates/jobs/src/bin/monitor.rs index 8a661949..f617a687 100644 --- a/crates/jobs/src/bin/monitor.rs +++ b/crates/jobs/src/bin/monitor.rs @@ -4,13 +4,12 @@ //! a separate pod per check. //! //! Sweeps run each tick: -//! - **reachability** — look at every monitored server's latest status row and -//! file (or close) the `(source=canopy, ref=reachability)` issue. Severity -//! escalates as the server stays unreported — Notice → Warning at the -//! sub-incident tiers, then Error (opens an incident) at "Down", then -//! Critical at "Gone". Independent of pingtask: most servers push their own -//! status, so the sweep operates on the resulting `statuses` rows regardless -//! of which path produced them. +//! - **staleness** — look at every monitored server's report freshness against +//! its down threshold and file (or close) the `(source=canopy, +//! ref=reachability)` check when nothing at all is reporting, plus a +//! `stale/` check per reporting source that has gone quiet. +//! Independent of pingtask: most servers push their own status, so the sweep +//! operates on the resulting rows regardless of which path produced them. //! - **backup staleness + reconcile** — `database::backup::sweep`: stale //! reported runs / maintenance, and report-vs-inventory reconciliation. //! - **tailnet key-expiry** — when the Tailscale directory is configured. @@ -109,10 +108,10 @@ pub fn spawn() -> JoinHandle<()> { continue; }; - match Status::sweep_reachability(&mut db).await { + match Status::sweep_staleness(&mut db).await { Ok(0) => {} - Ok(n) => debug!("filed {n} reachability events"), - Err(err) => error!("reachability sweep failed: {err}"), + Ok(n) => debug!("filed {n} staleness events"), + Err(err) => error!("staleness sweep failed: {err}"), } // Backup staleness + report-vs-inventory reconciliation: another diff --git a/crates/jobs/src/bin/slacker_outbox.rs b/crates/jobs/src/bin/slacker_outbox.rs index a4ba57a1..f5e169b5 100644 --- a/crates/jobs/src/bin/slacker_outbox.rs +++ b/crates/jobs/src/bin/slacker_outbox.rs @@ -15,7 +15,7 @@ use std::sync::atomic::{AtomicI64, Ordering}; use std::time::Duration; use clap::Parser; -use commons_types::issue::Severity; +use commons_types::status::CheckResult; use database::slack_outbox::{ KIND_INCIDENT_OPEN, KIND_INCIDENT_RESOLVE, KIND_SELF_ALERT_OPEN, KIND_SELF_ALERT_RESOLVE, SlackOutbox, @@ -55,19 +55,10 @@ const WATCHDOG_CHECK_EVERY: Duration = Duration::from_secs(30); /// Partial configuration is a hard error at startup — silently dropping rows /// for an unconfigured kind is exactly how we missed an entire month of /// resolve notifications previously, so a noisy startup failure is preferred. -/// -/// `SLACK_WEBHOOK_SELF_ALERT_URL` is deliberately outside that all-or-none -/// set: it postdates deployed configuration, and folding it in would -/// hard-fail every existing deploy on upgrade. Unset while incident hooks -/// are configured, self-alert rows are marked delivered without posting and -/// a warning is logged at startup — the alerts are still recorded and shown -/// in the operator UI. Fold it into the strict set once ops config carries -/// it everywhere. #[derive(Clone, Debug, Default)] struct Config { open: Option, resolve: Option, - self_alert: Option, private_url: Option, } @@ -76,7 +67,6 @@ impl Config { Self::build( std::env::var("SLACK_WEBHOOK_OPEN_URL").ok(), std::env::var("SLACK_WEBHOOK_RESOLVE_URL").ok(), - std::env::var("SLACK_WEBHOOK_SELF_ALERT_URL").ok(), std::env::var("PRIVATE_URL").ok(), ) } @@ -84,7 +74,6 @@ impl Config { fn build( open: Option, resolve: Option, - self_alert: Option, private_url: Option, ) -> miette::Result { let inputs: [(&str, &Option); 2] = [ @@ -92,7 +81,7 @@ impl Config { ("SLACK_WEBHOOK_RESOLVE_URL", &resolve), ]; let set_count = inputs.iter().filter(|(_, v)| v.is_some()).count(); - if set_count == 0 && self_alert.is_none() { + if set_count == 0 { return Ok(Self::default()); } if set_count != 0 && set_count != inputs.len() { @@ -118,22 +107,23 @@ impl Config { Ok(Self { open, resolve, - self_alert, private_url: Some(private_url), }) } fn any_hook(&self) -> bool { - self.open.is_some() || self.resolve.is_some() || self.self_alert.is_some() + self.open.is_some() || self.resolve.is_some() } - /// `Ok(Some(url))` — post there; `Ok(None)` — known kind whose optional - /// hook is unset (mark delivered without posting); `Err` — unknown kind. + /// `Ok(Some(url))` — post there; `Ok(None)` — known kind with no hook + /// (mark delivered without posting); `Err` — unknown kind. fn url_for(&self, kind: &str) -> Result, ()> { match kind { KIND_INCIDENT_OPEN => Ok(self.open.as_deref()), KIND_INCIDENT_RESOLVE => Ok(self.resolve.as_deref()), - KIND_SELF_ALERT_OPEN | KIND_SELF_ALERT_RESOLVE => Ok(self.self_alert.as_deref()), + // Legacy: nothing enqueues self-alert rows anymore; stragglers + // from before an upgrade drain as delivered without posting. + KIND_SELF_ALERT_OPEN | KIND_SELF_ALERT_RESOLVE => Ok(None), _ => Err(()), } } @@ -153,11 +143,6 @@ fn spawn(cfg: Config) -> JoinHandle<()> { let pool = database::init(); if !cfg.any_hook() { info!("no SLACK_WEBHOOK_*_URL set; slack outbox drainer running in no-op mode"); - } else if cfg.self_alert.is_none() { - warn!( - "SLACK_WEBHOOK_SELF_ALERT_URL unset; self-alerts will be recorded and \ - shown in the operator UI but not sent to Slack" - ); } let client = reqwest::Client::builder() .timeout(REQUEST_TIMEOUT) @@ -263,7 +248,10 @@ async fn file_self_event( database::self_alerts::raise( conn, database::self_alerts::SLACK_DELIVERY_FAILURE_REF, - Severity::Error, + CheckResult::Failed, + CheckResult::Failed, + false, + Some(database::self_alerts::SLACK_DELIVERY_FAILURE_DOC), &format!("Slack delivery permanently failed ({})", row.kind), &format!( "outbox row {} (kind={}, incident={:?}): gave up after {attempts} attempts. Last error: {}. Last response: {}", @@ -361,11 +349,11 @@ async fn deliver( }); } Ok(Some(url)) => url, - // The self-alert hook is optional (see Config docs): unset means - // record-only. Incident kinds unset while any hook is configured - // stay a hard error — that silence has bitten before. + // Legacy self-alert rows have no hook anymore: drain them as + // delivered without posting. Incident kinds unset while any hook + // is configured stay a hard error — that silence has bitten before. Ok(None) if row.kind == KIND_SELF_ALERT_OPEN || row.kind == KIND_SELF_ALERT_RESOLVE => { - debug!(id = %row.id, kind = %row.kind, "self-alert hook unset; row marked delivered without posting"); + debug!(id = %row.id, kind = %row.kind, "legacy self-alert row; marked delivered without posting"); return Ok(String::new()); } Ok(None) => { @@ -481,7 +469,6 @@ mod tests { let err = Config::build( Some("http://example/open".into()), None, - None, Some("https://canopy.test".into()), ) .expect_err("must require every incident SLACK_WEBHOOK_*_URL when any is set"); @@ -497,7 +484,6 @@ mod tests { Some("http://example/open".into()), Some("http://example/resolve".into()), None, - None, ) .expect_err("must require PRIVATE_URL"); assert!(err.to_string().contains("PRIVATE_URL")); @@ -505,7 +491,7 @@ mod tests { #[test] fn config_ok_when_no_hooks_set_even_without_private_url() { - let cfg = Config::build(None, None, None, None).expect("no-op mode is fine"); + let cfg = Config::build(None, None, None).expect("no-op mode is fine"); assert!(!cfg.any_hook()); } @@ -514,7 +500,6 @@ mod tests { let cfg = Config::build( Some("http://example/open".into()), Some("http://example/resolve".into()), - Some("http://example/self-alert".into()), Some("https://canopy.test".into()), ) .expect("complete config is fine"); @@ -527,14 +512,8 @@ mod tests { cfg.url_for(KIND_INCIDENT_RESOLVE), Ok(Some("http://example/resolve")) ); - assert_eq!( - cfg.url_for(KIND_SELF_ALERT_OPEN), - Ok(Some("http://example/self-alert")) - ); - assert_eq!( - cfg.url_for(KIND_SELF_ALERT_RESOLVE), - Ok(Some("http://example/self-alert")) - ); + assert_eq!(cfg.url_for(KIND_SELF_ALERT_OPEN), Ok(None)); + assert_eq!(cfg.url_for(KIND_SELF_ALERT_RESOLVE), Ok(None)); assert_eq!(cfg.url_for("mystery"), Err(())); } @@ -553,7 +532,6 @@ mod tests { let cfg = Config { open: Some(url), resolve: None, - self_alert: None, private_url: Some("https://canopy.example.ts.net".into()), }; let mut r = row( @@ -599,7 +577,6 @@ mod tests { let cfg = Config { open: Some(url), resolve: None, - self_alert: None, private_url: Some("https://new.example/".into()), }; let mut r = row( @@ -639,7 +616,6 @@ mod tests { let cfg = Config { open: Some("http://127.0.0.1:1/open-should-not-be-hit".into()), resolve: Some(resolve_url), - self_alert: None, private_url: Some("https://canopy.test".into()), }; let r = row( @@ -669,7 +645,6 @@ mod tests { let cfg = Config { open: Some(url), resolve: None, - self_alert: None, private_url: Some("https://canopy.test".into()), }; let r = row(KIND_INCIDENT_OPEN, serde_json::json!({})); @@ -693,7 +668,6 @@ mod tests { let cfg = Config { open: Some("http://127.0.0.1:1/should-not-be-hit".into()), resolve: Some("http://127.0.0.1:1/should-not-be-hit".into()), - self_alert: None, private_url: Some("https://canopy.test".into()), }; let r = row("bogus_kind", serde_json::json!({})); @@ -707,14 +681,13 @@ mod tests { } #[tokio::test(flavor = "multi_thread")] - async fn deliver_self_alert_without_hook_is_a_per_kind_noop() { - // The self-alert hook is optional: with incident hooks configured - // but SLACK_WEBHOOK_SELF_ALERT_URL unset, self-alert rows are - // marked delivered without posting rather than erroring forever. + async fn deliver_legacy_self_alert_row_is_a_per_kind_noop() { + // Nothing enqueues self-alert rows anymore, but stragglers from + // before an upgrade must drain as delivered rather than erroring + // forever. let cfg = Config { open: Some("http://127.0.0.1:1/should-not-be-hit".into()), resolve: Some("http://127.0.0.1:1/should-not-be-hit".into()), - self_alert: None, private_url: Some("https://canopy.test".into()), }; let mut r = row(KIND_SELF_ALERT_OPEN, serde_json::json!({})); @@ -725,53 +698,6 @@ mod tests { assert_eq!(body, ""); } - #[tokio::test(flavor = "multi_thread")] - async fn deliver_routes_self_alert_kinds_and_links_to_alerts() { - let (url, server, recorded) = one_shot_server(); - let cfg = Config { - open: Some("http://127.0.0.1:1/open-should-not-be-hit".into()), - resolve: Some("http://127.0.0.1:1/resolve-should-not-be-hit".into()), - self_alert: Some(url), - private_url: Some("https://canopy.example.ts.net".into()), - }; - let mut r = row( - KIND_SELF_ALERT_OPEN, - serde_json::json!({ - "state": "alert", - "severity": "Critical", - "source_ref": "canopy/preflight-identity", - "title": "Canopy IRSA identity broken", - "message": "boom", - }), - ); - r.incident_id = None; - deliver(&reqwest::Client::new(), &cfg, &r) - .await - .expect("deliver ok"); - server.join().unwrap(); - - let got = recorded.lock().unwrap().clone().expect("got a request"); - assert_eq!(got["state"], "alert"); - assert_eq!(got["source_ref"], "canopy/preflight-identity"); - assert_eq!( - got["link"], "https://canopy.example.ts.net/alerts", - "self-alert rows link to the alerts view, not an incident", - ); - } - - #[tokio::test(flavor = "multi_thread")] - async fn config_accepts_missing_self_alert_hook() { - let cfg = Config::build( - Some("http://example/open".into()), - Some("http://example/resolve".into()), - None, - Some("https://canopy.test".into()), - ) - .expect("self-alert hook is optional"); - assert!(cfg.any_hook()); - assert_eq!(cfg.url_for(KIND_SELF_ALERT_OPEN), Ok(None)); - } - #[tokio::test(flavor = "multi_thread")] async fn deliver_noop_swallows_unknown_kind() { // In dev / no-op mode (no hooks configured) any kind is a no-op — diff --git a/crates/private-server/src/fns/healthchecks.rs b/crates/private-server/src/fns/healthchecks.rs index 1362e3c3..1acefa50 100644 --- a/crates/private-server/src/fns/healthchecks.rs +++ b/crates/private-server/src/fns/healthchecks.rs @@ -1,5 +1,5 @@ -//! Operator-owned catalog of healthcheck names → severity. Read and -//! edit endpoints for the catalog page at /healthchecks. Ingestion +//! Operator-owned catalog of check policies per (source, check). Read +//! and edit endpoints for the catalog page at /healthchecks. Ingestion //! (in the public-server status handler) maintains the rows; this //! module exposes them to admins. @@ -8,8 +8,8 @@ use axum::extract::State; use canopy_utoipa_axum::{router::OpenApiRouter, routes}; use commons_errors::{AppError, ProblemDetailsSchema, Result}; use commons_servers::tailscale_auth::TailscaleAdmin; -use commons_types::issue::Severity; -use database::healthcheck_severities::{HealthcheckSeverity, IfLadder}; +use commons_types::status::CheckResult; +use database::check_policies::{CheckPolicy, IfLadder}; use database::servers::Server; use database::statuses::Status; use jiff::Timestamp; @@ -25,20 +25,30 @@ pub fn routes() -> OpenApiRouter { .routes(routes!(list)) .routes(routes!(update)) .routes(routes!(update_rules)) + .routes(routes!(update_documentation)) .routes(routes!(sample)) .routes(routes!(tag_keys)) } -/// A named healthcheck's alerting policy: the base severity assigned to -/// its failures, plus optional conditional rules that can override that -/// severity based on the details of a given failure. +/// One (source, check)'s policy: the ceiling capping its effective +/// result, the escalation flag, plus optional conditional rules that can +/// grade a given report differently. #[derive(Debug, Clone, Serialize, ToSchema)] -pub struct HealthcheckSeverityData { +pub struct CheckPolicyData { + /// The source that reports this check. + pub source: String, /// The healthcheck's name, exactly as reported by monitored servers. pub check_name: String, - /// The severity assigned to a failure of this check when no - /// conditional rule (see `rules`) overrides it. - pub severity: Severity, + /// The maximum effective result for this check when no conditional + /// rule (see `rules`) overrides it: an observed result more urgent + /// than the ceiling grades down to it. One of `failed`, `warning`, + /// `passed`, or `skipped` (`skipped` also tells the reporting source + /// it may stop running the check). + #[schema(value_type = String)] + pub ceiling: CheckResult, + /// Whether an effective failure of this check notifies immediately, + /// bypassing the incident grace period. + pub escalates: bool, /// When this check was first reported and this policy entry was /// created. pub first_seen: Timestamp, @@ -52,21 +62,26 @@ pub struct HealthcheckSeverityData { pub notes: Option, /// When this policy was last modified. pub updated_at: Timestamp, + /// Operator-authored documentation for this check: a single markdown + /// document. By convention it covers what the check observes, what + /// each result means, and hints for solving a failure, but no + /// structure is enforced. `null` when nobody has documented it yet. + pub documentation: Option, /// `true` if no operator has reviewed this policy yet. pub pending_review: bool, - /// Conditional rules that can assign a different severity than - /// `severity`, depending on the failing check's own fields, the - /// surrounding status report, or the reporting server's tags. `null` - /// means no conditional rules are configured, and `severity` always - /// applies. + /// Conditional rules that can grade a report to a different result + /// than the ceiling would — in any direction — depending on the + /// check's own fields, the surrounding status report, or the + /// reporting server's tags. `null` means no conditional rules are + /// configured, and the ceiling always applies. /// /// When present, this is a single-key object shaped like - /// `{"if": [condition_1, severity_1, condition_2, severity_2, ...]}`. - /// Conditions are tried in order, and the severity paired with the - /// first matching condition is used; if none match, the base - /// `severity` is used instead. There's no explicit "else" branch — - /// the fallback to `severity` plays that role — so the array must - /// have an even number of entries and at least one pair. + /// `{"if": [condition_1, result_1, condition_2, result_2, ...]}`. + /// Conditions are tried in order, and the result paired with the + /// first matching condition is used; if none match, the observed + /// result capped at the ceiling is used instead. There's no explicit + /// "else" branch — the ceiling fallback plays that role — so the + /// array must have an even number of entries and at least one pair. /// /// Each condition is a single-key object naming a comparison /// operator — one of `==`, `!=`, `<`, `<=`, `>`, `>=`, or `in_range` @@ -83,7 +98,7 @@ pub struct HealthcheckSeverityData { /// is treated the same as `null` (no conditional rules). #[schema(value_type = Option)] pub rules: Option, - /// Number of condition/severity branches in `rules`; `0` when + /// Number of condition/result branches in `rules`; `0` when /// `rules` is `null` or couldn't be parsed. Lets a caller tell /// whether conditional rules exist without parsing `rules` itself. pub rule_count: u32, @@ -96,18 +111,21 @@ fn rule_count(rules: &Option) -> u32 { .unwrap_or(0) } -impl From for HealthcheckSeverityData { - fn from(h: HealthcheckSeverity) -> Self { +impl From for CheckPolicyData { + fn from(h: CheckPolicy) -> Self { let pending_review = h.reviewed_at.is_none(); let rule_count = rule_count(&h.rules); Self { + source: h.source, check_name: h.check_name, - severity: h.severity, + ceiling: h.ceiling, + escalates: h.escalates, first_seen: h.first_seen, reviewed_at: h.reviewed_at, reviewed_by: h.reviewed_by, notes: h.notes, updated_at: h.updated_at, + documentation: h.documentation, pending_review, rules: h.rules, rule_count, @@ -115,11 +133,11 @@ impl From for HealthcheckSeverityData { } } -/// List the healthcheck severity catalog. +/// List the check policy catalog. /// -/// Returns every known healthcheck name together with its current -/// severity policy, ordered by name. An entry exists for every check name -/// any server has ever reported; new checks are added automatically the +/// Returns every known (source, check) together with its current policy, +/// ordered by source then name. An entry exists for every check any +/// source has ever reported; new checks are added automatically the /// first time they're seen, with a default policy pending review. #[utoipa::path( post, @@ -128,7 +146,7 @@ impl From for HealthcheckSeverityData { tag = "healthchecks", security(("tailscale-admin" = [])), responses( - (status = 200, description = "Catalog rows ordered by check_name.", body = Vec), + (status = 200, description = "Catalog rows ordered by source then check_name.", body = Vec), (status = 401, body = ProblemDetailsSchema), (status = 403, body = ProblemDetailsSchema), ), @@ -136,22 +154,30 @@ impl From for HealthcheckSeverityData { pub async fn list( State(state): State, _admin: TailscaleAdmin, -) -> Result>> { +) -> Result>> { let mut conn = state.db.get().await?; - let rows = HealthcheckSeverity::list(&mut conn).await?; + let rows = CheckPolicy::list(&mut conn).await?; Ok(Json(rows.into_iter().map(Into::into).collect())) } -/// Request body for updating a healthcheck's base severity policy. +/// Request body for updating a check's base policy. #[derive(Deserialize, ToSchema)] pub struct HealthcheckUpdateArgs { + /// The source whose check to update. + pub source: String, /// The healthcheck name to update; must already exist in the /// catalog. pub check_name: String, - /// The severity to assign to this check's failures when no - /// conditional rule overrides it. - pub severity: Severity, - /// Operator notes to store alongside the new severity. Omitting this + /// The ceiling to apply to this check's observed results when no + /// conditional rule overrides it: one of `failed`, `warning`, + /// `passed`, or `skipped`. + #[schema(value_type = String)] + pub ceiling: CheckResult, + /// Whether an effective failure of this check should notify + /// immediately, bypassing the incident grace period. + #[serde(default)] + pub escalates: bool, + /// Operator notes to store alongside the new policy. Omitting this /// or sending `null` clears any existing notes — there's no way to /// leave them unchanged implicitly, so resend the current value to /// keep it. @@ -159,12 +185,12 @@ pub struct HealthcheckUpdateArgs { pub notes: Option, } -/// Update a healthcheck's severity policy. +/// Update a check's policy. /// -/// Sets the base severity (and optionally notes) for the given check, and -/// marks it as reviewed by the caller. Saving with the same severity as -/// before still counts as a review, so an operator can acknowledge a -/// check without changing its policy. +/// Sets the ceiling and escalation flag (and optionally notes) for the +/// given (source, check), and marks it as reviewed by the caller. Saving +/// with the same values as before still counts as a review, so an +/// operator can acknowledge a check without changing its policy. #[utoipa::path( post, path = "/update", @@ -173,7 +199,7 @@ pub struct HealthcheckUpdateArgs { security(("tailscale-admin" = [])), request_body = HealthcheckUpdateArgs, responses( - (status = 200, description = "Updated catalog row.", body = HealthcheckSeverityData), + (status = 200, description = "Updated catalog row.", body = CheckPolicyData), (status = 401, body = ProblemDetailsSchema), (status = 403, body = ProblemDetailsSchema), ), @@ -182,12 +208,22 @@ pub async fn update( State(state): State, admin: TailscaleAdmin, Json(args): Json, -) -> Result> { +) -> Result> { + if !matches!( + args.ceiling, + CheckResult::Failed | CheckResult::Warning | CheckResult::Passed | CheckResult::Skipped + ) { + return Err(AppError::BadRequest( + "ceiling must be one of failed, warning, passed, skipped".into(), + )); + } let mut conn = state.db.get().await?; - let row = HealthcheckSeverity::update( + let row = CheckPolicy::update( &mut conn, + &args.source, &args.check_name, - args.severity, + args.ceiling, + args.escalates, args.notes.as_deref(), &admin.0.login, ) @@ -195,28 +231,79 @@ pub async fn update( Ok(Json(row.into())) } -/// Request body for replacing a healthcheck's conditional severity rules. +/// Request body for replacing a check's documentation. +#[derive(Deserialize, ToSchema)] +pub struct UpdateDocumentationArgs { + /// The source whose check to document. + pub source: String, + /// The healthcheck name to document; must already exist in the + /// catalog. + pub check_name: String, + /// The new markdown document, or `null` (or blank) to clear it. + #[serde(default)] + pub documentation: Option, +} + +/// Replace a check's documentation. +/// +/// Stores the markdown document presented alongside the check wherever +/// its state appears, and over MCP. Sending `null` or a blank document +/// clears it. Doesn't mark the policy as reviewed — documenting a check +/// is not the same as reviewing its grading. +#[utoipa::path( + post, + path = "/update_documentation", + operation_id = "healthcheck_update_documentation", + tag = "healthchecks", + security(("tailscale-admin" = [])), + request_body = UpdateDocumentationArgs, + responses( + (status = 200, description = "Updated catalog row.", body = CheckPolicyData), + (status = 401, body = ProblemDetailsSchema), + (status = 403, body = ProblemDetailsSchema), + ), +)] +pub async fn update_documentation( + State(state): State, + _admin: TailscaleAdmin, + Json(args): Json, +) -> Result> { + let documentation = args + .documentation + .as_deref() + .map(str::trim) + .filter(|d| !d.is_empty()); + let mut conn = state.db.get().await?; + let row = + CheckPolicy::update_documentation(&mut conn, &args.source, &args.check_name, documentation) + .await?; + Ok(Json(row.into())) +} + +/// Request body for replacing a check's conditional rules. #[derive(Deserialize, ToSchema)] pub struct UpdateRulesArgs { + /// The source whose check's rules to replace. + pub source: String, /// The healthcheck name whose rules to replace; must already exist /// in the catalog. pub check_name: String, /// The new conditional rules to store, or `null` to remove all - /// conditional rules and rely solely on the base severity. Same - /// shape as the `rules` field returned when listing checks. A ladder - /// with no condition/severity pairs is treated the same as `null`. + /// conditional rules and rely solely on the ceiling. Same shape as + /// the `rules` field returned when listing checks. A ladder with no + /// condition/result pairs is treated the same as `null`. #[schema(value_type = Option)] #[serde(default)] pub rules: Option, } -/// Replace a healthcheck's conditional severity rules. +/// Replace a check's conditional rules. /// -/// Stores a new set of conditional rules for the given check (or removes -/// them, if `rules` is `null`), and marks the check as reviewed by the -/// caller. Returns 400 if `rules` doesn't parse as a valid ladder — for -/// example an unknown comparison operator, a malformed variable -/// reference, or an odd number of entries. +/// Stores a new set of conditional rules for the given (source, check) +/// (or removes them, if `rules` is `null`), and marks the check as +/// reviewed by the caller. Returns 400 if `rules` doesn't parse as a +/// valid ladder — for example an unknown comparison operator, a +/// malformed variable reference, or an odd number of entries. #[utoipa::path( post, path = "/update_rules", @@ -225,7 +312,7 @@ pub struct UpdateRulesArgs { security(("tailscale-admin" = [])), request_body = UpdateRulesArgs, responses( - (status = 200, description = "Updated catalog row.", body = HealthcheckSeverityData), + (status = 200, description = "Updated catalog row.", body = CheckPolicyData), (status = 400, body = ProblemDetailsSchema), (status = 401, body = ProblemDetailsSchema), (status = 403, body = ProblemDetailsSchema), @@ -235,7 +322,7 @@ pub async fn update_rules( State(state): State, admin: TailscaleAdmin, Json(args): Json, -) -> Result> { +) -> Result> { let ladder: Option = match args.rules { None | Some(JsonValue::Null) => None, Some(v) => { @@ -251,8 +338,9 @@ pub async fn update_rules( } }; let mut conn = state.db.get().await?; - let row = HealthcheckSeverity::update_rules( + let row = CheckPolicy::update_rules( &mut conn, + &args.source, &args.check_name, ladder.as_ref(), &admin.0.login, @@ -264,13 +352,17 @@ pub async fn update_rules( /// Request body identifying which healthcheck to sample data for. #[derive(Deserialize, ToSchema)] pub struct SampleArgs { + /// The source that reports the check. A check's identity is the + /// (source, check) pair; another source's same-named check may carry + /// entirely different fields. + pub source: String, /// The healthcheck name to sample. pub check_name: String, } /// A real-world sample of the data a conditional rule can reference for a /// given healthcheck, taken from the most recent status report (across -/// all servers) that included it. +/// all servers) from the check's own source that included it. #[derive(Serialize, ToSchema)] pub struct HealthcheckSample { /// Additional top-level fields submitted with the status report that @@ -330,7 +422,9 @@ pub async fn sample( Json(args): Json, ) -> Result> { let mut conn = state.db.get().await?; - let Some(status) = Status::latest_for_check_name(&mut conn, &args.check_name).await? else { + let Some(status) = + Status::latest_for_check_name(&mut conn, &args.source, &args.check_name).await? + else { return Ok(Json(HealthcheckSampleResponse { check_name: args.check_name, sample: None, @@ -346,7 +440,7 @@ pub async fn sample( // by name; we don't require a failing result here so we still // surface the check's typical shape even on a passing push). Strip // the reserved fields and inject the normalised `result` so the UI - // sees exactly what the ingestion path passes to severity_for — + // sees exactly what the ingestion path passes to the policy — // including a `check.result` value on legacy (`healthy: bool`) // pushes. let check_extra = status diff --git a/crates/private-server/src/fns/incidents.rs b/crates/private-server/src/fns/incidents.rs index 15080089..567b7915 100644 --- a/crates/private-server/src/fns/incidents.rs +++ b/crates/private-server/src/fns/incidents.rs @@ -27,11 +27,12 @@ const DEFAULT_LIMIT: i64 = 100; pub struct IncidentData { /// Unique identifier of the incident. pub id: Uuid, - /// Identifier of the server group the incident belongs to. - pub server_group_id: Uuid, - /// Display name of the group this incident rolls up to. Empty only if - /// the group no longer exists, which should not happen in normal - /// operation. + /// Identifier of the server group the incident belongs to, or null for + /// a canopy-wide incident (aggregating canopy's self-alerts). + pub server_group_id: Option, + /// Display name of the group this incident rolls up to — `Canopy` for + /// a canopy-wide incident. Empty only if the group no longer exists, + /// which should not happen in normal operation. pub server_group_name: String, /// When the incident opened. pub opened_at: Timestamp, @@ -52,8 +53,6 @@ pub struct IncidentData { pub resolved_reason: Option, /// Number of distinct issues that have ever contributed to the incident. pub issue_count: i64, - /// Total number of events across all contributing issues. - pub event_count: i64, /// Combined count of notes on the incident itself plus notes on all its /// contributing issues. pub note_count: i64, @@ -91,7 +90,6 @@ impl IncidentData { resolved_by_pic: res_pic, resolved_reason: i.resolved_reason, issue_count: stats.issue_count, - event_count: stats.event_count, note_count: stats.note_count, notification_held_until, created_at: i.created_at, @@ -115,7 +113,7 @@ async fn enrich_incidents( pool: &database::Db, incidents: Vec, ) -> Result> { - let group_ids: Vec = incidents.iter().map(|i| i.server_group_id).collect(); + let group_ids: Vec = incidents.iter().filter_map(|i| i.server_group_id).collect(); let incident_ids: Vec = incidents.iter().map(|i| i.id).collect(); let groups = ServerGroup::list_by_ids(conn, &group_ids).await?; let group_names: std::collections::HashMap = @@ -128,10 +126,10 @@ async fn enrich_incidents( Ok(incidents .into_iter() .map(|i| { - let name = group_names - .get(&i.server_group_id) - .cloned() - .unwrap_or_default(); + let name = match i.server_group_id { + Some(gid) => group_names.get(&gid).cloned().unwrap_or_default(), + None => "Canopy".to_string(), + }; let s = stats.get(&i.id).copied().unwrap_or_default(); let held_until = held.get(&i.id).copied(); IncidentData::from_with(i, name, &users, s, held_until) @@ -144,7 +142,10 @@ async fn enrich_incident( pool: &database::Db, incident: Incident, ) -> Result { - let group = ServerGroup::get_by_id(conn, incident.server_group_id).await?; + let group_name = match incident.server_group_id { + Some(gid) => ServerGroup::get_by_id(conn, gid).await?.name, + None => "Canopy".to_string(), + }; let user_logins = collect_incident_user_logins(std::slice::from_ref(&incident)); let users = CachedTailscaleUser::by_logins(conn, &user_logins).await?; let mut stats = Incident::stats_for(pool, &[incident.id]).await?; @@ -153,7 +154,7 @@ async fn enrich_incident( database::slack_outbox::SlackOutbox::pending_opens_until(conn, &[incident.id]).await?; let held_until = held.remove(&incident.id); Ok(IncidentData::from_with( - incident, group.name, &users, s, held_until, + incident, group_name, &users, s, held_until, )) } diff --git a/crates/private-server/src/fns/issues.rs b/crates/private-server/src/fns/issues.rs index faa85a1b..69931919 100644 --- a/crates/private-server/src/fns/issues.rs +++ b/crates/private-server/src/fns/issues.rs @@ -3,12 +3,10 @@ use axum::extract::State; use canopy_utoipa_axum::{router::OpenApiRouter, routes}; use commons_errors::{AppError, ProblemDetailsSchema, Result}; use commons_servers::tailscale_auth::{TailscaleAdmin, TailscaleUser}; -use commons_types::{ - Uuid, - issue::{ResolvedReason, Severity}, -}; +use commons_types::{Uuid, issue::ResolvedReason, status::CheckResult}; use database::issues::{ - Event, Incident, Issue, IssueFilter, IssueIncidentRef, IssueListFilters, NewEvent, + CheckFiling, FilingScope, Incident, Issue, IssueFilter, IssueIncidentRef, IssueListFilters, + MANUAL_SOURCE, file_check, }; use database::notes::IssueNote; use database::servers::Server; @@ -17,7 +15,6 @@ use jiff::Timestamp; use serde::{Deserialize, Serialize}; use utoipa::ToSchema; -use crate::fns::Page; use crate::state::AppState; const DEFAULT_LIMIT: i64 = 100; @@ -52,8 +49,9 @@ pub struct IssueData { /// within its source and server. #[serde(rename = "ref")] pub r#ref: String, - /// Current severity level of the issue. - pub severity: Severity, + /// Whether the check's policy escalates: an effective failure notifies + /// immediately, bypassing incident grace. + pub escalates: bool, /// Short headline describing the issue, if one was given. pub description: Option, /// Latest human-readable message describing the issue's state. @@ -85,6 +83,18 @@ pub struct IssueData { pub created_at: Timestamp, /// When this issue record was last updated. pub updated_at: Timestamp, + /// The check this issue tracks, when it is check state (health-check + /// issues). Absent for issues that aren't check results yet. + pub check_name: Option, + /// The result the source reported on the latest filing, before policy. + #[schema(value_type = Option)] + pub observed_result: Option, + /// What policy made of the latest observed result — the result canopy + /// acts on. + #[schema(value_type = Option)] + pub effective_result: Option, + /// The check's own fields from the latest report, verbatim. + pub detail: Option, /// Incidents this issue is or was attached to, most recent first. Empty /// for issues that never escalated into an incident. pub incidents: Vec, @@ -135,7 +145,7 @@ impl IssueData { device_id: i.device_id, source: i.source, r#ref: i.r#ref, - severity: i.severity, + escalates: i.escalates, description: i.description, message: i.message, active: i.active, @@ -149,6 +159,10 @@ impl IssueData { snoozed_until: i.snoozed_until, created_at: i.created_at, updated_at: i.updated_at, + check_name: i.check_name, + observed_result: i.observed_result, + effective_result: i.effective_result, + detail: i.detail, incidents: e.incidents, } } @@ -261,57 +275,11 @@ pub(crate) async fn enrich_issue( )) } -/// A single recorded occurrence of an issue's underlying condition — one -/// push from a device or a manually submitted event. -#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)] -pub struct EventData { - /// Unique identifier for this event. - pub id: Uuid, - /// Id of the issue this event belongs to. - pub issue_id: Uuid, - /// When this event was recorded on the server. - pub created_at: Timestamp, - /// When the underlying condition actually occurred, if reported - /// separately from the time it was recorded. - pub occurred_at: Option, - /// Severity reported for this event. - pub severity: Severity, - /// Short headline for this event, if one was given. - pub description: Option, - /// Human-readable message describing this event. - pub message: String, - /// Whether the underlying condition was active as of this event. - pub active: bool, - /// Number of times this same condition has repeated and been coalesced - /// into this event rather than creating a new one. - pub occurrences: i32, - /// When this condition was last seen recurring. - pub last_seen: Timestamp, -} - -impl From for EventData { - fn from(e: Event) -> Self { - Self { - id: e.id, - issue_id: e.issue_id, - created_at: e.created_at, - occurred_at: e.occurred_at, - severity: e.severity, - description: e.description, - message: e.message, - active: e.active, - occurrences: e.occurrences, - last_seen: e.last_seen, - } - } -} - pub fn routes() -> OpenApiRouter { OpenApiRouter::new() .routes(routes!(list)) .routes(routes!(list_for_device)) .routes(routes!(list_for_server)) - .routes(routes!(list_events)) .routes(routes!(submit_manual_event)) .routes(routes!(resolve)) .routes(routes!(unresolve)) @@ -337,10 +305,11 @@ pub struct IssueListArgs { /// ones. Defaults to `true` (active issues only) when omitted. #[serde(default)] pub active_only: Option, - /// Restrict to issues at one of these severity levels. Omit to include - /// all severities. + /// Restrict to issues whose latest effective result is one of these. + /// Omit to include all results. #[serde(default)] - pub severities: Option>, + #[schema(value_type = Option>)] + pub results: Option>, /// Restrict to issues whose server belongs to this group. #[serde(default)] pub server_group_id: Option, @@ -375,7 +344,7 @@ pub async fn list( &mut conn, IssueListFilters { active_only: args.active_only.unwrap_or(true), - severities: args.severities, + results: args.results, server_group_id: args.server_group_id, since: None, }, @@ -476,87 +445,48 @@ pub async fn list_for_server( Ok(Json(enrich_issues(&mut conn, issues).await?)) } -/// Selects an issue and a page of its events. -#[derive(Deserialize, ToSchema)] -pub struct ListEventsArgs { - /// Id of the issue whose events to list. - pub issue_id: Uuid, - /// Number of events to skip, for pagination. Defaults to 0. - #[serde(default)] - pub offset: Option, - /// Maximum number of events to return. Defaults to 100 when omitted. - #[serde(default)] - pub limit: Option, -} - -/// List the events recorded against a specific issue, most recent first. -/// -/// Returns a page of events along with the total number of events recorded -/// for the issue, for pagination with `offset`/`limit`. -#[utoipa::path( - post, - path = "/list_events", - tag = "issues", - security(("tailscale-user" = [])), - request_body = ListEventsArgs, - responses( - (status = 200, body = Page), - ), -)] -pub async fn list_events( - State(state): State, - _user: TailscaleUser, - Json(args): Json, -) -> Result>> { - let mut conn = state.db_read.get().await?; - let total = Event::count_for_issue(&mut conn, args.issue_id).await? as u64; - let events = Event::list_for_issue( - &mut conn, - args.issue_id, - args.offset.unwrap_or(0), - args.limit.unwrap_or(DEFAULT_LIMIT), - ) - .await?; - let items = events.into_iter().map(EventData::from).collect(); - Ok(Json(Page { items, total })) -} - -/// A manually entered event to record against a server. +/// A manually raised condition to record against a server. #[derive(Deserialize, ToSchema)] #[serde(rename_all = "camelCase")] pub struct SubmitManualEventArgs { - /// Id of the server the event applies to. + /// Id of the server the condition applies to. pub server_id: Uuid, - /// Identifier for the underlying condition. Events with the same `ref` - /// on the same server are coalesced into the same issue rather than - /// opening a new one each time; use a fresh unique value if that - /// deduplication isn't wanted. + /// Identifier for the underlying condition. Reports with the same + /// `ref` on the same server update the same issue rather than opening + /// a new one each time; use a fresh unique value if that deduplication + /// isn't wanted. #[serde(rename = "ref")] pub r#ref: String, - /// Severity to record. If omitted, a default severity is used. + /// The condition's result: `failed` (can open an incident) or + /// `warning` (context only). Defaults to `failed`. `passed` records + /// the condition as cleared, same as `active: false`. + #[serde(default)] + pub result: Option, + /// Whether the condition's failures should notify immediately, + /// bypassing the incident grace period. Only consulted the first time + /// a `ref` is seen (it seeds the condition's catalog entry); adjust + /// later from the healthchecks catalog. #[serde(default)] - pub severity: Option, - /// Short, single-line headline for the event. Must not contain + pub escalates: Option, + /// Short, single-line headline for the condition. Must not contain /// newlines — use `message` for multi-line detail. #[serde(default)] pub description: Option, - /// Human-readable message describing the event. May be multi-line. + /// Human-readable message describing the condition. May be multi-line. pub message: String, /// Whether the underlying condition is currently active. Defaults to - /// `true` when omitted. + /// `true` when omitted; `false` records it as cleared regardless of + /// `result`. #[serde(default)] pub active: Option, - /// When the underlying condition actually occurred, if different from - /// the time of submission. Defaults to now when omitted. - #[serde(default)] - pub occurred_at: Option, } -/// Manually record an event against a server, creating or updating an issue. +/// Manually raise (or clear) a condition against a server. /// -/// Finds or creates an issue keyed by the server and the given `ref`, -/// appends this event to it, and returns the resulting issue. Returns 400 -/// if `ref` is empty or if `description` contains a newline. +/// Finds or creates an issue keyed by the server and the given `ref` +/// under the `manual` source, grades the chosen result through the +/// condition's catalog policy, and returns the resulting issue. Returns +/// 400 if `ref` is empty or if `description` contains a newline. #[utoipa::path( post, path = "/submit_manual_event", @@ -577,17 +507,33 @@ pub async fn submit_manual_event( return Err(AppError::custom("ref is required")); } - let event = NewEvent { - source: "manual".to_string(), - r#ref: args.r#ref, - severity: args.severity, - description: args.description, - message: args.message, - active: args.active, - occurred_at: args.occurred_at, + let observed = if args.active == Some(false) { + CheckResult::Passed + } else { + args.result.unwrap_or(CheckResult::Failed) }; let mut conn = state.db.get().await?; - let issue = event.save(&mut conn, args.server_id, None).await?; + let issue = file_check( + &mut conn, + CheckFiling { + source: MANUAL_SOURCE, + scope: FilingScope::Server { + server_id: args.server_id, + device_id: None, + }, + check: &args.r#ref, + observed, + title: args.description.as_deref(), + message: &args.message, + detail: None, + // The operator's chosen result passes through ungraded; the + // escalation choice seeds the catalog entry on first sight. + default_ceiling: CheckResult::Failed, + default_escalates: args.escalates.unwrap_or(false), + documentation: None, + }, + ) + .await?; Ok(Json(enrich_issue(&mut conn, issue).await?)) } diff --git a/crates/private-server/src/fns/self_alerts.rs b/crates/private-server/src/fns/self_alerts.rs index 0d43678e..dacce3d1 100644 --- a/crates/private-server/src/fns/self_alerts.rs +++ b/crates/private-server/src/fns/self_alerts.rs @@ -9,7 +9,8 @@ use canopy_utoipa_axum::{router::OpenApiRouter, routes}; use commons_errors::{ProblemDetailsSchema, Result}; use commons_servers::tailscale_auth::{TailscaleAdmin, TailscaleUser}; use commons_types::Uuid; -use commons_types::issue::{ResolvedReason, Severity}; +use commons_types::issue::ResolvedReason; +use commons_types::status::CheckResult; use database::issues::Issue; use jiff::Timestamp; use serde::{Deserialize, Serialize}; @@ -35,9 +36,15 @@ pub struct SelfAlertView { /// `mcp-token-expiry`. Stable across repeated raises of the same /// condition. pub r#ref: String, - /// How severe the condition is, from `critical` (most severe) down to - /// `debug` (least). - pub severity: Severity, + /// What canopy observed on the latest raise, before policy. + #[schema(value_type = Option)] + pub observed_result: Option, + /// What policy made of it — the result canopy acts on. + #[schema(value_type = Option)] + pub effective_result: Option, + /// Whether this condition's policy escalates: an effective failure + /// notifies immediately, bypassing incident grace. + pub escalates: bool, /// Single-line headline. pub title: Option, /// Full detail message describing the condition. @@ -63,7 +70,9 @@ impl From for SelfAlertView { Self { id: i.id, r#ref: i.r#ref, - severity: i.severity, + observed_result: i.observed_result, + effective_result: i.effective_result, + escalates: i.escalates, title: i.description, message: i.message, active: i.active, @@ -145,9 +154,7 @@ pub struct SelfAlertsResolveArgs { /// Mark a self-alert as resolved. /// -/// Records that an operator has resolved the given self-alert and cancels -/// its notification if one is still pending delivery (e.g. inside the -/// initial grace period before a low-severity alert is sent). Returns 404 +/// Records that an operator has resolved the given self-alert. Returns 404 /// if no self-alert with that id exists. #[utoipa::path( post, @@ -157,7 +164,7 @@ pub struct SelfAlertsResolveArgs { security(("tailscale-admin" = [])), request_body = SelfAlertsResolveArgs, responses( - (status = 200, description = "Alert marked operator-resolved; a pending notification is cancelled."), + (status = 200, description = "Alert marked operator-resolved."), (status = 401, body = ProblemDetailsSchema), (status = 403, body = ProblemDetailsSchema), (status = 404, body = ProblemDetailsSchema), @@ -170,13 +177,5 @@ pub async fn resolve( ) -> Result> { let mut conn = state.db.get().await?; Issue::resolve(&mut conn, args.id, &admin.login, ResolvedReason::Fixed).await?; - // If the alert's Slack open is still inside its grace window, the - // operator has dealt with it before anyone needed paging. - database::slack_outbox::SlackOutbox::cancel_pending_self_alert_open( - &mut conn, - args.id, - "cancelled: self-alert operator-resolved before the open had been delivered to Slack", - ) - .await?; Ok(Json(())) } diff --git a/crates/private-server/src/fns/servers.rs b/crates/private-server/src/fns/servers.rs index 78bb135a..b5802b5f 100644 --- a/crates/private-server/src/fns/servers.rs +++ b/crates/private-server/src/fns/servers.rs @@ -95,10 +95,6 @@ pub struct ServerInfo { /// reachability sweep skips it and its issues don't contribute to /// incidents. pub is_monitored: bool, - /// Whether this server may use the retired legacy `/status` format (a - /// push with no `health` array). Off by default; when on, such a push - /// only refreshes reachability and carries prior healthchecks forward. - pub allow_legacy_status: bool, /// Threshold in seconds for the reachability sweep to consider this /// server down. Always positive; only consulted when `is_monitored` /// is `true`. The default at creation is 600 (10 minutes). @@ -155,6 +151,9 @@ pub struct ServerLastStatusData { /// Per-check health breakdown from this push. Empty for reports /// predating structured per-check health. pub health: JsonValue, + /// The source that pushed this status (e.g. `alertd`). Silences on + /// the checks it carries are keyed by this source. + pub source: String, /// Additional endpoint-defined data included with this status push. pub extra: JsonValue, /// Operators identified as connected to the server as of this push, @@ -226,10 +225,6 @@ pub struct ServerDataUpdate { /// unchanged. #[serde(skip_serializing_if = "Option::is_none")] pub is_monitored: Option, - /// Whether to accept the retired legacy status format from this - /// server. Omit to leave unchanged. - #[serde(skip_serializing_if = "Option::is_none")] - pub allow_legacy_status: Option, /// New downtime threshold in seconds before this server is considered /// down. Omit to leave unchanged. #[serde(skip_serializing_if = "Option::is_none")] @@ -268,7 +263,6 @@ pub(super) fn server_to_info(s: Server) -> ServerInfo { cloud: s.cloud, geolocation: s.geolocation, is_monitored: s.is_monitored, - allow_legacy_status: s.allow_legacy_status, alert_when_down_for: s.alert_when_down_for.0.as_secs(), notes: s.notes, tags: s.tags, @@ -294,20 +288,15 @@ pub(super) async fn decorate_with_status( let statuses = Status::latest_for_servers(conn, &ids).await?; let by_server: std::collections::HashMap = statuses.iter().map(|s| (s.server_id, s)).collect(); - // Operator-silenced healthchecks don't count toward the health dot. + // Health comes from current check state across every source + // (silenced checks already skipped in the rollup). let server_groups: Vec<(Uuid, Option)> = infos.iter().map(|i| (i.id, i.group_id)).collect(); - let silenced = - database::silenced_refs::silenced_health_checks_for_servers(conn, &server_groups).await?; - let no_silences = std::collections::BTreeSet::new(); + let health = database::issues::health_from_check_state(conn, &server_groups).await?; for info in infos.iter_mut() { let st = by_server.get(&info.id).copied(); - let silenced_checks = silenced.get(&info.id).unwrap_or(&no_silences); info.up = Some(st.map(|s| s.short_status()).unwrap_or_default()); - info.health = Some( - st.map(|s| s.health_state_ignoring(silenced_checks)) - .unwrap_or_default(), - ); + info.health = Some(health.get(&info.id).copied().unwrap_or_default()); } Ok(()) } @@ -604,18 +593,15 @@ pub async fn get_detail( .as_ref() .map(|s| s.short_status()) .unwrap_or_default(); - // Operator-silenced healthchecks don't count toward the headline - // health chip; the checks table still shows them (as skipped). - let silenced_checks = database::silenced_refs::silenced_health_checks_for_server( - &mut conn, - server.id, - server.group_id, - ) - .await?; - let health = status - .as_ref() - .map(|s| s.health_state_ignoring(&silenced_checks)) - .unwrap_or_default(); + // The headline health chip rolls up current check state across every + // source; silenced checks are already skipped in the rollup, while + // the checks table still shows them (as skipped). + let health = + database::issues::health_from_check_state(&mut conn, &[(server.id, server.group_id)]) + .await? + .get(&server.id) + .copied() + .unwrap_or_default(); let device_with_info = if let Some(device_id) = device_id { Some(Device::get_with_info(&mut conn, device_id).await?) @@ -670,6 +656,7 @@ pub async fn get_detail( .and_then(|s| s.as_str().map(|s| s.to_string())), healthy: st.healthy, health: st.health.clone(), + source: st.source.clone(), extra: st.extra.clone(), operators, }) @@ -782,7 +769,6 @@ pub async fn update( cloud: args.data.cloud, geolocation: args.data.geolocation, is_monitored: args.data.is_monitored, - allow_legacy_status: args.data.allow_legacy_status, alert_when_down_for: args .data .alert_when_down_for @@ -925,7 +911,6 @@ pub async fn create( cloud: args.cloud, geolocation: args.geolocation, is_monitored: args.is_monitored.unwrap_or(true), - allow_legacy_status: false, alert_when_down_for: PgDuration(jiff::SignedDuration::from_secs( args.alert_when_down_for.unwrap_or(DEFAULT_ALERT_SECS), )), @@ -1252,7 +1237,6 @@ pub async fn attach_tailscale_device( cloud: None, geolocation: None, is_monitored: None, - allow_legacy_status: None, alert_when_down_for: None, notes: None, tags: None, diff --git a/crates/private-server/src/fns/statuses.rs b/crates/private-server/src/fns/statuses.rs index 66a40a9b..2ebbc583 100644 --- a/crates/private-server/src/fns/statuses.rs +++ b/crates/private-server/src/fns/statuses.rs @@ -6,7 +6,6 @@ use canopy_utoipa_axum::{router::OpenApiRouter, routes}; use commons_errors::{ProblemDetailsSchema, Result}; use commons_servers::tailscale_auth::TailscaleUser; use commons_types::{ - issue::Severity, server::{ cards::{FacilityServerStatus, ServerGroupCard}, rank::ServerRank, @@ -15,7 +14,7 @@ use commons_types::{ version::VersionStr, }; use database::{ - devices::DeviceConnection, healthcheck_severities::HealthcheckSeverity, issues::Issue, + check_policies::CheckPolicy, devices::DeviceConnection, issues::Issue, server_groups::ServerGroup, servers::Server, statuses::Status, tailscale_users::TailscaleUser as CachedTailscaleUser, versions::Version, }; @@ -220,13 +219,12 @@ pub async fn group_details( .into_iter() .map(|s| (s.server_id, s)) .collect(); - // Operator-silenced healthchecks don't count toward member health. + // Member health rolls up current check state across every source + // (silenced checks already skipped in the rollup). let member_groups: Vec<(Uuid, Option)> = servers.iter().map(|s| (s.id, s.group_id)).collect(); - let silenced = - database::silenced_refs::silenced_health_checks_for_servers(&mut conn, &member_groups) - .await?; - let no_silences = BTreeSet::new(); + let member_health = + database::issues::health_from_check_state(&mut conn, &member_groups).await?; // The card's headline version is the cached last reported version of the // group's canonical member (highest rank, then highest kind), maintained by @@ -251,14 +249,11 @@ pub async fn group_details( } _ => Vec::new(), }; - let silenced_checks = silenced.get(&s.id).unwrap_or(&no_silences); FacilityServerStatus { id: s.id, name: s.name.clone().unwrap_or_default(), up, - health: st - .map(|s| s.health_state_ignoring(silenced_checks)) - .unwrap_or_default(), + health: member_health.get(&s.id).copied().unwrap_or_default(), operators, rank: s.rank, kind: s.kind, @@ -294,49 +289,56 @@ pub struct CheckAttentionServerData { pub group_id: Option, /// The server's group name, if it belongs to one. pub group_name: Option, - /// The check's result on this server's latest status. The UI shows + /// The check's observed result on its latest report. The UI shows /// warning/failed/broken servers by default and puts passed/skipped /// ones behind a "show healthy" toggle. pub result: CheckResult, - /// The check's full `health[]` entry from this server's latest status, - /// verbatim (including the `check`/`healthy`/`result` keys), so the + /// The check's own fields from its latest report, verbatim, so the /// row can expand to the same per-check detail the server page shows. pub data: serde_json::Value, - /// When this check started failing on this server: the `first_seen` - /// of the still-active issue canopy filed at `(status, - /// health/)` when the check degraded. `None` for servers - /// currently reporting the check healthy, and for failing servers - /// with no active issue on file (e.g. the issue was - /// operator-resolved, or the ref is silenced so nothing was filed). + /// When the check's current degradation streak began. `None` for + /// servers currently reporting the check healthy. pub failing_since: Option, - /// When the reporting status was recorded. + /// When the check state last updated (the check's latest report). pub status_created_at: Timestamp, } /// Request body for [`check_attention`]. #[derive(Debug, Deserialize, ToSchema)] pub struct CheckAttentionArgs { + /// The source that reports the check. A check's identity is the + /// (source, check) pair — a same-named check from another source is + /// a different check. + pub source: String, /// The healthcheck name to look up, exactly as reported by devices in /// `health[].check` (an arbitrary, device/plugin-defined string). pub check: String, } -/// Response for [`check_attention`]: the queried check's catalog severity +/// Response for [`check_attention`]: the queried check's catalog policy /// (if it has one yet) and every live server whose latest status reports /// it, failing or healthy. #[derive(Debug, Clone, Serialize, Deserialize, ToSchema)] pub struct CheckAttentionData { - /// The check name that was queried, echoed back so the page can - /// render its heading without re-decoding the request. + /// The source that was queried, echoed back with `check` so the page + /// can render its heading without re-decoding the request. + pub source: String, + /// The check name that was queried. pub check: String, - /// The catalog's configured base severity for this check, or `None` - /// if no server has ever reported it yet (so it has no catalog row). - pub severity: Option, - /// Every live server whose latest status reports this check, at any - /// result, ordered as a TODO list: failed, warning, broken, passed, - /// skipped (most urgent first), then by group name then server name. - /// The client filters out the passed/skipped tail unless the "show - /// healthy" toggle is on. + /// The configured policy ceiling for this (source, check), or `None` + /// if the source has never reported it (so it has no catalog row). + #[schema(value_type = Option)] + pub ceiling: Option, + /// Whether this check's policy escalates its effective failures. + pub escalates: bool, + /// Operator-authored documentation for this (source, check) + /// (markdown), or `None` if nobody has written it yet. + pub documentation: Option, + /// Every live server whose latest state from this source reports + /// this check, at any result, ordered as a TODO list: failed, + /// warning, broken, passed, skipped (most urgent first), then by + /// group name then server name. The client filters out the + /// passed/skipped tail unless the "show healthy" toggle is on. pub servers: Vec, } @@ -354,16 +356,16 @@ fn check_result_rank(result: CheckResult) -> u8 { } } -/// List the servers whose latest status reports one named healthcheck. +/// List the servers whose check state reports one (source, check). /// /// Everything the per-healthcheck page needs: the catalog's configured -/// severity for `check` (if any) plus every live server whose **latest** -/// status reports it — the current, real-time picture, not a history of -/// past issues/events, though each failing server carries a -/// `failing_since` timestamp derived from its active issue. This is the -/// data behind the `/healthchecks/:check` "who's affected" page, which -/// doubles as an operator TODO list and as a way to correlate servers -/// sharing the same issue during a fleet-wide incident. +/// policy for the (source, check) (if any) plus every live server's +/// current state for it — the real-time picture, with each degraded row +/// carrying `failing_since` (the start of its current degradation +/// streak). This is the data behind the `/healthchecks/:source/:check` +/// "who's affected" page, which doubles as an operator TODO list and as +/// a way to correlate servers sharing the same issue during a +/// fleet-wide incident. #[utoipa::path( post, path = "/check_attention", @@ -371,7 +373,7 @@ fn check_result_rank(result: CheckResult) -> u8 { tag = "statuses", request_body = CheckAttentionArgs, responses( - (status = 200, description = "The check's catalog severity and the servers currently reporting it.", body = CheckAttentionData), + (status = 200, description = "The check's catalog policy and the servers currently reporting it.", body = CheckAttentionData), (status = 500, body = ProblemDetailsSchema), ), )] @@ -380,11 +382,26 @@ pub async fn check_attention( Json(args): Json, ) -> Result> { let mut conn = state.db_read.get().await?; - let reporting = Status::reporting_check_with_servers(&mut conn, &args.check).await?; + let states = Issue::check_state_for_check(&mut conn, &args.source, &args.check).await?; - let group_ids: Vec = reporting + // Live servers only: archived servers and canopy's own row never + // appear on the attention list. + let server_ids: Vec = states .iter() - .filter_map(|(s, _)| s.group_id) + .filter_map(|st| st.server_id) + .collect::>() + .into_iter() + .collect(); + let live: HashMap = + database::servers::Server::get_by_ids(&mut conn, &server_ids) + .await? + .into_iter() + .filter(|s| s.deleted_at.is_none() && s.id != Uuid::nil()) + .map(|s| (s.id, s)) + .collect(); + let group_ids: Vec = live + .values() + .filter_map(|s| s.group_id) .collect::>() .into_iter() .collect(); @@ -394,38 +411,27 @@ pub async fn check_attention( .map(|g| (g.id, g.name)) .collect(); - // "Failing since" comes from the issue the public-server status - // handler files at `(status, health/)` when a check degrades: - // an active issue's first_seen is exactly when the current failure - // streak began (recoveries close the issue, so a re-failure starts a - // fresh one). - let server_ids: Vec = reporting.iter().map(|(s, _)| s.id).collect(); - let failing_since: HashMap = Issue::list_by_source_ref( - &mut conn, - "status", - &format!("health/{}", args.check), - &server_ids, - ) - .await? - .into_iter() - .filter(|issue| issue.active) - .filter_map(|issue| issue.server_id.map(|sid| (sid, issue.first_seen))) - .collect(); - - let mut servers: Vec = reporting + let mut servers: Vec = states .into_iter() - .filter_map(|(server, status)| { - let (result, entry) = status.check_entry(&args.check)?; + .filter_map(|st| { + let server = st.server_id.and_then(|sid| live.get(&sid))?; + let result = st.observed_result?; let group_name = server.group_id.and_then(|g| group_names.get(&g).cloned()); + // failing_since is the current degradation streak; recovered + // rows have none. first_seen is the fallback for rows stamped + // before degraded_since existed. + let failing_since = st + .active + .then_some(st.degraded_since.unwrap_or(st.first_seen)); Some(CheckAttentionServerData { server_id: server.id, server_name: server.name.clone().unwrap_or_default(), group_id: server.group_id, group_name, result, - data: serde_json::Value::Object(entry), - failing_since: failing_since.get(&server.id).copied(), - status_created_at: status.created_at, + data: st.detail.unwrap_or_else(|| serde_json::json!({})), + failing_since, + status_created_at: st.last_seen, }) }) .collect(); @@ -436,13 +442,14 @@ pub async fn check_attention( .then_with(|| a.server_name.cmp(&b.server_name)) }); - let severity = HealthcheckSeverity::get(&mut conn, &args.check) - .await? - .map(|row| row.severity); + let policy = CheckPolicy::get(&mut conn, &args.source, &args.check).await?; Ok(Json(CheckAttentionData { + source: args.source, check: args.check, - severity, + ceiling: policy.as_ref().map(|p| p.ceiling), + escalates: policy.as_ref().is_some_and(|p| p.escalates), + documentation: policy.and_then(|p| p.documentation), servers, })) } @@ -493,11 +500,11 @@ pub struct StatusSnapshotData { /// with display name and profile picture filled in where known. Not /// filtered by recency — reflects this specific point-in-time snapshot. pub operators: Vec, - /// For each currently-unhealthy check in this push, the severity it - /// would be filed at if it turned into an issue. Healthy checks are - /// omitted. An unhealthy check with no severity listed here should be - /// treated as a default (warning-level) severity. - pub check_severities: std::collections::HashMap, + /// For each currently-unhealthy check in this push, the effective + /// result its policy grades it to. Healthy checks are omitted. An + /// unhealthy check not listed here should be treated as warning. + #[schema(value_type = std::collections::HashMap)] + pub check_results: std::collections::HashMap, /// Check names currently silenced for this server (at server or /// group scope). These don't count toward `health_state` and the UI /// renders them with its skipped affordance. Reflects the silence @@ -590,14 +597,17 @@ pub async fn snapshot( // Compute the per-unhealthy-check severity the rules engine would // file at given this push. Healthy checks are omitted; the UI // renders them with its 'passing' affordance regardless. - let check_severities = compute_check_severities(&mut conn, &server, &status).await?; + let check_results = compute_check_results(&mut conn, &server, &status).await?; // Operator-silenced healthchecks present as skipped and don't count // toward the rollup, even on historical snapshots — a silence // expresses current operator intent about the check, not the push. + // Scoped to the push's own source: a silence on another source's + // same-named check is about a different check. let silenced_checks = database::silenced_refs::silenced_health_checks_for_server( &mut conn, server.id, server.group_id, + &status.source, ) .await?; let health_state = status.health_state_ignoring(&silenced_checks); @@ -621,7 +631,7 @@ pub async fn snapshot( health: status.health, extra: status.extra, operators, - check_severities, + check_results, silenced_checks, }))) } @@ -655,20 +665,19 @@ pub(crate) async fn enrich_operators( Ok(()) } -/// For every warning/failed check on `status`, resolve the catalog + -/// rules severity given the snapshot's actual extras and the server's -/// resolved tag map. Mirrors the public-server ingestion path -/// (`file_health_events`) so the UI displays what *would* be filed. -/// Broken checks aren't included — they file at a fixed Warning and -/// the UI renders them from the result directly; passed/skipped file -/// nothing. -async fn compute_check_severities( +/// For every warning/failed check on `status`, resolve the policy + +/// rules grading given the snapshot's actual extras and the server's +/// resolved tag map: the effective result ingestion would file. Mirrors +/// the public-server ingestion path (`file_health_events`) so the UI +/// displays what *would* be filed. Broken checks aren't included — they +/// file as warnings and the UI renders them from the result directly. +async fn compute_check_results( conn: &mut database::diesel_async::AsyncPgConnection, server: &Server, status: &Status, -) -> commons_errors::Result> { +) -> commons_errors::Result> { use commons_types::status::CheckResult; - use database::healthcheck_severities::{EvaluationContext, HealthcheckSeverity}; + use database::check_policies::{CheckPolicy, EvaluationContext}; let Some(arr) = status.health.as_array() else { return Ok(Default::default()); @@ -723,8 +732,20 @@ async fn compute_check_severities( check_extra: &check_extra, tags: &tags, }; - let sev = HealthcheckSeverity::severity_for(conn, &name, result, &ctx).await?; - out.insert(name, sev); + let graded = CheckPolicy::apply_scoped( + conn, + &status.source, + &name, + result, + &ctx, + Some(server.id), + server.group_id, + ) + .await?; + match graded.effective { + CheckResult::Passed | CheckResult::Skipped => continue, + effective => out.insert(name, effective), + }; } Ok(out) } diff --git a/crates/private-server/tests/it/healthchecks.rs b/crates/private-server/tests/it/healthchecks.rs index 75e90162..1f590bec 100644 --- a/crates/private-server/tests/it/healthchecks.rs +++ b/crates/private-server/tests/it/healthchecks.rs @@ -7,10 +7,10 @@ use serde_json::json; async fn list_returns_catalog_rows_with_pending_review_flag() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name, severity) VALUES \ - ('disk_space', 'warning'), \ - ('reviewed_check', 'error'); \ - UPDATE healthcheck_severities \ + "INSERT INTO check_policies (source, check_name, ceiling) VALUES \ + ('alertd', 'disk_space', 'warning'), \ + ('alertd', 'reviewed_check', 'failed'); \ + UPDATE check_policies \ SET reviewed_at = NOW(), reviewed_by = 'alice@example.com' \ WHERE check_name = 'reviewed_check';", ) @@ -26,14 +26,15 @@ async fn list_returns_catalog_rows_with_pending_review_flag() { let body: Vec = response.json(); assert_eq!(body.len(), 2); - // Ordered by check_name. + // Ordered by source then check_name. + assert_eq!(body[0]["source"], "alertd"); assert_eq!(body[0]["check_name"], "disk_space"); - assert_eq!(body[0]["severity"], "warning"); + assert_eq!(body[0]["ceiling"], "warning"); assert_eq!(body[0]["pending_review"], true); assert!(body[0]["reviewed_at"].is_null()); assert_eq!(body[1]["check_name"], "reviewed_check"); - assert_eq!(body[1]["severity"], "error"); + assert_eq!(body[1]["ceiling"], "failed"); assert_eq!(body[1]["pending_review"], false); assert_eq!(body[1]["reviewed_by"], "alice@example.com"); }) @@ -41,10 +42,10 @@ async fn list_returns_catalog_rows_with_pending_review_flag() { } #[tokio::test(flavor = "multi_thread")] -async fn update_changes_severity_and_stamps_review_metadata() { +async fn update_changes_policy_and_stamps_review_metadata() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name) VALUES ('disk_space');", + "INSERT INTO check_policies (source, check_name) VALUES ('alertd', 'disk_space');", ) .await .unwrap(); @@ -52,15 +53,18 @@ async fn update_changes_severity_and_stamps_review_metadata() { let response = private .post("/api/healthchecks/update") .json(&json!({ + "source": "alertd", "check_name": "disk_space", - "severity": "error" + "ceiling": "failed", + "escalates": true })) .await; response.assert_status_ok(); let body: serde_json::Value = response.json(); assert_eq!(body["check_name"], "disk_space"); - assert_eq!(body["severity"], "error"); + assert_eq!(body["ceiling"], "failed"); + assert_eq!(body["escalates"], true); assert_eq!(body["pending_review"], false); assert!( !body["reviewed_at"].is_null(), @@ -73,10 +77,10 @@ async fn update_changes_severity_and_stamps_review_metadata() { } #[tokio::test(flavor = "multi_thread")] -async fn update_rejects_unknown_severity() { +async fn update_rejects_unknown_ceiling() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name) VALUES ('disk_space');", + "INSERT INTO check_policies (source, check_name) VALUES ('alertd', 'disk_space');", ) .await .unwrap(); @@ -84,8 +88,9 @@ async fn update_rejects_unknown_severity() { let response = private .post("/api/healthchecks/update") .json(&json!({ + "source": "alertd", "check_name": "disk_space", - "severity": "extremely_critical" + "ceiling": "extremely_critical" })) .await; response.assert_status_not_ok(); @@ -94,20 +99,21 @@ async fn update_rejects_unknown_severity() { } #[tokio::test(flavor = "multi_thread")] -async fn update_marks_reviewed_even_when_severity_unchanged() { +async fn update_marks_reviewed_even_when_policy_unchanged() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name) VALUES ('noisy_check');", + "INSERT INTO check_policies (source, check_name) VALUES ('alertd', 'noisy_check');", ) .await .unwrap(); - // Pass the same default severity to ack without changing. + // Pass the same default policy to ack without changing. let response = private .post("/api/healthchecks/update") .json(&json!({ + "source": "alertd", "check_name": "noisy_check", - "severity": "warning" + "ceiling": "warning" })) .await; response.assert_status_ok(); @@ -126,11 +132,11 @@ async fn update_marks_reviewed_even_when_severity_unchanged() { async fn list_returns_rules_and_rule_count() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name, rules) VALUES \ - ('no_rules', NULL), \ - ('one_rule', '{\"if\": [{\"==\": [{\"var\": \"check.x\"}, 1]}, \"error\"]}'::jsonb), \ - ('two_rules', '{\"if\": [{\"==\": [{\"var\": \"check.x\"}, 1]}, \"error\", {\"==\": [{\"var\": \"check.x\"}, 2]}, \"warning\"]}'::jsonb), \ - ('garbage_rules', '{\"and\": [true]}'::jsonb);", + "INSERT INTO check_policies (source, check_name, rules) VALUES \ + ('alertd', 'no_rules', NULL), \ + ('alertd', 'one_rule', '{\"if\": [{\"==\": [{\"var\": \"check.x\"}, 1]}, \"failed\"]}'::jsonb), \ + ('alertd', 'two_rules', '{\"if\": [{\"==\": [{\"var\": \"check.x\"}, 1]}, \"failed\", {\"==\": [{\"var\": \"check.x\"}, 2]}, \"warning\"]}'::jsonb), \ + ('alertd', 'garbage_rules', '{\"and\": [true]}'::jsonb);", ) .await .unwrap(); @@ -165,7 +171,7 @@ async fn list_returns_rules_and_rule_count() { async fn update_rules_accepts_valid_ladder() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name) VALUES ('disk_space');", + "INSERT INTO check_policies (source, check_name) VALUES ('alertd', 'disk_space');", ) .await .unwrap(); @@ -173,9 +179,10 @@ async fn update_rules_accepts_valid_ladder() { let response = private .post("/api/healthchecks/update_rules") .json(&json!({ + "source": "alertd", "check_name": "disk_space", "rules": {"if": [ - {">": [{"var": "check.used_pct"}, 95]}, "critical" + {">": [{"var": "check.used_pct"}, 95]}, "failed" ]} })) .await; @@ -193,15 +200,15 @@ async fn update_rules_accepts_valid_ladder() { async fn update_rules_with_null_clears_the_column() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name, rules) VALUES \ - ('disk_space', '{\"if\": [{\"==\": [{\"var\": \"check.x\"}, 1]}, \"error\"]}'::jsonb);", + "INSERT INTO check_policies (source, check_name, rules) VALUES \ + ('alertd', 'disk_space', '{\"if\": [{\"==\": [{\"var\": \"check.x\"}, 1]}, \"failed\"]}'::jsonb);", ) .await .unwrap(); let response = private .post("/api/healthchecks/update_rules") - .json(&json!({"check_name": "disk_space", "rules": null})) + .json(&json!({"source": "alertd", "check_name": "disk_space", "rules": null})) .await; response.assert_status_ok(); let body: serde_json::Value = response.json(); @@ -215,7 +222,7 @@ async fn update_rules_with_null_clears_the_column() { async fn update_rules_normalises_empty_ladder_to_null() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name) VALUES ('disk_space');", + "INSERT INTO check_policies (source, check_name) VALUES ('alertd', 'disk_space');", ) .await .unwrap(); @@ -223,7 +230,7 @@ async fn update_rules_normalises_empty_ladder_to_null() { // normalise it to null at write time. let response = private .post("/api/healthchecks/update_rules") - .json(&json!({"check_name": "disk_space", "rules": {"if": []}})) + .json(&json!({"source": "alertd", "check_name": "disk_space", "rules": {"if": []}})) .await; // Either the API rejects an empty ladder OR normalises it. Both // land at `rule_count == 0` and a null rules column. @@ -242,20 +249,20 @@ async fn update_rules_normalises_empty_ladder_to_null() { async fn update_rules_rejects_malformed_shapes() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name) VALUES ('disk_space');", + "INSERT INTO check_policies (source, check_name) VALUES ('alertd', 'disk_space');", ) .await .unwrap(); let cases: &[(serde_json::Value, &str)] = &[ (json!({"and": [true]}), "AND composition"), - (json!({"if": [{"if": [true, true]}, "error"]}), "nested if"), + (json!({"if": [{"if": [true, true]}, "failed"]}), "nested if"), ( - json!({"if": [{"==": [{"var": "BAD.x"}, 1]}, "error"]}), + json!({"if": [{"==": [{"var": "BAD.x"}, 1]}, "failed"]}), "unknown var namespace", ), ( - json!({"if": [{"==": [{"var": "check.x"}, 1]}, "not_a_severity"]}), - "bad severity", + json!({"if": [{"==": [{"var": "check.x"}, 1]}, "not_a_result"]}), + "bad result", ), ( json!({"if": [{"in_range": [{"var": "status.v"}, "not-a-range"]}, "warning"]}), @@ -265,7 +272,7 @@ async fn update_rules_rejects_malformed_shapes() { for (rules, label) in cases { let response = private .post("/api/healthchecks/update_rules") - .json(&json!({"check_name": "disk_space", "rules": rules})) + .json(&json!({"source": "alertd", "check_name": "disk_space", "rules": rules})) .await; assert!( !response.status_code().is_success(), @@ -282,13 +289,13 @@ async fn update_rules_rejects_malformed_shapes() { async fn sample_returns_null_when_no_server_has_reported_the_check() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name) VALUES ('uncharted_check');", + "INSERT INTO check_policies (source, check_name) VALUES ('alertd', 'uncharted_check');", ) .await .unwrap(); let response = private .post("/api/healthchecks/sample") - .json(&json!({"check_name": "uncharted_check"})) + .json(&json!({"source": "alertd", "check_name": "uncharted_check"})) .await; response.assert_status_ok(); let body: serde_json::Value = response.json(); @@ -318,7 +325,7 @@ async fn sample_materialises_latest_push_for_this_check() { let response = private .post("/api/healthchecks/sample") - .json(&json!({"check_name": "disk_space"})) + .json(&json!({"source": "alertd", "check_name": "disk_space"})) .await; response.assert_status_ok(); let body: serde_json::Value = response.json(); @@ -371,7 +378,7 @@ async fn sample_normalises_result_form_entries() { let response = private .post("/api/healthchecks/sample") - .json(&json!({"check_name": "queue_depth"})) + .json(&json!({"source": "alertd", "check_name": "queue_depth"})) .await; response.assert_status_ok(); let body: serde_json::Value = response.json(); @@ -402,7 +409,7 @@ async fn sample_picks_the_most_recent_push_across_servers() { let response = private .post("/api/healthchecks/sample") - .json(&json!({"check_name": "cert_expiry"})) + .json(&json!({"source": "alertd", "check_name": "cert_expiry"})) .await; response.assert_status_ok(); let body: serde_json::Value = response.json(); diff --git a/crates/private-server/tests/it/issues.rs b/crates/private-server/tests/it/issues.rs index 5197bc46..8b761a3d 100644 --- a/crates/private-server/tests/it/issues.rs +++ b/crates/private-server/tests/it/issues.rs @@ -12,10 +12,10 @@ async fn list_issues_for_device_and_server() { ('{device_id}', '\\x6b6579'::bytea, 'k', true); INSERT INTO servers (id, host, kind, device_id) VALUES \ ('{server_id}', 'https://example.com', 'central', '{device_id}'); - INSERT INTO issues (server_id, device_id, source, \"ref\", severity, message, active, first_seen, last_seen) VALUES \ - ('{server_id}', '{device_id}', 'src', 'a', 'error', 'newest', true, '2026-05-03T10:00:00Z', '2026-05-03T10:00:00Z'), - ('{server_id}', '{device_id}', 'src', 'b', 'warning', 'older', true, '2026-05-01T10:00:00Z', '2026-05-01T10:00:00Z'), - ('{server_id}', '{device_id}', 'src', 'c', 'info', 'gone', false, '2026-05-02T10:00:00Z', '2026-05-02T10:00:00Z');" + INSERT INTO issues (server_id, device_id, source, \"ref\", check_name, observed_result, effective_result, message, active, first_seen, last_seen, last_degraded_at) VALUES \ + ('{server_id}', '{device_id}', 'src', 'a', 'a', 'failed', 'failed', 'newest', true, '2026-05-03T10:00:00Z', '2026-05-03T10:00:00Z', '2026-05-03T10:00:00Z'), + ('{server_id}', '{device_id}', 'src', 'b', 'b', 'warning', 'warning', 'older', true, '2026-05-01T10:00:00Z', '2026-05-01T10:00:00Z', '2026-05-01T10:00:00Z'), + ('{server_id}', '{device_id}', 'src', 'c', 'c', 'passed', 'passed', 'gone', false, '2026-05-02T10:00:00Z', '2026-05-02T10:00:00Z', '2026-05-02T10:00:00Z');" )) .await .expect("seed"); @@ -66,7 +66,14 @@ async fn manual_event_submit_creates_issue_without_device() { let body: serde_json::Value = resp.json(); assert_eq!(body.get("source").and_then(|v| v.as_str()), Some("manual")); assert!(body.get("device_id").map_or(true, |v| v.is_null())); - assert_eq!(body.get("severity").and_then(|v| v.as_str()), Some("error")); + assert_eq!( + body.get("observed_result").and_then(|v| v.as_str()), + Some("failed") + ); + assert_eq!( + body.get("effective_result").and_then(|v| v.as_str()), + Some("failed") + ); }) .await; } @@ -98,7 +105,7 @@ async fn incident_groups_at_server_group() { .json(&serde_json::json!({ "serverId": server_b_id, "ref": "x", - "severity": "error", + "result": "failed", "message": "trouble in B", })) .await; @@ -160,7 +167,7 @@ async fn ungrouped_server_event_skips_incident() { .json(&serde_json::json!({ "serverId": server_id, "ref": "x", - "severity": "error", + "result": "failed", "message": "no group yet", })) .await; @@ -200,7 +207,7 @@ async fn assigning_group_opens_pending_incident() { .json(&serde_json::json!({ "serverId": server_id, "ref": "stuck", - "severity": "error", + "result": "failed", "message": "waiting to be grouped", })) .await; @@ -258,7 +265,7 @@ async fn issue_reopen_keeps_identity_and_joins_new_incident() { .json(&serde_json::json!({ "serverId": server_id, "ref": "x", - "severity": "error", + "result": "failed", "message": "trouble", })) .await; @@ -277,7 +284,7 @@ async fn issue_reopen_keeps_identity_and_joins_new_incident() { .json(&serde_json::json!({ "serverId": server_id, "ref": "x", - "severity": "error", + "result": "failed", "active": false, "message": "ok", })) @@ -298,7 +305,8 @@ async fn issue_reopen_keeps_identity_and_joins_new_incident() { .json(&serde_json::json!({ "serverId": server_id, "ref": "x", - "severity": "critical", + "result": "failed", + "escalates": true, "message": "back", })) .await; @@ -349,7 +357,7 @@ async fn low_severity_issue_joins_existing_open_incident() { .json(&serde_json::json!({ "serverId": server_id, "ref": "a", - "severity": "error", + "result": "failed", "message": "primary trouble", })) .await @@ -362,7 +370,7 @@ async fn low_severity_issue_joins_existing_open_incident() { .json(&serde_json::json!({ "serverId": server_id, "ref": "b", - "severity": "warning", + "result": "warning", "message": "ride-along", })) .await @@ -405,7 +413,7 @@ async fn low_severity_alone_does_not_open_incident() { .json(&serde_json::json!({ "serverId": server_id, "ref": "b", - "severity": "warning", + "result": "warning", "message": "minor", })) .await @@ -443,7 +451,7 @@ async fn severity_downgrade_keeps_issue_in_incident() { .json(&serde_json::json!({ "serverId": server_id, "ref": "x", - "severity": "error", + "result": "failed", "message": "trouble", })) .await @@ -455,7 +463,7 @@ async fn severity_downgrade_keeps_issue_in_incident() { .json(&serde_json::json!({ "serverId": server_id, "ref": "x", - "severity": "warning", + "result": "warning", "message": "less bad now", })) .await @@ -474,79 +482,6 @@ async fn severity_downgrade_keeps_issue_in_incident() { .await; } -#[tokio::test(flavor = "multi_thread")] -async fn list_events_returns_event_log() { - commons_tests::server::run(async |mut conn, _public, private| { - let server_id = Uuid::new_v4(); - conn.batch_execute(&format!( - "INSERT INTO servers (id, host, kind) VALUES \ - ('{server_id}', 'https://example.com', 'central');" - )) - .await - .expect("seed"); - - // Three distinct events. - for (sev, msg) in [("error", "a"), ("error", "b"), ("warning", "b")] { - private - .post("/api/issues/submit_manual_event") - .json(&serde_json::json!({ - "serverId": server_id, - "ref": "x", - "severity": sev, - "message": msg, - })) - .await - .assert_status_ok(); - } - - // Find the issue. - let issues = private - .post("/api/issues/list_for_server") - .json(&serde_json::json!({ "server_id": server_id })) - .await; - issues.assert_status_ok(); - let items: Vec = issues.json(); - assert_eq!(items.len(), 1); - let issue_id = items[0].get("id").unwrap().as_str().unwrap(); - - let events = private - .post("/api/issues/list_events") - .json(&serde_json::json!({ "issue_id": issue_id })) - .await; - events.assert_status_ok(); - let page: serde_json::Value = events.json(); - let items = page.get("items").and_then(|v| v.as_array()).unwrap(); - assert_eq!(items.len(), 3, "each distinct push is its own event row"); - assert_eq!(page.get("total").and_then(|v| v.as_u64()), Some(3)); - - // Pagination: limit=2 returns 2 items but total still reflects 3. - let page1 = private - .post("/api/issues/list_events") - .json(&serde_json::json!({ "issue_id": issue_id, "offset": 0, "limit": 2 })) - .await; - page1.assert_status_ok(); - let page1: serde_json::Value = page1.json(); - assert_eq!( - page1.get("items").and_then(|v| v.as_array()).unwrap().len(), - 2 - ); - assert_eq!(page1.get("total").and_then(|v| v.as_u64()), Some(3)); - - let page2 = private - .post("/api/issues/list_events") - .json(&serde_json::json!({ "issue_id": issue_id, "offset": 2, "limit": 2 })) - .await; - page2.assert_status_ok(); - let page2: serde_json::Value = page2.json(); - assert_eq!( - page2.get("items").and_then(|v| v.as_array()).unwrap().len(), - 1 - ); - assert_eq!(page2.get("total").and_then(|v| v.as_u64()), Some(3)); - }) - .await; -} - async fn open_issue( conn: &mut database::diesel_async::AsyncPgConnection, private: &commons_tests::axum_test::TestServer, @@ -567,7 +502,7 @@ async fn open_issue( .json(&serde_json::json!({ "serverId": server_id, "ref": "x", - "severity": "error", + "result": "failed", "message": "trouble", })) .await; @@ -664,25 +599,28 @@ async fn reopen_via_device_clears_resolved_fields() { .await .expect("seed"); - // Device opens the issue. - let opened = public - .post("/events") + // Device opens the issue by pushing a failing check. + public + .post(&format!("/status/{server_id}")) .add_header("mtls-certificate", &cert) .json(&serde_json::json!({ - "source": "watchdog", - "ref": "x", - "severity": "error", - "message": "trouble", + "health": [ { "check": "x", "result": "failed" } ], })) - .await; - opened.assert_status_ok(); - let issue_id = opened - .json::() - .get("id") - .unwrap() - .as_str() - .unwrap() - .to_string(); + .await + .assert_status_ok(); + let issue = database::issues::Issue::list_by_source_ref( + &mut conn, + "alertd", + "health/x", + &[server_id], + ) + .await + .expect("list issues") + .into_iter() + .next() + .expect("issue opened by the failing check"); + assert!(issue.active); + let issue_id = issue.id.to_string(); // Human resolves. private @@ -692,24 +630,32 @@ async fn reopen_via_device_clears_resolved_fields() { .assert_status_ok(); // Device pushes again — should clear resolved_* (Sentry-style reopen). - let reopened = public - .post("/events") + public + .post(&format!("/status/{server_id}")) .add_header("mtls-certificate", &cert) .json(&serde_json::json!({ - "source": "watchdog", - "ref": "x", - "severity": "error", - "message": "back again", + "health": [ { "check": "x", "result": "failed" } ], })) - .await; - reopened.assert_status_ok(); - let body: serde_json::Value = reopened.json(); + .await + .assert_status_ok(); + let reopened = database::issues::Issue::list_by_source_ref( + &mut conn, + "alertd", + "health/x", + &[server_id], + ) + .await + .expect("list issues") + .into_iter() + .next() + .expect("issue still present after reopen"); + assert!(reopened.active); assert!( - body.get("resolved_at").map_or(true, |v| v.is_null()), + reopened.resolved_at.is_none(), "reopen should clear resolved_at" ); - assert!(body.get("resolved_by").map_or(true, |v| v.is_null())); - assert!(body.get("resolved_reason").map_or(true, |v| v.is_null())); + assert!(reopened.resolved_by.is_none()); + assert!(reopened.resolved_reason.is_none()); }, ) .await; @@ -749,7 +695,8 @@ async fn snooze_leaves_incident_and_blocks_rejoin() { .json(&serde_json::json!({ "serverId": server_id, "ref": "x", - "severity": "critical", + "result": "failed", + "escalates": true, "message": "still flapping", })) .await @@ -810,7 +757,7 @@ async fn unmonitored_server_event_does_not_open_incident() { .json(&serde_json::json!({ "serverId": server_id, "ref": "ignored", - "severity": "error", + "result": "failed", "message": "should not open an incident", })) .await; @@ -858,7 +805,7 @@ async fn enabling_monitoring_opens_pending_incident() { .json(&serde_json::json!({ "serverId": server_id, "ref": "stuck", - "severity": "error", + "result": "failed", "message": "waiting to be re-enabled", })) .await; @@ -951,7 +898,7 @@ async fn silencing_server_ref_closes_only_matching_open_incident() { .json(&serde_json::json!({ "serverId": server_id, "ref": "other", - "severity": "error", + "result": "failed", "message": "second contributor", })) .await; @@ -1074,7 +1021,7 @@ async fn group_silence_blocks_events_from_all_members() { .json(&serde_json::json!({ "serverId": sid, "ref": "noisy", - "severity": "error", + "result": "failed", "message": "should not fire", })) .await; @@ -1093,7 +1040,7 @@ async fn group_silence_blocks_events_from_all_members() { .json(&serde_json::json!({ "serverId": server_a, "ref": "other", - "severity": "error", + "result": "failed", "message": "should still fire", })) .await; diff --git a/crates/private-server/tests/it/mcp.rs b/crates/private-server/tests/it/mcp.rs index e02f9457..92625584 100644 --- a/crates/private-server/tests/it/mcp.rs +++ b/crates/private-server/tests/it/mcp.rs @@ -346,11 +346,9 @@ async fn seed_incidents(conn: &mut impl SimpleAsyncConnection) { "INSERT INTO server_groups (id, name) VALUES ('{IGROUP}', 'Inc Group'); \ INSERT INTO servers (id, host, name, kind, group_id, is_monitored) VALUES \ ('{ISRV}', 'https://inc', 'Inc Server', 'central', '{IGROUP}', true); \ - INSERT INTO issues (id, created_at, updated_at, server_id, source, ref, severity, description, message, active, first_seen, last_seen) VALUES \ - ('{ISSUE1}', NOW(), NOW(), '{ISRV}', 'test', 'r1', 'error', 'Disk full', 'disk usage 98%', true, NOW() - interval '2 days', NOW() - interval '1 hour'), \ - ('{ISSUE2}', NOW(), NOW(), '{ISRV}', 'test', 'r2', 'warning', NULL, 'slow query', false, NOW() - interval '10 days', NOW() - interval '9 days'); \ - INSERT INTO events (id, created_at, issue_id, severity, message, active, hash, occurrences, last_seen) VALUES \ - (gen_random_uuid(), NOW() - interval '1 hour', '{ISSUE1}', 'error', 'disk usage 98%', true, '\\x00'::bytea, 3, NOW() - interval '1 hour'); \ + INSERT INTO issues (id, created_at, updated_at, server_id, source, ref, check_name, observed_result, effective_result, description, message, active, first_seen, last_seen, last_degraded_at) VALUES \ + ('{ISSUE1}', NOW(), NOW(), '{ISRV}', 'test', 'r1', 'r1', 'failed', 'failed', 'Disk full', 'disk usage 98%', true, NOW() - interval '2 days', NOW() - interval '1 hour', NOW() - interval '1 hour'), \ + ('{ISSUE2}', NOW(), NOW(), '{ISRV}', 'test', 'r2', 'r2', 'warning', 'warning', NULL, 'slow query', false, NOW() - interval '10 days', NOW() - interval '9 days', NOW() - interval '9 days'); \ INSERT INTO incidents (id, created_at, updated_at, server_group_id, opened_at, closed_at) VALUES \ ('{INC_OPEN}', NOW(), NOW(), '{IGROUP}', NOW() - interval '2 days', NULL), \ ('{INC_CLOSED}', NOW(), NOW(), '{IGROUP}', NOW() - interval '5 days', NOW() - interval '3 days'); \ @@ -436,7 +434,7 @@ async fn incidents_window_status_and_detail() { let issues = detail["issues"].as_array().unwrap(); assert_eq!(issues.len(), 1); assert_eq!(issues[0]["issue_id"], ISSUE1); - assert_eq!(issues[0]["severity"], "error"); + assert_eq!(issues[0]["effective_result"], "failed"); assert_eq!(issues[0]["ref"], "r1"); assert_eq!(issues[0]["server_name"], "Inc Server"); }) @@ -463,23 +461,22 @@ async fn issues_filter_and_detail() { let all_ids = ids_of(&all, "issues"); assert!(all_ids.contains(&ISSUE2.to_string())); - // severity filter. + // result filter. let warnings = call_tool!( private, "find_issues", - serde_json::json!({ "active_only": false, "severities": ["warning"] }) + serde_json::json!({ "active_only": false, "results": ["warning"] }) ); let w = warnings["issues"].as_array().unwrap(); - assert!(!w.is_empty() && w.iter().all(|i| i["severity"] == "warning")); + assert!(!w.is_empty() && w.iter().all(|i| i["effective_result"] == "warning")); - // Detail: events + the incident it belongs to. + // Detail: the issue's fields + the incident it belongs to. let detail = call_tool!( private, "get_issue", serde_json::json!({ "issue_id": ISSUE1 }) ); - assert_eq!(detail["severity"], "error"); - assert!(!detail["recent_events"].as_array().unwrap().is_empty()); + assert_eq!(detail["effective_result"], "failed"); let inc_ids: Vec = detail["incidents"] .as_array() .unwrap() diff --git a/crates/private-server/tests/it/notes.rs b/crates/private-server/tests/it/notes.rs index 9605899a..7fea9e5a 100644 --- a/crates/private-server/tests/it/notes.rs +++ b/crates/private-server/tests/it/notes.rs @@ -20,7 +20,7 @@ async fn seed_issue_and_incident( .json(&serde_json::json!({ "serverId": server_id, "ref": "x", - "severity": "error", + "result": "failed", "message": "trouble", })) .await; diff --git a/crates/private-server/tests/it/private_statuses.rs b/crates/private-server/tests/it/private_statuses.rs index 679df671..4e12ac55 100644 --- a/crates/private-server/tests/it/private_statuses.rs +++ b/crates/private-server/tests/it/private_statuses.rs @@ -1046,7 +1046,7 @@ struct CheckAttentionServer { #[derive(Debug, Deserialize)] struct CheckAttentionResponse { check: String, - severity: Option, + ceiling: Option, servers: Vec, } @@ -1055,12 +1055,12 @@ async fn check_attention_empty_database() { commons_tests::server::run(async |_conn, _, private| { let r = private .post("/api/statuses/check_attention") - .json(&serde_json::json!({"check": "postgres"})) + .json(&serde_json::json!({"source": "alertd", "check": "postgres"})) .await; r.assert_status_ok(); let data: CheckAttentionResponse = r.json(); assert_eq!(data.check, "postgres"); - assert_eq!(data.severity, None); + assert_eq!(data.ceiling, None); assert!(data.servers.is_empty()); }) .await @@ -1078,28 +1078,28 @@ async fn check_attention_lists_servers_reporting_that_check_ordered_failed_first ('33333333-3333-3333-3333-333333333333', 'Healthy Server', 'https://healthy.example.com', 'production', 'central', 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa'), ('44444444-4444-4444-4444-444444444444', 'Other Check Server', 'https://other.example.com', 'production', 'central', 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa'); - INSERT INTO statuses (server_id, created_at, healthy, health) VALUES - ('11111111-1111-1111-1111-111111111111', NOW(), true, - '[{\"check\":\"postgres\",\"result\":\"warning\"}]'::jsonb), - ('22222222-2222-2222-2222-222222222222', NOW(), true, - '[{\"check\":\"postgres\",\"result\":\"failed\",\"free_pct\":2}]'::jsonb), - ('33333333-3333-3333-3333-333333333333', NOW(), true, - '[{\"check\":\"postgres\",\"result\":\"passed\"}]'::jsonb), - ('44444444-4444-4444-4444-444444444444', NOW(), true, - '[{\"check\":\"disk_space\",\"result\":\"failed\"}]'::jsonb)", + INSERT INTO issues (server_id, source, \"ref\", check_name, observed_result, effective_result, detail, message, active, first_seen, last_seen, degraded_since, last_degraded_at) VALUES + ('11111111-1111-1111-1111-111111111111', 'alertd', 'health/postgres', 'postgres', 'warning', 'warning', + '{\"check\":\"postgres\",\"result\":\"warning\"}'::jsonb, 'warned', true, NOW(), NOW(), NOW(), NOW()), + ('22222222-2222-2222-2222-222222222222', 'alertd', 'health/postgres', 'postgres', 'failed', 'failed', + '{\"check\":\"postgres\",\"result\":\"failed\",\"free_pct\":2}'::jsonb, 'failed', true, NOW(), NOW(), NOW(), NOW()), + ('33333333-3333-3333-3333-333333333333', 'alertd', 'health/postgres', 'postgres', 'passed', 'passed', + '{\"check\":\"postgres\",\"result\":\"passed\"}'::jsonb, 'passing', false, NOW(), NOW(), NULL, NULL), + ('44444444-4444-4444-4444-444444444444', 'alertd', 'health/disk_space', 'disk_space', 'failed', 'failed', + '{\"check\":\"disk_space\",\"result\":\"failed\"}'::jsonb, 'failed', true, NOW(), NOW(), NOW(), NOW())", ) .await .unwrap(); let r = private .post("/api/statuses/check_attention") - .json(&serde_json::json!({"check": "postgres"})) + .json(&serde_json::json!({"source": "alertd", "check": "postgres"})) .await; r.assert_status_ok(); let data: CheckAttentionResponse = r.json(); assert_eq!(data.check, "postgres"); - assert_eq!(data.severity, None, "no catalog row was ever created"); + assert_eq!(data.ceiling, None, "no catalog row was ever created"); assert_eq!( data.servers.len(), 3, @@ -1123,14 +1123,14 @@ async fn check_attention_lists_servers_reporting_that_check_ordered_failed_first data.servers[0].server_id, "22222222-2222-2222-2222-222222222222" ); - // The full health[] entry rides along for the expandable row. + // The check's detail rides along for the expandable row. assert_eq!( data.servers[0].data, serde_json::json!({"check": "postgres", "result": "failed", "free_pct": 2}), ); - assert_eq!( - data.servers[0].failing_since, None, - "no issue on file for this failure" + assert!( + data.servers[0].failing_since.is_some(), + "a degraded state row carries its streak start" ); assert_eq!(data.servers[1].server_name, "Warning Server"); @@ -1151,27 +1151,21 @@ async fn check_attention_failing_since_comes_from_the_active_issue() { ('11111111-1111-1111-1111-111111111111', 'Failing Server', 'https://failing.example.com', 'production', 'central'), ('22222222-2222-2222-2222-222222222222', 'Recovered Issue Server', 'https://recovered.example.com', 'production', 'central'); - INSERT INTO statuses (server_id, created_at, healthy, health) VALUES - ('11111111-1111-1111-1111-111111111111', NOW(), true, - '[{\"check\":\"postgres\",\"result\":\"failed\"}]'::jsonb), - ('22222222-2222-2222-2222-222222222222', NOW(), true, - '[{\"check\":\"postgres\",\"result\":\"failed\"}]'::jsonb); - - -- Active issue: its first_seen is the failing-since timestamp. - INSERT INTO issues (server_id, source, \"ref\", severity, message, active, first_seen, last_seen) VALUES - ('11111111-1111-1111-1111-111111111111', 'status', 'health/postgres', 'error', - 'postgres check failing', true, NOW() - INTERVAL '3 hours', NOW()), - -- Inactive issue (check recovered then re-failed without a new - -- push being processed yet): must NOT be used. - ('22222222-2222-2222-2222-222222222222', 'status', 'health/postgres', 'error', - 'postgres check failing', false, NOW() - INTERVAL '9 hours', NOW() - INTERVAL '8 hours')", + -- Active state: its degraded_since is the failing-since timestamp. + INSERT INTO issues (server_id, source, \"ref\", check_name, observed_result, effective_result, message, active, first_seen, last_seen, degraded_since, last_degraded_at) VALUES + ('11111111-1111-1111-1111-111111111111', 'alertd', 'health/postgres', 'postgres', 'failed', 'failed', + 'postgres check failing', true, NOW() - INTERVAL '3 hours', NOW(), NOW() - INTERVAL '3 hours', NOW()), + -- Recovered state (inactive): shows the last observed result but + -- carries no streak. + ('22222222-2222-2222-2222-222222222222', 'alertd', 'health/postgres', 'postgres', 'failed', 'failed', + 'postgres check failing', false, NOW() - INTERVAL '9 hours', NOW() - INTERVAL '8 hours', NULL, NOW() - INTERVAL '8 hours')", ) .await .unwrap(); let r = private .post("/api/statuses/check_attention") - .json(&serde_json::json!({"check": "postgres"})) + .json(&serde_json::json!({"source": "alertd", "check": "postgres"})) .await; r.assert_status_ok(); let data: CheckAttentionResponse = r.json(); @@ -1200,7 +1194,7 @@ async fn check_attention_failing_since_comes_from_the_active_issue() { .unwrap(); assert_eq!( recovered.failing_since, None, - "inactive issues don't provide failing_since" + "recovered state doesn't provide failing_since" ); }) .await @@ -1216,18 +1210,16 @@ async fn check_attention_excludes_ungrouped_and_archived_servers() { UPDATE servers SET deleted_at = NOW() WHERE id = '22222222-2222-2222-2222-222222222222'; - INSERT INTO statuses (server_id, created_at, healthy, health) VALUES - ('11111111-1111-1111-1111-111111111111', NOW(), true, - '[{\"check\":\"postgres\",\"result\":\"failed\"}]'::jsonb), - ('22222222-2222-2222-2222-222222222222', NOW(), true, - '[{\"check\":\"postgres\",\"result\":\"failed\"}]'::jsonb)", + INSERT INTO issues (server_id, source, \"ref\", check_name, observed_result, effective_result, message, active, first_seen, last_seen, degraded_since, last_degraded_at) VALUES + ('11111111-1111-1111-1111-111111111111', 'alertd', 'health/postgres', 'postgres', 'failed', 'failed', 'failed', true, NOW(), NOW(), NOW(), NOW()), + ('22222222-2222-2222-2222-222222222222', 'alertd', 'health/postgres', 'postgres', 'failed', 'failed', 'failed', true, NOW(), NOW(), NOW(), NOW())", ) .await .unwrap(); let r = private .post("/api/statuses/check_attention") - .json(&serde_json::json!({"check": "postgres"})) + .json(&serde_json::json!({"source": "alertd", "check": "postgres"})) .await; r.assert_status_ok(); let data: CheckAttentionResponse = r.json(); @@ -1241,15 +1233,14 @@ async fn check_attention_excludes_ungrouped_and_archived_servers() { } #[tokio::test(flavor = "multi_thread")] -async fn check_attention_returns_catalog_severity_and_ignores_non_matching_check() { +async fn check_attention_returns_catalog_policy_and_ignores_non_matching_check() { commons_tests::server::run(async |mut conn, _, private| { conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name, severity) VALUES ('postgres', 'error'); + "INSERT INTO check_policies (source, check_name, ceiling) VALUES ('alertd', 'postgres', 'failed'); INSERT INTO servers (id, name, host, rank, kind) VALUES ('11111111-1111-1111-1111-111111111111', 'Failing Server', 'https://failing.example.com', 'production', 'central'); - INSERT INTO statuses (server_id, created_at, healthy, health) VALUES - ('11111111-1111-1111-1111-111111111111', NOW(), true, - '[{\"check\":\"postgres\",\"result\":\"failed\"}]'::jsonb)", + INSERT INTO issues (server_id, source, \"ref\", check_name, observed_result, effective_result, message, active, first_seen, last_seen, degraded_since, last_degraded_at) VALUES + ('11111111-1111-1111-1111-111111111111', 'alertd', 'health/postgres', 'postgres', 'failed', 'failed', 'failed', true, NOW(), NOW(), NOW(), NOW())", ) .await .unwrap(); @@ -1257,43 +1248,43 @@ async fn check_attention_returns_catalog_severity_and_ignores_non_matching_check // The check this server is actually failing. let r = private .post("/api/statuses/check_attention") - .json(&serde_json::json!({"check": "postgres"})) + .json(&serde_json::json!({"source": "alertd", "check": "postgres"})) .await; r.assert_status_ok(); let data: CheckAttentionResponse = r.json(); - assert_eq!(data.severity, Some("error".to_string())); + assert_eq!(data.ceiling, Some("failed".to_string())); assert_eq!(data.servers.len(), 1); // A different, never-reported check name: no servers, but the // catalog lookup still runs (and correctly finds nothing). let r = private .post("/api/statuses/check_attention") - .json(&serde_json::json!({"check": "unrelated_check"})) + .json(&serde_json::json!({"source": "alertd", "check": "unrelated_check"})) .await; r.assert_status_ok(); let data: CheckAttentionResponse = r.json(); assert_eq!(data.check, "unrelated_check"); - assert_eq!(data.severity, None); + assert_eq!(data.ceiling, None); assert!(data.servers.is_empty()); }) .await } #[tokio::test(flavor = "multi_thread")] -async fn snapshot_surfaces_per_check_severity() { +async fn snapshot_surfaces_per_check_results() { commons_tests::server::run(async |mut conn, _, private| { - // Seed catalog: catalog_only stays at the default warning; - // elevated has its base severity bumped to error so the snapshot - // returns the operator-set value, not the legacy heuristic; - // version_gated has a rules ladder firing on a specific - // status.bestoolVersion. + // Seed catalog: catalog_only stays at the default warning + // ceiling; elevated has its ceiling lifted to failed so the + // snapshot returns the operator-set grading; version_gated has a + // rules ladder firing on a specific status.bestoolVersion, and + // escalates. conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name, severity) VALUES \ - ('catalog_only', 'warning'), \ - ('elevated', 'error'), \ - ('version_gated', 'warning'); \ - UPDATE healthcheck_severities \ - SET rules = '{\"if\":[{\"in_range\":[{\"var\":\"status.bestoolVersion\"},\">=1.0.0 <2.0.0\"]},\"critical\"]}'::jsonb \ + "INSERT INTO check_policies (source, check_name, ceiling, escalates) VALUES \ + ('alertd', 'catalog_only', 'warning', FALSE), \ + ('alertd', 'elevated', 'failed', FALSE), \ + ('alertd', 'version_gated', 'warning', TRUE); \ + UPDATE check_policies \ + SET rules = '{\"if\":[{\"in_range\":[{\"var\":\"status.bestoolVersion\"},\">=1.0.0 <2.0.0\"]},\"failed\"]}'::jsonb \ WHERE check_name = 'version_gated';", ) .await @@ -1321,30 +1312,30 @@ async fn snapshot_surfaces_per_check_severity() { .await; r.assert_status_ok(); let body: serde_json::Value = r.json(); - let severities = &body["check_severities"]; - assert_eq!(severities["catalog_only"], "warning"); - assert_eq!(severities["elevated"], "error"); + let results = &body["check_results"]; + assert_eq!(results["catalog_only"], "warning"); + assert_eq!(results["elevated"], "failed"); // version_gated's rule fires because bestoolVersion 1.5.0 is in - // the >=1.0.0 <2.0.0 range — Critical wins over the base. - assert_eq!(severities["version_gated"], "critical"); + // the >=1.0.0 <2.0.0 range — graded failed. + assert_eq!(results["version_gated"], "failed"); // Passing checks must not appear in the map. assert!( - severities.get("passing").is_none(), - "healthy checks are omitted; got {severities}", + results.get("passing").is_none(), + "healthy checks are omitted; got {results}", ); }) .await } #[tokio::test(flavor = "multi_thread")] -async fn snapshot_check_severities_cover_result_form() { +async fn snapshot_check_results_cover_result_form() { commons_tests::server::run(async |mut conn, _, private| { // `elevated` has its catalog base bumped to error: a failed // result uses it, a warning result ignores it (fixed Warning). conn.batch_execute( - "INSERT INTO healthcheck_severities (check_name, severity) VALUES \ - ('elevated', 'error'), \ - ('degraded', 'error');", + "INSERT INTO check_policies (source, check_name, ceiling) VALUES \ + ('alertd', 'elevated', 'failed'), \ + ('alertd', 'degraded', 'failed');", ) .await .unwrap(); @@ -1372,17 +1363,17 @@ async fn snapshot_check_severities_cover_result_form() { .await; r.assert_status_ok(); let body: serde_json::Value = r.json(); - let severities = &body["check_severities"]; - assert_eq!(severities["elevated"], "error"); + let results = &body["check_results"]; + assert_eq!(results["elevated"], "failed"); assert_eq!( - severities["degraded"], "warning", - "warning result lands at fixed Warning regardless of catalog base" + results["degraded"], "warning", + "a warning observation is already below the ceiling" ); // Broken/skipped/passed don't go through the rules engine. for check in ["busted", "absent", "fine"] { assert!( - severities.get(check).is_none(), - "{check} must be omitted; got {severities}", + results.get(check).is_none(), + "{check} must be omitted; got {results}", ); } }) @@ -1404,7 +1395,10 @@ async fn get_detail_health_excludes_silenced_checks() { INSERT INTO statuses (server_id, version, healthy, health, extra, created_at) VALUES ('11111111-1111-1111-1111-111111111111', '1.0.0', true, - '[{\"check\": \"postgres\", \"result\": \"failed\"}]'::jsonb, '{}'::jsonb, NOW())", + '[{\"check\": \"postgres\", \"result\": \"failed\"}]'::jsonb, '{}'::jsonb, NOW()); + + INSERT INTO issues (server_id, source, ref, check_name, observed_result, effective_result, message, active, first_seen, last_seen, degraded_since, last_degraded_at) VALUES + ('11111111-1111-1111-1111-111111111111', 'alertd', 'health/postgres', 'postgres', 'failed', 'failed', 'postgres failed', true, NOW(), NOW(), NOW(), NOW())", ) .await .unwrap(); @@ -1423,8 +1417,8 @@ async fn get_detail_health_excludes_silenced_checks() { assert_eq!(body["health"], "unhealthy"); conn.batch_execute( - "INSERT INTO server_silenced_refs (server_id, source, ref) VALUES - ('11111111-1111-1111-1111-111111111111', 'status', 'health/postgres')", + "INSERT INTO scoped_check_policies (server_id, source, check_name, ceiling) VALUES + ('11111111-1111-1111-1111-111111111111', 'alertd', 'postgres', 'skipped')", ) .await .unwrap(); @@ -1437,7 +1431,9 @@ async fn get_detail_health_excludes_silenced_checks() { "raw check result must stay on the status payload" ); - conn.batch_execute("DELETE FROM server_silenced_refs").await.unwrap(); + conn.batch_execute("DELETE FROM scoped_check_policies") + .await + .unwrap(); let body = detail().await; assert_eq!(body["health"], "unhealthy"); }) @@ -1465,8 +1461,11 @@ async fn group_details_member_health_excludes_group_silenced_checks() { ('11111111-1111-1111-1111-111111111111', '1.0.0', true, '[{\"check\": \"disk\", \"result\": \"failed\"}]'::jsonb, '{}'::jsonb, NOW()); - INSERT INTO server_group_silenced_refs (server_group_id, source, ref) VALUES - ('aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa', 'status', 'health/disk')", + INSERT INTO issues (server_id, source, ref, check_name, observed_result, effective_result, message, active, first_seen, last_seen, degraded_since, last_degraded_at) VALUES + ('11111111-1111-1111-1111-111111111111', 'alertd', 'health/disk', 'disk', 'failed', 'failed', 'disk failed', true, NOW(), NOW(), NOW(), NOW()); + + INSERT INTO scoped_check_policies (server_group_id, source, check_name, ceiling) VALUES + ('aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa', 'alertd', 'disk', 'skipped')", ) .await .unwrap(); @@ -1495,8 +1494,8 @@ async fn snapshot_reports_and_excludes_silenced_checks() { ('11111111-1111-1111-1111-111111111111', '1.0.0', true, '[{\"check\": \"postgres\", \"result\": \"failed\"}, {\"check\": \"disk\", \"result\": \"passed\"}]'::jsonb, '{}'::jsonb, NOW()); - INSERT INTO server_silenced_refs (server_id, source, ref) VALUES - ('11111111-1111-1111-1111-111111111111', 'status', 'health/postgres')", + INSERT INTO scoped_check_policies (server_id, source, check_name, ceiling) VALUES + ('11111111-1111-1111-1111-111111111111', 'alertd', 'postgres', 'skipped')", ) .await .unwrap(); diff --git a/crates/private-server/tests/it/tagged_device_guard.rs b/crates/private-server/tests/it/tagged_device_guard.rs index 90372ea4..33de3eca 100644 --- a/crates/private-server/tests/it/tagged_device_guard.rs +++ b/crates/private-server/tests/it/tagged_device_guard.rs @@ -81,13 +81,9 @@ async fn public_mount_is_not_subject_to_tagged_device_guard() { // test fixture), NOT a 403 from the guard. commons_tests::server::run(async |_conn, _public, private| { let response = private - .post("/public/events") + .post(&format!("/public/status/{}", uuid::Uuid::new_v4())) .add_header("Forwarded", "for=100.64.0.42") - .json(&serde_json::json!({ - "source": "watchdog", - "ref": "x", - "message": "nope", - })) + .json(&serde_json::json!({ "health": [] })) .await; assert_eq!(response.status_code().as_u16(), 401); let body: serde_json::Value = response.json(); diff --git a/crates/private-server/tests/it/tailnet_device_auth.rs b/crates/private-server/tests/it/tailnet_device_auth.rs index 77b51105..3a7c0a75 100644 --- a/crates/private-server/tests/it/tailnet_device_auth.rs +++ b/crates/private-server/tests/it/tailnet_device_auth.rs @@ -19,28 +19,29 @@ async fn provision_server( } #[tokio::test(flavor = "multi_thread")] -async fn tailnet_server_device_can_post_event() { +async fn tailnet_server_device_can_push_status() { commons_tests::server::run_with_tailnet_device_auth( "server", async |mut conn, tailnet_ip, _node_id, device_id, _public, private| { let server_id = provision_server(&mut conn, device_id).await; let response = private - .post("/public/events") + .post(&format!("/public/status/{server_id}")) .add_header("Forwarded", &format!("for={tailnet_ip}")) .json(&serde_json::json!({ - "source": "watchdog", - "ref": "disk-/var", - "message": "less than 5% free", + "health": [ { "check": "disk", "result": "passed" } ], })) .await; response.assert_status_ok(); - let body: serde_json::Value = response.json(); - assert_eq!( - body.get("server_id").and_then(|v| v.as_str()), - Some(server_id.to_string().as_str()), - ); + // The push landed as a status row against the right server. + conn.batch_execute(&format!( + "DO $$ BEGIN IF NOT EXISTS (SELECT 1 FROM statuses \ + WHERE server_id = '{server_id}' AND device_id = '{device_id}') \ + THEN RAISE EXCEPTION 'status row not recorded'; END IF; END $$;" + )) + .await + .expect("status row recorded for the server"); }, ) .await @@ -98,13 +99,9 @@ async fn unknown_tailnet_node_is_rejected_without_creating_a_row() { let private = TestServer::new(private_router); let response = private - .post("/public/events") + .post(&format!("/public/status/{}", Uuid::new_v4())) .add_header("Forwarded", &format!("for={tailnet_ip}")) - .json(&serde_json::json!({ - "source": "watchdog", - "ref": "x", - "message": "first contact", - })) + .json(&serde_json::json!({ "health": [] })) .await; assert_eq!(response.status_code().as_u16(), 401); @@ -133,16 +130,12 @@ async fn non_tailnet_source_ip_rejected() { commons_tests::server::run_with_tailnet_device_auth( "server", async |mut conn, _tailnet_ip, _node_id, device_id, _public, private| { - provision_server(&mut conn, device_id).await; + let server_id = provision_server(&mut conn, device_id).await; let response = private - .post("/public/events") + .post(&format!("/public/status/{server_id}")) .add_header("Forwarded", "for=203.0.113.7") - .json(&serde_json::json!({ - "source": "watchdog", - "ref": "x", - "message": "spoofed", - })) + .json(&serde_json::json!({ "health": [] })) .await; assert_eq!(response.status_code().as_u16(), 401); let body: serde_json::Value = response.json(); diff --git a/crates/private-server/tests/it/tailnet_key_expiry_sweep.rs b/crates/private-server/tests/it/tailnet_key_expiry_sweep.rs index c531a3ee..b451bcc6 100644 --- a/crates/private-server/tests/it/tailnet_key_expiry_sweep.rs +++ b/crates/private-server/tests/it/tailnet_key_expiry_sweep.rs @@ -10,7 +10,6 @@ use commons_servers::{ }; use commons_tests::db::TestDb; use commons_tests::diesel_async::{AsyncPgConnection, SimpleAsyncConnection}; -use commons_types::issue::Severity; use database::issues::Issue; use uuid::Uuid; @@ -73,7 +72,11 @@ async fn sweep_opens_critical_issue_when_key_expiry_enabled() { assert_eq!(filed, 1); let issue = issue_for(&mut conn, server_id).await.expect("issue"); - assert_eq!(issue.severity, Severity::Critical); + assert_eq!( + issue.effective_result, + Some(commons_types::status::CheckResult::Failed) + ); + assert!(issue.escalates); assert_eq!(issue.device_id, Some(device_id)); assert!(issue.active); }) diff --git a/crates/public-server/openapi.json b/crates/public-server/openapi.json index 2bb8f05f..d1f70dab 100644 --- a/crates/public-server/openapi.json +++ b/crates/public-server/openapi.json @@ -370,83 +370,6 @@ } } }, - "/events": { - "post": { - "tags": [ - "events" - ], - "summary": "Report an event against the calling device's server.", - "description": "Requires a device certificate with the server role (or admin). Used to\npush a status update — a healthcheck result, an alert condition, or a\n\"condition cleared\" notice — which canopy records as an issue and, if\nthe server belongs to a group, may fold into an incident. An event\nwith the same `source` and `ref` as an already-open issue on this\nserver updates that issue instead of opening a new one.\n\nThe calling device must already be enrolled against exactly one\nserver, otherwise the request fails with 412. The `source` value\n`\"manual\"` is reserved for operator-entered events and is rejected\nwith 400 here, as is an empty `ref`.\n\nReturns the issue the event was recorded against (existing or newly\ncreated).", - "operationId": "submit_event", - "requestBody": { - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/NewEvent" - } - } - }, - "required": true - }, - "responses": { - "200": { - "description": "", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Issue" - } - } - } - }, - "400": { - "description": "", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/ProblemDetailsSchema" - } - } - } - }, - "401": { - "description": "", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/ProblemDetailsSchema" - } - } - } - }, - "403": { - "description": "", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/ProblemDetailsSchema" - } - } - } - }, - "412": { - "description": "Device is not registered against any server.", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/ProblemDetailsSchema" - } - } - } - } - }, - "security": [ - { - "server-device": [] - } - ] - } - }, "/restore-capabilities": { "post": { "tags": [ @@ -753,7 +676,7 @@ } ], "requestBody": { - "description": "Status push. A `health` array is required (it may be empty); a body without it is rejected with 400, unless the server has `allow_legacy_status` set, in which case the push refreshes reachability only and carries the last known healthchecks forward.", + "description": "Status push. A body without a `health` array is the legacy Tamanu direct-report format, treated as the `tamanu` source reporting a single always-passing `tasks` heartbeat check.", "content": { "application/json": { "schema": { @@ -818,7 +741,7 @@ "statuses" ], "summary": "Fetch the effective healthcheck severity mapping for a server.", - "description": "Returns, for every healthcheck canopy knows about, how a failure of that\ncheck is classified for this server: `skip` (the check is silenced for\nthis server — at server or group scope — or its severity is below\nwarning), `warn` (warning), or `fail` (error or critical). Keys are check\nnames as reported in `health[].check` on status pushes. Only the static\nseverity baseline is reflected; operator-defined conditional rules are\nevaluated per push and not included here. The same mapping also rides\nalong every status-push response as `check_severities`.\n\nThe calling device must be the one enrolled for this exact server (or\nhold the admin role).", + "description": "Returns, for every healthcheck the `alertd` source reports, how that\ncheck is handled for this server: `skip` (the check is silenced for\nthis server — at server or group scope — or its policy ceiling means it\nnever alerts), `warn` (graded at most a warning), or `fail` (failures\ncount as failures). Keys are check names as reported in\n`health[].check` on status pushes. Only the static policy ceiling is\nreflected; operator-defined conditional rules are evaluated per push\nand not included here. The same mapping also rides along every\nstatus-push response as `check_severities`, scoped to the pushing\nsource.\n\nThe calling device must be the one enrolled for this exact server (or\nhold the admin role).", "operationId": "check_severities", "parameters": [ { @@ -1421,7 +1344,7 @@ }, "CheckSeverity": { "type": "string", - "description": "How a server should treat one of its healthchecks, distilled from\ncanopy's operator-side configuration (the severity catalog and the\nsilence lists) into a three-level device-facing vocabulary.", + "description": "How a server should treat one of its healthchecks, distilled from\ncanopy's operator-side configuration (the policy catalog and the\nsilences) into a three-level device-facing vocabulary.", "enum": [ "skip", "warn", @@ -1544,7 +1467,7 @@ }, { "$ref": "#/components/schemas/CheckResult", - "description": "Outcome of the check: `passed`, `warning`, `failed`, `broken`, or\n`skipped`. Exactly one of `result` / `healthy` must be present per\nentry. `warning` and `failed` open an issue at the check's catalog\nseverity; `broken` (the check itself errored, not the system under\ntest) opens a separate issue at a fixed Warning severity without\nconfirming or clearing a known failure; `skipped` (a precondition was\nnot met) and `passed` open nothing and close prior issues." + "description": "Outcome of the check: `passed`, `warning`, `failed`, `broken`, or\n`skipped`. Exactly one of `result` / `healthy` must be present per\nentry. `warning` and `failed` open the check's issue as graded by\nits policy; `broken` (the check itself errored, not the system under\ntest) neither confirms nor clears a known failure — the issue stays\nopen, retaining its contribution; `skipped` (a precondition was\nnot met) and `passed` open nothing and close prior issues." } ] } @@ -1584,186 +1507,6 @@ } } }, - "Issue": { - "type": "object", - "description": "A tracked problem or condition on a server, or on a server group as a\nwhole. An issue is opened the first time its source reports it and stays\nopen (accumulating events) until the source reports it as no longer\nactive or an operator resolves it. Issues are the basic unit that drives\nincidents, Slack notifications, and the fleet health view.", - "required": [ - "id", - "created_at", - "updated_at", - "source", - "ref", - "severity", - "message", - "active", - "first_seen", - "last_seen" - ], - "properties": { - "active": { - "type": "boolean", - "description": "Whether the condition behind this issue is still ongoing. Set to\n`false` when the source reports the condition has cleared." - }, - "created_at": { - "type": "string", - "format": "date-time", - "description": "When this issue was first created." - }, - "description": { - "type": [ - "string", - "null" - ], - "description": "A short, single-line title for the issue, shown as its headline in\nthe UI and in Slack notifications. `None` if no title was given." - }, - "device_id": { - "type": [ - "string", - "null" - ], - "format": "uuid", - "description": "The device that reported this issue, if it was raised by a device\npush. `None` for issues raised by an operator or by the platform\nitself." - }, - "first_seen": { - "type": "string", - "format": "date-time", - "description": "When this issue was first reported." - }, - "id": { - "type": "string", - "format": "uuid", - "description": "Unique identifier for this issue." - }, - "last_seen": { - "type": "string", - "format": "date-time", - "description": "When this issue was most recently reported or updated." - }, - "message": { - "type": "string", - "description": "The full body text describing the issue." - }, - "ref": { - "type": "string", - "description": "A caller-chosen identifier for this issue within its `source`. The\nsame `(source, ref)` pair reported again updates this issue instead\nof creating a new one." - }, - "resolved_at": { - "type": [ - "string", - "null" - ], - "format": "date-time", - "description": "When an operator marked this issue resolved. `None` if it hasn't\nbeen resolved." - }, - "resolved_by": { - "type": [ - "string", - "null" - ], - "description": "The operator who resolved this issue. `None` if it hasn't been\nresolved." - }, - "resolved_reason": { - "type": [ - "string", - "null" - ], - "description": "The reason given when the issue was resolved (for example: fixed,\nfalse positive, won't fix). `None` if it hasn't been resolved." - }, - "server_group_id": { - "type": [ - "string", - "null" - ], - "format": "uuid", - "description": "The server group this issue is attached to, for a control-plane issue\n(e.g. backup corruption, a failed preflight check) that isn't tied to\nany single server. `None` for an ordinary server-scoped issue.\nGroup-scoped issues are always considered even if an individual\nserver in the group has monitoring turned off." - }, - "server_id": { - "type": [ - "string", - "null" - ], - "format": "uuid", - "description": "The server this issue is attached to. `None` for a group-scoped issue\n(see `server_group_id`) — exactly one of the two is always set." - }, - "severity": { - "$ref": "#/components/schemas/Severity", - "description": "The issue's current severity." - }, - "snoozed_until": { - "type": [ - "string", - "null" - ], - "format": "date-time", - "description": "If set, this issue is snoozed until this time: it's temporarily\nexcluded from incidents and notifications even while still active." - }, - "source": { - "type": "string", - "description": "Identifies what raised this issue — a healthcheck, a backup pipeline,\nan operator, etc. Used together with `ref` to detect repeat reports\nof the same underlying problem." - }, - "updated_at": { - "type": "string", - "format": "date-time", - "description": "When this issue was last modified." - } - } - }, - "NewEvent": { - "type": "object", - "description": "A single occurrence to report against an issue, sent either by a device\nor by an operator. If an issue with the same `source` and `ref` is\nalready open, this occurrence is folded into it (bumping its last-seen\ntime, and coalescing into the latest recorded event when the content is\nidentical); otherwise a new issue is created.", - "required": [ - "source", - "ref", - "message" - ], - "properties": { - "active": { - "type": [ - "boolean", - "null" - ], - "description": "Whether the condition this event describes is still ongoing.\nDefaults to `true`. Sending `false` marks the condition as cleared,\nwhich can close the issue's contribution to any open incident." - }, - "description": { - "type": [ - "string", - "null" - ], - "description": "A short, single-line title for this event, shown as the issue's\nheadline in the UI and as the subject of any Slack notification.\nMust not contain newlines. Use `message` for the full body text." - }, - "message": { - "type": "string", - "description": "The full body text for this event. Free-form, multi-line text is\nfine. If `description` is omitted, the UI falls back to using the\nfirst line of this field as the headline." - }, - "occurredAt": { - "type": [ - "string", - "null" - ], - "format": "date-time", - "description": "When the event actually happened, if known and different from when\nit was received. Defaults to the time the event was received." - }, - "ref": { - "type": "string", - "description": "A caller-chosen identifier for this specific problem within `source`,\nused to detect repeated reports of the same underlying condition.\nRequired — mint a UUID if deduplication isn't needed." - }, - "severity": { - "oneOf": [ - { - "type": "null" - }, - { - "$ref": "#/components/schemas/Severity", - "description": "Severity of this event. Defaults to a standard severity if omitted." - } - ] - }, - "source": { - "type": "string", - "description": "Identifies what is reporting this event — a healthcheck, a backup\npipeline, an operator, etc." - } - } - }, "ParamType": { "type": "string", "description": "The data type of a restore-replica configuration parameter, which\ndetermines how its value is validated. `duration` and `bytes` values must\nbe non-negative integers (a count of seconds and of bytes, respectively);\n`integer` accepts any whole number, positive or negative; `boolean` is a\nJSON boolean; `text` is a JSON string.", @@ -2002,17 +1745,6 @@ "dev" ] }, - "Severity": { - "type": "string", - "description": "Canopy's severity vocabulary, narrowed from RFC 5424 to a five-level\nset with operator semantics:\n\n- `Debug`: never participates in incidents (filed for the audit\n trail / per-server view only).\n- `Info`, `Warning`: join an open incident for context but don't\n open one or keep one open on their own.\n- `Error`: opens an incident; sits in the per-group `slack_open_delay`\n holding window before the Slack notification ships.\n- `Critical`: opens an incident and bypasses the holding window —\n the Slack notification is enqueued for immediate delivery.\n\nStored as text in Postgres; validated as this enum at the API layer.\nDefault is `Error` (the most common severity for a deliberately\nfiled event). The legacy syslog severities `emergency` / `alert` /\n`notice` have been retired — see the\n`2026-05-29-000000-0000_restrict_severities` migration; the\n`FromStr` impl still accepts them as aliases for forward-compat with\nany device that hasn't been updated.", - "enum": [ - "critical", - "error", - "warning", - "info", - "debug" - ] - }, "SnippetResponse": { "type": "object", "description": "A single named SQL snippet from the bestool snippet library.", @@ -2050,7 +1782,7 @@ "items": { "$ref": "#/components/schemas/HealthCheck" }, - "description": "Per-check breakdown. **Required** — a push without a `health` array is\nrejected with 400 (unless an operator has opted the server into the\nlegacy format, in which case such a push only refreshes reachability\nand carries the previously reported checks forward). May be empty\n(`[]`) for a server that genuinely runs no checks. Each entry must\ninclude a non-empty `check` name and exactly one of `result` /\n`healthy`; any additional fields per check (latency, free disk %,\ncertificate expiry, etc.) are passed through verbatim and shown in the\nstatus UI.\n\nEvery check name seen — whatever its result — is added to the\noperator-facing check catalog, where the severity assigned to its\nfailures can be reviewed and adjusted. A failed or warning check opens\n(or keeps open) an issue for that check at the catalog's current\nseverity; a broken check opens a separate issue at a fixed Warning\nseverity; passed and skipped results open nothing and close prior\nissues." + "description": "Per-check breakdown. A push without a `health` array is the legacy\nTamanu direct-report format: it is treated as the `tamanu` source\nreporting a single always-passing `tasks` heartbeat check. May be\nempty (`[]`) for a source that genuinely runs no checks — which\nrecovers every check it previously reported. Each entry must\ninclude a non-empty `check` name and exactly one of `result` /\n`healthy`; any additional fields per check (latency, free disk %,\ncertificate expiry, etc.) are passed through verbatim and shown in the\nstatus UI.\n\nEvery check name seen — whatever its result — is added to the\noperator-facing check catalog, where the policy grading its results\ncan be reviewed and adjusted. A check whose effective result is\nfailed or warning opens (or keeps open) its issue; a broken check\nkeeps the same issue open, retaining a known failure's contribution\nwhile warning the check itself is broken; effective passed and\nskipped results open nothing and close prior issues." }, "healthy": { "type": [ @@ -2058,6 +1790,13 @@ "null" ], "description": "Overall self-reported health of the server. **Absent means `true`**,\nso senders that predate this field are never treated as unhealthy by\nomission. Recorded for historical analysis and display, but **not\nconsulted for incident or severity decisions** — those are derived\nfrom the per-check results in `health`, with each check's severity\ncontrolled by an operator-managed catalog." + }, + "source": { + "type": [ + "string", + "null" + ], + "description": "The name of the source pushing this status: the reporting agent, e.g.\n`alertd`. Multiple sources may report on one server, each with its own\nset of checks; a source's push only opens and recovers its own checks.\n\n**Transitionally optional: this field will become mandatory.** A push\nwithout a `source` is attributed to `alertd`; new reporters must send\ntheir own name. Must be a non-empty string; the names `canopy` and\n`manual` are reserved for canopy itself and are rejected." } } } @@ -2078,7 +1817,7 @@ "items": { "type": "string" }, - "description": "Backup types the server should back up now: operator-requested\none-offs plus scheduled backups that are due. Each serializes as a\nplain string (e.g. `\"tamanu-postgres\"`). The device should run each\nlisted type, then report via `POST /backup-report`; an empty list\nmeans nothing to do." + "description": "Backup types the server should back up now: operator-requested\none-offs plus scheduled backups that are due. Each serializes as a\nplain string (e.g. `\"tamanu-postgres\"`). The device should run each\nlisted type, then report via `POST /backup-report`; an empty list\nmeans nothing to do. Only sent to `alertd` pushes (the agent that\nruns backups); other sources always receive an empty list." }, "check_severities": { "type": "object", @@ -2459,10 +2198,6 @@ "name": "bestool", "description": "Bestool SQL snippet read API." }, - { - "name": "events", - "description": "Device-pushed events; rolled up into issues and incidents server-side." - }, { "name": "restore", "description": "Managed restore replicas: consumer capability registration, worklist, and read-only restore credentials." diff --git a/crates/public-server/src/events.rs b/crates/public-server/src/events.rs deleted file mode 100644 index b3ba42d5..00000000 --- a/crates/public-server/src/events.rs +++ /dev/null @@ -1,73 +0,0 @@ -use axum::{Json, extract::State}; -use canopy_utoipa_axum::{router::OpenApiRouter, routes}; -use commons_errors::{AppError, ProblemDetailsSchema, Result}; -use commons_servers::device_auth::ServerDevice; -use database::{ - Db, - issues::{Issue, NewEvent}, - servers::Server, -}; - -use crate::state::AppState; - -pub fn routes() -> OpenApiRouter { - OpenApiRouter::new().routes(routes!(create)) -} - -/// Report an event against the calling device's server. -/// -/// Requires a device certificate with the server role (or admin). Used to -/// push a status update — a healthcheck result, an alert condition, or a -/// "condition cleared" notice — which canopy records as an issue and, if -/// the server belongs to a group, may fold into an incident. An event -/// with the same `source` and `ref` as an already-open issue on this -/// server updates that issue instead of opening a new one. -/// -/// The calling device must already be enrolled against exactly one -/// server, otherwise the request fails with 412. The `source` value -/// `"manual"` is reserved for operator-entered events and is rejected -/// with 400 here, as is an empty `ref`. -/// -/// Returns the issue the event was recorded against (existing or newly -/// created). -#[utoipa::path( - post, - path = "/events", - operation_id = "submit_event", - tag = "events", - security(("server-device" = [])), - request_body = NewEvent, - responses( - (status = 200, body = Issue), - (status = 400, body = ProblemDetailsSchema), - (status = 401, body = ProblemDetailsSchema), - (status = 403, body = ProblemDetailsSchema), - (status = 412, description = "Device is not registered against any server.", body = ProblemDetailsSchema), - ), -)] -async fn create( - State(db): State, - device: ServerDevice, - Json(event): Json, -) -> Result> { - if event.source.eq_ignore_ascii_case("manual") { - return Err(AppError::SourceManualForbidden); - } - if event.r#ref.trim().is_empty() { - return Err(AppError::custom("ref is required")); - } - - let mut conn = db.get().await?; - let device_id = device.0.0.id; - - // Strict: device must be registered against a server before reporting events. - // (servers.device_id is unique, so at most one server matches.) - let server = Server::get_by_device_id(&mut conn, device_id) - .await? - .into_iter() - .next() - .ok_or(AppError::DeviceHasNoServer)?; - - let issue = event.save(&mut conn, server.id, Some(device_id)).await?; - Ok(Json(issue)) -} diff --git a/crates/public-server/src/lib.rs b/crates/public-server/src/lib.rs index f381f660..c8d4cee8 100644 --- a/crates/public-server/src/lib.rs +++ b/crates/public-server/src/lib.rs @@ -7,7 +7,6 @@ use crate::state::AppState; pub mod artifacts; pub mod backup; pub mod bestool; -pub mod events; pub mod mcp; pub mod openapi; #[cfg(feature = "ui")] @@ -27,7 +26,6 @@ pub mod versions; pub fn routes() -> OpenApiRouter { #[cfg_attr(not(feature = "ui"), expect(unused_mut))] let mut router = OpenApiRouter::new() - .merge(events::routes()) .merge(backup::routes()) .merge(restore::routes()) .nest("/artifacts", artifacts::routes()) diff --git a/crates/public-server/src/openapi.rs b/crates/public-server/src/openapi.rs index 3f0e1f0e..ff90af29 100644 --- a/crates/public-server/src/openapi.rs +++ b/crates/public-server/src/openapi.rs @@ -18,7 +18,6 @@ use utoipa::{Modify, OpenApi, openapi::security::SecurityScheme}; (name = "artifacts", description = "Per-version artifact registration by releaser devices."), (name = "backup", description = "Device backup credential minting, target config, capability registration, and run reporting."), (name = "bestool", description = "Bestool SQL snippet read API."), - (name = "events", description = "Device-pushed events; rolled up into issues and incidents server-side."), (name = "restore", description = "Managed restore replicas: consumer capability registration, worklist, and read-only restore credentials."), (name = "servers", description = "Server registry — listing for the public, self-registration for server devices."), (name = "statuses", description = "Heartbeat / status submissions from server devices."), diff --git a/crates/public-server/src/statuses.rs b/crates/public-server/src/statuses.rs index dd522c8b..1a87e83a 100644 --- a/crates/public-server/src/statuses.rs +++ b/crates/public-server/src/statuses.rs @@ -13,19 +13,18 @@ use commons_servers::{ use commons_types::{ backup::BackupType, device::DeviceRole, - issue::Severity, server::TagMap, status::{CheckResult, CheckSeverity}, version::VersionStr, }; use database::{ Db, + check_policies::{CheckPolicy, EvaluationContext, GradedResult}, devices::Device, diesel_async::{AsyncConnection, AsyncPgConnection}, - healthcheck_severities::{EvaluationContext, HealthcheckSeverity}, - issues::{Issue, NewEvent}, + issues::{CheckStateStamp, Issue, NewEvent}, servers::Server, - silenced_refs::silenced_refs_with_prefix, + silenced_refs::silenced_health_checks_for_server, statuses::{NewStatus, Status}, }; use jiff::Timestamp; @@ -43,6 +42,16 @@ use crate::state::AppState; /// status data. #[derive(Debug, Deserialize, ToSchema)] pub struct StatusPayload { + /// The name of the source pushing this status: the reporting agent, e.g. + /// `alertd`. Multiple sources may report on one server, each with its own + /// set of checks; a source's push only opens and recovers its own checks. + /// + /// **Transitionally optional: this field will become mandatory.** A push + /// without a `source` is attributed to `alertd`; new reporters must send + /// their own name. Must be a non-empty string; the names `canopy` and + /// `manual` are reserved for canopy itself and are rejected. + pub source: Option, + /// Overall self-reported health of the server. **Absent means `true`**, /// so senders that predate this field are never treated as unhealthy by /// omission. Recorded for historical analysis and display, but **not @@ -51,23 +60,23 @@ pub struct StatusPayload { /// controlled by an operator-managed catalog. pub healthy: Option, - /// Per-check breakdown. **Required** — a push without a `health` array is - /// rejected with 400 (unless an operator has opted the server into the - /// legacy format, in which case such a push only refreshes reachability - /// and carries the previously reported checks forward). May be empty - /// (`[]`) for a server that genuinely runs no checks. Each entry must + /// Per-check breakdown. A push without a `health` array is the legacy + /// Tamanu direct-report format: it is treated as the `tamanu` source + /// reporting a single always-passing `tasks` heartbeat check. May be + /// empty (`[]`) for a source that genuinely runs no checks — which + /// recovers every check it previously reported. Each entry must /// include a non-empty `check` name and exactly one of `result` / /// `healthy`; any additional fields per check (latency, free disk %, /// certificate expiry, etc.) are passed through verbatim and shown in the /// status UI. /// /// Every check name seen — whatever its result — is added to the - /// operator-facing check catalog, where the severity assigned to its - /// failures can be reviewed and adjusted. A failed or warning check opens - /// (or keeps open) an issue for that check at the catalog's current - /// severity; a broken check opens a separate issue at a fixed Warning - /// severity; passed and skipped results open nothing and close prior - /// issues. + /// operator-facing check catalog, where the policy grading its results + /// can be reviewed and adjusted. A check whose effective result is + /// failed or warning opens (or keeps open) its issue; a broken check + /// keeps the same issue open, retaining a known failure's contribution + /// while warning the check itself is broken; effective passed and + /// skipped results open nothing and close prior issues. pub health: Vec, /// Free-form additional data (uptime, database version, timezone, @@ -93,10 +102,10 @@ pub struct HealthCheck { pub check: String, /// Outcome of the check: `passed`, `warning`, `failed`, `broken`, or /// `skipped`. Exactly one of `result` / `healthy` must be present per - /// entry. `warning` and `failed` open an issue at the check's catalog - /// severity; `broken` (the check itself errored, not the system under - /// test) opens a separate issue at a fixed Warning severity without - /// confirming or clearing a known failure; `skipped` (a precondition was + /// entry. `warning` and `failed` open the check's issue as graded by + /// its policy; `broken` (the check itself errored, not the system under + /// test) neither confirms nor clears a known failure — the issue stays + /// open, retaining its contribution; `skipped` (a precondition was /// not met) and `passed` open nothing and close prior issues. pub result: Option, /// Legacy pass/fail form: `true` means `passed`, `false` means `failed`. @@ -110,19 +119,25 @@ pub struct HealthCheck { pub extra: serde_json::Map, } -/// `source` value for the events filed below. Distinct from -/// `canopy` (reachability sweep) so operators can tell apart "we -/// couldn't reach you" from "you told us you're sick". -const STATUS_SOURCE: &str = "status"; -/// Prefix for per-check refs. Each failed check is filed at -/// `(status, health/)`. +/// The source a push is attributed to when it names none: the reporter +/// deployed before the `source` field existed. Also the migration value for +/// pre-source history. Transitional — the field will become mandatory. +const DEFAULT_SOURCE: &str = "alertd"; +/// The source legacy-format pushes (no `health` array) are attributed to: +/// they come from Tamanu's own direct reporting, not from alertd. +const LEGACY_SOURCE: &str = "tamanu"; +/// The synthetic check a legacy push reports: a liveness heartbeat, +/// always passing on receipt. Its value is that it stops — a Tamanu +/// server that goes quiet trips the source-staleness net. +const LEGACY_CHECK: &str = "tasks"; +/// Source names a push may not claim: `canopy` is canopy's own +/// determinations (reachability sweep etc.), `manual` is operator-entered. +const RESERVED_SOURCES: &[&str] = &[database::statuses::CANOPY_SOURCE, "manual"]; +/// Prefix for per-check refs. Each check is filed at +/// `(, health/)` — one thread per check, brokenness +/// included (a broken check retains the previous definite result's +/// contribution while additionally warning that the check is broken). const HEALTH_REF: &str = "health"; -/// Prefix for broken-check refs. A check reporting `result: broken` -/// files at `(status, health-broken/)` — a separate ref -/// from the check's failure issue, so a known failure can stay open -/// (unconfirmed either way) while the check itself is broken, and so -/// operators can silence the two independently. -const BROKEN_REF: &str = "health-broken"; /// The status-push response: only the return-path instructions the device /// can act on. The stored status record is deliberately not echoed back — @@ -133,7 +148,8 @@ pub struct StatusResponse { /// one-offs plus scheduled backups that are due. Each serializes as a /// plain string (e.g. `"tamanu-postgres"`). The device should run each /// listed type, then report via `POST /backup-report`; an empty list - /// means nothing to do. + /// means nothing to do. Only sent to `alertd` pushes (the agent that + /// runs backups); other sources always receive an empty list. #[schema(value_type = Vec)] pub backup_now: Vec, /// The effective handling of every healthcheck canopy knows about, keyed @@ -188,7 +204,7 @@ pub fn routes() -> OpenApiRouter { ), request_body( content = StatusPayload, - description = "Status push. A `health` array is required (it may be empty); a body without it is rejected with 400, unless the server has `allow_legacy_status` set, in which case the push refreshes reachability only and carries the last known healthchecks forward.", + description = "Status push. A body without a `health` array is the legacy Tamanu direct-report format, treated as the `tamanu` source reporting a single always-passing `tasks` heartbeat check.", ), responses( (status = 200, body = StatusResponse), @@ -216,18 +232,8 @@ async fn create( )); } - // Tell the device which backup types to run now (operator one-offs + - // schedule-due), riding the heartbeat response. Empty for an ungrouped - // server or one whose group has no `ready` backup config. - let backup_now = match server.group_id { - Some(group_id) => { - backups_due_now_for_server(&mut db, server_id, group_id, Timestamp::now()).await? - } - None => Vec::new(), - }; - let raw = body.map(|j| j.0).unwrap_or(serde_json::Value::Null); - let (healthy, health, extra) = split_health_from_extra(raw)?; + let (source, healthy, health, extra) = split_health_from_extra(raw)?; // The server version canopy tracks (and compares against the published // version catalog) is the Tamanu version. Prefer the payload's @@ -235,22 +241,17 @@ async fn create( // reporters that predate carrying it in the body. Either may be absent. let version = resolve_version(&extra, current_version.map(|v| v.0)); - let Some(health) = health else { - // Legacy format (no `health` array). Off by default — only servers an - // operator has explicitly opted in via `allow_legacy_status` may use - // it, until their reporter speaks the new format. - if !server.allow_legacy_status { - return Err(AppError::BadRequest("`health` array is required".into())); - } - create_legacy_status(&mut db, server_id, id, extra, version).await?; - let check_severities = - effective_check_severities(&mut db, server_id, server.group_id).await?; - let tags = crate::tags::effective_tags_for_server(&mut db, &server).await?; - return Ok(Json(StatusResponse { - backup_now, - check_severities, - tags, - })); + // Legacy format (no `health` array): Tamanu's direct reporting. It + // becomes a heartbeat from the `tamanu` source — a single `tasks` + // check that always passes on receipt — and flows through the normal + // path from here, so it records state, registers its catalog entry, + // and participates in source staleness like any source. + let (source, health) = match health { + Some(health) => (source, health), + None => ( + LEGACY_SOURCE.to_string(), + serde_json::json!([{ "check": LEGACY_CHECK, "result": "passed" }]), + ), }; // Resolve the server's effective tag map outside the write transaction. @@ -273,19 +274,33 @@ async fn create( extra, healthy, health, + source: source.clone(), } .save(conn) .await?; - file_health_events(conn, server_id, Some(id), &status, &tags).await?; + file_health_events(conn, server_id, server.group_id, Some(id), &status, &tags).await?; Ok(()) }) .await?; + // Tell the device which backup types to run now (operator one-offs + + // schedule-due), riding the heartbeat response. Only alertd runs + // backups — other sources (the tamanu heartbeat, seedling) would + // treat an instruction they can't act on as noise at best. Empty for + // an ungrouped server or one whose group has no `ready` backup config. + let backup_now = match server.group_id { + Some(group_id) if source == DEFAULT_SOURCE => { + backups_due_now_for_server(&mut db, server_id, group_id, Timestamp::now()).await? + } + _ => Vec::new(), + }; + // Computed after the transaction so checks first seen on this very push // (upserted into the catalog above) are already in the map. - let check_severities = effective_check_severities(&mut db, server_id, server.group_id).await?; + let check_severities = + effective_check_severities(&mut db, server_id, server.group_id, &source).await?; let effective_tags = crate::tags::effective_tags_for_server(&mut db, &server).await?; Ok(Json(StatusResponse { @@ -297,14 +312,16 @@ async fn create( /// Fetch the effective healthcheck severity mapping for a server. /// -/// Returns, for every healthcheck canopy knows about, how a failure of that -/// check is classified for this server: `skip` (the check is silenced for -/// this server — at server or group scope — or its severity is below -/// warning), `warn` (warning), or `fail` (error or critical). Keys are check -/// names as reported in `health[].check` on status pushes. Only the static -/// severity baseline is reflected; operator-defined conditional rules are -/// evaluated per push and not included here. The same mapping also rides -/// along every status-push response as `check_severities`. +/// Returns, for every healthcheck the `alertd` source reports, how that +/// check is handled for this server: `skip` (the check is silenced for +/// this server — at server or group scope — or its policy ceiling means it +/// never alerts), `warn` (graded at most a warning), or `fail` (failures +/// count as failures). Keys are check names as reported in +/// `health[].check` on status pushes. Only the static policy ceiling is +/// reflected; operator-defined conditional rules are evaluated per push +/// and not included here. The same mapping also rides along every +/// status-push response as `check_severities`, scoped to the pushing +/// source. /// /// The calling device must be the one enrolled for this exact server (or /// hold the admin role). @@ -339,71 +356,39 @@ async fn check_severities( )); } - let map = effective_check_severities(&mut db, server_id, server.group_id).await?; + let map = + effective_check_severities(&mut db, server_id, server.group_id, DEFAULT_SOURCE).await?; Ok(Json(map)) } -/// Build the effective per-check severity map for a server: every check in -/// the operator catalog mapped from its static base severity (error and -/// above → `fail`, warning → `warn`, below warning → `skip`), then any check -/// silenced for this server (at server or group scope) forced to `skip`. -/// Conditional severity rules are deliberately not consulted — they depend -/// on each push's contents, so only the static baseline can be mapped ahead -/// of time. +/// Build the effective per-check map for a server and source: every check +/// in the source's catalog mapped from its static policy ceiling (`failed` +/// → `fail`, `warning`/`broken` → `warn`, `passed`/`skipped` → `skip`), +/// then any check silenced for this server (at server or group scope) +/// forced to `skip`. Conditional rules are deliberately not consulted — +/// they depend on each push's contents, so only the static ceiling can be +/// mapped ahead of time. async fn effective_check_severities( db: &mut AsyncPgConnection, server_id: Uuid, group_id: Option, + source: &str, ) -> Result> { - let mut map: BTreeMap = HealthcheckSeverity::base_severity_map(db) + let mut map: BTreeMap = CheckPolicy::ceiling_map_for_source(db, source) .await? .into_iter() - .map(|(name, severity)| (name, severity.into())) + .map(|(name, ceiling)| (name, ceiling.into())) .collect(); - let health_prefix = format!("{HEALTH_REF}/"); - for r#ref in - silenced_refs_with_prefix(db, server_id, group_id, STATUS_SOURCE, &health_prefix).await? - { - if let Some(check) = r#ref.strip_prefix(&health_prefix) { - map.insert(check.to_string(), CheckSeverity::Skip); - } + // Silences are keyed per (source, check): only this source's own + // silences force its checks to skip. + for check in silenced_health_checks_for_server(db, server_id, group_id, source).await? { + map.insert(check, CheckSeverity::Skip); } Ok(map) } -/// Store a legacy-format push (no `health` array) for a server that's opted -/// into [`Server::allow_legacy_status`]. The push only refreshes reachability: -/// the new row carries the server's last known `healthy`/`health` forward -/// (defaulting to "healthy, no checks" if the server has never reported the -/// new format) rather than wiping them, and no health events are filed. So a -/// server straddling an old and a new reporter doesn't flap its per-check -/// issues every time the legacy endpoint pings. -async fn create_legacy_status( - db: &mut AsyncPgConnection, - server_id: Uuid, - device_id: Uuid, - extra: serde_json::Value, - version: Option, -) -> Result<()> { - let (healthy, health) = match Status::latest_for_server(db, server_id).await? { - Some(prior) => (prior.healthy, prior.health), - None => (true, serde_json::Value::Array(Vec::new())), - }; - NewStatus { - server_id, - device_id: Some(device_id), - version, - extra, - healthy, - health, - } - .save(db) - .await?; - Ok(()) -} - /// Resolve the server version to record on this status. Prefers the payload's /// `tamanuVersion` extra (the version bestool now carries in the body), parsed /// as a semver; falls back to the `X-Version` header for reporters that still @@ -421,23 +406,28 @@ fn resolve_version(extra: &serde_json::Value, header: Option) -> Opt /// Per-push event filing. Warning/failed checks land at /// `(status, health/)`; recoveries close those issues. Broken /// checks (`result: broken` — the check itself errored, not the -/// system under test) land at `(status, health-broken/)` at a -/// fixed Warning; a broken check neither confirms nor clears a known -/// failure, so its `health/` issue stays open. Skipped checks +/// system under test) neither confirm nor clear a known failure: the +/// check's issue stays open, retaining its contribution. Skipped checks /// (`result: skipped` — precondition not met) file nothing and close -/// both refs. +/// the check's issue. /// -/// Severity for each warning/failed check comes from the -/// operator-owned `healthcheck_severities` catalog (see -/// [`HealthcheckSeverity::severity_for`] for the rules/fallback -/// contract). Every check seen on a push — whatever its result — -/// upserts a default catalog row so new checks are visible to -/// operators immediately at the default Warning severity. -/// `status.healthy` is intentionally not consulted: the catalog is -/// canopy's single source of truth for per-check severity. +/// Each check's effective result comes from applying the operator-owned +/// `check_policies` catalog entry for `(source, check)` (see +/// [`CheckPolicy::apply`] for the rules/ceiling contract) to the +/// observed result. Every check seen on a push — whatever its result — +/// upserts a default catalog row so new checks are visible to operators +/// immediately at the default warning ceiling. `status.healthy` is +/// intentionally not consulted: the catalog is canopy's single source +/// of truth for per-check grading. +/// +/// Until issues themselves carry results, the effective result maps to +/// the issue severity: failed → error (critical when the policy +/// escalates), warning and broken → warning; passed and skipped file +/// nothing and close prior issues. async fn file_health_events( conn: &mut AsyncPgConnection, server_id: Uuid, + group_id: Option, device_id: Option, status: &Status, tags: &std::collections::HashMap, @@ -446,22 +436,20 @@ async fn file_health_events( let occurred_at = Some(status.created_at); // Upsert a catalog row for every check name seen on this push, - // whatever its result. New checks land at default Warning; - // operators can review and adjust from the /healthchecks page. + // whatever its result. New checks land at the default warning + // ceiling; operators can review and adjust from the /healthchecks + // page. for check_name in curr_check_results.keys() { - HealthcheckSeverity::upsert_default(conn, check_name).await?; + CheckPolicy::upsert_default(conn, &status.source, check_name).await?; } // Status-level extras are shared across every per-check evaluation. let empty_map = serde_json::Map::new(); let status_extra = status.extra.as_object().unwrap_or(&empty_map); - // Per-check opens: warning and failed file on the same ref (one - // thread per check; the filed severity is what differs). - for (check, result) in curr_check_results - .iter() - .filter(|(_, r)| matches!(r, CheckResult::Warning | CheckResult::Failed)) - { + // Grade every check in the push through its policy. + let mut effective: BTreeMap<&String, GradedResult> = BTreeMap::new(); + for (check, result) in &curr_check_results { let entry = find_health_entry(&status.health, check); // Strip the reserved `check` / `healthy` keys, and replace any // wire-form `result` with the normalised value so rules see a @@ -479,110 +467,164 @@ async fn file_health_events( check_extra: &check_extra, tags, }; - let severity = HealthcheckSeverity::severity_for(conn, check, *result, &ctx).await?; - let described = match result { - CheckResult::Warning => "warned", - _ => "failed", - }; - NewEvent { - source: STATUS_SOURCE.into(), - r#ref: format!("{HEALTH_REF}/{check}"), - severity: Some(severity), - description: Some(format!("Health check '{check}' {described}")), - message: per_check_description(entry).unwrap_or_default(), - active: Some(true), - occurred_at, - } - .save(conn, server_id, device_id) - .await?; - } - - // Broken opens: separate ref, fixed Warning. A broken check is a - // monitoring blind spot — actionable (fix the check), but not the - // system under test failing, so it doesn't inherit the catalog - // severity and can't open incidents. - for (check, _) in curr_check_results - .iter() - .filter(|(_, r)| matches!(r, CheckResult::Broken)) - { - let entry = find_health_entry(&status.health, check); - NewEvent { - source: STATUS_SOURCE.into(), - r#ref: format!("{BROKEN_REF}/{check}"), - severity: Some(Severity::Warning), - description: Some(format!("Health check '{check}' is broken")), - message: per_check_description(entry).unwrap_or_default(), - active: Some(true), - occurred_at, - } - .save(conn, server_id, device_id) + let graded = CheckPolicy::apply_scoped( + conn, + &status.source, + check, + *result, + &ctx, + Some(server_id), + group_id, + ) .await?; + effective.insert(check, graded); } - // Per-check closes are derived from the issues that are actually - // open, not from the previous status row: an issue can stay open - // across pushes that don't re-file it (failed → broken keeps the - // failure open), so the previous push alone can't tell us what - // needs closing. - - // Failure closes: an open `health/` closes when the check - // is now passed, skipped, or unmentioned ("trust the reporter"). - // Broken does NOT close a prior failure — the check can't confirm - // the failure either way while it's broken. + // The pushing source's previously-open issues: consulted for close + // messages ("recovered" vs "was never trouble") and for the + // unmentioned-check closes below. let health_prefix = format!("{HEALTH_REF}/"); - for r#ref in - Issue::active_refs_with_prefix(conn, server_id, STATUS_SOURCE, &health_prefix).await? - { - let Some(check) = r#ref.strip_prefix(&health_prefix) else { - continue; + let previously_active: std::collections::BTreeSet = + Issue::active_refs_with_prefix(conn, server_id, &status.source, &health_prefix) + .await? + .into_iter() + .filter_map(|r| { + r.strip_prefix(&health_prefix) + .map(|check| check.to_string()) + }) + .collect(); + + // File every check in the push — passing ones included, so the state + // row records the current result and when it was last reported. An + // effective broken result neither confirms nor clears the previous + // definite result: the filing retains an open effective failure's + // contribution, or counts as a warning when there was nothing to + // retain (broken contributes as a warning in the rollups). + // + // Degraded checks file before recoveries: when one failure swaps for + // another in a single push, the incoming failure must join the open + // incident before the outgoing one leaves, or the incident closes + // and reopens as two. + let filing_order = effective.iter().filter(|(_, g)| { + matches!( + g.effective, + CheckResult::Warning | CheckResult::Failed | CheckResult::Broken + ) + }); + let filing_order = filing_order.chain( + effective + .iter() + .filter(|(_, g)| matches!(g.effective, CheckResult::Passed | CheckResult::Skipped)), + ); + for (check, graded) in filing_order { + let was_active = previously_active.contains(*check); + let (effective, escalates, active, description, message) = match graded.effective { + CheckResult::Failed => ( + CheckResult::Failed, + graded.escalates, + true, + Some(format!("Health check '{check}' failed")), + None, + ), + CheckResult::Warning => ( + CheckResult::Warning, + graded.escalates, + true, + Some(format!("Health check '{check}' warned")), + None, + ), + CheckResult::Broken => { + let retained = Issue::list_by_source_ref( + conn, + &status.source, + &format!("{HEALTH_REF}/{check}"), + &[server_id], + ) + .await? + .into_iter() + .next() + .filter(|i| i.active && i.effective_result == Some(CheckResult::Failed)); + let (effective, escalates) = match retained { + Some(prior) => (CheckResult::Failed, prior.escalates), + None => (CheckResult::Broken, graded.escalates), + }; + ( + effective, + escalates, + true, + Some(format!("Health check '{check}' is broken")), + None, + ) + } + CheckResult::Passed => ( + CheckResult::Passed, + graded.escalates, + false, + None, + Some(if was_active { + format!("Health check '{check}' recovered") + } else { + format!("Health check '{check}' passing") + }), + ), + CheckResult::Skipped => ( + CheckResult::Skipped, + graded.escalates, + false, + None, + Some(if was_active { + format!("Health check '{check}' is now skipped") + } else { + format!("Health check '{check}' skipped") + }), + ), }; - let curr = curr_check_results.get(check); - if matches!( - curr, - Some(CheckResult::Warning | CheckResult::Failed | CheckResult::Broken) - ) { - continue; - } - let message = if matches!(curr, Some(CheckResult::Skipped)) { - format!("Health check '{check}' is now skipped") - } else { - format!("Health check '{check}' recovered") + let entry = find_health_entry(&status.health, check); + let stamp = CheckStateStamp { + check: (*check).clone(), + observed: curr_check_results[*check], + effective, + escalates, + detail: entry.cloned().map(serde_json::Value::Object), }; NewEvent { - source: STATUS_SOURCE.into(), - r#ref, - severity: Some(Severity::Info), - description: None, - message, - active: Some(false), + source: status.source.clone(), + r#ref: format!("{HEALTH_REF}/{check}"), + description, + message: message + .or_else(|| per_check_description(entry)) + .unwrap_or_default(), + active: Some(active), occurred_at, } - .save(conn, server_id, device_id) + .save_with_state(conn, server_id, device_id, Some(&stamp)) .await?; } - // Broken closes: any result other than broken (or absence) means - // the check itself is no longer broken. - let broken_prefix = format!("{BROKEN_REF}/"); - for r#ref in - Issue::active_refs_with_prefix(conn, server_id, STATUS_SOURCE, &broken_prefix).await? - { - let Some(check) = r#ref.strip_prefix(&broken_prefix) else { - continue; - }; - if matches!(curr_check_results.get(check), Some(CheckResult::Broken)) { + // Unmentioned closes: a check the source previously reported but + // omits from this push has recovered ("trust the reporter"). Scoped + // to the pushing source: one source's push says nothing about + // another's checks. + for check in &previously_active { + if curr_check_results.contains_key(check) { continue; } + let stamp = CheckStateStamp { + check: check.clone(), + observed: CheckResult::Passed, + effective: CheckResult::Passed, + escalates: false, + detail: None, + }; NewEvent { - source: STATUS_SOURCE.into(), - r#ref: format!("{BROKEN_REF}/{check}"), - severity: Some(Severity::Info), + source: status.source.clone(), + r#ref: format!("{HEALTH_REF}/{check}"), description: None, - message: format!("Health check '{check}' is no longer broken"), + message: format!("Health check '{check}' recovered"), active: Some(false), occurred_at, } - .save(conn, server_id, device_id) + .save_with_state(conn, server_id, device_id, Some(&stamp)) .await?; } @@ -639,11 +681,15 @@ fn per_check_description( (!lines.is_empty()).then(|| lines.join("\n")) } -/// Pulls the reserved `healthy` and `health` keys out of the incoming -/// status body and returns them alongside the rest of the payload +/// Pulls the reserved `source`, `healthy`, and `health` keys out of the +/// incoming status body and returns them alongside the rest of the payload /// (`extra`). Validates types per the contract: /// -/// - missing or `null` body → `healthy = true`, `health = []`, `extra = {}` +/// - missing or `null` body → `source = alertd`, `healthy = true`, +/// `health = []`, `extra = {}` +/// - `source` absent ⇒ `alertd` (transitional — the field will become +/// mandatory); present must be a non-empty string and not one of the +/// reserved names (`canopy`, `manual`) /// - `healthy` absent ⇒ `true` (legacy compat — non-negotiable, this is /// what stops every legacy server from false-positiving unhealthy on /// the day we deploy) @@ -657,7 +703,7 @@ fn per_check_description( /// verbatim. fn split_health_from_extra( raw: serde_json::Value, -) -> Result<(bool, Option, serde_json::Value)> { +) -> Result<(String, bool, Option, serde_json::Value)> { let mut obj = match raw { serde_json::Value::Null => serde_json::Map::new(), serde_json::Value::Object(m) => m, @@ -668,18 +714,34 @@ fn split_health_from_extra( } }; + let source = match obj.remove("source") { + None => DEFAULT_SOURCE.to_string(), + Some(serde_json::Value::String(s)) if !s.is_empty() => { + if RESERVED_SOURCES.iter().any(|r| s.eq_ignore_ascii_case(r)) { + return Err(AppError::BadRequest(format!( + "`source` must not be a reserved name ({})", + RESERVED_SOURCES.join(", "), + ))); + } + s + } + Some(_) => { + return Err(AppError::BadRequest( + "`source` must be a non-empty string".into(), + )); + } + }; + let healthy = match obj.remove("healthy") { None => true, Some(serde_json::Value::Bool(b)) => b, Some(_) => return Err(AppError::BadRequest("`healthy` must be a boolean".into())), }; - // A push without a `health` key is the retired legacy format. We don't - // reject it here — the caller decides, per the server's - // `allow_legacy_status` flag, whether to accept it (reachability-only, - // carrying prior healthchecks forward) or 400 it. + // A push without a `health` key is the legacy Tamanu direct-report + // format; the caller transforms it into the tamanu/tasks heartbeat. let Some(health_value) = obj.remove("health") else { - return Ok((healthy, None, serde_json::Value::Object(obj))); + return Ok((source, healthy, None, serde_json::Value::Object(obj))); }; let health_arr = match health_value { serde_json::Value::Array(a) => a, @@ -732,6 +794,7 @@ fn split_health_from_extra( } Ok(( + source, healthy, Some(serde_json::Value::Array(health_arr)), serde_json::Value::Object(obj), diff --git a/crates/public-server/tests/it/check_severities.rs b/crates/public-server/tests/it/check_severities.rs index 6e407f34..7dcec9c6 100644 --- a/crates/public-server/tests/it/check_severities.rs +++ b/crates/public-server/tests/it/check_severities.rs @@ -37,22 +37,22 @@ async fn insert_server( server_id } -/// Seed a catalog row at a given severity, optionally with a conditional +/// Seed a policy row at a given ceiling, optionally with a conditional /// rules ladder (which the mapping must ignore). async fn seed_catalog( conn: &mut diesel_async::AsyncPgConnection, check_name: &str, - severity: &str, + ceiling: &str, rules: Option, ) { sql_query( - "INSERT INTO healthcheck_severities (check_name, severity, rules, reviewed_at, reviewed_by) \ - VALUES ($1, $2, $3, NOW(), 'test') \ - ON CONFLICT (check_name) DO UPDATE \ - SET severity = EXCLUDED.severity, rules = EXCLUDED.rules", + "INSERT INTO check_policies (source, check_name, ceiling, rules, reviewed_at, reviewed_by) \ + VALUES ('alertd', $1, $2, $3, NOW(), 'test') \ + ON CONFLICT (source, check_name) DO UPDATE \ + SET ceiling = EXCLUDED.ceiling, rules = EXCLUDED.rules", ) .bind::(check_name) - .bind::(severity) + .bind::(ceiling) .bind::, _>(rules) .execute(conn) .await @@ -67,42 +67,36 @@ async fn endpoint_maps_catalog_severities_and_silences() { let group_id = insert_group(&mut conn).await; let server_id = insert_server(&mut conn, Some(device_id), Some(group_id)).await; - seed_catalog(&mut conn, "disk_space", "error", None).await; + seed_catalog(&mut conn, "disk_space", "failed", None).await; seed_catalog(&mut conn, "cert_expiry", "warning", None).await; - seed_catalog(&mut conn, "chatty", "info", None).await; - seed_catalog(&mut conn, "verbose", "debug", None).await; - seed_catalog(&mut conn, "flaky", "critical", None).await; - seed_catalog(&mut conn, "groupwide", "error", None).await; - // A conditional ladder that would escalate to critical at push + seed_catalog(&mut conn, "chatty", "passed", None).await; + seed_catalog(&mut conn, "verbose", "skipped", None).await; + seed_catalog(&mut conn, "flaky", "failed", None).await; + seed_catalog(&mut conn, "groupwide", "failed", None).await; + // A conditional ladder that would grade up to failed at push // time must not leak into the static mapping. seed_catalog( &mut conn, "ruled", "warning", Some(serde_json::json!({"if": [ - {"==": [{"var": "check.result"}, "failed"]}, "critical", + {"==": [{"var": "check.result"}, "failed"]}, "failed", ]})), ) .await; // Silences: flaky at server scope, groupwide at group scope; a - // silence on disk_space's *broken* thread must not skip the - // check itself. - ServerSilencedRef::add(&mut conn, server_id, "status", "health/flaky", None) + // silence on a canopy check (a reserved source, outside the + // health/ namespace) must not leak into alertd's map. + ServerSilencedRef::add(&mut conn, server_id, "alertd", "health/flaky", None) .await .expect("server silence"); - ServerGroupSilencedRef::add(&mut conn, group_id, "status", "health/groupwide", None) + ServerGroupSilencedRef::add(&mut conn, group_id, "alertd", "health/groupwide", None) .await .expect("group silence"); - ServerSilencedRef::add( - &mut conn, - server_id, - "status", - "health-broken/disk_space", - None, - ) - .await - .expect("broken silence"); + ServerSilencedRef::add(&mut conn, server_id, "canopy", "reachability", None) + .await + .expect("canopy silence"); let response = public .get(&format!("/status/{server_id}/check-severities")) @@ -133,8 +127,8 @@ async fn endpoint_works_for_ungrouped_server() { "server", async |mut conn, cert, device_id, public, _| { let server_id = insert_server(&mut conn, Some(device_id), None).await; - seed_catalog(&mut conn, "disk_space", "error", None).await; - ServerSilencedRef::add(&mut conn, server_id, "status", "health/disk_space", None) + seed_catalog(&mut conn, "disk_space", "failed", None).await; + ServerSilencedRef::add(&mut conn, server_id, "alertd", "health/disk_space", None) .await .expect("silence"); @@ -158,9 +152,9 @@ async fn status_response_carries_check_severities() { let group_id = insert_group(&mut conn).await; let server_id = insert_server(&mut conn, Some(device_id), Some(group_id)).await; - seed_catalog(&mut conn, "disk_space", "error", None).await; - seed_catalog(&mut conn, "cert_expiry", "critical", None).await; - ServerSilencedRef::add(&mut conn, server_id, "status", "health/cert_expiry", None) + seed_catalog(&mut conn, "disk_space", "failed", None).await; + seed_catalog(&mut conn, "cert_expiry", "failed", None).await; + ServerSilencedRef::add(&mut conn, server_id, "alertd", "health/cert_expiry", None) .await .expect("silence"); diff --git a/crates/public-server/tests/it/events.rs b/crates/public-server/tests/it/events.rs deleted file mode 100644 index 6cc71c99..00000000 --- a/crates/public-server/tests/it/events.rs +++ /dev/null @@ -1,481 +0,0 @@ -use diesel::{QueryableByName, sql_query, sql_types}; -use diesel_async::RunQueryDsl; -use uuid::Uuid; - -#[derive(QueryableByName)] -#[allow(dead_code)] -struct IssueRow { - #[diesel(sql_type = sql_types::Uuid)] - id: Uuid, - #[diesel(sql_type = sql_types::Uuid)] - server_id: Uuid, - #[diesel(sql_type = sql_types::Nullable)] - device_id: Option, - #[diesel(sql_type = sql_types::Text)] - source: String, - #[diesel(sql_type = sql_types::Text)] - severity: String, - #[diesel(sql_type = sql_types::Text)] - message: String, - #[diesel(sql_type = sql_types::Bool)] - active: bool, -} - -async fn provision_server(conn: &mut diesel_async::AsyncPgConnection, device_id: Uuid) -> Uuid { - // Put each provisioned server in its own group so events can promote - // to incidents (incidents are group-keyed; ungrouped servers don't). - let group_id = Uuid::new_v4(); - sql_query("INSERT INTO server_groups (id, name) VALUES ($1, 'test-group')") - .bind::(group_id) - .execute(conn) - .await - .expect("insert group"); - let server_id = Uuid::new_v4(); - sql_query( - "INSERT INTO servers (id, host, kind, device_id, group_id) \ - VALUES ($1, 'https://test.example.com', 'central', $2, $3)", - ) - .bind::(server_id) - .bind::(device_id) - .bind::(group_id) - .execute(conn) - .await - .expect("insert server"); - server_id -} - -async fn issue_by_id(conn: &mut diesel_async::AsyncPgConnection, id: Uuid) -> IssueRow { - sql_query( - "SELECT id, server_id, device_id, source, severity, message, active \ - FROM issues WHERE id = $1", - ) - .bind::(id) - .get_result(conn) - .await - .expect("fetch issue") -} - -#[tokio::test(flavor = "multi_thread")] -async fn submit_event_creates_issue() { - commons_tests::server::run_with_device_auth( - "server", - async |mut conn, cert, device_id, public, _| { - let server_id = provision_server(&mut conn, device_id).await; - - let response = public - .post("/events") - .add_header("mtls-certificate", &cert) - .json(&serde_json::json!({ - "source": "watchdog", - "ref": "disk-/var", - "message": "less than 5% free", - })) - .await; - response.assert_status_ok(); - - let body: serde_json::Value = response.json(); - let issue_id = Uuid::parse_str(body.get("id").unwrap().as_str().unwrap()).unwrap(); - assert_eq!( - body.get("server_id").and_then(|v| v.as_str()), - Some(server_id.to_string().as_str()) - ); - assert_eq!( - body.get("severity").and_then(|v| v.as_str()), - Some("error") // default - ); - assert_eq!(body.get("active").and_then(|v| v.as_bool()), Some(true)); - - let row = issue_by_id(&mut conn, issue_id).await; - assert_eq!(row.server_id, server_id); - assert_eq!(row.device_id, Some(device_id)); - assert_eq!(row.source, "watchdog"); - assert_eq!(row.severity, "error"); - assert_eq!(row.message, "less than 5% free"); - assert!(row.active); - }, - ) - .await -} - -#[tokio::test(flavor = "multi_thread")] -async fn submit_event_dedups_by_ref() { - commons_tests::server::run_with_device_auth( - "server", - async |mut conn, cert, device_id, public, _| { - let server_id = provision_server(&mut conn, device_id).await; - - let first = public - .post("/events") - .add_header("mtls-certificate", &cert) - .json(&serde_json::json!({ - "source": "watchdog", - "ref": "disk-/var", - "message": "less than 5% free", - })) - .await; - first.assert_status_ok(); - let first_id = first - .json::() - .get("id") - .unwrap() - .as_str() - .unwrap() - .to_string(); - - let second = public - .post("/events") - .add_header("mtls-certificate", &cert) - .json(&serde_json::json!({ - "source": "watchdog", - "ref": "disk-/var", - "severity": "critical", - "message": "less than 1% free", - })) - .await; - second.assert_status_ok(); - let second_id = second - .json::() - .get("id") - .unwrap() - .as_str() - .unwrap() - .to_string(); - - assert_eq!( - first_id, second_id, - "same (server, source, ref) should be one issue" - ); - - let row = issue_by_id(&mut conn, Uuid::parse_str(&first_id).unwrap()).await; - assert_eq!(row.server_id, server_id); - assert_eq!(row.severity, "critical"); - assert_eq!(row.message, "less than 1% free"); - - #[derive(QueryableByName)] - struct Counts { - #[diesel(sql_type = sql_types::BigInt)] - event_count: i64, - } - let counts: Counts = - sql_query("SELECT COUNT(*) AS event_count FROM events WHERE issue_id = $1") - .bind::(Uuid::parse_str(&first_id).unwrap()) - .get_result(&mut conn) - .await - .expect("count events"); - assert_eq!(counts.event_count, 2, "different content → two event rows"); - }, - ) - .await -} - -#[tokio::test(flavor = "multi_thread")] -async fn submit_event_coalesces_identical_pushes() { - commons_tests::server::run_with_device_auth( - "server", - async |mut conn, cert, device_id, public, _| { - provision_server(&mut conn, device_id).await; - - // Three identical pushes. - let mut issue_id = String::new(); - for _ in 0..3 { - let r = public - .post("/events") - .add_header("mtls-certificate", &cert) - .json(&serde_json::json!({ - "source": "watchdog", - "ref": "x", - "message": "same content", - })) - .await; - r.assert_status_ok(); - issue_id = r.json::().get("id").unwrap().as_str().unwrap().to_string(); - } - - #[derive(QueryableByName)] - struct Counts { - #[diesel(sql_type = sql_types::BigInt)] - event_count: i64, - #[diesel(sql_type = sql_types::Integer)] - latest_occurrences: i32, - } - let counts: Counts = sql_query( - "SELECT \ - (SELECT COUNT(*) FROM events WHERE issue_id = $1) AS event_count, \ - (SELECT occurrences FROM events WHERE issue_id = $1 ORDER BY created_at DESC LIMIT 1) AS latest_occurrences", - ) - .bind::(Uuid::parse_str(&issue_id).unwrap()) - .get_result(&mut conn) - .await - .expect("count"); - assert_eq!(counts.event_count, 1, "identical pushes collapse into one row"); - assert_eq!(counts.latest_occurrences, 3); - }, - ) - .await -} - -#[tokio::test(flavor = "multi_thread")] -async fn submit_event_active_false_resolves_issue() { - commons_tests::server::run_with_device_auth( - "server", - async |mut conn, cert, device_id, public, _| { - provision_server(&mut conn, device_id).await; - - let opened = public - .post("/events") - .add_header("mtls-certificate", &cert) - .json(&serde_json::json!({ - "source": "watchdog", - "ref": "x", - "message": "trouble", - })) - .await; - opened.assert_status_ok(); - let issue_id = opened - .json::() - .get("id") - .unwrap() - .as_str() - .unwrap() - .to_string(); - - let resolved = public - .post("/events") - .add_header("mtls-certificate", &cert) - .json(&serde_json::json!({ - "source": "watchdog", - "ref": "x", - "active": false, - "message": "trouble", - })) - .await; - resolved.assert_status_ok(); - assert_eq!( - resolved - .json::() - .get("id") - .unwrap() - .as_str() - .unwrap(), - issue_id - ); - - let row = issue_by_id(&mut conn, Uuid::parse_str(&issue_id).unwrap()).await; - assert!(!row.active, "issue should be inactive after active:false"); - }, - ) - .await -} - -#[tokio::test(flavor = "multi_thread")] -async fn submit_event_rejects_manual_source() { - commons_tests::server::run_with_device_auth( - "server", - async |mut conn, cert, device_id, public, _| { - provision_server(&mut conn, device_id).await; - let response = public - .post("/events") - .add_header("mtls-certificate", &cert) - .json(&serde_json::json!({ - "source": "manual", - "ref": "x", - "message": "nope", - })) - .await; - assert_eq!(response.status_code().as_u16(), 400); - }, - ) - .await -} - -#[tokio::test(flavor = "multi_thread")] -async fn submit_event_rejects_device_with_no_server() { - commons_tests::server::run_with_device_auth( - "server", - async |_conn, cert, _device_id, public, _| { - // No server provisioned for this device. - let response = public - .post("/events") - .add_header("mtls-certificate", &cert) - .json(&serde_json::json!({ - "source": "watchdog", - "ref": "x", - "message": "should fail", - })) - .await; - assert_eq!(response.status_code().as_u16(), 412); - }, - ) - .await -} - -#[tokio::test(flavor = "multi_thread")] -async fn submit_event_rejects_invalid_severity() { - commons_tests::server::run_with_device_auth( - "server", - async |mut conn, cert, device_id, public, _| { - provision_server(&mut conn, device_id).await; - let response = public - .post("/events") - .add_header("mtls-certificate", &cert) - .json(&serde_json::json!({ - "source": "watchdog", - "ref": "x", - "severity": "kaboom", - "message": "should fail", - })) - .await; - assert!( - !response.status_code().is_success(), - "expected non-success, got {}", - response.status_code() - ); - }, - ) - .await -} - -#[tokio::test(flavor = "multi_thread")] -async fn submit_event_opens_incident_at_error() { - commons_tests::server::run_with_device_auth( - "server", - async |mut conn, cert, device_id, public, _| { - let server_id = provision_server(&mut conn, device_id).await; - - let r = public - .post("/events") - .add_header("mtls-certificate", &cert) - .json(&serde_json::json!({ - "source": "watchdog", - "ref": "x", - "severity": "error", - "message": "trouble", - })) - .await; - r.assert_status_ok(); - - #[derive(QueryableByName)] - struct IncidentCheck { - #[diesel(sql_type = sql_types::Uuid)] - id: Uuid, - #[diesel(sql_type = sql_types::Bool)] - is_open: bool, - } - let inc: IncidentCheck = sql_query( - "SELECT i.id, i.closed_at IS NULL AS is_open \ - FROM incidents i JOIN servers s ON i.server_group_id = s.group_id \ - WHERE s.id = $1", - ) - .bind::(server_id) - .get_result(&mut conn) - .await - .expect("one incident"); - assert!(inc.is_open, "incident should still be open"); - - #[derive(QueryableByName)] - struct LinkCheck { - #[diesel(sql_type = sql_types::BigInt)] - count: i64, - } - let links: LinkCheck = sql_query( - "SELECT COUNT(*) AS count FROM incident_issues WHERE incident_id = $1 AND left_at IS NULL", - ) - .bind::(inc.id) - .get_result(&mut conn) - .await - .expect("count"); - assert_eq!(links.count, 1); - }, - ) - .await -} - -#[tokio::test(flavor = "multi_thread")] -async fn submit_event_at_warning_does_not_open_incident() { - commons_tests::server::run_with_device_auth( - "server", - async |mut conn, cert, device_id, public, _| { - let server_id = provision_server(&mut conn, device_id).await; - - let r = public - .post("/events") - .add_header("mtls-certificate", &cert) - .json(&serde_json::json!({ - "source": "watchdog", - "ref": "x", - "severity": "warning", - "message": "minor", - })) - .await; - r.assert_status_ok(); - - #[derive(QueryableByName)] - struct Counts { - #[diesel(sql_type = sql_types::BigInt)] - count: i64, - } - let counts: Counts = sql_query( - "SELECT COUNT(*) AS count FROM incidents i \ - JOIN servers s ON i.server_group_id = s.group_id WHERE s.id = $1", - ) - .bind::(server_id) - .get_result(&mut conn) - .await - .expect("count"); - assert_eq!(counts.count, 0, "warning shouldn't open an incident"); - }, - ) - .await -} - -#[tokio::test(flavor = "multi_thread")] -async fn incident_closes_when_last_issue_resolves() { - commons_tests::server::run_with_device_auth( - "server", - async |mut conn, cert, device_id, public, _| { - let server_id = provision_server(&mut conn, device_id).await; - - public - .post("/events") - .add_header("mtls-certificate", &cert) - .json(&serde_json::json!({ - "source": "watchdog", - "ref": "x", - "severity": "error", - "message": "trouble", - })) - .await - .assert_status_ok(); - - public - .post("/events") - .add_header("mtls-certificate", &cert) - .json(&serde_json::json!({ - "source": "watchdog", - "ref": "x", - "severity": "error", - "active": false, - "message": "resolved", - })) - .await - .assert_status_ok(); - - #[derive(QueryableByName)] - struct IncidentCheck { - #[diesel(sql_type = sql_types::Bool)] - is_closed: bool, - } - let inc: IncidentCheck = sql_query( - "SELECT closed_at IS NOT NULL AS is_closed \ - FROM incidents i JOIN servers s ON i.server_group_id = s.group_id \ - WHERE s.id = $1", - ) - .bind::(server_id) - .get_result(&mut conn) - .await - .expect("incident"); - assert!(inc.is_closed, "incident should be closed"); - }, - ) - .await -} diff --git a/crates/public-server/tests/it/main.rs b/crates/public-server/tests/it/main.rs index 51afaf94..abbaf035 100644 --- a/crates/public-server/tests/it/main.rs +++ b/crates/public-server/tests/it/main.rs @@ -9,7 +9,6 @@ mod backup_secrets; mod check_severities; mod device_key_auth; mod error_scenarios; -mod events; mod health; mod index; mod mcp; diff --git a/crates/public-server/tests/it/openapi_spec.rs b/crates/public-server/tests/it/openapi_spec.rs index 4363ea36..7d405933 100644 --- a/crates/public-server/tests/it/openapi_spec.rs +++ b/crates/public-server/tests/it/openapi_spec.rs @@ -38,7 +38,6 @@ fn spec_has_one_path_per_module() { let spec = build_spec(); let paths = &spec["paths"]; for p in [ - "/events", "/servers", "/bestool/snippets", "/status/{server_id}", diff --git a/crates/public-server/tests/it/server_enrollment.rs b/crates/public-server/tests/it/server_enrollment.rs index 16fce39d..07820800 100644 --- a/crates/public-server/tests/it/server_enrollment.rs +++ b/crates/public-server/tests/it/server_enrollment.rs @@ -35,7 +35,6 @@ fn new_server(host: &str) -> Server { cloud: None, geolocation: None, is_monitored: true, - allow_legacy_status: false, alert_when_down_for: PgDuration(SignedDuration::from_secs(600)), notes: String::new(), tags: TagMap::default(), diff --git a/crates/public-server/tests/it/statuses.rs b/crates/public-server/tests/it/statuses.rs index 7c1e6712..41ab64d1 100644 --- a/crates/public-server/tests/it/statuses.rs +++ b/crates/public-server/tests/it/statuses.rs @@ -24,8 +24,8 @@ struct HealthResult { #[derive(QueryableByName, Debug)] struct IssueRow { - #[diesel(sql_type = sql_types::Text)] - severity: String, + #[diesel(sql_type = sql_types::Bool)] + escalates: bool, #[diesel(sql_type = sql_types::Bool)] active: bool, #[diesel(sql_type = sql_types::Text)] @@ -34,6 +34,10 @@ struct IssueRow { description: Option, #[diesel(sql_type = sql_types::Bool)] is_resolved: bool, + #[diesel(sql_type = sql_types::Nullable)] + observed_result: Option, + #[diesel(sql_type = sql_types::Nullable)] + effective_result: Option, } async fn fetch_issue( @@ -44,7 +48,8 @@ async fn fetch_issue( ) -> Option { sql_query( r#" - SELECT severity, active, message, description, (resolved_at IS NOT NULL) AS is_resolved + SELECT escalates, active, message, description, (resolved_at IS NOT NULL) AS is_resolved, + observed_result, effective_result FROM issues WHERE server_id = $1 AND source = $2 AND ref = $3 "#, @@ -81,29 +86,40 @@ struct IncidentRow { id: Uuid, } -/// Pre-seed (or update) a catalog row so a check's failure severity is -/// known up-front. v1 ingestion would otherwise auto-insert at the -/// default Warning, which only opens an incident when one already -/// exists. Tests that want to exercise Error-class behaviour seed -/// here. +/// Pre-seed (or update) a policy row so a check's grading is known +/// up-front, expressed in the old severity vocabulary the tests speak: +/// critical → failed + escalates, error → failed, warning → warning, +/// info → passed, debug → skipped. Ingestion would otherwise +/// auto-insert at the default warning ceiling, which only opens an +/// incident when one already exists. async fn set_check_severity( conn: &mut diesel_async::AsyncPgConnection, check_name: &str, severity: &str, ) { + let (ceiling, escalates) = match severity { + "critical" => ("failed", true), + "error" => ("failed", false), + "warning" => ("warning", false), + "info" => ("passed", false), + "debug" => ("skipped", false), + other => panic!("unknown severity {other}"), + }; sql_query( - "INSERT INTO healthcheck_severities (check_name, severity, reviewed_at, reviewed_by) \ - VALUES ($1, $2, NOW(), 'test') \ - ON CONFLICT (check_name) DO UPDATE \ - SET severity = EXCLUDED.severity, \ + "INSERT INTO check_policies (source, check_name, ceiling, escalates, reviewed_at, reviewed_by) \ + VALUES ('alertd', $1, $2, $3, NOW(), 'test') \ + ON CONFLICT (source, check_name) DO UPDATE \ + SET ceiling = EXCLUDED.ceiling, \ + escalates = EXCLUDED.escalates, \ reviewed_at = EXCLUDED.reviewed_at, \ reviewed_by = EXCLUDED.reviewed_by", ) .bind::(check_name) - .bind::(severity) + .bind::(ceiling) + .bind::(escalates) .execute(conn) .await - .expect("seed catalog severity"); + .expect("seed catalog policy"); } async fn fetch_open_incident( @@ -703,34 +719,6 @@ async fn submit_status_rejects_non_bool_healthy() { .await } -#[tokio::test(flavor = "multi_thread")] -async fn submit_status_rejects_missing_health() { - commons_tests::server::run_with_device_auth( - "server", - async |mut conn, cert, device_id, public, _| { - let server_id = insert_health_test_server(&mut conn, device_id).await; - - // A push that carries no `health` array at all is rejected - // outright — even though `healthy` and other fields are valid. - let response = public - .post(&format!("/status/{}", server_id)) - .add_header("mtls-certificate", &cert) - .json(&serde_json::json!({ "healthy": true, "uptime": 1 })) - .await; - response.assert_status_bad_request(); - - // An empty body (⇒ `{}`) is rejected for the same reason. - let response = public - .post(&format!("/status/{}", server_id)) - .add_header("mtls-certificate", &cert) - .json(&serde_json::json!({})) - .await; - response.assert_status_bad_request(); - }, - ) - .await -} - #[tokio::test(flavor = "multi_thread")] async fn submit_status_rejects_non_array_health() { commons_tests::server::run_with_device_auth( @@ -790,48 +778,67 @@ async fn submit_status_rejects_health_entry_missing_healthy() { } // ----------------------------------------------------------------- -// Legacy format opt-in (`servers.allow_legacy_status`). +// Legacy format (no `health` array): the tamanu/tasks heartbeat. // ----------------------------------------------------------------- -async fn enable_legacy_status(conn: &mut diesel_async::AsyncPgConnection, server_id: Uuid) { - sql_query("UPDATE servers SET allow_legacy_status = TRUE WHERE id = $1") - .bind::(server_id) - .execute(conn) - .await - .expect("enable legacy status"); -} - -/// Default (opt-out): a legacy push — no `health` array — is rejected even -/// when the rest of the body is valid. +/// A legacy push — no `health` array — is accepted unconditionally and +/// transformed into the `tamanu` source reporting a single always-passing +/// `tasks` heartbeat check, with the push's extras recorded verbatim. #[tokio::test(flavor = "multi_thread")] -async fn submit_status_legacy_rejected_without_optin() { +async fn submit_status_legacy_transforms_to_tamanu_heartbeat() { commons_tests::server::run_with_device_auth( "server", async |mut conn, cert, device_id, public, _| { let server_id = insert_health_test_server(&mut conn, device_id).await; - let response = public - .post(&format!("/status/{}", server_id)) - .add_header("mtls-certificate", &cert) - .json(&serde_json::json!({ "healthy": true, "uptime": 5 })) - .await; - response.assert_status_bad_request(); + post_status( + &public, + &cert, + server_id, + serde_json::json!({ "healthy": false, "uptime": 7 }), + ) + .await; + + let row = fetch_latest_health(&mut conn, server_id).await; + assert_eq!( + row.health, + serde_json::json!([{ "check": "tasks", "result": "passed" }]), + ); + assert_eq!(row.extra.get("uptime").and_then(|v| v.as_i64()), Some(7)); + + // The heartbeat records healthy state under the tamanu source. + let tasks = fetch_issue(&mut conn, server_id, "tamanu", "health/tasks") + .await + .expect("heartbeat state recorded"); + assert!(!tasks.active, "the heartbeat is not an issue"); + assert!(fetch_open_incident(&mut conn, server_id).await.is_none()); + + #[derive(QueryableByName)] + struct SourceRow { + #[diesel(sql_type = sql_types::Text)] + source: String, + } + let row: SourceRow = sql_query( + "SELECT source FROM statuses WHERE server_id = $1 ORDER BY created_at DESC LIMIT 1", + ) + .bind::(server_id) + .get_result(&mut conn) + .await + .expect("status row"); + assert_eq!(row.source, "tamanu"); }, ) .await } -/// With the opt-in on, a legacy push is accepted but does not disturb the -/// healthchecks: it carries the last real push's `health`/`healthy` forward -/// and files no events, so a failing check filed by an earlier new-style push -/// stays open instead of flapping closed. +/// A legacy push must not disturb another source's checks: with per-source +/// scoping, the tamanu heartbeat says nothing about alertd's open issues. #[tokio::test(flavor = "multi_thread")] -async fn submit_status_legacy_allowed_carries_health_forward() { +async fn submit_status_legacy_leaves_other_sources_alone() { commons_tests::server::run_with_device_auth( "server", async |mut conn, cert, device_id, public, _| { let server_id = insert_health_test_server(&mut conn, device_id).await; - enable_legacy_status(&mut conn, server_id).await; // New-style push with a failing check files a per-check issue. post_status( @@ -843,13 +850,12 @@ async fn submit_status_legacy_allowed_carries_health_forward() { }), ) .await; - let before = fetch_issue(&mut conn, server_id, "status", "health/disk") + let before = fetch_issue(&mut conn, server_id, "alertd", "health/disk") .await .expect("per-check issue filed"); assert!(before.active); - assert_eq!(count_issues_for_server(&mut conn, server_id).await, 1); - // Legacy push (no `health` array) only refreshes reachability. + // Legacy push: heartbeat under tamanu only. post_status( &public, &cert, @@ -858,57 +864,10 @@ async fn submit_status_legacy_allowed_carries_health_forward() { ) .await; - // The disk issue is untouched — not closed by the legacy push. - let after = fetch_issue(&mut conn, server_id, "status", "health/disk") + let after = fetch_issue(&mut conn, server_id, "alertd", "health/disk") .await .expect("per-check issue still present"); assert!(after.active, "legacy push must not close the failing check"); - assert_eq!( - count_issues_for_server(&mut conn, server_id).await, - 1, - "legacy push must not file or close any issue" - ); - - // The newest row carries the prior healthchecks forward (so the - // snapshot keeps showing them) while recording the legacy extras. - let row = fetch_latest_health(&mut conn, server_id).await; - let arr = row.health.as_array().expect("health carried forward"); - assert_eq!(arr.len(), 1); - assert_eq!(arr[0]["check"], "disk"); - assert_eq!(arr[0]["free_pct"], 4); - assert_eq!(row.extra.get("uptime").and_then(|v| v.as_i64()), Some(99)); - }, - ) - .await -} - -/// A server that has only ever spoken the legacy format (no prior row to carry -/// forward) is accepted and stored with an empty healthcheck set. -#[tokio::test(flavor = "multi_thread")] -async fn submit_status_legacy_allowed_without_prior_health() { - commons_tests::server::run_with_device_auth( - "server", - async |mut conn, cert, device_id, public, _| { - let server_id = insert_health_test_server(&mut conn, device_id).await; - enable_legacy_status(&mut conn, server_id).await; - - post_status( - &public, - &cert, - server_id, - serde_json::json!({ "healthy": false, "uptime": 7 }), - ) - .await; - - let row = fetch_latest_health(&mut conn, server_id).await; - assert_eq!(row.health, serde_json::json!([])); - assert!(row.healthy, "no prior row ⇒ defaults to healthy"); - assert_eq!(row.extra.get("uptime").and_then(|v| v.as_i64()), Some(7)); - assert_eq!( - count_issues_for_server(&mut conn, server_id).await, - 0, - "legacy push files no issues" - ); }, ) .await @@ -976,10 +935,10 @@ async fn submit_status_warning_check_only() { ) .await; - let per_check = fetch_issue(&mut conn, server_id, "status", "health/disk") + let per_check = fetch_issue(&mut conn, server_id, "alertd", "health/disk") .await .expect("per-check issue filed"); - assert_eq!(per_check.severity, "warning"); + assert_eq!(per_check.effective_result.as_deref(), Some("warning")); assert!(per_check.active); assert!(!per_check.is_resolved); assert!( @@ -992,7 +951,7 @@ async fn submit_status_warning_check_only() { // Rollup is retired — never filed in any case. assert!( - fetch_issue(&mut conn, server_id, "status", "health") + fetch_issue(&mut conn, server_id, "alertd", "health") .await .is_none() ); @@ -1033,26 +992,26 @@ async fn submit_status_unhealthy_with_checks_opens_incident() { // The (status, health) rollup was retired — only per-check issues now. assert!( - fetch_issue(&mut conn, server_id, "status", "health") + fetch_issue(&mut conn, server_id, "alertd", "health") .await .is_none(), "rollup issue must not be created" ); for check in ["database", "disk"] { - let i = fetch_issue(&mut conn, server_id, "status", &format!("health/{check}")) + let i = fetch_issue(&mut conn, server_id, "alertd", &format!("health/{check}")) .await .unwrap_or_else(|| panic!("per-check issue for {check} missing")); - assert_eq!(i.severity, "error", "{check}"); + assert_eq!(i.effective_result.as_deref(), Some("failed"), "{check}"); + assert!(!i.escalates, "{check}"); assert!(i.active, "{check}"); } - // Passing check shouldn't manifest as a resolved-from-birth issue. - assert!( - fetch_issue(&mut conn, server_id, "status", "health/tls") - .await - .is_none(), - "passing check must not create an issue" - ); + // Passing checks get a state row too — but an inactive one that + // never degraded, which the issue listings exclude. + let tls = fetch_issue(&mut conn, server_id, "alertd", "health/tls") + .await + .expect("passing check records state"); + assert!(!tls.active, "passing state is not an active issue"); // Per-check failures at the catalog's Error severity open an incident. assert!(fetch_open_incident(&mut conn, server_id).await.is_some()); }, @@ -1079,7 +1038,7 @@ async fn submit_status_unhealthy_no_checks_files_nothing() { // failures to file individual issues against, so the unhealthy flag // on its own produces nothing. assert!( - fetch_issue(&mut conn, server_id, "status", "health") + fetch_issue(&mut conn, server_id, "alertd", "health") .await .is_none(), "rollup issue must not be created" @@ -1104,11 +1063,11 @@ async fn submit_status_with_all_failing_checks_silenced_opens_no_incident() { // Pre-silence both failing checks at server scope. for check in ["database", "disk"] { sql_query( - "INSERT INTO server_silenced_refs (server_id, source, ref) \ - VALUES ($1, 'status', $2)", + "INSERT INTO scoped_check_policies (server_id, source, check_name, ceiling) \ + VALUES ($1, 'alertd', $2, 'skipped')", ) .bind::(server_id) - .bind::(format!("health/{check}")) + .bind::(check) .execute(&mut conn) .await .expect("seed silence"); @@ -1133,13 +1092,16 @@ async fn submit_status_with_all_failing_checks_silenced_opens_no_incident() { let row = fetch_latest_health(&mut conn, server_id).await; assert!(!row.healthy); - // Per-check issues exist (silence doesn't gate row creation) - // but the silence prevents them from joining an incident. + // Check state still records (silence doesn't gate row creation) + // with the observation intact, but the silence grades the + // effective result to skipped so nothing raises. for check in ["database", "disk"] { - let i = fetch_issue(&mut conn, server_id, "status", &format!("health/{check}")) + let i = fetch_issue(&mut conn, server_id, "alertd", &format!("health/{check}")) .await - .unwrap_or_else(|| panic!("per-check issue for {check} missing")); - assert!(i.active, "{check}"); + .unwrap_or_else(|| panic!("per-check state for {check} missing")); + assert!(!i.active, "{check} is silenced, so it must not raise"); + assert_eq!(i.observed_result.as_deref(), Some("failed"), "{check}"); + assert_eq!(i.effective_result.as_deref(), Some("skipped"), "{check}"); } assert!(fetch_open_incident(&mut conn, server_id).await.is_none()); }, @@ -1158,8 +1120,8 @@ async fn submit_status_with_partial_silence_opens_incident_for_unsilenced() { // Silence only one of the two failing checks. sql_query( - "INSERT INTO server_silenced_refs (server_id, source, ref) \ - VALUES ($1, 'status', 'health/database')", + "INSERT INTO scoped_check_policies (server_id, source, check_name, ceiling) \ + VALUES ($1, 'alertd', 'database', 'skipped')", ) .bind::(server_id) .execute(&mut conn) @@ -1186,16 +1148,17 @@ async fn submit_status_with_partial_silence_opens_incident_for_unsilenced() { let row = fetch_latest_health(&mut conn, server_id).await; assert!(!row.healthy); assert!( - fetch_issue(&mut conn, server_id, "status", "health") + fetch_issue(&mut conn, server_id, "alertd", "health") .await .is_none(), "rollup must not be created" ); // Disk is unsilenced and configured at Error severity → opens an incident. - let disk = fetch_issue(&mut conn, server_id, "status", "health/disk") + let disk = fetch_issue(&mut conn, server_id, "alertd", "health/disk") .await .expect("disk issue filed"); - assert_eq!(disk.severity, "error"); + assert_eq!(disk.effective_result.as_deref(), Some("failed")); + assert!(!disk.escalates); assert!(fetch_open_incident(&mut conn, server_id).await.is_some()); }, ) @@ -1226,10 +1189,10 @@ async fn submit_status_per_check_severity_is_catalog_driven() { }), ) .await; - let after_first = fetch_issue(&mut conn, server_id, "status", "health/disk") + let after_first = fetch_issue(&mut conn, server_id, "alertd", "health/disk") .await .expect("per-check issue"); - assert_eq!(after_first.severity, "warning"); + assert_eq!(after_first.effective_result.as_deref(), Some("warning")); // Second push: bestool reports overall healthy. Same severity — // the catalog is the source of truth, not the top-level flag. @@ -1243,14 +1206,14 @@ async fn submit_status_per_check_severity_is_catalog_driven() { }), ) .await; - let disk = fetch_issue(&mut conn, server_id, "status", "health/disk") + let disk = fetch_issue(&mut conn, server_id, "alertd", "health/disk") .await .expect("per-check issue still present"); - assert_eq!(disk.severity, "warning"); + assert_eq!(disk.effective_result.as_deref(), Some("warning")); assert!(disk.active, "still failing, must stay active"); assert!( - fetch_issue(&mut conn, server_id, "status", "health") + fetch_issue(&mut conn, server_id, "alertd", "health") .await .is_none(), "rollup is retired and must not exist" @@ -1288,7 +1251,7 @@ async fn submit_status_check_recovery_explicit() { ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/db") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await .expect("per-check issue exists"); assert!(!issue.active, "explicit healthy=true closes the issue"); @@ -1327,7 +1290,7 @@ async fn submit_status_check_recovery_via_drop() { ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/db") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await .expect("per-check issue exists"); assert!(!issue.active); @@ -1401,7 +1364,7 @@ async fn submit_status_reachability_to_health_handoff() { // Initial state: server silent → reachability sweep files a // canopy/reachability issue at Critical (severity for `Gone`), // which opens an incident on the server's group. - database::statuses::Status::sweep_reachability(&mut conn) + database::statuses::Status::sweep_staleness(&mut conn) .await .expect("reachability sweep"); let reach_before = fetch_issue(&mut conn, server_id, "canopy", "reachability") @@ -1432,7 +1395,7 @@ async fn submit_status_reachability_to_health_handoff() { ) .await; - let per_check = fetch_issue(&mut conn, server_id, "status", "health/db") + let per_check = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await .expect("per-check issue opened"); assert!(per_check.active); @@ -1447,7 +1410,7 @@ async fn submit_status_reachability_to_health_handoff() { // Reachability sweep runs again. Server's latest status is fresh // so the sweep closes the reachability issue. The incident must // stay open because the per-check issue is still contributing. - database::statuses::Status::sweep_reachability(&mut conn) + database::statuses::Status::sweep_staleness(&mut conn) .await .expect("reachability sweep (recovery)"); @@ -1516,11 +1479,11 @@ async fn submit_status_keeps_incident_open_when_failure_swaps() { "same incident, not a close+reopen flicker" ); - let db = fetch_issue(&mut conn, server_id, "status", "health/db") + let db = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await .expect("db issue"); assert!(!db.active, "db has recovered"); - let disk = fetch_issue(&mut conn, server_id, "status", "health/disk") + let disk = fetch_issue(&mut conn, server_id, "alertd", "health/disk") .await .expect("disk issue"); assert!(disk.active, "disk is new failure"); @@ -1532,7 +1495,9 @@ async fn submit_status_keeps_incident_open_when_failure_swaps() { #[derive(QueryableByName, Debug)] struct CatalogRow { #[diesel(sql_type = sql_types::Text)] - severity: String, + source: String, + #[diesel(sql_type = sql_types::Text)] + ceiling: String, #[diesel(sql_type = sql_types::Bool)] pending_review: bool, } @@ -1542,8 +1507,8 @@ async fn fetch_catalog( check_name: &str, ) -> Option { sql_query( - "SELECT severity, reviewed_at IS NULL AS pending_review \ - FROM healthcheck_severities WHERE check_name = $1", + "SELECT source, ceiling, reviewed_at IS NULL AS pending_review \ + FROM check_policies WHERE check_name = $1", ) .bind::(check_name) .get_result(conn) @@ -1582,13 +1547,15 @@ async fn submit_status_seeds_catalog_for_new_checks() { let failing = fetch_catalog(&mut conn, "brand_new_check") .await .expect("failing check seeded in catalog"); - assert_eq!(failing.severity, "warning"); + assert_eq!(failing.source, "alertd"); + assert_eq!(failing.ceiling, "warning"); assert!(failing.pending_review); let passing = fetch_catalog(&mut conn, "passing_check") .await .expect("passing check seeded in catalog"); - assert_eq!(passing.severity, "warning"); + assert_eq!(passing.source, "alertd"); + assert_eq!(passing.ceiling, "warning"); assert!(passing.pending_review); }, ) @@ -1617,10 +1584,11 @@ async fn submit_status_uses_catalog_severity_on_failure() { ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/tunable_check") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/tunable_check") .await .expect("per-check issue filed"); - assert_eq!(issue.severity, "critical"); + assert_eq!(issue.effective_result.as_deref(), Some("failed")); + assert!(issue.escalates); }, ) .await @@ -1638,14 +1606,14 @@ async fn set_check_rules( ) { // Ensure the catalog row exists. sql_query( - "INSERT INTO healthcheck_severities (check_name) VALUES ($1) \ - ON CONFLICT (check_name) DO NOTHING", + "INSERT INTO check_policies (source, check_name) VALUES ('alertd', $1) \ + ON CONFLICT (source, check_name) DO NOTHING", ) .bind::(check_name) .execute(conn) .await .expect("ensure catalog row"); - sql_query("UPDATE healthcheck_severities SET rules = $1::jsonb WHERE check_name = $2") + sql_query("UPDATE check_policies SET rules = $1::jsonb WHERE check_name = $2") .bind::(rules.to_string()) .bind::(check_name) .execute(conn) @@ -1679,7 +1647,7 @@ async fn submit_status_rule_on_check_extra_overrides_base() { &mut conn, "disk_space", serde_json::json!({"if": [ - {">": [{"var": "check.used_pct"}, 90]}, "critical" + {">": [{"var": "check.used_pct"}, 90]}, "failed" ]}), ) .await; @@ -1694,10 +1662,12 @@ async fn submit_status_rule_on_check_extra_overrides_base() { }), ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/disk_space") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/disk_space") .await .expect("per-check issue filed"); - assert_eq!(issue.severity, "critical"); + assert_eq!(issue.effective_result.as_deref(), Some("failed")); + + assert!(!issue.escalates); // Below-threshold push falls back to base (default warning). post_status( @@ -1710,10 +1680,10 @@ async fn submit_status_rule_on_check_extra_overrides_base() { }), ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/disk_space") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/disk_space") .await .expect("per-check issue still present"); - assert_eq!(issue.severity, "warning"); + assert_eq!(issue.effective_result.as_deref(), Some("warning")); }, ) .await @@ -1751,10 +1721,10 @@ async fn submit_status_rule_on_bestool_version_range() { }), ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/tamanu_service") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/tamanu_service") .await .expect("per-check issue filed"); - assert_eq!(issue.severity, "warning"); + assert_eq!(issue.effective_result.as_deref(), Some("warning")); // Outside range → falls back to base (error). post_status( @@ -1768,10 +1738,11 @@ async fn submit_status_rule_on_bestool_version_range() { }), ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/tamanu_service") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/tamanu_service") .await .expect("per-check issue still present"); - assert_eq!(issue.severity, "error"); + assert_eq!(issue.effective_result.as_deref(), Some("failed")); + assert!(!issue.escalates); }, ) .await @@ -1795,7 +1766,7 @@ async fn submit_status_rule_on_server_tag() { &mut conn, "cert_expiry", serde_json::json!({"if": [ - {"==": [{"var": "tag.environment"}, "prod"]}, "error" + {"==": [{"var": "tag.environment"}, "prod"]}, "failed" ]}), ) .await; @@ -1810,11 +1781,12 @@ async fn submit_status_rule_on_server_tag() { }), ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/cert_expiry") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/cert_expiry") .await .expect("per-check issue filed"); assert_eq!( - issue.severity, "error", + issue.effective_result.as_deref(), + Some("failed"), "tag.environment=prod fires the rule" ); }, @@ -1833,7 +1805,7 @@ async fn submit_status_tiered_ladder() { &mut conn, "cert_expiry", serde_json::json!({"if": [ - {"<": [{"var": "check.days_remaining"}, 7]}, "error", + {"<": [{"var": "check.days_remaining"}, 7]}, "failed", {"<": [{"var": "check.days_remaining"}, 30]}, "warning" ]}), ) @@ -1850,10 +1822,12 @@ async fn submit_status_tiered_ladder() { }), ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/cert_expiry") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/cert_expiry") .await .expect("issue"); - assert_eq!(issue.severity, "error"); + assert_eq!(issue.effective_result.as_deref(), Some("failed")); + + assert!(!issue.escalates); // Within 30 days but not 7 → warning. post_status( @@ -1866,10 +1840,10 @@ async fn submit_status_tiered_ladder() { }), ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/cert_expiry") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/cert_expiry") .await .expect("issue"); - assert_eq!(issue.severity, "warning"); + assert_eq!(issue.effective_result.as_deref(), Some("warning")); }, ) .await @@ -1953,10 +1927,11 @@ async fn submit_status_result_failed_uses_catalog_severity() { ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/db") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await .expect("per-check issue filed"); - assert_eq!(issue.severity, "error"); + assert_eq!(issue.effective_result.as_deref(), Some("failed")); + assert!(!issue.escalates); assert!(issue.active); assert!( issue @@ -1990,10 +1965,10 @@ async fn submit_status_result_warning_ignores_catalog_severity() { ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/db") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await .expect("per-check issue filed"); - assert_eq!(issue.severity, "warning"); + assert_eq!(issue.effective_result.as_deref(), Some("warning")); assert!(issue.active); assert!( issue @@ -2008,7 +1983,8 @@ async fn submit_status_result_warning_ignores_catalog_severity() { } /// Custom rules can condition on the normalised `check.result` — and -/// they win over both the fixed-Warning default and the catalog base. +/// they win over both the observed result and the ceiling. A warning +/// graded down to passed files nothing at all. #[tokio::test(flavor = "multi_thread")] async fn submit_status_result_rule_on_check_result() { commons_tests::server::run_with_device_auth( @@ -2019,7 +1995,7 @@ async fn submit_status_result_rule_on_check_result() { &mut conn, "db", serde_json::json!({"if": [ - {"==": [{"var": "check.result"}, "warning"]}, "info" + {"==": [{"var": "check.result"}, "warning"]}, "passed" ]}), ) .await; @@ -2034,22 +2010,26 @@ async fn submit_status_result_rule_on_check_result() { ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/db") + let db = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await - .expect("per-check issue filed"); - assert_eq!(issue.severity, "info", "rule overrides the fixed default"); + .expect("state row recorded"); + assert!( + !db.active, + "a warning graded to passed records healthy state, not an issue", + ); }, ) .await } #[tokio::test(flavor = "multi_thread")] -async fn submit_status_result_broken_files_separate_ref_at_warning() { +async fn submit_status_result_broken_warns_on_the_check_ref() { commons_tests::server::run_with_device_auth( "server", async |mut conn, cert, device_id, public, _| { let server_id = insert_health_test_server(&mut conn, device_id).await; - // Catalog severity is irrelevant to broken checks. + // The ceiling grades definite results; a broken check with no + // prior contribution warns regardless. set_check_severity(&mut conn, "db", "critical").await; post_status( @@ -2062,28 +2042,25 @@ async fn submit_status_result_broken_files_separate_ref_at_warning() { ) .await; - let broken = fetch_issue(&mut conn, server_id, "status", "health-broken/db") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await - .expect("broken issue filed"); - assert_eq!(broken.severity, "warning"); - assert!(broken.active); + .expect("broken files on the check's own ref"); + assert_eq!( + issue.effective_result.as_deref(), + Some("broken"), + "nothing to retain: brokenness itself counts as a warning", + ); + assert!(issue.active); assert!( - broken + issue .description .as_ref() .is_some_and(|d| d.contains("broken")) ); - assert!(broken.message.contains("config not found")); - // No failure issue, no incident. - assert!( - fetch_issue(&mut conn, server_id, "status", "health/db") - .await - .is_none(), - "broken must not file at the failure ref" - ); + assert!(issue.message.contains("config not found")); assert!(fetch_open_incident(&mut conn, server_id).await.is_none()); - // Recovery closes the broken ref. + // Recovery closes it. post_status( &public, &cert, @@ -2093,24 +2070,26 @@ async fn submit_status_result_broken_files_separate_ref_at_warning() { }), ) .await; - let broken = fetch_issue(&mut conn, server_id, "status", "health-broken/db") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await - .expect("broken issue still exists"); - assert!(!broken.active); - assert!(broken.message.contains("no longer broken")); + .expect("issue still exists"); + assert!(!issue.active); + assert!(issue.message.contains("recovered")); }, ) .await } -/// failed→broken: the failure issue stays open (the broken check can't -/// confirm the failure either way) while a separate broken issue opens. +/// failed→broken: the issue stays open at the failure's contribution +/// (the broken check can't confirm the failure either way), and a later +/// definite pass closes it. #[tokio::test(flavor = "multi_thread")] -async fn submit_status_failed_then_broken_keeps_failure_open() { +async fn submit_status_failed_then_broken_retains_the_failure() { commons_tests::server::run_with_device_auth( "server", async |mut conn, cert, device_id, public, _| { let server_id = insert_health_test_server(&mut conn, device_id).await; + set_check_severity(&mut conn, "db", "error").await; post_status( &public, @@ -2131,16 +2110,28 @@ async fn submit_status_failed_then_broken_keeps_failure_open() { ) .await; - let failure = fetch_issue(&mut conn, server_id, "status", "health/db") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await - .expect("failure issue exists"); - assert!(failure.active, "broken must not close the failure"); - let broken = fetch_issue(&mut conn, server_id, "status", "health-broken/db") - .await - .expect("broken issue filed"); - assert!(broken.active); + .expect("issue exists"); + assert!(issue.active, "broken must not close the failure"); + assert_eq!( + issue.effective_result.as_deref(), + Some("failed"), + "the failure's contribution is retained while broken", + ); + assert!( + issue + .description + .as_ref() + .is_some_and(|d| d.contains("broken")), + "the headline says the check is broken", + ); + assert!( + fetch_open_incident(&mut conn, server_id).await.is_some(), + "the retained error keeps the incident open", + ); - // passed closes both. + // passed closes it and the incident follows. post_status( &public, &cert, @@ -2150,14 +2141,11 @@ async fn submit_status_failed_then_broken_keeps_failure_open() { }), ) .await; - let failure = fetch_issue(&mut conn, server_id, "status", "health/db") - .await - .expect("failure issue exists"); - assert!(!failure.active); - let broken = fetch_issue(&mut conn, server_id, "status", "health-broken/db") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await - .expect("broken issue exists"); - assert!(!broken.active); + .expect("issue exists"); + assert!(!issue.active); + assert!(fetch_open_incident(&mut conn, server_id).await.is_none()); }, ) .await @@ -2189,7 +2177,7 @@ async fn submit_status_result_skipped_closes_failure_with_message() { ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/cert") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/cert") .await .expect("per-check issue exists"); assert!(!issue.active, "skipped closes the failure"); @@ -2234,10 +2222,10 @@ async fn submit_status_result_skipped_closes_broken() { ) .await; - let broken = fetch_issue(&mut conn, server_id, "status", "health-broken/cert") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/cert") .await - .expect("broken issue exists"); - assert!(!broken.active); + .expect("issue exists"); + assert!(!issue.active); }, ) .await @@ -2273,7 +2261,7 @@ async fn submit_status_legacy_to_result_transition() { ) .await; - let issue = fetch_issue(&mut conn, server_id, "status", "health/db") + let issue = fetch_issue(&mut conn, server_id, "alertd", "health/db") .await .expect("per-check issue exists"); assert!(!issue.active, "result form closes legacy-form failure"); @@ -2312,7 +2300,7 @@ async fn submit_status_result_all_kinds_upsert_catalog() { check_name: String, } let rows: Vec = - sql_query("SELECT check_name FROM healthcheck_severities ORDER BY check_name") + sql_query("SELECT check_name FROM check_policies ORDER BY check_name") .get_results(&mut conn) .await .expect("list catalog"); @@ -2413,6 +2401,40 @@ async fn status_signals_backup_now_for_due_schedule() { .await } +/// Only alertd runs backups: pushes from any other source (a named one, +/// or the legacy tamanu heartbeat) never receive the signal even when a +/// backup is due. +#[tokio::test(flavor = "multi_thread")] +async fn status_backup_now_only_for_alertd() { + commons_tests::server::run_with_device_auth( + "server", + async |mut conn, cert, device_id, public, _| { + let (server_id, group_id) = seed_server_in_group(&mut conn, device_id).await; + seed_backup_config(&mut conn, group_id, "ready").await; + enable_backup_capability(&mut conn, server_id, "tamanu-postgres").await; + + // A named non-alertd source. + let resp = public + .post(&format!("/status/{server_id}")) + .add_header("mtls-certificate", &cert) + .json(&serde_json::json!({ "source": "seedling", "health": [] })) + .await; + resp.assert_status_ok(); + assert!(backup_now(&resp.json()).is_empty()); + + // The legacy (no health array) push, attributed to tamanu. + let resp = public + .post(&format!("/status/{server_id}")) + .add_header("mtls-certificate", &cert) + .json(&serde_json::json!({ "healthy": true })) + .await; + resp.assert_status_ok(); + assert!(backup_now(&resp.json()).is_empty()); + }, + ) + .await +} + /// A recent successful backup is within the interval ⇒ not due ⇒ no signal. #[tokio::test(flavor = "multi_thread")] async fn status_no_backup_now_when_recent_success() { @@ -2655,3 +2677,195 @@ async fn submit_status_versionless_when_neither_present() { ) .await } + +/// Two sources reporting disjoint check sets for the same server must not +/// close each other's issues: a push's "unmentioned means recovered" only +/// applies to the pushing source's own checks. +#[tokio::test(flavor = "multi_thread")] +async fn multi_source_pushes_do_not_flap() { + commons_tests::server::run_with_device_auth( + "server", + async |mut conn, cert, device_id, public, _| { + let server_id = insert_health_test_server(&mut conn, device_id).await; + + // Source-less push: attributed to alertd, files its failure there. + public + .post(&format!("/status/{server_id}")) + .add_header("mtls-certificate", &cert) + .json(&serde_json::json!({ "health": [{ "check": "db", "result": "failed" }] })) + .await + .assert_status_ok(); + assert!( + fetch_issue(&mut conn, server_id, "alertd", "health/db") + .await + .expect("alertd issue filed") + .active + ); + + // A second source pushes a disjoint check set: its own failure + // files under its name, and alertd's open issue is untouched. + public + .post(&format!("/status/{server_id}")) + .add_header("mtls-certificate", &cert) + .json(&serde_json::json!({ + "source": "seedling", + "health": [{ "check": "disk", "result": "failed" }], + })) + .await + .assert_status_ok(); + assert!( + fetch_issue(&mut conn, server_id, "seedling", "health/disk") + .await + .expect("seedling issue filed") + .active + ); + assert!( + fetch_issue(&mut conn, server_id, "alertd", "health/db") + .await + .expect("alertd issue still present") + .active, + "another source's push must not recover alertd's checks", + ); + + // The second source recovering only closes its own issue. + public + .post(&format!("/status/{server_id}")) + .add_header("mtls-certificate", &cert) + .json(&serde_json::json!({ "source": "seedling", "health": [] })) + .await + .assert_status_ok(); + assert!( + !fetch_issue(&mut conn, server_id, "seedling", "health/disk") + .await + .expect("seedling issue closed") + .active + ); + assert!( + fetch_issue(&mut conn, server_id, "alertd", "health/db") + .await + .expect("alertd issue survives") + .active + ); + + // Each stored status row records the source that pushed it. + #[derive(QueryableByName)] + struct SourceRow { + #[diesel(sql_type = sql_types::Text)] + source: String, + } + let sources: Vec = sql_query( + "SELECT source FROM statuses WHERE server_id = $1 ORDER BY created_at, source", + ) + .bind::(server_id) + .load(&mut conn) + .await + .expect("load status sources"); + let sources: Vec<&str> = sources.iter().map(|r| r.source.as_str()).collect(); + assert_eq!(sources, ["alertd", "seedling", "seedling"]); + }, + ) + .await +} + +/// The `source` field must be a non-empty, non-reserved string when present. +#[tokio::test(flavor = "multi_thread")] +async fn source_field_validation() { + commons_tests::server::run_with_device_auth( + "server", + async |mut conn, cert, device_id, public, _| { + let server_id = insert_health_test_server(&mut conn, device_id).await; + + for source in ["canopy", "Manual", ""] { + public + .post(&format!("/status/{server_id}")) + .add_header("mtls-certificate", &cert) + .json(&serde_json::json!({ "source": source, "health": [] })) + .await + .assert_status_bad_request(); + } + + public + .post(&format!("/status/{server_id}")) + .add_header("mtls-certificate", &cert) + .json(&serde_json::json!({ "source": 42, "health": [] })) + .await + .assert_status_bad_request(); + }, + ) + .await +} + +/// Filings stamp the issue's check-state columns: the observed result, +/// the policy-effective result (which can diverge), and the check's +/// detail from the push. +#[tokio::test(flavor = "multi_thread")] +async fn filings_stamp_check_state_columns() { + commons_tests::server::run_with_device_auth( + "server", + async |mut conn, cert, device_id, public, _| { + let server_id = insert_health_test_server(&mut conn, device_id).await; + // Ceiling warning (the default) grades failures down: observed + // and effective diverge. + public + .post(&format!("/status/{server_id}")) + .add_header("mtls-certificate", &cert) + .json(&serde_json::json!({ + "health": [{ "check": "db", "result": "failed", "latency_ms": 42 }], + })) + .await + .assert_status_ok(); + + #[derive(QueryableByName, Debug)] + struct StateRow { + #[diesel(sql_type = sql_types::Nullable)] + check_name: Option, + #[diesel(sql_type = sql_types::Nullable)] + observed_result: Option, + #[diesel(sql_type = sql_types::Nullable)] + effective_result: Option, + #[diesel(sql_type = sql_types::Nullable)] + detail: Option, + } + let fetch = async |conn: &mut diesel_async::AsyncPgConnection| -> StateRow { + sql_query( + "SELECT check_name, observed_result, effective_result, detail \ + FROM issues WHERE server_id = $1 AND ref = 'health/db'", + ) + .bind::(server_id) + .get_result(conn) + .await + .expect("issue row") + }; + + let row = fetch(&mut conn).await; + assert_eq!(row.check_name.as_deref(), Some("db")); + assert_eq!(row.observed_result.as_deref(), Some("failed")); + assert_eq!( + row.effective_result.as_deref(), + Some("warning"), + "default warning ceiling grades the failure down", + ); + assert_eq!( + row.detail + .as_ref() + .and_then(|d| d.get("latency_ms")) + .and_then(|v| v.as_i64()), + Some(42), + ); + + // Recovery stamps the pass. + public + .post(&format!("/status/{server_id}")) + .add_header("mtls-certificate", &cert) + .json(&serde_json::json!({ + "health": [{ "check": "db", "result": "passed" }], + })) + .await + .assert_status_ok(); + let row = fetch(&mut conn).await; + assert_eq!(row.observed_result.as_deref(), Some("passed")); + assert_eq!(row.effective_result.as_deref(), Some("passed")); + }, + ) + .await +} diff --git a/docs/plans/events-retention.md b/docs/plans/events-retention.md deleted file mode 100644 index 3ee9bb6c..00000000 --- a/docs/plans/events-retention.md +++ /dev/null @@ -1,46 +0,0 @@ -# Events retention & partitioning (draft) - -**Status: draft.** `events` grows without bound — one row per issue -state-change and per contributing device push. This plan bounds it with -time-based partitioning and drop-by-partition retention. - -## Sizing - -Observed ≈ **6M rows/year**, extrapolated from the first ~1½ months of -`events` data. That's the low end of the theoretical range (worst case ≈ 100 -devices × 1 push/minute with no coalescing ≈ 50M rows/year; steady-state -coalescing cuts that roughly 10×, though a flapping issue doesn't). Re-check -the rate as the fleet and event mix grow: - -```sql -SELECT count(*) * EXTRACT(EPOCH FROM ('1 year'::interval)) - / EXTRACT(EPOCH FROM (max(created_at) - min(created_at))) -FROM events; -``` - -## Approach - -1. **Partition `events` by `created_at`, weekly** — the same pattern - `statuses` and `device_connections` already use (cron-maintained partition - manager, proven in-tree; see the weekly-partition migrations). -2. **Retention by partition drop.** Keep N weeks (e.g. 26 ≈ 6 months), drop - older partitions. The count is configurable. -3. **Indexes.** `(issue_id, created_at DESC)` within each partition; the "list - events for an issue" query wants recent rows, so partition pruning helps. - -## Non-goals - -- Don't keep daily aggregates while dropping individual rows — that's a - separate analytical concern. If long-term event-rate trends are wanted, - ship them as a materialized view that survives the partition drops. - -## Open question - -- Partition `incident_issues` too? Probably not — it grows far slower (one row - per join/leave, not per push). `incidents` likewise. - -## Effort - -Medium-small: the infrastructure exists in-repo, so it's copy-and-adapt from -the `statuses` partitioning. The retention number is an ops decision, not a -code one. diff --git a/migrations/2026-07-08-063717-0000_statuses_source/down.sql b/migrations/2026-07-08-063717-0000_statuses_source/down.sql new file mode 100644 index 00000000..faf0c55e --- /dev/null +++ b/migrations/2026-07-08-063717-0000_statuses_source/down.sql @@ -0,0 +1,5 @@ +UPDATE server_group_silenced_refs SET source = 'status' WHERE source = 'alertd'; +UPDATE server_silenced_refs SET source = 'status' WHERE source = 'alertd'; +UPDATE issues SET source = 'status' WHERE source = 'alertd'; + +ALTER TABLE statuses DROP COLUMN source; diff --git a/migrations/2026-07-08-063717-0000_statuses_source/up.sql b/migrations/2026-07-08-063717-0000_statuses_source/up.sql new file mode 100644 index 00000000..cf54e3fb --- /dev/null +++ b/migrations/2026-07-08-063717-0000_statuses_source/up.sql @@ -0,0 +1,11 @@ +-- Every status row records the source that pushed it. Pushes that name no +-- source are attributed to 'alertd' (the transitional default), and all +-- pre-source history was pushed by alertd's doctor sweep. +ALTER TABLE statuses ADD COLUMN source TEXT NOT NULL DEFAULT 'alertd'; + +-- Health-check issues and silences were filed under the fixed source +-- 'status'; they now live under the source that reports them, which for +-- everything to date is alertd. +UPDATE issues SET source = 'alertd' WHERE source = 'status'; +UPDATE server_silenced_refs SET source = 'alertd' WHERE source = 'status'; +UPDATE server_group_silenced_refs SET source = 'alertd' WHERE source = 'status'; diff --git a/migrations/2026-07-08-070653-0000_drop_events/down.sql b/migrations/2026-07-08-070653-0000_drop_events/down.sql new file mode 100644 index 00000000..ebc7083f --- /dev/null +++ b/migrations/2026-07-08-070653-0000_drop_events/down.sql @@ -0,0 +1,16 @@ +-- Recreates the table shape only; the audit history is not recoverable. +CREATE TABLE events ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(), + occurred_at TIMESTAMP WITH TIME ZONE, + issue_id UUID NOT NULL REFERENCES issues (id) ON DELETE CASCADE ON UPDATE CASCADE, + severity TEXT NOT NULL, + description TEXT, + message TEXT NOT NULL, + active BOOLEAN NOT NULL, + hash BYTEA NOT NULL, + occurrences INTEGER NOT NULL DEFAULT 1, + last_seen TIMESTAMP WITH TIME ZONE NOT NULL +); + +CREATE INDEX events_issue_created ON events (issue_id, created_at DESC); diff --git a/migrations/2026-07-08-070653-0000_drop_events/up.sql b/migrations/2026-07-08-070653-0000_drop_events/up.sql new file mode 100644 index 00000000..5f51372f --- /dev/null +++ b/migrations/2026-07-08-070653-0000_drop_events/up.sql @@ -0,0 +1,5 @@ +-- Events were an append-only audit log of issue state changes; nothing +-- derived state from them, and issue state itself is the durable record. +-- Statuses hold the per-check history. Unpartitioned and unpruned, the +-- table only grew; it goes away rather than gaining a retention scheme. +DROP TABLE events; diff --git a/migrations/2026-07-08-072903-0000_check_policies/down.sql b/migrations/2026-07-08-072903-0000_check_policies/down.sql new file mode 100644 index 00000000..7edae493 --- /dev/null +++ b/migrations/2026-07-08-072903-0000_check_policies/down.sql @@ -0,0 +1,56 @@ +-- Best-effort reversal: ceilings map back to base severities (escalating +-- failures to critical), rules branches map back to severity strings, and +-- per-source entries collapse to one row per check name (alertd wins, +-- otherwise an arbitrary source's entry). +CREATE TABLE healthcheck_severities ( + check_name TEXT PRIMARY KEY, + severity TEXT NOT NULL DEFAULT 'warning', + first_seen TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(), + reviewed_at TIMESTAMP WITH TIME ZONE, + reviewed_by TEXT, + notes TEXT, + updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(), + rules JSONB +); + +INSERT INTO healthcheck_severities + (check_name, severity, first_seen, reviewed_at, reviewed_by, notes, updated_at, rules) +SELECT DISTINCT ON (check_name) + check_name, + CASE ceiling + WHEN 'failed' THEN CASE WHEN escalates THEN 'critical' ELSE 'error' END + WHEN 'warning' THEN 'warning' + WHEN 'passed' THEN 'info' + WHEN 'skipped' THEN 'debug' + ELSE 'warning' + END, + first_seen, + reviewed_at, + reviewed_by, + notes, + updated_at, + ( + SELECT jsonb_build_object( + 'if', + jsonb_agg( + CASE + WHEN ord % 2 = 1 THEN elem + ELSE to_jsonb( + CASE elem #>> '{}' + WHEN 'failed' THEN 'error' + WHEN 'warning' THEN 'warning' + WHEN 'passed' THEN 'info' + WHEN 'skipped' THEN 'debug' + ELSE 'warning' + END + ) + END + ORDER BY ord + ) + ) + FROM jsonb_array_elements(rules -> 'if') WITH ORDINALITY AS t(elem, ord) + ) +FROM check_policies +ORDER BY check_name, (source = 'alertd') DESC; + +DROP TABLE check_policies; diff --git a/migrations/2026-07-08-072903-0000_check_policies/up.sql b/migrations/2026-07-08-072903-0000_check_policies/up.sql new file mode 100644 index 00000000..28958b8a --- /dev/null +++ b/migrations/2026-07-08-072903-0000_check_policies/up.sql @@ -0,0 +1,82 @@ +-- The severity catalog becomes the check-policy catalog: keyed per +-- (source, check) now that checks are keyed per reporting source, and +-- speaking result transforms instead of severities. The ceiling is the +-- maximum effective result (failed = no cap, warning = failures grade as +-- warnings, passed = recorded but never alerting, skipped = the source +-- may stop running the check); escalates marks checks whose effective +-- failure notifies immediately, bypassing incident grace — the residue +-- of the Critical severity. +CREATE TABLE check_policies ( + source TEXT NOT NULL, + check_name TEXT NOT NULL, + ceiling TEXT NOT NULL DEFAULT 'warning', + escalates BOOLEAN NOT NULL DEFAULT FALSE, + rules JSONB, + notes TEXT, + first_seen TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(), + reviewed_at TIMESTAMP WITH TIME ZONE, + reviewed_by TEXT, + updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(), + PRIMARY KEY (source, check_name) +); + +-- Existing catalog rows were all reported by alertd. Severity → policy: +-- Critical → failed + escalates, Error → failed, Warning → warning, +-- Info → passed, Debug → skipped. An entry whose rules ladder could +-- output critical also escalates (escalation is per-entry now, not +-- per-branch). +INSERT INTO check_policies + (source, check_name, ceiling, escalates, rules, notes, first_seen, reviewed_at, reviewed_by, updated_at) +SELECT + 'alertd', + check_name, + CASE severity + WHEN 'critical' THEN 'failed' + WHEN 'error' THEN 'failed' + WHEN 'warning' THEN 'warning' + WHEN 'info' THEN 'passed' + WHEN 'debug' THEN 'skipped' + ELSE 'warning' + END, + severity = 'critical' OR EXISTS ( + SELECT 1 + FROM jsonb_array_elements(rules -> 'if') WITH ORDINALITY AS t(elem, ord) + WHERE ord % 2 = 0 AND elem #>> '{}' = 'critical' + ), + rules, + notes, + first_seen, + reviewed_at, + reviewed_by, + updated_at +FROM healthcheck_severities; + +-- Rules ladders output results now: rewrite each branch's severity +-- string (the even-ordinal elements of the "if" array) to the result it +-- maps to. +UPDATE check_policies +SET rules = ( + SELECT jsonb_build_object( + 'if', + jsonb_agg( + CASE + WHEN ord % 2 = 1 THEN elem + ELSE to_jsonb( + CASE elem #>> '{}' + WHEN 'critical' THEN 'failed' + WHEN 'error' THEN 'failed' + WHEN 'warning' THEN 'warning' + WHEN 'info' THEN 'passed' + WHEN 'debug' THEN 'skipped' + ELSE 'warning' + END + ) + END + ORDER BY ord + ) + ) + FROM jsonb_array_elements(rules -> 'if') WITH ORDINALITY AS t(elem, ord) +) +WHERE rules IS NOT NULL; + +DROP TABLE healthcheck_severities; diff --git a/migrations/2026-07-08-085731-0000_issues_global_scope/down.sql b/migrations/2026-07-08-085731-0000_issues_global_scope/down.sql new file mode 100644 index 00000000..3685bb3d --- /dev/null +++ b/migrations/2026-07-08-085731-0000_issues_global_scope/down.sql @@ -0,0 +1,9 @@ +DROP INDEX issues_global_source_ref; + +UPDATE issues SET server_id = '00000000-0000-0000-0000-000000000000' + WHERE server_id IS NULL AND server_group_id IS NULL; + +ALTER TABLE issues DROP CONSTRAINT issues_scope_at_most_one; +ALTER TABLE issues ADD CONSTRAINT issues_scope_exactly_one CHECK ( + (server_id IS NULL) <> (server_group_id IS NULL) +); diff --git a/migrations/2026-07-08-085731-0000_issues_global_scope/up.sql b/migrations/2026-07-08-085731-0000_issues_global_scope/up.sql new file mode 100644 index 00000000..68f42ff4 --- /dev/null +++ b/migrations/2026-07-08-085731-0000_issues_global_scope/up.sql @@ -0,0 +1,15 @@ +-- Issues gain a third scope: canopy-wide (neither server nor group). +-- Self-alerts previously squatted on the nil "meta" server row; they +-- become true global-scope issues. +ALTER TABLE issues DROP CONSTRAINT issues_scope_exactly_one; +ALTER TABLE issues ADD CONSTRAINT issues_scope_at_most_one CHECK ( + server_id IS NULL OR server_group_id IS NULL +); + +UPDATE issues SET server_id = NULL + WHERE server_id = '00000000-0000-0000-0000-000000000000'; + +-- Global-scope issues coalesce per (source, ref), mirroring the +-- per-server and per-group unique keys. +CREATE UNIQUE INDEX issues_global_source_ref ON issues (source, ref) + WHERE server_id IS NULL AND server_group_id IS NULL; diff --git a/migrations/2026-07-08-092535-0000_issues_check_state/down.sql b/migrations/2026-07-08-092535-0000_issues_check_state/down.sql new file mode 100644 index 00000000..8372aa46 --- /dev/null +++ b/migrations/2026-07-08-092535-0000_issues_check_state/down.sql @@ -0,0 +1,4 @@ +ALTER TABLE issues DROP COLUMN detail; +ALTER TABLE issues DROP COLUMN effective_result; +ALTER TABLE issues DROP COLUMN observed_result; +ALTER TABLE issues DROP COLUMN check_name; diff --git a/migrations/2026-07-08-092535-0000_issues_check_state/up.sql b/migrations/2026-07-08-092535-0000_issues_check_state/up.sql new file mode 100644 index 00000000..ffcae71d --- /dev/null +++ b/migrations/2026-07-08-092535-0000_issues_check_state/up.sql @@ -0,0 +1,35 @@ +-- Issues grow toward the check-state model: each row records which check +-- it tracks and both sides of the policy transform (what the source +-- observed, what policy made of it), plus the check's latest detail. +-- Nullable while the severity vocabulary is still authoritative; filing +-- populates them from here on. +ALTER TABLE issues ADD COLUMN check_name TEXT; +ALTER TABLE issues ADD COLUMN observed_result TEXT; +ALTER TABLE issues ADD COLUMN effective_result TEXT; +ALTER TABLE issues ADD COLUMN detail JSONB; + +-- Backfill health-check rows: the check is the ref minus its prefix, and +-- the effective result is read back from the filed severity (the reverse +-- of the transitional severity mapping). The observed result is not +-- recoverable from an issue row; assume it matched the effective one. +UPDATE issues SET + check_name = substring(ref from 8), + observed_result = CASE + WHEN NOT active THEN 'passed' + WHEN severity IN ('critical', 'error') THEN 'failed' + ELSE 'warning' + END, + effective_result = CASE + WHEN NOT active THEN 'passed' + WHEN severity IN ('critical', 'error') THEN 'failed' + ELSE 'warning' + END + WHERE ref LIKE 'health/%'; + +-- Broken-thread rows observed a broken check; their fixed-Warning filing +-- is the brokenness itself counting as a warning. +UPDATE issues SET + check_name = substring(ref from 15), + observed_result = CASE WHEN active THEN 'broken' ELSE 'passed' END, + effective_result = CASE WHEN active THEN 'broken' ELSE 'passed' END + WHERE ref LIKE 'health-broken/%'; diff --git a/migrations/2026-07-08-095359-0000_merge_broken_thread/down.sql b/migrations/2026-07-08-095359-0000_merge_broken_thread/down.sql new file mode 100644 index 00000000..50a170ee --- /dev/null +++ b/migrations/2026-07-08-095359-0000_merge_broken_thread/down.sql @@ -0,0 +1,3 @@ +-- The broken-thread rows and their silences are not recoverable; undoing +-- only reopens nothing. No-op. +SELECT 1; diff --git a/migrations/2026-07-08-095359-0000_merge_broken_thread/up.sql b/migrations/2026-07-08-095359-0000_merge_broken_thread/up.sql new file mode 100644 index 00000000..88273c9c --- /dev/null +++ b/migrations/2026-07-08-095359-0000_merge_broken_thread/up.sql @@ -0,0 +1,81 @@ +-- Retire the separate `health-broken/` issue thread: brokenness +-- now lives on the check's own `health/` state row (sticky — +-- retaining the previous definite result's contribution while broken). +-- The companion code change no longer files the separate ref; this +-- migration cleans up the rows that already exist, mirroring the +-- overall-health rollup retirement. + +-- 1. Deactivate and resolve the broken-thread issues. +UPDATE issues +SET active = false, + resolved_at = NOW(), + resolved_by = 'migration:2026-07-08-merge_broken_thread', + resolved_reason = 'expected', + last_seen = NOW(), + updated_at = NOW() +WHERE ref LIKE 'health-broken/%' AND active = true; + +-- 2. Mark incident links as left for these issues, so the orphan-close +-- in step 3 sees an accurate "remaining contributors" count. +UPDATE incident_issues +SET left_at = NOW() +WHERE left_at IS NULL + AND issue_id IN ( + SELECT id FROM issues + WHERE ref LIKE 'health-broken/%' + AND resolved_by = 'migration:2026-07-08-merge_broken_thread' + ); + +-- 3. Close incidents this migration just orphaned. No Slack resolve — +-- this is a code-retirement cleanup, not an operational recovery. +UPDATE incidents +SET closed_at = NOW(), updated_at = NOW() +WHERE closed_at IS NULL + AND EXISTS ( + SELECT 1 FROM incident_issues ii + JOIN issues i ON i.id = ii.issue_id + WHERE ii.incident_id = incidents.id + AND i.ref LIKE 'health-broken/%' + AND i.resolved_by = 'migration:2026-07-08-merge_broken_thread' + ) + AND NOT EXISTS ( + SELECT 1 FROM incident_issues ii + WHERE ii.incident_id = incidents.id + AND ii.left_at IS NULL + ); + +-- 4. Cancel pending Slack opens for incidents just closed. +UPDATE slack_outbox +SET gave_up_at = NOW(), + last_error = 'cancelled: incident closed by broken-thread merge migration' +WHERE gave_up_at IS NULL + AND delivered_at IS NULL + AND kind = 'incident_open' + AND incident_id IN ( + SELECT id FROM incidents + WHERE closed_at IS NOT NULL + AND EXISTS ( + SELECT 1 FROM incident_issues ii + JOIN issues i ON i.id = ii.issue_id + WHERE ii.incident_id = incidents.id + AND i.ref LIKE 'health-broken/%' + AND i.resolved_by = 'migration:2026-07-08-merge_broken_thread' + ) + ); + +-- 5. Fold broken-thread silences into the check's own silence: a check +-- silenced only on its broken thread becomes silenced outright (the +-- two are one thread now), then the old rows go away. +INSERT INTO server_silenced_refs (server_id, source, ref, created_at, created_by) +SELECT server_id, source, 'health/' || substring(ref from 15), created_at, created_by +FROM server_silenced_refs +WHERE ref LIKE 'health-broken/%' +ON CONFLICT DO NOTHING; +DELETE FROM server_silenced_refs WHERE ref LIKE 'health-broken/%'; + +INSERT INTO server_group_silenced_refs (server_group_id, source, ref, created_at, created_by) +SELECT server_group_id, source, 'health/' || substring(ref from 15), created_at, created_by +FROM server_group_silenced_refs +WHERE ref LIKE 'health-broken/%' +ON CONFLICT DO NOTHING; +DELETE FROM server_group_silenced_refs WHERE ref LIKE 'health-broken/%'; diff --git a/migrations/2026-07-08-101747-0000_issues_degraded_timestamps/down.sql b/migrations/2026-07-08-101747-0000_issues_degraded_timestamps/down.sql new file mode 100644 index 00000000..e2d3a383 --- /dev/null +++ b/migrations/2026-07-08-101747-0000_issues_degraded_timestamps/down.sql @@ -0,0 +1,2 @@ +ALTER TABLE issues DROP COLUMN last_degraded_at; +ALTER TABLE issues DROP COLUMN degraded_since; diff --git a/migrations/2026-07-08-101747-0000_issues_degraded_timestamps/up.sql b/migrations/2026-07-08-101747-0000_issues_degraded_timestamps/up.sql new file mode 100644 index 00000000..4d93aca3 --- /dev/null +++ b/migrations/2026-07-08-101747-0000_issues_degraded_timestamps/up.sql @@ -0,0 +1,11 @@ +-- With passing checks about to get state rows, "when did this start +-- failing" and "has this ever been trouble" stop being derivable from +-- first_seen / row existence. degraded_since tracks the current +-- degradation streak (null while healthy); last_degraded_at never +-- clears, distinguishing a recovered issue from always-healthy state. +ALTER TABLE issues ADD COLUMN degraded_since TIMESTAMP WITH TIME ZONE; +ALTER TABLE issues ADD COLUMN last_degraded_at TIMESTAMP WITH TIME ZONE; + +-- Every existing row was filed because it degraded at some point. +UPDATE issues SET last_degraded_at = last_seen; +UPDATE issues SET degraded_since = first_seen WHERE active; diff --git a/migrations/2026-07-08-105204-0000_incidents_global_target/down.sql b/migrations/2026-07-08-105204-0000_incidents_global_target/down.sql new file mode 100644 index 00000000..e6dfd88a --- /dev/null +++ b/migrations/2026-07-08-105204-0000_incidents_global_target/down.sql @@ -0,0 +1,13 @@ +DROP INDEX incidents_open_global; + +-- Canopy-wide incidents have no group to return to; they must go before +-- the column can be NOT NULL again. +DELETE FROM incident_issues WHERE incident_id IN ( + SELECT id FROM incidents WHERE server_group_id IS NULL +); +DELETE FROM incident_notes WHERE incident_id IN ( + SELECT id FROM incidents WHERE server_group_id IS NULL +); +DELETE FROM incidents WHERE server_group_id IS NULL; + +ALTER TABLE incidents ALTER COLUMN server_group_id SET NOT NULL; diff --git a/migrations/2026-07-08-105204-0000_incidents_global_target/up.sql b/migrations/2026-07-08-105204-0000_incidents_global_target/up.sql new file mode 100644 index 00000000..0156fa13 --- /dev/null +++ b/migrations/2026-07-08-105204-0000_incidents_global_target/up.sql @@ -0,0 +1,8 @@ +-- Incidents generalise to per-target: a server group, or canopy as a +-- whole (server_group_id NULL). Canopy-wide issues (self-alerts) get the +-- full incident lifecycle instead of their bespoke direct-Slack path. +ALTER TABLE incidents ALTER COLUMN server_group_id DROP NOT NULL; + +-- At most one open canopy-wide incident, mirroring the per-group rule. +CREATE UNIQUE INDEX incidents_open_global ON incidents ((1)) + WHERE server_group_id IS NULL AND closed_at IS NULL; diff --git a/migrations/2026-07-08-151912-0000_drop_allow_legacy_status/down.sql b/migrations/2026-07-08-151912-0000_drop_allow_legacy_status/down.sql new file mode 100644 index 00000000..e4d59a45 --- /dev/null +++ b/migrations/2026-07-08-151912-0000_drop_allow_legacy_status/down.sql @@ -0,0 +1 @@ +ALTER TABLE servers ADD COLUMN allow_legacy_status BOOLEAN NOT NULL DEFAULT FALSE; diff --git a/migrations/2026-07-08-151912-0000_drop_allow_legacy_status/up.sql b/migrations/2026-07-08-151912-0000_drop_allow_legacy_status/up.sql new file mode 100644 index 00000000..2ebe11e1 --- /dev/null +++ b/migrations/2026-07-08-151912-0000_drop_allow_legacy_status/up.sql @@ -0,0 +1,4 @@ +-- Legacy-format pushes are no longer gated: they transform into the +-- tamanu source's always-passing `tasks` heartbeat unconditionally, so +-- the opt-in flag has no job left. +ALTER TABLE servers DROP COLUMN allow_legacy_status; diff --git a/migrations/2026-07-09-034816-0000_scoped_check_policies/down.sql b/migrations/2026-07-09-034816-0000_scoped_check_policies/down.sql new file mode 100644 index 00000000..750a04cb --- /dev/null +++ b/migrations/2026-07-09-034816-0000_scoped_check_policies/down.sql @@ -0,0 +1,39 @@ +CREATE TABLE server_silenced_refs ( + server_id UUID NOT NULL REFERENCES servers (id) ON DELETE CASCADE ON UPDATE CASCADE, + source TEXT NOT NULL, + ref TEXT NOT NULL, + created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(), + created_by TEXT, + PRIMARY KEY (server_id, source, ref) +); + +CREATE TABLE server_group_silenced_refs ( + server_group_id UUID NOT NULL REFERENCES server_groups (id) ON DELETE CASCADE ON UPDATE CASCADE, + source TEXT NOT NULL, + ref TEXT NOT NULL, + created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(), + created_by TEXT, + PRIMARY KEY (server_group_id, source, ref) +); + +-- Silences (skipped-ceiling scoped policies) go back to ref rows; other +-- scoped transforms have no downgrade representation and are dropped. +INSERT INTO server_silenced_refs (server_id, source, ref, created_at, created_by) +SELECT + server_id, source, + CASE WHEN source IN ('canopy', 'manual') THEN check_name ELSE 'health/' || check_name END, + created_at, created_by +FROM scoped_check_policies +WHERE server_id IS NOT NULL AND ceiling = 'skipped' +ON CONFLICT DO NOTHING; + +INSERT INTO server_group_silenced_refs (server_group_id, source, ref, created_at, created_by) +SELECT + server_group_id, source, + CASE WHEN source IN ('canopy', 'manual') THEN check_name ELSE 'health/' || check_name END, + created_at, created_by +FROM scoped_check_policies +WHERE server_group_id IS NOT NULL AND ceiling = 'skipped' +ON CONFLICT DO NOTHING; + +DROP TABLE scoped_check_policies; diff --git a/migrations/2026-07-09-034816-0000_scoped_check_policies/up.sql b/migrations/2026-07-09-034816-0000_scoped_check_policies/up.sql new file mode 100644 index 00000000..3377dc4d --- /dev/null +++ b/migrations/2026-07-09-034816-0000_scoped_check_policies/up.sql @@ -0,0 +1,60 @@ +-- Scoped check policy: a result transform scoped to one target — a +-- server, a server group, or canopy-wide — applied after the fleet +-- catalog (fleet, then group, then server; each acting on the previous +-- effective result). Either side of the transform may be present: a +-- ceiling, a rules ladder, or both. +-- +-- The operator-facing silence is a scoped ceiling of 'skipped': the +-- check keeps recording its observed results, but its effective result +-- is skipped so it raises nothing and counts nowhere. Arbitrary scoped +-- transforms are admitted by the model; the UI only offers silences. +CREATE TABLE scoped_check_policies ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(), + updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(), + source TEXT NOT NULL, + check_name TEXT NOT NULL, + -- The scope: exactly one of server/group, or neither for canopy-wide. + server_id UUID REFERENCES servers (id) ON DELETE CASCADE, + server_group_id UUID REFERENCES server_groups (id) ON DELETE CASCADE, + ceiling TEXT, + rules JSONB, + created_by TEXT, + CHECK (server_id IS NULL OR server_group_id IS NULL), + CHECK (ceiling IS NOT NULL OR rules IS NOT NULL) +); + +-- One transform per (scope, source, check). +CREATE UNIQUE INDEX scoped_check_policies_server + ON scoped_check_policies (server_id, source, check_name) + WHERE server_id IS NOT NULL; +CREATE UNIQUE INDEX scoped_check_policies_group + ON scoped_check_policies (server_group_id, source, check_name) + WHERE server_group_id IS NOT NULL; +CREATE UNIQUE INDEX scoped_check_policies_global + ON scoped_check_policies (source, check_name) + WHERE server_id IS NULL AND server_group_id IS NULL; + +-- Silences were (source, ref) rows in two sibling tables; they become +-- skipped-ceiling scoped policies keyed by check name (the ref minus +-- the health/ namespace prefix for source-reported checks). +INSERT INTO scoped_check_policies + (source, check_name, server_id, ceiling, created_by, created_at, updated_at) +SELECT + source, + CASE WHEN ref LIKE 'health/%' THEN substring(ref from 8) ELSE ref END, + server_id, 'skipped', created_by, created_at, created_at +FROM server_silenced_refs +ON CONFLICT DO NOTHING; + +INSERT INTO scoped_check_policies + (source, check_name, server_group_id, ceiling, created_by, created_at, updated_at) +SELECT + source, + CASE WHEN ref LIKE 'health/%' THEN substring(ref from 8) ELSE ref END, + server_group_id, 'skipped', created_by, created_at, created_at +FROM server_group_silenced_refs +ON CONFLICT DO NOTHING; + +DROP TABLE server_silenced_refs; +DROP TABLE server_group_silenced_refs; diff --git a/migrations/2026-07-09-042028-0000_issues_escalates/down.sql b/migrations/2026-07-09-042028-0000_issues_escalates/down.sql new file mode 100644 index 00000000..a6bbb0af --- /dev/null +++ b/migrations/2026-07-09-042028-0000_issues_escalates/down.sql @@ -0,0 +1 @@ +ALTER TABLE issues DROP COLUMN escalates; diff --git a/migrations/2026-07-09-042028-0000_issues_escalates/up.sql b/migrations/2026-07-09-042028-0000_issues_escalates/up.sql new file mode 100644 index 00000000..21f9d1af --- /dev/null +++ b/migrations/2026-07-09-042028-0000_issues_escalates/up.sql @@ -0,0 +1,7 @@ +-- Check state records whether its policy escalates: an effective failure +-- of an escalating check notifies immediately, bypassing incident grace. +-- Stamped from the catalog on every filing, so incident semantics can key +-- on (effective_result, escalates) instead of the severity vocabulary. +-- Backfill from the transitional mapping's only escalation severity. +ALTER TABLE issues ADD COLUMN escalates BOOLEAN NOT NULL DEFAULT FALSE; +UPDATE issues SET escalates = TRUE WHERE severity = 'critical'; diff --git a/migrations/2026-07-09-050226-0000_drop_issue_severity/down.sql b/migrations/2026-07-09-050226-0000_drop_issue_severity/down.sql new file mode 100644 index 00000000..f9bbc5b5 --- /dev/null +++ b/migrations/2026-07-09-050226-0000_drop_issue_severity/down.sql @@ -0,0 +1,7 @@ +ALTER TABLE issues ADD COLUMN severity TEXT NOT NULL DEFAULT 'info'; +UPDATE issues SET severity = CASE + WHEN effective_result = 'failed' AND escalates THEN 'critical' + WHEN effective_result = 'failed' THEN 'error' + WHEN effective_result IN ('warning', 'broken') THEN 'warning' + ELSE 'info' +END; diff --git a/migrations/2026-07-09-050226-0000_drop_issue_severity/up.sql b/migrations/2026-07-09-050226-0000_drop_issue_severity/up.sql new file mode 100644 index 00000000..c3eb9776 --- /dev/null +++ b/migrations/2026-07-09-050226-0000_drop_issue_severity/up.sql @@ -0,0 +1,21 @@ +-- The severity vocabulary is fully retired: incident semantics key on +-- (effective_result, escalates), and every surface speaks results. +-- Backfill rows that predate the check-state model (never stamped by a +-- filing) so their incident participation carries over, then drop. +UPDATE issues SET + check_name = ref, + observed_result = CASE + WHEN NOT active THEN 'passed' + WHEN severity IN ('critical', 'error') THEN 'failed' + WHEN severity = 'warning' THEN 'warning' + ELSE 'skipped' + END, + effective_result = CASE + WHEN NOT active THEN 'passed' + WHEN severity IN ('critical', 'error') THEN 'failed' + WHEN severity = 'warning' THEN 'warning' + ELSE 'skipped' + END + WHERE effective_result IS NULL; + +ALTER TABLE issues DROP COLUMN severity; diff --git a/migrations/2026-07-09-052945-0000_check_documentation/down.sql b/migrations/2026-07-09-052945-0000_check_documentation/down.sql new file mode 100644 index 00000000..271ff611 --- /dev/null +++ b/migrations/2026-07-09-052945-0000_check_documentation/down.sql @@ -0,0 +1 @@ +ALTER TABLE check_policies DROP COLUMN documentation; diff --git a/migrations/2026-07-09-052945-0000_check_documentation/up.sql b/migrations/2026-07-09-052945-0000_check_documentation/up.sql new file mode 100644 index 00000000..1078e5bf --- /dev/null +++ b/migrations/2026-07-09-052945-0000_check_documentation/up.sql @@ -0,0 +1,5 @@ +-- Operator-authored documentation for a catalogued (source, check): a +-- single markdown document. Convention (seeded by the UI editor, not +-- enforced): what the check observes, what each result means, and hints +-- for solving a failure. +ALTER TABLE check_policies ADD COLUMN documentation TEXT; diff --git a/private-web/e2e/health.spec.ts b/private-web/e2e/health.spec.ts index 48baeabc..7f73a769 100644 --- a/private-web/e2e/health.spec.ts +++ b/private-web/e2e/health.spec.ts @@ -1,5 +1,6 @@ import { expect, test } from "./test-fixtures"; import { + seedCheckPolicy, resetSeededTables, seedGroupSilencedRef, seedServer, @@ -97,6 +98,61 @@ test.describe("server detail checks table", () => { await expect(page.getByText("free_pct: 2")).toBeVisible(); }); + test("the ? button pops up the check's rendered documentation", async ({ + page, + sql, + }) => { + await seedVersion(sql, { major: 1, minor: 0, patch: 0 }); + const server = await seedServer(sql, { + name: "documented-server", + kind: "central", + }); + await seedStatus(sql, { + serverId: server.id, + health: [ + { check: "postgres", result: "failed" }, + { check: "disk", result: "passed" }, + ], + }); + await seedCheckPolicy(sql, { + checkName: "postgres", + documentation: + "## Description\n\nWatches the PostgreSQL connection.\n\n## Solve\n\nCheck pg_hba.conf.", + }); + // Another source documents a same-named "disk" check: that document + // must NOT leak into alertd's undocumented one below — documentation + // is keyed per (source, check). + await seedCheckPolicy(sql, { + checkName: "disk", + source: "seedling", + documentation: "## Description\n\nSeedling's unrelated disk check.", + }); + + await page.goto(`/servers/${server.id}`); + await page + .getByRole("button", { name: "Documentation for postgres" }) + .click(); + await expect( + page.getByText("Watches the PostgreSQL connection."), + ).toBeVisible(); + await expect( + page.getByRole("link", { name: "Edit documentation" }), + ).toBeVisible(); + + // An undocumented check still gets the affordance, prompting for + // the missing document instead of hiding the icon — and another + // source's same-named document doesn't leak in. + await page.keyboard.press("Escape"); + await page.getByRole("button", { name: "Documentation for disk" }).click(); + await expect( + page.getByText("Nobody has documented this check yet."), + ).toBeVisible(); + await expect( + page.getByText("Seedling's unrelated disk check."), + ).not.toBeVisible(); + await expect(page.getByRole("link", { name: "Write it" })).toBeVisible(); + }); + test("sorts skipped checks last, after passing ones", async ({ page, sql, diff --git a/private-web/e2e/healthcheck-attention.spec.ts b/private-web/e2e/healthcheck-attention.spec.ts index 4d0fcf86..eb7fd831 100644 --- a/private-web/e2e/healthcheck-attention.spec.ts +++ b/private-web/e2e/healthcheck-attention.spec.ts @@ -1,8 +1,7 @@ import { expect, test } from "./test-fixtures"; import { resetSeededTables, - seedHealthcheckSeverity, - seedIssue, + seedCheckPolicy, seedServer, seedServerGroup, seedStatus, @@ -16,7 +15,7 @@ test.describe("healthcheck attention page", () => { test("renders an empty state when no server flags the check", async ({ page, }) => { - await page.goto("/healthchecks/postgres"); + await page.goto("/healthchecks/alertd/postgres"); await expect( page.getByText("No servers currently flag"), ).toBeVisible(); @@ -69,7 +68,7 @@ test.describe("healthcheck attention page", () => { health: [{ check: "disk_space", result: "failed", free_pct: 3 }], }); - await page.goto("/healthchecks/postgres"); + await page.goto("/healthchecks/alertd/postgres"); const failingLink = page.getByRole("link", { name: "Korolevu Facility" }); const warningLink = page.getByRole("link", { name: "Sigatoka Facility" }); @@ -124,7 +123,7 @@ test.describe("healthcheck attention page", () => { health: [{ check: "postgres", result: "passed", latency_ms: 12 }], }); - await page.goto("/healthchecks/postgres"); + await page.goto("/healthchecks/alertd/postgres"); await expect( page.getByRole("link", { name: "Mendi Facility" }), ).toBeVisible(); @@ -175,7 +174,7 @@ test.describe("healthcheck attention page", () => { health: [{ check: "disk_space", result: "warning" }], }); - await page.goto("/healthchecks/disk_space"); + await page.goto("/healthchecks/alertd/disk_space"); await expect( page.getByRole("link", { name: "Labasa Facility" }), ).toBeVisible(); @@ -222,23 +221,29 @@ test.describe("healthcheck attention page", () => { { check: "postgres", result: "failed", error: "connection refused" }, ], }); - await seedIssue(sql, { - serverId: failing.id, - source: "status", - ref: "health/postgres", - message: "postgres check failing", - firstSeen: new Date(Date.now() - 3 * 3_600_000).toISOString(), - }); + // The failing server's degradation streak started three hours ago; + // the other server's state row predates the check-state stamps + // (inactive, no streak), which renders the placeholder. + await sql.query( + `UPDATE issues SET degraded_since = NOW() - INTERVAL '3 hours' + WHERE server_id = $1 AND ref = 'health/postgres'`, + [failing.id], + ); + await sql.query( + `UPDATE issues SET active = false, degraded_since = NULL + WHERE server_id = $1 AND ref = 'health/postgres'`, + [fresh.id], + ); - await page.goto("/healthchecks/postgres"); + await page.goto("/healthchecks/alertd/postgres"); - // The issue-backed row shows a relative failing-since; the row - // without an issue shows the em-dash placeholder. + // The state-backed row shows a relative failing-since; the row + // without an active streak shows the em-dash placeholder. await expect(page.getByText("3h ago")).toBeVisible(); await expect(page.getByText("—")).toBeVisible(); }); - test("shows the catalog severity when one is configured", async ({ + test("shows the catalog ceiling when one is configured", async ({ page, sql, }) => { @@ -247,13 +252,75 @@ test.describe("healthcheck attention page", () => { serverId: server.id, health: [{ check: "disk_space", result: "failed", free_pct: 2 }], }); - await seedHealthcheckSeverity(sql, { + await seedCheckPolicy(sql, { + checkName: "disk_space", + ceiling: "failed", + escalates: true, + }); + + await page.goto("/healthchecks/alertd/disk_space"); + // The heading chip renders the configured ceiling, plus the + // escalates marker. + await expect( + page.getByRole("heading", { name: "disk_space" }), + ).toBeVisible(); + await expect(page.getByText("escalates", { exact: true })).toBeVisible(); + }); + + test("shows operator documentation in an expandable panel", async ({ + page, + sql, + }) => { + const server = await seedServer(sql, { name: "Doc Facility" }); + await seedStatus(sql, { + serverId: server.id, + health: [{ check: "disk_space", result: "failed" }], + }); + await seedCheckPolicy(sql, { checkName: "disk_space", - severity: "critical", + documentation: + "## Description\n\nWatches free disk space.\n\n## Solve\n\nClear old backups.", }); - await page.goto("/healthchecks/disk_space"); - await expect(page.getByText("critical", { exact: true })).toBeVisible(); + await page.goto("/healthchecks/alertd/disk_space"); + await page.getByText("About this check").click(); + await expect(page.getByText("Watches free disk space.")).toBeVisible(); + await expect(page.getByText("Clear old backups.")).toBeVisible(); + }); + + test("documentation is written from the healthcheck settings page, seeded with the template", async ({ + page, + sql, + }) => { + await seedCheckPolicy(sql, { checkName: "disk_space" }); + + await page.goto("/settings/healthchecks/disk_space"); + await expect( + page.getByText("Nobody has documented this check yet", { exact: false }), + ).toBeVisible(); + await page.getByRole("button", { name: "Write documentation" }).click(); + + // The editor seeds the conventional template. + const editor = page.getByRole("textbox", { + name: "Documentation markdown", + }); + await expect(editor).toHaveValue(/## Description/); + await expect(editor).toHaveValue(/## Results/); + await expect(editor).toHaveValue(/## Solve/); + + await editor.fill("## Description\n\nDisk space watcher."); + await page.getByRole("button", { name: "Save documentation" }).click(); + + // The saved document renders as markdown. + await expect( + page.getByRole("heading", { name: "Description" }), + ).toBeVisible(); + await expect(page.getByText("Disk space watcher.")).toBeVisible(); + + const rows = await sql.query<{ documentation: string | null }>( + "SELECT documentation FROM check_policies WHERE source = 'alertd' AND check_name = 'disk_space'", + ); + expect(rows[0]!.documentation).toContain("Disk space watcher."); }); test("URL-encodes check names with special characters", async ({ @@ -267,7 +334,7 @@ test.describe("healthcheck attention page", () => { health: [{ check: checkName, result: "failed" }], }); - await page.goto(`/healthchecks/${encodeURIComponent(checkName)}`); + await page.goto(`/healthchecks/alertd/${encodeURIComponent(checkName)}`); await expect( page.getByRole("heading", { name: checkName }), ).toBeVisible(); @@ -293,6 +360,6 @@ test.describe("healthcheck attention page", () => { const checkLink = page.getByRole("link", { name: "postgres" }); await expect(checkLink).toBeVisible(); await checkLink.click(); - await expect(page).toHaveURL(/\/healthchecks\/postgres$/); + await expect(page).toHaveURL(/\/healthchecks\/alertd\/postgres$/); }); }); diff --git a/private-web/e2e/issues.spec.ts b/private-web/e2e/issues.spec.ts index 4cda70b0..e3ff4b33 100644 --- a/private-web/e2e/issues.spec.ts +++ b/private-web/e2e/issues.spec.ts @@ -1,6 +1,8 @@ import { resetSeededTables, + seedCheckPolicy, seedIssue, + seedServer, seedServerGroup, } from "./seed"; import { expect, test } from "./test-fixtures"; @@ -8,6 +10,38 @@ import { expect, test } from "./test-fixtures"; // Group-scoped issues (server_id NULL, server_group_id set) must not render a // per-server link (`/servers/null`) or a per-server silence action — both only // make sense for server-scoped issues. +test.describe("issue check documentation", () => { + test.beforeEach(async ({ sql }) => { + await resetSeededTables(sql); + }); + + test("the ? button on an issue row pops up the check's documentation", async ({ + page, + sql, + }) => { + const server = await seedServer(sql, { name: "doc-issue-server" }); + await seedIssue(sql, { + serverId: server.id, + ref: "health/postgres", + message: "postgres is down", + }); + await seedCheckPolicy(sql, { + checkName: "postgres", + documentation: + "## Description\n\nWatches the PostgreSQL connection.\n\n## Solve\n\nCheck pg_hba.conf.", + }); + + await page.goto("/incidents?showAll=1"); + await expect(page.getByText("postgres is down")).toBeVisible(); + await page + .getByRole("button", { name: "Documentation for postgres" }) + .click(); + await expect( + page.getByText("Watches the PostgreSQL connection."), + ).toBeVisible(); + }); +}); + test.describe("group-scoped issue rendering", () => { test.beforeEach(async ({ sql }) => { await resetSeededTables(sql); @@ -59,3 +93,46 @@ test.describe("group-scoped issue rendering", () => { ).toHaveCount(0); }); }); + +// The manual-condition form speaks the result vocabulary: an operator +// raises a failed (optionally escalating) or warning condition, which +// files under the manual source and grades through the catalog. +test.describe("manual conditions", () => { + test.beforeEach(async ({ sql }) => { + await resetSeededTables(sql); + }); + + test("raising a failed condition from the server page files an issue", async ({ + page, + sql, + }) => { + const group = await seedServerGroup(sql, { name: "ManualGroup" }); + const server = await seedServer(sql, { + name: "manual-target", + kind: "central", + groupId: group.id, + }); + + await page.goto(`/servers/${server.id}`); + await page.getByRole("button", { name: /new incident/i }).click(); + + const dialog = page.getByRole("dialog"); + await expect(dialog.getByLabel("Result")).toBeVisible(); + // A failed condition offers the immediate-notify (escalates) toggle. + await expect( + dialog.getByRole("switch", { name: /notify immediately/i }), + ).toBeVisible(); + await dialog + .getByLabel("Description (short)") + .fill("Manually raised trouble"); + await dialog.getByLabel("Message").fill("operator saw smoke"); + await dialog.getByRole("button", { name: /^submit$/i }).click(); + await expect(dialog).not.toBeVisible(); + + // The condition filed as a failure: it appears in the fleet issue + // list (the never-checked-in server page shows setup instructions, + // not an issues panel). + await page.goto("/incidents?showAll=1"); + await expect(page.getByText("Manually raised trouble")).toBeVisible(); + }); +}); diff --git a/private-web/e2e/seed.ts b/private-web/e2e/seed.ts index fc20f829..ec54729a 100644 --- a/private-web/e2e/seed.ts +++ b/private-web/e2e/seed.ts @@ -47,7 +47,7 @@ function randomLabel(prefix: string): string { * statement with CASCADE. */ export async function resetSeededTables(sql: Sql): Promise { await sql.query( - "TRUNCATE statuses, issues, device_keys, servers, server_groups, devices, versions, tailscale_users, healthcheck_severities, server_group_backup_config, server_group_backup_schedule, server_backup_capabilities, backup_requests, backup_runs, backup_repo_stats, backup_maintenance_runs, backup_credential_issuances, restore_replicas, restore_consumer_capabilities, backup_restore_checks, recovery_vault_writes RESTART IDENTITY CASCADE", + "TRUNCATE statuses, issues, device_keys, servers, server_groups, devices, versions, tailscale_users, check_policies, scoped_check_policies, server_group_backup_config, server_group_backup_schedule, server_backup_capabilities, backup_requests, backup_runs, backup_repo_stats, backup_maintenance_runs, backup_credential_issuances, restore_replicas, restore_consumer_capabilities, backup_restore_checks, recovery_vault_writes RESTART IDENTITY CASCADE", ); // The truncate takes the migration-seeded nil "Canopy" server with it; // self-alerts attach to that row, so put it back. @@ -234,37 +234,92 @@ export async function seedStatus( RETURNING created_at`, params, ); + + // Mirror ingestion: each check in the push has a check-state row, which + // is what the health rollup and attention pages read. Degraded checks + // carry the degraded-streak stamps; healthy ones record inactive state. + for (const entry of (opts.health ?? []) as Record[]) { + const check = entry.check; + if (typeof check !== "string") continue; + const result = + typeof entry.result === "string" + ? entry.result + : typeof entry.healthy === "boolean" + ? entry.healthy + ? "passed" + : "failed" + : null; + if (result === null) continue; + const degraded = ["failed", "warning", "broken"].includes(result); + await sql.query( + `INSERT INTO issues + (server_id, source, ref, check_name, observed_result, effective_result, detail, message, active, first_seen, last_seen, degraded_since, last_degraded_at) + VALUES ($1, 'alertd', $2, $3, $4, $4, $5::jsonb, $6, $7, NOW(), NOW(), $8, $9) + ON CONFLICT DO NOTHING`, + [ + opts.serverId, + `health/${check}`, + check, + result, + JSON.stringify(entry), + `Health check '${check}' ${degraded ? "degraded" : "recorded"}`, + degraded, + degraded ? new Date().toISOString() : null, + degraded ? new Date().toISOString() : null, + ], + ); + } + return { id, createdAt: String(rows[0]!.created_at) }; } -export interface SeededHealthcheckSeverity { +export interface SeededCheckPolicy { + source: string; checkName: string; } -/** Catalog row for a healthcheck name, as ingestion would have upserted - * (plus an operator-set severity). Upserts so tests can call it without +/** Policy row for a (source, check), as ingestion would have upserted + * (plus an operator-set ceiling). Upserts so tests can call it without * worrying whether ingestion already created a default row. */ -export async function seedHealthcheckSeverity( +export async function seedCheckPolicy( sql: Sql, opts: { checkName: string; - severity?: string; + source?: string; + ceiling?: string; + escalates?: boolean; notes?: string | null; + documentation?: string | null; }, -): Promise { +): Promise { + const source = opts.source ?? "alertd"; await sql.query( - `INSERT INTO healthcheck_severities (check_name, severity, notes) - VALUES ($1, $2, $3) - ON CONFLICT (check_name) - DO UPDATE SET severity = EXCLUDED.severity, notes = EXCLUDED.notes`, - [opts.checkName, opts.severity ?? "warning", opts.notes ?? null], + `INSERT INTO check_policies (source, check_name, ceiling, escalates, notes, documentation) + VALUES ($1, $2, $3, $4, $5, $6) + ON CONFLICT (source, check_name) + DO UPDATE SET ceiling = EXCLUDED.ceiling, escalates = EXCLUDED.escalates, notes = EXCLUDED.notes, documentation = EXCLUDED.documentation`, + [ + source, + opts.checkName, + opts.ceiling ?? "warning", + opts.escalates ?? false, + opts.notes ?? null, + opts.documentation ?? null, + ], ); - return { checkName: opts.checkName }; + return { source, checkName: opts.checkName }; +} + +/** The check name a silence ref maps to in scoped-policy storage: + * refs of source-reported checks carry the `health/` prefix. */ +function refToCheck(ref: string): string { + return ref.startsWith("health/") ? ref.slice("health/".length) : ref; } /** Server-scope silence for a `(source, ref)` pair, as the UI's - * silence button would create. For a healthcheck, pass - * `ref: "health/"` (source defaults to "status"). */ + * silence button would create: a scoped check policy with a skipped + * ceiling. For a healthcheck, pass `ref: "health/"` (source + * defaults to "alertd"). */ export async function seedServerSilencedRef( sql: Sql, opts: { @@ -275,10 +330,15 @@ export async function seedServerSilencedRef( }, ): Promise { await sql.query( - `INSERT INTO server_silenced_refs (server_id, source, ref, created_by) - VALUES ($1, $2, $3, $4) + `INSERT INTO scoped_check_policies (server_id, source, check_name, ceiling, created_by) + VALUES ($1, $2, $3, 'skipped', $4) ON CONFLICT DO NOTHING`, - [opts.serverId, opts.source ?? "status", opts.ref, opts.createdBy ?? null], + [ + opts.serverId, + opts.source ?? "alertd", + refToCheck(opts.ref), + opts.createdBy ?? null, + ], ); } @@ -294,10 +354,15 @@ export async function seedGroupSilencedRef( }, ): Promise { await sql.query( - `INSERT INTO server_group_silenced_refs (server_group_id, source, ref, created_by) - VALUES ($1, $2, $3, $4) + `INSERT INTO scoped_check_policies (server_group_id, source, check_name, ceiling, created_by) + VALUES ($1, $2, $3, 'skipped', $4) ON CONFLICT DO NOTHING`, - [opts.groupId, opts.source ?? "status", opts.ref, opts.createdBy ?? null], + [ + opts.groupId, + opts.source ?? "alertd", + refToCheck(opts.ref), + opts.createdBy ?? null, + ], ); } @@ -325,10 +390,11 @@ export async function seedIssue( sql: Sql, opts: { /** Server-scoped issue. Mutually exclusive with `serverGroupId` — the - * `issues` scope CHECK requires exactly one set. */ + * `issues` scope CHECK allows at most one set. */ serverId?: string | null; /** Group-scoped issue (e.g. a backup issue spanning the group). When set, - * leave `serverId` unset so the row satisfies the scope constraint. */ + * leave `serverId` unset so the row satisfies the scope constraint. + * Leaving both unset seeds a canopy-wide issue (a self-alert). */ serverGroupId?: string | null; source?: string; ref?: string; @@ -352,25 +418,41 @@ export async function seedIssue( ): Promise { const id = randomUUID(); const resolved = opts.resolved ?? false; + const active = resolved ? false : (opts.active ?? true); + const severity = opts.severity ?? "error"; + // Mirror the filing path's result stamping so seeded rows look like + // real check state: active rows degraded per the legacy severity the + // caller speaks, closed rows recovered. Critical means escalating. + const result = !active + ? "passed" + : severity === "warning" + ? "warning" + : "failed"; + const check = (opts.ref ?? "health").replace(/^health\//, ""); + // Every seeded issue was degraded at some point — that's what makes it + // an issue rather than healthy check state, which the listings exclude. await sql.query( `INSERT INTO issues - (id, server_id, server_group_id, device_id, source, ref, severity, message, description, active, first_seen, last_seen, resolved_at, resolved_by, resolved_reason) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, COALESCE($11::timestamptz, NOW()), NOW(), $12, $13, $14)`, + (id, server_id, server_group_id, device_id, source, ref, check_name, observed_result, effective_result, escalates, message, description, active, first_seen, last_seen, resolved_at, resolved_by, resolved_reason, degraded_since, last_degraded_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $8, $9, $10, $11, $12, COALESCE($13::timestamptz, NOW()), NOW(), $14, $15, $16, $17, NOW())`, [ id, opts.serverId ?? null, opts.serverGroupId ?? null, opts.deviceId ?? null, - opts.source ?? "status", + opts.source ?? "alertd", opts.ref ?? "health", - opts.severity ?? "error", + check, + result, + severity === "critical", opts.message ?? "Issue message", opts.description ?? null, - resolved ? false : (opts.active ?? true), + active, opts.firstSeen ?? null, resolved ? new Date().toISOString() : null, resolved ? (opts.resolvedBy ?? null) : null, resolved ? (opts.resolvedReason ?? null) : null, + active ? (opts.firstSeen ?? new Date().toISOString()) : null, ], ); return { id }; diff --git a/private-web/e2e/self-alerts.spec.ts b/private-web/e2e/self-alerts.spec.ts index cdfeffb6..c20b7ed3 100644 --- a/private-web/e2e/self-alerts.spec.ts +++ b/private-web/e2e/self-alerts.spec.ts @@ -3,9 +3,9 @@ import { expect, test } from "./test-fixtures"; const NIL = "00000000-0000-0000-0000-000000000000"; -// Self-alerts (issues on the nil "Canopy" server) get their own surface: a -// banner on every page and the /alerts view. They must NOT appear in the -// fleet issue listing on /incidents. +// Self-alerts (canopy-wide issues, scoped to neither server nor group) +// get their own surface: a banner on every page and the /alerts view. +// They must NOT appear in the fleet issue listing on /incidents. test.describe("self-alerts", () => { test.beforeEach(async ({ sql }) => { await resetSeededTables(sql); @@ -16,7 +16,6 @@ test.describe("self-alerts", () => { sql, }) => { await seedIssue(sql, { - serverId: NIL, source: "canopy", ref: "mcp-token-expiry", severity: "error", @@ -40,7 +39,6 @@ test.describe("self-alerts", () => { sql, }) => { await seedIssue(sql, { - serverId: NIL, source: "canopy", ref: "preflight-identity", severity: "critical", @@ -58,7 +56,6 @@ test.describe("self-alerts", () => { test("resolve clears the banner", async ({ page, sql }) => { await seedIssue(sql, { - serverId: NIL, source: "canopy", ref: "slack-delivery-failure", severity: "error", diff --git a/private-web/e2e/servers.spec.ts b/private-web/e2e/servers.spec.ts index aabed5be..af8ac0b3 100644 --- a/private-web/e2e/servers.spec.ts +++ b/private-web/e2e/servers.spec.ts @@ -121,36 +121,31 @@ test.describe("server edit page", () => { await expect(page.getByLabel(/^Name(\s*\*)?$/i)).toHaveValue(server.name); }); - test("toggling 'Allow status from Tamanu' on persists to the server", async ({ + test("saving a renamed server persists the change", async ({ page, sql, }) => { // Saving requires a group, so seed one and put the server in it. - const group = await seedServerGroup(sql, { name: "legacy-group" }); + const group = await seedServerGroup(sql, { name: "edit-group" }); const server = await seedServer(sql, { - name: "legacy-target", + name: "edit-save-target", groupId: group.id, }); await page.goto(`/servers/${server.id}/edit`); - // Off by default for a freshly-seeded server. - const toggle = page.getByRole("checkbox", { - name: "Allow status from Tamanu", - }); - await expect(toggle).not.toBeChecked(); - - await toggle.check(); + const nameField = page.getByLabel(/^Name(\s*\*)?$/i); + await nameField.fill("edit-save-renamed"); await page.getByRole("button", { name: /^save$/i }).click(); // Save navigates to the detail page. await page.waitForURL(`**/servers/${server.id}`); - const rows = await sql.query<{ allow_legacy_status: boolean }>( - "SELECT allow_legacy_status FROM servers WHERE id = $1", + const rows = await sql.query<{ name: string }>( + "SELECT name FROM servers WHERE id = $1", [server.id], ); - expect(rows[0]!.allow_legacy_status).toBe(true); + expect(rows[0]!.name).toBe("edit-save-renamed"); }); }); diff --git a/private-web/openapi.json b/private-web/openapi.json index cdaf8e62..12558ef1 100644 --- a/private-web/openapi.json +++ b/private-web/openapi.json @@ -2512,18 +2512,18 @@ "tags": [ "healthchecks" ], - "summary": "List the healthcheck severity catalog.", - "description": "Returns every known healthcheck name together with its current\nseverity policy, ordered by name. An entry exists for every check name\nany server has ever reported; new checks are added automatically the\nfirst time they're seen, with a default policy pending review.", + "summary": "List the check policy catalog.", + "description": "Returns every known (source, check) together with its current policy,\nordered by source then name. An entry exists for every check any\nsource has ever reported; new checks are added automatically the\nfirst time they're seen, with a default policy pending review.", "operationId": "healthcheck_list", "responses": { "200": { - "description": "Catalog rows ordered by check_name.", + "description": "Catalog rows ordered by source then check_name.", "content": { "application/json": { "schema": { "type": "array", "items": { - "$ref": "#/components/schemas/HealthcheckSeverityData" + "$ref": "#/components/schemas/CheckPolicyData" } } } @@ -2669,8 +2669,8 @@ "tags": [ "healthchecks" ], - "summary": "Update a healthcheck's severity policy.", - "description": "Sets the base severity (and optionally notes) for the given check, and\nmarks it as reviewed by the caller. Saving with the same severity as\nbefore still counts as a review, so an operator can acknowledge a\ncheck without changing its policy.", + "summary": "Update a check's policy.", + "description": "Sets the ceiling and escalation flag (and optionally notes) for the\ngiven (source, check), and marks it as reviewed by the caller. Saving\nwith the same values as before still counts as a review, so an\noperator can acknowledge a check without changing its policy.", "operationId": "healthcheck_update", "requestBody": { "content": { @@ -2688,7 +2688,64 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/HealthcheckSeverityData" + "$ref": "#/components/schemas/CheckPolicyData" + } + } + } + }, + "401": { + "description": "", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ProblemDetailsSchema" + } + } + } + }, + "403": { + "description": "", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ProblemDetailsSchema" + } + } + } + } + }, + "security": [ + { + "tailscale-admin": [] + } + ] + } + }, + "/api/healthchecks/update_documentation": { + "post": { + "tags": [ + "healthchecks" + ], + "summary": "Replace a check's documentation.", + "description": "Stores the markdown document presented alongside the check wherever\nits state appears, and over MCP. Sending `null` or a blank document\nclears it. Doesn't mark the policy as reviewed — documenting a check\nis not the same as reviewing its grading.", + "operationId": "healthcheck_update_documentation", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/UpdateDocumentationArgs" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Updated catalog row.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/CheckPolicyData" } } } @@ -2726,8 +2783,8 @@ "tags": [ "healthchecks" ], - "summary": "Replace a healthcheck's conditional severity rules.", - "description": "Stores a new set of conditional rules for the given check (or removes\nthem, if `rules` is `null`), and marks the check as reviewed by the\ncaller. Returns 400 if `rules` doesn't parse as a valid ladder — for\nexample an unknown comparison operator, a malformed variable\nreference, or an odd number of entries.", + "summary": "Replace a check's conditional rules.", + "description": "Stores a new set of conditional rules for the given (source, check)\n(or removes them, if `rules` is `null`), and marks the check as\nreviewed by the caller. Returns 400 if `rules` doesn't parse as a\nvalid ladder — for example an unknown comparison operator, a\nmalformed variable reference, or an odd number of entries.", "operationId": "healthcheck_update_rules", "requestBody": { "content": { @@ -2745,7 +2802,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/HealthcheckSeverityData" + "$ref": "#/components/schemas/CheckPolicyData" } } } @@ -3263,43 +3320,6 @@ ] } }, - "/api/issues/list_events": { - "post": { - "tags": [ - "issues" - ], - "summary": "List the events recorded against a specific issue, most recent first.", - "description": "Returns a page of events along with the total number of events recorded\nfor the issue, for pagination with `offset`/`limit`.", - "operationId": "list_events", - "requestBody": { - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/ListEventsArgs" - } - } - }, - "required": true - }, - "responses": { - "200": { - "description": "", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Page_EventData" - } - } - } - } - }, - "security": [ - { - "tailscale-user": [] - } - ] - } - }, "/api/issues/list_for_device": { "post": { "tags": [ @@ -3499,8 +3519,8 @@ "tags": [ "issues" ], - "summary": "Manually record an event against a server, creating or updating an issue.", - "description": "Finds or creates an issue keyed by the server and the given `ref`,\nappends this event to it, and returns the resulting issue. Returns 400\nif `ref` is empty or if `description` contains a newline.", + "summary": "Manually raise (or clear) a condition against a server.", + "description": "Finds or creates an issue keyed by the server and the given `ref`\nunder the `manual` source, grades the chosen result through the\ncondition's catalog policy, and returns the resulting issue. Returns\n400 if `ref` is empty or if `description` contains a newline.", "operationId": "submit_manual_event", "requestBody": { "content": { @@ -4132,7 +4152,7 @@ "self_alerts" ], "summary": "Mark a self-alert as resolved.", - "description": "Records that an operator has resolved the given self-alert and cancels\nits notification if one is still pending delivery (e.g. inside the\ninitial grace period before a low-severity alert is sent). Returns 404\nif no self-alert with that id exists.", + "description": "Records that an operator has resolved the given self-alert. Returns 404\nif no self-alert with that id exists.", "operationId": "self_alerts_resolve", "requestBody": { "content": { @@ -4146,7 +4166,7 @@ }, "responses": { "200": { - "description": "Alert marked operator-resolved; a pending notification is cancelled." + "description": "Alert marked operator-resolved." }, "401": { "description": "", @@ -5572,8 +5592,8 @@ "tags": [ "statuses" ], - "summary": "List the servers whose latest status reports one named healthcheck.", - "description": "Everything the per-healthcheck page needs: the catalog's configured\nseverity for `check` (if any) plus every live server whose **latest**\nstatus reports it — the current, real-time picture, not a history of\npast issues/events, though each failing server carries a\n`failing_since` timestamp derived from its active issue. This is the\ndata behind the `/healthchecks/:check` \"who's affected\" page, which\ndoubles as an operator TODO list and as a way to correlate servers\nsharing the same issue during a fleet-wide incident.", + "summary": "List the servers whose check state reports one (source, check).", + "description": "Everything the per-healthcheck page needs: the catalog's configured\npolicy for the (source, check) (if any) plus every live server's\ncurrent state for it — the real-time picture, with each degraded row\ncarrying `failing_since` (the start of its current degradation\nstreak). This is the data behind the `/healthchecks/:source/:check`\n\"who's affected\" page, which doubles as an operator TODO list and as\na way to correlate servers sharing the same issue during a\nfleet-wide incident.", "operationId": "status_check_attention", "requestBody": { "content": { @@ -5587,7 +5607,7 @@ }, "responses": { "200": { - "description": "The check's catalog severity and the servers currently reporting it.", + "description": "The check's catalog policy and the servers currently reporting it.", "content": { "application/json": { "schema": { @@ -6886,44 +6906,62 @@ "type": "object", "description": "Request body for [`check_attention`].", "required": [ + "source", "check" ], "properties": { "check": { "type": "string", "description": "The healthcheck name to look up, exactly as reported by devices in\n`health[].check` (an arbitrary, device/plugin-defined string)." + }, + "source": { + "type": "string", + "description": "The source that reports the check. A check's identity is the\n(source, check) pair — a same-named check from another source is\na different check." } } }, "CheckAttentionData": { "type": "object", - "description": "Response for [`check_attention`]: the queried check's catalog severity\n(if it has one yet) and every live server whose latest status reports\nit, failing or healthy.", + "description": "Response for [`check_attention`]: the queried check's catalog policy\n(if it has one yet) and every live server whose latest status reports\nit, failing or healthy.", "required": [ + "source", "check", + "escalates", "servers" ], "properties": { + "ceiling": { + "type": [ + "string", + "null" + ], + "description": "The configured policy ceiling for this (source, check), or `None`\nif the source has never reported it (so it has no catalog row)." + }, "check": { "type": "string", - "description": "The check name that was queried, echoed back so the page can\nrender its heading without re-decoding the request." + "description": "The check name that was queried." + }, + "documentation": { + "type": [ + "string", + "null" + ], + "description": "Operator-authored documentation for this (source, check)\n(markdown), or `None` if nobody has written it yet." + }, + "escalates": { + "type": "boolean", + "description": "Whether this check's policy escalates its effective failures." }, "servers": { "type": "array", "items": { "$ref": "#/components/schemas/CheckAttentionServerData" }, - "description": "Every live server whose latest status reports this check, at any\nresult, ordered as a TODO list: failed, warning, broken, passed,\nskipped (most urgent first), then by group name then server name.\nThe client filters out the passed/skipped tail unless the \"show\nhealthy\" toggle is on." + "description": "Every live server whose latest state from this source reports\nthis check, at any result, ordered as a TODO list: failed,\nwarning, broken, passed, skipped (most urgent first), then by\ngroup name then server name. The client filters out the\npassed/skipped tail unless the \"show healthy\" toggle is on." }, - "severity": { - "oneOf": [ - { - "type": "null" - }, - { - "$ref": "#/components/schemas/Severity", - "description": "The catalog's configured base severity for this check, or `None`\nif no server has ever reported it yet (so it has no catalog row)." - } - ] + "source": { + "type": "string", + "description": "The source that was queried, echoed back with `check` so the page\ncan render its heading without re-decoding the request." } } }, @@ -6939,7 +6977,7 @@ ], "properties": { "data": { - "description": "The check's full `health[]` entry from this server's latest status,\nverbatim (including the `check`/`healthy`/`result` keys), so the\nrow can expand to the same per-check detail the server page shows." + "description": "The check's own fields from its latest report, verbatim, so the\nrow can expand to the same per-check detail the server page shows." }, "failing_since": { "type": [ @@ -6947,7 +6985,7 @@ "null" ], "format": "date-time", - "description": "When this check started failing on this server: the `first_seen`\nof the still-active issue canopy filed at `(status,\nhealth/)` when the check degraded. `None` for servers\ncurrently reporting the check healthy, and for failing servers\nwith no active issue on file (e.g. the issue was\noperator-resolved, or the ref is silenced so nothing was filed)." + "description": "When the check's current degradation streak began. `None` for\nservers currently reporting the check healthy." }, "group_id": { "type": [ @@ -6966,7 +7004,7 @@ }, "result": { "$ref": "#/components/schemas/CheckResult", - "description": "The check's result on this server's latest status. The UI shows\nwarning/failed/broken servers by default and puts passed/skipped\nones behind a \"show healthy\" toggle." + "description": "The check's observed result on its latest report. The UI shows\nwarning/failed/broken servers by default and puts passed/skipped\nones behind a \"show healthy\" toggle." }, "server_id": { "type": "string", @@ -6980,7 +7018,91 @@ "status_created_at": { "type": "string", "format": "date-time", - "description": "When the reporting status was recorded." + "description": "When the check state last updated (the check's latest report)." + } + } + }, + "CheckPolicyData": { + "type": "object", + "description": "One (source, check)'s policy: the ceiling capping its effective\nresult, the escalation flag, plus optional conditional rules that can\ngrade a given report differently.", + "required": [ + "source", + "check_name", + "ceiling", + "escalates", + "first_seen", + "updated_at", + "pending_review", + "rule_count" + ], + "properties": { + "ceiling": { + "type": "string", + "description": "The maximum effective result for this check when no conditional\nrule (see `rules`) overrides it: an observed result more urgent\nthan the ceiling grades down to it. One of `failed`, `warning`,\n`passed`, or `skipped` (`skipped` also tells the reporting source\nit may stop running the check)." + }, + "check_name": { + "type": "string", + "description": "The healthcheck's name, exactly as reported by monitored servers." + }, + "documentation": { + "type": [ + "string", + "null" + ], + "description": "Operator-authored documentation for this check: a single markdown\ndocument. By convention it covers what the check observes, what\neach result means, and hints for solving a failure, but no\nstructure is enforced. `null` when nobody has documented it yet." + }, + "escalates": { + "type": "boolean", + "description": "Whether an effective failure of this check notifies immediately,\nbypassing the incident grace period." + }, + "first_seen": { + "type": "string", + "format": "date-time", + "description": "When this check was first reported and this policy entry was\ncreated." + }, + "notes": { + "type": [ + "string", + "null" + ], + "description": "Free-form operator notes about this check." + }, + "pending_review": { + "type": "boolean", + "description": "`true` if no operator has reviewed this policy yet." + }, + "reviewed_at": { + "type": [ + "string", + "null" + ], + "format": "date-time", + "description": "When an operator last reviewed or updated this policy. `null` if\nit has never been reviewed." + }, + "reviewed_by": { + "type": [ + "string", + "null" + ], + "description": "The operator who last reviewed this policy. `null` if it has\nnever been reviewed." + }, + "rule_count": { + "type": "integer", + "format": "int32", + "description": "Number of condition/result branches in `rules`; `0` when\n`rules` is `null` or couldn't be parsed. Lets a caller tell\nwhether conditional rules exist without parsing `rules` itself.", + "minimum": 0 + }, + "rules": { + "description": "Conditional rules that can grade a report to a different result\nthan the ceiling would — in any direction — depending on the\ncheck's own fields, the surrounding status report, or the\nreporting server's tags. `null` means no conditional rules are\nconfigured, and the ceiling always applies.\n\nWhen present, this is a single-key object shaped like\n`{\"if\": [condition_1, result_1, condition_2, result_2, ...]}`.\nConditions are tried in order, and the result paired with the\nfirst matching condition is used; if none match, the observed\nresult capped at the ceiling is used instead. There's no explicit\n\"else\" branch — the ceiling fallback plays that role — so the\narray must have an even number of entries and at least one pair.\n\nEach condition is a single-key object naming a comparison\noperator — one of `==`, `!=`, `<`, `<=`, `>`, `>=`, or `in_range`\n— whose value is a two-element array: a variable reference and a\nvalue to compare it against. A variable reference has the shape\n`{\"var\": \".\"}`, where `` is one of\n`check` (a field on the failing check itself), `status` (a\ntop-level field on the status report that contained it), or\n`tag` (a tag on the reporting server, merged with its group's\ntags). `in_range` compares a version-like string against a\nsemantic version range (e.g. `\">=1.2.0 <2.0.0\"`). If the named\nvariable isn't present in the data being evaluated, the condition\ndoesn't match. A `rules` value that doesn't parse into this shape\nis treated the same as `null` (no conditional rules)." + }, + "source": { + "type": "string", + "description": "The source that reports this check." + }, + "updated_at": { + "type": "string", + "format": "date-time", + "description": "When this policy was last modified." } } }, @@ -7541,74 +7663,6 @@ } } }, - "EventData": { - "type": "object", - "description": "A single recorded occurrence of an issue's underlying condition — one\npush from a device or a manually submitted event.", - "required": [ - "id", - "issue_id", - "created_at", - "severity", - "message", - "active", - "occurrences", - "last_seen" - ], - "properties": { - "active": { - "type": "boolean", - "description": "Whether the underlying condition was active as of this event." - }, - "created_at": { - "type": "string", - "format": "date-time", - "description": "When this event was recorded on the server." - }, - "description": { - "type": [ - "string", - "null" - ], - "description": "Short headline for this event, if one was given." - }, - "id": { - "type": "string", - "format": "uuid", - "description": "Unique identifier for this event." - }, - "issue_id": { - "type": "string", - "format": "uuid", - "description": "Id of the issue this event belongs to." - }, - "last_seen": { - "type": "string", - "format": "date-time", - "description": "When this condition was last seen recurring." - }, - "message": { - "type": "string", - "description": "Human-readable message describing this event." - }, - "occurred_at": { - "type": [ - "string", - "null" - ], - "format": "date-time", - "description": "When the underlying condition actually occurred, if reported\nseparately from the time it was recorded." - }, - "occurrences": { - "type": "integer", - "format": "int32", - "description": "Number of times this same condition has repeated and been coalesced\ninto this event rather than creating a new one." - }, - "severity": { - "$ref": "#/components/schemas/Severity", - "description": "Severity reported for this event." - } - } - }, "ExecuteArgs": { "type": "object", "description": "Request body for running a query in the SQL playground.", @@ -7869,7 +7923,7 @@ }, "HealthcheckSample": { "type": "object", - "description": "A real-world sample of the data a conditional rule can reference for a\ngiven healthcheck, taken from the most recent status report (across\nall servers) that included it.", + "description": "A real-world sample of the data a conditional rule can reference for a\ngiven healthcheck, taken from the most recent status report (across\nall servers) from the check's own source that included it.", "required": [ "status_extra", "check_extra", @@ -7950,95 +8004,37 @@ } } }, - "HealthcheckSeverityData": { + "HealthcheckUpdateArgs": { "type": "object", - "description": "A named healthcheck's alerting policy: the base severity assigned to\nits failures, plus optional conditional rules that can override that\nseverity based on the details of a given failure.", + "description": "Request body for updating a check's base policy.", "required": [ + "source", "check_name", - "severity", - "first_seen", - "updated_at", - "pending_review", - "rule_count" + "ceiling" ], "properties": { - "check_name": { - "type": "string", - "description": "The healthcheck's name, exactly as reported by monitored servers." - }, - "first_seen": { + "ceiling": { "type": "string", - "format": "date-time", - "description": "When this check was first reported and this policy entry was\ncreated." - }, - "notes": { - "type": [ - "string", - "null" - ], - "description": "Free-form operator notes about this check." + "description": "The ceiling to apply to this check's observed results when no\nconditional rule overrides it: one of `failed`, `warning`,\n`passed`, or `skipped`." }, - "pending_review": { - "type": "boolean", - "description": "`true` if no operator has reviewed this policy yet." - }, - "reviewed_at": { - "type": [ - "string", - "null" - ], - "format": "date-time", - "description": "When an operator last reviewed or updated this policy. `null` if\nit has never been reviewed." - }, - "reviewed_by": { - "type": [ - "string", - "null" - ], - "description": "The operator who last reviewed this policy. `null` if it has\nnever been reviewed." - }, - "rule_count": { - "type": "integer", - "format": "int32", - "description": "Number of condition/severity branches in `rules`; `0` when\n`rules` is `null` or couldn't be parsed. Lets a caller tell\nwhether conditional rules exist without parsing `rules` itself.", - "minimum": 0 - }, - "rules": { - "description": "Conditional rules that can assign a different severity than\n`severity`, depending on the failing check's own fields, the\nsurrounding status report, or the reporting server's tags. `null`\nmeans no conditional rules are configured, and `severity` always\napplies.\n\nWhen present, this is a single-key object shaped like\n`{\"if\": [condition_1, severity_1, condition_2, severity_2, ...]}`.\nConditions are tried in order, and the severity paired with the\nfirst matching condition is used; if none match, the base\n`severity` is used instead. There's no explicit \"else\" branch —\nthe fallback to `severity` plays that role — so the array must\nhave an even number of entries and at least one pair.\n\nEach condition is a single-key object naming a comparison\noperator — one of `==`, `!=`, `<`, `<=`, `>`, `>=`, or `in_range`\n— whose value is a two-element array: a variable reference and a\nvalue to compare it against. A variable reference has the shape\n`{\"var\": \".\"}`, where `` is one of\n`check` (a field on the failing check itself), `status` (a\ntop-level field on the status report that contained it), or\n`tag` (a tag on the reporting server, merged with its group's\ntags). `in_range` compares a version-like string against a\nsemantic version range (e.g. `\">=1.2.0 <2.0.0\"`). If the named\nvariable isn't present in the data being evaluated, the condition\ndoesn't match. A `rules` value that doesn't parse into this shape\nis treated the same as `null` (no conditional rules)." - }, - "severity": { - "$ref": "#/components/schemas/Severity", - "description": "The severity assigned to a failure of this check when no\nconditional rule (see `rules`) overrides it." - }, - "updated_at": { - "type": "string", - "format": "date-time", - "description": "When this policy was last modified." - } - } - }, - "HealthcheckUpdateArgs": { - "type": "object", - "description": "Request body for updating a healthcheck's base severity policy.", - "required": [ - "check_name", - "severity" - ], - "properties": { "check_name": { "type": "string", "description": "The healthcheck name to update; must already exist in the\ncatalog." }, + "escalates": { + "type": "boolean", + "description": "Whether an effective failure of this check should notify\nimmediately, bypassing the incident grace period." + }, "notes": { "type": [ "string", "null" ], - "description": "Operator notes to store alongside the new severity. Omitting this\nor sending `null` clears any existing notes — there's no way to\nleave them unchanged implicitly, so resend the current value to\nkeep it." + "description": "Operator notes to store alongside the new policy. Omitting this\nor sending `null` clears any existing notes — there's no way to\nleave them unchanged implicitly, so resend the current value to\nkeep it." }, - "severity": { - "$ref": "#/components/schemas/Severity", - "description": "The severity to assign to this check's failures when no\nconditional rule overrides it." + "source": { + "type": "string", + "description": "The source whose check to update." } } }, @@ -8124,11 +8120,9 @@ "description": "An operational incident: a group-scoped roll-up of related issues.\n\nAn incident opens when an issue on a server in the group crosses the\nseverity threshold, gathers further contributing issues while open, and\ncloses automatically once the last serious contributor clears. Operators\ncan additionally mark an incident resolved with a reason.", "required": [ "id", - "server_group_id", "server_group_name", "opened_at", "issue_count", - "event_count", "note_count", "created_at", "updated_at" @@ -8147,11 +8141,6 @@ "format": "date-time", "description": "When the incident record was created." }, - "event_count": { - "type": "integer", - "format": "int64", - "description": "Total number of events across all contributing issues." - }, "id": { "type": "string", "format": "uuid", @@ -8217,13 +8206,16 @@ "description": "Reason given when resolving: one of `fixed`, `wont_fix`, `expected`,\n`duplicate`, or `flapping`." }, "server_group_id": { - "type": "string", + "type": [ + "string", + "null" + ], "format": "uuid", - "description": "Identifier of the server group the incident belongs to." + "description": "Identifier of the server group the incident belongs to, or null for\na canopy-wide incident (aggregating canopy's self-alerts)." }, "server_group_name": { "type": "string", - "description": "Display name of the group this incident rolls up to. Empty only if\nthe group no longer exists, which should not happen in normal\noperation." + "description": "Display name of the group this incident rolls up to — `Canopy` for\na canopy-wide incident. Empty only if the group no longer exists,\nwhich should not happen in normal operation." }, "updated_at": { "type": "string", @@ -8453,7 +8445,7 @@ "server_host", "source", "ref", - "severity", + "escalates", "message", "active", "first_seen", @@ -8467,6 +8459,13 @@ "type": "boolean", "description": "Whether the underlying condition is still ongoing. `false` once the\ncondition has stopped recurring." }, + "check_name": { + "type": [ + "string", + "null" + ], + "description": "The check this issue tracks, when it is check state (health-check\nissues). Absent for issues that aren't check results yet." + }, "created_at": { "type": "string", "format": "date-time", @@ -8479,6 +8478,9 @@ ], "description": "Short headline describing the issue, if one was given." }, + "detail": { + "description": "The check's own fields from the latest report, verbatim." + }, "device_id": { "type": [ "string", @@ -8487,6 +8489,17 @@ "format": "uuid", "description": "Id of the device that reported the underlying event, if the issue\noriginated from a device push rather than a manual entry." }, + "effective_result": { + "type": [ + "string", + "null" + ], + "description": "What policy made of the latest observed result — the result canopy\nacts on." + }, + "escalates": { + "type": "boolean", + "description": "Whether the check's policy escalates: an effective failure notifies\nimmediately, bypassing incident grace." + }, "first_seen": { "type": "string", "format": "date-time", @@ -8513,6 +8526,13 @@ "type": "string", "description": "Latest human-readable message describing the issue's state." }, + "observed_result": { + "type": [ + "string", + "null" + ], + "description": "The result the source reported on the latest filing, before policy." + }, "ref": { "type": "string", "description": "Identifier used to match new incoming events to this issue; unique\nwithin its source and server." @@ -8587,10 +8607,6 @@ ], "description": "Display name of the affected server, when one is set. Falls back to\n`server_host` when absent." }, - "severity": { - "$ref": "#/components/schemas/Severity", - "description": "Current severity level of the issue." - }, "snoozed_until": { "type": [ "string", @@ -8685,23 +8701,23 @@ "format": "int64", "description": "Maximum number of issues to return. Defaults to 100 when omitted." }, - "serverGroupId": { + "results": { "type": [ - "string", + "array", "null" ], - "format": "uuid", - "description": "Restrict to issues whose server belongs to this group." + "items": { + "type": "string" + }, + "description": "Restrict to issues whose latest effective result is one of these.\nOmit to include all results." }, - "severities": { + "serverGroupId": { "type": [ - "array", + "string", "null" ], - "items": { - "$ref": "#/components/schemas/Severity" - }, - "description": "Restrict to issues at one of these severity levels. Omit to include\nall severities." + "format": "uuid", + "description": "Restrict to issues whose server belongs to this group." } } }, @@ -8933,36 +8949,6 @@ } } }, - "ListEventsArgs": { - "type": "object", - "description": "Selects an issue and a page of its events.", - "required": [ - "issue_id" - ], - "properties": { - "issue_id": { - "type": "string", - "format": "uuid", - "description": "Id of the issue whose events to list." - }, - "limit": { - "type": [ - "integer", - "null" - ], - "format": "int64", - "description": "Maximum number of events to return. Defaults to 100 when omitted." - }, - "offset": { - "type": [ - "integer", - "null" - ], - "format": "int64", - "description": "Number of events to skip, for pagination. Defaults to 0." - } - } - }, "ListForDeviceArgs": { "type": "object", "description": "Filters for listing the issues raised by one device.", @@ -9351,94 +9337,6 @@ } } }, - "Page_EventData": { - "type": "object", - "description": "A single page of a paginated list response.", - "required": [ - "items", - "total" - ], - "properties": { - "items": { - "type": "array", - "items": { - "type": "object", - "description": "A single recorded occurrence of an issue's underlying condition — one\npush from a device or a manually submitted event.", - "required": [ - "id", - "issue_id", - "created_at", - "severity", - "message", - "active", - "occurrences", - "last_seen" - ], - "properties": { - "active": { - "type": "boolean", - "description": "Whether the underlying condition was active as of this event." - }, - "created_at": { - "type": "string", - "format": "date-time", - "description": "When this event was recorded on the server." - }, - "description": { - "type": [ - "string", - "null" - ], - "description": "Short headline for this event, if one was given." - }, - "id": { - "type": "string", - "format": "uuid", - "description": "Unique identifier for this event." - }, - "issue_id": { - "type": "string", - "format": "uuid", - "description": "Id of the issue this event belongs to." - }, - "last_seen": { - "type": "string", - "format": "date-time", - "description": "When this condition was last seen recurring." - }, - "message": { - "type": "string", - "description": "Human-readable message describing this event." - }, - "occurred_at": { - "type": [ - "string", - "null" - ], - "format": "date-time", - "description": "When the underlying condition actually occurred, if reported\nseparately from the time it was recorded." - }, - "occurrences": { - "type": "integer", - "format": "int32", - "description": "Number of times this same condition has repeated and been coalesced\ninto this event rather than creating a new one." - }, - "severity": { - "$ref": "#/components/schemas/Severity", - "description": "Severity reported for this event." - } - } - }, - "description": "The items in this page." - }, - "total": { - "type": "integer", - "format": "int64", - "description": "The total number of items across all pages, not just this one — use\nthis to render page counts without a separate request.", - "minimum": 0 - } - } - }, "Page_ServerInfo": { "type": "object", "description": "A single page of a paginated list response.", @@ -9457,7 +9355,6 @@ "kind", "display_host", "is_monitored", - "allow_legacy_status", "alert_when_down_for", "notes", "tags", @@ -9469,10 +9366,6 @@ "format": "int64", "description": "Threshold in seconds for the reachability sweep to consider this\nserver down. Always positive; only consulted when `is_monitored`\nis `true`. The default at creation is 600 (10 minutes)." }, - "allow_legacy_status": { - "type": "boolean", - "description": "Whether this server may use the retired legacy `/status` format (a\npush with no `health` array). Off by default; when on, such a push\nonly refreshes reachability and carries prior healthchecks forward." - }, "archived": { "type": "boolean", "description": "Whether the server is archived (soft-deleted)." @@ -10864,12 +10757,17 @@ "type": "object", "description": "Request body identifying which healthcheck to sample data for.", "required": [ + "source", "check_name" ], "properties": { "check_name": { "type": "string", "description": "The healthcheck name to sample." + }, + "source": { + "type": "string", + "description": "The source that reports the check. A check's identity is the\n(source, check) pair; another source's same-named check may carry\nentirely different fields." } } }, @@ -10949,7 +10847,7 @@ "required": [ "id", "ref", - "severity", + "escalates", "message", "active", "first_seen", @@ -10960,6 +10858,17 @@ "type": "boolean", "description": "Whether the underlying condition is still ongoing. Becomes `false`\nonce the condition has cleared on its own, independently of whether\nan operator has resolved the alert." }, + "effective_result": { + "type": [ + "string", + "null" + ], + "description": "What policy made of it — the result canopy acts on." + }, + "escalates": { + "type": "boolean", + "description": "Whether this condition's policy escalates: an effective failure\nnotifies immediately, bypassing incident grace." + }, "first_seen": { "type": "string", "format": "date-time", @@ -10979,6 +10888,13 @@ "type": "string", "description": "Full detail message describing the condition." }, + "observed_result": { + "type": [ + "string", + "null" + ], + "description": "What canopy observed on the latest raise, before policy." + }, "ref": { "type": "string", "description": "The stable identifier of the underlying condition, e.g.\n`mcp-token-expiry`. Stable across repeated raises of the same\ncondition." @@ -10998,10 +10914,6 @@ ], "description": "The login of the operator who resolved this alert, or `null` if it\nhas not been resolved." }, - "severity": { - "$ref": "#/components/schemas/Severity", - "description": "How severe the condition is, from `critical` (most severe) down to\n`debug` (least)." - }, "title": { "type": [ "string", @@ -11114,13 +11026,6 @@ "format": "int64", "description": "New downtime threshold in seconds before this server is considered\ndown. Omit to leave unchanged." }, - "allow_legacy_status": { - "type": [ - "boolean", - "null" - ], - "description": "Whether to accept the retired legacy status format from this\nserver. Omit to leave unchanged." - }, "cloud": { "type": [ "boolean", @@ -11555,7 +11460,6 @@ "kind", "display_host", "is_monitored", - "allow_legacy_status", "alert_when_down_for", "notes", "tags", @@ -11567,10 +11471,6 @@ "format": "int64", "description": "Threshold in seconds for the reachability sweep to consider this\nserver down. Always positive; only consulted when `is_monitored`\nis `true`. The default at creation is 600 (10 minutes)." }, - "allow_legacy_status": { - "type": "boolean", - "description": "Whether this server may use the retired legacy `/status` format (a\npush with no `health` array). Off by default; when on, such a push\nonly refreshes reachability and carries prior healthchecks forward." - }, "archived": { "type": "boolean", "description": "Whether the server is archived (soft-deleted)." @@ -11722,6 +11622,7 @@ "created_at", "healthy", "health", + "source", "extra", "operators" ], @@ -11785,6 +11686,10 @@ ], "description": "PostgreSQL version the server reported, if any." }, + "source": { + "type": "string", + "description": "The source that pushed this status (e.g. `alertd`). Silences on\nthe checks it carries are keyed by this source." + }, "timezone": { "type": [ "string", @@ -12030,17 +11935,6 @@ } } }, - "Severity": { - "type": "string", - "description": "Canopy's severity vocabulary, narrowed from RFC 5424 to a five-level\nset with operator semantics:\n\n- `Debug`: never participates in incidents (filed for the audit\n trail / per-server view only).\n- `Info`, `Warning`: join an open incident for context but don't\n open one or keep one open on their own.\n- `Error`: opens an incident; sits in the per-group `slack_open_delay`\n holding window before the Slack notification ships.\n- `Critical`: opens an incident and bypasses the holding window —\n the Slack notification is enqueued for immediate delivery.\n\nStored as text in Postgres; validated as this enum at the API layer.\nDefault is `Error` (the most common severity for a deliberately\nfiled event). The legacy syslog severities `emergency` / `alert` /\n`notice` have been retired — see the\n`2026-05-29-000000-0000_restrict_severities` migration; the\n`FromStr` impl still accepts them as aliases for forward-compat with\nany device that hasn't been updated.", - "enum": [ - "critical", - "error", - "warning", - "info", - "debug" - ] - }, "ShortStatus": { "type": "string", "description": "Reachability of a server, based on how recently it last reported a status update.", @@ -12235,15 +12129,15 @@ "health", "extra", "operators", - "check_severities", + "check_results", "silenced_checks" ], "properties": { - "check_severities": { + "check_results": { "type": "object", - "description": "For each currently-unhealthy check in this push, the severity it\nwould be filed at if it turned into an issue. Healthy checks are\nomitted. An unhealthy check with no severity listed here should be\ntreated as a default (warning-level) severity.", + "description": "For each currently-unhealthy check in this push, the effective\nresult its policy grades it to. Healthy checks are omitted. An\nunhealthy check not listed here should be treated as warning.", "additionalProperties": { - "$ref": "#/components/schemas/Severity" + "type": "string" }, "propertyNames": { "type": "string" @@ -12362,7 +12256,7 @@ }, "SubmitManualEventArgs": { "type": "object", - "description": "A manually entered event to record against a server.", + "description": "A manually raised condition to record against a server.", "required": [ "serverId", "ref", @@ -12374,46 +12268,45 @@ "boolean", "null" ], - "description": "Whether the underlying condition is currently active. Defaults to\n`true` when omitted." + "description": "Whether the underlying condition is currently active. Defaults to\n`true` when omitted; `false` records it as cleared regardless of\n`result`." }, "description": { "type": [ "string", "null" ], - "description": "Short, single-line headline for the event. Must not contain\nnewlines — use `message` for multi-line detail." + "description": "Short, single-line headline for the condition. Must not contain\nnewlines — use `message` for multi-line detail." }, - "message": { - "type": "string", - "description": "Human-readable message describing the event. May be multi-line." - }, - "occurredAt": { + "escalates": { "type": [ - "string", + "boolean", "null" ], - "format": "date-time", - "description": "When the underlying condition actually occurred, if different from\nthe time of submission. Defaults to now when omitted." + "description": "Whether the condition's failures should notify immediately,\nbypassing the incident grace period. Only consulted the first time\na `ref` is seen (it seeds the condition's catalog entry); adjust\nlater from the healthchecks catalog." }, - "ref": { + "message": { "type": "string", - "description": "Identifier for the underlying condition. Events with the same `ref`\non the same server are coalesced into the same issue rather than\nopening a new one each time; use a fresh unique value if that\ndeduplication isn't wanted." + "description": "Human-readable message describing the condition. May be multi-line." }, - "serverId": { + "ref": { "type": "string", - "format": "uuid", - "description": "Id of the server the event applies to." + "description": "Identifier for the underlying condition. Reports with the same\n`ref` on the same server update the same issue rather than opening\na new one each time; use a fresh unique value if that deduplication\nisn't wanted." }, - "severity": { + "result": { "oneOf": [ { "type": "null" }, { - "$ref": "#/components/schemas/Severity", - "description": "Severity to record. If omitted, a default severity is used." + "$ref": "#/components/schemas/CheckResult", + "description": "The condition's result: `failed` (can open an incident) or\n`warning` (context only). Defaults to `failed`. `passed` records\nthe condition as cleared, same as `active: false`." } ] + }, + "serverId": { + "type": "string", + "format": "uuid", + "description": "Id of the server the condition applies to." } } }, @@ -12647,6 +12540,31 @@ } } }, + "UpdateDocumentationArgs": { + "type": "object", + "description": "Request body for replacing a check's documentation.", + "required": [ + "source", + "check_name" + ], + "properties": { + "check_name": { + "type": "string", + "description": "The healthcheck name to document; must already exist in the\ncatalog." + }, + "documentation": { + "type": [ + "string", + "null" + ], + "description": "The new markdown document, or `null` (or blank) to clear it." + }, + "source": { + "type": "string", + "description": "The source whose check to document." + } + } + }, "UpdateKeyNameArgs": { "type": "object", "description": "Request to rename (or clear the name of) a device key.", @@ -12670,8 +12588,9 @@ }, "UpdateRulesArgs": { "type": "object", - "description": "Request body for replacing a healthcheck's conditional severity rules.", + "description": "Request body for replacing a check's conditional rules.", "required": [ + "source", "check_name" ], "properties": { @@ -12680,7 +12599,11 @@ "description": "The healthcheck name whose rules to replace; must already exist\nin the catalog." }, "rules": { - "description": "The new conditional rules to store, or `null` to remove all\nconditional rules and rely solely on the base severity. Same\nshape as the `rules` field returned when listing checks. A ladder\nwith no condition/severity pairs is treated the same as `null`." + "description": "The new conditional rules to store, or `null` to remove all\nconditional rules and rely solely on the ceiling. Same shape as\nthe `rules` field returned when listing checks. A ladder with no\ncondition/result pairs is treated the same as `null`." + }, + "source": { + "type": "string", + "description": "The source whose check's rules to replace." } } }, diff --git a/private-web/src/App.tsx b/private-web/src/App.tsx index 229ecd24..371a69cc 100644 --- a/private-web/src/App.tsx +++ b/private-web/src/App.tsx @@ -185,7 +185,7 @@ export default function App() { } /> } /> } /> - } /> + } /> } /> } /> }> diff --git a/private-web/src/api-types.ts b/private-web/src/api-types.ts index ab26d004..a4197321 100644 --- a/private-web/src/api-types.ts +++ b/private-web/src/api-types.ts @@ -1265,10 +1265,10 @@ export interface paths { get?: never; put?: never; /** - * List the healthcheck severity catalog. - * @description Returns every known healthcheck name together with its current - * severity policy, ordered by name. An entry exists for every check name - * any server has ever reported; new checks are added automatically the + * List the check policy catalog. + * @description Returns every known (source, check) together with its current policy, + * ordered by source then name. An entry exists for every check any + * source has ever reported; new checks are added automatically the * first time they're seen, with a default policy pending review. */ post: operations["healthcheck_list"]; @@ -1337,11 +1337,11 @@ export interface paths { get?: never; put?: never; /** - * Update a healthcheck's severity policy. - * @description Sets the base severity (and optionally notes) for the given check, and - * marks it as reviewed by the caller. Saving with the same severity as - * before still counts as a review, so an operator can acknowledge a - * check without changing its policy. + * Update a check's policy. + * @description Sets the ceiling and escalation flag (and optionally notes) for the + * given (source, check), and marks it as reviewed by the caller. Saving + * with the same values as before still counts as a review, so an + * operator can acknowledge a check without changing its policy. */ post: operations["healthcheck_update"]; delete?: never; @@ -1350,6 +1350,29 @@ export interface paths { patch?: never; trace?: never; }; + "/api/healthchecks/update_documentation": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** + * Replace a check's documentation. + * @description Stores the markdown document presented alongside the check wherever + * its state appears, and over MCP. Sending `null` or a blank document + * clears it. Doesn't mark the policy as reviewed — documenting a check + * is not the same as reviewing its grading. + */ + post: operations["healthcheck_update_documentation"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; "/api/healthchecks/update_rules": { parameters: { query?: never; @@ -1360,12 +1383,12 @@ export interface paths { get?: never; put?: never; /** - * Replace a healthcheck's conditional severity rules. - * @description Stores a new set of conditional rules for the given check (or removes - * them, if `rules` is `null`), and marks the check as reviewed by the - * caller. Returns 400 if `rules` doesn't parse as a valid ladder — for - * example an unknown comparison operator, a malformed variable - * reference, or an odd number of entries. + * Replace a check's conditional rules. + * @description Stores a new set of conditional rules for the given (source, check) + * (or removes them, if `rules` is `null`), and marks the check as + * reviewed by the caller. Returns 400 if `rules` doesn't parse as a + * valid ladder — for example an unknown comparison operator, a + * malformed variable reference, or an odd number of entries. */ post: operations["healthcheck_update_rules"]; delete?: never; @@ -1632,27 +1655,6 @@ export interface paths { patch?: never; trace?: never; }; - "/api/issues/list_events": { - parameters: { - query?: never; - header?: never; - path?: never; - cookie?: never; - }; - get?: never; - put?: never; - /** - * List the events recorded against a specific issue, most recent first. - * @description Returns a page of events along with the total number of events recorded - * for the issue, for pagination with `offset`/`limit`. - */ - post: operations["list_events"]; - delete?: never; - options?: never; - head?: never; - patch?: never; - trace?: never; - }; "/api/issues/list_for_device": { parameters: { query?: never; @@ -1770,10 +1772,11 @@ export interface paths { get?: never; put?: never; /** - * Manually record an event against a server, creating or updating an issue. - * @description Finds or creates an issue keyed by the server and the given `ref`, - * appends this event to it, and returns the resulting issue. Returns 400 - * if `ref` is empty or if `description` contains a newline. + * Manually raise (or clear) a condition against a server. + * @description Finds or creates an issue keyed by the server and the given `ref` + * under the `manual` source, grades the chosen result through the + * condition's catalog policy, and returns the resulting issue. Returns + * 400 if `ref` is empty or if `description` contains a newline. */ post: operations["submit_manual_event"]; delete?: never; @@ -2109,9 +2112,7 @@ export interface paths { put?: never; /** * Mark a self-alert as resolved. - * @description Records that an operator has resolved the given self-alert and cancels - * its notification if one is still pending delivery (e.g. inside the - * initial grace period before a low-severity alert is sent). Returns 404 + * @description Records that an operator has resolved the given self-alert. Returns 404 * if no self-alert with that id exists. */ post: operations["self_alerts_resolve"]; @@ -2874,15 +2875,15 @@ export interface paths { get?: never; put?: never; /** - * List the servers whose latest status reports one named healthcheck. + * List the servers whose check state reports one (source, check). * @description Everything the per-healthcheck page needs: the catalog's configured - * severity for `check` (if any) plus every live server whose **latest** - * status reports it — the current, real-time picture, not a history of - * past issues/events, though each failing server carries a - * `failing_since` timestamp derived from its active issue. This is the - * data behind the `/healthchecks/:check` "who's affected" page, which - * doubles as an operator TODO list and as a way to correlate servers - * sharing the same issue during a fleet-wide incident. + * policy for the (source, check) (if any) plus every live server's + * current state for it — the real-time picture, with each degraded row + * carrying `failing_since` (the start of its current degradation + * streak). This is the data behind the `/healthchecks/:source/:check` + * "who's affected" page, which doubles as an operator TODO list and as + * a way to correlate servers sharing the same issue during a + * fleet-wide incident. */ post: operations["status_check_attention"]; delete?: never; @@ -3663,27 +3664,46 @@ export interface components { * `health[].check` (an arbitrary, device/plugin-defined string). */ check: string; + /** + * @description The source that reports the check. A check's identity is the + * (source, check) pair — a same-named check from another source is + * a different check. + */ + source: string; }; /** - * @description Response for [`check_attention`]: the queried check's catalog severity + * @description Response for [`check_attention`]: the queried check's catalog policy * (if it has one yet) and every live server whose latest status reports * it, failing or healthy. */ CheckAttentionData: { /** - * @description The check name that was queried, echoed back so the page can - * render its heading without re-decoding the request. + * @description The configured policy ceiling for this (source, check), or `None` + * if the source has never reported it (so it has no catalog row). */ + ceiling?: string | null; + /** @description The check name that was queried. */ check: string; /** - * @description Every live server whose latest status reports this check, at any - * result, ordered as a TODO list: failed, warning, broken, passed, - * skipped (most urgent first), then by group name then server name. - * The client filters out the passed/skipped tail unless the "show - * healthy" toggle is on. + * @description Operator-authored documentation for this (source, check) + * (markdown), or `None` if nobody has written it yet. + */ + documentation?: string | null; + /** @description Whether this check's policy escalates its effective failures. */ + escalates: boolean; + /** + * @description Every live server whose latest state from this source reports + * this check, at any result, ordered as a TODO list: failed, + * warning, broken, passed, skipped (most urgent first), then by + * group name then server name. The client filters out the + * passed/skipped tail unless the "show healthy" toggle is on. */ servers: components["schemas"]["CheckAttentionServerData"][]; - severity?: null | components["schemas"]["Severity"]; + /** + * @description The source that was queried, echoed back with `check` so the page + * can render its heading without re-decoding the request. + */ + source: string; }; /** * @description One server whose latest status reports [`CheckAttentionData::check`], @@ -3691,19 +3711,14 @@ export interface components { */ CheckAttentionServerData: { /** - * @description The check's full `health[]` entry from this server's latest status, - * verbatim (including the `check`/`healthy`/`result` keys), so the + * @description The check's own fields from its latest report, verbatim, so the * row can expand to the same per-check detail the server page shows. */ data: unknown; /** * Format: date-time - * @description When this check started failing on this server: the `first_seen` - * of the still-active issue canopy filed at `(status, - * health/)` when the check degraded. `None` for servers - * currently reporting the check healthy, and for failing servers - * with no active issue on file (e.g. the issue was - * operator-resolved, or the ref is silenced so nothing was filed). + * @description When the check's current degradation streak began. `None` for + * servers currently reporting the check healthy. */ failing_since?: string | null; /** @@ -3715,7 +3730,7 @@ export interface components { /** @description The server's group name, if it belongs to one. */ group_name?: string | null; /** - * @description The check's result on this server's latest status. The UI shows + * @description The check's observed result on its latest report. The UI shows * warning/failed/broken servers by default and puts passed/skipped * ones behind a "show healthy" toggle. */ @@ -3729,10 +3744,104 @@ export interface components { server_name: string; /** * Format: date-time - * @description When the reporting status was recorded. + * @description When the check state last updated (the check's latest report). */ status_created_at: string; }; + /** + * @description One (source, check)'s policy: the ceiling capping its effective + * result, the escalation flag, plus optional conditional rules that can + * grade a given report differently. + */ + CheckPolicyData: { + /** + * @description The maximum effective result for this check when no conditional + * rule (see `rules`) overrides it: an observed result more urgent + * than the ceiling grades down to it. One of `failed`, `warning`, + * `passed`, or `skipped` (`skipped` also tells the reporting source + * it may stop running the check). + */ + ceiling: string; + /** @description The healthcheck's name, exactly as reported by monitored servers. */ + check_name: string; + /** + * @description Operator-authored documentation for this check: a single markdown + * document. By convention it covers what the check observes, what + * each result means, and hints for solving a failure, but no + * structure is enforced. `null` when nobody has documented it yet. + */ + documentation?: string | null; + /** + * @description Whether an effective failure of this check notifies immediately, + * bypassing the incident grace period. + */ + escalates: boolean; + /** + * Format: date-time + * @description When this check was first reported and this policy entry was + * created. + */ + first_seen: string; + /** @description Free-form operator notes about this check. */ + notes?: string | null; + /** @description `true` if no operator has reviewed this policy yet. */ + pending_review: boolean; + /** + * Format: date-time + * @description When an operator last reviewed or updated this policy. `null` if + * it has never been reviewed. + */ + reviewed_at?: string | null; + /** + * @description The operator who last reviewed this policy. `null` if it has + * never been reviewed. + */ + reviewed_by?: string | null; + /** + * Format: int32 + * @description Number of condition/result branches in `rules`; `0` when + * `rules` is `null` or couldn't be parsed. Lets a caller tell + * whether conditional rules exist without parsing `rules` itself. + */ + rule_count: number; + /** + * @description Conditional rules that can grade a report to a different result + * than the ceiling would — in any direction — depending on the + * check's own fields, the surrounding status report, or the + * reporting server's tags. `null` means no conditional rules are + * configured, and the ceiling always applies. + * + * When present, this is a single-key object shaped like + * `{"if": [condition_1, result_1, condition_2, result_2, ...]}`. + * Conditions are tried in order, and the result paired with the + * first matching condition is used; if none match, the observed + * result capped at the ceiling is used instead. There's no explicit + * "else" branch — the ceiling fallback plays that role — so the + * array must have an even number of entries and at least one pair. + * + * Each condition is a single-key object naming a comparison + * operator — one of `==`, `!=`, `<`, `<=`, `>`, `>=`, or `in_range` + * — whose value is a two-element array: a variable reference and a + * value to compare it against. A variable reference has the shape + * `{"var": "."}`, where `` is one of + * `check` (a field on the failing check itself), `status` (a + * top-level field on the status report that contained it), or + * `tag` (a tag on the reporting server, merged with its group's + * tags). `in_range` compares a version-like string against a + * semantic version range (e.g. `">=1.2.0 <2.0.0"`). If the named + * variable isn't present in the data being evaluated, the condition + * doesn't match. A `rules` value that doesn't parse into this shape + * is treated the same as `null` (no conditional rules). + */ + rules?: unknown; + /** @description The source that reports this check. */ + source: string; + /** + * Format: date-time + * @description When this policy was last modified. + */ + updated_at: string; + }; /** * @description Outcome of a single health check reported in a server's status update. * @@ -4052,52 +4161,6 @@ export interface components { */ ticket: string; }; - /** - * @description A single recorded occurrence of an issue's underlying condition — one - * push from a device or a manually submitted event. - */ - EventData: { - /** @description Whether the underlying condition was active as of this event. */ - active: boolean; - /** - * Format: date-time - * @description When this event was recorded on the server. - */ - created_at: string; - /** @description Short headline for this event, if one was given. */ - description?: string | null; - /** - * Format: uuid - * @description Unique identifier for this event. - */ - id: string; - /** - * Format: uuid - * @description Id of the issue this event belongs to. - */ - issue_id: string; - /** - * Format: date-time - * @description When this condition was last seen recurring. - */ - last_seen: string; - /** @description Human-readable message describing this event. */ - message: string; - /** - * Format: date-time - * @description When the underlying condition actually occurred, if reported - * separately from the time it was recorded. - */ - occurred_at?: string | null; - /** - * Format: int32 - * @description Number of times this same condition has repeated and been coalesced - * into this event rather than creating a new one. - */ - occurrences: number; - /** @description Severity reported for this event. */ - severity: components["schemas"]["Severity"]; - }; /** @description Request body for running a query in the SQL playground. */ ExecuteArgs: { /** @description The query to execute. */ @@ -4261,7 +4324,7 @@ export interface components { /** * @description A real-world sample of the data a conditional rule can reference for a * given healthcheck, taken from the most recent status report (across - * all servers) that included it. + * all servers) from the check's own source that included it. */ HealthcheckSample: { /** @@ -4306,102 +4369,33 @@ export interface components { check_name: string; sample?: null | components["schemas"]["HealthcheckSample"]; }; - /** - * @description A named healthcheck's alerting policy: the base severity assigned to - * its failures, plus optional conditional rules that can override that - * severity based on the details of a given failure. - */ - HealthcheckSeverityData: { - /** @description The healthcheck's name, exactly as reported by monitored servers. */ - check_name: string; - /** - * Format: date-time - * @description When this check was first reported and this policy entry was - * created. - */ - first_seen: string; - /** @description Free-form operator notes about this check. */ - notes?: string | null; - /** @description `true` if no operator has reviewed this policy yet. */ - pending_review: boolean; - /** - * Format: date-time - * @description When an operator last reviewed or updated this policy. `null` if - * it has never been reviewed. - */ - reviewed_at?: string | null; - /** - * @description The operator who last reviewed this policy. `null` if it has - * never been reviewed. - */ - reviewed_by?: string | null; - /** - * Format: int32 - * @description Number of condition/severity branches in `rules`; `0` when - * `rules` is `null` or couldn't be parsed. Lets a caller tell - * whether conditional rules exist without parsing `rules` itself. - */ - rule_count: number; - /** - * @description Conditional rules that can assign a different severity than - * `severity`, depending on the failing check's own fields, the - * surrounding status report, or the reporting server's tags. `null` - * means no conditional rules are configured, and `severity` always - * applies. - * - * When present, this is a single-key object shaped like - * `{"if": [condition_1, severity_1, condition_2, severity_2, ...]}`. - * Conditions are tried in order, and the severity paired with the - * first matching condition is used; if none match, the base - * `severity` is used instead. There's no explicit "else" branch — - * the fallback to `severity` plays that role — so the array must - * have an even number of entries and at least one pair. - * - * Each condition is a single-key object naming a comparison - * operator — one of `==`, `!=`, `<`, `<=`, `>`, `>=`, or `in_range` - * — whose value is a two-element array: a variable reference and a - * value to compare it against. A variable reference has the shape - * `{"var": "."}`, where `` is one of - * `check` (a field on the failing check itself), `status` (a - * top-level field on the status report that contained it), or - * `tag` (a tag on the reporting server, merged with its group's - * tags). `in_range` compares a version-like string against a - * semantic version range (e.g. `">=1.2.0 <2.0.0"`). If the named - * variable isn't present in the data being evaluated, the condition - * doesn't match. A `rules` value that doesn't parse into this shape - * is treated the same as `null` (no conditional rules). - */ - rules?: unknown; - /** - * @description The severity assigned to a failure of this check when no - * conditional rule (see `rules`) overrides it. - */ - severity: components["schemas"]["Severity"]; + /** @description Request body for updating a check's base policy. */ + HealthcheckUpdateArgs: { /** - * Format: date-time - * @description When this policy was last modified. + * @description The ceiling to apply to this check's observed results when no + * conditional rule overrides it: one of `failed`, `warning`, + * `passed`, or `skipped`. */ - updated_at: string; - }; - /** @description Request body for updating a healthcheck's base severity policy. */ - HealthcheckUpdateArgs: { + ceiling: string; /** * @description The healthcheck name to update; must already exist in the * catalog. */ check_name: string; /** - * @description Operator notes to store alongside the new severity. Omitting this + * @description Whether an effective failure of this check should notify + * immediately, bypassing the incident grace period. + */ + escalates?: boolean; + /** + * @description Operator notes to store alongside the new policy. Omitting this * or sending `null` clears any existing notes — there's no way to * leave them unchanged implicitly, so resend the current value to * keep it. */ notes?: string | null; - /** - * @description The severity to assign to this check's failures when no - * conditional rule overrides it. - */ - severity: components["schemas"]["Severity"]; + /** @description The source whose check to update. */ + source: string; }; /** @description Pagination parameters for browsing the shared query history. */ HistoryArgs: { @@ -4472,11 +4466,6 @@ export interface components { * @description When the incident record was created. */ created_at: string; - /** - * Format: int64 - * @description Total number of events across all contributing issues. - */ - event_count: number; /** * Format: uuid * @description Unique identifier of the incident. @@ -4527,13 +4516,14 @@ export interface components { resolved_reason?: string | null; /** * Format: uuid - * @description Identifier of the server group the incident belongs to. + * @description Identifier of the server group the incident belongs to, or null for + * a canopy-wide incident (aggregating canopy's self-alerts). */ - server_group_id: string; + server_group_id?: string | null; /** - * @description Display name of the group this incident rolls up to. Empty only if - * the group no longer exists, which should not happen in normal - * operation. + * @description Display name of the group this incident rolls up to — `Canopy` for + * a canopy-wide incident. Empty only if the group no longer exists, + * which should not happen in normal operation. */ server_group_name: string; /** @@ -4692,6 +4682,11 @@ export interface components { * condition has stopped recurring. */ active: boolean; + /** + * @description The check this issue tracks, when it is check state (health-check + * issues). Absent for issues that aren't check results yet. + */ + check_name?: string | null; /** * Format: date-time * @description When this issue record was created. @@ -4699,12 +4694,24 @@ export interface components { created_at: string; /** @description Short headline describing the issue, if one was given. */ description?: string | null; + /** @description The check's own fields from the latest report, verbatim. */ + detail?: unknown; /** * Format: uuid * @description Id of the device that reported the underlying event, if the issue * originated from a device push rather than a manual entry. */ device_id?: string | null; + /** + * @description What policy made of the latest observed result — the result canopy + * acts on. + */ + effective_result?: string | null; + /** + * @description Whether the check's policy escalates: an effective failure notifies + * immediately, bypassing incident grace. + */ + escalates: boolean; /** * Format: date-time * @description When the issue was first raised. @@ -4727,6 +4734,8 @@ export interface components { last_seen: string; /** @description Latest human-readable message describing the issue's state. */ message: string; + /** @description The result the source reported on the latest filing, before policy. */ + observed_result?: string | null; /** * @description Identifier used to match new incoming events to this issue; unique * within its source and server. @@ -4777,8 +4786,6 @@ export interface components { * `server_host` when absent. */ server_name?: string | null; - /** @description Current severity level of the issue. */ - severity: components["schemas"]["Severity"]; /** * Format: date-time * @description If set, the issue is snoozed and won't demand attention again until @@ -4845,16 +4852,16 @@ export interface components { * @description Maximum number of issues to return. Defaults to 100 when omitted. */ limit?: number | null; + /** + * @description Restrict to issues whose latest effective result is one of these. + * Omit to include all results. + */ + results?: string[] | null; /** * Format: uuid * @description Restrict to issues whose server belongs to this group. */ serverGroupId?: string | null; - /** - * @description Restrict to issues at one of these severity levels. Omit to include - * all severities. - */ - severities?: components["schemas"]["Severity"][] | null; }; /** @description Filters for listing the issues raised against one server. */ IssueListForServerArgs: { @@ -5000,24 +5007,6 @@ export interface components { */ limit?: number | null; }; - /** @description Selects an issue and a page of its events. */ - ListEventsArgs: { - /** - * Format: uuid - * @description Id of the issue whose events to list. - */ - issue_id: string; - /** - * Format: int64 - * @description Maximum number of events to return. Defaults to 100 when omitted. - */ - limit?: number | null; - /** - * Format: int64 - * @description Number of events to skip, for pagination. Defaults to 0. - */ - offset?: number | null; - }; /** @description Filters for listing the issues raised by one device. */ ListForDeviceArgs: { /** @@ -5252,58 +5241,6 @@ export interface components { total: number; }; /** @description A single page of a paginated list response. */ - Page_EventData: { - /** @description The items in this page. */ - items: { - /** @description Whether the underlying condition was active as of this event. */ - active: boolean; - /** - * Format: date-time - * @description When this event was recorded on the server. - */ - created_at: string; - /** @description Short headline for this event, if one was given. */ - description?: string | null; - /** - * Format: uuid - * @description Unique identifier for this event. - */ - id: string; - /** - * Format: uuid - * @description Id of the issue this event belongs to. - */ - issue_id: string; - /** - * Format: date-time - * @description When this condition was last seen recurring. - */ - last_seen: string; - /** @description Human-readable message describing this event. */ - message: string; - /** - * Format: date-time - * @description When the underlying condition actually occurred, if reported - * separately from the time it was recorded. - */ - occurred_at?: string | null; - /** - * Format: int32 - * @description Number of times this same condition has repeated and been coalesced - * into this event rather than creating a new one. - */ - occurrences: number; - /** @description Severity reported for this event. */ - severity: components["schemas"]["Severity"]; - }[]; - /** - * Format: int64 - * @description The total number of items across all pages, not just this one — use - * this to render page counts without a separate request. - */ - total: number; - }; - /** @description A single page of a paginated list response. */ Page_ServerInfo: { /** @description The items in this page. */ items: { @@ -5314,12 +5251,6 @@ export interface components { * is `true`. The default at creation is 600 (10 minutes). */ alert_when_down_for: number; - /** - * @description Whether this server may use the retired legacy `/status` format (a - * push with no `health` array). Off by default; when on, such a push - * only refreshes reachability and carries prior healthchecks forward. - */ - allow_legacy_status: boolean; /** @description Whether the server is archived (soft-deleted). */ archived: boolean; /** @description Whether this server runs in a cloud environment, if known. */ @@ -6230,6 +6161,12 @@ export interface components { SampleArgs: { /** @description The healthcheck name to sample. */ check_name: string; + /** + * @description The source that reports the check. A check's identity is the + * (source, check) pair; another source's same-named check may carry + * entirely different fields. + */ + source: string; }; /** * @description Request body for creating a new snippet, or saving a new version of an @@ -6279,6 +6216,13 @@ export interface components { * an operator has resolved the alert. */ active: boolean; + /** @description What policy made of it — the result canopy acts on. */ + effective_result?: string | null; + /** + * @description Whether this condition's policy escalates: an effective failure + * notifies immediately, bypassing incident grace. + */ + escalates: boolean; /** * Format: date-time * @description When this condition was first raised. @@ -6296,6 +6240,8 @@ export interface components { last_seen: string; /** @description Full detail message describing the condition. */ message: string; + /** @description What canopy observed on the latest raise, before policy. */ + observed_result?: string | null; /** * @description The stable identifier of the underlying condition, e.g. * `mcp-token-expiry`. Stable across repeated raises of the same @@ -6313,11 +6259,6 @@ export interface components { * has not been resolved. */ resolved_by?: string | null; - /** - * @description How severe the condition is, from `critical` (most severe) down to - * `debug` (least). - */ - severity: components["schemas"]["Severity"]; /** @description Single-line headline. */ title?: string | null; }; @@ -6402,11 +6343,6 @@ export interface components { * down. Omit to leave unchanged. */ alert_when_down_for?: number | null; - /** - * @description Whether to accept the retired legacy status format from this - * server. Omit to leave unchanged. - */ - allow_legacy_status?: boolean | null; /** * @description Whether the server runs in a cloud environment, or `null` to clear. * Omit to leave unchanged. @@ -6641,12 +6577,6 @@ export interface components { * is `true`. The default at creation is 600 (10 minutes). */ alert_when_down_for: number; - /** - * @description Whether this server may use the retired legacy `/status` format (a - * push with no `health` array). Off by default; when on, such a push - * only refreshes reachability and carries prior healthchecks forward. - */ - allow_legacy_status: boolean; /** @description Whether the server is archived (soft-deleted). */ archived: boolean; /** @description Whether this server runs in a cloud environment, if known. */ @@ -6759,6 +6689,11 @@ export interface components { platform?: string | null; /** @description PostgreSQL version the server reported, if any. */ postgres?: string | null; + /** + * @description The source that pushed this status (e.g. `alertd`). Silences on + * the checks it carries are keyed by this source. + */ + source: string; /** @description Timezone the server reported, if any. */ timezone?: string | null; version?: null | components["schemas"]["VersionStr"]; @@ -6894,29 +6829,6 @@ export interface components { /** @description Backup type these defaults apply to. */ type: string; }; - /** - * @description Canopy's severity vocabulary, narrowed from RFC 5424 to a five-level - * set with operator semantics: - * - * - `Debug`: never participates in incidents (filed for the audit - * trail / per-server view only). - * - `Info`, `Warning`: join an open incident for context but don't - * open one or keep one open on their own. - * - `Error`: opens an incident; sits in the per-group `slack_open_delay` - * holding window before the Slack notification ships. - * - `Critical`: opens an incident and bypasses the holding window — - * the Slack notification is enqueued for immediate delivery. - * - * Stored as text in Postgres; validated as this enum at the API layer. - * Default is `Error` (the most common severity for a deliberately - * filed event). The legacy syslog severities `emergency` / `alert` / - * `notice` have been retired — see the - * `2026-05-29-000000-0000_restrict_severities` migration; the - * `FromStr` impl still accepts them as aliases for forward-compat with - * any device that hasn't been updated. - * @enum {string} - */ - Severity: "critical" | "error" | "warning" | "info" | "debug"; /** * @description Reachability of a server, based on how recently it last reported a status update. * @enum {string} @@ -7047,13 +6959,12 @@ export interface components { */ StatusSnapshotData: { /** - * @description For each currently-unhealthy check in this push, the severity it - * would be filed at if it turned into an issue. Healthy checks are - * omitted. An unhealthy check with no severity listed here should be - * treated as a default (warning-level) severity. + * @description For each currently-unhealthy check in this push, the effective + * result its policy grades it to. Healthy checks are omitted. An + * unhealthy check not listed here should be treated as warning. */ - check_severities: { - [key: string]: components["schemas"]["Severity"]; + check_results: { + [key: string]: string; }; /** * Format: date-time @@ -7130,39 +7041,41 @@ export interface components { */ version_distance?: number | null; }; - /** @description A manually entered event to record against a server. */ + /** @description A manually raised condition to record against a server. */ SubmitManualEventArgs: { /** * @description Whether the underlying condition is currently active. Defaults to - * `true` when omitted. + * `true` when omitted; `false` records it as cleared regardless of + * `result`. */ active?: boolean | null; /** - * @description Short, single-line headline for the event. Must not contain + * @description Short, single-line headline for the condition. Must not contain * newlines — use `message` for multi-line detail. */ description?: string | null; - /** @description Human-readable message describing the event. May be multi-line. */ - message: string; /** - * Format: date-time - * @description When the underlying condition actually occurred, if different from - * the time of submission. Defaults to now when omitted. + * @description Whether the condition's failures should notify immediately, + * bypassing the incident grace period. Only consulted the first time + * a `ref` is seen (it seeds the condition's catalog entry); adjust + * later from the healthchecks catalog. */ - occurredAt?: string | null; + escalates?: boolean | null; + /** @description Human-readable message describing the condition. May be multi-line. */ + message: string; /** - * @description Identifier for the underlying condition. Events with the same `ref` - * on the same server are coalesced into the same issue rather than - * opening a new one each time; use a fresh unique value if that - * deduplication isn't wanted. + * @description Identifier for the underlying condition. Reports with the same + * `ref` on the same server update the same issue rather than opening + * a new one each time; use a fresh unique value if that deduplication + * isn't wanted. */ ref: string; + result?: null | components["schemas"]["CheckResult"]; /** * Format: uuid - * @description Id of the server the event applies to. + * @description Id of the server the condition applies to. */ serverId: string; - severity?: null | components["schemas"]["Severity"]; }; /** @description Fleet-wide summary of software versions currently running in production. */ SummaryData: { @@ -7282,6 +7195,18 @@ export interface components { /** @description Exact version string to update (e.g. `"1.2.3"`). */ version: string; }; + /** @description Request body for replacing a check's documentation. */ + UpdateDocumentationArgs: { + /** + * @description The healthcheck name to document; must already exist in the + * catalog. + */ + check_name: string; + /** @description The new markdown document, or `null` (or blank) to clear it. */ + documentation?: string | null; + /** @description The source whose check to document. */ + source: string; + }; /** @description Request to rename (or clear the name of) a device key. */ UpdateKeyNameArgs: { /** @@ -7292,7 +7217,7 @@ export interface components { /** @description New display name for the key, or null to clear it. */ name?: string | null; }; - /** @description Request body for replacing a healthcheck's conditional severity rules. */ + /** @description Request body for replacing a check's conditional rules. */ UpdateRulesArgs: { /** * @description The healthcheck name whose rules to replace; must already exist @@ -7301,11 +7226,13 @@ export interface components { check_name: string; /** * @description The new conditional rules to store, or `null` to remove all - * conditional rules and rely solely on the base severity. Same - * shape as the `rules` field returned when listing checks. A ladder - * with no condition/severity pairs is treated the same as `null`. + * conditional rules and rely solely on the ceiling. Same shape as + * the `rules` field returned when listing checks. A ladder with no + * condition/result pairs is treated the same as `null`. */ rules?: unknown; + /** @description The source whose check's rules to replace. */ + source: string; }; /** @description Identifies a version and the publication status to set on it. */ UpdateStatusArgs: { @@ -9196,13 +9123,13 @@ export interface operations { }; requestBody?: never; responses: { - /** @description Catalog rows ordered by check_name. */ + /** @description Catalog rows ordered by source then check_name. */ 200: { headers: { [name: string]: unknown; }; content: { - "application/json": components["schemas"]["HealthcheckSeverityData"][]; + "application/json": components["schemas"]["CheckPolicyData"][]; }; }; 401: { @@ -9318,7 +9245,47 @@ export interface operations { [name: string]: unknown; }; content: { - "application/json": components["schemas"]["HealthcheckSeverityData"]; + "application/json": components["schemas"]["CheckPolicyData"]; + }; + }; + 401: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["ProblemDetailsSchema"]; + }; + }; + 403: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["ProblemDetailsSchema"]; + }; + }; + }; + }; + healthcheck_update_documentation: { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + requestBody: { + content: { + "application/json": components["schemas"]["UpdateDocumentationArgs"]; + }; + }; + responses: { + /** @description Updated catalog row. */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["CheckPolicyData"]; }; }; 401: { @@ -9358,7 +9325,7 @@ export interface operations { [name: string]: unknown; }; content: { - "application/json": components["schemas"]["HealthcheckSeverityData"]; + "application/json": components["schemas"]["CheckPolicyData"]; }; }; 400: { @@ -9683,29 +9650,6 @@ export interface operations { }; }; }; - list_events: { - parameters: { - query?: never; - header?: never; - path?: never; - cookie?: never; - }; - requestBody: { - content: { - "application/json": components["schemas"]["ListEventsArgs"]; - }; - }; - responses: { - 200: { - headers: { - [name: string]: unknown; - }; - content: { - "application/json": components["schemas"]["Page_EventData"]; - }; - }; - }; - }; list_for_device: { parameters: { query?: never; @@ -10263,7 +10207,7 @@ export interface operations { }; }; responses: { - /** @description Alert marked operator-resolved; a pending notification is cancelled. */ + /** @description Alert marked operator-resolved. */ 200: { headers: { [name: string]: unknown; @@ -11251,7 +11195,7 @@ export interface operations { }; }; responses: { - /** @description The check's catalog severity and the servers currently reporting it. */ + /** @description The check's catalog policy and the servers currently reporting it. */ 200: { headers: { [name: string]: unknown; diff --git a/private-web/src/components/CheckDocButton.tsx b/private-web/src/components/CheckDocButton.tsx new file mode 100644 index 00000000..73ab7856 --- /dev/null +++ b/private-web/src/components/CheckDocButton.tsx @@ -0,0 +1,104 @@ +import HelpOutlinedIcon from "@mui/icons-material/HelpOutlined"; +import { + Box, + IconButton, + LinearProgress, + Link as MuiLink, + Popover, + Tooltip, + Typography, +} from "@mui/material"; +import { useState } from "react"; +import { Link as RouterLink } from "react-router-dom"; +import { useApi } from "../api"; +import Markdown from "./Markdown"; + +/** `?` affordance shown next to a check name wherever its state is + * presented: opens a popover with the check's rendered operator + * documentation (see the healthcheck settings page, where it's + * authored). Renders unconditionally — an undocumented check pops a + * prompt to write the missing document instead of hiding the icon, + * which would make documented and undocumented checks look different + * for no discoverable reason. */ +export default function CheckDocButton({ + source, + check, +}: { + /** The source that reports this check. Documentation is keyed per + * (source, check); only that exact entry is shown — a same-named + * check from another source may describe something else entirely. */ + source: string; + check: string; +}) { + const [anchor, setAnchor] = useState(null); + return ( + <> + + { + e.stopPropagation(); + setAnchor(e.currentTarget); + }} + > + + + + setAnchor(null)} + anchorOrigin={{ vertical: "bottom", horizontal: "left" }} + // Card rows toggle on click; don't let clicks inside the + // popover bubble back into the row underneath. + onClick={(e) => e.stopPropagation()} + slotProps={{ paper: { sx: { maxWidth: 480, p: 2 } } }} + > + {anchor !== null && } + + + ); +} + +/** Mounted only while the popover is open, so the catalog fetch is + * lazy: nothing is requested until the operator first asks. */ +function DocContent({ source, check }: { source: string; check: string }) { + const list = useApi("healthchecks", "list"); + if (list.status === "loading" || list.status === "idle") { + return ; + } + if (list.status === "error") { + return ( + + {list.error.message} + + ); + } + // Documentation is keyed per (source, check): only the reporting + // source's own entry applies. A same-named check from another source + // may describe something else entirely, so no fallback. + const documentation = + list.data.find((r) => r.source === source && r.check_name === check) + ?.documentation ?? null; + return ( + + {documentation ? ( + {documentation} + ) : ( + + Nobody has documented this check yet. + + )} + + + {documentation ? "Edit documentation" : "Write it"} + + + + ); +} diff --git a/private-web/src/components/CheckResultChip.tsx b/private-web/src/components/CheckResultChip.tsx new file mode 100644 index 00000000..e071a888 --- /dev/null +++ b/private-web/src/components/CheckResultChip.tsx @@ -0,0 +1,28 @@ +import { Chip, Tooltip } from "@mui/material"; +import { CHECK_RESULT_INTENT, type CheckResult } from "../types"; + +type Color = "error" | "warning" | "success" | "default"; + +/// Broken reads as a warning since it says nothing about the system +/// under test, just the check itself; passed/skipped read calm. +const COLOR: Record = { + failed: "error", + warning: "warning", + broken: "warning", + passed: "success", + skipped: "default", +}; + +export default function CheckResultChip({ + result, + variant, +}: { + result: CheckResult; + variant?: "filled" | "outlined"; +}) { + return ( + + + + ); +} diff --git a/private-web/src/components/IncidentCard.tsx b/private-web/src/components/IncidentCard.tsx index 923d96e5..1f703b44 100644 --- a/private-web/src/components/IncidentCard.tsx +++ b/private-web/src/components/IncidentCard.tsx @@ -1,7 +1,6 @@ import { Box, Stack, Tooltip, Typography } from "@mui/material"; import BugReportIcon from "@mui/icons-material/BugReport"; import NotesIcon from "@mui/icons-material/StickyNote2"; -import TimelineIcon from "@mui/icons-material/Timeline"; import { Link as RouterLink } from "react-router-dom"; import TimeAgo from "./TimeAgo"; import { useIsNotificationHeld } from "../hooks/useIsNotificationHeld"; @@ -62,12 +61,7 @@ export default function IncidentCard({ incident }: { incident: IncidentData }) { value={incident.issue_count ?? 0} noun="issue" /> - } - value={incident.event_count ?? 0} - noun="event" - /> - } value={incident.note_count ?? 0} noun="note" diff --git a/private-web/src/components/IssueRow.tsx b/private-web/src/components/IssueRow.tsx index 55bdfa73..595a995d 100644 --- a/private-web/src/components/IssueRow.tsx +++ b/private-web/src/components/IssueRow.tsx @@ -5,7 +5,6 @@ import { IconButton, Link as MuiLink, MenuItem, - Pagination, Stack, TextField, Typography, @@ -17,14 +16,15 @@ import NotificationsOffOutlinedIcon from "@mui/icons-material/NotificationsOffOu import SnoozeIcon from "@mui/icons-material/Snooze"; import { Fragment, useState } from "react"; import { Link as RouterLink } from "react-router-dom"; -import { useApi, useApiAction } from "../api"; +import { useApiAction } from "../api"; import { useIsAdmin } from "../hooks/useIsAdmin"; import { humanDuration } from "../lib/humanDuration"; import MessageView from "./MessageView"; import NotesList, { AddNoteButton } from "./NotesList"; import ResolverAvatar from "./ResolverAvatar"; import ServerNameWithGroup from "./ServerNameWithGroup"; -import SeverityChip from "./SeverityChip"; +import CheckDocButton from "./CheckDocButton"; +import CheckResultChip from "./CheckResultChip"; import StatusSnapshotPanel, { StatusSnapshotButton } from "./StatusSnapshot"; import TimeAgo from "./TimeAgo"; import { @@ -32,6 +32,7 @@ import { RESOLVED_REASON_LABEL, healthcheckNameFromRef, healthcheckPath, + type CheckResult, type IssueData, type IssueIncidentLink, type ResolvedReason, @@ -50,10 +51,10 @@ function headline(issue: IssueData): string { } /** One issue. Expanded by default: shows a provenance line (source, ref, - * incident links), the message body, action buttons and a two-column - * Events / Notes panel. When collapsed, only the header row is visible — + * incident links), the message body, action buttons and the notes panel. + * When collapsed, only the header row is visible — * even the action buttons are hidden. Header layout: - * `[toggle] [server] [headline] [resolver-avatar?] [snapshot] [severity] [time]`. + * `[toggle] [server] [headline] [resolver-avatar?] [snapshot] [result] [time]`. * The headline is struck through when the issue is inactive or resolved. * The server is always shown because incidents can span child servers in * a group — relying on a page H1 to identify the server is insufficient. @@ -220,9 +221,19 @@ function Header({ tooltip="Status snapshot when this issue was last seen" /> - - - + {issue.effective_result && ( + + + + )} + {headerCheckName(issue) && ( + + + + )} )} - - + {checkName ? ( - + {issue.ref} ) : ( @@ -641,151 +648,3 @@ function IncidentRef({ inc }: { inc: IssueIncidentLink }) { ); } - -const COLLAPSED_EVENT_COUNT = 3; -const EXPANDED_PAGE_SIZE = 10; - -function EventLog({ - issueId, - serverId, -}: { - issueId: string; - serverId?: string; -}) { - const [expanded, setExpanded] = useState(false); - const [page, setPage] = useState(0); - const limit = expanded ? EXPANDED_PAGE_SIZE : COLLAPSED_EVENT_COUNT; - const offset = expanded ? page * EXPANDED_PAGE_SIZE : 0; - const result = useApi( - "issues", - "list_events", - { issue_id: issueId, offset, limit }, - [issueId, offset, limit], - ); - const [snapshotOpen, setSnapshotOpen] = useState>( - () => new Set(), - ); - const toggleSnapshot = (eventId: string) => - setSnapshotOpen((prev) => { - const next = new Set(prev); - if (next.has(eventId)) next.delete(eventId); - else next.add(eventId); - return next; - }); - const closeSnapshot = (eventId: string) => - setSnapshotOpen((prev) => { - if (!prev.has(eventId)) return prev; - const next = new Set(prev); - next.delete(eventId); - return next; - }); - - if (result.status === "loading" || result.status === "idle") - return ( - - Loading… - - ); - if (result.status === "error") - return {result.error.message}; - const { items, total } = result.data; - if (total === 0) - return ( - - No events recorded. - - ); - - const pageCount = Math.max(1, Math.ceil(total / EXPANDED_PAGE_SIZE)); - const hiddenCount = Math.max(0, total - COLLAPSED_EVENT_COUNT); - - return ( - - {items.map((e) => { - const at = e.occurred_at ?? e.created_at; - const open = snapshotOpen.has(e.id); - return ( - - - - {e.occurrences > 1 && ( - - ×{e.occurrences} - - )} - - toggleSnapshot(e.id)} - tooltip="Status snapshot at this event" - /> - - - - - - - - {open && serverId && ( - closeSnapshot(e.id)} - /> - )} - - ); - })} - {!expanded && hiddenCount > 0 && ( - - )} - {expanded && ( - - - {pageCount > 1 && ( - setPage(p - 1)} - /> - )} - - )} - - ); -} diff --git a/private-web/src/components/ManualEventForm.tsx b/private-web/src/components/ManualEventForm.tsx index 9dff0098..d10ca636 100644 --- a/private-web/src/components/ManualEventForm.tsx +++ b/private-web/src/components/ManualEventForm.tsx @@ -2,13 +2,19 @@ import { Alert as MuiAlert, Box, Button, + FormControlLabel, MenuItem, Stack, + Switch, TextField, } from "@mui/material"; import { useState } from "react"; import { useApiAction } from "../api"; -import { SEVERITIES, type Severity } from "../types"; + +/// Results an operator can raise a manual condition at: a failure can +/// open an incident; a warning joins one for context only. +const MANUAL_RESULTS = ["failed", "warning"] as const; +type ManualResult = (typeof MANUAL_RESULTS)[number]; export default function ManualEventForm({ serverId, @@ -17,7 +23,8 @@ export default function ManualEventForm({ serverId: string; onSubmitted?: () => void; }) { - const [severity, setSeverity] = useState("error"); + const [result, setResult] = useState("failed"); + const [escalates, setEscalates] = useState(false); const [message, setMessage] = useState(""); const [description, setDescription] = useState(""); @@ -30,7 +37,8 @@ export default function ManualEventForm({ // Each manual submission is its own issue. Operators that need to // add context to an existing issue use the notes panel instead. ref: crypto.randomUUID(), - severity, + result, + escalates, description: description.trim() === "" ? null : description.trim(), message, }); @@ -46,20 +54,34 @@ export default function ManualEventForm({ return ( - setSeverity(e.target.value as Severity)} - sx={{ minWidth: 140, alignSelf: "flex-start" }} - > - {SEVERITIES.map((s) => ( - - {s} - - ))} - + + setResult(e.target.value as ManualResult)} + sx={{ minWidth: 140 }} + > + {MANUAL_RESULTS.map((r) => ( + + {r} + + ))} + + {result === "failed" && ( + setEscalates(e.target.checked)} + size="small" + /> + } + label="Notify immediately" + /> + )} + {active.data.map((alert) => ( - + Canopy: {alert.title ?? alert.ref} diff --git a/private-web/src/components/SeverityChip.tsx b/private-web/src/components/SeverityChip.tsx deleted file mode 100644 index 94e15762..00000000 --- a/private-web/src/components/SeverityChip.tsx +++ /dev/null @@ -1,20 +0,0 @@ -import { Chip, Tooltip } from "@mui/material"; -import { SEVERITY_INTENT, type Severity } from "../types"; - -type Color = "error" | "warning" | "info" | "default"; - -const COLOR: Record = { - critical: "error", - error: "error", - warning: "warning", - info: "info", - debug: "default", -}; - -export default function SeverityChip({ severity }: { severity: Severity }) { - return ( - - - - ); -} diff --git a/private-web/src/components/StatusSnapshot.tsx b/private-web/src/components/StatusSnapshot.tsx index 8cbf3ee0..023524a7 100644 --- a/private-web/src/components/StatusSnapshot.tsx +++ b/private-web/src/components/StatusSnapshot.tsx @@ -12,7 +12,6 @@ import CancelIcon from "@mui/icons-material/Cancel"; import CheckCircleIcon from "@mui/icons-material/CheckCircle"; import CloseIcon from "@mui/icons-material/Close"; import CircleIcon from "@mui/icons-material/Circle"; -import ErrorIcon from "@mui/icons-material/Error"; import InfoIcon from "@mui/icons-material/Info"; import NotificationsOffIcon from "@mui/icons-material/NotificationsOff"; import PreviewIcon from "@mui/icons-material/Preview"; @@ -28,11 +27,10 @@ import TimeAgo from "./TimeAgo"; import TimezoneTooltip from "./TimezoneTooltip"; import VersionIndicator from "./VersionIndicator"; import { + CHECK_RESULT_INTENT, CHECK_RESULT_ORDER, - SEVERITY_INTENT, checkResultOf, type CheckResult, - type Severity, type StatusSnapshotData, } from "../types"; @@ -120,7 +118,7 @@ function PanelBody({ @@ -189,12 +187,12 @@ function Field({ function ChecksBlock({ health, - severities, + results, operators, silencedChecks, }: { health: StatusSnapshotData["health"]; - severities: StatusSnapshotData["check_severities"]; + results: StatusSnapshotData["check_results"]; operators: StatusSnapshotData["operators"]; silencedChecks: StatusSnapshotData["silenced_checks"]; }) { @@ -244,7 +242,9 @@ function ChecksBlock({ > @@ -348,11 +348,11 @@ function parseChecks( /// reported — they don't count toward the server's health. function CheckIcon({ result, - severity, + effective, silenced = false, }: { result: CheckResult; - severity: Severity | null; + effective: CheckResult | null; silenced?: boolean; }) { if (silenced) { @@ -391,34 +391,33 @@ function CheckIcon({ case "failed": break; } - const sev: Severity = severity ?? "warning"; - const tooltip = `${result} at ${sev} — ${SEVERITY_INTENT[sev]}`; - switch (sev) { - case "critical": - return ( - - - - ); - case "error": + // A degraded observation renders by what policy grades it to. + const eff: CheckResult = effective ?? "warning"; + const tooltip = + result === eff + ? `${result} — ${CHECK_RESULT_INTENT[eff]}` + : `${result}, graded ${eff} — ${CHECK_RESULT_INTENT[eff]}`; + switch (eff) { + case "failed": return ( ); case "warning": + case "broken": return ( ); - case "info": + case "passed": return ( ); - case "debug": + case "skipped": return ( diff --git a/private-web/src/lib/healthcheck-rule-eval.ts b/private-web/src/lib/healthcheck-rule-eval.ts index 643fe5a3..df2ce4f7 100644 --- a/private-web/src/lib/healthcheck-rule-eval.ts +++ b/private-web/src/lib/healthcheck-rule-eval.ts @@ -1,5 +1,5 @@ // Client-side mirror of the Rust IfLadder evaluator (see -// crates/database/src/healthcheck_severities.rs). Used by the rule +// crates/database/src/check_policies.rs). Used by the rule // editor to preview whether a candidate rule would match a sampled // status push. // diff --git a/private-web/src/routes/HealthcheckAttention.tsx b/private-web/src/routes/HealthcheckAttention.tsx index 709ef01d..35305c22 100644 --- a/private-web/src/routes/HealthcheckAttention.tsx +++ b/private-web/src/routes/HealthcheckAttention.tsx @@ -1,4 +1,7 @@ import { + Accordion, + AccordionDetails, + AccordionSummary, Alert, Box, Chip, @@ -23,46 +26,34 @@ import { useState } from "react"; import { Link as RouterLink, useParams } from "react-router-dom"; import { useApi } from "../api"; import CheckExtrasList, { checkEntryExtras } from "../components/CheckExtras"; +import Markdown from "../components/Markdown"; +import CheckResultChip from "../components/CheckResultChip"; import ServerNameWithGroup from "../components/ServerNameWithGroup"; -import SeverityChip from "../components/SeverityChip"; import TimeAgo from "../components/TimeAgo"; import { usePageTitle } from "../hooks/usePageTitle"; import { useReloadInterval } from "../hooks/useReloadInterval"; import { type CheckAttentionServerData, type CheckResult } from "../types"; -/// MUI chip colour per check result. Broken reads as a warning since it -/// says nothing about the system under test, just the check itself; -/// passed/skipped (visible behind the "show healthy" toggle) read calm. -const CHECK_CHIP_COLOR: Record< - CheckResult, - "error" | "warning" | "success" | "default" -> = { - failed: "error", - warning: "warning", - broken: "warning", - passed: "success", - skipped: "default", -}; - const HEALTHY_RESULTS: readonly string[] = ["passed", "skipped"]; -/// Dedicated page for a single healthcheck: every live server whose -/// *current* status flags it, most urgent first, with the servers +/// Dedicated page for a single healthcheck — one (source, check), since +/// that pair is the check's identity: every live server whose *current* +/// state from that source flags it, most urgent first, with the servers /// reporting it healthy behind a toggle. Doubles as an operator TODO /// list for normalising those servers back to healthy, and as a way to /// see who's sharing the same issue during a fleet-wide incident. /// Linked from wherever a check name shows up — server detail, issue /// rows, and the healthchecks settings catalog. export default function HealthcheckAttention() { - const { check } = useParams<{ check: string }>(); + const { source, check } = useParams<{ source: string; check: string }>(); usePageTitle(check ?? "Healthcheck"); const tick = useReloadInterval(30_000, "canopy-data-changed"); const [showHealthy, setShowHealthy] = useState(false); const result = useApi( "statuses", "check_attention", - { check: check ?? "" }, - [check, tick], + { source: source ?? "", check: check ?? "" }, + [source, check, tick], ); return ( @@ -75,21 +66,45 @@ export default function HealthcheckAttention() { {check} - {result.status === "ok" && result.data.severity && ( - + + reported by {source} + + {result.status === "ok" && result.data.ceiling && ( + + )} + {result.status === "ok" && result.data.escalates && ( + )} - Servers whose latest status currently flags this check.{" "} + Servers whose latest report from this source currently flags this + check.{" "} - Configure severity / rules + Configure ceiling / rules / documentation + {result.status === "ok" && result.data.documentation && ( + + }> + About this check + + + {result.data.documentation} + + + )} + - diff --git a/private-web/src/routes/HealthcheckDetail.tsx b/private-web/src/routes/HealthcheckDetail.tsx index cad6f367..0cf33ab1 100644 --- a/private-web/src/routes/HealthcheckDetail.tsx +++ b/private-web/src/routes/HealthcheckDetail.tsx @@ -8,6 +8,7 @@ import { DialogActions, DialogContent, DialogTitle, + FormControlLabel, IconButton, InputAdornment, LinearProgress, @@ -15,6 +16,7 @@ import { Paper, Select, Stack, + Switch, Table, TableBody, TableCell, @@ -31,25 +33,29 @@ import DeleteIcon from "@mui/icons-material/Delete"; import EditIcon from "@mui/icons-material/Edit"; import WarningAmberIcon from "@mui/icons-material/WarningAmber"; import { useMemo, useState } from "react"; +import Markdown from "../components/Markdown"; import { Link as RouterLink, useParams } from "react-router-dom"; import { ApiError, useApi, useApiAction } from "../api"; -import SeverityChip from "../components/SeverityChip"; +import CheckResultChip from "../components/CheckResultChip"; import TimeAgo from "../components/TimeAgo"; import { useIsAdmin } from "../hooks/useIsAdmin"; import { usePageTitle } from "../hooks/usePageTitle"; import { evaluate as evalCondition } from "../lib/healthcheck-rule-eval"; import { - SEVERITIES, - SEVERITY_INTENT, + CEILINGS, + CEILING_INTENT, + CHECK_RESULT_INTENT, + CHECK_RESULT_ORDER, healthcheckPath, + type Ceiling, + type CheckPolicyData, + type CheckResult, type HealthcheckSample, - type HealthcheckSeverityData, - type Severity, } from "../types"; // ── Constrained JsonLogic shape ─────────────────────────────────────────── // -// The Rust deserialiser only accepts {if: [c1, s1, c2, s2, …]} ladders +// The Rust deserialiser only accepts {if: [c1, r1, c2, r2, …]} ladders // where each condition is {: [{var: ""}, ]}. The // helpers below parse the raw JSON returned by the API into the typed // shape this page edits, and serialize it back for /update_rules. @@ -71,7 +77,7 @@ interface Branch { varPath: string; op: RuleOp; value: unknown; - severity: Severity; + result: CheckResult; } function parseRules(raw: unknown): { branches: Branch[]; error: string | null } { @@ -90,20 +96,23 @@ function parseRules(raw: unknown): { branches: Branch[]; error: string | null } const branches: Branch[] = []; for (let i = 0; i < args.length; i += 2) { const condRaw = args[i]; - const sevRaw = args[i + 1]; + const resultRaw = args[i + 1]; const cond = parseCondition(condRaw); if (cond.error) return { branches: [], error: `branch ${i / 2}: ${cond.error}` }; - if (typeof sevRaw !== "string" || !(SEVERITIES as readonly string[]).includes(sevRaw)) { + if ( + typeof resultRaw !== "string" || + !(CHECK_RESULT_ORDER as readonly string[]).includes(resultRaw) + ) { return { branches: [], - error: `branch ${i / 2}: invalid severity '${String(sevRaw)}'`, + error: `branch ${i / 2}: invalid result '${String(resultRaw)}'`, }; } branches.push({ varPath: cond.varPath, op: cond.op, value: cond.value, - severity: sevRaw as Severity, + result: resultRaw as CheckResult, }); } return { branches, error: null }; @@ -149,7 +158,7 @@ function serializeRules(branches: Branch[]): unknown | null { const args: unknown[] = []; for (const b of branches) { args.push({ [b.op]: [{ var: b.varPath }, b.value] }); - args.push(b.severity); + args.push(b.result); } return { if: args }; } @@ -185,8 +194,8 @@ export default function HealthcheckDetail() { const isAdmin = useIsAdmin() === true; const list = useApi("healthchecks", "list"); - const row: HealthcheckSeverityData | undefined = - list.status === "ok" ? list.data.find((r) => r.check_name === checkName) : undefined; + const rows: CheckPolicyData[] = + list.status === "ok" ? list.data.filter((r) => r.check_name === checkName) : []; return ( @@ -197,35 +206,54 @@ export default function HealthcheckDetail() { {checkName} - - - See servers currently flagging this check - - {list.status === "loading" || list.status === "idle" ? ( ) : list.status === "error" ? ( {list.error.message} - ) : row == null ? ( + ) : rows.length === 0 ? ( No healthcheck named {checkName} in the catalog yet — it'll appear here once a server reports it. ) : ( - <> - - - - - + rows.map((row) => ( + + {rows.length > 1 && ( + + source: {row.source} + + )} + + + See servers currently flagging this check + {rows.length > 1 && ` (as reported by ${row.source})`} + + + + + See servers currently flagging this check + {rows.length > 1 && ` (as reported by ${row.source})`} + + + + + + + + + )) )} ); } -function RowMetadata({ row }: { row: HealthcheckSeverityData }) { +function RowMetadata({ row }: { row: CheckPolicyData }) { return ( void; }) { const update = useApiAction("healthchecks", "update"); - const [severity, setSeverity] = useState(row.severity); + const [ceiling, setCeiling] = useState(row.ceiling as Ceiling); + const [escalates, setEscalates] = useState(row.escalates); const save = async () => { try { - await update.call({ check_name: row.check_name, severity, notes: row.notes }); + await update.call({ + source: row.source, + check_name: row.check_name, + ceiling, + escalates, + notes: row.notes, + }); onChanged(); } catch { - setSeverity(row.severity); + setCeiling(row.ceiling as Ceiling); + setEscalates(row.escalates); } }; return ( - Base severity + Ceiling - Used when this check fails and no rule below matches. + The maximum effective result for this check when no rule below + matches: anything more urgent grades down to it. {canEdit ? ( ) : ( - + )} + setEscalates(e.target.checked)} + disabled={!canEdit || update.pending} + /> + } + label="Escalates: an effective failure notifies immediately, bypassing the incident grace period" + slotProps={{ typography: { variant: "caption" } }} + /> {canEdit && ( + )} + + {editing ? ( + <> + setDraft(e.target.value)} + disabled={update.pending} + slotProps={{ + htmlInput: { + style: { fontFamily: "monospace" }, + "aria-label": "Documentation markdown", + }, + }} + /> + + + + + + ) : row.documentation ? ( + {row.documentation} + ) : ( + + Nobody has documented this check yet. What does it observe, what + does each result mean, and how does one fix a failure? Shown + wherever the check appears, and to agents over MCP. + + )} + {update.error && ( + + {formatError(update.error)} + + )} + + ); +} + // ── Rules editor ────────────────────────────────────────────────────────── function RulesCard({ @@ -374,7 +539,7 @@ function RulesCard({ canEdit, onChanged, }: { - row: HealthcheckSeverityData; + row: CheckPolicyData; canEdit: boolean; onChanged: () => void; }) { @@ -387,7 +552,10 @@ function RulesCard({ // the most recent status push (across all servers) that reported // this check. Powers the Add/Edit dialog's autocomplete + validation. // Fetched lazily; the dialog handles a null sample gracefully. - const sampleResp = useApi("healthchecks", "sample", { check_name: row.check_name }); + const sampleResp = useApi("healthchecks", "sample", { + source: row.source, + check_name: row.check_name, + }); const sample: HealthcheckSample | null = sampleResp.status === "ok" ? (sampleResp.data.sample ?? null) : null; @@ -404,7 +572,11 @@ function RulesCard({ const save = async () => { try { - await update.call({ check_name: row.check_name, rules: serializeRules(branches) }); + await update.call({ + source: row.source, + check_name: row.check_name, + rules: serializeRules(branches), + }); onChanged(); } catch { // keep local state for retry @@ -413,7 +585,7 @@ function RulesCard({ const deleteAll = async () => { setBranches([]); try { - await update.call({ check_name: row.check_name, rules: null }); + await update.call({ source: row.source, check_name: row.check_name, rules: null }); onChanged(); } catch { setBranches(parsed.branches); @@ -438,8 +610,9 @@ function RulesCard({ )} - Evaluated top-to-bottom on every failing push for this check. First matching - branch's severity wins; if none match, the base severity above is used. + Evaluated top-to-bottom on every push for this check. The first matching + branch's result wins — rules can grade in either direction; if none + match, the observed result is capped at the ceiling above. {sample ? ( @@ -456,11 +629,11 @@ function RulesCard({ {parsed.error && ( Stored rules are malformed ({parsed.error}) — ingestion falls back to the - base severity until you save valid rules from this page. + ceiling until you save valid rules from this page. )} {branches.length === 0 ? ( - No rules. The base severity is used for every push. + No rules. The ceiling applies to every push. ) : ( @@ -468,7 +641,7 @@ function RulesCard({ # Condition - Severity + Result {canEdit && } @@ -480,7 +653,7 @@ function RulesCard({ {b.varPath} {OP_LABEL[b.op]} {valueToInputText(b.value)} - + {canEdit && ( @@ -610,7 +783,7 @@ function BranchDialog({ const [valueText, setValueText] = useState( initial ? valueToInputText(initial.value) : "", ); - const [severity, setSeverity] = useState(initial?.severity ?? "error"); + const [result, setResult] = useState(initial?.result ?? "failed"); const varValid = VAR_PATTERN.test(varPath); // Did the sample carry a value for this exact var path? Powers the @@ -636,7 +809,7 @@ function BranchDialog({ varPath, op, value: parseValueInput(valueText), - severity, + result, }); }; @@ -730,18 +903,18 @@ function BranchDialog({ setSeverity(e.target.value as Severity)} + value={result} + onChange={(e) => setResult(e.target.value as CheckResult)} sx={{ minWidth: 320 }} > - {SEVERITIES.map((s) => ( - + {CHECK_RESULT_ORDER.map((r) => ( + - + - {SEVERITY_INTENT[s]} + {CHECK_RESULT_INTENT[r]} @@ -753,7 +926,7 @@ function BranchDialog({ varPath={varPath} op={op} valueText={valueText} - severity={severity} + result={result} /> )} @@ -773,13 +946,13 @@ function PreviewBox({ varPath, op, valueText, - severity, + result: branchResult, }: { sample: HealthcheckSample; varPath: string; op: RuleOp; valueText: string; - severity: Severity; + result: CheckResult; }) { const result = useMemo( () => @@ -802,11 +975,11 @@ function PreviewBox({ {result.matched ? ( - would file at - + would grade to + ) : ( - would not match → base severity used + would not match → ceiling applies )} rows.filter((r) => r.pending_review).length, [rows], @@ -50,9 +50,10 @@ export default function Healthchecks() { Healthchecks - Catalog of healthcheck names reported by servers and the severity - each check's failures are filed at. New checks land here at the - default warning severity, marked pending review. + Catalog of healthchecks, one entry per reporting source and check + name, with the ceiling each check's results are graded at. New + checks land here at the default warning ceiling, + marked pending review. @@ -94,8 +95,9 @@ export default function Healthchecks() {
+ Source Check name - Severity + Ceiling First seen Reviewed @@ -103,7 +105,7 @@ export default function Healthchecks() { {visible.map((row) => ( list.reload()} @@ -123,25 +125,29 @@ function HealthcheckRow({ canEdit, onChanged, }: { - row: HealthcheckSeverityData; + row: CheckPolicyData; canEdit: boolean; onChanged: () => void; }) { const update = useApiAction("healthchecks", "update"); - const [localSeverity, setLocalSeverity] = useState(row.severity); + const [localCeiling, setLocalCeiling] = useState(row.ceiling as Ceiling); + const [localEscalates, setLocalEscalates] = useState(row.escalates); const save = async () => { try { await update.call({ + source: row.source, check_name: row.check_name, - severity: localSeverity, + ceiling: localCeiling, + escalates: localEscalates, notes: row.notes, }); onChanged(); } catch { - // Revert the dropdown selection on failure so the row's + // Revert the editor selection on failure so the row's // rendered state matches the server's. - setLocalSeverity(row.severity); + setLocalCeiling(row.ceiling as Ceiling); + setLocalEscalates(row.escalates); } }; @@ -149,6 +155,7 @@ function HealthcheckRow({ return ( + {row.source} {row.check_name} @@ -161,28 +168,46 @@ function HealthcheckRow({ - · base + · ceiling - + + {row.escalates && } ) : ( {canEdit ? ( - + <> + + setLocalEscalates(e.target.checked)} + disabled={update.pending} + /> + } + label="escalates" + slotProps={{ typography: { variant: "caption" } }} + /> + ) : ( - + <> + + {row.escalates && } + )} {canEdit && (