TAM-6867: Seedling x Canopy status channel — STA spec

.workhorse/specs/public-server/status.md

@@ -0,0 +1,45 @@
+---
+id: STA
+---
+
+# Status reporting
+
+A server device sends Canopy periodic status heartbeats: a self-reported health flag, named health checks, and free-form host facts (versions, uptime, and the like).
+Canopy records each, derives incidents from failing checks, and infers reachability from how recently a device reported.
+
+More than one agent may report for one host: on a Tamanu server the bestool alert daemon and Seedling run against the same device identity, each with its own view.
+This spec covers how Canopy keeps those reports distinct and replies to each.
+
+## Scope
+
+This spec covers the device-facing status contract: how a heartbeat is attributed to a reporting agent, how Canopy keeps concurrent agents under one server distinct, and what it returns in response.
+
+It does not cover how a device enrols or authenticates (see [DTR](../private-server/device-trust.md)), nor what a device backs up (see [BAK](backup.md)).
+
+## Identity and clients
+
+A heartbeat authenticates as a server device over either transport Canopy accepts, exactly as every device request does; identity is never taken from the request body.
+
+Under one authenticated server, a heartbeat is attributed to a named **client**, the agent that produced it.
+The set of client names is open; the two Canopy expects are `bestool` (the alert daemon) and `seedling`.
+
+A heartbeat that names no client is attributed to `bestool`, so agents already deployed, which name none, keep reporting unchanged until they are rebuilt.
+
+The client is a label, not a second identity: authorisation stays the server binding, so a device reports only for the server it is bound to, and an admin for any.
+
+## Independent streams
+
+Canopy records status per `(server, client)`, so two agents reporting for one server never overwrite each other: each keeps its own latest heartbeat, health checks, and history.
+
+A server is not treated as down while any of its clients is still reporting, so one agent going quiet does not by itself make the server look down.
+A quiet agent is still surfaced: when the server is reporting but its `bestool` client — the stream Canopy's health view reads — has gone quiet past the server's down threshold, Canopy raises an issue for that client, distinct from server reachability, and resolves it when the client reports again.
+A server that has never had a `bestool` client raises nothing.
+
+## Response
+
+Canopy knows the reporting client from the request, so it returns only what that client needs, and does not give one client another's concerns.
+A client acts on the parts of the response it understands and ignores the rest; it is sent nothing meant only for another client, and relies on receiving nothing beyond its own.
+
+The backup types due now go to the client that runs backups (`bestool`), not to one such as Seedling that does not.
+A client that reports health checks is told the severities Canopy applies to them.
+The response carries only such return-path instructions; the recorded status itself is not echoed back.

crates/database/src/bin/seed.rs

@@ -864,6 +864,7 @@ async fn seed_statuses(
 				extra,
 				healthy,
 				health,
+				client: database::statuses::DEFAULT_CLIENT.to_owned(),
 			})
 			.execute(conn)
 			.await?;

crates/database/src/schema.rs

@@ -538,6 +538,7 @@ diesel::table! {
 		device_id -> Nullable<Uuid>,
 		healthy -> Bool,
 		health -> Jsonb,
+		client -> Text,
 	}
 }
 

crates/database/src/server_groups.rs

@@ -235,10 +235,10 @@ impl ServerGroup {
 				.map_err(AppError::from)?;
 
 			if !member_ids.is_empty() {
-				// A member is "gone" iff it's absent from `latest_for_servers`
-				// (no status in the last 7 days). Allow the cascade only when
-				// every live member is gone; any recent reporter blocks it.
-				let recent = Status::latest_for_servers(conn, &member_ids).await?;
+				// A member is "gone" iff no client of it has reported in the
+				// last 7 days. Allow the cascade only when every live member
+				// is gone; any recent reporter blocks it.
+				let recent = Status::last_report_for_servers(conn, &member_ids).await?;
 				if !recent.is_empty() {
 					return Err(AppError::Conflict(format!(
 						"group {group_id} has {} server(s) that reported within the last \

crates/database/src/statuses.rs

@@ -31,6 +31,20 @@ pub const CANOPY_SOURCE: &str = "canopy";
 /// same issue row.
 pub const REACHABILITY_REF: &str = "reachability";
 
+/// The client stream canopy's own views read: `bestool` is the agent whose
+/// heartbeats carry the authoritative health checks and version. Other
+/// clients' streams are stored and answered per client, but do not (yet)
+/// feed the status board, version tracking, or health issues.
+pub const DEFAULT_CLIENT: &str = "bestool";
+
+/// Ref value for the one authoritative-client-quiet issue per server. Filed
+/// when the server is still reporting (some client is fresh) but the
+/// [`DEFAULT_CLIENT`] stream has gone quiet past the server's down threshold,
+/// so a dead health reporter isn't masked by another agent keeping the server
+/// "up". Distinct from [`REACHABILITY_REF`], which covers the fully quiet
+/// server.
+pub const CLIENT_STALE_REF: &str = "client-stale/bestool";
+
 fn server_label(s: &Server) -> String {
 	s.name
 		.clone()
@@ -91,6 +105,9 @@ pub struct Status {
 	/// `check` name and a result; any extra per-check fields are passed
 	/// through verbatim.
 	pub health: serde_json::Value,
+	/// The reporting agent this status is attributed to; `bestool` when the
+	/// report named none.
+	pub client: String,
 }
 
 #[derive(Debug, Insertable)]
@@ -104,6 +121,7 @@ pub struct NewStatus {
 	pub extra: serde_json::Value,
 	pub healthy: bool,
 	pub health: serde_json::Value,
+	pub client: String,
 }
 
 impl Default for NewStatus {
@@ -115,6 +133,7 @@ impl Default for NewStatus {
 			extra: serde_json::Value::Object(Default::default()),
 			healthy: true,
 			health: serde_json::Value::Array(Default::default()),
+			client: DEFAULT_CLIENT.to_owned(),
 		}
 	}
 }
@@ -162,6 +181,9 @@ impl Status {
 					// to avoid false-positive unhealthy events from this path.
 					healthy: true,
 					health: serde_json::Value::Array(Default::default()),
+					// Canopy-generated reachability status; attributed to the
+					// default client stream.
+					client: DEFAULT_CLIENT.to_owned(),
 				})
 			}
 			Err(err) => {
@@ -237,7 +259,9 @@ impl Status {
 		}
 
 		let server_ids: Vec<Uuid> = monitored.iter().map(|s| s.id).collect();
-		let statuses = Self::latest_for_servers(db, &server_ids).await?;
+		// Any-client freshness: a server is reachable while any of its
+		// clients is still reporting.
+		let statuses = Self::last_report_for_servers(db, &server_ids).await?;
 		let status_map: std::collections::HashMap<Uuid, Status> =
 			statuses.into_iter().map(|s| (s.server_id, s)).collect();
 		let existing_issues =
@@ -249,6 +273,20 @@ impl Status {
 			.filter_map(|i| i.server_id.map(|sid| (sid, i)))
 			.collect();
 
+		// The authoritative client's own freshness, for the client-stale pass
+		// below (`latest_for_servers` is pinned to [`DEFAULT_CLIENT`]).
+		let bestool_statuses = Self::latest_for_servers(db, &server_ids).await?;
+		let bestool_map: std::collections::HashMap<Uuid, Status> = bestool_statuses
+			.into_iter()
+			.map(|s| (s.server_id, s))
+			.collect();
+		let stale_issues =
+			Issue::list_by_source_ref(db, CANOPY_SOURCE, CLIENT_STALE_REF, &server_ids).await?;
+		let stale_issue_map: std::collections::HashMap<Uuid, &Issue> = stale_issues
+			.iter()
+			.filter_map(|i| i.server_id.map(|sid| (sid, i)))
+			.collect();
+
 		let now = Timestamp::now();
 		let mut filed = 0usize;
 		for server in &monitored {
@@ -265,18 +303,18 @@ impl Status {
 			let existing = issue_map.get(&server.id).copied();
 
 			let event = match (down, existing) {
-				(false, None) => continue,
-				(false, Some(issue)) if !issue.active => continue,
-				(false, Some(_)) => NewEvent {
+				(false, None) => None,
+				(false, Some(issue)) if !issue.active => None,
+				(false, Some(_)) => Some(NewEvent {
 					source: CANOPY_SOURCE.into(),
 					r#ref: REACHABILITY_REF.into(),
 					severity: Some(Severity::Info),
 					description: None,
 					message: format!("Server {} is reachable again", server_label(server)),
 					active: Some(false),
 					occurred_at: Some(now),
-				},
-				(true, _) => NewEvent {
+				}),
+				(true, _) => Some(NewEvent {
 					source: CANOPY_SOURCE.into(),
 					r#ref: REACHABILITY_REF.into(),
 					severity: Some(Severity::Error),
@@ -296,10 +334,60 @@ impl Status {
 					},
 					active: Some(true),
 					occurred_at: Some(now),
-				},
+				}),
 			};
-			event.save(db, server.id, None).await?;
-			filed += 1;
+			if let Some(event) = event {
+				event.save(db, server.id, None).await?;
+				filed += 1;
+			}
+
+			// Client-stale pass: the server-down issue above covers a fully
+			// quiet server; this covers the authoritative stream going quiet
+			// while another client keeps the server up.
+			if !down {
+				let bestool_elapsed: Option<SignedDuration> = bestool_map
+					.get(&server.id)
+					.map(|s| now.duration_since(s.created_at).abs());
+				let existing_stale = stale_issue_map.get(&server.id).copied();
+				let stale_event = match (bestool_elapsed, existing_stale) {
+					// Nothing in the window: either the client never existed
+					// (nothing went quiet) or it's been gone so long its rows
+					// aged out — in which case the already-open issue stays
+					// open until the client actually reports again.
+					(None, _) => None,
+					(Some(e), _) if e >= threshold => Some(NewEvent {
+						source: CANOPY_SOURCE.into(),
+						r#ref: CLIENT_STALE_REF.into(),
+						severity: Some(Severity::Error),
+						description: None,
+						message: format!(
+							"Client {DEFAULT_CLIENT} on server {} has not reported for {} (threshold {}), though the server is still reporting",
+							server_label(server),
+							format_secs(e.as_secs()),
+							format_secs(threshold.as_secs()),
+						),
+						active: Some(true),
+						occurred_at: Some(now),
+					}),
+					(Some(_), Some(issue)) if issue.active => Some(NewEvent {
+						source: CANOPY_SOURCE.into(),
+						r#ref: CLIENT_STALE_REF.into(),
+						severity: Some(Severity::Info),
+						description: None,
+						message: format!(
+							"Client {DEFAULT_CLIENT} on server {} is reporting again",
+							server_label(server),
+						),
+						active: Some(false),
+						occurred_at: Some(now),
+					}),
+					(Some(_), _) => None,
+				};
+				if let Some(event) = stale_event {
+					event.save(db, server.id, None).await?;
+					filed += 1;
+				}
+			}
 		}
 		Ok(filed)
 	}
@@ -360,6 +448,7 @@ impl Status {
 			.filter(
 				server_id
 					.eq(server)
+					.and(client.eq(DEFAULT_CLIENT))
 					.and(created_at.ge(diesel::dsl::sql("NOW() - INTERVAL '7 days'")))
 					.and(id.ne(Uuid::nil())),
 			)
@@ -385,6 +474,7 @@ impl Status {
 			.filter(
 				server_id
 					.eq(server)
+					.and(client.eq(DEFAULT_CLIENT))
 					.and(created_at.le(jiff_diesel::Timestamp::from(at)))
 					.and(id.ne(Uuid::nil())),
 			)
@@ -410,6 +500,7 @@ impl Status {
 			.filter(
 				server_id
 					.eq(server)
+					.and(client.eq(DEFAULT_CLIENT))
 					.and(version.is_not_null())
 					.and(id.ne(Uuid::nil())),
 			)
@@ -433,6 +524,36 @@ impl Status {
 		// status row in the 7-day window for each server. The LIMIT 1 under
 		// the lateral join reads one row per weekly partition through the
 		// (server_id, created_at DESC) composite index.
+		let query = diesel::sql_query(
+			"SELECT st.* FROM unnest($1) AS s(id) \
+			 CROSS JOIN LATERAL ( \
+				SELECT * FROM statuses \
+				WHERE server_id = s.id \
+				AND client = $2 \
+				AND created_at >= NOW() - INTERVAL '7 days' \
+				AND id != '00000000-0000-0000-0000-000000000000' \
+				ORDER BY created_at DESC LIMIT 1 \
+			 ) st",
+		)
+		.bind::<diesel::sql_types::Array<diesel::sql_types::Uuid>, _>(server_ids)
+		.bind::<diesel::sql_types::Text, _>(DEFAULT_CLIENT);
+
+		query.load::<Status>(db).await.map_err(AppError::from)
+	}
+
+	/// Most recent report per server from **any** client, for freshness
+	/// decisions only: a server is reporting while any of its clients is,
+	/// so one agent going quiet does not by itself make the server look
+	/// down. The returned row belongs to whichever client reported last —
+	/// read its timestamp, not its content.
+	pub async fn last_report_for_servers(
+		db: &mut AsyncPgConnection,
+		server_ids: &[Uuid],
+	) -> Result<Vec<Status>> {
+		if server_ids.is_empty() {
+			return Ok(Vec::new());
+		}
+
 		let query = diesel::sql_query(
 			"SELECT st.* FROM unnest($1) AS s(id) \
 			 CROSS JOIN LATERAL ( \