feat(runtime): separate liveness, readiness, and degraded health

CHANGELOG.md

@@ -9,6 +9,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- Distinct liveness, readiness, recoverable degradation, and sticky fatal health
+  (`src/health/`) with a fake-clock state-machine test for every transition and
+  recovery path. Optional `control_bind` listener serves `/livez`, `/readyz`,
+  `/health`, and `/metrics` (the repository's first control surface; none existed
+  before). Contract, transition table, and example snapshots: [`docs/health.md`](docs/health.md).
 - Deterministic fake-clock / fake-core fault-injection harness (`src/runtime`)
   that injects failures at initialization, checkpoint validation, ingress,
   tick execution, metric publication, and shutdown. Restart preserves

README.md

@@ -18,6 +18,7 @@ Headless spiking neural-network runtime written in Rust.
 - Live-mode restore of Distill sidecar `snn_model.json` before the tick loop (explicit `simulation` mode for blank networks)
 - Optional **ZeroMQ PUB/SUB** networking via `corpus-ipc`
 - Headless **`brainstem-daemon`** binary for background execution
+- Distinct **liveness / readiness / degraded / fatal** health snapshots (library handle plus optional `control_bind` listener)
 
 ---
 
@@ -99,6 +100,10 @@ runtime_mode   = "live"    # default when omitted; "simulation" is the only blan
 tick_rate_hz   = 1000      # loop frequency
 log_level      = "info"    # error|warn|info|debug|trace
 
+# Optional process control surface (unset = no extra socket; historical default).
+# Serves /livez, /readyz, /health, /metrics. See docs/health.md.
+# control_bind   = "127.0.0.1:9464"
+
 # ZMQ (still required in TOML; no-ops under the default stub backend)
 spine_sub_port = 5555      # stimuli in
 spine_pub_port = 5556      # spikes out
@@ -132,19 +137,21 @@ max_payload_len     = 4096
 
 Default Cargo features are empty (`default = []` in `Cargo.toml`). That path uses the in-memory **stub** backend (`StubStimulusSource` + `NoopSpikeSink`) and does **not** need ZeroMQ. The optional `corpus-ipc` feature (same as `--all-features` today) pulls `corpus-ipc` **from crates.io** (`0.1`, `features = ["zmq"]`) plus this crate's optional `zmq` dependency. Published `corpus-ipc` compiles libzmq via `zmq-sys` / `zeromq-src` (a C++ compiler is required; a system `libzmq` package is not). It does not vendor ZeroMQ as a git submodule.
 
-`DaemonConfig` deserialization is **not** feature-gated: `spine_sub_port`, `spine_pub_port`, and `model_path` are still required in TOML even on the stub path (`services` and `runtime_mode` are optional; `runtime_mode` defaults to `live`; `ingress` defaults to the bounded-queue table below). Effect at runtime depends on which backend is **wired** and on `runtime_mode`.
+`DaemonConfig` deserialization is **not** feature-gated: `spine_sub_port`, `spine_pub_port`, and `model_path` are still required in TOML even on the stub path (`services`, `runtime_mode`, `ingress`, and `control_bind` are optional; `runtime_mode` defaults to `live`; `ingress` defaults to the bounded-queue table below; `control_bind` unset = no extra socket). Effect at runtime depends on which backend is **wired** and on `runtime_mode`.
 
 #### Feature truth table
 
 | Cargo flags | Wired backend | `libzmq` | Binary (`brainstem-daemon`) | Library `BrainstemDaemon::new()` / `try_new()` |
 |---|---|---|---|---|
-| default / `--no-default-features` | stub | not required | no sockets; logs `🔌 Using stub backend` | stub |
+| default / `--no-default-features` | stub | not required | no backend sockets by default; `control_bind` opens the control listener; logs `🔌 Using stub backend` | stub |
 | `--features corpus-ipc` | ZMQ / `corpus-ipc` | required | SUB via env, PUB on `spine_pub_port`; logs `📡 Using ZMQ corpus-ipc backend` | **still stub** |
 | `--all-features` | same as `corpus-ipc` | required | same as `--features corpus-ipc` | **still stub** |
 
 Enabling the feature does **not** change `BrainstemDaemon::new()` or `try_new()`. Those always inject `BackendPair::stub()`. Only `src/bin/brainstem_daemon.rs` constructs `ZmqStimulusSource` + `ZmqSpikeSink` when `corpus-ipc` is on.
 
-Library users who want live ZMQ must build that pair themselves under `#[cfg(feature = "corpus-ipc")]` and pass it to `with_backend` / `try_with_backend`. Call `StimulusSource::initialize(...)` on the source first (as the binary does). Neither constructor nor `run` calls `initialize`; skipping it makes ingress fail with `ZmqIpcBackend not initialized`.
+Library users who want live ZMQ must build that pair themselves under `#[cfg(feature = "corpus-ipc")]` and pass it to `with_backend` / `try_with_backend`. `BrainstemDaemon::run` / `run_with_restored_network` is the sole caller of `StimulusSource::initialize` (the binary no longer initializes first, so the pinned ZMQ backend is not reconnected). A failing `initialize` marks health **fatal** and never becomes ready.
+
+Health snapshots, probe paths, and the transition table live in [`docs/health.md`](docs/health.md).
 
 #### Config keys and env vars
 
@@ -155,7 +162,8 @@ Library users who want live ZMQ must build that pair themselves under `#[cfg(fea
 | `tick_rate_hz` | used | used |
 | `log_level` | binary tracing init only; unused by `::new()` / `run` | binary tracing init only; unused by `::new()` / `run` |
 | `services` | used (`ServiceRegistry`) | used |
-| `ingress` | used (bounded class queues in the tick loop) | used (same queues wrap backend packets before the network step) |
+| `control_bind` | optional HTTP control surface; unset = no listener | same |
+| `ingress` | used (bounded class queues in the tick loop; health reports aggregate fill) | used (same queues wrap backend packets before the network step) |
 | `spine_sub_port` | parsed, **no-op** | sets `CORPUS_IPC_ZMQ_READOUT_IPC` to `tcp://127.0.0.1:<port>` (also sets legacy `SPIKENAUT_ZMQ_READOUT_IPC` for compatibility) |
 | `spine_pub_port` | parsed, **no-op** | binds ZMQ PUB `tcp://*:<port>` |
 | `model_path` | used in **live** mode (sidecar JSON); ignored in **simulation** (`StubStimulusSource::initialize` still ignores it) | same live/simulation gate, then passed literally to `initialize` (no `~` expansion); published `ZmqIpcBackend` currently ignores `_model_path` |

docs/health.md

@@ -0,0 +1,126 @@
+# Runtime health
+
+Supervisors should treat **liveness** and **readiness** as independent. A live
+`brainstem-daemon` process has a health reporter; it is ready only after
+`StimulusSource::initialize` succeeds **and** a checkpoint is validated
+(`restore_network` in live mode, or the explicit simulation blank network).
+Live mode uses the Distill sidecar SHA-256 as `checkpoint.digest`. Simulation
+mode reports `digest: null`.
+
+This repository had no HTTP/metrics server before this surface. When
+`control_bind` is set, `BrainstemDaemon::run` starts **one** listener:
+
+| Path | Meaning |
+|---|---|
+| `GET /livez` | `200` if live, `503` otherwise |
+| `GET /readyz` | `200` if ready, `503` otherwise |
+| `GET /health` | `200` JSON [`HealthSnapshot`](../src/health/snapshot.rs) (inspect `phase`); bounded `503 busy` if the control plane cannot snapshot |
+| `GET /metrics` | Prometheus text; labels are phase/reason codes only; same `503 busy` exception |
+
+Leave `control_bind` unset to preserve the historical no-extra-socket default.
+Do not add a second control server beside this one.
+
+Library embedders can also clone [`HealthHandle`](../src/health/handle.rs) from
+`BrainstemDaemon::health()` and call `snapshot()` / `try_snapshot()` without
+waiting on the tick loop's backend or `SpikingNetwork::step`. Control probes
+use `try_snapshot()` so they never wait on an in-flight `apply`.
+
+## Transition table
+
+| From | Event | To | live | ready | Notes |
+|---|---|---|---|---|---|
+| (unstarted) | `ProcessStarted` | `starting` | true | false | Construction. Live does not imply ready. |
+| (unstarted) | init/checkpoint success or failure | (unstarted) | false | false | Ignored until `ProcessStarted`; the fatal transition below applies only after `ProcessStarted`. |
+| `starting` | `InitializationCompleted` | `loading_checkpoint` | true | false | `initialize()` succeeded. Checkpoint restore is a separate gate. |
+| `starting` | `InitializationFailed` | `fatal` | true | false | Sticky. Detail is JSON-only, never a metric label. |
+| `loading_checkpoint` | `CheckpointValidated` | `running` | true | true | Ready only after this gate. Live identity is sidecar `model_id` plus SHA-256. |
+| `loading_checkpoint` | `CheckpointRejected` | `fatal` | true | false | Sticky. |
+| `running` | clock ≥ `stale_after` without ingress | `degraded` | true | true | Reason `stale_input`. Ready stays true. |
+| `degraded` (stale) | `IngressObserved` | `running` (if no other reasons) | true | true | Ticks without ingress do **not** clear stale. Empty stub packets do not count. |
+| `running` | queue fill ≥ `overload_high` | `degraded` | true | true | Reason `overload`. Fill is the aggregate of the four bounded ingress classes. |
+| `degraded` (overload) | fill ≤ `overload_low` | `running` (if no other reasons) | true | true | Hysteresis: mid-band does not recover. |
+| `running` / `degraded` | `BeginDrain` | `draining` | true | false | SIGTERM/SIGINT. Does not return to ready. |
+| any started non-fatal | `Fatal` / init or checkpoint failure | `fatal` | true | false | Subsequent validate/tick/drain cannot restore ready. Unstarted init/checkpoint failures stay ignored. |
+
+After `BeginDrain`, `BrainstemDaemon::run` stops the control listener. External
+`/readyz` probes may get connection refused rather than `503`. In-process
+`HealthHandle::snapshot()` still reports `phase: draining`. There is no probe
+grace period. If all 32 in-flight control slots are busy, up to 4 extra
+connections get a short `503` (`busy`); further accepts are closed immediately.
+The same `503` is used when a snapshot read would block on an in-flight `apply`.
+
+Recoverable reasons (`stale_input`, `overload`) are independent: clearing one
+leaves the other. `capacity == 0` means “no queue instrumented” and never
+counts as overload. After LIM-1216 the tick loop reports the sum of the four
+class depths and capacities.
+
+Fatal and draining are sticky for **this process**. A new process starts in
+`starting` again.
+
+## Checkpoint identity
+
+Live mode copies [`ModelProvenance::model_id`](../src/checkpoint/mod.rs) and
+the sidecar SHA-256 into `checkpoint`. Simulation mode uses
+`id = "simulation/blank"` and `digest: null`.
+
+## Example snapshots
+
+Healthy (ready to consume events) after a live sidecar restore:
+
+```json
+{
+  "live": true,
+  "ready": true,
+  "phase": "running",
+  "reasons": [],
+  "last_successful_tick_ms": 0,
+  "tick_age_ms": 0,
+  "checkpoint": { "id": "spikenaut-snn:soma16", "digest": "abc123" },
+  "input_freshness": { "age_ms": 0, "stale": false },
+  "queue_pressure": { "depth": 0, "capacity": 216, "ratio": 0.0, "overloaded": false },
+  "fatal": null,
+  "observed_at_ms": 0
+}
+```
+
+Degraded (still ready; supervisors should not bounce the process):
+
+```json
+{
+  "live": true,
+  "ready": true,
+  "phase": "degraded",
+  "reasons": ["stale_input", "overload"],
+  "last_successful_tick_ms": 0,
+  "tick_age_ms": 0,
+  "checkpoint": { "id": "spikenaut-snn:soma16", "digest": "abc123" },
+  "input_freshness": { "age_ms": 100, "stale": true },
+  "queue_pressure": { "depth": 95, "capacity": 100, "ratio": 0.95, "overloaded": true },
+  "fatal": null,
+  "observed_at_ms": 100
+}
+```
+
+Fatal (never returns to ready in this process):
+
+```json
+{
+  "live": true,
+  "ready": false,
+  "phase": "fatal",
+  "reasons": ["fatal"],
+  "last_successful_tick_ms": null,
+  "tick_age_ms": null,
+  "checkpoint": null,
+  "input_freshness": { "age_ms": null, "stale": false },
+  "queue_pressure": { "depth": 0, "capacity": 0, "ratio": null, "overloaded": false },
+  "fatal": { "code": "checkpoint_invalid", "detail": "blank weights" },
+  "observed_at_ms": 0
+}
+```
+
+`fatal.detail` belongs in JSON/logs. Prometheus `/metrics` exposes
+`brainstem_fatal 1` and `brainstem_phase{phase="fatal"} 1` without the detail
+string. Optional gauges (`last_successful_tick_ms`, `tick_age_ms`,
+`input_age_ms`) emit `NaN` when the observation is missing so supervisors do
+not treat startup as a tick at time zero.

src/bin/brainstem_daemon.rs

@@ -12,8 +12,6 @@ use brainstem_daemon::daemon::{BrainstemDaemon, DaemonConfig};
 use brainstem_daemon::daemon::{CORPUS_IPC_READOUT_ENV, LEGACY_SPIKENAUT_READOUT_ENV};
 
 use anyhow::Context;
-#[cfg(feature = "corpus-ipc")]
-use brainstem_daemon::StimulusSource;
 use clap::Parser;
 use tracing::info;
 use tracing_subscriber::EnvFilter;
@@ -83,19 +81,15 @@ async fn run(cfg: DaemonConfig, config_path: PathBuf) -> anyhow::Result<()> {
         // Build a real ZMQ pair (binary is responsible for the SUB endpoint via env).
         // We still need to create the PUB side here because the default `new()` path
         // is intentionally conservative.
-        let mut source = brainstem_daemon::backend::ZmqStimulusSource::with_channels(cfg.channels);
+        let source = brainstem_daemon::backend::ZmqStimulusSource::with_channels(cfg.channels);
 
-        // Pass the model path through for the ZMQ backend handshake. SNN
-        // restoration is owned by `restore_network` /
+        // `BrainstemDaemon::run` owns `StimulusSource::initialize` so the pinned
+        // ZMQ backend is not reconnected here (repeat initialize replaces the SUB
+        // socket). SNN restoration is owned by `restore_network` /
         // `BrainstemDaemon::run_with_restored_network`, not by
         // `ZmqIpcBackend::initialize` (published corpus-ipc 0.1 still names
         // the argument `_model_path` and ignores it).
 
-        let model_path = cfg.model_path.to_string_lossy();
-        source
-            .initialize(Some(model_path.as_ref()))
-            .map_err(|e| anyhow::anyhow!("failed to initialize ZMQ stimulus source: {e}"))?;
-
         let zmq_context = zmq::Context::new();
         let pub_socket = zmq_context
             .socket(zmq::PUB)

src/checkpoint/tests.rs

@@ -36,6 +36,7 @@ fn live_config(model_path: PathBuf, lif: usize, channels: usize) -> DaemonConfig
         services: vec![ServiceConfig::named("telemetry")],
         runtime_mode: RuntimeMode::Live,
         ingress: IngressConfig::default(),
+        control_bind: None,
     }
 }
 
@@ -395,6 +396,7 @@ fn simulation_uses_blank_with_dimensions_and_does_not_claim_spikenaut() {
         services: Vec::new(),
         runtime_mode: RuntimeMode::Simulation,
         ingress: IngressConfig::default(),
+        control_bind: None,
     };
     let (network, provenance) = restore_network(&cfg).expect("simulation");
     assert_eq!(network.neurons.len(), 4);