write-protocol lifetime ownership and recovery barriers

.github/workflows/ci.yml

@@ -306,6 +306,7 @@ jobs:
           echo "$output" | grep -Fq "fold_delta_remeasure_bytes=402653184" \
             || { echo "::error::Gate R0 RSS evidence line or 384-MiB tripwire changed"; exit 1; }
           cargo test --locked -p omnigraph-cluster --features failpoints --test failpoints
+          cargo test --locked -p omnigraph-server --features failpoints --test failpoints
 
       - name: Commit regenerated openapi.json to PR branch
         if: |

AGENTS.md

@@ -35,7 +35,7 @@ OmniGraph is a typed property-graph engine built as a coordination layer over ma
 - **Atomic per-query writes**: `mutate_as` and `load` accumulate insert/update batches into an in-memory `MutationStaging.pending` per touched table. Strict insert and upsert both route through the sealed exact-`id`, filter-bearing adapter; bare Lance Append is test-only. Their RFC-022 adapter resolves or rejects relevant recovery intents before base capture, captures `(native branch id, exact graph_head, schema identity)`, then rechecks recovery and authority under schema → branch → table gates. Mutation/Load keeps one keyed transaction per table and rejects more than 8,192 rows or 32 MiB before recovery arm; mutation update scans stream into the remaining table budget after pending-key shadowing, with blob sizes checked before payload reads. It then arms an identity-bearing recovery-v9 sidecar containing exact Lance transaction identities + pre-minted lineage, commits each table with zero transparent conflict retries, confirms the achieved effects, and publishes under the same token. Unrelated retryable pre-effect authority movement may fully reprepare without changing logical mode. A proven effect-free strict conflict returns `KeyConflict` only after a fresh exact-ID probe; no exact match triggers bounded full strict-mode reprepare, while an effect-free upsert conflict also fully reprepares. Any earlier effect or ambiguity returns `RecoveryRequired` with the sidecar retained. Strict read-set conflicts return `ReadSetChanged`. Deletes stage through the same path. D₂ at parse time remains the constructive (insert/update) XOR destructive (delete) boundary.
 - **RFC-026 private streaming core**: Phase A's main-only adapter enrolls one exact-`id` table into one empty unsharded Lance MemWAL under recovery-v10, then publishes its physical binding and `OPEN` lifecycle. Phase B1 adds a root-singleflight worker for one no-roll generation (8,192 rows / 32 MiB of logical dense-slice Arrow data), watcher success plus the same writer's post-durability `check_fenced()` before clean acknowledgement, conservative replay/fold-only recovery, and the bounded fold mechanics. Cheap raw bounds and exact post-tombstone validation run before recovery; put ownership follows charge → shared admission → same-key queue → worker mode; cold replay installs exact fold-only accounting; exclusive admission spans seal, drain proof, both table effects, and publication. Admission, replay, and fold all charge `ArrayData::get_slice_memory_size`; fold densifies retained arrays. Backing-buffer capacity and physical allocation are not the limit, while isolated fold RSS remains a 384-MiB remeasurement tripwire. The 8,192-row high-entropy near-cap closure cell remains green. Internal schema v9 adds the private B2 compare-and-chain core: canonical payload and token digests, grammar-impossible trusted hidden row metadata (`__omnigraph_stream_v1$`), same-generation token overlays, and a manifest-selected graph-global `_stream_tokens.lance` authority. Exact base/token lookups and recovery validation materialize at most the requested rows plus one and cap both Arrow batch bytes and cumulative retained bytes. Admission recaptures lifecycle/binding/HEAD authority only after acquiring shared admission and the same-key queue, so a stale provisional capture cannot authorize a WAL put. Recovery-v12 owns exact pre-minted base and token transactions; only their exact joint outcome may drive the sole `__manifest` CAS that advances both pointers, lifecycle state-v2, graph lineage, and a durable fold-attribution summary. Recovery-v11 is historical v8 state and is refused under v9 because it lacks the token participant and complete state-v2 authority. Post-invocation ambiguity is `AckUnknown`; unresolved recovery blocks progress. The topology remains main-only, unsharded, one resident writer and one live writer process, with no fresh reads or generation GC. The private **B2a unbounded retain-all profile is active**: OmniGraph imposes no retained-byte, object-count, file-count, or history quota and never deletes a canonical durable `_mem_wal` object. Lance may clean only its losing `.binpb.tmp.<uuid>` atomic-CAS staging. Complete and partial unreferenced generation residue stays non-authoritative and untouched through recovery/reopen; provider exhaustion is loud. The genuine v8↔v9 refusal/rebuild gate preserves logical rows—including an ordinary v8 `__omnigraph_stream_v1` user property—vectors, and PK metadata while omitting hidden trusted stream metadata from export. The core remains crate-private behind one feature-gated, doc-hidden test seam. Explicit production enrollment, lifecycle management/correction/status, authorization, and SDK/CLI/HTTP/OpenAPI surfaces remain inactive.
 - **RFC-026 B2 preprocessing bound**: before blob materialization or canonical encoding, the private adapter reserves a 128-MiB worst-case envelope (original 32-MiB Arrow row + possible 32-MiB replacement + 64-MiB canonical payload) and an inflight slot. The private profile admits exactly two envelopes (256 MiB root-wide), preserving the two-caller stale-authority race without unbounded preprocessing. The slot transfers into queued/worker ownership; scratch releases after digest derivation. Pressure fails effect-free as typed `stream_b2_preprocessing_bytes`; this is process-memory admission, not retained-storage GC or quota.
-- **HTTP server**: Axum + utoipa OpenAPI, bearer auth (SHA-256 hashed, optional AWS Secrets Manager). Cedar policy enforcement is engine-wide — every `_as` writer calls `Omnigraph::enforce(action, scope, actor)`, so HTTP, CLI, and embedded SDK consumers all hit the same gate. **Cluster-only boot** (RFC-011): the server always boots from a cluster directory (`--cluster <dir | s3://…>`, RFC-005) and serves N graphs (N ≥ 1) under multi-graph routes (`/graphs/{graph_id}/...` + read-only `GET /graphs` enumeration); there are no single-graph flat routes and no positional-URI boot. Per-graph + server-level Cedar policies. Runtime add/remove (`POST /graphs`, `DELETE /graphs/{id}`) is not exposed — operators run `cluster apply` and restart.
+- **HTTP server**: Axum + utoipa OpenAPI, bearer auth (SHA-256 hashed, optional AWS Secrets Manager). Cedar policy enforcement is engine-wide — every `_as` writer calls `Omnigraph::enforce(action, scope, actor)`, so HTTP, CLI, and embedded SDK consumers all hit the same gate. **Cluster-only boot** (RFC-011): the server always boots from a cluster directory (`--cluster <dir | s3://…>`, RFC-005) and serves N configured graphs (N ≥ 1; the serving subset may transiently be empty in lenient mode while open-quarantined graphs converge under supervision) under multi-graph routes (`/graphs/{graph_id}/...` + read-only `GET /graphs` enumeration with per-graph serving/quarantined status); there are no single-graph flat routes and no positional-URI boot. Per-graph + server-level Cedar policies. Runtime add/remove (`POST /graphs`, `DELETE /graphs/{id}`) is not exposed — operators run `cluster apply` and restart.
 - **CLI** with two-surface config (RFC-007/008): the team-owned cluster directory (`cluster.yaml`) plus the per-operator `~/.omnigraph/config.yaml` (servers, clusters, credentials, actor, profiles, aliases, defaults). Graphs are addressed via `--store`/`--server`/`--cluster`/`--profile`/operator defaults (RFC-011). Multi-format output (json/jsonl/csv/kv/table).
 
 Throughout the docs, capabilities are split into **L1 — Inherited from Lance** vs **L2 — Added by OmniGraph**.
@@ -294,7 +294,7 @@ omnigraph policy explain --cluster ./company-brain --graph knowledge --actor act
 | Three-way row-level merge | — | `OrderedTableCursor` + `StagedTableWriter`, structured `MergeConflictKind` |
 | Change feeds | — | `diff_between` / `diff_commits` with manifest fast path + ID streaming |
 | Cedar policy | — | Per-graph actions plus server-scoped actions (see [docs/user/operations/policy.md](docs/user/operations/policy.md) for the current list), branch / target_branch / protected scopes, validate/test/explain CLI. **Engine-wide enforcement** (MR-722): every `_as` writer (`apply_schema_as`, `mutate_as`, `load_as` — the deprecated `ingest_as` shims route through it — `branch_create_as` / `branch_create_from_as`, `branch_delete_as`, `branch_merge_as`) calls `Omnigraph::enforce(action, scope, actor)` — HTTP, CLI, embedded SDK all hit the same gate. |
-| HTTP server | — | Axum, OpenAPI via utoipa, bearer auth (SHA-256, AWS Secrets Manager option), `authorize_request` at the HTTP boundary (resolves bearer→actor, applies admission control), NDJSON streaming export, **cluster-only boot (RFC-011): always `--cluster <dir | s3://…>`, serving N graphs (N ≥ 1) under multi-graph routes + read-only `GET /graphs` enumeration + per-graph + server-level Cedar policies. Add/remove graphs via `cluster apply` and restart.** |
+| HTTP server | — | Axum, OpenAPI via utoipa, bearer auth (SHA-256, AWS Secrets Manager option), `authorize_request` at the HTTP boundary (resolves bearer→actor, applies admission control), NDJSON streaming export, **cluster-only boot (RFC-011): always `--cluster <dir | s3://…>`, serving N configured graphs (N ≥ 1; lenient mode may transiently serve none while quarantined graphs converge) under multi-graph routes + read-only `GET /graphs` enumeration with per-graph serving/quarantined status + per-graph + server-level Cedar policies. Add/remove graphs via `cluster apply` and restart.** |
 | CLI with config | — | two-surface config (team `cluster.yaml` dir + per-operator `~/.omnigraph/config.yaml`), scope addressing (`--store`/`--server`/`--cluster`/`--profile`/defaults, RFC-011), aliases, multi-format output (json/jsonl/csv/kv/table) |
 | Audit / actor tracking | — | `_as` write APIs + actor map in commit graph |
 | Local S3 testing | — | run RustFS/MinIO + the `AWS_*` env; see [docs/user/deployment.md](docs/user/deployment.md) → *Testing against S3 locally* |

crates/omnigraph-api-types/src/lib.rs

@@ -761,14 +761,49 @@ pub fn read_target_output(target: &ReadTarget) -> ReadTargetOutput {
 
 // ─── MR-668 — management endpoint shapes ──────────────────────────────────
 
+/// Serving state of a configured graph (RFC-030 W3).
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize, ToSchema)]
+#[serde(rename_all = "snake_case")]
+pub enum GraphStatus {
+    /// Open and answering requests.
+    #[default]
+    Serving,
+    /// Configured but not serving: the open failed and the server's
+    /// supervision loop is retrying with capped backoff. Routes under the
+    /// graph answer 503 meanwhile.
+    Quarantined,
+}
+
+/// Quarantine detail on a [`GraphInfo`] whose `status` is `quarantined`
+/// (RFC-030 W3). Timestamps are seconds since the Unix epoch.
+#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
+pub struct GraphQuarantineInfo {
+    /// When the graph entered quarantine (this boot), Unix seconds.
+    pub since_unix_secs: u64,
+    /// Open attempts so far (the failed boot open counts as the first).
+    pub attempts: u32,
+    /// The most recent open error, verbatim.
+    pub last_error: String,
+    /// When the supervision loop will retry, Unix seconds.
+    pub retry_at_unix_secs: u64,
+}
+
 /// One entry in the response from `GET /graphs`. Cluster operators
 /// consume this list to discover which graphs the server is currently
-/// serving. The shape is intentionally minimal — `graph_id` and `uri`
-/// are the only fields a routing client needs.
+/// serving — including graphs that are configured but quarantined
+/// (RFC-030 W3), so an unhealthy graph is visible instead of silently
+/// absent.
 #[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
 pub struct GraphInfo {
     pub graph_id: String,
     pub uri: String,
+    /// Additive (RFC-030): absent in older responses, so deserialization
+    /// defaults to `serving` for wire compatibility.
+    #[serde(default)]
+    pub status: GraphStatus,
+    /// Present only when `status` is `quarantined`.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub quarantine: Option<GraphQuarantineInfo>,
 }
 
 /// Response from `GET /graphs`. Lists every graph registered with the

crates/omnigraph-server/Cargo.toml

@@ -17,6 +17,12 @@ default = []
 # Enables the AWS Secrets Manager bearer-token source. Off by default — on-prem
 # and local-dev builds don't pay the AWS SDK compile cost.
 aws = ["dep:aws-config", "dep:aws-sdk-secretsmanager"]
+# Test-only fault injection: exposes the engine's failpoint registry
+# (`omnigraph::failpoints::{names, ScopedFailPoint}`) to this crate's
+# feature-gated integration tests (`tests/failpoints.rs`). The server defines
+# no failpoint hooks of its own, so unlike omnigraph-cluster there is no
+# `dep:fail` here. Never enabled in production builds.
+failpoints = ["omnigraph/failpoints"]
 
 [dependencies]
 omnigraph = { package = "omnigraph-engine", path = "../omnigraph", version = "0.9.0" }