fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained

.changeset/ready-primary-datasource-drain.md

@@ -0,0 +1,65 @@
+---
+"@objectstack/objectql": minor
+"@objectstack/runtime": minor
+---
+
+fix(runtime,objectql): `/api/v1/ready` drains only on the PRIMARY datasource's failure; a secondary/tenant datasource is reported, not drained (#13408)
+
+On a multi-datasource deployment, one datasource whose driver could not start
+pinned `/api/v1/ready` to 503 on **every replica** — so a readiness-checked load
+balancer drained every upstream and took the whole deployment offline, while
+Postgres and the app itself were healthy (`/api/v1/health` 200, direct reads
+working). One tenant's misconfiguration = total outage. Observed on a live
+3-replica EE deployment and recovered only by restarting every process.
+
+Ruled 2026-08-31 (第 6 场总监席决裁批 #12, maintainer verbatim 「同意」), Option B:
+
+> `/api/v1/ready` 只在**主/默认数据源**不健康时摘流量;次要/租户数据源的故障照常
+> **上报**(`/ready` 响应 body、日志、告警)但不 drain 节点。
+
+**What changed.** When a driver reports itself unhealthy, `/ready` now asks
+which datasource is the deployment's primary one before choosing a status:
+
+- primary unhealthy, or the criterion unresolvable ⇒ **503**, unchanged envelope;
+- primary healthy, only a secondary down ⇒ **200**, with the failed drivers named
+  in a new `degraded` block: `{ status, state, degraded: { drivers, primaryDatasource } }`.
+
+The failed driver is never hidden — the rejected alternative of filtering it out
+of the response stays rejected. `degraded` appears **only** on that branch; an
+all-healthy 200 body is byte-identical to before.
+
+**The criterion is a readable fact, not a heuristic.** `ObjectQL.resolvePrimaryDatasource()`
+(new, exported with `PrimaryDatasourceVerdict` / `PrimaryDatasourceUnresolvedReason`)
+answers *where this deployment's platform system objects actually live*, resolved
+through the same five-step routing order every query uses — never registration
+order, never the driver flagged default at registration. The ADR-0057 §3.6 system
+ledgers (`audit` / `telemetry` / `event`) are excluded because they are
+deliberately routed off the primary. Disagreement, silence, or a name with no
+driver behind it return an **unresolved verdict**, never a guess.
+
+**Fail toward draining.** Every way of not knowing — an engine that predates the
+probe, a probe that throws, a malformed verdict, a genuinely split deployment —
+falls back to the old whole-node 503. Staying in rotation requires a *positive*
+reading; the absence of a negative one is not permission. Pinned in both
+directions, including an inversion ablation.
+
+**framework#3756 is not overturned.** Its quantified reason — "a replica that
+would fail 100% of its requests" — still holds in the single-datasource
+deployment it was measured on, where the primary *is* the only source; that
+deployment's answer is unchanged down to the response body. What #3756's
+reasoning never covered is the multi-datasource shape, and that is the only
+branch this carves out.
+
+**Grade: `minor` for both, proposed rather than assumed** — this changes when a
+published operational probe drains a node, so an operator whose alerting keys on
+`/ready` returning 503 for *any* driver failure will see 200 + `degraded`
+instead, and `degraded` is a new response field. Nothing is removed or renamed,
+no declared contract key is added (the ruling explicitly declines that: 「⛔ 本裁不
+加契约键」), single-datasource behaviour is bit-identical, and no migration is
+required — so `major` overstates it and `patch` understates a deliberate change
+to an availability control surface.
+
+⚠️ Out of scope, tracked separately as **#13578**: `DELETE` of a datasource still
+does not evict the stuck driver from the in-memory engine registry, so the
+datasource keeps appearing in `/ready`'s report until the process restarts. That
+half is a defect under either answer to this card and is queued independently.

content/docs/permissions/system-context.mdx

@@ -109,18 +109,18 @@ that silently does not happen.
 
 | # | Behaviour when `isSystem` | Package | What you get / what you lose | Anchor |
 |:--|:---|:---|:---|:---|
-| 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10616` |
-| 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:10778` |
-| 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9509` |
+| 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10712` |
+| 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:10874` |
+| 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9605` |
 | 21 | **`readonly` strip bypassed — INSERT (protocol ingress)** | metadata-protocol | `isSystem` is the **only** exemption here. `preserveAudit` is deliberately not read on this path (#6640) — a non-system historical import is still stripped on create | `metadata-protocol/src/protocol.ts:1576` |
-| 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9546`, `readonly-strict-errors.ts:66` |
-| 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5638` |
-| 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3573`, `:3583`, `:3610` |
+| 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9642`, `readonly-strict-errors.ts:66` |
+| 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5639` |
+| 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3574`, `:3584`, `:3611` |
 | 25 | Engine-owned / append-only write guard bypassed | plugin-security | Get: generic writes to `managedBy` engine-owned objects | `system-write-guard.ts:96`, `:120` |
 | 26 | Identity write guard bypassed (ADR-0092) | plugin-auth | Get: direct writes to identity tables through the generic data path | `identity-write-guard.ts:98` |
-| 27 | Search-companion column **kept** in a read's rows when it was explicitly requested | objectql | Get: the internal companion column is readable. Lose: nothing for app code — this is the engine reading its own index | `objectql/src/engine.ts:6336` |
-| 28 | Dependent-count disclosure on a blocked delete | objectql | Get: the count of blocking children. Nothing was elevated past the caller, so nothing is withheld | `objectql/src/engine.ts:11364` |
-| 29 | Reference-cleanup log attributes the write to `'system'` | objectql | Get: an honest actor label instead of `anonymous` when the context carries neither `userId` nor `actor` | `objectql/src/engine.ts:11293` |
+| 27 | Search-companion column **kept** in a read's rows when it was explicitly requested | objectql | Get: the internal companion column is readable. Lose: nothing for app code — this is the engine reading its own index | `objectql/src/engine.ts:6337` |
+| 28 | Dependent-count disclosure on a blocked delete | objectql | Get: the count of blocking children. Nothing was elevated past the caller, so nothing is withheld | `objectql/src/engine.ts:11460` |
+| 29 | Reference-cleanup log attributes the write to `'system'` | objectql | Get: an honest actor label instead of `anonymous` when the context carries neither `userId` nor `actor` | `objectql/src/engine.ts:11389` |
 
 ### 3. Sharing (`plugin-sharing`)
 
@@ -179,8 +179,8 @@ a reader tracing where elevation travels needs them.
 
 | # | Site | Package | What it does |
 |:--|:---|:---|:---|
-| 62 | `objectql/src/engine.ts:3405` | objectql | Propagates `isSystem` into the hook session so hooks can tell engine self-writes from user writes |
-| 63 | `objectql/src/engine.ts:13705` | objectql | `ScopedContext.isSystem` getter — re-exposes the underlying execution context's flag |
+| 62 | `objectql/src/engine.ts:3406` | objectql | Propagates `isSystem` into the hook session so hooks can tell engine self-writes from user writes |
+| 63 | `objectql/src/engine.ts:13801` | objectql | `ScopedContext.isSystem` getter — re-exposes the underlying execution context's flag |
 | 64 | `plugin-reports/src/report-service.ts:556` | plugin-reports | Threads the flag into the engine call that runs a report |
 | 65 | `body-runner.ts:279` | runtime | Rebuilds an `ExecutionContext` from a hook session, carrying the flag across |
 
@@ -195,7 +195,7 @@ assuming `isSystem` covers it is a documented source of bugs.
 |:---|:---|:---|
 | "It suppresses triggers / record-change automation" | **No.** Only `skipTriggers` does. A bare `{ isSystem: true }` on a seed write re-fired automation on freshly seeded rows and wedged first boot | `metadata-protocol/src/seed-loader.ts:1909` (rationale at `:1819`–`1821`, #3760), `flow.zod.ts:685` |
 | "It skips the state machine" | **No.** That is `skipStateMachine`, carried by seed replay and by `treatAsHistorical` imports | `objectql/src/engine.ts` FSM gate; see [State Machine](/docs/protocol/objectql/state-machine) |
-| "It skips validation rules" | **No.** Field shape, `format`, `script` and the rest still run. The `readonly` strip runs *before* validation precisely so a discarded value is not judged | `objectql/src/engine.ts:9492`–`9509` |
+| "It skips validation rules" | **No.** Field shape, `format`, `script` and the rest still run. The `readonly` strip runs *before* validation precisely so a discarded value is not judged | `objectql/src/engine.ts:9588`–`9605` |
 | "It preserves a supplied `updated_at` / `updated_by`" | **No.** That is `preserveAudit`, a separate opt-in — and an UPDATE-path exemption only | `field.zod.ts:1451` (#3493 / #6640) |
 | "It stamps `created_by`" | **No.** Audit stamping reads `userId` from the context. A user-less system write stamps nothing — that is today's behaviour, not an error | `runtime-identity.ts:280`–`281` |
 | "It bypasses every guard" | **No.** The last-admin guard applies to **every** context, `isSystem` included — the deprovision path that actually locks an org out is the system one | `last-admin-guard.ts:286` |

packages/objectql/src/driver-connect-errors.ts

@@ -19,6 +19,54 @@ export interface DriverHealth {
   skipped?: boolean;
 }
 
+/**
+ * Why {@link PrimaryDatasourceVerdict} could not name a primary datasource.
+ *
+ * Every member means the SAME thing to a readiness probe — *we cannot tell* —
+ * and the ruling on #13408 fixes what a probe must do with that answer: fall
+ * back to the old whole-node 503 (fail toward draining), never to "don't
+ * drain". They are distinguished only so the reason can be logged and reported;
+ * ⛔ a caller must not branch on them to keep a replica in rotation.
+ */
+export type PrimaryDatasourceUnresolvedReason =
+  /** No platform system object is registered here — nothing to read the fact off. */
+  | 'no-system-objects-registered'
+  /** The platform's system objects are split across more than one datasource. */
+  | 'system-objects-split'
+  /** A registered system object routes nowhere: no binding and no default driver. */
+  | 'system-object-unbound'
+  /** The name resolved, but no driver is registered under it, so nothing probes it. */
+  | 'no-driver-registered';
+
+/**
+ * WHICH datasource carries this deployment's platform system objects — the
+ * machine-readable "primary/default datasource" fact the #13408 ruling requires
+ * (2026-08-31, 第 6 场总监席决裁批 #12), stated as a verdict rather than a
+ * `string | undefined` so "we could not tell" can never be mistaken for a name.
+ *
+ * ⛔ The ruling forbids deriving this from a heuristic such as "the first
+ * datasource registered". The one implementation is
+ * `ObjectQL.resolvePrimaryDatasource()`.
+ */
+export type PrimaryDatasourceVerdict =
+  | {
+      resolved: true;
+      /** The datasource name, which is also a registered driver's name. */
+      datasource: string;
+      /**
+       * How many registered platform system objects agreed on it. Never 0 — a
+       * verdict read off nothing is `no-system-objects-registered`, not a name,
+       * so a caller cannot act on a vacuous agreement.
+       */
+      witnesses: number;
+    }
+  | {
+      resolved: false;
+      reason: PrimaryDatasourceUnresolvedReason;
+      /** The distinct datasource names seen, when more than one disagreed. */
+      candidates?: readonly string[];
+    };
+
 /** `error.message` when it is an Error, its string form otherwise. */
 function failureMessage(error: unknown): string {
   if (error instanceof Error) return error.message || error.name;