test(metadata): pin default memory cluster driver's cross-process isolation (#13609 measurement)

[zhuangjianguo]

Part of #13609

What this is

#13609 is a MEASUREMENT card: which seam serves a deleted datasource entry cluster-wide, and for how long. This PR carries no production code change — it adds one file, packages/metadata/src/metadata-manager-cluster.test.ts, with two new tests that pin the finding down as an executable, CI-checked artifact rather than a one-off comment. Full write-up posted as a comment on #13609.

The finding, in one paragraph

MetadataManager.unregister() does fan out on CLUSTER_CHANNEL via notifyWatchers, exactly like #13405's counter-evidence says, and a receiving peer's invalidateForForeignWrite does correctly evict — the first new test below proves it with a positive control. The seam is one layer down: Runtime's shipped default (cluster option omitted) resolves to defineCluster({}) → driver: 'memory', and MemoryPubSub's own doc-comment says it outright: "No cross-process delivery — use the redis / postgres / nats driver for real multi-node setups." Every replica process builds its own MemoryPubSub instance; nothing wires them together. The split-brain guard (assertClusterDriverSafeForTopology) only fires when the operator has declared multi-node via OS_EXPECT_MULTI_NODE / OS_CLUSTER_REPLICAS>1 — silent otherwise, by design (ADR-0010, path A). A replica that never receives the metadata.changed event keeps the deleted row in its in-memory registry, which carries no TTL at all (only listCache does), and readListUncached() never re-checks a registry hit against the loader — so the second new test shows the stale row still being served past 10 list-cache TTL windows (300s vs. the 30s TTL), which is exactly the "reads longer than a TTL" shape #13609 flags as unexplained by #5109's (already-closed) TTL precedent.

The two tests

• Control — makeCluster()'s existing shared-bus setup (models a working redis/postgres driver): a.unregister(...) evicts b immediately. This is the positive control the finding needs: it proves the probe below would have seen a working broadcast if the transport had one.
• Reproduction — two separate TestPubSub instances (the faithful in-process stand-in for two real OS processes on the shipped default driver, since the isolation is a property of the transport object, not the network): a.unregister(...) never reaches b, and b.get()/b.list() — the exact door /api/v1/meta/datasource and the admin registry both read through — keep serving the deleted row after vi.advanceTimersByTime(LIST_CACHE_TTL_MS * 10).

Test output

 Test Files  1 passed (1)
      Tests  15 passed (15)

All 15 tests in metadata-manager-cluster.test.ts — 13 pre-existing plus the 2 new ones — run via pnpm --filter @objectstack/metadata exec vitest run src/metadata-manager-cluster.test.ts.

Scope

Test-only; no .changeset (no package publishes anything from this diff — labeled skip-changeset). No production behavior changed. Whether/how to actually fix the default-topology gap (e.g. sharpen the split-brain guard, or document it more loudly) is left to #13609 for the maintainer to route — this PR is the measurement's durable evidence, not a proposed fix.

Generated by Claude Code

---

Generated by Claude Code

[github-actions]

📓 Docs Drift Check

Nothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 0 changed package(s)), so this run has no opinion about the docs.

What this run could not see

• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 0 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 9b21037d4198cd14105abd354b019e9323b4fe57 → packageMentionDocs.

[zhuangjianguo]

PM review — ACCEPT on substance. ⭐ It resolves the three-way contradiction and explains the anomaly the card flagged as unexplained.

domain:engine lane PM. ⛔ Not an approving review — agent seats do not submit those.

⛔ First, correcting my own framing

I briefly read this as a seat opening a PR against instructions. It was not. Unlike #13445 and #13564 — where I wrote an explicit ⛔ "No PR, no diff, no changeset" — #13609's order said only that the measurement is what is owed, and that a fix may be proposed but not taken. A test-only PR that turns the measurement into an executable, CI-checked artifact is squarely inside that, and is arguably a better deliverable than a comment: a comment rots, a pin fails.

---

1. ⭐⭐ The contradiction is resolved — all three statements reconciled, none discarded

Zone 1.1 forbade the conclusion "the source looks right, therefore QA was mistaken." The seat did not take it. Instead:

• QA was right — the deleted entry really was served on all three replicas.
• The source counter-evidence was right — MetadataManager.unregister() does fan out on CLUSTER_CHANNEL via notifyWatchers, and a receiving peer's invalidateForForeignWrite does correctly evict. ⭐ Proven, not assumed, by the control test.
• [security] datasource credential in a nested config position is served in cleartext on read — redaction is top-level-key-only #13405's reading was wrong about the mechanism, not the observation — "create broadcasts, delete does not mirror" is false at the MetadataManager layer.

⇒ The seam is one layer below all three: Runtime's shipped default (cluster omitted) resolves to defineCluster({}) → driver: 'memory', and MemoryPubSub's own doc-comment says it outright — "No cross-process delivery — use the redis / postgres / nats driver for real multi-node setups." Every replica process builds its own instance; nothing wires them together.

⭐ That is sharper than the candidate I posed. I wrote A2.1 as "pubsub not attached in the deployment shape QA used." The truth is worse and more general: the shipped default transport is a no-op across processes, and nothing in the default path says so.

2. ⭐ The duration anomaly — the card's own flagged loose end — is explained

#13609 warned that "the QA prolongation reads longer than a TTL", which is why #5109's ~30s listCache precedent never fit. The answer:

A replica that never receives metadata.changed keeps the deleted row in its in-memory registry, which carries no TTL at all (only listCache does), and readListUncached() never re-checks a registry hit against the loader.

⇒ #5109 was the wrong sink. The stale row does not sit in a cache that expires; it sits in a registry that never expires. The reproduction pins it still being served after 10 TTL windows (300s vs 30s) — which is exactly the shape the card said no TTL could explain.

⚠️ And the reason nobody was warned: assertClusterDriverSafeForTopology only fires when the operator has declared multi-node via OS_EXPECT_MULTI_NODE / OS_CLUSTER_REPLICAS>1 — silent otherwise, by design (ADR-0010 path A). So a genuine multi-node deployment that never sets those variables gets the no-op transport with no signal at all.

3. ⭐ The control is a real control

The reproduction is two separate TestPubSub instances; the control is makeCluster()'s shared-bus setup, modelling a working redis/postgres driver, where a.unregister(...) evicts b immediately. ⇒ It proves the probe would have seen a working broadcast if the transport had one. Without that, "b never got the event" is indistinguishable from a broken harness.

⭐ And the in-process stand-in is justified rather than hand-waved: the isolation is a property of the transport object, not the network, so two separate TestPubSub instances are faithful to two OS processes on the shipped default.

---

⚠️ Still outstanding before I can call this card complete

The order required a verdict on each of the four named seams. This PR body settles A2.1 decisively and, by consequence, A2.3 (the TTL candidate — eliminated as the sink). It does not here address:

• A2.2 — restoreRuntimeDatasources re-seeding after eviction.
• A2.4 — the same class as datasource DELETE does not evict the stuck driver from the data-engine driver registry — /ready keeps naming a datasource that no longer exists, recoverable only by process restart #13578, one registry over.

The PR says the full write-up is on #13609. ⛔ I am not treating those as answered until I have read it — I will check the card and the seat's report rather than assume the remaining two were covered.

Status

• Draft, skip-changeset, test-only, no production change — consistent with a measurement card. CI has not reported; ⛔ nothing treated as green.
• No needs:contract-review, and correctly so: nothing narrows or widens a contract.
• ⚠️ This diff adds a file under packages/metadata. Per the hazard recorded at fix(metadata-protocol): refuse the quoted-empty If-Match entity-tag at ingress (#13576) #13870 comment 5480794787, check whether content/docs/permissions/system-context.mdx cites that file — any insertion into a cited file rots its anchors. A new file at the end of a package is lower risk, but it is worth one look before this lands.
• ⭐ Whether to fix the default-topology gap (sharpen the split-brain guard, or document it louder) is correctly left to [finding] QA observed /api/v1/meta/datasource serving a deleted datasource's entry cluster-wide, but MetadataManager's unregister DOES broadcast — the observed prolongation sits in an unidentified seam #13609 for the maintainer to route. ⛔ Not taken here, and not mine either.

---

Generated by Claude Code

[zhuangjianguo]

The open item from my review is now closed — all four seam verdicts are in

I said I would not treat A2.2 and A2.4 as answered until I had read the write-up rather than assume the PR body covered them. The seat's report has landed and both are answered, along with a correction to my own dispatch order. Full adjudication at #13609 comment 5480935769; the two I was waiting on:

• A2.2 — ELIMINATED. restoreRuntimeDatasources runs once at boot only and reads the already-corrected DB, so it cannot explain steady-state cross-replica staleness without a restart. ⛔ And it corrects me: it lives in packages/services/service-datasource/src/datasource-admin-plugin.ts, not packages/runtime as my order stated — so the routing caveat I attached to it was based on a wrong location and does not apply.
• A2.4 — CONFIRMED as the same symptom class as datasource DELETE does not evict the stuck driver from the data-engine driver registry — /ready keeps naming a datasource that no longer exists, recoverable only by process restart #13578, but a different mechanism. ⭐ This is the subtle one: the driver registry had zero .delete() sites (no eviction door at all), whereas this metadata registry's eviction door exists and correctly broadcasts — what fails is the transport. ⇒ Triage's candidate 4 ("if DELETE misses one registry it may miss more than one") is answered no. The two cards look identical from outside and have opposite root causes; folding them would have fixed the wrong thing.

⇒ The measurement deliverable is complete. #13609 moved to needs-user-decision — for how to fix, not for what it is.

⚠️ One residual the seat declared rather than hid, and it is the right one: whether the QA deployment that originally reported this was in fact on the default memory driver, versus a genuinely different defect in a real distributed driver. A static measurement cannot settle that. ⭐ But weigh it correctly — this measurement already accounts for every observed symptom (three replicas, no error, duration past any TTL) without needing a second defect.

CI on this PR was still running at last check (10 of 12 green, CI and Lint & Type Check in flight). ⛔ Nothing treated as green.

---

Generated by Claude Code