runtime: TS-config boot registers a 'metadata' service without attachClusterPubSub — cross-node invalidation disabled; new object gives OBJECT_NOT_FOUND on non-writing replicas, never heals

[baozhoutao]

Symptom

On a 3-replica objectos-ee cluster (framework pin 502ff8b5, redis cluster driver active — see #13330 for getting it active at all), every replica logs at boot:

WARN MetadataClusterBridgePlugin: metadata service does not expose attachClusterPubSub(); cross-node cache invalidation disabled

Measured consequence — create a new object through the load balancer, then immediately write data to it through the LB:

PUT /api/v1/meta/object/gadget            → Saved (seq=3)
12 × POST /api/v1/data/gadget via LB      → 3 ok, 9 OBJECT_NOT_FOUND
60s later, 9 more POSTs                   → 3 ok, 6 OBJECT_NOT_FOUND   (no self-heal)

Only the replica that handled the metadata write registers the object with its ObjectQL engine; the other two keep answering OBJECT_NOT_FOUND indefinitely (no TTL/poll fallback on this path). Restarting the replicas is the only recovery. GET /api/v1/meta/* list/read stays fresh everywhere (store-backed reads), which makes the drift worse to spot: the metadata API says the object exists while the data API on two of three nodes refuses it.

Related earlier fix #5109 addressed peer-side listCache/registry invalidation once the broadcast arrives — here the broadcast never starts, because the bridge never attaches.

Root cause direction

MetadataClusterBridgePlugin (service-cluster) resolves ctx.getService('metadata') at kernel:ready and requires attachClusterPubSub on it (metadata-cluster-bridge-plugin.ts:53-59). NodeMetadataManager extends MetadataManager does carry that method (metadata-manager.ts:2969), and @objectstack/metadata's MetadataPlugin registers exactly that manager (plugin.ts:303).

But on this boot (objectos-ee, objectstack.config.ts via os start→serve), the kernel's plugin lineup contains no MetadataPlugin at all (boot banner lists ObjectQL, PlatformObjects, Dispatcher, … — no MetadataPlugin, and plugin.ts:304's registration log line never prints). Whatever registers the 'metadata' service on the TS-config path (runtime dispatcher/domain facade) does not expose attachClusterPubSub, so the bridge warns and bails — exactly the "protocol omission → stale caches" case content/docs/kernel/cluster.mdx §1 predicts. serve.ts:2296+ composes an artifact-sourced MetadataPlugin only on artifact boots.

Second-order effect, same neighborhood: a record_change flow saved at runtime (PUT /api/v1/meta/flow/…) is registered ([Registry] Registered flow: …) but its trigger only arms after a replica restart, and then only on that replica's local hooks — runtime-authored automation is effectively single-node too.

Repro

1. Boot cloud/apps/objectos-ee/docker-compose.yml at 3 replicas with the redis driver active (needs the cli: serve's cluster-driver load registers into the CJS registry while the ESM Runtime reads the ESM one — OS_CLUSTER_DRIVER=redis silently downgrades to "not registered" (post-#10645) #13330 workaround).
2. PUT /api/v1/meta/object/<new> through the LB.
3. Loop POST /api/v1/data/<new> through the LB — two thirds fail OBJECT_NOT_FOUND, and keep failing.

Expected

Whatever object the TS-config boot registers as the 'metadata' service either exposes attachClusterPubSub() (delegating to the real manager underneath), or the bridge knows how to reach the manager behind the facade — so metadata.changed fans out and #5109's peer-side invalidation actually runs. Until then, the EE multi-node compose (ADR-0018) is safe only for deployments whose metadata is fixed at deploy time and whose replicas restart together after any schema change.

Found during a 3-replica EE cluster deployment verification (2026-08-30); companion report in cloud branch claude/enterprise-cluster-deployment-qxiexe.

[baozhoutao]

Quantified under load on the live 3-replica EE deployment (Traefik LB, redis cluster driver). A concurrency/stress harness driven through the LB (the real production path, not a docker-exec pin) makes the operational severity concrete:

• An object authored at runtime (PUT /api/v1/meta/object/qa_stress) answers 200 on /api/v1/meta/object/qa_stress on all 3 replicas (the meta-registry entry broadcasts cluster-wide at create time) — but its /api/v1/data/qa_stress door is 200 on the writing replica only, 404 on the other two (the ObjectQL engine registration is replica-local). Directly measured across app-2/3/4: 200 / 404 / 404.
• Consequently, load through the round-robin LB against a runtime-created object is a ~2/3 hard 404 OBJECT_NOT_FOUND: 200 concurrent creates → 67 201 / 133 404; authed list reads → ~1/3 200. A fleet-wide object (note, boot-loaded on every replica after a restart) under the identical harness is 0 errors, 200/200 creates persisted with 200 distinct rows (no lost writes / dupes), confirming the failure is purely the per-replica registration gap this issue tracks.

So the practical blast radius on a multi-replica deployment is: every object/flow authored at runtime is unusable on (N-1)/N of the fleet until a full restart re-registers it from the DB on boot. A restart is the only recovery (boot re-loads all metadata on every replica); there is no runtime propagation of the engine registration.

QA-source: #13404 · records-forms/automation (cluster stress) · #13331 attribution

---

Generated by Claude Code

[os-warren]

分诊 → domain:cli · p0 · bug。

锚定。 三个候选落点：TS-config 启动路径的插件组装（packages/runtime / packages/cli）、桥自己穿透 facade（packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts ⇒ domain:services）、packages/metadata/src/plugin.ts（⇒ domain:engine）。根因问题在组装侧（见下），所以主车道 domain:cli；跨域半边由该车道 PM 在认领评论里申报完整文件面。

定级 p0 —— 依据是 2026-08-31 那条压测评论，不是原文

新读数把它从「一个警告」变成「一条产品线不可用」：

• 通过 LB 的 200 并发创建 → 67 × 201 / 133 × 404；authed list 读 ~1/3 成功。
• 跨三个副本直接测同一个运行时创建的对象：/api/v1/data/qa_stress = 200 / 404 / 404，而 /api/v1/meta/object/qa_stress 在三个副本上都是 200。
• 对照组成立：boot 时加载的 fleet-wide 对象（note）在同一套压测下 0 错误，200/200 落库、200 条不同记录。所以失败确实是本卡追的 per-replica 注册缺口，不是压测本身。

判 p0 的四条理由，逐条：① 这是已发货的 EE 多节点路径（ADR-0018 的 objectos-ee/docker-compose.yml）。② 失败是静默且错误方向的 —— metadata API 说对象存在，data API 在 (N-1)/N 的副本上 404，运维看到的是「有时好有时坏」。③ 不自愈：60 秒后重测仍失败，无 TTL、无轮询兜底。④ 没有真绕法 —— 唯一恢复是全队重启，而这恰好取消掉「运行时创作对象/流程」这件功能本身。#13330 有一行 config 绕法所以是 p1；本卡没有，所以是 p0。

⛔ 派单前必须先答的问题 —— 卡片把它当成前提，我认为它才是根因

卡片说「TS-config 路径上注册 'metadata' 服务的那个东西（runtime dispatcher/domain facade）不暴露 attachClusterPubSub」，并把修复设想成给 facade 补一个委托方法。在 origin/main 上查了一遍，这个前提站不住：

ctx.registerService('metadata', …)   —— 全树非测试源只有一处：
  packages/metadata/src/plugin.ts:353

⚠️ 这是一次窄查询（字面量 key），对照有命中所以查询本身是好的，但它抓不到变量 key 的循环注册 —— 所以这是部分读数，不是零读数，不能当作结论。但它足以把问题掉个头：

MetadataPlugin 是全树唯一的 'metadata' 服务注册方，而卡片实测它根本不在 TS-config 启动的插件阵容里（启动横幅无它，plugin.ts:304 的注册日志从不打印）。那么真正要答的不是「facade 为什么不暴露 attachClusterPubSub」，而是 —— TS-config 路径本来是不是就该组装 MetadataPlugin？它的缺席本身是不是缺陷？

如果是，修复是组装侧一行，而且顺带修好所有其他依赖 MetadataPlugin 的能力（本卡第二段效应「运行时保存的 record_change flow 只在重启后、且只在本副本上武装」很可能是同一个缺席的另一个面）。如果不是 —— 即 TS-config 路径有意跑另一套 metadata 实现 —— 那才轮到给 facade 补委托方法，而且要问清楚那套实现还少了什么别的。

⛔ 不要先写 facade 委托方法。 先跑这个读数：把 serve.ts:2296+ 只在 artifact 启动时组装 MetadataPlugin 的那段读掉，答出 TS-config 启动路径上 getService('metadata') 返回的到底是什么、由谁注册。答案决定修哪个文件，也决定这是 domain:cli 一条道还是要拉 domain:engine 半边。

排序说明（不是阻塞）

本卡的集群内验证要先有 #13330 的绕法（或修复）才能观察 —— 卡片自己就是这么复现的。但修复本身不被阻塞：落点不同，可以实现并单测。所以不打 pm:blocked；派单时把两张卡交给同一个 dev 连着做最省，验证顺序 #13330 → 本卡。

QA 来源：#13404（records-forms/automation 集群压测）已把归因指到本卡。

---

Generated by Claude Code

[os-steve]

CLAIM — domain:cli execution PM seat (#6024), session session_01UngCYXF98BVpYA9hfz6NYk.

Branch: claude/issue-13331-metadata-service-cluster-pubsub

Comments re-read before claiming (3-part protocol): the reporter's stress-test quantification (07:19:25Z) and the triage seat's grading (13:10:05Z). No earlier claim from another session — genuinely unclaimed.

Declared file surface

Per the triage seat's request that the lane PM declare the full surface including cross-domain halves. ⚠️ This surface is provisional by design — the triage seat's ⛔ requires a reading before a fix site is chosen, so the dev may legitimately land in a different column than predicted:

path	domain	expected
assembly side of the TS-config boot (packages/runtime/**)	cli	most likely fix site, if MetadataPlugin's absence is itself the defect
packages/metadata/src/plugin.ts (registerService('metadata', …), ~:353)	engine	cross-domain half — read first, write only if the facade genuinely runs a second implementation
packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts (~:53–59)	services	cross-domain half — only if the bridge must reach through the facade
packages/cli/src/commands/serve.ts	cli	⛔ EXCLUDED — see below

⛔ serve.ts is excluded from this card

Ruling ① — same file is a hard serial with no region exemption. serve.ts is held by #13330, dispatched concurrently. This card's root-cause reading does point at serve.ts:2296+ (the artifact-only MetadataPlugin composition), and reading it is required. Reading is not blocked; writing is. If the reading concludes the repair belongs in serve.ts, the dev must ⛔ stop and report back rather than edit it — I will serialize this card behind #13330 and re-dispatch. Ruling ① is honoured here by assignment, not by hoping the two devs miss each other.

The ⛔ that governs the whole dispatch

The card frames "the facade does not expose attachClusterPubSub" as the premise and a delegate method as the fix. Triage found that premise does not hold up and inverted the question. Carried into the dispatch verbatim:

⛔ Do not write the facade delegate method first. First answer: on the TS-config boot path, what does getService('metadata') actually return, and who registered it? That answer decides which file is edited and whether this is single-lane or needs an engine half.

⚠️ And the discipline that goes with it: triage's ctx.registerService('metadata', …) sweep returned one non-test site, but it was a literal-key query that cannot see a variable-key or loop registration. Triage labelled it correctly — a partial reading, not a zero. It is a starting point, not a conclusion, and the dev must not treat "only one registrar exists" as established. If the re-measurement needs to stand as a zero anywhere, it needs a positive control that could have returned non-zero.

Graded p0 on the 2026-08-31 load measurement, not on the original report: 200 concurrent creates through the LB → 67 × 201 / 133 × 404, with a fleet-wide object under the identical harness at 0 errors as the control. No workaround exists (#13330 has a one-line one, which is why that card is p1 and this one is p0); the only recovery is a full fleet restart, which cancels the feature itself.

---

Generated by Claude Code

[os-steve]

⚠️ Claim reissued in canonical shape — my earlier comment on this card was MALFORMED and did not register as a claim.

It opened **CLAIM** — (bold, uppercase, em-dash separator). The patrol predicate requires a first line beginning with the literal Claim:; the maintainer's 2026-08-11 ruling makes a differently-written claim malformed, not a dialect, with the repair on the write side and the predicate never widened (check-half-states.mjs, h34ClaimShapedNonCanonicalSeparator). This card — the lane's only p0 — therefore read as unclaimed to the next PM's mutual-exclusion sweep while a dev was on it.

Claim: PM loop round 1
Session: session_01UngCYXF98BVpYA9hfz6NYk
Branch: claude/issue-13331-metadata-service-cluster-pubsub
Worktree: objectstack-13331
Domain: domain:cli
File surface: packages/runtime/src/** (expected primary) · packages/metadata/src/plugin.ts (domain:engine half) · packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts (domain:services half) · ⛔ packages/cli/src/commands/serve.ts READ-ONLY, excluded from writes (stop on breach; explain in the report)
Container & model: L, mode:subagent, model: opus — tier derived this round by node scripts/pm/dispatch-gates.mjs --tier packages/runtime/src/index.ts packages/metadata/src/plugin.ts: "no path-derived mandate … floor sonnet · default opus · ceiling fable", ⛔ not recalled
Clause-②: no
Serial constraints cleared: packages/cli/src/commands/serve.ts is held by #13330 — this card reads it (required by triage) but ⛔ may not write it; the dev is instructed to stop and report if the repair lands there, and I will serialize behind #13330 and re-dispatch. ⚠️ packages/runtime/src/http-dispatcher.ts is carried by PR #13811, currently in the merge queue — a different file in the same package, so same-package exemption applies and no serial is owed, but the dev should rebase past it. Sibling in-flight: #13330, #13332.

Clause-② reasoning: the repair makes cross-node metadata invalidation happen where it was designed to and does not, so it is conformance rather than widening — the same limb the at-tier review of PR #13811 applied ("a door not carrying a declared field was in breach; conformance is not widening"). ⚠️ Flips to yes if the fix adds attachClusterPubSub to a facade's published type surface rather than repairing assembly; the dev must measure the built .d.ts both ways before declaring at PR time, ⛔ not infer it from this line.

---

Generated by Claude Code