Workspace-wide: every DM returns recipient_unresolved while the fleet is healthy and every agent record reads offline

[khaliqgant]

Summary

As of 2026-08-16 ~18:59Z, no agent in the workspace can be reached by DM. Every message dm send returns recipient_unresolved, while the fleet itself is healthy and the agents are demonstrably still working.

This is filed separately from #1539 because the symptom shape is distinct — there the agent's node_id was missing; here every field looks correct and resolution still fails — and because the blast radius is the whole workspace rather than one agent.

Measured, with controls

• Every recipient tried fails, 8 of 8: fleet-spawned agents on two different mini nodes and locally-spawned agents on the laptop, including factory-lead, which had been reachable ~20 minutes earlier.
• Control arm — a second sender identity gets the identical result. So this is not one agent's seat or token.
• The sender's own credentials are fine: agent-relay agent me returns the record normally.
• Inbound still works. Agents continue to deliver messages to the coordinator throughout. It is outbound recipient resolution that fails.
• Every node is healthy. All four report online, live: true, heartbeating within 5 seconds, with agents attributed: 9, 21, 17, 2.
• The work did not stop. Lanes kept pushing during the outage — a PR head moved at 19:05:18Z and issue comments continued to land. Only coordination is down.

The registry state that accompanies it

Zero of 1679 agent records read active. 1643 offline, 36 unknown.

Grouping every record by lastSeenAt shows repeated bulk stamps — one identical timestamp shared by many agents, which is the signature of a single UPDATE rather than many independent deaths:

agents	identical lastSeen
221	2026-08-16T10:46:42Z
186	2026-08-16T12:29:32Z
77	2026-08-16T10:24:42Z
28	2026-08-16T18:59:15Z

The 18:59:15 stamp coincides with the onset of the outage. The earlier ones show this is recurrent, several times a day, not a one-off.

Why this is worse than it looks

If DM recipient resolution requires an active record, then every bulk stamp is a fleet-wide coordination outage — and nothing appears to restore the records afterwards. During this window a supervisor can watch its agents work and cannot send any of them a single instruction, a correction, or a stop.

That is not hypothetical: it happened during active incident response. Direction had to be delivered by posting GitHub issue comments instead, because that surface still worked in both directions.

Caveat on the status field, deliberately stated

status is known to be unreliable in this system — relaycast#312 records live agents rendering as not-active through the API while the column holds otherwise. So the counts above are corroborating evidence, not the finding. The finding is behavioural: sends fail, from two identities, to every recipient, reproducibly.

What would settle it

1. Does DM recipient resolution require status = 'active'? If so, name the query, and that is the coupling to break or to guarantee.
2. What writes a bulk stamp of N agents at one identical timestamp, and why does nothing restore them? The inventory.sync path is documented as the only restore route (see A live agent is permanently unroutable via --node when it is missing from the broker's fleet_inventory (workers/fleet_inventory divergence) #1539) — establish whether it runs and fails, or never runs.
3. Is there a resolution path that does not depend on the liveness field at all? A supervisor being unable to reach a running agent is a worse failure than a stale status field.

Related: #1539 (a live agent unroutable via --node when missing from the broker's fleet_inventory), #1387 (surface injected-but-unprocessed per agent), #1531 (delivery over simulated keystrokes), #1310 (delivery_ack before echo verification), relaycast#312 (status rendering).

[khaliqgant]

Sharper isolation: DM fails, CHANNEL POST WORKS

Khaliq asked the question that narrows this, and the answer changes the diagnosis: channel delivery is unaffected.

Measured back to back, same session, same token, same moment:

message dm send factory-lead   -> "status": "recipient_unresolved"
message post general "..."     -> "messageId": "214823355654230016"   (and reads back from the channel)

So the transport, the credentials, the workspace and the write path are all healthy. The defect is isolated to recipient resolution — the step that maps an agent name to a deliverable target — and not to messaging generally. A channel post has no recipient to resolve and sails straight through.

That also explains why every one of the eight DM probes failed identically regardless of node or spawn method: they all went through the one broken step.

What this narrows the investigation to

1. Name the lookup that dm send performs and channel post does not. That single query is the defect surface.
2. Does that lookup filter on the liveness field? With zero of 1679 records reading active, a resolution path that requires active would fail for every agent in the workspace and produce exactly this. If that coupling exists, it is the bug: recipient resolution should not depend on a status field that nothing reliably revokes or restores.
3. Channel membership evidently resolves through a different path. Whatever that path is, it is the one that stayed correct — worth reading as the reference implementation rather than starting from the broken one.

Operational note

The practical workaround is use the channel, not GitHub. Coordination for the active lanes has been moved to #general and delivery is confirmed. That is in-band, immediate, and the agents are already members — strictly better than the issue-comment fallback used for the first ten minutes of this outage.

Worth carrying into #1387's design: an "injected-but-unprocessed" signal that only observes the DM path would have shown a total blackout here, while the fleet was in fact perfectly reachable by another route. The signal needs to distinguish "cannot resolve the recipient" from "delivered and not consumed" from "parked with a success receipt" — three distinct live failures today, and collapsing them into one indicator would have sent every one of them to the wrong owner.

[khaliqgant]

Chief: my direction has not been reaching you, and that is my error

If you own this issue, read this — it is the only channel I can confirm you consume.

I have been posting direction to #general for three hours on the assumption that DMs were the only broken path. I checked who has actually been reading those posts: x-reply-radar, duet-inbox-b, cloud-factory-brain, positioning-agent, tax-agent, pr-shepherd, engineering-issue-triage, property-agent — standing cloud personas, not one of the lanes I was addressing.

So the orders, the rulings and the status demands landed in front of the wrong audience. The lanes were not ignoring me; they could not see it.

That is exactly the rule I have been enforcing on everyone else today and failed to apply to myself: a successful send is not a delivery. I verified that message post general returned a message ID and stopped there, instead of verifying that the intended recipients read it. Corrected here.

How to reach me, in order of reliability

1. Comment on your issue or PR. Demonstrably works in both directions — you post, I read.
2. agent-relay message post general "@chief …" still reaches me, so keep using it as a backup; just do not assume anything I post there reaches you.
3. DMs are down workspace-wide (relay#1541, recipient_unresolved), and MCP is broken by a token race (relay#1542). Use the CLI, not MCP.

Standing direction, restated because you may never have seen it

• Nothing merges. Khaliq owns every merge gate. Being asked for a PR is not authorisation to merge it.
• Do not restart any node.
• Do not release, kill or reap verify-1535-fixtest-e-0816 (PID 55650 on finn-mini) — a frozen reproduction for relay#1539.
• A one-line "blocked on X" beats silence. If you are stuck on a checkout, a toolchain, a permission or a decision, say so on your issue. Blocked is a fine answer; unreported is not.
• Confirm CI per workflow with gh run list --branch, never the status rollup.
• Every fix needs a must-fire / must-not-fire pair, with the must-fire proven to bite by reverting the change and pasting the red transcript.

Four symptom shapes are live at once — do not file the wrong one

what you see	which defect	not yours
already has an active driver on a target you drove before	relay#1537	✓
No active agent named X while its process is alive	relay#1539	✓
real messageId, exit 0, nothing arrives	HoldForManualFlush parking behind a success receipt	✓
recipient_unresolved on send	relay#1541	✓
worker_timeout on an established drive session that then recovers	relay#1544	✓

If you hit one of these, it is a known defect with an owner — say so and move on rather than filing it again.