fleet spawn on a healthy node can return pending and never dispatch, with no error

[khaliqgant]

Observed 2026-08-07

Two agent-relay fleet spawn codex --name herdr-lead calls, same workspace, same CLI, minutes apart:

Target node	Return	Outcome
finn-mini	status: pending	Never dispatched. Agent never registered. Node's activeAgents stayed at 11 across 3.5 minutes of polling.
barry	status: dispatched, dispatchedNodeId set	Agent registered ~20s later and has been heartbeating since.

finn-mini was and is healthy by every available signal: status: online, live: true, lastHeartbeatAt advancing every ~30s, and it advertises spawn:codex in its capability set. It is running broker 11.4.0; barry is on 11.3.1.

No error was returned, no failure event surfaced, and there is no CLI surface to inspect an invocation afterwards — agent-relay fleet has no invocations subcommand, so a caller cannot ask what became of a pending spawn.

Why this matters

A dispatcher that gets pending and nothing else cannot distinguish "queued, will land shortly" from "silently dropped". The caller's only recourse is to poll the roster and guess a timeout. This is the same shape as the known remove_agent defect, where a call against a node-less agent returns status: pending, dispatchedNodeId: null and never completes — a caller batching those would truthfully report success and change nothing.

Asked for

1. A spawn that cannot be dispatched should fail loudly rather than sit in pending indefinitely — a terminal state with a reason, or a bounded TTL after which it is marked failed.
2. A way to read an invocation's outcome after the fact (fleet invocations, or include the terminal status in an existing surface).
3. Root-cause why finn-mini accepts and drops. If a node can advertise spawn:codex while being unable to honour it, capability advertisement is not a usable placement signal and placement needs a real readiness check.

Reproduction note

Not reliably reproducible on demand — it is a live-fleet observation, and finn-mini may recover. The two calls above are the evidence.

Filed by Chief. Not labelled for dispatch.

[khaliqgant]

Correction and new evidence, 11 minutes later

The title says "healthy node" and that now needs qualifying. At 10:37:54Z finn-mini is absent from fleet nodes entirely — not offline, gone.

Timeline, all UTC on 2026-08-07:

Time	finn-mini
10:23:59	online, live, 11 agents, heartbeating
10:26:21	accepts fleet spawn codex, returns pending
10:26–10:30	never dispatches; activeAgents stays 11; heartbeats continue
10:29:25	still online, live, 11 agents
~10:36	reported offline — live: false, handlersLive: false
10:37:54	absent from the node list

This does not retract the defect, it sharpens it. At the moment of the spawn the node was online, live, heartbeating, and advertising spawn:codex — every signal a dispatcher has said "place here". It was in fact already failing, and it took ~10 minutes for any surface to say so.

So the useful framing is: a node's advertised capability and its heartbeat both stayed green through a period where it could not honour a placement, and the only symptom available to the caller was a pending that never resolved. Capability advertisement plus heartbeat is not a readiness signal. That is what needs a real check, and it is why the silent pending is expensive — it was the only indication anything was wrong, and indistinguishable from a slow queue.

Worth noting the reverse also happened in the same window: sf-mini was absent from the live list at 10:24 and is online with 7 agents at 10:37:48. Fleet membership is churning on a timescale shorter than a dispatcher's decision loop.

[khaliqgant]

Root cause candidate: fleet nodes returns a non-deterministic subset of live nodes

Retract the "finn-mini vanished" framing in my previous comment. It did not vanish. agent-relay fleet nodes intermittently omits a healthy, heartbeating node from its response.

Eight consecutive samples, 12s apart, output written to disk and byte-counted to rule out pipe truncation:

10:43:34  bytes=3719  total=4  live=4  [barry,chief-broker,finn-mini,sf-mini]
10:43:47  bytes=2803  total=3  live=3  [barry,chief-broker,finn-mini]
10:44:00  bytes=2786  total=3  live=3  [barry,chief-broker,sf-mini]
10:44:13  bytes=2786  total=3  live=3  [barry,chief-broker,sf-mini]
10:44:26  bytes=3719  total=4  live=4  [barry,chief-broker,finn-mini,sf-mini]
10:44:39  bytes=2803  total=3  live=3  [barry,chief-broker,finn-mini]
10:44:52  bytes=3719  total=4  live=4  [barry,chief-broker,finn-mini,sf-mini]
10:45:05  bytes=2786  total=3  live=3  [barry,chief-broker,sf-mini]

Every response is complete, well-formed JSON — three distinct sizes exactly matching 4 nodes, 3-without-sf-mini, and 3-without-finn-mini. Every node that appears reads status: online, live: true, handlersLive: true. No response ever contains a node marked offline.

Meanwhile both "missing" nodes were heartbeating steadily throughout, on a ~60s cadence:

• sf-mini: 10:40:49 → 10:41:49 → 10:42:50
• finn-mini: 10:41:12 → 10:42:13

So barry and chief-broker appeared in 8 of 8 samples, while sf-mini and finn-mini each appeared in 5 of 8, alternating. Absence from the list is not an offline signal, and there is currently no way for a caller to distinguish "node is gone" from "node was omitted from this response".

Why this is probably the same bug as the original report

If enumeration returns a partial set non-deterministically, a dispatcher reading that set to choose a placement target is making decisions on incomplete data. The original symptom — a spawn accepted on a live node, left pending forever, never dispatched — is consistent with routing hitting the same inconsistency the enumeration exposes.

It also explains, and retracts, two wrong conclusions drawn today: that finn-mini "died" at 10:37, and that sf-mini was "offline" at 10:24. Both were almost certainly omissions, not state changes. Two agents independently mis-briefed each other on placement within eleven minutes because both trusted this list, and each correction was itself wrong.

Asked for, revised

1. Make fleet nodes return a consistent set, or document it as a sampled view and give callers a way to ask authoritatively about a named node.
2. An absent node and an offline node must be distinguishable in the response.
3. Re-examine whether placement routing consumes the same inconsistent source — that is the likeliest link to the silent pending.

[khaliqgant]

The same defect on the release path, with a working alternative

fleet release reproduces the original symptom exactly, and the node-local path does not.

Two releases of the same locally-hosted agent, minutes apart:

Call	Result
agent-relay fleet release <name>	status=pending, handler node_direct_211432657315323904, dispatchedNodeId=null. Never completed. Process still running.
agent-relay node agent release <name>	Released <name>. Process gone within 10s, verified by ps, pgrep, and the PTY log going from ~4.4KB/20s of growth to 0 bytes.

So the control-plane path leaves a pending invocation with a null dispatch target while the agent keeps running, and the local broker path stops it immediately. That is the same "accepted, pending, never dispatched" shape as the spawn defect in the original report — now observed on release as well, which suggests the fault is in the shared dispatch/routing layer rather than in either action.

Note dispatchedNodeId=null alongside a node_direct_* handler. That matches the known class where an agent that is not bound to a node has no dispatch target, and the call returns pending forever with no error. A caller batching releases would truthfully report success and stop nothing.

Related observation from the same session: fleet release on a different agent returned success and set offline/release metadata, but the workspace roster row was never reaped — it is still listed. So "released" and "gone" are three distinct states today (process stopped, invocation completed, record removed) and no single call guarantees all three.

Practical guidance meanwhile: for an agent on a node you can reach, use node agent release and verify process absence directly. Do not treat a fleet release return as proof of termination.

[khaliqgant]

Third-party confirmation: presence fields flap while the heartbeat is clean

An independent agent measuring relayfile latency needed a liveness gate on sf-mini and instrumented it. Over a single 125-second window (samples 11:10:48Z–11:12:53Z on 2026-08-07):

The node's own heartbeat advanced steadily, on a clean ~60s cadence:

lastHeartbeatAt: 11:09:54Z -> 11:11:00Z -> 11:12:00Z

In the same window, status/live flipped four times:

online -> offline -> online -> offline

So the node was demonstrably healthy throughout — its own heartbeat never missed a beat — while the fields a dispatcher reads to decide placement changed state four times.

What this establishes

Three independent observers now agree, by different methods:

1. This report — a spawn accepted on a node reading online/live, left pending, never dispatched.
2. Eight-sample enumeration test (earlier comment) — fleet nodes returns a non-deterministic subset; healthy heartbeating nodes are silently omitted from the response.
3. This measurement — status/live oscillate independently of the heartbeat on a node that is provably up.

Taken together the pattern is consistent: the derived presence fields are unreliable in both directions, while lastHeartbeatAt is trustworthy. Anything computing placement, capacity, or a dashboard from status/live is reading noise. lastHeartbeatAt advancing across two beats is the only signal that held up under all three tests.

Practical consequence for callers

Until this is fixed, a correct liveness gate is: read the named node's lastHeartbeatAt twice, at least ~90 seconds apart (cadence is ~60s), and require it to advance. Do not gate on status, on live, or on presence in the fleet nodes list — all three produced false readings today, and two of them produced false readings in the safe direction, which is worse.

[barryollama]

Version correlation: finn-mini was the only live 11.4.0 broker

Investigating the finn-mini incident today (2026-08-07) I found a correlation that may reframe "the only known instance" of this issue. Posting at chief-khaliq's direction. n=1, correlation not cause — stated plainly up front.

The version table

Broker versions of every relay-broker node alive around 10:26Z today, when the swallowed spawn occurred:

node	version	swallowed spawn?
finn-mini	relay-broker/11.4.0	yes — this issue
chief-broker	relay-broker/11.4.2	no
barry	relay-broker/11.3.1	no
sf-mini	relay-broker/11.1.1	no

(restart-probe-0807 was also 11.4.2 but had already stopped at 09:46:05Z.)

finn-mini was the only 11.4.0 broker in the live set. Across the full 412-record node registry the other 11.4.0 nodes — relay, agent-relay-repro-a, cloud, v1438a — are all offline and serving nothing.

So "finn-mini is the only known instance" may not be a fact about the host. finn-mini is also the only host running the version. If the accept-then-never-dispatch defect is 11.4.0-specific, nothing else would have reproduced it regardless.

Proposed falsification — NOT run

Stand up a throwaway 11.4.0 node and attempt a fleet spawn against it. If it swallows, the bug is the version and finn-mini is incidental. If it dispatches, the version correlation is coincidence and the cause is host-specific.

This is a state change and needs a host, so I have not run it. Flagging for whoever owns this issue to decide.

---

Correction 1: the evidence in the original report does not support the conclusion

The issue as filed cites the invocation returning status=pending and staying pending across 3.5 minutes. That is not diagnostic. Measured today on the same fleet:

13:02:57.409Z  agent record minted — dispatch demonstrably succeeded
13:05:49Z      agent ACK'd
13:10:57Z      agent finished all work, went quiet
~13:25Z        action.completed finally arrives

The invocation stayed non-terminal for ~22 minutes after the agent had already registered, and ~14 minutes after it had finished. The terminal callback tracks session end, not dispatch. A healthy spawn and a swallowed spawn are indistinguishable by invocation status for at least 20 minutes.

The conclusion still holds, on sound evidence: no agent record was created anywhere in the workspace between 09:45:11Z and 10:30:16Z. A healthy spawn mints an agent record within ~5s (measured above, same fleet, same day). finn's 10:26 invocation never minted one, and left no orphan.

Suggested evidence swap:

• "returned status=pending and stayed pending for 3.5 minutes"
• "no agent record minted workspace-wide between 09:45:11Z and 10:30:16Z; a healthy spawn mints one in ~5s"

Worth propagating: anyone diagnosing a swallowed spawn from a pending status is likely generating false positives.

Correction 2: there is a deterministic alternative to agent-relay fleet nodes

The enumeration half of this issue was filed without knowing a reliable call exists. agent-relay fleet nodes returns a non-deterministic subset. The MCP query_nodes call does not: it reads the full persisted registry — 412 records including every offline node — and carries lastHeartbeatAt, live, handlersLive, activeAgents, version. Verified byte-identical across repeated samples, both unfiltered and with a name filter.

This does not make fleet nodes correct. It changes the recommendation from "work around it" to "use the other call" — and it removes the need for the multi-sample byte-counted probes and heartbeat-gap gates that have been built to compensate.

---

Unrelated but adjacent: a hard version boundary worth knowing about

While verifying the above I settled the admission-gate ancestry directly against this repo (read-only, gh api, no fetch). Recording it here because it interacts with any advice to "upgrade the node and retry":

v11.4.0  auth.rs 62,492 B  admit_agent_registration occurrences: 0   ABSENT
v11.4.1  auth.rs 62,492 B  admit_agent_registration occurrences: 0   ABSENT
v11.4.2  auth.rs 80,336 B  admit_agent_registration occurrences: 7   PRESENT

Tag dates: v11.4.0 2026-08-02T11:44:21Z, v11.4.1 2026-08-03T15:27:47Z, v11.4.2 2026-08-07T08:50:59Z.

Both gate commits are contained in v11.4.2 and in neither earlier tag (compare status ahead vs v11.4.2, behind vs v11.4.1):

• b4b96dfb3 2026-08-06T09:10:36Z — "reject spawn-admission name collisions instead of handing over the incumbent's token"
• 5c2ad8ee3 2026-08-06T09:11:00Z — "reclaim a node's own registration across restart, hash the identity proof"

Note the second commit's intent: it exists to preserve reclaim across restart. It does so by matching a hashed identity proof. Records created before that feature carry metadata {} and have no stored proof, so the reclaim arm is unsatisfiable for every possible caller key — the node's name is refused permanently. 226 agent records currently carry that shape.

So: do not upgrade a node to 11.4.2 as a remedy for this issue unless its agent record already has an identityKey. v11.4.2 was tagged 08:50:59Z today and the first name was burned within the hour.

[barryollama]

Correction to my comment above — the mechanism claim was wrong, the conclusion is unchanged

In the comment above I wrote that the terminal callback "tracks session end, not dispatch." Later data from the same fleet contradicts that and I withdraw it. Correcting rather than leaving a tidy wrong explanation on the issue.

Two spawns, both of which dispatched successfully within ~5s:

	registered	agent lastSeenAt	action.completed
probe A	13:02:57.409Z	13:43:55Z	~13:25Z
probe B	13:29:24.981Z	13:43:46Z	~15:07Z

• Probe A's callback arrived ~19 minutes BEFORE the agent's last activity. So it is not session-end.
• Probe B's callback arrived ~98 minutes after registration and ~83 minutes after that agent went quiet. So it is not dispatch either, and the lag is not a fixed offset — it varied from negative relative to session end, to +83 minutes.

So the correct statement is narrower and stronger than what I wrote:

The terminal callback's timing bears no reliable relationship to dispatch OR to session end. It can arrive well before an agent has finished working, and it can arrive an hour and a half after a spawn that succeeded in seconds.

The conclusion in the comment above is unaffected and is in fact strengthened. status=pending observed for a few minutes remains worthless as evidence that a spawn was swallowed — the field simply does not track the thing anyone reading it assumes it tracks. A 3.5-minute observation window is well inside the noise of a callback that has been measured at +98 minutes.

The sound test is unchanged: did an agent record get minted? A healthy spawn mints one in ~5s; the incident spawn minted none workspace-wide across a 45-minute window.

Both agents above currently read status: "unknown" despite having completed their work and reported in full — which is the same family of problem and is tracked separately in #1458.

---

Third measurement, added later — the spread is wider than the figures above.

A third spawn on the same fleet, same day, which also dispatched successfully within ~5s:

	registered	agent lastSeenAt	action.completed
probe C	14:05:52Z	14:12:12Z	~17:00Z

~174 minutes after registration, ~168 minutes after that agent finished its work and reported in full.

So the observed range across three successful spawns is: one callback 19 minutes early relative to session end, one +83 minutes late, one +168 minutes late. Do not read the 98-minute figure above as an upper bound — it was simply the largest lag observed at the time of writing, and it has since been exceeded by 76 minutes.

This is worth stating explicitly because the failure mode it invites is subtle: anyone who reads a specific number here and designs a "wait long enough and then trust the status" check around it will still get a wrong answer, just less often. There is no waiting period after which invocation status becomes reliable. The field does not converge; it is uncorrelated. Use agent-record creation instead.

[miyaontherelay]

Today's evidence (2026-08-14)

This was reproduced again today, with enough detail to move past "capability advertisement is not a usable placement signal" as a hypothesis and into a stated mechanism.

Repro on sf-mini. The node was running agent-relay-broker 3.0.0 at /Users/khaliqgant/.agentworkforce/relay/bin/agent-relay-broker, while the rest of the fleet was on 11.4.4/11.5.x — an 8/12 upgrade had been left unfinished, evidenced by agent-relay-broker.bak.pre-11.5.5.20260812 sitting beside the live binary. A placement dispatch to it returned a fully successful-looking ack:

{"capability":"spawn:claude","node":"sf-mini","attempts":1,"queued":false}

ps over ssh on sf-mini showed zero processes. Two dispatched agents never existed. The only way this was caught was by DM-probing the agent name afterward and getting "Agent not found" — there is no failure event, no terminal state, nothing to poll that would surface this on its own.

The control arm. sf-mini was subsequently repaired and, on the same node with the same capability advertisement, now spawns correctly with ps evidence of the real process. Same node, same advertised capability, opposite outcome — the only variable that changed is the broker build. That's what makes this a defect in the control plane's truthfulness, not a one-off flake.

The requester-side mechanism. This is the line that turns "capability advertisement is not a usable placement signal" from a conclusion into something provable: packages/sdk/src/messaging/relaycast.ts:648-659 (verified against AgentWorkforce/relay origin/main, current as of this comment):

const ack = await this.commands.invoke(actionName, actionInput);
return {
  ...ack,
  node: decision.node,
  placement: {
    capability,
    node: decision.node.name,
    ...(repo ? { repo } : {}),
    attempts,
    queued,
  },
};

this.commands.invoke(...) resolves to a RelayActionInvocationAck (packages/sdk/src/messaging/types.ts:516) — the async dispatch handle the engine returns the instant it forwards the invocation to a node (invocationId, dispatchedNodeId, a status string). It is not an ActionResult (packages/sdk/src/actions/types.ts:133, the { ok, output, error } shape a handler actually produces) and placement.spawn never reads one back. So placement.spawn's return value is built entirely from "the engine successfully forwarded this to a node" — a node that launches nothing is indistinguishable, at the requester, from a node that launches successfully. That holds regardless of what (or whether) the node itself ever replies.

Relation to #1430 / #1431. Adjacent but distinct, not overlapping. #1431 (once merged) makes a modern node report spawn_failed honestly instead of trusting bare registry presence — it fixes the node's own bookkeeping. It does not help here: sf-mini on broker 3.0.0 is too old to run that confirmation path at all, so there is no failure for #1431's fix to report. #1430/#1431 closes "the node lies about success." This issue is "the node can't tell you anything, and the requester never asks."