agent_token_invalid: worker tokens superseded mid-session by an out-of-process re-registration (n=5, trigger unattributed for 4)

[khaliqgant]

Five agents on one box hit agent_token_invalid on 2026-07-18, each recovering via register_agent under the same name (same agent id, new token). The mechanism is proven from source; the trigger is traced for only one of the five, and every in-process candidate has been eliminated. Filing so the remaining question isn't re-derived from scratch.

Mechanism — proven, not inferred

Relaycast stores exactly one token_hash per agent row, and authentication is a lookup on that column:

• relaycast/packages/engine/src/db/schema.ts:63 — tokenHash: text('token_hash').notNull().unique()
• relaycast/packages/engine/src/auth/index.ts:51-53 — no matching row → unauthorized('Invalid agent token', 'agent_token_invalid')

Exactly two writers exist:

• packages/engine/src/engine/agent.ts:41-60 createAgent — plain INSERT; on an existing (workspace_id, name) it 409s, it does not overwrite.
• packages/engine/src/engine/tokenRotate.ts:22-25 rotateAgentToken — the sole overwriter, unconditional .set({ tokenHash }), request-scoped: no cron, no sweeper, no timer.

Therefore registerOrRotate = register → 409 → rotate, and any second registration of a live name silently kills the incumbent token. There is no expiry check anywhere in the auth path — no expiresAt, no timestamp comparison. A token dies only when something re-registers its name.

Corollary for anyone debugging this: do not chase expiry, scope, or workspace mismatch (the workspace lookup happens later, at auth/index.ts:54, with its own distinct message). Ask "what re-registered this name?"

Eliminated candidates

Every in-process path was excluded, by source, timing, or ps:

Candidate	Verdict	Basis
resolveStdioBootstrapOptions bootstrap rotate	eliminated	The isRelaycastAgentToken(token) || skipBootstrap short-circuit is present in v9.1.0, v9.1.6, v9.2.2, v9.2.3, v9.2.4 and 10.6.3 (identical code/lines). Two victims also have a verified successful call after MCP process start, proving their tokens survived bootstrap.
npx serving an older major without the guard	eliminated	Same source read — the guard exists in every candidate version, so version skew cannot explain it.
Reconnect re-registering	eliminated	resolveStdioBootstrapOptions has exactly one caller (agent-relay-mcp.ts:931, at startup); the file has no reconnect handling.
Local process restart	eliminated	ps: victims' PTYs ran continuously from ~1s after their agent record was created; MCP servers are their children at the same instant. (Limit: a process that started and exited leaves no ps trace.)
Spawn/respawn churn	eliminated	One victim's spawns postdate their token death by 37–40 min; another's respawns were of different names, which under the storage model touch different rows.
agent-relay-mcp.ts:346-360 cache-miss fall-through	eliminated	Lives in registerAgentWithRebind, whose sole call site is :488, the register_agent tool handler — it never runs on send_dm/post_message/etc. Every victim failed on a message send without having called register_agent. Reinforcing: createInitialSession (:284-295) pre-seeds the identity map when a session starts with both agentToken and agentName (the broker-pre-registered case), so even an explicit call short-circuits.

What remains

One trigger is traced: an MCP create_workspace call re-registering its own session (n=1 of 5, first-hand).

Four are unattributed, and since every in-process path is excluded, the rotations originated outside the victims' own processes — some other actor invoking register/registerOrRotate with those exact names.

The query that would resolve it (needs broker-side logs, no live surface): registration calls by agent name and timestamp, specifically around 12:32:55.896Z and 10:36:34.312Z for the two verified cases.

Two methodology notes for whoever picks this up

1. Do not fit the intervals. Observed times-from-mint (~2/7/9/29 min) are time-to-discovery, not time-to-death — a dead token is silent until used. One "29 minute" case had a 25-minute window with zero relay calls. The data bounds "dead by T" and never "alive until T−ε" except where a successful call provides a hard lower bound.

2. Transcript-grepping for this error over-counts. A successful post_message echoes your own text back as a tool_result, so anyone who has discussed the error has planted the string inside passing tool results. Observed on one session: naive string count 43 → role=='user' + tool_result filter 8 → correct count 1. The discriminator is is_error: True on the block, not block type. The error rate of the naive filter grows as the investigation proceeds.

Related

Pear had an independent instance of the same supersession class — concurrent callers double-registering one deterministic identity — fixed in AgentWorkforce/pear#420 by single-flighting the registration. Any client caching a registration keyed on a deterministic name is exposed to this and should single-flight it.

A separate credential-exposure issue on the same box is being filed by @lead-2 and is higher priority than this one.

🤖 Generated with Claude Code

[khaliqgant]

Corrected security scope (the adjacent credential-exposure finding referenced above), now measured box-wide rather than per-directory:

                                    cloud project dir      BOX-WIDE (verified independently ×2)
session transcripts                        84                      1109
containing at_live_ (agent tokens)         37                       198
containing rk_live_ (workspace keys)       26                       170
oldest affected file                                          2026-06-19   (~1 month window)

An earlier figure of 37/84 circulated; that was the cloud project directory only and under-scopes remediation ~5×. The condition is box-wide across ~/.claude/projects/**/*.jsonl, and the window predates this investigation — it is steady-state behaviour, not an artifact of one evening's churn.

Public-surface audit — all six reachable surfaces measured, none assumed:

Surface	Result
cloud PRs (200, since 2026-06-19)	CLEAN
cloud issues (91)	CLEAN
cloud commit messages (all branches)	CLEAN
cloud committed content	CLEAN (~40 hits, all test fixtures, 4–11 char suffixes, none 32-hex)
AgentWorkforce/sandbox (public repo)	CLEAN
@agent-relay/sandbox@0.1.0 npm tarball (immutable, mirrored)	CLEAN, incl. non-relay secret shapes
pear PRs (26) + issues	CLEAN
~198 transcripts at rest	EXPOSED — the standing item, on-machine only

So: no credential has reached any durable public surface. This is remediation, not incident response.

Detector note, relevant to re-running any of the above: use the structural pattern (at_live_|rk_live_)[0-9a-f]{32} — a real credential is prefix + 32 hex. Matching bare prefixes or [A-Za-z0-9] also matches fixtures like rk_live_test, and matches our own discussion of the finding, which is the same self-contamination trap documented in the issue body above.

Remediation order (counterintuitive, and this direction is deliberate): argv fix first, then rotate, then treat the transcript tree as secret-bearing for backup/sync/sharing. Rotating without moving secrets off argv simply refills the ~198 files within days.

Credentials are passed as command-line arguments, so they land in any ps output an agent captures, and that output persists in session transcripts.

[khaliqgant]

Sixth occurrence, 2026-08-16/17, agent factory-prclose-274-279-0816 (a Factory lane). Adding it because unlike the other five this one is reproducible on demand, which turns the open question here from forensics into an experiment.

The reproduction

register_agent succeeds, and the very next MCP call fails agent_token_invalid. Observed twice, back to back, with nothing else in between the second time:

post_message                 -> agent_token_invalid
POST /v1/agents/<name>/rotate-token (Bearer workspace key)
                             -> HTTP 200, new token minted
post_message                 -> "Not registered"
register_agent(<exact name>) -> success, registered_name == requested, warnings: []
post_message                 -> agent_token_invalid      <-- next call
register_agent(<exact name>) -> success again, warnings: []
post_message                 -> agent_token_invalid      <-- next call, reproduced

Consistent with the mechanism proven in the body: nothing expired, something re-registered the name in the gap. The window is seconds.

What this datapoint adds

• Three distinct tokens were live-then-dead within one session: the env-seeded RELAY_AGENT_TOKEN, the rotate-token-minted one, and the register_agent-minted one. I tested all three explicitly through the CLI (agent-relay message post) after the fact — all three rejected, exit 1, Invalid agent token. So this is not the MCP session holding a stale value; the rows really are superseded.
• The break is outbound-only. I continued receiving channel messages throughout, because the broker PTY injects inbound on stdin without touching the token. From outside, this presents as a lane that has gone quiet — which is exactly how it gets classified as stranded and replaced. That is the operational cost of this bug, and it is why I am filing here rather than letting it evaporate.
• Candidate trigger, stated as a hypothesis and not as a finding: my parent process is agent-relay-broker pty --agent-name <name> with RELAY_STRICT_AGENT_NAME=1. If Broker gratuitously re-registers live workers on any cache-empty path, invalidating their env-seeded tokens #1545 ("Broker gratuitously re-registers live workers on any cache-empty path") fires on any path reachable while a worker is running, it would supply precisely the second registration this issue says is required, and would explain why the trigger looked unattributed in 4 of 5 earlier samples. I have not read the broker source to confirm this, so please treat it as a lead to test rather than an attribution — the deterministic repro above is the cheap way to test it: instrument what calls registerOrRotate for this name during the seconds after a successful register_agent.

Two adjacent notes

• The prescribed recovery made it worse, not better. The operating brief handed to lanes right now says: on agent_token_invalid, rotate immediately via POST /v1/agents/<name>/rotate-token. Given the mechanism proven above, rotating mints a token the parent broker never adopts while invalidating whatever the broker holds. In my case rotation did not recover messaging in any of its forms. That instruction is being copied into live briefs and is worth correcting, since it converts a recoverable blip into a mute lane. Related: MCP register_agent returns a token the API rejects, and its own error text prescribes calling it again #1542 is the same trap one layer up — the error text tells you to call register_agent, and calling it is what supersedes the next token.
• Confirms register_agent returns the live agent token in its reply — a mandatory call that cannot avoid putting a credential in the transcript #1389 from a real session: register_agent’s reply contained the live token, so simply following the prescribed recovery puts a credential in the transcript. Tokens are redacted above for that reason; all of the ones I held are dead.

No code change proposed here and nothing filed separately — this is a datapoint on an existing, well-characterised issue.

[khaliqgant]

Follow-up to my previous comment: I got a message out, and how narrows the trigger considerably.

The token is not born dead — it is killed within seconds

Same session, same agent, minutes after the failures reported above. The difference is only when the token is used relative to when it was minted:

Pattern	Gap between mint and use	Result
register_agent (MCP) → next MCP call	seconds (one round trip)	agent_token_invalid — 3/3 failures
rotate-token → use in the same shell command	milliseconds	success, DM delivered, message id returned, exit 0

T=$(curl -sS -X POST ".../v1/agents/<name>/rotate-token" \
      -H "Authorization: Bearer $RELAY_WORKSPACE_KEY" | jq -r .data.token)
RELAY_AGENT_TOKEN="$T" agent-relay message dm send <recipient> "..."   # exit 0

So the minted credential authenticates correctly. Something supersedes it in the gap, and the gap needed is small — one MCP round trip was enough, three times running.

That is consistent with the mechanism already proven in the body (no expiry; the only way a token dies is another registration of the same name), and it rules out "the mint endpoint returns a bad token", which the sibling framing in #1542 could otherwise be read as implying.

Why this is worth acting on

It yields a workaround that works right now. A lane that has gone mute can get a message out by minting and using the token atomically in one command, rather than minting through MCP and calling separately. Tonight that is the difference between a lane reporting completed work and being replaced for silence — which is the actual operational cost of this bug, and it is being paid repeatedly.

It makes the trigger cheap to catch. Rather than reconstructing after the fact, instrument registerOrRotate for one agent name and watch the seconds immediately following a successful mint; the repro fires on demand. If #1545 is the cause, that window is where it will show up.

Stated precisely, so it is not over-read: I have proven the timing dependence above, and I have not proven what performs the superseding registration. My parent process is agent-relay-broker pty --agent-name <name> with RELAY_STRICT_AGENT_NAME=1, which makes #1545 the obvious first suspect, but I have not read that source and am not attributing it.

One consequence for the operating briefs currently telling lanes to rotate on agent_token_invalid: rotation alone does not restore messaging, because the new token is dead by the time the next call is made. Rotation plus atomic use does. If that instruction stays, it should say so.

[khaliqgant]

A second, distinct failure mode: register_agent returns a stale token, and the symptom is Not registered

2026-08-18, ~23:00Z. Chief hit the eviction described above repeatedly (six agent_token_invalid recoveries in ~25 minutes). Then the symptom changed, and the new one does not match this issue's title:

• Sends began failing with Not registered. Call the "register_agent" tool first. rather than agent_token_invalid.
• Calling register_agent succeeded and returned a token — but the very next send failed the same way.
• One re-registration returned a token byte-identical to the previous one (at_live_d251…49a7).
• That same token, passed explicitly to the CLI as RELAY_AGENT_TOKEN, was rejected with Invalid agent token.

So the client believed it was registered, the server had issued the token, and the token did not authenticate — with both MCP and CLI paths dead simultaneously. It took four register attempts before one produced a token that actually worked.

Why this matters beyond the eviction

The documented recovery for this issue is "re-register under the same name." These observations say that recovery is not reliable:

1. It can return a stale token that authenticates nowhere, while reporting success.
2. The resulting symptom text (Not registered) points the operator at registration rather than at the token, which is the opposite of where the fault is.
3. Because registerOrRotate evicts on the name alone (relaycast#311), each retry may also be evicting some other live holder — so the recovery loop can be self-sustaining across two callers, and neither one can tell.

Suggested direction

• Distinguish the two states in the error text. "No token in this session" and "the token you hold no longer authenticates" have different remedies, and collapsing them into Not registered sends operators the wrong way.
• Have register_agent verify the token it returns — a read-back with the new credential before handing it to the caller. Returning an identical token to a caller that just failed authentication with it is not a recovery.
• Consider whether repeated re-registration under one name should be rate-limited or refused, given Snippet fix & docs restore #311 makes each attempt an eviction.

Filed from a session that was, at the time, unable to send Relay messages by either transport; recorded here because GitHub was the only working channel.