Skip to content

Cross-node --mode drive is unproven: prove it end to end, and support multiple concurrent drivers #1535

Description

@khaliqgant

Goal

Cross-node --mode drive is not proven. --mode view was proven on 2026-08-16 (relay#1521, broker 11.6.7, live VT screens both directions between chief-broker/sf-mini/finn-mini). Drive has never completed once. This issue owns proving it end to end, and fixing whatever blocks it.

Khaliq's product ruling, stated 2026-08-16: multi-driver must be supported — several people driving one agent at once is the intended behaviour, not an error case.

The observed failure

$ agent-relay node agent attach --node sf-mini relayfile-adapters-263-index-labels-fix --mode drive
Error: could not switch 'relayfile-adapters-263-index-labels-fix' to auto_inject mode: [object Object]
Error: cli-exit:1

A plain view attach against the same target in the same window returned to the prompt silently — no screen, no error, exit status unread.

The Cloud dashboard separately renders Terminal sessions unavailable: internal.

What is verified (read on origin/main, not a local checkout)

  1. A drive-exclusivity lock exists. relaycast-cloud packages/relaycast/src/durable-objects/node.ts:560-566:

    if (mode !== 'view' && activeSessions.some((s) => s.agent === agent && s.mode !== 'view')) {
      return Response.json(
        { ok: false, error: { code: 'drive_in_use', message: `Agent '${agent}' already has an active driver` } },
        { status: 409 },
      );
    }

    This is the line that has to change for multi-driver.

  2. It is the only error on that handler shaped as an object. Every sibling returns a plain string — 'invalid terminal session' (:534), 'terminal session limit reached' (:558). That asymmetry is why an object-vs-string formatting bug would surface only on the drive path.

  3. Sessions are swept lazily, so the lock self-clears. TERMINAL_SESSION_TTL_MS = 10 * 60_000 (:141); expired sessions are deleted on the next admission attempt (:539-548). MAX_TERMINAL_SESSIONS = 32 (:143). Nothing needs unsticking by hand.

  4. The CLI's error formatter is NOT the cause — this falsifies the obvious theory. mapBrokerSdkFailure (relay packages/cli/src/cli/lib/attach-broker.ts:36) routes through describeError (packages/cli/src/cli/lib/describe-error.ts:43), whose contract is literally "Never [object Object]" and which recurses into a nested error object. Executed against the exact drive_in_use payload in the installed 11.6.3 build, it returns:

    Agent 'x' already has an active driver (code drive_in_use) (status 409)
    

    So the [object Object] the operator saw did not come from this path. Some other formatter, or some other payload shape, produced it. This is the first thing to find.

  5. The message-injection path to a stranded driver is healthy. The target agent read a DM at 11:03:46Z with a read receipt (not a messageId) during the failure window. Whatever is broken, it is not inbound delivery.

What is NOT established — do not assume any of it

  • The "leaked session" chain is unproven and the code argues against it. runDriveSession (packages/cli/src/cli/lib/attach-drive.ts:1476) performs the delivery-mode flip before creating any terminal session. If the local path is representative, aborting at the flip leaves no session to leak — so "the first attempt leaked a session that blocks all later ones" needs the --node path read directly before anyone believes it.
  • Which binary produced the error. agent-relay on PATH is 11.6.3; ~/.agentworkforce/relay/bin/agent-relay is 11.6.7. An operator shell runs the former, the fleet runs the latter. The reported failure may already be fixed. Establish the build before debugging the source.
  • Whether the node is the variable at all. The failing target is a codex harness on sf-mini; the known-attachable control (attach-target-0816, finn-mini) is claude. Harness type and node are confounded in every observation so far.

Definition of done

  1. A real cross-node drive session: keystrokes typed on one machine reach an agent on another and its screen updates. Evidence is the recipient-side transcript or screen, never a sender-side receipt or exit code.
  2. Multi-driver works, per Khaliq's ruling — two drivers on one agent concurrently.
  3. The input-arbitration question is answered in writing before the if is deleted. Two drivers typing into one PTY is an unarbitrated byte stream. Last-writer-wins, a floor/token, or per-driver echo — this is a design call that must be specified, not defaulted into by removing a guard. State it, then implement it.
  4. [object Object] and internal are both replaced by the real upstream text, with a test that fails on the opaque form. Note the existing cloud test asserts a real string (packages/web/app/dashboard/chief/organization-panel.test.tsx:568 expects "Terminal sessions unavailable: Relaycast returned 503") while organization-tree.tsx:396 interpolates whatever arrives — so the current test cannot catch internal.
  5. The silent view-mode return is explained, with the harness-vs-node confound separated by a control arm.

Method

Two commands, run as a pair, settle the confound before any code is read:

agent-relay node agent attach --node sf-mini   <codex-target>       --mode view; echo "exit=$?"
agent-relay node agent attach --node finn-mini attach-target-0816   --mode view; echo "exit=$?"

Capture the exit codes without a pipe — head and pipelines report the wrong status and have manufactured phantom findings here before.

Cross-repo

Fixes likely land in relaycast-cloud (the lock and the error shape) and cloud (the dashboard string), with the proof driven from relay. Open the PRs in the owning repos and link them here.

mergePolicy: never. Open PRs, stop at review. Khaliq owns every merge.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions