From 7fc22ed0ecbb97f990af25641e8a0d2e07315867 Mon Sep 17 00:00:00 2001 From: Khaliq Date: Thu, 6 Aug 2026 12:57:41 +0200 Subject: [PATCH 1/4] docs(orchestrating-agent-relay): fleet node enrollment + availability diagnostics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the two things the skill's fleet section was missing, both learned while recovering a Mac mini fleet node after a reboot. "Is the node actually available?" — online is not the same as available for placement. Documents checking the spawn:* capability list rather than the status field, the two listing traps (fleet nodes hides offline records; output truncates at 64KB through a pipe), and an end-to-end placement proof that spawns from a different machine and verifies the process on the target host. "Enrolling a new machine as a fleet node" — the mint/redeem API flow (POST /api/v1/fleet/enrollment-tokens then /api/v1/fleet/register), linking cloud/dev-stack/fleet-node-bootstrap/README.md as authoritative rather than duplicating it. Calls out the mandatory reaper guard: node up kills every broker whose CWD is the resolved project root, which is relay#1328. Plus five Common Mistakes rows for the CLI behaviours that cost real time, including node agent subcommands ignoring --state-dir and AGENT_RELAY_DATA_DIR (filed separately as relay#1446). Co-Authored-By: Claude Opus 5 --- README.md | 2 +- prpm.json | 2 +- skills/orchestrating-agent-relay/SKILL.md | 68 +++++++++++++++++++++++ 3 files changed, 70 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 5d385fb..99a0441 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ Package metadata lives in [prpm.json](prpm.json). The repo currently publishes ` | [writing-agent-relay-workflows](skills/writing-agent-relay-workflows/SKILL.md) | 1.6.18 | Build multi-agent workflows with WorkflowBuilder, DAG dependencies, Relayfile-backed Slack human assistance, integration subscriptions, waitFor gates, review-depth review/fix loops, channels, and chat-native coordination recipes. | | [setting-up-relayfile](skills/setting-up-relayfile/SKILL.md) | 1.1.1 | Set up Relayfile mounts and writeback for provider files through local filesystem access. | | [using-agent-relay](skills/using-agent-relay/SKILL.md) | 1.4.0 | Participant-side MCP reference for a **registered** relay agent (spawned worker / registered lead): messaging, channels, threads, reactions, search, inbox, actions, and worker spawn/release. Counterpart to `orchestrating-agent-relay`. | -| [orchestrating-agent-relay](skills/orchestrating-agent-relay/SKILL.md) | 2.2.0 | The canonical way to run agent-relay: self-bootstrap the broker (`agent-relay node up`) and autonomously spawn, monitor, and coordinate a worker team over the relay MCP without human intervention. | +| [orchestrating-agent-relay](skills/orchestrating-agent-relay/SKILL.md) | 2.3.0 | The canonical way to run agent-relay: self-bootstrap the broker (`agent-relay node up`) and autonomously spawn, monitor, and coordinate a worker team over the relay MCP without human intervention. | | [relay-80-100-workflow](skills/relay-80-100-workflow/SKILL.md) | 1.0.8 | Author workflows that close the 80-to-100 validation gap with repair-aware test, verify, review-depth review/fix with test hardening, and commit gates. | | [review-fix-signoff-loop](skills/review-fix-signoff-loop/SKILL.md) | 1.0.2 | Loop review, repair, validation, and fresh-context dual-agent signoff until independent reviewers both satisfy the verdict contract. | | [trigger-autocomplete-catalog](skills/trigger-autocomplete-catalog/SKILL.md) | 1.0.0 | Enforce webhook/event trigger autocomplete coverage through KNOWN_TRIGGER_CATALOG in @relayfile/adapter-core. | diff --git a/prpm.json b/prpm.json index 0339e99..b448b1b 100644 --- a/prpm.json +++ b/prpm.json @@ -89,7 +89,7 @@ }, { "name": "orchestrating-agent-relay", - "version": "2.2.0", + "version": "2.3.0", "description": "The canonical way to run agent-relay - self-bootstrap the local broker and autonomously spawn, monitor, and coordinate a team of worker agents without human intervention. Covers infrastructure startup, agent spawning, lifecycle monitoring, message-based reading via the relay MCP, and team coordination.", "format": "claude", "subtype": "skill", diff --git a/skills/orchestrating-agent-relay/SKILL.md b/skills/orchestrating-agent-relay/SKILL.md index 6eeb350..fc2cd64 100644 --- a/skills/orchestrating-agent-relay/SKILL.md +++ b/skills/orchestrating-agent-relay/SKILL.md @@ -447,6 +447,69 @@ From the relay MCP, `query_nodes` finds nodes by capability or name and `spawn` invokes the fleet spawn action — the engine places it on an eligible node (or a named `target_node`). +### Is the node actually available? + +`online` is not the same as **available for placement**. A node can be live and +still never receive a spawn. Check the capability list, not the status field: + +```bash +# `fleet nodes` HIDES offline/non-fleet records by default (it hid 385 of 390 +# on a real workspace), so a node you are looking for may simply not be printed. +agent-relay fleet nodes --all > /tmp/nodes.raw # redirect: output truncates at 64KB through a pipe +python3 -c 'import json;raw=open("/tmp/nodes.raw").read();d=json.loads(raw[raw.find("{"):]); +[print(n["name"], n["status"], n.get("live"), [c["name"] for c in n.get("capabilities",[])]) for n in d["nodes"]]' +``` + +A healthy placement target carries `spawn:claude`, `spawn:codex`, `spawn:gemini`, +`spawn:opencode`, `release`, `relay:delivery-cursor-v1`. A record with no +`spawn:*` capability is registered but cannot be placed on. + +Prove placement end to end rather than trusting the roster — spawn **from a +different machine** so you are testing placement and not a local spawn, confirm +`dispatchedNodeId` matches the target's node id, then verify on the target host +that the process actually exists, and release: + +```bash +RELAY_AGENT_TOKEN= agent-relay fleet spawn claude \ + --name placement-proof --node --channel general --task "Run hostname -s and reply with its output only." +# on the target host: +pgrep -fl placement-proof # broker pty + CLI process must be present +agent-relay node agent release placement-proof +``` + +### Enrolling a new machine as a fleet node + +Enrollment is a two-step API flow — mint on the control plane, redeem from the +node: + +1. `POST /api/v1/fleet/enrollment-tokens` → single-use `ocl_node_enr_…` +2. `POST /api/v1/fleet/register`, from the machine being enrolled + +The node-side script and the authoritative reference is +`cloud/dev-stack/fleet-node-bootstrap/README.md` (supports Daytona, CF +Containers, the local dev-stack runner, and Mac minis). Inputs are env-only so +secrets never reach `ps` argv: + +```bash +RELAY_ENROLLMENT_TOKEN='ocl_node_enr_…' \ +RELAY_ENROLLMENT_URL='https:///api/v1/fleet/register' \ +RELAY_NODE_NAME='' sandbox-node-bootstrap.sh enroll +``` + +> **Never skip `sandbox-node-bootstrap.sh preflight` on a machine that already +> runs brokers.** `agent-relay node up` calls `killOrphanedBrokerProcesses(projectRoot)` +> at startup, terminating **every broker whose CWD is that root**. `findProjectRoot()` +> walks up for markers (`.git`, `package.json`, `.agentworkforce/relay`), so a +> `$HOME`-rooted workdir resolves `projectRoot=$HOME` and reaps every +> `$HOME`-rooted broker. That is relay#1328 — a real incident that killed +> production brokers on a shared machine. Pin `AGENT_RELAY_PROJECT` to a unique +> per-instance dir and drop a physical `.agentworkforce/relay` marker there. + +Enrollment persists to `~/.agentworkforce/relay/fleet-enrollments.json` (holds a +live `nt_live_…` node token — never echo this file). Once enrolled, a +`com.agentrelay.fleet-node` LaunchAgent brings the node back automatically across +reboots; a rebooted machine does **not** need re-enrolling. + ## Common Mistakes | Mistake | Fix | @@ -468,6 +531,11 @@ named `target_node`). | `node status` says running but `node agent list`/MCP calls return empty or `Failed to query broker session` | The CLI is dialing a **stale/wrong broker** — leftover `.agentworkforce/relay/connection.json` from a prior run on an old port, or a second broker process. `ps aux \| grep -c '[a]gent-relay-broker'` (>1 ⇒ kill extras), compare `.agentworkforce/relay/connection.json` to the actual listening port, then `agent-relay node down --force`, delete `.agentworkforce/relay/`, `agent-relay node up` clean | | `Invalid agent token` while broker + workers keep working | The orchestrator shell has an **unresolved `${RELAY_WORKSPACE_KEY}`-style template** being used as a literal key (broker/workers hold real tokens). Ensure the workspace key/token is actually resolved in the orchestrator env | | New worker appears in `node agent list` but no ACK yet | Expected — appearing means process up (~5s); the CLI cold-starts for another 30–45s before its first ACK DM. Wait ≥60s before troubleshooting a fresh worker | +| A node you know exists is missing from `agent-relay fleet nodes` | The default view hides offline/non-fleet records (385 of 390 hidden on a real workspace) — and the node may be present but past the cut. Use `agent-relay fleet nodes --all` | +| `fleet nodes` JSON fails to parse mid-object | Output **truncates at 64KB** through a pipe. Redirect to a file first (`agent-relay fleet nodes --all > /tmp/nodes.raw`) and parse the file, never the pipe | +| `node agent list`/`release` says `No running broker found (…/relay/connection.json does not exist)` while the fleet node is clearly running | These subcommands only read the **default** `~/.agentworkforce/relay/connection.json`. They reject `--state-dir` and ignore `AGENT_RELAY_DATA_DIR`, so a node started with `--state-dir` (as the `com.agentrelay.fleet-node` LaunchAgent does) is unreachable. Temporarily symlink the node's real connection file to the default path, run the command, then remove the symlink | +| Targeted `fleet spawn` fails with `Targeted Fleet spawn requires an agent token` | Pass `--token` or set `RELAY_AGENT_TOKEN`; mint one with `agent-relay agent register --type system` (capture it without echoing). `--task` is also mandatory and the error only surfaces one problem at a time | +| Node shows `online` but never receives a spawn | `online` ≠ available. Check `capabilities` contains `spawn:*` — a record can be live with no spawn capacity. Confirm with a throwaway targeted spawn, verified by `pgrep` **on the target host**, then release | | Harness blocks `sleep 25; check_inbox ...` | Bare foreground `sleep` wait loops are disallowed in harnessed environments. Run the poll loop with `run_in_background` (or Monitor + until-loop); the inline `sleep` snippets show logic only | | Worker self-removed; can't send review fixes | Instruct workers not to self-remove until told. If already gone, spawn a fresh worker and re-inject branch + commit SHA + full verdict (see Multi-Round Review Loops) | | Worker died silently; loop hangs | Inbox polling fires on messages only. Poll `agent-relay node agent list` for liveness and set a wall-clock fallback (~30 min ScheduleWakeup) | From c7927c67788135251561ea0ad8486a1b4b7a110f Mon Sep 17 00:00:00 2001 From: Khaliq Date: Thu, 6 Aug 2026 13:12:07 +0200 Subject: [PATCH 2/4] docs(orchestrating-agent-relay): address PR #93 review Four review findings, all valid and all fixed. Capability requirement was overstated: a node advertising only spawn:claude is a valid target for `fleet spawn claude`. Now requires the spawn: matching the request, with release and relay:delivery-cursor-v1 called out as separate lifecycle capabilities. Tokens no longer appear inline. Both the placement-proof and enrollment examples now populate the token with a silent `read` plus an unset trap, so it stays out of shell history as well as out of ps argv. The enrollment prose claimed env-only inputs while the example contradicted it. Release path was self-contradictory: the proof used `node agent release` on the target host while the same document states that subcommand cannot reach a --state-dir broker. Now uses the control-plane `agent-relay fleet release`, which needs no local connection file. The symlink workaround was actively unsafe. It used `ln -sf`, which destroys a pre-existing connection.json -- and that file is a real regular file on at least one host in this fleet. Replaced with a guarded form that refuses when the path exists and removes only a link it created. Verified both branches: refuses and preserves on the host with the real file, links/runs/cleans on the host without. Bootstrap assets are not shipped with this skill, so the bare relative path was unresolvable for anyone installing it. Now names the AgentWorkforce/cloud repo and states the access requirement. Skipped: SkillSpector npx-pinning warnings (lines 54/75/85/549) are pre-existing and outside this PR's diff. Co-Authored-By: Claude Opus 5 --- skills/orchestrating-agent-relay/SKILL.md | 53 ++++++++++++++++------- 1 file changed, 38 insertions(+), 15 deletions(-) diff --git a/skills/orchestrating-agent-relay/SKILL.md b/skills/orchestrating-agent-relay/SKILL.md index fc2cd64..e83fb7e 100644 --- a/skills/orchestrating-agent-relay/SKILL.md +++ b/skills/orchestrating-agent-relay/SKILL.md @@ -460,9 +460,12 @@ python3 -c 'import json;raw=open("/tmp/nodes.raw").read();d=json.loads(raw[raw.f [print(n["name"], n["status"], n.get("live"), [c["name"] for c in n.get("capabilities",[])]) for n in d["nodes"]]' ``` -A healthy placement target carries `spawn:claude`, `spawn:codex`, `spawn:gemini`, -`spawn:opencode`, `release`, `relay:delivery-cursor-v1`. A record with no -`spawn:*` capability is registered but cannot be placed on. +A placement target must carry the `spawn:` capability for the spawn +you are requesting — a node advertising only `spawn:claude` is a valid target for +`fleet spawn claude` and not for `fleet spawn codex`. `release` and +`relay:delivery-cursor-v1` are separate lifecycle capabilities, needed to manage +the worker once placed. A record with no `spawn:*` capability at all is +registered but cannot receive a spawn. Prove placement end to end rather than trusting the roster — spawn **from a different machine** so you are testing placement and not a local spawn, confirm @@ -470,11 +473,21 @@ different machine** so you are testing placement and not a local spawn, confirm that the process actually exists, and release: ```bash -RELAY_AGENT_TOKEN= agent-relay fleet spawn claude \ - --name placement-proof --node --channel general --task "Run hostname -s and reply with its output only." -# on the target host: -pgrep -fl placement-proof # broker pty + CLI process must be present -agent-relay node agent release placement-proof +# Read the token without leaving it in shell history or `ps` argv. +read -r -s -p 'Agent token: ' RELAY_AGENT_TOKEN; printf '\n' +export RELAY_AGENT_TOKEN +trap 'unset RELAY_AGENT_TOKEN' EXIT + +agent-relay fleet spawn claude \ + --name placement-proof --node --channel general \ + --task "Run hostname -s and reply with its output only." +# confirm dispatchedNodeId in the response matches 's id, then ON THE TARGET HOST: +pgrep -fl placement-proof # broker pty + CLI process must both be present + +# Release from the control plane — works regardless of how the node's broker was +# started. Do NOT use `node agent release` here: a fleet node started with +# --state-dir (as the LaunchAgent does) is unreachable from that subcommand. +agent-relay fleet release placement-proof ``` ### Enrolling a new machine as a fleet node @@ -485,15 +498,25 @@ node: 1. `POST /api/v1/fleet/enrollment-tokens` → single-use `ocl_node_enr_…` 2. `POST /api/v1/fleet/register`, from the machine being enrolled -The node-side script and the authoritative reference is -`cloud/dev-stack/fleet-node-bootstrap/README.md` (supports Daytona, CF -Containers, the local dev-stack runner, and Mac minis). Inputs are env-only so -secrets never reach `ps` argv: +The node-side script is `sandbox-node-bootstrap.sh`, with +`README.md` alongside it as the authoritative reference. Both live at +`dev-stack/fleet-node-bootstrap/` in the **`AgentWorkforce/cloud`** repository — +they are not shipped with this skill, so you need access to that repo to run an +enrollment. It supports Daytona, CF Containers, the local dev-stack runner, and +Mac minis. + +The script takes its inputs from the environment so secrets never reach `ps` +argv. Populate the token with a silent read so it does not land in shell history +either: ```bash -RELAY_ENROLLMENT_TOKEN='ocl_node_enr_…' \ +read -r -s -p 'Enrollment token: ' RELAY_ENROLLMENT_TOKEN; printf '\n' +trap 'unset RELAY_ENROLLMENT_TOKEN' EXIT + +RELAY_ENROLLMENT_TOKEN="$RELAY_ENROLLMENT_TOKEN" \ RELAY_ENROLLMENT_URL='https:///api/v1/fleet/register' \ -RELAY_NODE_NAME='' sandbox-node-bootstrap.sh enroll +RELAY_NODE_NAME='' \ + sandbox-node-bootstrap.sh enroll ``` > **Never skip `sandbox-node-bootstrap.sh preflight` on a machine that already @@ -533,7 +556,7 @@ reboots; a rebooted machine does **not** need re-enrolling. | New worker appears in `node agent list` but no ACK yet | Expected — appearing means process up (~5s); the CLI cold-starts for another 30–45s before its first ACK DM. Wait ≥60s before troubleshooting a fresh worker | | A node you know exists is missing from `agent-relay fleet nodes` | The default view hides offline/non-fleet records (385 of 390 hidden on a real workspace) — and the node may be present but past the cut. Use `agent-relay fleet nodes --all` | | `fleet nodes` JSON fails to parse mid-object | Output **truncates at 64KB** through a pipe. Redirect to a file first (`agent-relay fleet nodes --all > /tmp/nodes.raw`) and parse the file, never the pipe | -| `node agent list`/`release` says `No running broker found (…/relay/connection.json does not exist)` while the fleet node is clearly running | These subcommands only read the **default** `~/.agentworkforce/relay/connection.json`. They reject `--state-dir` and ignore `AGENT_RELAY_DATA_DIR`, so a node started with `--state-dir` (as the `com.agentrelay.fleet-node` LaunchAgent does) is unreachable. Temporarily symlink the node's real connection file to the default path, run the command, then remove the symlink | +| `node agent list`/`release` says `No running broker found (…/relay/connection.json does not exist)` while the fleet node is clearly running | These subcommands only read the **default** `~/.agentworkforce/relay/connection.json`. They reject `--state-dir` and ignore `AGENT_RELAY_DATA_DIR`, so a node started with `--state-dir` (as the `com.agentrelay.fleet-node` LaunchAgent does) is unreachable (`relay#1446`). **For fleet-spawned agents use the control plane instead — `agent-relay fleet release ` and `agent-relay fleet nodes` need no local connection file.** Only if you genuinely need a node-local subcommand, guard the symlink: refuse to proceed if the default path already exists (it is a real file on some hosts and `ln -sf` would destroy it), then remove only the link you created — `[ -e ~/.agentworkforce/relay/connection.json ] && echo "pre-existing, do not clobber" \|\| { ln -s /connection.json ~/.agentworkforce/relay/connection.json; agent-relay node agent list; [ -L ~/.agentworkforce/relay/connection.json ] && rm ~/.agentworkforce/relay/connection.json; }` | | Targeted `fleet spawn` fails with `Targeted Fleet spawn requires an agent token` | Pass `--token` or set `RELAY_AGENT_TOKEN`; mint one with `agent-relay agent register --type system` (capture it without echoing). `--task` is also mandatory and the error only surfaces one problem at a time | | Node shows `online` but never receives a spawn | `online` ≠ available. Check `capabilities` contains `spawn:*` — a record can be live with no spawn capacity. Confirm with a throwaway targeted spawn, verified by `pgrep` **on the target host**, then release | | Harness blocks `sleep 25; check_inbox ...` | Bare foreground `sleep` wait loops are disallowed in harnessed environments. Run the poll loop with `run_in_background` (or Monitor + until-loop); the inline `sleep` snippets show logic only | From 3532a5e1cc68d87dab0117ded0ed773f2e6ff1d7 Mon Sep 17 00:00:00 2001 From: Khaliq Date: Thu, 6 Aug 2026 13:31:58 +0200 Subject: [PATCH 3/4] docs(orchestrating-agent-relay): address second review round (PR #93) Proof recipe now demonstrates the check it promised. The prose told readers to spawn from a different machine and confirm dispatchedNodeId, then showed neither. Now numbered STEP 0-3: an explicit different-machine instruction, a parser that extracts dispatchedNodeId from the response (which carries a human-readable preamble before the JSON, so it parses from the first brace), the on-host pgrep, and the release. Adds a note that steps 1 and 2 are separate claims -- control plane records dispatch whether or not anything ran, which is exactly how a broken node looks healthy. Moves the --state-dir workaround out of the table cell into its own subsection. It had outgrown a single row, and a multi-line guarded procedure could not be written safely inside one. The subsection leads with the control-plane path and treats the symlink as the fallback it is. Adds mkdir -p for the parent directory, which may not exist on a freshly provisioned node, and states explicitly why ln -sf must never be used here. Table row shrinks to a pointer at that subsection. Validated, not assumed: the parser against a real response shape, and both guard branches on real hosts -- refuses and preserves on the host with a real connection file (untouched, still dated 2026-08-03), links/runs/cleans on the host without one. Co-Authored-By: Claude Opus 5 --- skills/orchestrating-agent-relay/SKILL.md | 58 +++++++++++++++++++++-- 1 file changed, 53 insertions(+), 5 deletions(-) diff --git a/skills/orchestrating-agent-relay/SKILL.md b/skills/orchestrating-agent-relay/SKILL.md index e83fb7e..0ac1a2e 100644 --- a/skills/orchestrating-agent-relay/SKILL.md +++ b/skills/orchestrating-agent-relay/SKILL.md @@ -473,6 +473,9 @@ different machine** so you are testing placement and not a local spawn, confirm that the process actually exists, and release: ```bash +# STEP 0 — run everything below from a machine OTHER than . Spawning on the +# same host you are testing proves nothing about placement. + # Read the token without leaving it in shell history or `ps` argv. read -r -s -p 'Agent token: ' RELAY_AGENT_TOKEN; printf '\n' export RELAY_AGENT_TOKEN @@ -480,16 +483,28 @@ trap 'unset RELAY_AGENT_TOKEN' EXIT agent-relay fleet spawn claude \ --name placement-proof --node --channel general \ - --task "Run hostname -s and reply with its output only." -# confirm dispatchedNodeId in the response matches 's id, then ON THE TARGET HOST: + --task "Run hostname -s and reply with its output only." > /tmp/spawn.json + +# STEP 1 — the control plane says it dispatched where you asked. The response has a +# human-readable preamble before the JSON, so parse from the first brace. +python3 -c 'import json;raw=open("/tmp/spawn.json").read();i=json.loads(raw[raw.find("{"):])["invocation"]; +print("dispatched to:", i["dispatchedNodeId"], "| name:", i["node"]["name"], "| status:", i["status"])' +# That id must equal 's id in `agent-relay fleet nodes --all`. A mismatch means +# placement ignored your target; a match still proves nothing about execution — hence step 2. + +# STEP 2 — the process actually exists. Run this ON THE TARGET HOST. pgrep -fl placement-proof # broker pty + CLI process must both be present -# Release from the control plane — works regardless of how the node's broker was -# started. Do NOT use `node agent release` here: a fleet node started with +# STEP 3 — release from the control plane. Works regardless of how the node's broker +# was started. Do NOT use `node agent release` here: a fleet node started with # --state-dir (as the LaunchAgent does) is unreachable from that subcommand. agent-relay fleet release placement-proof ``` +Steps 1 and 2 are separate claims. Step 1 alone is the mistake that makes a broken +node look healthy — dispatch is recorded by the control plane whether or not +anything ran. + ### Enrolling a new machine as a fleet node Enrollment is a two-step API flow — mint on the control plane, redeem from the @@ -533,6 +548,39 @@ live `nt_live_…` node token — never echo this file). Once enrolled, a `com.agentrelay.fleet-node` LaunchAgent brings the node back automatically across reboots; a rebooted machine does **not** need re-enrolling. +### Reaching a `--state-dir` broker + +`agent-relay node up --state-dir ` (how the `com.agentrelay.fleet-node` +LaunchAgent starts every fleet node) writes its connection file to +`/connection.json`. Every `node agent` subcommand except `attach` reads only +the **default** `~/.agentworkforce/relay/connection.json`, rejects `--state-dir`, +and ignores `AGENT_RELAY_DATA_DIR` — so those subcommands report +`No running broker found` against a perfectly healthy broker (`relay#1446`). + +**Prefer the control plane.** `agent-relay fleet nodes`, `fleet spawn` and +`fleet release` need no local connection file and work on any node regardless of +how its broker was started. Reach for the workaround below only for a node-local +subcommand that has no fleet equivalent. + +```bash +DEF=~/.agentworkforce/relay/connection.json +SD= # the --state-dir the broker was started with + +if [ -e "$DEF" ]; then + echo "REFUSING: $DEF already exists — on some hosts this is a real connection file" + echo "and clobbering it would break the default broker. Inspect it before proceeding." +else + mkdir -p "$(dirname "$DEF")" # may not exist yet on a freshly provisioned node + ln -s "$SD/connection.json" "$DEF" + agent-relay node agent list # ... or whichever node-local subcommand you need + [ -L "$DEF" ] && rm "$DEF" # remove ONLY a symlink, and only one we created +fi +``` + +Never use `ln -sf` here. The `-f` silently destroys a pre-existing connection file, +and that file is a real regular file on some hosts — not a stale leftover. Delete +this whole workaround once `relay#1446` lands rather than letting it outlive the bug. + ## Common Mistakes | Mistake | Fix | @@ -556,7 +604,7 @@ reboots; a rebooted machine does **not** need re-enrolling. | New worker appears in `node agent list` but no ACK yet | Expected — appearing means process up (~5s); the CLI cold-starts for another 30–45s before its first ACK DM. Wait ≥60s before troubleshooting a fresh worker | | A node you know exists is missing from `agent-relay fleet nodes` | The default view hides offline/non-fleet records (385 of 390 hidden on a real workspace) — and the node may be present but past the cut. Use `agent-relay fleet nodes --all` | | `fleet nodes` JSON fails to parse mid-object | Output **truncates at 64KB** through a pipe. Redirect to a file first (`agent-relay fleet nodes --all > /tmp/nodes.raw`) and parse the file, never the pipe | -| `node agent list`/`release` says `No running broker found (…/relay/connection.json does not exist)` while the fleet node is clearly running | These subcommands only read the **default** `~/.agentworkforce/relay/connection.json`. They reject `--state-dir` and ignore `AGENT_RELAY_DATA_DIR`, so a node started with `--state-dir` (as the `com.agentrelay.fleet-node` LaunchAgent does) is unreachable (`relay#1446`). **For fleet-spawned agents use the control plane instead — `agent-relay fleet release ` and `agent-relay fleet nodes` need no local connection file.** Only if you genuinely need a node-local subcommand, guard the symlink: refuse to proceed if the default path already exists (it is a real file on some hosts and `ln -sf` would destroy it), then remove only the link you created — `[ -e ~/.agentworkforce/relay/connection.json ] && echo "pre-existing, do not clobber" \|\| { ln -s /connection.json ~/.agentworkforce/relay/connection.json; agent-relay node agent list; [ -L ~/.agentworkforce/relay/connection.json ] && rm ~/.agentworkforce/relay/connection.json; }` | +| `node agent list`/`release` says `No running broker found (…/relay/connection.json does not exist)` while the fleet node is clearly running | The subcommand is reading the **default** connection path, not the broker's `--state-dir` one (`relay#1446`). Use the control-plane equivalent — `agent-relay fleet release ` / `fleet nodes` — which needs no local connection file. See [Reaching a `--state-dir` broker](#reaching-a---state-dir-broker) for the guarded workaround when only a node-local subcommand will do | | Targeted `fleet spawn` fails with `Targeted Fleet spawn requires an agent token` | Pass `--token` or set `RELAY_AGENT_TOKEN`; mint one with `agent-relay agent register --type system` (capture it without echoing). `--task` is also mandatory and the error only surfaces one problem at a time | | Node shows `online` but never receives a spawn | `online` ≠ available. Check `capabilities` contains `spawn:*` — a record can be live with no spawn capacity. Confirm with a throwaway targeted spawn, verified by `pgrep` **on the target host**, then release | | Harness blocks `sleep 25; check_inbox ...` | Bare foreground `sleep` wait loops are disallowed in harnessed environments. Run the poll loop with `run_in_background` (or Monitor + until-loop); the inline `sleep` snippets show logic only | From 1a0fb00b366526cd84c14680f507f8bb76f8c611 Mon Sep 17 00:00:00 2001 From: Khaliq Date: Thu, 6 Aug 2026 14:25:28 +0200 Subject: [PATCH 4/4] docs(orchestrating-agent-relay): address third review round (PR #93) Parsers no longer crash on unexpected output. Both the roster and STEP 1 probes used raw.find("{"), which assumes the CLI preamble contains no brace and that the call succeeded. Neither holds. Now they locate the first brace at start-of-line via regex and degrade to printing the raw output. The STEP 1 case mattered most: a traceback there exits non-zero and skips the STEP 3 release, leaking the running agent the recipe just spawned -- a diagnostic that leaks the thing it is diagnosing. The new form always exits 0. Roster parser now prints node id. STEP 1 told readers to compare dispatchedNodeId against the node's id while the roster command printed only name, so the mandated comparison was impossible. Also states that dispatchedNodeId is an id, not a name. Symlink guard now tests -L as well as -e. A dangling link is what an interrupted run leaves behind, and -e is false for one, so the guard passed and ln -s then failed with "File exists" while the subcommand silently never ran. Validated on real hosts, Python 3.9.6 (stock macOS): - failed-spawn response with a brace in the preamble: old parser raised JSONDecodeError, new one reports the failure and exits 0 - planted dangling symlink: [ -e ] false, [ -L ] true, new guard refuses - roster parser prints id=node_d4190c4c... for finn-mini, matching the dispatchedNodeId observed in a real spawn Co-Authored-By: Claude Opus 5 --- skills/orchestrating-agent-relay/SKILL.md | 52 ++++++++++++++++++----- 1 file changed, 42 insertions(+), 10 deletions(-) diff --git a/skills/orchestrating-agent-relay/SKILL.md b/skills/orchestrating-agent-relay/SKILL.md index 0ac1a2e..b7439e2 100644 --- a/skills/orchestrating-agent-relay/SKILL.md +++ b/skills/orchestrating-agent-relay/SKILL.md @@ -456,10 +456,21 @@ still never receive a spawn. Check the capability list, not the status field: # `fleet nodes` HIDES offline/non-fleet records by default (it hid 385 of 390 # on a real workspace), so a node you are looking for may simply not be printed. agent-relay fleet nodes --all > /tmp/nodes.raw # redirect: output truncates at 64KB through a pipe -python3 -c 'import json;raw=open("/tmp/nodes.raw").read();d=json.loads(raw[raw.find("{"):]); -[print(n["name"], n["status"], n.get("live"), [c["name"] for c in n.get("capabilities",[])]) for n in d["nodes"]]' +python3 - <<'PY' +import json, re +raw = open("/tmp/nodes.raw").read() +m = re.search(r"^\{", raw, re.M) # first brace at start of a line, not inside the preamble +if not m: + raise SystemExit("No JSON in output. Raw:\n" + raw[:500]) +for n in json.loads(raw[m.start():]).get("nodes", []): + caps = [c["name"] for c in n.get("capabilities", [])] + print(f'{n.get("name")} id={n.get("id")} {n.get("status")} live={n.get("live")} {caps}') +PY ``` +`id` is printed because that is the field you compare against in the placement +proof below — `dispatchedNodeId` is a node **id**, not a name. + A placement target must carry the `spawn:` capability for the spawn you are requesting — a node advertising only `spawn:claude` is a valid target for `fleet spawn claude` and not for `fleet spawn codex`. `release` and @@ -485,12 +496,29 @@ agent-relay fleet spawn claude \ --name placement-proof --node --channel general \ --task "Run hostname -s and reply with its output only." > /tmp/spawn.json -# STEP 1 — the control plane says it dispatched where you asked. The response has a -# human-readable preamble before the JSON, so parse from the first brace. -python3 -c 'import json;raw=open("/tmp/spawn.json").read();i=json.loads(raw[raw.find("{"):])["invocation"]; -print("dispatched to:", i["dispatchedNodeId"], "| name:", i["node"]["name"], "| status:", i["status"])' -# That id must equal 's id in `agent-relay fleet nodes --all`. A mismatch means -# placement ignored your target; a match still proves nothing about execution — hence step 2. +# STEP 1 — the control plane says it dispatched where you asked. The response carries +# a human-readable preamble before the JSON. Never abort here: a failed spawn is a +# result, and a traceback would skip the STEP 3 release and leak a running agent. +python3 - <<'PY' +import json, re +raw = open("/tmp/spawn.json").read() +m = re.search(r"^\{", raw, re.M) # first brace at start of a line +inv = None +if m: + try: + inv = json.loads(raw[m.start():]).get("invocation") + except ValueError: + pass +if not inv: + print("Spawn did not return an invocation — it likely failed. Raw output:\n" + raw) +else: + print("dispatched to:", inv.get("dispatchedNodeId"), + "| name:", (inv.get("node") or {}).get("name"), + "| status:", inv.get("status")) +PY +# `dispatchedNodeId` must equal 's `id` from the roster command above — it is an +# id (`node_…`), not a name. A mismatch means placement ignored your target; a match +# still proves nothing about execution, hence STEP 2. # STEP 2 — the process actually exists. Run this ON THE TARGET HOST. pgrep -fl placement-proof # broker pty + CLI process must both be present @@ -566,9 +594,13 @@ subcommand that has no fleet equivalent. DEF=~/.agentworkforce/relay/connection.json SD= # the --state-dir the broker was started with -if [ -e "$DEF" ]; then +# -e alone is FALSE for a dangling symlink, which is exactly what a previous run +# leaves behind if it died before its cleanup — so test -L as well, or `ln -s` +# fails with "File exists" and the subcommand silently never runs. +if [ -e "$DEF" ] || [ -L "$DEF" ]; then echo "REFUSING: $DEF already exists — on some hosts this is a real connection file" - echo "and clobbering it would break the default broker. Inspect it before proceeding." + echo "and clobbering it would break the default broker. If it is a dangling symlink" + echo "from an interrupted run, remove it; otherwise inspect it before proceeding." else mkdir -p "$(dirname "$DEF")" # may not exist yet on a freshly provisioned node ln -s "$SD/connection.json" "$DEF"