Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/skills/stuck-crewmate-recovery/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Recover a genuinely stuck remote mate only through `bin/fm-spawn.sh <id> --secon

Treat the digest's endpoint result as a presence signal, not proof that the task's work or validation run is gone.
Read the targeted current state with `bin/fm-crew-state.sh <id>` before deciding to relaunch.
A no-mistakes run matched to the crew's branch and current code remains authoritative when the endpoint is dead: handle a terminal or parked run through the normal lifecycle, and keep supervising an active run instead of creating a duplicate worker.
A no-mistakes run attributed to the crew by `bin/fm-crew-state.sh` remains authoritative when the endpoint is dead: handle a terminal or parked run through the normal lifecycle, and keep supervising an active run instead of creating a duplicate worker.

When no authoritative run accounts for the task, inspect only its recorded backend and worktree inventory.
Use `treehouse status` for treehouse-backed tmux, herdr, zellij, or cmux tasks, and use the recorded `orca_worktree_id=` and `terminal=` for Orca tasks.
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -356,7 +356,7 @@ Send the same worker one exact decision naming the decision key, step, action, a
Require the matching `resolved` event, forbid `--yes`, and require the worker to process every synchronous return until completion or a genuinely new escalation.
Resume fleet supervision immediately after the decision lands.

Judge validation by the current-code-matched run step through `bin/fm-crew-state.sh`, not by shell liveness or the last status event.
Judge validation by the current run step reported through `bin/fm-crew-state.sh`, not by shell liveness or the last status event.
Running, fixing, or CI states remain working; parked approval or fix-review states require the worker to follow the active gate help; passed or checks-passed is done; failed or cancelled is failed.
A worker hand-editing, committing, aborting, or restarting during an active validation run duplicates pipeline ownership outside the supersession sequence above; steer it back to the gate response flow.
The worker reports the PR when CI first becomes green rather than waiting for merge monitoring to finish.
Expand Down
94 changes: 33 additions & 61 deletions bin/fm-crew-state.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
# or blocked and the crew resumes (responds to the gate, the pipeline fixes, it
# re-validates), the log's last line stays stale. This helper never infers the
# current state from a tail of the log: it reads the authoritative source (a
# no-mistakes run-step attributed to this crew's branch and current code
# identity, else the pane busy-signature) and reconciles the possibly-stale log
# against it.
# no-mistakes run-step attributed to this crew's branch under the reporting
# policy in fm-nm-run-lib.sh, else the pane busy-signature) and reconciles the
# possibly-stale log against it.
#
# The determinism lives entirely here - only run-step / pane / log reads plus
# fixed mapping logic, no heuristics and no LLM. Output is one stable, parseable,
Expand All @@ -27,14 +27,12 @@
# to the routed status log; dead/missing report the remote verdict; an
# unreachable or unreadable remote reports unknown-remote, never a false
# gone/dead.
# 2. Matching no-mistakes run for this crew's branch AND current code identity,
# active or terminal (from `axi status`, or the coarse `no-mistakes runs`
# fallback)? Branch name alone is not enough: a historical run on a reused
# branch whose head was rewritten or diverged must not be attributed.
# A run matches when its head equals the worktree HEAD, or the worktree HEAD
# is an ancestor of the run head (pipeline fix commits advanced the run on
# the same line of history). Local work that advanced past the run head, or
# diverged from it, invalidates attribution.
# 2. Reportable no-mistakes run for this crew's branch, active or terminal
# (from `axi status`, or the coarse `no-mistakes runs` fallback)? A missing
# head or locally resolvable rewritten/diverged head is not attributed.
# A nonempty head unavailable in the local object store remains reportable
# because it may be a live pipeline commit not yet fetched here. The shared
# library owns the exact reporting and strict teardown matching policies.
# The run-step is AUTHORITATIVE: running/fixing -> working, ci -> working,
# awaiting_approval/fix_review -> parked (with gate findings), terminal
# passed/checks-passed -> done, failed/cancelled -> failed. EXCEPT: while
Expand Down Expand Up @@ -217,10 +215,10 @@ crew_busy_verdict() { # <target>
fm_busy_classify "$TASK_BACKEND" "$1" "$HARNESS" "$ID" "$STATE" "$tail40"
}

# --- no-mistakes run lookup (authoritative when a run matches this branch) --
# --- no-mistakes run lookup (authoritative when reporting attributes a run) -
# trim, strip_quotes, the bounded nm_run call, nm_field's TOON parse, and the
# branch+head attribution rule below are thin wrappers over the ONE owner in
# bin/fm-nm-run-lib.sh, shared with fm-teardown.sh's pre-teardown run abort.
# reporting attribution rule below are thin wrappers over the owner in
# bin/fm-nm-run-lib.sh. Teardown uses that library's stricter policy.

trim() { fm_nm_trim "$@"; }
strip_quotes() { fm_nm_strip_quotes "$@"; }
Expand Down Expand Up @@ -353,38 +351,13 @@ nm_ci_checks_state() {
*) printf 'unknown' ;;
esac
}
# Coarse fallback for cross-branch attribution. `no-mistakes axi status` (bare)
# reports the active-or-most-recent run for the CURRENT branch when one
# exists, else falls back to some other branch's run purely as informational
# display (verified empirically: querying a worktree with its own active run
# reliably returns that run, even under concurrent load from several other
# validating crews on the same underlying repo). A crew whose branch genuinely
# has no run yet therefore sees another branch's answer here.
#
# This fallback used to shell out to `no-mistakes axi` (bare, no subcommand)
# expecting a `runs[N]{id,branch,status,...}:` TOON table and re-query the
# matched id via `axi status --run <id>`. Verified against the real installed
# CLI (v1.32.2): the `axi` surface exposes only abort/logs/respond/run/status -
# there is no runs-listing subcommand under `axi` at all, so that table never
# appears and the lookup was silently dead code; whenever the bare `axi
# status` answer was not this crew's own branch, attribution always failed and
# the caller fell straight through to the pane/log fallback below. (The
# PRIMARY cause of the 2026-07 herdr false-surface incidents turned out to be
# a separate bug in bin/fm-watch.sh's stale_is_terminal precedence - see that
# file's history - but this cross-branch path was independently confirmed
# dead code and is worth having actually work.)
#
# The real run-listing command is the top-level `no-mistakes runs` (verified:
# `no-mistakes --help` lists it separately from `axi`). It is plain, human-
# oriented text - no run id, no JSON/TOON, newest-first, columns
# "<status> <branch> <short-sha> <date> [<pr-url>]" separated by runs of
# spaces (verified: no quoting, so splitting on the first two whitespace runs
# is exact) - but branch + coarse status is exactly what this predicate needs:
# is a run for THIS branch active right now. Echoes the first (most recent)
# matching row's status word (running/completed/cancelled/failed), or empty
# when the branch has no run within FM_CREW_STATE_RUNS_LIMIT rows.
# `axi status` can describe another branch's run, so the fallback scans the
# top-level `no-mistakes runs` rows for this branch. Its plain rows provide only
# a status, branch, and short SHA. Return a reportably matching running row in
# preference to any terminal row, regardless of list order, so stale terminal
# output cannot hide live validation.
nm_runs_status_for_branch() { # <branch>
local branch=$1 out row st rest br sha
local branch=$1 out row st rest br sha terminal=''
out=$(nm_run runs --limit "$FM_CREW_STATE_RUNS_LIMIT")
[ -n "$out" ] || return 0
while IFS= read -r row; do
Expand All @@ -397,37 +370,37 @@ nm_runs_status_for_branch() { # <branch>
rest=${rest#* }
rest=$(trim "$rest")
sha=${rest%% *}
if [ "$br" = "$branch" ]; then
# Same code-identity rule as axi status: skip a same-branch row whose
# short-sha does not match this worktree (rewritten or advanced tip).
if ! nm_coarse_head_matches_worktree "$sha"; then
continue
fi
[ "$br" = "$branch" ] || continue
# A missing or locally incompatible head cannot be attributed; the shared
# reporting rule accepts only heads unavailable in the local object store.
nm_coarse_head_matches_worktree "$sha" || continue
if [ "$st" = running ]; then
printf '%s' "$st"
return 0
fi
[ -n "$terminal" ] || terminal=$st
done <<< "$out"
[ -n "$terminal" ] && printf '%s' "$terminal"
return 0
}

# CREW_BRANCH is empty at detached HEAD (a just-spawned crew, or a scout's
# scratch worktree); with no branch there is no run to attribute to this crew.
CREW_BRANCH=$(git -C "$WT" symbolic-ref --quiet --short HEAD 2>/dev/null || true)

# 0 if the active axi-status run's head field matches this worktree's code
# identity. Branch match is a precondition (caller). Rule owned by
# fm_nm_head_matches_worktree in bin/fm-nm-run-lib.sh.
# 0 if the active axi-status run's head is reportable for this worktree.
# Branch match is a precondition (caller).
nm_run_head_matches_worktree() {
local run_head
run_head=$(strip_quotes "$(nm_field head)")
fm_nm_head_matches_worktree "$WT" "$run_head"
fm_nm_head_matches_worktree_reporting "$WT" "$run_head"
}

# Coarse runs-list rows are "<status> <branch> <short-sha> ...". 0 if the short
# sha for this branch row matches the worktree head under the same rules as
# nm_run_head_matches_worktree (equal, or local is ancestor of run tip).
# SHA for this branch row is reportable under the same rules as
# nm_run_head_matches_worktree.
nm_coarse_head_matches_worktree() { # <short-sha>
fm_nm_head_matches_worktree "$WT" "$1"
fm_nm_head_matches_worktree_reporting "$WT" "$1"
}

HAVE_RUN=0
Expand All @@ -446,9 +419,8 @@ if [ "$KIND" = ship ] && [ -n "$CREW_BRANCH" ] && command -v no-mistakes >/dev/n
if [ -n "$run_branch" ] && [ "$run_branch" = "$CREW_BRANCH" ] && nm_run_head_matches_worktree; then
HAVE_RUN=1
else
# The active-or-most-recent run is for another branch, or same branch with
# a rewritten/diverged head (the CLI is alive and answered; only the
# attribution missed) - try the coarse fallback.
# The reported run belongs to another branch or is not reportable here.
# The CLI answered, so try the coarse fallback.
# Deliberately nested inside `[ -n "$RUN_OUT" ]`: an empty/timed-out
# primary call means the CLI itself did not respond, so retrying it
# immediately with a second bounded call would just double the wait
Expand Down
29 changes: 21 additions & 8 deletions bin/fm-nm-run-lib.sh
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
#!/usr/bin/env bash
# Shared no-mistakes axi run attribution primitives.
#
# ONE owner for the branch+code-identity matching rule that decides whether a
# no-mistakes run belongs to a given worktree, used by fm-crew-state.sh
# (read-only current-state reporting) and fm-teardown.sh (pre-teardown run
# abort, see its "Fix 1" header comment). Getting this wrong in either
# direction is unsafe: a false negative hides a genuinely parked run, and a
# false positive lets teardown act on a run it does not own.
# Owns the branch-and-code-identity policies that decide whether a no-mistakes
# run belongs to a worktree. `fm_nm_head_matches_worktree` is the strict policy
# required before teardown can abort a parked run. Read-only reporting uses the
# reporting variant: it accepts a nonempty run head that is unavailable in the
# local object store, so a live pipeline-owned run is not reported as diverged.
# A locally resolvable head must still pass the strict policy. Never use the
# reporting variant for a destructive action.
#
# Bounded call to `no-mistakes "$@"` in dir $1, timeout $2 seconds. The bounded
# form preserves stdout, stderr, and exit status; the checked form discards
Expand Down Expand Up @@ -55,8 +56,7 @@ fm_nm_field() { # <toon-output> <key>
printf '%s\n' "$1" | sed -n "s/^[[:space:]]*$2:[[:space:]]*\(.*\)/\1/p" | head -1
}

# 0 if run head $2 matches worktree $1's code identity, per the same rule
# everywhere this attribution is needed:
# 0 if run head $2 strictly matches worktree $1's code identity:
# - missing/empty head: cannot bind; reject
# - equal commits (short or full SHA): match
# - worktree HEAD is an ancestor of run head: match (pipeline fix commits on
Expand All @@ -71,3 +71,16 @@ fm_nm_head_matches_worktree() { # <worktree> <run_head>
[ "$run_full" = "$local_full" ] && return 0
git -C "$wt" merge-base --is-ancestor "$local_full" "$run_full" 2>/dev/null
}

# 0 for read-only reporting when the strict rule matches or when a nonempty
# run head cannot be resolved locally. An unresolvable head may be a pipeline
# commit not yet fetched into this worktree, so it must not be treated as a
# proven divergence. This must never authorize teardown or another mutation.
fm_nm_head_matches_worktree_reporting() { # <worktree> <run_head>
local wt=$1 run_head=$2
fm_nm_head_matches_worktree "$wt" "$run_head" && return 0
[ -n "$run_head" ] || return 1
git -C "$wt" rev-parse HEAD >/dev/null 2>&1 || return 1
git -C "$wt" rev-parse --verify "${run_head}^{commit}" >/dev/null 2>&1 && return 1
return 0
}
6 changes: 3 additions & 3 deletions bin/fm-teardown.sh
Original file line number Diff line number Diff line change
Expand Up @@ -107,9 +107,9 @@
# alone: no-mistakes drives those against its own gate-repo clone, not the
# crew's worktree, so they are not orphaned by removing the worktree.
# conclude_task_no_mistakes_run attributes the active-or-most-recent run to
# THIS task only when its branch AND code identity (bin/fm-nm-run-lib.sh's
# fm_nm_head_matches_worktree, the same rule bin/fm-crew-state.sh uses) both
# match this worktree, then runs `no-mistakes axi abort --run <id>` for
# THIS task only when its branch and code identity pass
# bin/fm-nm-run-lib.sh's strict fm_nm_head_matches_worktree policy, then
# runs `no-mistakes axi abort --run <id>` for
# that verified run instance. A run already terminal
# (an outcome is set) or not parked at a gate is left untouched. Idempotent:
# an already-aborted run reads back terminal and is skipped on retry.
Expand Down
5 changes: 3 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,9 @@ The explicit resolution is written by the actor that answers, not the busy worke
This home's answerer close, pending-reply escalation close, and captain-held transfer use the provenance-guarded append owned by `bin/fm-wake-lib.sh`, so they advance the watcher marker only across their own bytes when all earlier bytes were already announced; pending or interleaved foreign bytes fail toward an ordinary wake.
A turn-ended-only queue row omits its historical status annotation when that status file exactly matches the same seen marker.
Any direct or remaining historical annotation prints every status line unread at the presentation cursor instead of replaying only the latest line.
`bin/fm-crew-state.sh <id>` is the cheap current-state read for an actionable heartbeat review: it attributes a no-mistakes run, active or terminal, only when it matches the crew's branch and current code identity, then keeps that run-step authoritative even if the pane has closed.
The script header owns the exact run-head ancestry rules.
`bin/fm-crew-state.sh <id>` is the cheap current-state read for an actionable heartbeat review: it attributes an active or terminal no-mistakes run to the crew's branch and keeps that run-step authoritative even if the pane has closed.
Read-only reporting accepts a run whose nonempty head is unavailable in the local object store, while teardown requires a strict locally resolved code-identity match before it may abort a run.
`bin/fm-nm-run-lib.sh` owns those exact attribution policies.
During no-mistakes' `ci` monitor phase, it also reads the ci step log tail because `axi status` reports both "still waiting on checks" and "checks green, waiting on merge" as `ci,running`.
The most recent recognized ci log marker wins, so checks-green monitoring reports done while a later re-arm, failed-check, or issue marker returns the crew to working.
Only when no matching run exists does it consult semantic busy state; exact busy reports working, exact idle permits fallback to a status-log event whose verb maps to a recognized run-state, and unknown or a dead pane stays unknown instead of trusting a stale log.
Expand Down
2 changes: 1 addition & 1 deletion docs/scripts.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize
| `fm-supervisor-target-lib.sh` | Resolve the shared supervisor target and backend for the daemon and launcher |
| `fm-supervise-daemon.sh` | Presence-gated away-mode sub-supervisor: self-handle routine wakes, guard injection by the detected primary harness, escalate batched digests, alert on failed delivery |
| `fm-crew-state.sh` | Print one deterministic current-state line for a crew |
| `fm-nm-run-lib.sh` | Shared branch-and-code-identity attribution for no-mistakes runs |
| `fm-nm-run-lib.sh` | Shared strict teardown and read-only reporting attribution for no-mistakes runs |
| `fm-tangle-lib.sh` | Shared default-branch resolution and primary-checkout tangle classification |
| `fm-timeout-lib.sh` | Single owner of hard-bounded command execution and its fallback watchdog |
| `fm-timing-lib.sh` | Single owner of the deferred network stage's per-step elapsed-time records, inert unless a run asks for them |
Expand Down
Loading