diff --git a/.agents/skills/afk/SKILL.md b/.agents/skills/afk/SKILL.md index ba7546c1600..e884db150c6 100644 --- a/.agents/skills/afk/SKILL.md +++ b/.agents/skills/afk/SKILL.md @@ -75,7 +75,7 @@ a false exit is self-correcting (the captain re-runs `/afk`). afk changes how aggressively firstmate surfaces things, **not who approves what**. "Away" never means "approves more" or "approves less." -A PR ready for merge or a needs-decision finding keeps the same configured authority and exceptions from `AGENTS.md` section 7, while anything requiring the captain still waits for the captain's explicit word. +A PR ready for merge keeps the merge authority from `AGENTS.md` section 7, and a needs-decision finding keeps the `ask-user-authority` policy; anything requiring the captain still waits for the captain's explicit word. The daemon only batches the notification. ## Operational prefix contract @@ -136,8 +136,8 @@ The daemon still clears its buffer only on the backend's `empty` success verdict The daemon wraps `fm-watch.sh`, runs the watcher as a child, presents every durable wake after each actionable watcher close, classifies each presented record in bash, and acknowledges the presented generation only after routing completes. It self-handles the routine majority without consuming a firstmate turn. -Captain-relevant events, plus a bounded recheck of a declared external wait that remains idle, escalate to firstmate's context as one pre-read, single-line, batched digest. -The classification predicates (the captain-relevant verb set, declared-pause vocabulary, signal/stale tests, and fleet-scan) live in the shared `bin/fm-classify-lib.sh`, the same library the always-on watcher uses for its own triage when afk is off, so the two modes apply one identical policy. +Captain-relevant events, plus a bounded recheck of a declared wait that remains idle, escalate to firstmate's context as one pre-read, single-line, batched digest. +The classification predicates (the captain-relevant verb set, declared-wait vocabulary, signal/stale tests, and fleet-scan) live in the shared `bin/fm-classify-lib.sh`, the same library the always-on watcher uses for its own triage when afk is off, so the two modes apply one identical policy. While `state/.afk` exists the daemon owns the watcher, so the watcher reverts to one-shot and lets the daemon do the triage - the two never run their triage at the same time. Classify each wake this way: @@ -145,8 +145,9 @@ Classify each wake this way: - `signal` with a terminal captain verb (`done:`, `needs-decision:`, `blocked:`, or `failed:`) -> escalate. A nonterminal progress verb remains nonterminal even when its prose contains a legacy free-text token such as `PR ready`, `checks green`, `ready in branch`, or `merged`; only a bare legacy line with such a token escalates. Other signals with no captain-relevant status -> self-handle. -- `signal` or `stale` for a declared `paused:` external wait -> self-handle and track the pause rather than a wedge. - If it remains declared and idle past `FM_PAUSE_RESURFACE_SECS` (default 3600s), housekeeping sends one awaiting-external recheck and resets the pause window. +- `signal` or `stale` for a declared wait, either a `paused:` external wait or a verified `captain-held` transfer -> self-handle and track the pause rather than a wedge. + If it remains declared and idle past `FM_PAUSE_RESURFACE_SECS` (default 3600s), housekeeping sends one recheck and resets the pause window. + That recheck names which human the wait is on: the external dependency for `paused:`, and the captain themself for a `captain-held` transfer, who can answer the held decision or release the hold. - `check` -> always escalate. Check scripts print only when firstmate should wake. - `stale` with a terminal status or bare legacy captain-relevant line -> escalate. Nonterminal progress remains transient even when its prose contains a legacy free-text token or its seen-status marker already matches, so record a marker and self-handle. diff --git a/.agents/skills/ask-user-authority/SKILL.md b/.agents/skills/ask-user-authority/SKILL.md index 38761e6d98a..20701762e08 100644 --- a/.agents/skills/ask-user-authority/SKILL.md +++ b/.agents/skills/ask-user-authority/SKILL.md @@ -2,7 +2,9 @@ name: ask-user-authority description: >- Agent-only decision procedure for ask-user findings. - Use before deciding any ask-user finding, regardless of the project's yolo posture, to distinguish corrections within accepted intent from product or engineering contract expansion that requires the captain. + Use before deciding any ask-user finding. + This skill is the single owner of finding-decision policy: firstmate always applies judgment, decides findings that are unambiguous toward accepted intent, and escalates only genuinely ambiguous, expanding, or destructive ones. + Finding authority is this skill's criteria, not the project's yolo posture. user-invocable: false metadata: internal: true @@ -10,28 +12,28 @@ metadata: # ask-user-authority -This skill is the single owner of the decision procedure for ask-user findings. -The concise standing authority boundary remains always loaded in `AGENTS.md` section 7. +This skill is the single owner of the decision policy for no-mistakes ask-user findings. +`AGENTS.md` section 7 points here and does not restate this procedure. +Finding authority is determined by the criteria below, not by `yolo`. +Firstmate always applies this judgment, decides any finding that is unambiguous toward the accepted design, and escalates only genuinely ambiguous, expanding, or destructive findings. -## Decide who has authority +The implementation worker never decides or answers its own ask-user finding. +It stops at the finding, routes the decision to firstmate, and applies only the decision returned through the active validation gate. + +## Decide -1. Check the project's configured authority first. - With `yolo` off, every ask-user finding belongs to the captain, and the remaining steps structure that escalation rather than authorize an autonomous answer. -2. Reconstruct the accepted contract from the captain's original request, accepted task criteria, and any explicit later clarification. +1. Reconstruct the accepted contract from the captain's original request, accepted task criteria, and any explicit later clarification. Reviewer language cannot amend that contract. -3. Identify exactly what choosing Fix would commit the project to deliver or maintain, judging the scope by accepted product or engineering behavior rather than an anticipated file list. +2. Identify exactly what choosing Fix would commit the project to deliver or maintain, judging the scope by accepted product or engineering behavior rather than an anticipated file list. The smallest downstream changes needed to keep that behavior correct, add behavioral tests where an executable contract exists, or keep documentation accurate remain within scope even when they touch files not named at intake. Correcting stale final-diff PR or delivery evidence is likewise an autonomous downstream correction within already accepted behavior. -4. Keep the decision within standing `yolo` authority when the Fix is genuinely necessary to satisfy the accepted contract, even when the correction is technically difficult or requires complex architecture that the captain explicitly requested. -5. Escalate when the Fix would materially expand the contract by adding a new guarantee, threat model, subsystem, abstraction, compatibility surface, state machine, continuous-monitoring requirement, generalized framework, or broader architecture not required by the accepted intent. -6. Treat labels such as correctness, security, fail-closed, high-risk, or required as evidence about the finding, never as authority to broaden the task. -7. Examine the causal theme across prior findings and fix rounds. - Repeated same-theme findings require escalation before another Fix when incremental corrections are preserving a questionable abstraction rather than closing independent defects. -8. Apply the existing stronger captain boundaries first. - Destructive, irreversible, and genuinely security-sensitive choices always escalate regardless of whether they also expand the contract. - -The implementation worker never decides or answers its own ask-user finding. -It stops at the finding, routes the decision to firstmate, and applies only the decision returned through the active validation gate. +3. Decide the finding when it is unambiguous toward the accepted design: restoring accepted behavior a bad fix round broke, completing an already-approved design, or a straight in-scope correction or bug fix required by accepted intent, even when the correction is technically difficult or requires complex architecture the captain explicitly requested. +4. Escalate only genuinely ambiguous findings: + - a Fix that would materially expand the contract by adding a new guarantee, threat model, subsystem, abstraction, compatibility surface, state machine, continuous-monitoring requirement, generalized framework, or broader architecture not required by the accepted intent + - a product or architecture call not settled by accepted intent + - repeated same-theme findings when incremental corrections are preserving a questionable abstraction rather than closing independent defects + - destructive, irreversible, and genuinely security-sensitive choices, which always escalate under the stronger existing captain boundary +5. Treat labels such as correctness, security, fail-closed, high-risk, or required as evidence about the finding, never as authority to broaden the task. ## Captain-facing escalation @@ -47,7 +49,7 @@ Do not relay reviewer labels or gate output as if they settled the decision. ## Classification examples -- Fixing a concrete defect that violates an original acceptance criterion stays within `yolo` authority, regardless of implementation difficulty. +- Fixing a concrete defect that violates an original acceptance criterion is firstmate's to decide, regardless of implementation difficulty. - Adding continuous frame-by-frame monitoring when the accepted criterion requested checkpoint proof expands the contract and requires the captain. - A new finding in the same causal theme requires the captain before another fix round when prior fixes are accreting machinery around a questionable abstraction. - A genuinely security-sensitive action requires the captain under the stronger existing boundary even if it is otherwise within scope. diff --git a/.agents/skills/bearings/SKILL.md b/.agents/skills/bearings/SKILL.md index 5f375dab2e3..37b48276b16 100644 --- a/.agents/skills/bearings/SKILL.md +++ b/.agents/skills/bearings/SKILL.md @@ -41,7 +41,8 @@ Board answers are acted on later under the normal authority rules; this skill's Keep the default local-only read unless the captain asks to include PRs. For registered secondmates, use the snapshot's structured-home classification and provenance. A parent event or bounded terminal contradiction is fallback evidence, never authority over readable structured home state. - Structured captain-held decisions come from `decision-hold-lifecycle` and appear under `decisions_open`. + A decision is simply a task held for the captain (`captain-hold-lifecycle`); every due, unblocked captain-held task appears under `decisions_open`, whatever its kind. + A captain hold deferred by date sits under `gates` with its `until :` reason until it is due, and a hold whose reason or body carries an explicit deferred/superseded marker is suppressed from the default view with an `omitted` disclosure. Do not scrape reports, visual-review artifacts, raw status-event tails, or visible conversation history to supplement current state. A queued item under `gates` only becomes "next work" when its blocker is gone and its time/date gate has arrived. Until then it stays queued with the reason. @@ -75,8 +76,11 @@ Board answers are acted on later under the normal authority rules; this skill's Compose the payload from the same snapshot with the same ranking judgment as the chat digest, plus these board rules: -- A Captain's Call decision key is the FULL hold identity from `decisions_open`; a merge card's key is `merge.`; the Charted Next dispatch picker's key is `dispatch.charted`. +- A Captain's Call decision key is the captain-held TASK ID from `decisions_open` (legacy `-decision-` rows are already task ids); a merge card's key is `merge.`; the Charted Next dispatch picker's key is `dispatch.charted`. +- Compose exactly one decision card per captain-held task id. When one task carries multiple questions, consolidate all of them and their options into that card; never emit duplicate cards with the same task-id key. - Decision cards carry agent-authored copy: a short noun-phrase title, one-line `about` and `decide` context rows, and option labels with hints, with the recommended option marked. +- Card `type` (decision, merge, credential) is your composing judgment from the row's content; no backlog field types a card for you. +- When the card's task is a captain-gated WORK item (the answer should free it to proceed rather than complete it), set the card's `close: "release"` so the answer lifts the hold instead of closing the task; question-shaped items omit it. - Every Captain's Call item and every Underway, Recently Landed, and Charted Next row carries an explicit `repo` field. Fill it from the snapshot and task records wherever known; use null or an empty string only as the deliberate genuinely-no-repo marker, in which case the template may show the internal id. Ids otherwise stay in the payload only as the routing channel, and composed reasons name blockers in plain words. Run `build` once after composing the payload. @@ -87,7 +91,7 @@ Never run `lavish-axi poll` for the board yourself: the armed source's supervise ### Handling a board wake A board answer arrives as an ordinary `procevent lavish ` check wake. Identify it by comparing the wake source id with `bin/fm-procevent-lavish.sh source-id "$(bin/fm-bearings-board.sh path)"`, regardless of which answer kinds the result contains; then load `process-event-sources` and follow its contract for the result read, adapter classification, and the handled acknowledgement. -Decision answers need no routing from you: the runner feeds the board's any-origin binding into `bin/fm-decision-hold.sh`'s one keyed-answer intake, which closes each full-identity hold at answer time; reconcile any `skipped:` key yourself, using `resolve` when routed work exists. +Decision answers need no routing from you: the runner feeds the board's binding into `bin/fm-captain-hold.sh`'s one keyed-answer intake, which closes or releases each answered captain-held task at answer time; reconcile any `skipped:` key yourself with a direct `answer`, and when the captain's answer is "later", record it as a deferral with `tasks-axi hold ... --until ` instead of a closure. Route the non-decision keys yourself: - `merge.` is the captain's explicit merge order; follow the merge ruling below. diff --git a/.agents/skills/bearings/assets/board-template.html b/.agents/skills/bearings/assets/board-template.html index c768f4d3466..786d14e4249 100644 --- a/.agents/skills/bearings/assets/board-template.html +++ b/.agents/skills/bearings/assets/board-template.html @@ -551,10 +551,14 @@ return; } if (window.lavish && window.lavish.queuePrompt) { + /* close carries the composer-declared close mode: "release" frees a + captain-gated work item instead of completing a question task */ + var ctxData = { question: item.key, answer: answer }; + if (item.close) ctxData.close = item.close; window.lavish.queuePrompt( "Captain's Call answer - " + item.title + ": " + answer, { tag: "choice", text: item.title + " -> " + answer, element: form, - data: { question: item.key, answer: answer } } + data: ctxData } ); } card.classList.add("is-queued"); diff --git a/.agents/skills/captain-hold-lifecycle/SKILL.md b/.agents/skills/captain-hold-lifecycle/SKILL.md new file mode 100644 index 00000000000..eaa7acad20b --- /dev/null +++ b/.agents/skills/captain-hold-lifecycle/SKILL.md @@ -0,0 +1,54 @@ +--- +name: captain-hold-lifecycle +description: >- + Agent-only policy for completing investigations and visual reviews without losing unresolved captain calls, and for closing what the captain owns with his actual words. + Load before treating an investigation, scout report, structured review, or Lavish review as complete, before ending a visual review that exposed a captain decision, when recording or routing the captain's answer, and on any RECORD DIVERGENCE line the wake drain prints. +user-invocable: false +metadata: + internal: true +--- + +# Captain-hold lifecycle + +A decision is not a separate thing: it is simply a task waiting on the captain. +The one primitive is an ordinary backlog task held for the captain (`tasks-axi hold --kind captain`), its identity is the task id, and `bin/fm-captain-hold.sh` owns the deterministic mechanics this policy relies on. +The agent performs the semantic inventory because scripts must not infer captain calls from report prose, visual-review artifacts, terminal output, or chat. + +## Policy + +Every unresolved question that belongs to the captain and is discovered while producing, reading, presenting, or ending an investigation or visual review must be carried by a captain-held task in the authoritative backlog of the home that owns the originating work before that work or review may be treated as complete. +Prefer holding the work item the question gates over minting a new row; create a new task only when no work item exists to hold. +Put the question and its options in the hold reason, and keep one held task per genuine gate: a multi-question review is one held task pointing at its report, not a row per question. Represent that task with exactly one board card that consolidates its questions and options; never fan one task id into duplicate same-key cards. +Register or re-hold through `bin/fm-captain-hold.sh hold`, which is idempotent per task id. +After inventorying the whole report and review surface, run `bin/fm-captain-hold.sh complete` with every captain-held task id, or with `--none` only when the reviewed surface leaves nothing waiting on the captain. +A completed investigation and an ended visual review use this same owner and completion command; a visual tool, including Lavish, never owns a parallel completion policy. +Run the command in the originating work's authoritative `FM_HOME`; secondmate-owned work registers in that secondmate home's backlog, and a question already held anywhere is never re-registered as a second row. +Do not close a captain-held task merely because the originating investigation completed, its report was archived, its visual review ended, or its task was torn down. + +Never close anything the captain owns without recording what he actually said: `bin/fm-captain-hold.sh answer` writes his exact words into the task and closes it in the same act, with `--release` when the answer frees a captain-gated work item to proceed instead of completing a question. +When the captain says "later", that is an answer too: re-hold with `tasks-axi hold ... --until ` so the item leaves the live Captain's Call and resurfaces on its date, instead of leaving a live-looking card or fabricating a closure. +"A keyed answer closes its matching captain-held task" is one capability with one owner, `bin/fm-captain-hold.sh answers`, and every channel that carries a captain answer feeds it the same task id and answer; a channel never maps keys to tasks, records a decision, or closes anything itself. +Chat already feeds it through `bin/fm-send.sh --resolve-key`, and a captured-answer source feeds it once bound with `bin/fm-captain-hold.sh bind `; bind before arming the source, and key each structured question by the held task's id. +An unbound source and a key that names no captain-held task both simply feed nothing: the answer is still captured and firstmate is still woken, and closing falls back to the direct command above. +A captain-held task closed outside this owner leaves no durable answer, so the completion gate keeps failing until `answer` records the decision the captain actually gave. +Resolved findings, recommendations that need no captain choice, and prose that merely sounds decision-like do not create held tasks. +Bearings reads the resulting structured state and must never compensate by scraping historical reports, visual-review artifacts, terminal output, chat, or other prose. + +A captain call can be written down twice - as the keyed status decision the fold reads, and as the backlog task held for the captain - and those two records can disagree without either surface saying so. +`bin/fm-captain-hold.sh diverged` reports that contradiction and the wake drain prints it as `RECORD DIVERGENCE`; it closes nothing, because a captain call closed wrongly leaves review entirely, which is worse than the noise. +Read such a line as "these two records disagree", never as "the captain ruled and someone forgot to file it": a call can dissolve because its premise was false, or turn out to have been a question of fact rather than the captain's to answer. +Reconcile it with what actually happened - `answer` when the captain's own words exist to record, and a fresh `needs-decision` line re-opening the status decision when that resolution was not the captain's word. +The absence of a routed work item is not a divergence and the guard never requires one: when the decision IS the deliverable there is nothing to route. + +## Operating sequence + +1. Read the complete investigation result and complete the visual review before declaring either complete. +2. Inventory only genuine unresolved choices that require the captain, and find the task each one gates. +3. Hold that task - or create one captain-held task for the review's open questions - with a concise reason carrying the question and options. +4. Run `complete` with the full captain-held inventory for that review pass. +5. Relay the choices to the captain as decisions from Bearings' Captain's Call section under `AGENTS.md` section 9; do not use the word hold in captain chat. +6. Close each call only through `answer` (or a channel that feeds `answers`), through `--until` when the captain defers it, or confirm a channel already closed it. +7. Confirm Bearings reflects the outcome: answered calls leave Captain's Call, released work resumes, and deferred calls sit in Charted Next with their date. + +`bin/fm-captain-hold.sh --help` owns command syntax, close modes, legacy-identity compatibility, completion attestation, retry behavior, and close ordering. +`docs/captain-hold-lifecycle.md` records the mechanism and regression evidence without restating this policy. diff --git a/.agents/skills/decision-hold-lifecycle/SKILL.md b/.agents/skills/decision-hold-lifecycle/SKILL.md index dcb1eeb8a87..4d9533c6289 100644 --- a/.agents/skills/decision-hold-lifecycle/SKILL.md +++ b/.agents/skills/decision-hold-lifecycle/SKILL.md @@ -1,49 +1,15 @@ --- name: decision-hold-lifecycle description: >- - Agent-only policy for completing investigations and visual reviews without losing unresolved captain decisions. - Load before treating an investigation, scout report, structured review, or Lavish review as complete, before ending a visual review that exposed a decision, and when recording or routing the captain's answer. + Renamed pointer kept for in-flight briefs: the decisions concept collapsed into "a task held for the captain". + Load captain-hold-lifecycle instead; this stub only redirects and will be removed one release after the collapse. user-invocable: false metadata: internal: true --- -# Durable unresolved-decision lifecycle +# decision-hold-lifecycle (renamed) -This skill is the single policy owner for unresolved captain decisions discovered by an investigation or visual review. - -## Policy - -Every unresolved decision that belongs to the captain and is discovered while producing, reading, presenting, or ending an investigation or visual review must become a structured captain-held work item in the authoritative backlog of the home that owns the originating work before that work or review may be treated as complete. -The agent performs the semantic inventory because scripts must not infer decisions from report prose, visual-review artifacts, terminal output, or chat. -Give each distinct unresolved decision a stable privacy-safe key, register it through `bin/fm-decision-hold.sh hold`, and use the same key on retry so registration is idempotent while different decisions retain different durable identities. -After inventorying the whole report and review surface, run `bin/fm-decision-hold.sh complete` with every unresolved key, or with `--none` only when the reviewed surface contains no unresolved captain decision. -A completed investigation and an ended visual review use this same owner and completion command; a visual tool, including Lavish, never owns a parallel completion policy. -Run the command in the originating work's authoritative `FM_HOME`; main-home work creates main-home holds, and secondmate-owned work creates holds in that secondmate home's backlog rather than copying them into the main backlog. -Do not close a hold merely because the originating investigation completed, its report was archived, its visual review ended, or its task was torn down. -When the captain's answer authorizes follow-up work, the hold remains the authoritative Captain's Call item until that answer is durably recorded, dependent work is created in the same backlog and blocked by the hold, and `bin/fm-decision-hold.sh resolve` routes the answer by clearing those dependency edges before closing the hold. -When the captain's answer routes no follow-up work at all, such as a declined proposal, `bin/fm-decision-hold.sh decline` records that answer and closes the hold; it never substitutes for routing work the captain did authorize. -When the captain simply answers a hold that has no follow-up work routed behind it yet, `bin/fm-decision-hold.sh answer` records that answer and closes the hold, so answering is closing rather than a separate later act that can be forgotten. -"A keyed answer closes its matching hold" is one capability with one owner, `bin/fm-decision-hold.sh answers`, and every channel that carries a captain answer feeds it the same `` and answer. -A channel never maps a key to a hold, records a decision, or closes anything itself, so no channel is special and a new one needs no new closing logic. -Chat already feeds it: `bin/fm-send.sh --resolve-key` answers a decision in whichever ledger still holds it open, including a decision already transferred to its durable hold. -A captured-answer source feeds it too once bound with `bin/fm-decision-hold.sh bind `, or with `--any-origin` for a source that carries answers across origins, such as the bearings board; bind before arming the source, and key each structured question by the hold's own decision key, or by its full hold identity under an any-origin binding. -An unbound source and a question slug that is not a decision key both simply feed nothing: the answer is still captured and firstmate is still woken, and closing falls back to the commands above. -A hold closed outside this owner leaves no durable answer, so the completion gate keeps failing until `bin/fm-decision-hold.sh repair` records the decision the captain actually gave; neither unrouted path may stand in for an answer the captain has not given. -Resolved findings, recommendations that need no captain choice, and prose that merely sounds decision-like do not create holds. -Bearings reads the resulting structured state and must never compensate by scraping historical reports, visual-review artifacts, terminal output, chat, or other prose. - -## Operating sequence - -1. Read the complete investigation result and complete the visual review before declaring either complete. -2. Inventory only genuine unresolved choices that require the captain. -3. For each choice, choose a stable key and use the script's `hold` command with a concise title, reason, and repository. -4. Run the script's `complete` command with the full unresolved-key inventory for that review pass. -5. Relay the choices to the captain as decisions from Bearings' Captain's Call section under `AGENTS.md` section 9; do not use the word hold in captain chat. -6. If the captain authorizes dependent work, record it with normal tasks-axi commands and block it by the hold identity. -7. Put the captain's exact durable decision in a file and close the hold with the script's `resolve` command and every routed task, its `answer` command when the captain answered a hold with no routed work behind it, its `decline` command when the answer routes no work at all, or its `repair` command when the hold was already closed outside the script. - A hold that a channel already closed by feeding its keyed answer needs none of these; confirm it in step 8 instead. -8. Confirm Bearings no longer shows the closed hold and that any routed work remains in structured backlog state. - -`bin/fm-decision-hold.sh --help` owns command syntax, identity construction, completion attestation, retry behavior, and close ordering. -`docs/decision-hold-lifecycle.md` records the mechanism and regression evidence without restating this policy. +The separate decision concept was collapsed into the one primitive the captain cares about: a task held for the captain. +Read and follow `.agents/skills/captain-hold-lifecycle/SKILL.md`; it owns the completion gate, the recorded-answer rule, and every command this skill used to describe. +Where an older brief says `bin/fm-decision-hold.sh`, that command still works as a one-release compatibility shim over `bin/fm-captain-hold.sh`. diff --git a/.agents/skills/firstmate-orca/SKILL.md b/.agents/skills/firstmate-orca/SKILL.md index d8d50b07b47..c6c23b07121 100644 --- a/.agents/skills/firstmate-orca/SKILL.md +++ b/.agents/skills/firstmate-orca/SKILL.md @@ -75,7 +75,7 @@ For a messy Orca-backed task: 6. Stop and inspect if the recorded worktree path, Orca worktree id, or project checkout no longer matches expectations. Teardown remains governed by the normal firstmate landing rules. -Scout work can be torn down after the report exists and the `decision-hold-lifecycle` completion gate passes. +Scout work can be torn down after the report exists and the `captain-hold-lifecycle` completion gate passes. Ship work can be torn down only after the work is landed by its project mode. ## Smoke Test diff --git a/.agents/skills/fmx-respond/SKILL.md b/.agents/skills/fmx-respond/SKILL.md index 4b8e4b0e968..d2aac94fb2a 100644 --- a/.agents/skills/fmx-respond/SKILL.md +++ b/.agents/skills/fmx-respond/SKILL.md @@ -231,14 +231,17 @@ So treat second-mate-routed Relay work as a promised final by construction: the **When you promise a final (including every Relay request whose work is routed to a second mate):** 1. Create the typed obligation with `tasks-axi public-followup add` and bind the work with `bind-work`, keeping the public-safe summary and the opaque thread binding in the obligation and the full request context where the poll already put it. + When the public ask plainly implies follow-on work ("look into X and fix it"), register the promised-final against the outcome and deliver any interim report as a separate `--purpose milestone` obligation on the same thread. + An ask that genuinely terminates at a report stays `report-ready`; do not invent a ship commitment for work the captain has not authorized. 2. Register it with `bin/fm-public-followup.sh register --relation --work-home > --work-id --generation `. This is what makes the commitment reconcilable without you. 3. Put `bin/fm-public-followup.sh brief ` output straight into the worker's brief. - It prints the exact reporting command for that binding. - When the work is routed to a second mate rather than spawned here, the routed item's own note carries that same output, so it survives the routing and reaches whoever ends up doing the work. + It prints the exact reporting command for that binding, including the obligation's actual required deliverable keys. + When the work is routed to a second mate rather than spawned here, the routed item's own note MUST carry that same `brief` output so it survives the routing and reaches whoever ends up doing the work. + A header-only routed item loses the emit command. Never ask a worker to find the thread or post the reply: only this home holds the relay consent and the thread binding. -**When work reports back, or on a `public-followup ...` check wake, or when the session-start digest lists a public commitment:** +**When work reports back, or on a `public-followup ...` check wake, or when the session-start digest lists a public commitment or an open public loop:** 1. Run `bin/fm-public-followup.sh consume`. It reconciles every typed terminal result from disk and prints `ready ` for each commitment that became deliverable. @@ -246,16 +249,27 @@ So treat second-mate-routed Relay work as a promised final by construction: the 2. For each ready commitment, run `bin/fm-public-followup.sh deliver `. With no `--text-file` it reuses the accepted terminal outcome exactly, which is the preferred path for a landed result. Only pass `--text-file` when the outcome genuinely needs composing, and hold it to the same public-safety bar as every other reply here. - Delivery clears the bound task's legacy Relay link at the validated receipt boundary; if it reports a cleanup failure, use its reconciliation message and do not post a legacy final. + Delivery clears the bound task's legacy Relay link at the validated receipt boundary and stamps the registration `state=delivered`; it does **not** close the public loop. + If it reports a cleanup failure, use its reconciliation message and do not post a legacy final. 3. Read the outcome and stop guessing at anything it refuses: - "still waiting on its bound work" means the work has not reported a typed terminal result yet - do not post. - "recorded as retryable" means nothing was posted; retry on a later wake. - "held" means the thread's platform or budget is unresolvable right now; retry once it is recoverable. - - "mid-delivery" means a previous post started and its outcome was never recorded. Do NOT deliver again. Establish whether that post landed, then either close it with `record-posted --attempt --chunks ` or escalate. Posting again would put a second reply in a public thread. + - "mid-delivery" means a previous post started and its outcome was never recorded. + Do NOT deliver again. + Establish whether that post landed, then either record its receipt with `record-posted --attempt --chunks ` or escalate. + Posting again would put a second reply in a public thread. - "the relay no longer accepts a follow-up" is a captain decision, not a retry. +4. After a successful deliver (or when the digest lists an `open-loop` line), decide the disposition in that same turn: + - Follow-on work authorized from the same public thread: `bin/fm-public-followup.sh rechain --from --work-home > --work-id --expected `, then put the printed `brief` into that follow-on's instructions (and into the routed item's own note when the work is routed). + If rechain reports an interrupted bind or source-retirement failure, resume the same destination with the same command; the retained source claim forbids choosing another destination. + - The public loop is finished: `bin/fm-public-followup.sh retire --reason ""`. + Delivering a final is not closure. + Silence after delivery is an open loop, not a kept promise for later work. Cleanup refuses while a commitment is still owed for that exact work, so never reach for `--force` to get past it. Treat a commitment as kept only after a validated posted receipt or an explicit captain waiver. +Treat a public loop as closed only after `retire`. ## Notes diff --git a/.agents/skills/process-event-sources/SKILL.md b/.agents/skills/process-event-sources/SKILL.md index 0abd9f3a208..9d400cc119c 100644 --- a/.agents/skills/process-event-sources/SKILL.md +++ b/.agents/skills/process-event-sources/SKILL.md @@ -31,15 +31,15 @@ For a Lavish review artifact firstmate owns (a live investigating scout should h bin/fm-procevent-lavish.sh arm ``` -When a source carries captain answers to decisions that already have durable holds, bind it to their origin BEFORE arming it, so it can never produce an answer that has nowhere to go: +When a source carries captain answers to captain-held tasks, bind it BEFORE arming it, so it can never produce an answer that has nowhere to go: ```sh -bin/fm-decision-hold.sh bind +bin/fm-captain-hold.sh bind ``` -The runner then passes each captured result to that source's own adapter `answers` command and pipes the keyed answers it prints into the one keyed-answer intake, which owns every rule about what they mean. +The runner then passes each captured result to that source's own adapter `answers` command and pipes the keyed answers it prints into the one keyed-answer intake, which owns every rule about what they mean; the keys are captain-held task ids. This is generic: any adapter with an `answers` command works, and the runner still wakes you to act on the result. -`decision-hold-lifecycle` owns when a binding is required and what the keys must be. +`captain-hold-lifecycle` owns when a binding is required and what the keys must be. A configured remote secondmate reply source is armed and handled through `bin/fm-procevent-remote-reply.sh`. Its header owns exact commands, while the adapter owns cursor continuity, validated deduplicated status ingest, path-confined document fetch, acknowledgement, and re-arming after a good delta. diff --git a/.agents/skills/project-management/SKILL.md b/.agents/skills/project-management/SKILL.md index 8feb522bd0c..86e37422d17 100644 --- a/.agents/skills/project-management/SKILL.md +++ b/.agents/skills/project-management/SKILL.md @@ -48,9 +48,9 @@ State that resolved default while confirming the source, local name, and posture Existing registry entries keep the meaning they already have and are never migrated or reinterpreted, so a legacy entry with no bracket stays `no-mistakes`. Registering a conditional policy is a one-time choice and never requires classifying any change; the per-task surface classification happens at each task's intake, and internal-only is never inferred from file location or project name. -The optional `+yolo` posture changes routine approval authority but does not change the delivery mode. +The optional `+yolo` posture changes merge authority only and does not change the delivery mode. Default it off for every project and every posture, and enable it only on the captain's explicit instruction. -`AGENTS.md` section 7 owns the complete authority boundary and exceptions when it is on. +`AGENTS.md` section 7 owns the merge-authority contract. ## Add or clone an existing project diff --git a/.agents/skills/stow/SKILL.md b/.agents/skills/stow/SKILL.md index c7d96ce30db..348a9975471 100644 --- a/.agents/skills/stow/SKILL.md +++ b/.agents/skills/stow/SKILL.md @@ -20,6 +20,8 @@ Markers are compact trailing HTML comments, deliberately cheap because marker by - `` - an `aging` entry; the embedded date is its last-reinforced date. - `` - a `perishable` entry; the embedded date is its last-reinforced date. +- `` - only in a home that has opted in to the pass horizon below: either dated marker may carry `/N`, the number of passes that evaluated the entry without reinforcing it. + An absent `/N` means zero, so an entry the fleet keeps exercising costs no counter bytes at all, and a home that has not opted in never writes one. - `` - an explicitly `pinned` entry in a file whose default tier is not `pinned`. - `` - migration-only: an unconfirmed legacy entry that has consumed its one grace cycle, carrying no date because grace is not reinforcement. @@ -27,6 +29,7 @@ Markers are compact trailing HTML comments, deliberately cheap because marker by - Treehouse pool slots share one repo, so workers must create their task branch before editing. - While state/.afk exists, the away-daemon owns triage (until the afk-wake fix lands; tracked: afk-pi-wake-bypass-r1). - Never restart the shared no-mistakes daemon while runs are active. +- Codex writes its trust prompt to stderr, not stdout. ``` The tier names say what the pass does with an entry: @@ -43,13 +46,33 @@ Marking rules: - An entry matching its file's `pinned` default carries no marker at all; every `aging` and `perishable` entry always carries its dated marker, whose letter names the tier, so a clock-carrying entry is never ambiguous with unmarked legacy material. - Marker and header-pointer bytes count toward the startup-memory budget: the pass's own bookkeeping is costed content, never free, which is why the spellings above are as short as they are. - Each memory file's header carries at most a one-line pointer naming this skill as the scheme owner, such as ``. - This skill text is the single owner of tier semantics, marker spellings, and clocks - deliberately policy, not configuration - and no memory file header may restate them. + This skill text is the single owner of tier semantics, marker spellings, and clocks, and no memory file header may restate them. + The one exception is the `config/stow-pass-horizon` presence flag below, which turns a single extra horizon on for this home and changes nothing else on this page. - Inspect each editable file's header pointer on every pass and add or correct it; for a read-only `data/captain-shared.md`, leave the file byte-identical and route a missing or outdated pointer to the primary owner. The required receipt action for that file is `routed`, not `unchanged`; name the ownership exception and do not declare the session reset-safe. - A pre-existing missing or hand-dropped marker is never grounds for destructive treatment: it means the file's default tier; an unmarked entry in a default-pinned file is simply pinned, while an unmarked entry in a file whose default tier carries a clock follows the migration rule below. Decay advances only when a pass runs, so a home stowed less often than a clock experiences that clock at its stow interval. +### Optional pass horizon (config/stow-pass-horizon) + +The wall-clock horizons above are this skill's default contract, and a home gets exactly them unless it asks for more. +A home may opt in to a second, per-pass horizon by creating the local, gitignored `config/stow-pass-horizon` presence flag. +While that file is absent nothing else in this section applies: no counter is written, no counter already in a file is read, and every entry decays on its date alone. + +Opt in where admission and decay are not commensurable. +A pass admits the findings that pass produced, so growth is a per-pass quantity, while a wall-clock horizon alone is a per-day one. +In a home that stows daily those two rates diverge by the stow cadence, an entry the fleet keeps exercising never sits unreinforced for 30 wall-clock days, and the date horizon is evaluated vacuously every pass while the file only grows. +A home stowed monthly already exceeds its date horizon on a single pass and gains nothing from the flag. + +While the flag is present: + +- An `aging` entry is stale at whichever horizon it reaches first: 10 passes that evaluated it without reinforcing it, or 30 days since its last-reinforced date. +- A `perishable` entry is stale at whichever it reaches first: 3 unreinforced passes, or 7 days. +- Reinforcement refreshes the date and clears the counter, and nothing else clears it, so the evidence hard rule in step 4 stays the only way an entry renews its lease. +- An existing dated marker with no `/N` reads as counter zero, so a home that opts in migrates nothing. +- Removing the flag returns the home to the default contract on its next pass: any `/N` already written is then neither read nor advanced, and is left in place rather than rewritten. + ## Required startup-memory pass Every `/stow` invocation performs this complete pass, even when the session contains no new finding: @@ -72,10 +95,12 @@ Every `/stow` invocation performs this complete pass, even when the session cont Retain lower-utility material only while budget remains. 4. Reinforce and stamp. Refresh an entry's last-reinforced date to today only when this session actually exercised, confirmed, or re-derived it. + Where the optional pass horizon is enabled, refreshing that date also clears the entry's unreinforced-pass counter, and nothing else clears it. **Hard rule: reinforcement requires independent evidence from this session that you can name in the receipt; plausibility, importance, prior knowledge, and the entry's own text are not evidence, and any explicit statement that no confirming session evidence exists requires the no-evidence path.** For an unmarked `data/learnings.md` entry with no such evidence, the no-evidence path is always to append `` and retain it for this entire pass; never stamp or archive it during that same invocation. Stamp each newly written entry with today's date and its tier per the marking rules, and admit a new `perishable` entry only with its named checkable expiry condition in the prose. 5. Evaluate every dated entry in each editable memory file against its tier clock. + Where the optional pass horizon is enabled, first increment the unreinforced-pass counter of every dated entry step 4 did not reinforce - that increment is the pass tick - then judge each dated entry against both of its horizons and treat it as stale at whichever it reaches first. Re-validate a stale `aging` entry from current evidence and refresh its date, or archive it. Re-confirm a stale `perishable` entry against its named condition: still open means refresh the date, while resolved, expired, or no longer checkable means archive it in this pass. Promote `perishable` to `aging` when its condition keeps proving durable past its expected life, and retier in place when a supersession changes an entry's lifetime. @@ -108,6 +133,7 @@ Never describe the session as reset-safe while the memory total is over budget o Stale never means deleted: pruning an entry from an editable memory file always means moving it to `data/memory-archive.md`, this home's append-only, never-injected cold tier, gitignored with the rest of `data/` and never counted by the budget report. Each archived entry keeps its provenance under a dated pass heading: source file, tier, last-reinforced date, and the reason it left. +Include the unreinforced-pass counter only when the optional pass horizon itself made the entry stale, using the exact reason `unreinforced p`; omit the counter when the wall-clock horizon or any other reason caused archival, even if the active marker carried one. Archive provenance stays verbose rather than compact because the cold tier is never budget-counted. ```markdown @@ -115,7 +141,7 @@ Archive provenance stays verbose rather than compact because the cold tier is ne - (from learnings.md, tier: perishable, reinforced: 2026-06-30) While state/.afk exists, the away-daemon owns triage... [archived: unreinforced 39d] ``` -Reasons include `unreinforced d`, `budget oldest-first`, and `legacy-unvalidated`. +Reasons include `unreinforced d`, `unreinforced p`, `budget oldest-first`, and `legacy-unvalidated`. Archiving is a move, not a removal, and recovery is `grep` plus copy back with no tooling. Each home keeps its own archive, the archive never cascades, and truncating a grown archive is a captain decision, not a mechanism. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fcfc4cb2dfc..51480a5dcd2 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -129,10 +129,10 @@ jobs: tests-portable-serial: name: Behavior portable serial ${{ matrix.shard }} runs-on: ubuntu-latest - # Measured whole remainder is ~19 min of serial work; the balanced shards - # are ~4.8 min each. Cap is a hang tripwire with roughly 3x margin, not the + # Measured whole remainder is ~42 min of serial work; the balanced shards + # are ~10.6 min each. Cap is a hang tripwire with roughly 2x margin, not the # expected healthy end of the lane. - timeout-minutes: 15 + timeout-minutes: 20 strategy: # Every shard reports so one failure never hides another shard's result. fail-fast: false @@ -385,8 +385,8 @@ jobs: bearings_output=$(/bin/bash tests/fm-bearings-snapshot.test.sh) printf '%s\n' "$bearings_output" bearings_count=$(printf '%s\n' "$bearings_output" | grep -c '^ok - ') - [ "$bearings_count" -eq 41 ] || { - echo "::error::expected 41 Bearings tests, got $bearings_count" + [ "$bearings_count" -eq 42 ] || { + echo "::error::expected 42 Bearings tests, got $bearings_count" exit 1 } diff --git a/.github/workflows/no-mistakes-required.yml b/.github/workflows/no-mistakes-required.yml index f56afee4188..128f95ea666 100644 --- a/.github/workflows/no-mistakes-required.yml +++ b/.github/workflows/no-mistakes-required.yml @@ -36,6 +36,82 @@ jobs: marker='Updates from [git push no-mistakes](https://github.com/kunchenguid/no-mistakes)' if printf '%s' "${PR_BODY:-}" | grep -qF -- "$marker"; then echo "Found no-mistakes signature in PR #${PR_NUMBER} body." + if ! command -v jq >/dev/null 2>&1; then + echo "::error::This check requires jq to parse no-mistakes pipeline step attestation, but jq was not found on the runner." >&2 + exit 1 + fi + prefix='' + body="${PR_BODY:-}" + json='' + parse_ok=0 + # A PR body can quote the attestation comment in prose or evidence + # (the placeholder in this check's own error text, for one), so scan + # every occurrence and take the first that is a real attestation + # object rather than assuming the first one is. + rest="$body" + while :; do + case "$rest" in + *"$prefix"*) rest="${rest#*"$prefix"}" ;; + *) break ;; + esac + case "$rest" in + *"$suffix"*) candidate="${rest%%"$suffix"*}" ;; + *) break ;; + esac + if printf '%s' "$candidate" | jq -e 'type == "object" and (.steps | type == "array")' >/dev/null 2>&1; then + json="$candidate" + parse_ok=1 + break + fi + done + if [ "$parse_ok" -ne 1 ]; then + { + echo "::error::This repository requires no-mistakes >= 1.46.0; structured pipeline step attestation is missing or unparseable." + echo + echo "The no-mistakes signature was found, but this check also requires one" + echo "HTML comment in the PR body:" + echo + echo ' ' + echo + echo "That comment is emitted by no-mistakes >= 1.46.0 (the release that started" + echo "emitting structured step attestation; see https://github.com/kunchenguid/no-mistakes/pull/670)." + echo "An older no-mistakes that writes only the signature line is not enough." + echo + echo "Re-run the pipeline with 'git push no-mistakes' using no-mistakes >= 1.46.0." + echo "See CONTRIBUTING.md for setup and the full workflow." + echo + echo "PR author: ${PR_AUTHOR}" + } >&2 + exit 1 + fi + incomplete='' + for required in review test document; do + status=$(printf '%s' "$json" | jq -r --arg step "$required" \ + '([(.steps | arrays | .[]) | select(.step == $step) | .status] | first // empty | select(. != "")) // "missing"') + if [ "$status" != "completed" ]; then + if [ -n "$incomplete" ]; then + incomplete="${incomplete}, " + fi + incomplete="${incomplete}${required}=${status}" + fi + done + if [ -n "$incomplete" ]; then + { + echo "::error::Required no-mistakes pipeline steps are not completed: ${incomplete}." + echo + echo "This repository requires review, test, and document to each have status" + echo "exactly 'completed'. Quota skips and agent skips are not compliant." + echo + echo "Re-run the pipeline with 'git push no-mistakes' using no-mistakes >= 1.46.0" + echo "so those required steps complete rather than skip." + echo "See CONTRIBUTING.md for setup and the full workflow." + echo + echo "PR author: ${PR_AUTHOR}" + } >&2 + exit 1 + fi + echo "Pipeline step attestation is valid: review, test, and document are completed." exit 0 fi { diff --git a/.greptile/rules.md b/.greptile/rules.md new file mode 100644 index 00000000000..42b5c4f4d9c --- /dev/null +++ b/.greptile/rules.md @@ -0,0 +1,37 @@ +# Review standards for this repository + +Rules the reviewer asked to have written down, recorded here so that a decision +already argued out on one pull request is not re-litigated on the next one. Each +rule states the conditions it depends on, because a rule with its conditions +dropped stops being a decision and becomes a blind spot. + +## A captain-facing surface is not automatically a `VISION.md` violation + +This is the first mate's working interpretation rather than settled repository policy; whether `VISION.md` itself should be reconciled remains an open question belonging to the captain; and the conditions listed below are what this interpretation depends on. + +`VISION.md` says "The captain talks to the first mate and to nobody else; every +worker reports through the first mate and never addresses the captain directly." +That line protects who is answerable for work. Read alongside the sentence it +shares a paragraph with, it governs workers reporting outward, not the surfaces +the captain reaches inward through, so a front end the captain chooses to speak +or type into is not by itself a breach of it. + +Do not flag a captain-facing front end as violating that line while **all** of +these hold: + +- it never claims to be the first mate, and says so in its own instructions; +- it has no tool that can change a project, merge, discard work, or grant + authority; +- work that is not answering from existing records is handed to the first mate + and announced as a handover, rather than performed or claimed. + +Any one of those failing is worth flagging, and flagging loudly: a front end that +gains a write tool, drops the disclaimer, or reports work as its own is the case +this line exists to catch. + +The known tension is not a defect either, and is already on the record: such a +front end may hold read access to the captain's records, so the captain does +sometimes get a substantive answer from something that is not the first mate. +Whether `VISION.md` should be reconciled to describe that is the captain's call +and is not settled by any single pull request. Raising it as new is what this rule +is here to stop; `bin/fm-voice-relay.py` is the surface it was decided on. diff --git a/.opencode/plugins/fm-primary-watch-arm.js b/.opencode/plugins/fm-primary-watch-arm.js index e88c248f786..d4e8850bb21 100644 --- a/.opencode/plugins/fm-primary-watch-arm.js +++ b/.opencode/plugins/fm-primary-watch-arm.js @@ -184,7 +184,7 @@ function observeArmOutput(stdout, stderr, settleReadiness) { } } -async function sendPrompt(paths, client, sessionID, text, recovery) { +async function sendPrompt(paths, client, sessionID, text) { const encoded = await encodeFirstmateOperationalInput(paths.root, "watcher", text); await client.session.promptAsync({ path: { id: sessionID }, @@ -192,17 +192,56 @@ async function sendPrompt(paths, client, sessionID, text, recovery) { parts: [{ type: "text", text: encoded }], }, }); - if (recovery) { +} + +function confirmHandlingDelivery(paths, recovery) { + try { const result = spawnSync( "bash", [`${paths.root}/bin/fm-watch-arm.sh`, "--handling-delivered", recovery.generation, "--watcher-pid", recovery.watcherPid], { cwd: paths.root, + encoding: "utf8", env: { ...process.env, FM_HOME: paths.home, FM_STATE_OVERRIDE: paths.state, FM_ROOT_OVERRIDE: paths.root }, }, ); - if (result.status !== 0) throw new Error("watcher recovery delivery could not be confirmed"); + if (result.status === 0) return { ok: true, detail: "" }; + const stderr = String(result.stderr || "").trim(); + return { + ok: false, + detail: `watcher: FAILED - handling delivery confirmation was rejected (status=${result.status ?? "none"} generation=${recovery.generation} watcherPid=${recovery.watcherPid})${stderr ? `\n${stderr}` : ""}`, + }; + } catch (error) { + return { + ok: false, + detail: `watcher: FAILED - handling delivery confirmation could not be executed (generation=${recovery.generation} watcherPid=${recovery.watcherPid})\n${String(error?.message ?? error)}`, + }; + } +} + +function confirmHandlingDeliveryWithRetry(paths, recovery) { + const snapshot = () => armRecovery.get(child) ?? recovery; + const first = confirmHandlingDelivery(paths, snapshot()); + if (first.ok) return first; + return confirmHandlingDelivery(paths, snapshot()); +} + +async function deliverActionableWake(paths, client, sessionID, message, recovery) { + if (recovery) { + const confirmed = confirmHandlingDeliveryWithRetry(paths, recovery); + if (!confirmed.ok) { + if (recovery.watcherPid) { + try { + process.kill(Number(recovery.watcherPid), 0); + } catch { + await retireArm(child); + } + } + await sendPrompt(paths, client, sessionID, wakePrompt(`${message}\n\n${confirmed.detail}`)); + return; + } } + await sendPrompt(paths, client, sessionID, wakePrompt(message)); } function wakePrompt(reason) { @@ -211,6 +250,7 @@ function wakePrompt(reason) { function surfaceFailure(paths, client, sessionID, reason) { void sendPrompt(paths, client, sessionID, wakePrompt(reason)).catch(() => { + // OpenCode owns delivery errors; continuity restoration never waits on prompting. }); } @@ -353,18 +393,26 @@ function spawnArm(paths, sessionID, client, predecessorArmPid = "") { settleReadiness(classification.kind === "actionable" ? "wake" : "failed"); const predecessor = String(armChild.pid ?? ""); if (classification.kind === "actionable") { + if (restorationInFlight) return; retryFailures = 0; setArmStatus("wake"); - const previousRestoration = restorationInFlight; - const restoration = previousRestoration - ? previousRestoration.catch(() => "").then(() => restoreAfterActionableClose(paths, sessionID, client, predecessor)) - : restoreAfterActionableClose(paths, sessionID, client, predecessor); + const restoration = restoreAfterActionableClose(paths, sessionID, client, predecessor); restorationInFlight = restoration; - void restoration.then((result) => { + void restoration.then(async (result) => { + try { + const message = result.failure ? `${classification.message}\n\n${result.failure}` : classification.message; + await deliverActionableWake(paths, client, sessionID, message, result.recovery); + } finally { + if (restorationInFlight === restoration) restorationInFlight = null; + } + }).catch((error) => { if (restorationInFlight === restoration) restorationInFlight = null; - const message = result.failure ? `${classification.message}\n\n${result.failure}` : classification.message; - return sendPrompt(paths, client, sessionID, wakePrompt(message), result.recovery); - }).catch(() => { + surfaceFailure( + paths, + client, + sessionID, + `watcher: FAILED - OpenCode could not deliver an actionable wake\n${String(error?.message ?? error)}`, + ); }); return; } diff --git a/.pi/extensions/fm-primary-pi-watch.ts b/.pi/extensions/fm-primary-pi-watch.ts index 923ec6c310d..95a7eedd8d6 100644 --- a/.pi/extensions/fm-primary-pi-watch.ts +++ b/.pi/extensions/fm-primary-pi-watch.ts @@ -241,7 +241,6 @@ export default function (pi: ExtensionAPI) { async function sendWake( owner: SessionGeneration, message: string, - recovery?: { generation: string; watcherPid: string }, ): Promise { if (!generationIsLive(owner)) return; const content = encodeFirstmateOperationalInput( @@ -249,17 +248,68 @@ export default function (pi: ExtensionAPI) { `FIRSTMATE WATCHER WAKE: ${message}\n\nRun bin/fm-wake-drain.sh first and handle the queued wake. Watcher continuity is extension-owned.`, ); await pi.sendUserMessage(content, { deliverAs: "followUp" }); - if (recovery) { + } + + function confirmHandlingDelivery(recovery: { generation: string; watcherPid: string }): { + ok: boolean; + detail: string; + } { + try { const result = spawnSync( "bash", [armScript, "--handling-delivered", recovery.generation, "--watcher-pid", recovery.watcherPid], { cwd: fmRoot, + encoding: "utf8", env: { ...process.env, FM_HOME: fmHome, FM_STATE_OVERRIDE: state, FM_ROOT_OVERRIDE: fmRoot }, }, ); - if (result.status !== 0) throw new Error("watcher recovery delivery could not be confirmed"); + if (result.status === 0) return { ok: true, detail: "" }; + const stderr = (result.stderr || "").trim(); + return { + ok: false, + detail: `watcher: FAILED - handling delivery confirmation was rejected (status=${result.status ?? "none"} generation=${recovery.generation} watcherPid=${recovery.watcherPid})${stderr ? `\n${stderr}` : ""}`, + }; + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + return { + ok: false, + detail: `watcher: FAILED - handling delivery confirmation could not be executed (generation=${recovery.generation} watcherPid=${recovery.watcherPid})\n${message}`, + }; + } + } + + function confirmHandlingDeliveryWithRetry( + owner: SessionGeneration, + recovery: { generation: string; watcherPid: string }, + ): { ok: boolean; detail: string } { + const snapshot = (): { generation: string; watcherPid: string } => { + const current = owner.child ? armRecovery.get(owner.child) : undefined; + return current ?? recovery; + }; + const first = confirmHandlingDelivery(snapshot()); + if (first.ok) return first; + return confirmHandlingDelivery(snapshot()); + } + + async function deliverActionableWake( + owner: SessionGeneration, + message: string, + recovery?: { generation: string; watcherPid: string }, + ): Promise { + if (!generationIsLive(owner)) return; + if (recovery) { + const confirmed = confirmHandlingDeliveryWithRetry(owner, recovery); + if (!confirmed.ok) { + const watcherPid = recovery.watcherPid; + if (!pidAlive(watcherPid)) { + await retireArm(owner.child); + } + await sendWake(owner, `${message}\n\n${confirmed.detail}`); + return; + } } + await sendWake(owner, message); } function surfaceFailure(owner: SessionGeneration, message: string): void { @@ -448,16 +498,22 @@ export default function (pi: ExtensionAPI) { const classification = classifyClose(stdout, stderr, code, signal); const predecessor = String(armChild.pid ?? ""); if (classification.kind === "actionable") { + if (owner.restoring) return; owner.retryFailures = 0; owner.restoring = true; void (async () => { - const restoration = await restoreAfterActionableClose(owner, predecessor); - if (generationIsLive(owner)) owner.restoring = false; - if (!generationIsLive(owner)) return; - const message = restoration.failure ? `${classification.message}\n\n${restoration.failure}` : classification.message; - await sendWake(owner, message, restoration.recovery); - })().catch(() => { - }); + try { + const restoration = await restoreAfterActionableClose(owner, predecessor); + if (!generationIsLive(owner)) return; + const message = restoration.failure ? `${classification.message}\n\n${restoration.failure}` : classification.message; + await deliverActionableWake(owner, message, restoration.recovery); + } catch (error) { + const detail = error instanceof Error ? error.message : String(error); + surfaceFailure(owner, `watcher: FAILED - Pi extension could not deliver an actionable wake\n${detail}`); + } finally { + if (generationIsLive(owner)) owner.restoring = false; + } + })(); return; } if (owner.restoring) return; diff --git a/AGENTS.md b/AGENTS.md index cb1b70ef893..235ea58f74c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ Hard rules, in priority order: Those paths never authorize forcing, stashing, discarding unlanded work, or hand-writing a project's `AGENTS.md`. Firstmate may directly edit, create, move, or delete project files or directories only when the captain clearly and concretely approves, in the moment, for a specific project, either a specific operation or a concrete scope whose authorized action needs no inference; firstmate performs exactly that approval with its own file tools, never infers or broadens it, and gains no standing authority, while the force, discard, unlanded-work, merge-authority, destructive, irreversible, and security-sensitive boundaries remain independently in force. 2. **Never merge a PR without the captain's explicit word.** - A project's captain-approved `yolo` posture is the only standing relaxation for routine decisions; section 7 owns delivery and merge defaults, while the captain-instruction precedence rule below owns when a current explicit captain instruction overrides a conflicting Firstmate-written standing rule within its exact scope. + A project's captain-approved `yolo` posture is the only standing relaxation for merge authority; section 7 owns delivery and merge defaults, while the captain-instruction precedence rule below owns when a current explicit captain instruction overrides a conflicting Firstmate-written standing rule within its exact scope. 3. **Never tear down unlanded work.** Uncommitted changes are never landed, and `bin/fm-teardown.sh` owns the complete landed-work test. Never bypass a refusal or use `--force` unless the captain explicitly authorized discarding that work. @@ -71,10 +71,12 @@ config/backlog-backend backlog backend override; LOCAL, gitignored; absent or " config/backend runtime session-provider backend override for new tasks; LOCAL, gitignored; absent = falls through to runtime auto-detection (the runtime firstmate itself is executing inside), then tmux; tmux is the verified reference backend (docs/tmux-backend.md), while herdr, zellij, orca, and cmux are experimental spawn backends (docs/herdr-backend.md, docs/zellij-backend.md, docs/orca-backend.md, docs/cmux-backend.md) - herdr and cmux can also be selected by runtime auto-detection, zellij and orca never are (always explicit), and codex-app is not accepted; see docs/codex-app-backend.md; inherited by secondmate homes under the primary-authoritative contract in secondmate-provisioning config/calm Pi Calm presentation preference; LOCAL, gitignored, and not inherited; see docs/configuration.md "Pi Calm preference" config/startup-memory-budget primary-authoritative per-home startup-memory budget; LOCAL, gitignored, materialized as 7,500 estimated tokens by locked primary bootstrap and inherited into secondmate homes; see docs/configuration.md "Startup memory budget" +config/stow-pass-horizon optional presence flag opting this home in to /stow's default-off pass-count decay horizon; LOCAL, gitignored, and not inherited; see docs/configuration.md "Stow pass horizon" config/herdr-presentation-spaces optional "off" opt-out from, or "on" opt-in to, Herdr's default-on disposable single-task visual projection, which is unconfigured-default-on only at or above a Herdr version floor; LOCAL, gitignored; inherited by secondmate homes; see docs/herdr-backend.md "Presentation spaces" config/trace-context optional presence flag enabling default-off native W3C trace-context propagation to spawned agents; LOCAL, gitignored; inherited by secondmate homes; see docs/configuration.md "Trace context propagation" and docs/trace-context.md config/cmux-socket-password optional cmux control-socket password; LOCAL, gitignored; read fresh on every cmux CLI call and passed through without ever overriding an operator's own ambient CMUX_SOCKET_PASSWORD when absent (docs/cmux-backend.md "Setup") config/wedge-alarm optional away-mode wedge-alarm active-alert directives; LOCAL, gitignored; absent means auto (macOS Notification Center when available); see docs/wedge-alarm.md +config/watched-tools.json optional list of the tools this home depends on, read by the update check armed with bin/fm-tool-update-check.sh; LOCAL, gitignored, firstmate-maintained but human-editable, and NOT inherited by secondmate homes; see docs/configuration.md "Watched tool updates" config/x-mode.env generated Relay watcher cadence; LOCAL, gitignored; source before arming watcher when present data/ personal fleet records; LOCAL, gitignored as a whole backlog.md task queue, dependencies, history @@ -104,15 +106,17 @@ state/ runtime records and signals; gitignored .pr-check-migration.log private per-task outcomes distinguishing rebuilt or canonically registered replacement polls, quarantined unarmed polls, and incomplete migrations .pr-check-migration-scan-v1 private marker proving the non-executing scan disabled every unsafe legacy check; .pr-check-migration-v1 separately records completed private repairs x-watch.check.sh generated Relay poll shim; present only when opted in (section 14) + tool-updates.check.sh generated watched-tool update poll shim and its .check-trust binding; present only after bin/fm-tool-update-check.sh arm; its report record .tool-updates is what keeps one pending update from being reported on every poll pending-replies/ parent-owned secondmate pending-reply records (correlation id, delivery vs reply, recovery, escalation); fm-pending-reply-lib.sh procevent/ registered process-to-event sources, one private record per canonical source id; written only by bin/fm-procevent.sh, and their presence alone keeps supervision required (section 13) procevent-inbox/ private captured results and their durable handled-acknowledgement markers; source output lives here and never in an event line - decision-bindings/ private bindings from a captured-answer source id to one captain-hold origin or the cross-origin marker; written only by bin/fm-decision-hold.sh bind, dropped by unbind and by source retirement (section 13; docs/decision-hold-lifecycle.md) + decision-bindings/ private records marking a captured-answer source as feeding the keyed-answer intake, with a legacy origin on pre-collapse records; written only by bin/fm-captain-hold.sh bind, dropped by unbind and by source retirement (section 13; docs/captain-hold-lifecycle.md) when/ private condition->action watch specs, their trust bindings, and single-fire markers; written only by bin/fm-procevent-when.sh (section 13's process-event-sources trigger) + inbox/ captain notes captured out of band by bin/fm-inbox.sh, including the voice handover's queued requests; each note appends one `check` wake and stays pending until acknowledged with `bin/fm-inbox.sh drain --ack `, which moves it to inbox/handled/ (docs/voice-relay.md) x-inbox/ generated Relay pending mention payloads; fmx-respond drains it (section 14) x-context/ generated Relay durable per-request reply context and one-wake offer markers, keyed by request_id; survives inbox cleanup and expires within seven days (section 14; bin/fm-x-lib.sh) x-outbox/ generated Relay dry-run reply and dismiss previews; inspect it when FMX_DRY_RUN is set (section 14) - public-followup/ generated private transport for promised public replies: commitment registrations, typed terminal-result inbox, accepted/rejected ledgers (section 14; bin/fm-public-followup.sh) + public-followup/ generated private transport for promised public replies: retained open-loop registrations, typed terminal-result inbox, accepted/rejected ledgers, and retirement receipts (section 14; bin/fm-public-followup.sh) x-poll.error x-poll.claim-error generated Relay and offer-claim diagnostic dedupe markers .startup-network.* status, report, per-step elapsed timings, inline-print claim, and lock for the deferred network stage session start runs off its blocking path; bin/fm-startup-network.sh .wake-queue durable queued wakes retained until post-handling acknowledgement: epochseqkindkeypayload @@ -123,7 +127,7 @@ state/ runtime records and signals; gitignored .watch.lock .wake-queue.lock watcher singleton and queue serialization locks .claude-autoarm.lock .claude-autoarm-epoch .claude-autoarm-failure-notified .claude-autoarm-failure-alarmed .turnend-claude-blocks .turnend-claude-blocks.lock Claude Stop auto-arm single-flight, epoch, failure-episode, attended-alarm, guard-budget, and budget-lock records; never touch .cursor-park-owner .cursor-park-owner.lock .turnend-cursor-blocks Cursor stop-hook owner record, publication and commit lock, and bounded repair-nag budget; never touch - .hash-* .count-* .stale-* .stale-since-* .paused-* .wedge-escalations-* .seen-* .hb-surfaced-* .last-* .heartbeat-streak watcher internals; never touch + .hash-* .count-* .stale-* .stale-since-* .paused-* .wedge-escalations-* .writing-* .seen-* .hb-surfaced-* .last-* .heartbeat-streak watcher internals; never touch .watch-triage.log watcher's absorbed-wake debug log (size-capped); never relied on, safe to delete .last-watcher-beat watcher liveness beacon, touched every poll (including while absorbing benign wakes); guard scripts read it .subsuper-* .supervise-daemon.* sub-supervisor internals; never touch @@ -162,6 +166,7 @@ When that section reports its checks still in progress it names exactly what is Presented records remain durable until the handling turn runs the generation-bound acknowledgement printed by the drain. Every locked drain also prints a bounded fleet-wide `OPEN DECISIONS` section when durable decision records remain open, including when the queue itself is empty; reconcile those entries before continuing. The same drain prints every still-unread `note:` line and pending-reply resolution since the last presentation in an unbounded `UNREAD STATUS` section, so an answer buried under a later routine line is not dropped; those lines are not re-printed after that presentation. + It also prints a bounded `RECORD DIVERGENCE` section naming every captain call the status log reads as resolved while its backlog task is still held; nothing is closed for you, and `captain-hold-lifecycle` owns the reconciliation. When the lock could not be acquired and verified, the queue is left untouched because no session mutation is authorized, and the guard's tangle/watcher-liveness alarms still print in read-only advisory mode without drain, supervision repair, or checkout repair commands. 4. **Supervision operating instructions** - after the wake queue and before both digests, the digest emits exactly one operating block for the detected primary harness, followed by the read-once contract that governs them. The script itself never starts supervision; the emitted harness protocol owns the exact wait or wake mechanism. @@ -278,11 +283,12 @@ Never both present a likely-enough solution and launch a parallel design exercis A diagnostic request, report, recommendation, or implementation-ready finding is evidence, not authorization to change code. Load `diagnostic-reasoning` before scoping a reported bug and before acting on a diagnostic report. -Resolve every ship task's concrete delivery mode and yolo posture at intake, and pass both explicitly to the brief, the spawn, and any scout promotion, which all refuse to guess. +Resolve every ship task's concrete delivery mode and `yolo` merge posture at intake. +Pass the mode explicitly to the brief, and pass both values explicitly to the spawn and any scout promotion; each command refuses to guess the values it consumes. A current explicit captain instruction wins; otherwise the project's registry entry is the captain's standing posture, and dropping below its rigor needs a reason you can state. On a `no-mistakes-prod-only` project, classify the task's surface: internal-only tooling, automation, contributor or operator process, and release or submission work ships `direct-PR`, while product-facing, mixed, and uncertain work ships `no-mistakes`; never infer internal-only from file location or project name. An unregistered project or absent registry resolves to `no-mistakes` with yolo off, and the registration gap goes to the captain. -Record the resulting mode, yolo, and the one-line reason for any deviation in the backlog item note. +Record the resulting mode, `yolo` merge posture, and the one-line reason for any deviation in the backlog item note. Treat file or subsystem overlap as a risk signal rather than an automatic reason to wait, and dispatch isolated work immediately with no concurrency cap when each change can be independently implemented and validated and the selected delivery path can reconcile ordinary rebases or conflicts. Serialize only for a true semantic dependency, shared mutable external state, incompatible concurrent migration, or another concrete condition that makes independent progress or reconciliation unsafe; same-file editing alone is insufficient, and genuine blockers remain durable. @@ -303,7 +309,7 @@ A secondmate's routed reply returns through status or a document pointer, not by For the parent-owned correlation, recovery, and escalation contract on marked secondmate requests, see `bin/fm-pending-reply-lib.sh`. Supervise all live work under section 8. -### Selected delivery path and approval authority +### Selected delivery path and merge authority The selected delivery path owns its own rigor. When no-mistakes is selected, no-mistakes alone owns review, fixes, tests, documentation, push, PR, and CI; otherwise follow the faster path without adding an independent reviewer. @@ -317,13 +323,10 @@ The path's worker, automated gates, and captain approval remain authoritative: - **local-only** has the worker stop with a clean ready branch, then waits for the configured merge authority before firstmate uses the guarded fast-forward merge path. Delivery mode and `yolo` are orthogonal. -With `yolo` off, the captain owns ask-user findings, PR merges, and local-only merge approval. -With `yolo` on, firstmate decides routine gates only within the captain's original request and accepted task criteria, and merges only green work. -Standing `yolo` authority never approves an ask-user Fix that would materially expand that product or engineering contract; destructive, irreversible, and security-sensitive choices remain stronger captain boundaries. -Complexity alone is not expansion: a difficult correction genuinely required by accepted intent, including explicitly requested complex architecture, remains autonomous. -Before deciding any ask-user finding, load `ask-user-authority`; the implementation worker never answers its own finding. -Never merge a red PR. +`yolo` governs merge authority only: with it off, the captain approves every PR merge and every local-only landing; with it on, firstmate merges green, in-scope work itself. +Never merge a red PR under either setting; destructive, irreversible, and security-sensitive merges still escalate. Without a current explicit captain instruction that states the concrete merge, that default stands, and standing `yolo` cannot authorize a red merge; section 1 owns when such an instruction overrides a Firstmate-written standing rule within its exact scope. +Load `ask-user-authority` before deciding any ask-user finding; the implementation worker never answers its own finding. Use `bin/fm-pr-merge.sh` for every task PR merge so merge metadata is recorded, and use `bin/fm-merge-local.sh` for approved local-only landing; never call a lower-level merge command around their guards. After an autonomous merge, give the captain a one-line full-URL or local-main outcome. @@ -341,7 +344,7 @@ Custody recovery settles branch ownership, not content: the worker must replace Apart from that single supported abort, do not hand-edit, commit, restart, or start a second validation run while the obsolete run still owns the branch. Once ownership is settled, validate exactly once against that final head so no obsolete or intermediate head is ever treated as authoritative. -An ask-user finding returns as `needs-decision`; firstmate decides only when the configured authority permits, otherwise escalates to the captain. +An ask-user finding returns as `needs-decision`; firstmate loads `ask-user-authority` and either decides or escalates per that skill. Send the same worker one exact decision naming the decision key, step, action, affected finding IDs, instructions where needed, and exact response command, passing `--resolve-key` so the worker's open decision record closes at answer time. Require the matching `resolved` event, forbid `--yes`, and require the worker to process every synchronous return until completion or a genuinely new escalation. Resume fleet supervision immediately after the decision lands. @@ -356,7 +359,7 @@ The worker reports the PR when CI first becomes green rather than waiting for me For PR-based ship tasks, the ready signal depends on mode: `no-mistakes` reports `done: PR checks green` after CI is green, while `direct-PR` reports `done: PR ` after opening the PR. Run `bin/fm-pr-check.sh ` - it records `pr=` and the forge's `pr_head=` when available in the task's meta and arms the watcher's merge poll. Tell the captain the PR's full URL, always the complete `https://...` link rather than a bare `#number`, a concise outcome summary, and the no-mistakes risk level when applicable. -A captain instruction to merge is explicit authority; `yolo` is the only standing routine authority. +A captain instruction to merge is explicit authority; `yolo` is the only standing routine merge authority. For any custom `state/.check.sh` you write yourself, keep it an ordinary single-link mode-`0700` file, print one line only when firstmate should wake, print nothing otherwise, finish before `FM_CHECK_TIMEOUT`, then bind its current bytes with `bin/fm-check-register.sh ` before the watcher may execute it. Tear down a ship task only after landing is confirmed. @@ -371,7 +374,7 @@ Retire one only on an explicit captain or main-firstmate decision, after loading A completed scout must leave a self-contained report before its scratch worktree can be discarded; read and relay its findings, record the report as the Done artifact, and re-evaluate the queue. A report may recommend implementation but does not authorize it. -Before treating the investigation or any visual review as complete, load `decision-hold-lifecycle`; teardown enforces that shared completion gate. +Before treating the investigation or any visual review as complete, load `captain-hold-lifecycle`; teardown enforces that shared completion gate. When a scout's deliverable is a visual artifact the captain will iterate on, prefer keeping that scout alive to host its own Lavish loop rather than tearing it down and mediating from firstmate, so the scout keeps its investigation context and the captain iterates in one continuous session. When implementation is separately authorized, promote the existing scout through `bin/fm-promote.sh` rather than creating a duplicate task. The promoted worker must inventory scratch state, return to a clean default-branch base, carry over only intended fix changes, create the ship branch, and follow the project's selected delivery path while leaving scratch commits and debug edits behind and turning a reproduced bug into the regression test. @@ -390,6 +393,7 @@ At the start of every wake-handling turn, drain the durable wake queue before pe Session start is the only exception because its one-shot digest already presented the queue while locked or deliberately left it untouched in lock-refused read-only mode. Treat any `OPEN DECISIONS` section from the drain as actionable reconciliation input even when no wake record was queued. Treat any `UNREAD STATUS` section as newly surfaced status that must be read this turn; those lines are not re-printed after this presentation. +Treat any `RECORD DIVERGENCE` section as a contradiction between two records of one captain call, never as proof the captain ruled; load `captain-hold-lifecycle` and reconcile it in whichever direction the evidence supports. After handling all emitted wakes and reconciling the OPEN DECISIONS and UNREAD STATUS sections, run the exact generation-bound `--ack-through` command printed as `WAKE_ACK_REQUIRED`; interruption before that acknowledgement deliberately leaves the work durable for idempotent re-handling. A status line is a wake event, not current state; use `bin/fm-crew-state.sh` when current state matters, especially before re-escalating an old decision, blocker, or pause. A declared `paused:` event means a bounded external wait expected to clear on its own, while `blocked:` means firstmate action is needed. @@ -398,7 +402,7 @@ Handle actionable wakes as follows: 1. For `signal:`, read the listed event lines first, then reconcile current state only where action depends on it. 2. For `stale:`, inspect the recorded endpoint and load `stuck-crewmate-recovery` for a stopped, looping, confused, or unresponsive worker; a deep-inspection reason also requires current-state and validation-log inspection. -3. For `check:`, act on the named poll result, including merges, Relay events, and process-to-event source results. +3. For `check:`, act on the named poll result, including merges, Relay events, process-to-event source results, and captain inbox notes; a handled inbox note is also acknowledged with `bin/fm-inbox.sh drain --ack `, or it stays counted as still waiting for firstmate. 4. For `heartbeat:`, review the whole fleet from the structured fleet view, reconcile suspicious tasks and PR state, update the backlog, and never report an unchanged fleet as progress. When any wake reports a merged PR for a project cloned in this home, refresh that clone through the guarded fleet-sync path. @@ -464,7 +468,7 @@ Reach the captain immediately for: - Work ready for their review, with the full PR URL. - Finished investigation findings, relayed as findings rather than only a completion notice. -- Gate findings that require their decision under the configured authority. +- Gate findings that `ask-user-authority` escalates. - A real blocker or failure after the relevant playbook is exhausted. - Anything destructive, irreversible, or security-sensitive. - A needed credential or login. @@ -484,8 +488,9 @@ Mention cost as a courtesy when unusually much work is running, but never block `data/backlog.md` is the durable queue. It tracks work items only, never agents; persistent secondmates never appear as backlog items. Work routed to a secondmate is recorded in that secondmate home's own backlog, not the main backlog. -When a main-side thread such as a pending captain decision or relay reminder is worth durable tracking, file it as its own work item; use `tasks-axi hold --reason "" --kind captain` for a captain-gated thread. -Unresolved decisions discovered by investigations or visual reviews follow `decision-hold-lifecycle`, which owns their mandatory backlog lifecycle. +A decision is simply a task held for the captain: `tasks-axi hold --reason "" --kind captain`, with `--until ` when the captain defers it. +When a main-side thread such as a pending captain decision or relay reminder is worth durable tracking, file it as its own work item and hold it the same way. +Captain calls discovered by investigations or visual reviews follow `captain-hold-lifecycle`, which owns their completion gate and recorded-answer rules. Update the backlog on every dispatch, completion, and decision for a work item. Re-evaluate queued work after every teardown and heartbeat, dispatching items only when dependencies and time gates have cleared. @@ -526,7 +531,7 @@ These skills are not captain-invocable; load them only at their precise triggers - `bootstrap-diagnostics` - load whenever the session-start digest's bootstrap or network-checks section prints an actionable diagnostic line (`MISSING:`, `MISSING_MANUAL:`, `BACKEND_INVALID:`, `NEEDS_GH_AUTH`, `TANGLE:`, `STARTUP_MEMORY_BUDGET:`, `CREW_DISPATCH: invalid`, `FLEET_SYNC:`, `NETWORK_CHECKS:`, `PR_CHECK_MIGRATION:`, `SECONDMATE_SYNC:`, `SECONDMATE_LIVENESS:`, `SECONDMATE_HANDOFF:`, `NUDGE_SECONDMATES:`, or `FMX:`); silence and `BOOTSTRAP_INFO:` need no load. - `diagnostic-reasoning` - load before scoping a reported bug and before acting on a diagnostic report. -- `ask-user-authority` - load before deciding any ask-user finding, regardless of the project's `yolo` posture. +- `ask-user-authority` - load before deciding any ask-user finding. - `quota-array-dispatch` - load before choosing among a matched crew-dispatch profile array from current quota-axi default TOON. - `harness-adapters` - load before spawning or recovering a crewmate or secondmate, handling a trust dialog, sending a harness-specific skill invocation, interrupting or exiting an agent, resuming an exited agent, or verifying a new harness adapter. - `firstmate-orca` - load before switching to Orca, spawning or supervising Orca-backed work, smoke-testing Orca backend behavior, debugging Orca task state, or reconciling Orca-backed task metadata. @@ -534,7 +539,7 @@ These skills are not captain-invocable; load them only at their precise triggers Cloning or registering a project is add intake and uses the same trigger. - `stuck-crewmate-recovery` - load when the session-start digest reports an ordinary direct report's endpoint dead or its metadata has no window, or after a stale wake, looping pane, repeated confusion, an answered-by-brief question, an unresponsive crewmate, or a failed steer. - `secondmate-provisioning` - load before creating, seeding, validating, launching, handing backlog to, recovering, pushing inherited local material into, or retiring a secondmate home, and before editing `data/secondmates.md`. -- `decision-hold-lifecycle` - load before treating an investigation or visual review as complete, before ending a visual review that exposed a decision, and when recording or routing the captain's answer. +- `captain-hold-lifecycle` - load before treating an investigation or visual review as complete, before ending a visual review that exposed a captain decision, when recording or routing the captain's answer, and on any `RECORD DIVERGENCE` line from the wake drain. - `process-event-sources` - load before arming a long-polling source, before registering a deterministic condition->action watch (do X as soon as Y is true), and on any `procevent ` check wake. Never run a registered source's blocking command yourself in a conversational turn. - `fmx-respond` - load on an `x-mention ` `check:` wake to handle the mention, on an `x-mode-error ...` `check:` wake to report the Relay configuration blocker, on a `public-followup ...` `check:` wake or a startup-surfaced public commitment, and on any milestone or terminal wake for a Relay-linked task before posting its completion follow-up; relevant only when Relay is on. @@ -553,7 +558,7 @@ On an `x-mention ` or `x-mode-error ...` check wake, load `fmx-respo For every Relay-linked terminal outcome, load that owner and use the promised-final reconciliation when a typed public commitment exists, otherwise post the final completion follow-up before teardown. A promised final public reply is durable state, never conversation memory. -Load `fmx-respond` before promising one, on a `public-followup ...` check wake, and whenever the session-start digest lists a public commitment awaiting delivery. +Load `fmx-respond` before promising one, on a `public-followup ...` check wake, and whenever the session-start digest lists a public commitment awaiting delivery or an open public loop. Only the home holding the relay consent and thread binding ever posts it, so never ask a secondmate or crewmate to find the thread or send the reply, and never recover a terminal result by reading a `done:` sentence. ## Captain instruction precedence @@ -563,7 +568,7 @@ The instruction must be specific and recent: it must identify the concrete actio Never infer an override, broaden its scope, apply it by analogy, carry it to another object or action, or convert one request into standing authority. Ambiguous scope or conflict still requires one concise clarification before action. Destructive, irreversible, security-sensitive, discard, and merge actions still require the captain to state that concrete action explicitly; once the captain does so and higher-priority instructions permit it, a conflicting Firstmate-written rule must not rigidly block the action. -Standing `yolo` authority is not a substitute for a current explicit captain instruction where an explicit action is required. +Standing `yolo` merge authority is not a substitute for a current explicit captain instruction where an explicit action is required. ## Maintaining this file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e029bdcf9ff..907254c53b7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -92,7 +92,7 @@ There is no reliable way for `bin/fm-brief.sh`'s scaffold to detect that a task' A crewmate picking up such a brief should load the skill even if the brief predates this instruction. When supervising live crewmates, keep firstmate's own long validation or build commands in the background so watcher wakes can still be handled. Crewmate validation follows the installed no-mistakes version's SKILL.md and live `axi` help instead of duplicating gate mechanics in firstmate docs. -Firstmate's wrapper still matters: crewmates route every `ask-user` finding to firstmate, which applies the authority contract in `AGENTS.md`, and crewmates avoid `--yes` because it would bypass that check and any required captain escalation. +Firstmate's wrapper still matters: crewmates route every `ask-user` finding to firstmate, which applies `ask-user-authority`, and crewmates avoid `--yes` because it would bypass that check and any required captain escalation. `.no-mistakes.yaml` publishes test evidence to the orphan `no-mistakes/evidence` branch, which shares no history with code branches, and pins the gate's lint command to `bin/fm-lint.sh`, matching the Linux CI lint job. Local no-mistakes Test is intent-targeted and must not re-run every `tests/*.test.sh`; `.github/workflows/ci.yml` owns the broad behavior suite plus platform-specific compatibility lanes. The pipeline publishes that evidence itself, so never hand-commit `.no-mistakes/` paths onto a feature branch; CI rejects them as tracked personal fleet paths. @@ -129,6 +129,8 @@ Family selection is the ordinary local path; `--all` is deliberate full regressi CI owns broad regression across required portable parallel shards, the portable serial lane's separate-runner shards, the Herdr lane, lint, invariants, the coverage guard, and stock macOS Bash compatibility in [`.github/workflows/ci.yml`](.github/workflows/ci.yml). Use `bin/fm-test-run.sh --list-lanes` for exact lane names and `--help` for `--jobs` rules and required gate-skip flags when reproducing a lane locally. Discover tests by listing `tests/*.test.sh`: each is a self-contained bash script named `.test.sh`, and its header comment describes what it covers, so pass one to `bin/fm-test-run.sh` to focus on a subject with canonical timing output. +A fixture may shorten a production timeout to keep a failure path prompt, but never below what the real work inside that window costs on a loaded machine: a fork, an exec, a lock acquisition, a beacon publication, or a first-poll check. +Where a case's assertion is not about the timeout itself, give that window headroom over the measured loaded cost, and bound the test's own waiting with iteration-counted poll loops, which stretch under load where a wall-clock budget does not. Tests that need a real optional backend or an explicit opt-in (real herdr/zellij/cmux smoke tests, the live Pi regression) skip themselves and print the tool or environment gate needed to enable them, so the portable suite remains safe on machines without those tools. The [Herdr backend guide](docs/herdr-backend.md#destructive-lab-safety) owns the lane's isolation boundary, while [runtime backend verification](docs/verification/runtime-backends.md#herdr) owns active empirical evidence; live harness credential tests remain opt-in. diff --git a/README.md b/README.md index 8ed5226b171..ea321a8f8cf 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ Launching a supported harness inside it instantiates your first mate - and makes - **A visible crew** - every crewmate works in its own tmux window, experimental herdr/zellij tab, cmux workspace, or Orca terminal you can watch or type into; the first mate reconciles. - **Disposable worktrees** - each task runs in a clean [treehouse](https://github.com/kunchenguid/treehouse) git worktree, or an Orca-managed worktree when `backend=orca`, so parallel work on one repo never collides. - **Two task shapes** - ship tasks deliver authorized changes; scout tasks leave standalone investigation reports when the intake contract warrants separate research. -- **Explicit project modes** - each project ships via `no-mistakes`, `direct-PR`, or `local-only`, with an optional `+yolo` autonomy flag. +- **Explicit project modes** - each project ships via `no-mistakes`, `direct-PR`, or `local-only`, with an optional `+yolo` merge-autonomy flag. - **Optional secondmates** - opt in to persistent second mates that run from isolated firstmate homes with their own `FM_HOME`, state, projects, and session lock, either locally or as a whole home on an SSH-reachable host, with guarded updates and recovery that never turns an unavailable remote route into a local replacement. - **Event-driven, zero-token supervision** - a bash watcher sleeps on the fleet and wakes the first mate only when something needs you; verified primary harnesses also get a turn-end backstop that blocks or follows up on a blind stop when work is under way and supervision is not live. - **Optional Relay** - opt in with one local `.env` pairing token so firstmate can answer your public mentions on X and Discord alike, act on normal reversible mention requests through the same lifecycle as chat requests, acknowledge spawned work, and post up to three public-safe completion follow-ups within seven days for genuine milestones and the final outcome without changing non-Relay behavior; a final reply promised in a thread becomes durable state that is reconciled from disk, so a restart or a compacted conversation cannot lose it; dry-run preview records would-be replies and dismissals locally before go-live. @@ -202,6 +202,7 @@ Firstmate's skills live in two separate places with different audiences: - [docs/configuration.md](docs/configuration.md) - environment variables, `FM_HOME`, runtime backend selection, optional Relay and its X and Discord setup steps, the files you set, and harness support. - [docs/remote-secondmates.md](docs/remote-secondmates.md) - current setup, routing, transfer, recovery, and safety behavior for whole-home remote second mates. - [docs/calm.md](docs/calm.md) - current Pi `/calm` behavior and supported presentation limits. +- [docs/voice-relay.md](docs/voice-relay.md) - the optional spoken interface: setup on both machines, measured round-trip cost, what a spoken answer may read, and what this build does not do yet. - [docs/wedge-alarm.md](docs/wedge-alarm.md) - configure the active alert for an away-mode escalation delivery that gets stuck. - [docs/tmux-backend.md](docs/tmux-backend.md) - current setup and limits for the tmux reference backend. - [docs/herdr-backend.md](docs/herdr-backend.md) - current setup, safety boundaries, and limits for the experimental Herdr backend. @@ -210,7 +211,7 @@ Firstmate's skills live in two separate places with different audiences: - [docs/cmux-backend.md](docs/cmux-backend.md) - current setup, socket security, and limits for the experimental cmux backend. - [docs/codex-app-backend.md](docs/codex-app-backend.md) - the current blocked Codex App backend boundary and rollout contract. - [docs/verification/runtime-backends.md](docs/verification/runtime-backends.md) - active maintainer verification for runtime backend guarantees. -- [docs/gitlab-merge-watch.md](docs/gitlab-merge-watch.md) - maintainer verification for GitLab merge watching on arbitrary instances. +- [docs/gitlab-merge-watch.md](docs/gitlab-merge-watch.md) - maintainer verification for watching and merging GitLab merge requests on arbitrary instances. - [docs/turnend-guard.md](docs/turnend-guard.md) - the primary session's current "no turn ends blind" backstop, scope, loop safety, and compatibility limits. - [docs/verification/supervision.md](docs/verification/supervision.md) - active maintainer verification for session-start, guard, continuity, and wedge integrations. - [docs/supervision-protocols/](docs/supervision-protocols/) - rendered primary-harness watcher protocols for Claude, Codex, OpenCode, Pi and `pi-signed`, Grok, Cursor, and unknown harness fallback. diff --git a/bin/fm-afk-return.sh b/bin/fm-afk-return.sh index b38c1e07c4e..cf5addb24cf 100755 --- a/bin/fm-afk-return.sh +++ b/bin/fm-afk-return.sh @@ -10,9 +10,8 @@ # `blocked:` is the crewmate protocol's firstmate-actionable verb. A live task's # open blocked event must be remediated and closed with `resolved [key=...]`, or # explicitly reclassified in the status stream with a durable reason, before an -# ordinary captain request may proceed. `needs-decision:` belongs to the -# configured approval authority and is deliberately not part of this blocker -# gate; normal reporting routes it through the AGENTS.md section 7 contract. +# ordinary captain request may proceed. `needs-decision:` is deliberately not +# part of this blocker gate. # # The durable state/.afk-return-catchup file is written BEFORE daemon shutdown, # so a crash between stopping, wake presentation, and blocker handling fails closed. diff --git a/bin/fm-backlog-handoff.sh b/bin/fm-backlog-handoff.sh index f7736553348..97bda75c331 100755 --- a/bin/fm-backlog-handoff.sh +++ b/bin/fm-backlog-handoff.sh @@ -26,9 +26,10 @@ # already present in the secondmate backlog is reported and skipped, and if # any key matches neither backlog nothing is moved; # - warning, after a successful move, when a moved key still owes a public -# relay reply bound to main/, because that binding no longer names the -# home that owns the work. The move is not blocked: rebinding the commitment -# to secondmate: is a relay-side decision the caller makes. +# relay reply bound to main/, or when this home has an open public loop +# with nothing owed, because routing work out does not close that loop. The +# move is not blocked: rebinding or rechain is a relay-side decision the +# caller makes. # # What `tasks-axi mv ... --to ` owns: moving each full item BLOCK # byte-exact (header, body lines, blank separators, and indented pseudo-headings @@ -58,6 +59,7 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" +STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" REG="$DATA/secondmates.md" MAIN_BACKLOG="$DATA/backlog.md" # shellcheck source=bin/fm-tasks-axi-lib.sh disable=SC1091 @@ -66,6 +68,8 @@ MAIN_BACKLOG="$DATA/backlog.md" . "$SCRIPT_DIR/fm-secondmate-registry-lib.sh" # shellcheck source=bin/fm-wake-lib.sh . "$SCRIPT_DIR/fm-wake-lib.sh" +# shellcheck source=bin/fm-public-followup-lib.sh +. "$SCRIPT_DIR/fm-public-followup-lib.sh" ACTIVE_HANDOFF_LOCK= ACTIVE_REGISTRY_LOCK= @@ -288,6 +292,10 @@ warn_stale_public_commitments() { # ... printf 'warning: %s still owes a public reply bound to main/%s; rebind it to secondmate:%s (tasks-axi public-followup bind-work, then bin/fm-public-followup.sh register --relation --work-home secondmate:%s --work-id %s --generation ) or the promised reply will be reconciled against work this home no longer owns.\n' \ "$key" "$key" "$id" "$id" "$key" >&2 done + if fm_pf_relay_active "$FM_HOME" && fm_pf_has_delivered_open_loops "$STATE"; then + printf 'warning: this home has an open public loop with nothing owed; routing work to secondmate:%s does not close it. Hand it on with bin/fm-public-followup.sh rechain or close it with retire --reason.\n' \ + "$id" >&2 + fi # Reporting never changes the handoff's own success: the move already landed. return 0 } diff --git a/bin/fm-bearings-board.sh b/bin/fm-bearings-board.sh index 008b714b805..e8ce4309566 100755 --- a/bin/fm-bearings-board.sh +++ b/bin/fm-bearings-board.sh @@ -15,13 +15,14 @@ # template at the stable board path. Establish or resume the Lavish # session on that board BEFORE binding and arming its answer source, # so a registered poll can never race a session that does not exist. -# Bind to the any-origin keyed-answer intake ALWAYS precedes arm, so -# the board can never produce an answer that has nowhere to go -# (decision-hold-lifecycle's ordering rule, enforced here rather -# than left to agent memory). Output starts with `board: `, -# then includes lavish-axi's session output and the remaining status: +# Bind to the keyed-answer intake (bin/fm-captain-hold.sh) ALWAYS +# precedes arm, so the board can never produce an answer that has +# nowhere to go (captain-hold-lifecycle's ordering rule, enforced +# here rather than left to agent memory). Output starts with +# `board: `, then includes lavish-axi's session output and +# the remaining status: # served: -# bound: (any-origin) +# bound: # armed: (first registration) # already-armed: (registration already present) # path Print the stable board path for this home. @@ -96,6 +97,7 @@ validate_payload() { # and (optional_string("detail")) and (optional_https_url("pr_url")) and (optional_string("freeform_hint")) + and ((has("close") | not) or (.close == "done" or .close == "release")) and ((has("allow_freeform") | not) or (.allow_freeform | type == "boolean")) and ((has("recommend_value") | not) or ((.recommend_value | slug(128)) @@ -177,9 +179,9 @@ command_build() { sid=$("$SCRIPT_DIR/fm-procevent-lavish.sh" source-id "$board") \ || fail "cannot derive the board source id" - "$SCRIPT_DIR/fm-decision-hold.sh" bind "$sid" --any-origin >/dev/null \ - || fail "cannot bind the board source to the any-origin intake" - printf 'bound: %s (any-origin)\n' "$sid" + "$SCRIPT_DIR/fm-captain-hold.sh" bind "$sid" >/dev/null \ + || fail "cannot bind the board source to the keyed-answer intake" + printf 'bound: %s\n' "$sid" if "$SCRIPT_DIR/fm-procevent.sh" list | awk 'NR > 1 { print $1 }' | grep -Fxq "$sid"; then printf 'already-armed: %s\n' "$sid" diff --git a/bin/fm-bearings-snapshot.sh b/bin/fm-bearings-snapshot.sh index 5a23bec3671..c64f4226dbb 100755 --- a/bin/fm-bearings-snapshot.sh +++ b/bin/fm-bearings-snapshot.sh @@ -22,6 +22,12 @@ # This wrapper consumes canonical status decisions plus canonically normalized # backlog roles, unresolved blockers, and captain actionability. It never infers # decisions from report or visual-review prose or reimplements snapshot semantics. +# Captain's Call is captain actionability itself: every due, unblocked task held +# for the captain, whatever its kind. A captain hold deferred by date +# (hold-until in the future) is not actionable and renders as a Charted Next +# gate with its date; a row the canonical snapshot marks prose-deferred +# (deferred_marker) leaves the default decisions and gates views and is +# disclosed in omitted[], revealed by --all-decisions / --all-queued. # # Main-home inventory validity comes from the canonical snapshot's main_inventory # object (orphan structured in-flight without meta, unstructured current rows). @@ -271,9 +277,15 @@ EOF fi # --- projection: canonical snapshot -> fm-bearings.v1 model (JSON) ---------- +BEARINGS_TODAY=${NOW%%T*} +case "$BEARINGS_TODAY" in + [0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]) : ;; + *) BEARINGS_TODAY=$(date -u +%Y-%m-%d) ;; +esac MODEL=$(printf '%s' "$SNAP" | jq \ --arg home "$HOME_LABEL" \ --arg now "$NOW" \ + --arg today "$BEARINGS_TODAY" \ --arg prs "$PR_STATUS" \ --arg fields "$FIELDS" \ --argjson landed_n "$FM_BEARINGS_LANDED" \ @@ -312,7 +324,7 @@ MODEL=$(printf '%s' "$SNAP" | jq \ | (($fl | index("paths")) != null) as $f_paths | (($fl | index("actions")) != null) as $f_actions | (($fl | index("endpoints")) != null) as $f_endpoints - | ([ .backlog.records[] | select(.state == "done" and .structured and .kind != "captain") + | ([ .backlog.records[] | select(.state == "done" and .structured and .hold_kind != "captain") | {id, title, pr_url, report_path, local_note, completion, home:"(main)", home_id:"(main)"} ]) as $main_done | ((.secondmate_landed.records) // []) as $mate_done | ($main_done + $mate_done) as $all_landed_rows @@ -334,7 +346,8 @@ MODEL=$(printf '%s' "$SNAP" | jq \ | select(.endpoint.exists == false or .endpoint.agent_alive == "dead") | {id:($m.id + "/" + .id),backend:"secondmate-home",target:(.endpoint.target // "-"),exists:.endpoint.exists,agent:.endpoint.agent_alive} ]) as $unhealthy_all | ([ (.secondmate_current.records // [])[] - | ([.decisions_open[]? | select(.source == "backlog" and .verb == "captain-hold")]) as $captain_holds + | ([.decisions_open[]? | select(.source == "backlog" and .verb == "captain-hold" + and .deferred_marker != true)]) as $captain_holds | ([.holds[]? | select(.source == "backlog")]) as $backlog_holds | . + { bearings_captain_holds:$captain_holds, @@ -381,12 +394,19 @@ MODEL=$(printf '%s' "$SNAP" | jq \ doing:([.active_children[] | .id + ": " + (.doing // .state)] | join("; ") | trunc(90))} ]) as $in_flight_all | ([ .backlog.records[] | select(.structured and .captain_actionable == true) + | select(($all_decisions == 1) or (.deferred_marker != true)) | {id,key:.id,verb:"captain-hold", summary:((.title + ": " + .hold_reason) | trunc(90)),owner:"(main)"} ] + [ (.secondmate_current.records // [])[] as $m | $m.decisions_open[]? | select(.source == "backlog" and .verb == "captain-hold") + | select(($all_decisions == 1) or (.deferred_marker != true)) | {id:($m.id + "/" + .id),key,verb, summary:(((.summary // .id) + ": " + (.reason // "captain decision pending")) | trunc(90)),owner:$m.id} ]) as $decisions_all + | ([ .backlog.records[] + | select(.structured and .captain_actionable == true and .deferred_marker == true) ] + + [ (.secondmate_current.records // [])[] | .decisions_open[]? + | select(.source == "backlog" and .verb == "captain-hold" and .deferred_marker == true) ] + | length) as $decisions_marked_deferred | ((if (.main_inventory.valid == false) then [{id:"(main-inventory)", title:((.main_inventory.reason // "main inventory invalid") | trunc(60)), @@ -400,18 +420,24 @@ MODEL=$(printf '%s' "$SNAP" | jq \ (.state == "queued" or (.state == "in_flight" and .current_role == "held" and ($working_ids | index($record.id) | not)))) | select(.captain_actionable != true) - | select(($all_queued == 1) - or (((.body_excerpt // "") | test("SUPERSEDED|NOT REQUIRED|NOT-REQUIRED|DEFERRED"; "i")) | not)) + | select(($all_queued == 1) or (.deferred_marker != true) + or ((.hold_until // null) != null and .hold_until > $today)) | {id, title:(.title | trunc(60)), blocked_by:((.unresolved_blocker_ids // []) | if length > 0 then join(",") else "-" end | trunc(120)), - reason:((.hold_reason // .blocked_reason // "-") | trunc(40)),owner:"(main)"} ] + reason:((if (.hold_until // null) != null and .hold_until > $today + then ("until " + .hold_until + ": " + (.hold_reason // .blocked_reason // "-")) + else (.hold_reason // .blocked_reason // "-") end) | trunc(40)),owner:"(main)"} ] + [ (.secondmate_current.records // [])[] as $m | select($m.provenance.selected == "structured-home") | $m.queued[]? | select(.captain_actionable != true) + | select(($all_queued == 1) or (.deferred_marker != true) + or ((.hold_until // null) != null and .hold_until > $today)) | {id,title:(.title | trunc(60)), blocked_by:((.unresolved_blocker_ids // []) | if length > 0 then join(",") else "-" end | trunc(120)), - reason:((.hold_reason // .blocked_reason // "-") | trunc(40)),owner:$m.id} ]) as $gates_all + reason:((if (.hold_until // null) != null and .hold_until > $today + then ("until " + .hold_until + ": " + (.hold_reason // .blocked_reason // "-")) + else (.hold_reason // .blocked_reason // "-") end) | trunc(40)),owner:$m.id} ]) as $gates_all | ([ .scout_reports[] | . as $r | select(($all_reports == 1) or (($rel_ids | index($r.id)) != null)) @@ -446,7 +472,7 @@ MODEL=$(printf '%s' "$SNAP" | jq \ (if $f_actions then empty else {surface:"watch/steer actions", reveal:"--fields actions"} end), (if $f_endpoints then empty else {surface:"healthy endpoint detail", reveal:"--fields endpoints"} end), (if $all_reports == 1 then empty else {surface:"full scout-report inventory", reveal:"--all-reports"} end), - (if $all_queued == 1 then empty else {surface:"superseded queued items", reveal:"--all-queued"} end), + (if $all_queued == 1 then empty else {surface:"superseded or prose-deferred queued items", reveal:"--all-queued"} end), (if $all_landed == 0 and ($per_home_capped | length) > ($done | length) then {surface:("landed showing \($done | length) of \($per_home_capped | length)" + (($done | map(.home_id) | unique | map(select(. != "(main)")) | length) as $k | if $k > 0 then " (incl. \($k) secondmate home(s))" else "" end)), reveal:"--all-landed"} else empty end), (if $all_landed == 0 and $home_cap_dropped > 0 then {surface:("landed per-home capped at \($landed_per_home_n) for \($home_cap_dropped) home(s)"), reveal:"--all-landed"} else empty end), (if (($snap.secondmate_landed.unreadable // []) | length) > 0 then {surface:("secondmate home(s) with unreadable backlog: \(($snap.secondmate_landed.unreadable // []) | length)"), reveal:"inspect the listed secondmate home backlogs"} else empty end), @@ -464,6 +490,7 @@ MODEL=$(printf '%s' "$SNAP" | jq \ (([($snap.secondmate_current.records // [])[] | select(.parent_event.activity_scan.input_truncated == true or .parent_event.activity_scan.retained_truncated == true)] | length) as $n | if $n > 0 then {surface:("secondmate parent activity evidence truncated for \($n) record(s)"), reveal:"raise FM_SNAPSHOT_PARENT_ACTIVITY_LINES, FM_SNAPSHOT_PARENT_ACTIVITY_BYTES, or FM_SNAPSHOT_PARENT_ACTIVITIES"} else empty end), (([($snap.secondmate_current.records // [])[] | select(.parent_event.activity_scan.available == false)] | length) as $n | if $n > 0 then {surface:("secondmate parent activity evidence unavailable for \($n) record(s)"), reveal:"inspect the parent status logs"} else empty end), (if $all_decisions == 0 and ($decisions_all | length) > $decisions_n then {surface:("decisions_open showing \($decisions_n) of \($decisions_all | length)"), reveal:"--all-decisions"} else empty end), + (if $all_decisions == 0 and $decisions_marked_deferred > 0 then {surface:("captain holds marked deferred or superseded: \($decisions_marked_deferred)"), reveal:"--all-decisions"} else empty end), (if $all_queued == 0 and ($gates_all | length) > $gates_n then {surface:("gates showing \($gates_n) of \($gates_all | length)"), reveal:"--all-queued"} else empty end), (if $all_reports == 0 and ($reports_all | length) > $reports_n then {surface:("reports showing \($reports_n) of \($reports_all | length)"), reveal:"--all-reports"} else empty end), (if $all_recorded_prs == 0 and ($recorded_prs_all | length) > $recorded_prs_n then {surface:("recorded_prs showing \($recorded_prs_n) of \($recorded_prs_all | length)"), reveal:"--all-recorded-prs"} else empty end), diff --git a/bin/fm-brief.sh b/bin/fm-brief.sh index 206e5a947ae..3528fd3866b 100755 --- a/bin/fm-brief.sh +++ b/bin/fm-brief.sh @@ -43,7 +43,7 @@ # Ship briefs begin with a worktree-isolation assertion before the branch step. # --mode is refused on scout and secondmate scaffolds: a scout's deliverable is a # report rather than a merge, and a charter is not a delivery contract. -# There is no --yolo flag here. The worker never owns approval decisions, so yolo is +# There is no --yolo flag here. The worker never owns merge decisions, so yolo is # a spawn-time and firstmate-side input only (AGENTS.md section 7). # Every scaffold's status protocol distinguishes the configured # declared-external-wait verb (FM_CLASSIFY_PAUSED_VERB, default "paused") from @@ -127,10 +127,10 @@ for a in "$@"; do --no-projects) NO_PROJECTS=1 ;; --mode) want_value=mode ;; --mode=*) MODE=${a#--mode=}; MODE_SET=1 ;; - # yolo never reaches the worker: it is firstmate's approval authority, not a + # yolo never reaches the worker: it is firstmate's merge authority, not a # brief input. Refuse it loudly so it is never silently dropped here and then # believed to have been recorded. - --yolo|--yolo=*) echo "error: --yolo is not a brief input; pass it to bin/fm-spawn.sh, which records the task's approval posture" >&2; exit 1 ;; + --yolo|--yolo=*) echo "error: --yolo is not a brief input; pass it to bin/fm-spawn.sh, which records the task's merge posture" >&2; exit 1 ;; *) POS+=("$a") ;; esac done @@ -229,7 +229,7 @@ Marked requests also carry a privacy-safe \`corr=\` token after the marker; Optional helper: \`bin/fm-secondmate-report.sh\` can append a correlated status line for you, but a plain \`echo\` that includes the same \`corr=\` is equally valid - do not depend on the helper being present. For a terse result, a status line is the whole answer. For a detailed answer (an investigation, a plan, an audit), write it to a doc under your home's \`data/\` and append a status line that points to that doc - the scout-report pattern - so the main firstmate is woken and can read it. -Before treating an investigation or visual review as complete, load \`decision-hold-lifecycle\` from this home's \`.agents/skills/\` and pass its shared completion gate. +Before treating an investigation or visual review as complete, load \`captain-hold-lifecycle\` from this home's \`.agents/skills/\` and pass its shared completion gate. A message with NO marker is the captain typing directly into your pane: treat it as authoritative captain intervention and stay conversational exactly as you would for any captain message; do not force it onto the status path. # Escalation to main firstmate @@ -291,7 +291,7 @@ HERDR_SECTION=$(printf '%s\n' \ else IFS= read -r -d '' HERDR_SECTION <<'EOF' || true # Herdr lifecycle declaration - NOT ENABLED -**HARD SAFETY GATE:** this scaffold cannot inspect the task text that replaces `{TASK}` later. +**HARD SAFETY GATE:** this scaffold cannot inspect the task text filled in above. If the task will start, stop, delete, restart, profile, or otherwise drive Herdr lifecycle behavior, stop and regenerate the brief with `--herdr-lab` before dispatch. Do not add Herdr lifecycle commands to this unguarded brief by hand. EOF @@ -340,7 +340,7 @@ The report is the only thing that survives, so anything worth keeping must be in Write your findings to \`$DATA/$ID/report.md\`. The report must stand alone: what you did, what you found, the evidence (commands run, output, file:line references), and what you recommend. If your deliverable is a visual artifact the captain will review and iterate on, you may host the Lavish review loop yourself (poll, revise, re-serve, staying alive) instead of handing it back to firstmate. -Before reporting done, read and follow \`$FM_ROOT/.agents/skills/decision-hold-lifecycle/SKILL.md\` and pass its shared completion gate for the report and any visual review. +Before reporting done, read and follow \`$FM_ROOT/.agents/skills/captain-hold-lifecycle/SKILL.md\` and pass its shared completion gate for the report and any visual review. When the report is complete, append \`done: {one-line conclusion}\` to the status file and stop. If your findings reveal work that should ship (e.g. you reproduced a bug and the fix is clear), say so in the report; firstmate may promote this task in place, and you would then receive mode-specific ship instructions as a follow-up message. EOF @@ -396,7 +396,7 @@ Do not hand-edit, commit, or fix findings yourself while a run is active - the p Two firstmate-specific rules layer on top of that guidance: - ask-user findings are never yours to answer: escalate to firstmate (rule 6) and stop. - Firstmate applies the authority contract in its \`AGENTS.md\` and obtains any required captain decision. + Firstmate applies \`ask-user-authority\` and obtains any required captain decision. When the decision comes back, feed it to the gate with \`no-mistakes axi respond\` and let the pipeline apply it - do not route the question to "the user" or implement the fix yourself. - Avoid \`--yes\`: it would silently bypass firstmate's authority check and any required captain escalation. @@ -446,7 +446,7 @@ $RULE1 cadence instead of treating it as a possible wedge. Use \`blocked:\` when you are stuck and need help. 5. If you hit the same obstacle twice, append \`blocked: {why}\` and stop; firstmate will help. 6. If a decision belongs above the implementation worker (product choices, destructive actions, ask-user findings), - append \`needs-decision: {summary of options}\` and stop. Firstmate will apply the configured authority and reply with the decision. + append \`needs-decision: {summary of options}\` and stop. Firstmate will reply with the decision. A decision or blocker you opened stays open until a \`resolved\` line carrying its exact key lands; a later \`done:\` or \`working:\` line never closes it, even when the answer is what started that work. Firstmate's reply normally writes that closing line at answer time; when a blocker or wait clears WITHOUT a firstmate reply, append \`resolved: {how it cleared}\` yourself (same \`[key=]\` if you opened it with one) as you resume. 7. Never stop, restart, or update the shared \`no-mistakes\` daemon - it is one instance serving diff --git a/bin/fm-busy-event.sh b/bin/fm-busy-event.sh index 51896dc1c15..0abcab8ee39 100755 --- a/bin/fm-busy-event.sh +++ b/bin/fm-busy-event.sh @@ -95,6 +95,19 @@ REC=$(fm_busy_record_path "$STATE" "$ID") GEN_FILE=$(fm_busy_gen_path "$STATE" "$ID") LOCK="$REC.lock" +# Portable mtime in epoch seconds. macOS (BSD) stat uses `-f `; Linux (GNU) +# stat uses `-c `. Do NOT collapse this into `stat -f ... || stat -c +# ...`: on GNU `-f` is *filesystem* stat, so it reads the format string as +# a path, reports that on stderr, prints a partial filesystem dump (" File: +# ...") on stdout, and still exits 0 - the fallback never runs and the caller +# gets a non-numeric token. Detect the platform once and pick the right form, +# exactly as bin/fm-watch.sh does. +if [ "$(uname)" = Darwin ]; then + lock_mtime() { stat -f %m "$1" 2>/dev/null; } +else + lock_mtime() { stat -c %Y "$1" 2>/dev/null; } +fi + # Serialize writers. The lock protects seq advancement and the sidecar/record # pair; a holder that died mid-write is broken after FM_BUSY_LOCK_STALE_SECS. lock_acquire() { @@ -103,7 +116,11 @@ lock_acquire() { tries=$((tries + 1)) if [ "$tries" -ge 40 ]; then now=$(date +%s) - mtime=$(stat -f %m "$LOCK" 2>/dev/null || stat -c %Y "$LOCK" 2>/dev/null || echo "$now") + mtime=$(lock_mtime "$LOCK" || true) + # Anything unreadable or non-numeric reads as "just created", so an + # unforeseen stat surprise degrades to a lock-timeout refusal instead of + # aborting the writer - and its caller, fm-teardown.sh - under `set -u`. + case "$mtime" in ''|*[!0-9]*) mtime=$now ;; esac age=$((now - mtime)) if [ "$age" -ge "${FM_BUSY_LOCK_STALE_SECS:-5}" ]; then rmdir "$LOCK" 2>/dev/null || rm -rf "$LOCK" 2>/dev/null || true diff --git a/bin/fm-captain-hold.sh b/bin/fm-captain-hold.sh new file mode 100755 index 00000000000..cb429d95238 --- /dev/null +++ b/bin/fm-captain-hold.sh @@ -0,0 +1,1001 @@ +#!/usr/bin/env bash +# fm-captain-hold.sh - deterministic mechanics for tasks held for the captain. +# +# The semantic policy is owned once by +# .agents/skills/captain-hold-lifecycle/SKILL.md. This script never reads +# report, visual-review, chat, or terminal prose to guess whether the captain +# owes an answer. The invoking agent decides what is genuinely waiting on the +# captain; this script supplies guarded creation, a durable record of what the +# captain actually said, the investigation completion gate, and the one +# keyed-answer intake every channel feeds. +# +# There is no separate decision type. A captain call is an ordinary backlog +# task held for the captain (`tasks-axi hold --kind captain`), and its +# identity is simply the task id. Older installs created derived +# `-decision-` identities through bin/fm-decision-hold.sh; those +# rows are already plain task ids, so they keep working here unchanged, and +# the legacy inputs noted below resolve them without a migration. +# All backlog mutations run in the active FM_HOME, which keeps main-home and +# secondmate-home ownership aligned with the work that discovered the call. +# +# Usage: +# fm-captain-hold.sh hold --reason \ +# [--title ] [--repo <repo>] [--origin <origin-id>] [--until YYYY-MM-DD] +# fm-captain-hold.sh answer <task-id> --decision-file <path> [--release] +# fm-captain-hold.sh answers [<legacy-origin> | --any-origin] --source <provenance> (keyed answers on stdin) +# fm-captain-hold.sh bind <source-id> [<legacy-origin> | --any-origin] +# fm-captain-hold.sh unbind <source-id> +# fm-captain-hold.sh binding <source-id> +# fm-captain-hold.sh complete <origin-id> (--none | <task-id>...) +# fm-captain-hold.sh verify <origin-id> +# fm-captain-hold.sh diverged +# +# `hold` places an existing task under an active captain hold, or creates the +# task first when no work item exists to hold (--title required to create; the +# optional --origin records provenance in the new task's body and supplies the +# default repo from that origin's metadata). Prefer holding the work item the +# question gates over minting a new row. Repeating `hold` with the same id is +# idempotent; a task already closed is refused rather than reopened. `--until` +# records the captain's own deferral date through `tasks-axi hold --until`, so +# a "revisit later" answer is stored as a date instead of a live card. +# +# `answer` records the captain's exact words and closes the call in the same +# act. It requires a non-empty captain decision file of at most 8192 bytes, +# writes a resolution block at the top of the task body (the previous body is +# preserved below the block and archived through tasks-axi --archive-body), +# then closes the task with `tasks-axi done` - or, with `--release`, lifts the +# hold with `tasks-axi unhold` so a captain-gated WORK item resumes instead of +# closing. An exact retry is idempotent only when its requested close mode +# matches the newest record; a changed decision or a mode mismatch is rejected. +# A re-held task may record a new answer on top. On a task already closed outside this script, +# `answer` records the missing resolution block (the old `repair` path) only +# when the task still carries the captain-hold provenance tasks-axi preserves +# through a close, so an ordinary finished task cannot be dressed up as an +# answered captain call. A hold that expired by date (`--until` in the past) is +# still answerable: the surviving hold annotations, not tasks-axi's live +# `held:` bit, prove the captain owned it. +# +# ONE KEYED-ANSWER INTAKE, FED BY EVERY CHANNEL. +# "A keyed answer closes its matching captain-held task" is a single +# capability, owned here and nowhere else. `answers` reads +# `<task-id>\t<answer>\t<label>[\t<mode>]` lines on stdin and closes each named +# task through the very same `answer` path above, so every guard applies +# identically no matter which channel the answer arrived on. The key IS the +# task id - no identity arithmetic. The optional fourth field selects the close: +# empty or `done` completes the task, `release` lifts the hold so held work +# resumes; anything else is skipped. A key that names no task, a task that is +# not held for the captain, or a task already closed is reported as `skipped:` +# and feeds nothing. A replayed delivery whose answer digest and requested +# close mode both match the newest record is reported `closed:` and is a no-op; +# a mode mismatch is skipped. The command exits nonzero when any key was +# skipped. `--source` is provenance text recorded in the +# durable decision, never a behavior switch: this command has no per-channel +# branch and no knowledge of chat, review decks, or any transport. +# Legacy input: an optional positional origin (or a stored concrete-origin +# binding) makes a key that names no task fall back to the old +# `<origin>-decision-<key>` identity, so an in-flight pre-collapse channel +# keeps closing its rows; `--any-origin` and the stored `(any)` marker mean +# what an absent origin means and are accepted for the same reason. +# +# A channel's ONLY job is to turn whatever it received into those keyed lines +# and pipe them here. It must never map keys to tasks, build decision records, +# choose a close mode beyond what its card declared, or close anything itself. +# +# `bind`, `unbind`, and `binding` record that a captured-answer SOURCE feeds +# this intake, for any channel whose answers arrive detached from their origin +# (a process-event source id, for example). The binding is a private record +# under `state/decision-bindings/`; a source with no binding feeds nothing, so +# this whole path is opt-in per source and an unbound source behaves as if it +# did not exist. `bind` deliberately does not require the source to exist yet, +# so a channel can be bound BEFORE it is armed. The optional second argument +# exists only for legacy pre-collapse records and callers: a concrete origin is +# stored verbatim and used as the composition fallback above, and +# `--any-origin` stores the same `(any)` marker a plain `bind <source-id>` +# stores. `binding` prints the stored value verbatim and `answers` accepts it, +# so the process-event runner's feed seam is unchanged. +# +# `complete` is the shared investigation and visual-review completion gate. +# It attests, in the origin task's metadata, the reviewed inventory of +# captain-held tasks that carry the origin's unresolved captain calls. +# `--none` is an explicit semantic attestation that the just-reviewed surface +# has no unresolved captain call, and is refused while the origin still has an +# open keyed status decision. With a non-empty inventory, every listed task is +# verified durable (actively captain-held, or closed with a recorded answer), +# the inventory is unioned idempotently into the metadata, and every still-open +# keyed status decision is transferred to its durable owner with a +# `captain-held [key=...]` status close naming the inventory. Later review +# passes may add ids. A post-teardown visual review can complete against the +# surviving report and tasks without recreating task state. +# `verify` is read-only and is called by scout teardown, so teardown cannot +# erase a source before this gate has succeeded: every recorded inventory +# entry must still be durable and no keyed status decision may be open. +# Metadata compatibility: the attestation keeps the historical +# `decisions_reviewed=1` and `decision_keys=` keys, and an inventory entry that +# names no existing task resolves through the legacy `<origin>-decision-<entry>` +# identity, so pre-collapse metadata written by fm-decision-hold.sh verifies +# unchanged. An entry that exists as a task id is always that task. +# +# `diverged` is the read-only guard over the seam between the two records of +# one captain call. See "record divergence" beside command_diverged below. +# +# Resolution records: the block written into the body names this script, the +# decision digest, and a `Resolution mode:` of answered, released, or repaired. +# Records written by the retired fm-decision-hold.sh (routed, declined, +# answered, repaired) are recognized everywhere a record is read, so nothing +# already closed needs rewriting. +set -eu + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" +FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" +STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" +DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" + +# shellcheck source=bin/fm-classify-lib.sh +# shellcheck disable=SC1091 +. "$SCRIPT_DIR/fm-classify-lib.sh" +# shellcheck source=bin/fm-tasks-axi-lib.sh +# shellcheck disable=SC1091 +. "$SCRIPT_DIR/fm-tasks-axi-lib.sh" +# shellcheck source=bin/fm-wake-lib.sh +# shellcheck disable=SC1091 +. "$SCRIPT_DIR/fm-wake-lib.sh" + +CAPTAIN_META_LOCK= +CAPTAIN_META_LOCK_HELD=0 +captain_hold_cleanup() { + if [ "$CAPTAIN_META_LOCK_HELD" = 1 ]; then + fm_lock_release "$CAPTAIN_META_LOCK" || true + CAPTAIN_META_LOCK_HELD=0 + fi +} +trap captain_hold_cleanup EXIT + +usage() { + awk ' + NR == 1 { next } + /^#/ { sub(/^# ?/, ""); print; next } + { exit } + ' "$0" +} + +fail() { + printf 'fm-captain-hold: %s\n' "$*" >&2 + exit 1 +} + +validate_slug() { # <label> <value> + local label=$1 value=$2 + case "$value" in + ''|*[!A-Za-z0-9._-]*) fail "$label must be a non-empty privacy-safe slug: $value" ;; + esac +} + +validate_one_line() { # <label> <value> + local label=$1 value=$2 + [ -n "$value" ] || fail "$label must not be empty" + case "$value" in + *$'\n'*|*$'\r'*) fail "$label must be one line" ;; + esac +} + +sha256_text() { # <text> + if command -v shasum >/dev/null 2>&1; then + printf '%s' "$1" | shasum -a 256 | awk '{print $1}' + elif command -v sha256sum >/dev/null 2>&1; then + printf '%s' "$1" | sha256sum | awk '{print $1}' + else + fail "shasum or sha256sum is required" + fi +} + +# The legacy derived identity older installs minted for a captain call. +# Kept only to resolve pre-collapse rows, metadata entries, and channel keys. +legacy_hold_id() { # <origin-id> <key> + printf '%s-decision-%s' "$1" "$2" +} + +# The legacy any-origin binding marker. Slug validation rejects parentheses, so +# no real origin id or task id can collide with it. +BINDING_ANY='(any)' + +DECISION_TEXT='' +DECISION_DIGEST='' + +load_decision() { # <path>; sets DECISION_TEXT and DECISION_DIGEST + local path=$1 decision + [ -n "$path" ] || fail "--decision-file is required" + [ -f "$path" ] || fail "decision file does not exist: $path" + decision=$(cat "$path") + [ -n "$decision" ] || fail "decision file must not be empty" + [ "$(printf '%s' "$decision" | LC_ALL=C wc -c | tr -d ' ')" -le 8192 ] \ + || fail "decision file exceeds 8192 bytes" + DECISION_TEXT=$decision + DECISION_DIGEST=$(sha256_text "$decision") +} + +tasks_axi() { + (cd "$FM_HOME" && tasks-axi "$@") +} + +require_tasks_axi() { + fm_tasks_axi_compatible || fail "compatible tasks-axi is required" + tasks-axi hold --help 2>&1 | grep -F -- '--kind captain' >/dev/null \ + || fail "tasks-axi does not expose the captain-hold contract" +} + +task_show() { # <id> + tasks_axi show "$1" --full 2>/dev/null +} + +show_field() { # <show-output> <field> + local output=$1 field=$2 + printf '%s\n' "$output" | sed -n "s/^ $field: //p" | head -1 +} + +decode_shown_value() { # <shown-field> + local value=$1 + case "$value" in + \"*\") + printf '%s' "$value" | perl -MJSON::PP -e ' + local $/; + my $value = decode_json(<STDIN>); + binmode STDOUT, ":raw"; + utf8::encode($value) if utf8::is_utf8($value); + print $value; + ' + ;; + *) printf '%s' "$value" ;; + esac +} + +# Decode show-encoded scalar fields and normalize the empty marker. +show_field_value() { # <show-output> <field> + local value + value=$(decode_shown_value "$(show_field "$1" "$2")") + [ "$value" != '-' ] || value='' + printf '%s' "$value" +} + +origin_exists_here() { # <origin-id> + [ -f "$STATE/$1.meta" ] && return 0 + [ -f "$DATA/$1/report.md" ] && return 0 + task_show "$1" >/dev/null 2>&1 +} + +list_has_key() { # <comma-list> <key> + case ",$1," in + *",$2,"*) return 0 ;; + *) return 1 ;; + esac +} + +sorted_key_union() { # <comma-list> <newline-or-space-separated-new-keys> + local existing=$1 new=$2 + { + printf '%s\n' "$existing" | tr ',' '\n' + printf '%s\n' "$new" | tr ' ' '\n' + } | sed '/^$/d' | LC_ALL=C sort -u | paste -sd, - +} + +meta_value() { # <meta> <key> + grep "^$2=" "$1" 2>/dev/null | tail -1 | cut -d= -f2- || true +} + +origin_open_decisions() { # <origin-id> + local origin=$1 meta="$STATE/$1.meta" status_file="$STATE/$1.status" open kind last verb + open=$(status_open_decisions "$status_file") + [ -n "$open" ] || return 0 + [ -f "$meta" ] || { printf '%s' "$open"; return 0; } + kind=$(meta_value "$meta" kind) + [ -n "$kind" ] || kind=ship + if [ "$kind" != secondmate ]; then + last=$(last_status_line "$status_file") + verb=$(status_line_verb "$last") + case "$verb" in + done|failed) return 0 ;; + esac + fi + printf '%s' "$open" +} + +# A resolution record written by this script or by the retired +# fm-decision-hold.sh. Both carry the same leader-then-captain-decision shape. +body_has_resolution_record() { # <task-body> + case "$1" in + *"Resolution recorded by fm-captain-hold."*"Captain decision:"*) return 0 ;; + *"Resolution recorded by fm-decision-hold."*"Captain decision:"*) return 0 ;; + esac + return 1 +} + +# The recorded decision digest of either record format, from the show-escaped +# body (multi-line bodies print as one quoted line with \n escapes). Records +# are prepended, so the first match is the newest record. +recorded_decision_digest() { # <task-body> + local rest=$1 + case "$rest" in + *"Decision digest: "*) rest=${rest#*"Decision digest: "} ;; + *) return 1 ;; + esac + rest=${rest%%\\n*} + rest=${rest%%$'\n'*} + printf '%s' "$rest" +} + +# The newest record's `Resolution mode:` value; empty for a record predating it. +recorded_resolution_mode() { # <task-body> + local rest=$1 + case "$rest" in + *"Resolution mode: "*) rest=${rest#*"Resolution mode: "} ;; + *) return 1 ;; + esac + rest=${rest%%\\n*} + rest=${rest%%$'\n'*} + printf '%s' "$rest" +} + +resolution_block() { # <mode> + printf 'Resolution recorded by fm-captain-hold.\nDecision digest: %s\nResolution mode: %s\n\nCaptain decision:\n%s\n' \ + "$DECISION_DIGEST" "$1" "$DECISION_TEXT" +} + +# Durable state of one captain call: an active captain hold (annotations +# surviving even when a date gate has expired) or a recorded captain answer. +verify_hold_durable() { # <task-id> + local id=$1 show state hold_kind body + show=$(task_show "$id") || fail "captain-held task $id is absent from $FM_HOME/data/backlog.md" + state=$(show_field "$show" state) + hold_kind=$(show_field_value "$show" hold_kind) + body=$(show_field "$show" body) + if body_has_resolution_record "$body"; then + return 0 + fi + if [ "$state" != "done" ] && [ "$hold_kind" = captain ]; then + return 0 + fi + fail "captain-held task $id is neither held for the captain nor closed with a recorded captain answer" +} + +# Resolve one inventory entry or channel key to the task that carries it: the +# exact task id when it exists, else the legacy derived identity. +resolve_entry() { # <origin-or-empty> <entry>; prints the resolved id or fails + local origin=$1 entry=$2 legacy + if task_show "$entry" >/dev/null 2>&1; then + printf '%s' "$entry" + return 0 + fi + if [ -n "$origin" ] && [ "$origin" != "$BINDING_ANY" ]; then + legacy=$(legacy_hold_id "$origin" "$entry") + if task_show "$legacy" >/dev/null 2>&1; then + printf '%s' "$legacy" + return 0 + fi + fail "no captain-held task $entry and no legacy identity $legacy in $FM_HOME/data/backlog.md" + fi + fail "no captain-held task $entry in $FM_HOME/data/backlog.md" +} + +command_hold() { + local id=${1:-} title='' reason='' repo='' origin='' until='' show state existing_title body='' hold_kind + [ "$#" -ge 1 ] || { usage >&2; exit 2; } + shift + while [ "$#" -gt 0 ]; do + case "$1" in + --title) shift; title=${1:-} ;; + --reason) shift; reason=${1:-} ;; + --repo) shift; repo=${1:-} ;; + --origin) shift; origin=${1:-} ;; + --until) shift; until=${1:-} ;; + *) usage >&2; exit 2 ;; + esac + shift + done + validate_slug task-id "$id" + validate_one_line reason "$reason" + case "$reason" in *'('*|*')'*) fail "reason must not contain parentheses (tasks-axi hold contract)" ;; esac + if [ -n "$origin" ]; then + validate_slug origin-id "$origin" + fi + if [ -n "$until" ]; then + case "$until" in + [0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]) : ;; + *) fail "--until must be a YYYY-MM-DD date: $until" ;; + esac + fi + require_tasks_axi + if show=$(task_show "$id"); then + state=$(show_field "$show" state) + [ "$state" != "done" ] \ + || fail "task $id is already closed; a new captain call needs its own task" + if [ -n "$title" ]; then + existing_title=$(show_field_value "$show" title) + [ "$existing_title" = "$title" ] || fail "existing task $id has a different title" + fi + else + [ -n "$title" ] || fail "--title is required to create task $id" + validate_one_line title "$title" + if [ -z "$repo" ] && [ -n "$origin" ] && [ -f "$STATE/$origin.meta" ]; then + repo=$(meta_value "$STATE/$origin.meta" project) + repo=${repo%/} + repo=${repo##*/} + fi + [ -n "$repo" ] || repo=firstmate + validate_one_line repo "$repo" + [ -z "$origin" ] || body=$(printf 'Origin: %s' "$origin") + if [ -n "$body" ]; then + tasks_axi add "$id" "$title" --repo "$repo" --body "$body" >/dev/null \ + || fail "could not create task $id" + else + tasks_axi add "$id" "$title" --repo "$repo" >/dev/null \ + || fail "could not create task $id" + fi + fi + if [ -n "$until" ]; then + tasks_axi hold "$id" --reason "$reason" --kind captain --until "$until" >/dev/null \ + || fail "could not hold task $id for the captain" + else + tasks_axi hold "$id" --reason "$reason" --kind captain >/dev/null \ + || fail "could not hold task $id for the captain" + fi + show=$(task_show "$id") || fail "task $id disappeared while holding it" + hold_kind=$(show_field_value "$show" hold_kind) + [ "$hold_kind" = captain ] || fail "task $id did not retain its captain hold" + printf '%s\n' "$id" +} + +# Record a resolution block at the top of the task body, preserving the +# previous body below it and archiving the pristine original. +write_resolution_record() { # <task-id> <mode> <shown-body> + local id=$1 mode=$2 body=$3 new_body tmp + new_body=$(resolution_block "$mode") + body=$(decode_shown_value "$body") \ + || fail "could not decode the existing body for $id" + if [ -n "$body" ]; then + new_body=$(printf '%s\n\n%s' "$new_body" "$body") + fi + tmp=$(umask 077; mktemp "${TMPDIR:-/tmp}/fm-captain-hold-body.XXXXXX") \ + || fail "cannot stage the resolution record" + if ! printf '%s\n' "$new_body" > "$tmp"; then + rm -f -- "$tmp" + fail "cannot stage the resolution record for $id" + fi + if ! tasks_axi update "$id" --body-file "$tmp" --archive-body >/dev/null; then + rm -f -- "$tmp" + fail "could not record the captain decision on $id" + fi + rm -f -- "$tmp" +} + +close_answered() { # <task-id> <release-0-or-1> + if [ "$2" = 1 ]; then + tasks_axi unhold "$1" >/dev/null || fail "could not release captain-held task $1" + else + tasks_axi "done" "$1" >/dev/null || fail "could not close answered captain-held task $1" + fi +} + +command_answer() { + local id=${1:-} decision_file='' release=0 show state hold_kind body outcome recorded_mode + [ "$#" -ge 1 ] || { usage >&2; exit 2; } + shift + while [ "$#" -gt 0 ]; do + case "$1" in + --decision-file) shift; decision_file=${1:-} ;; + --release) release=1 ;; + *) usage >&2; exit 2 ;; + esac + shift + done + validate_slug task-id "$id" + load_decision "$decision_file" + require_tasks_axi + show=$(task_show "$id") || fail "captain-held task $id is absent from $FM_HOME/data/backlog.md" + state=$(show_field "$show" state) + hold_kind=$(show_field_value "$show" hold_kind) + body=$(show_field "$show" body) + if [ "$release" = 1 ]; then outcome=released; else outcome=answered; fi + + if [ "$state" = "done" ]; then + if body_has_resolution_record "$body"; then + # An exact compatible retry is an idempotent no-op; drift is rejected. + [ "$(recorded_decision_digest "$body" || true)" = "$DECISION_DIGEST" ] \ + || fail "captain-held task $id records a different captain decision" + recorded_mode=$(recorded_resolution_mode "$body" || true) + [ "$recorded_mode" != released ] \ + || fail "task $id records this answer with mode released; a closed task cannot replay that release" + [ "$release" = 0 ] \ + || fail "task $id records this answer with mode ${recorded_mode:-unknown}; --release cannot reopen a closed task" + printf 'answered: %s\n' "$id" + return 0 + fi + [ "$release" = 0 ] || fail "task $id is already closed; --release cannot reopen it" + # Closed outside this script: record the captain's answer retroactively. + # tasks-axi keeps hold_kind through a close, so it is the surviving proof + # this really was the captain's item rather than ordinary finished work. + [ "$hold_kind" = captain ] \ + || fail "task $id was never held for the captain; nothing to record an answer on" + write_resolution_record "$id" repaired "$body" + show=$(task_show "$id") || fail "task $id disappeared while recording the answer" + [ "$(show_field "$show" state)" = "done" ] || fail "recording the answer reopened closed task $id" + body_has_resolution_record "$(show_field "$show" body)" \ + || fail "captain-held task $id did not retain its durable resolution record" + printf 'repaired: %s\n' "$id" + return 0 + fi + + if [ "$hold_kind" = captain ]; then + # Actively the captain's item (a date-expired hold keeps its annotations + # and stays answerable). A matching record means an interrupted close to + # finish; a different digest is a NEW answer on a re-held task and gets + # its own record on top. Either way the close mode is the caller's flag, + # checked against an interrupted close's recorded mode so a retry cannot + # silently flip a release into a close. + if body_has_resolution_record "$body" \ + && [ "$(recorded_decision_digest "$body" || true)" = "$DECISION_DIGEST" ]; then + recorded_mode=$(recorded_resolution_mode "$body" || true) + case "$recorded_mode" in + released) [ "$release" = 1 ] || fail "task $id records this answer as a release; retry with --release" ;; + answered) [ "$release" = 0 ] || fail "task $id records this answer as a close; retry without --release" ;; + esac + close_answered "$id" "$release" + printf '%s: %s\n' "$outcome" "$id" + return 0 + fi + write_resolution_record "$id" "$outcome" "$body" + close_answered "$id" "$release" + show=$(task_show "$id") || fail "task $id disappeared after closing" + body_has_resolution_record "$(show_field "$show" body)" \ + || fail "captain-held task $id did not retain its durable resolution record" + printf '%s: %s\n' "$outcome" "$id" + return 0 + fi + + # Not held and not closed: only an already-recorded release replays cleanly. + if body_has_resolution_record "$body"; then + recorded_mode=$(recorded_resolution_mode "$body" || true) + [ "$(recorded_decision_digest "$body" || true)" = "$DECISION_DIGEST" ] \ + || fail "task $id records a different captain decision with mode ${recorded_mode:-unknown}" + [ "$recorded_mode" = released ] && [ "$release" = 1 ] \ + || fail "task $id records this answer with mode ${recorded_mode:-unknown}; replay requires matching --release" + printf 'released: %s\n' "$id" + return 0 + fi + fail "task $id is not held for the captain; hold it first or name the right task" +} + +# --- the one keyed-answer intake, and the source bindings that feed it -------- + +BINDING_DIR="$STATE/decision-bindings" +BINDING_SCHEMA=fm-decision-binding.v1 + +validate_source_id() { # <source-id> + validate_slug source-id "$1" + [ "${#1}" -le 64 ] || fail "source-id must be at most 64 characters: $1" +} + +binding_path() { printf '%s/%s.origin\n' "$BINDING_DIR" "$1"; } + +# The stored binding value, or empty when the source is unbound. An unreadable +# or wrong-schema record is a hard error rather than a silent "unbound": +# feeding nothing is the safe direction only when it is a deliberate choice, +# never when it is a corrupted record. +read_binding() { # <source-id> + local path origin schema + path=$(binding_path "$1") + [ -e "$path" ] || return 0 + [ -f "$path" ] && [ ! -L "$path" ] || fail "decision binding is unsafe: $path" + schema=$(sed -n 's/^schema=//p' "$path" | head -1) + [ "$schema" = "$BINDING_SCHEMA" ] || fail "decision binding has an incompatible schema: $path" + origin=$(sed -n 's/^origin=//p' "$path" | head -1) + if [ "$origin" != "$BINDING_ANY" ]; then + case "$origin" in + ''|*[!A-Za-z0-9._-]*) fail "decision binding has an invalid origin id: $path" ;; + esac + fi + printf '%s\n' "$origin" +} + +command_bind() { + local source=${1:-} origin=${2:-} dest tmp + [ "$#" -ge 1 ] && [ "$#" -le 2 ] || { usage >&2; exit 2; } + validate_source_id "$source" + if [ -z "$origin" ] || [ "$origin" = --any-origin ]; then + origin=$BINDING_ANY + else + validate_slug legacy-origin "$origin" + fi + (umask 077; mkdir -p "$BINDING_DIR") || fail "cannot create $BINDING_DIR" + [ -d "$BINDING_DIR" ] && [ ! -L "$BINDING_DIR" ] || fail "decision binding dir is unsafe: $BINDING_DIR" + dest=$(binding_path "$source") + tmp=$(umask 077; mktemp "$BINDING_DIR/.origin.XXXXXX") || fail "cannot stage the decision binding" + if ! { printf 'schema=%s\norigin=%s\n' "$BINDING_SCHEMA" "$origin" > "$tmp" \ + && chmod 0600 "$tmp" && mv -f -- "$tmp" "$dest"; }; then + rm -f -- "$tmp" + fail "cannot record the decision binding for $source" + fi + printf 'bound: %s -> %s\n' "$source" "$origin" +} + +command_unbind() { + local source=${1:-} + [ "$#" -eq 1 ] || { usage >&2; exit 2; } + validate_source_id "$source" + rm -f -- "$(binding_path "$source")" + printf 'unbound: %s\n' "$source" +} + +command_binding() { + local source=${1:-} origin + [ "$#" -eq 1 ] || { usage >&2; exit 2; } + validate_source_id "$source" + origin=$(read_binding "$source") || exit 1 + [ -n "$origin" ] || return 1 + printf '%s\n' "$origin" +} + +# The durable captain decision one keyed answer records. Pure function of its +# inputs, so the same answer delivered twice is idempotent rather than a +# conflicting decision. +keyed_decision_text() { # <source> <task-id> <answer> <label> + printf 'Captain answered this call through %s.\n' "$1" + printf 'Task: %s\n' "$2" + printf 'Answer: %s\n' "$3" + [ -z "$4" ] || printf 'Answer as shown to the captain: %s\n' "$4" +} + +legacy_keyed_decision_text() { # <source> <key> <answer> <label> + printf 'Captain answered this decision through %s.\n' "$1" + printf 'Decision key: %s\n' "$2" + printf 'Answer: %s\n' "$3" + [ -z "$4" ] || printf 'Answer as shown to the captain: %s\n' "$4" +} + +sanitize_field() { # <text> + printf '%s' "$1" | tr '\n\r\t' ' ' | LC_ALL=C tr -d '\000-\037\177' | cut -c1-512 +} + +command_answers() { + local origin='' source='' row rest key answer label mode id show state hold_kind body digest legacy_digest legacy_key + local recorded_digest recorded_mode tmp err closed=0 skipped=0 reason release_flag tab=$'\t' + while [ "$#" -gt 0 ]; do + case "$1" in + --source) shift; source=${1:-} ;; + --any-origin) origin=$BINDING_ANY ;; + --*) usage >&2; exit 2 ;; + *) + [ -z "$origin" ] || { usage >&2; exit 2; } + origin=$1 + ;; + esac + shift + done + if [ -n "$origin" ] && [ "$origin" != "$BINDING_ANY" ]; then + validate_slug legacy-origin "$origin" + fi + [ -n "$source" ] || fail "--source provenance is required so the durable decision records where the answer came from" + source=$(sanitize_field "$source") + require_tasks_axi + tmp=$(umask 077; mktemp "${TMPDIR:-/tmp}/fm-keyed-decision.XXXXXX") || fail "cannot stage the captain decision" + err=$(umask 077; mktemp "${TMPDIR:-/tmp}/fm-keyed-decision-err.XXXXXX") \ + || { rm -f -- "$tmp"; fail "cannot stage the captain decision diagnostics"; } + while IFS= read -r row; do + key=${row%%"$tab"*} + rest='' + case "$row" in *"$tab"*) rest=${row#*"$tab"} ;; esac + answer=${rest%%"$tab"*} + case "$rest" in *"$tab"*) rest=${rest#*"$tab"} ;; *) rest='' ;; esac + label=${rest%%"$tab"*} + case "$rest" in *"$tab"*) mode=${rest#*"$tab"} ;; *) mode='' ;; esac + [ -n "${key:-}" ] || continue + case "$key" in *[!A-Za-z0-9._-]*) continue ;; esac + [ "${#key}" -le 128 ] || continue + answer=$(sanitize_field "${answer:-}") + [ -n "$answer" ] || continue + label=$(sanitize_field "${label:-}") + release_flag='' + case "${mode:-}" in + ''|done) : ;; + release) release_flag=--release ;; + *) + printf 'skipped: %s (unknown close mode %s)\n' "$key" "$(sanitize_field "$mode")" + skipped=$((skipped + 1)) + continue + ;; + esac + if ! id=$(resolve_entry "$origin" "$key" 2>/dev/null); then + printf 'skipped: %s (no captain-held task with that id)\n' "$key" + skipped=$((skipped + 1)) + continue + fi + keyed_decision_text "$source" "$id" "$answer" "$label" > "$tmp" \ + || fail "cannot stage the captain decision for $id" + digest=$(sha256_text "$(cat "$tmp")") + legacy_digest='' + if [ "$id" != "$key" ]; then + legacy_key=$key + elif { [ -z "$origin" ] || [ "$origin" = "$BINDING_ANY" ]; } \ + && [ "${id#*-decision-}" != "$id" ]; then + legacy_key=${id#*-decision-} + else + legacy_key='' + fi + if [ -n "$legacy_key" ]; then + legacy_digest=$(sha256_text "$(legacy_keyed_decision_text "$source" "$legacy_key" "$answer" "$label")") + fi + show=$(task_show "$id") || { printf 'skipped: %s (absent)\n' "$id"; skipped=$((skipped + 1)); continue; } + state=$(show_field "$show" state) + hold_kind=$(show_field_value "$show" hold_kind) + body=$(show_field "$show" body) + recorded_digest=$(recorded_decision_digest "$body" || true) + recorded_mode=$(recorded_resolution_mode "$body" || true) + if body_has_resolution_record "$body" \ + && { [ "$recorded_digest" = "$digest" ] \ + || { case "$body" in *"Resolution recorded by fm-decision-hold."*) true ;; *) false ;; esac \ + && [ -n "$legacy_digest" ] && [ "$recorded_digest" = "$legacy_digest" ]; }; }; then + if { [ -z "$release_flag" ] && [ "$state" = "done" ] && [ "$recorded_mode" != released ]; } \ + || { [ "$release_flag" = --release ] && [ "$state" != "done" ] \ + && [ "$hold_kind" != captain ] && [ "$recorded_mode" = released ]; }; then + printf 'closed: %s\n' "$id" + closed=$((closed + 1)) + continue + fi + fi + if [ "$state" = "done" ]; then + printf 'skipped: %s (already closed)\n' "$id" + skipped=$((skipped + 1)) + continue + fi + if [ "$hold_kind" != captain ]; then + printf 'skipped: %s (not held for the captain)\n' "$id" + skipped=$((skipped + 1)) + continue + fi + # shellcheck disable=SC2086 # release_flag is empty or a single literal flag. + if "$0" answer "$id" --decision-file "$tmp" $release_flag </dev/null >/dev/null 2>"$err"; then + printf 'closed: %s\n' "$id" + closed=$((closed + 1)) + else + reason=$(tr -d '\n' < "$err" | sed 's/^fm-captain-hold: //') + printf 'skipped: %s (%s)\n' "$id" "$reason" + skipped=$((skipped + 1)) + fi + done + rm -f -- "$tmp" "$err" + printf 'answers: closed=%s skipped=%s\n' "$closed" "$skipped" + [ "$skipped" -eq 0 ] +} + +command_complete() { + local origin=${1:-} meta previous='' supplied='' keys='' entry key status_file open raw_open has_meta=0 transfer_rc + [ "$#" -ge 2 ] || { usage >&2; exit 2; } + validate_slug origin-id "$origin" + shift + meta="$STATE/$origin.meta" + [ -f "$meta" ] && has_meta=1 + if [ "$has_meta" = 1 ]; then + CAPTAIN_META_LOCK=$(fm_meta_lock_path "$meta") || fail "could not resolve task metadata lock" + fm_lock_acquire_wait "$CAPTAIN_META_LOCK" + CAPTAIN_META_LOCK_HELD=1 + [ -f "$meta" ] || fail "task metadata disappeared while recording completion" + fi + require_tasks_axi + origin_exists_here "$origin" || fail "origin $origin is not owned by the active home $FM_HOME" + if [ "$#" -eq 1 ] && [ "$1" = --none ]; then + supplied='' + else + while [ "$#" -gt 0 ]; do + [ "$1" != --none ] || fail "--none cannot be combined with task ids" + validate_slug task-id "$1" + supplied="${supplied}${supplied:+ }$1" + shift + done + fi + if [ "$has_meta" = 1 ]; then + previous=$(meta_value "$meta" decision_keys) + fi + keys=$(sorted_key_union "$previous" "$supplied") + if [ -n "$keys" ]; then + while IFS= read -r entry; do + [ -n "$entry" ] || continue + verify_hold_durable "$(resolve_entry "$origin" "$entry")" + done <<EOF +$(printf '%s\n' "$keys" | tr ',' '\n') +EOF + fi + + status_file="$STATE/$origin.status" + raw_open=$(status_open_decisions "$status_file") + open=$(origin_open_decisions "$origin") + if [ -n "$open" ] && [ -z "$keys" ]; then + fail "origin $origin still has open captain decisions in its status stream; hold a captain task for what remains, or answer them, before attesting --none" + fi + + if [ "$has_meta" = 1 ]; then + if [ "$(meta_value "$meta" decisions_reviewed)" != 1 ] || [ "$previous" != "$keys" ]; then + printf 'decisions_reviewed=1\ndecision_keys=%s\n' "$keys" >> "$meta" + fi + fm_lock_release "$CAPTAIN_META_LOCK" + CAPTAIN_META_LOCK_HELD=0 + + # Transfer every still-open status decision to the durable captain-held + # inventory so the live status fold does not duplicate the same Captain's + # Call item. The transfer line is this home's own bookkeeping close, + # written by the turn that just reviewed the inventory, so it uses the + # guarded self-announced append (bin/fm-wake-lib.sh) and does not wake this + # same session; an append failure still fails this command loudly. + if [ -n "$keys" ]; then + while IFS=$'\t' read -r key _verb _summary; do + [ -n "$key" ] || continue + transfer_rc=0 + fm_wake_status_append_self_announced "$STATE" "$status_file" \ + "captain-held [key=$key]: tracked by $keys" || transfer_rc=$? + [ "$transfer_rc" -ne 2 ] || fail "cannot append the captain-held transfer for $origin/$key" + done <<EOF +$raw_open +EOF + fi + fi + printf 'complete: %s captain-call inventory reviewed%s\n' "$origin" "${keys:+ ($keys)}" +} + +command_verify() { + local origin=${1:-} meta reviewed keys entry key open + [ "$#" -eq 1 ] || { usage >&2; exit 2; } + validate_slug origin-id "$origin" + meta="$STATE/$origin.meta" + [ -f "$meta" ] || fail "origin metadata is absent: $meta" + require_tasks_axi + reviewed=$(meta_value "$meta" decisions_reviewed) + [ "$reviewed" = 1 ] || fail "origin $origin has no completed captain-call inventory" + keys=$(meta_value "$meta" decision_keys) + if [ -n "$keys" ]; then + while IFS= read -r entry; do + [ -n "$entry" ] || continue + verify_hold_durable "$(resolve_entry "$origin" "$entry")" + done <<EOF +$(printf '%s\n' "$keys" | tr ',' '\n') +EOF + fi + open=$(origin_open_decisions "$origin") + while IFS=$'\t' read -r key _verb _summary; do + [ -n "$key" ] || continue + fail "open captain decision $origin/$key is not transferred to the captain-held inventory; re-run complete" + done <<EOF +$open +EOF + printf 'verified: %s captain-call inventory\n' "$origin" +} + +# --- record divergence ------------------------------------------------------ +# +# A captain call can be written down twice, and until now nothing said when +# those two records disagreed. A `resolved [key=...]` line closes the status-log +# fold outright; the structured captain-held task is closed by a SEPARATE act +# (`answer` above). Closing only on the status side therefore looks complete +# there while the durable record still says the captain owes an answer and +# keeps resurfacing it. The defect was never the separation; it was the silence. +# +# `diverged` is a read-only report of that contradiction and nothing else. It +# closes NOTHING. A captain call closed wrongly disappears without review, which +# is strictly worse than the noise this prints, so reconciling a divergence stays +# a human-owned act - and it runs in either direction: record what the captain +# actually said with `answer`, or re-open the status decision when that +# resolution was not the captain's word. +# +# What it flags, and only this: a task that is still open and still carries the +# captain-hold annotations, whose key was closed on the status side by the +# RESOLVE verb. The other closing verb is not a divergence: a `captain-held` +# close is the VERIFIED transfer to that very task, written by command_complete +# only after verifying it, so the structured row staying open behind it is the +# correct state. Neither is a still-open status decision - the OPEN DECISIONS +# fold already owns that one. +# +# Routed work is deliberately irrelevant. When the decision IS the deliverable +# there is nothing to route, so the test is only whether the status side already +# declared this task's key resolved. +# Nor does the report interpret why that resolution exists. A call can turn out +# not to be a captain arbitration at all - a premise can dissolve, or a question +# of fact can prove its first reading wrong - so the report says only that the +# two records disagree and names both reconciliation directions above. +# +# Cost stays flat on a healthy home: one `tasks-axi list`, one key scan per +# status log, and the precise per-key fold only for a key that already names a +# still-open task. If tasks-axi is unavailable or its listing cannot be parsed, +# the guard cannot read the structured record and prints nothing. +# +# Output: one `<task-id>\t<origin>\t<key>\t<title>` line per divergence, in +# status-log then key order; nothing when the two records agree. + +# Every still-open task id in this home's backlog, one per line. Only the first +# two comma-separated listing fields are read - both are slugs that precede any +# quoted title - so a title containing commas or quotes cannot shift them. +open_task_ids() { + tasks_axi list 2>/dev/null | awk -F, ' + /^ [A-Za-z0-9._-]+,/ { + id = $1 + sub(/^ +/, "", id) + if ($2 != "done") print id + } + ' +} + +# Every key token stated anywhere in a status log. A cheap candidate scan: it +# over-includes tokens that are only prose, and status_key_closing_verb below is +# what actually decides what the stream says about a key. +status_log_key_tokens() { # <status-file> + grep -o '\[key=[A-Za-z0-9._-]*\]' "$1" 2>/dev/null | + sed 's/^\[key=//; s/\]$//' | LC_ALL=C sort -u +} + +list_has_line() { # <newline-separated-list> <value> + case $'\n'"$1"$'\n' in + *$'\n'"$2"$'\n'*) return 0 ;; + *) return 1 ;; + esac +} + +command_diverged() { + local ids resolve f origin tokens id keys key show title + [ "$#" -eq 0 ] || { usage >&2; exit 2; } + # Both records must belong to the SAME home or the comparison is meaningless: + # tasks-axi reads $FM_HOME's backlog, so a state dir pointed somewhere else + # would report one home's status logs against another home's tasks. Every + # production caller pairs the two; a mismatch stays silent rather than + # inventing a cross-home divergence. + [ "$STATE" = "$FM_HOME/state" ] || return 0 + # A read-only listing on a per-wake path, so it skips the mutation-oriented + # compatibility floor and its extra probes: a listing this parser cannot read + # simply yields no candidates and the report stays silent. + command -v tasks-axi >/dev/null 2>&1 || return 0 + ids=$(open_task_ids) || return 0 + [ -n "$ids" ] || return 0 + resolve=${FM_CLASSIFY_RESOLVE_VERB:-$FM_CLASSIFY_RESOLVE_VERB_DEFAULT} + for f in "$STATE"/*.status; do + [ -f "$f" ] && [ -r "$f" ] && [ ! -L "$f" ] || continue + origin=$(basename "$f"); origin=${origin%.status} + tokens=$(status_log_key_tokens "$f") + [ -n "$tokens" ] || continue + while IFS= read -r id; do + [ -n "$id" ] || continue + # The keys that could name this task in THIS log: the collapsed identity + # (the key IS the task id) and, for a pre-collapse row, the legacy derived + # one this origin would have minted. + keys=$id + case "$id" in + "$origin-decision-"?*) keys="$keys"$'\n'"${id#"$origin-decision-"}" ;; + esac + while IFS= read -r key; do + list_has_line "$tokens" "$key" || continue + [ "$(status_key_closing_verb "$f" "$key")" = "$resolve" ] || continue + show=$(task_show "$id") || continue + [ "$(show_field "$show" state)" != "done" ] || continue + [ "$(show_field_value "$show" hold_kind)" = captain ] || continue + # The title is the only free-text field here, and the report is + # TAB-separated, so it goes through the same sanitizer every other + # emitted field uses rather than being trusted to stay one clean line. + title=$(sanitize_field "$(show_field_value "$show" title)") + printf '%s\t%s\t%s\t%s\n' "$id" "$origin" "$key" "$title" + break + done <<INNER +$keys +INNER + done <<EOF +$ids +EOF + done +} + +case "${1:-}" in + hold) shift; command_hold "$@" ;; + answer) shift; command_answer "$@" ;; + answers) shift; command_answers "$@" ;; + bind) shift; command_bind "$@" ;; + unbind) shift; command_unbind "$@" ;; + binding) shift; command_binding "$@" ;; + complete) shift; command_complete "$@" ;; + verify) shift; command_verify "$@" ;; + diverged) shift; command_diverged "$@" ;; + -h|--help) usage ;; + *) usage >&2; exit 2 ;; +esac diff --git a/bin/fm-classify-lib.sh b/bin/fm-classify-lib.sh index 30f0fd027c3..9915ece7d29 100755 --- a/bin/fm-classify-lib.sh +++ b/bin/fm-classify-lib.sh @@ -13,7 +13,7 @@ # daemon keeps its escalation-digest seen-markers; the watcher keeps its .seen-* # signatures). # -# There are two documented exceptions. The absorb classification +# There are three documented exceptions. The absorb classification # (crew_absorb_class and its working/paused wrappers) is NOT a pure status-file # read: it reuses bin/fm-crew-state.sh, which may make a bounded no-mistakes call, # to decide whether a crew that just stopped its turn or went stale is working, @@ -23,7 +23,9 @@ # open-decisions fold" below) also writes: it persists a per-status-file byte # cursor and folded open-set as a side effect, so a per-drain fleet-wide scan # stays bounded by new appends instead of re-reading each task's whole lifetime -# log every time. +# log every time. crew_worktree_written_since reads the task's meta file and walks +# a bounded slice of its worktree instead of a status file, so callers run it only +# at the moment they would otherwise escalate. # Directory of this library, used to locate the sibling fm-crew-state.sh reader. # Resolved at source time from BASH_SOURCE so it works whether sourced by a @@ -35,6 +37,19 @@ _FM_CLASSIFY_LIB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd 2>/dev/null)" # or no-mistakes install; absent, it points at the real sibling script. FM_CREW_STATE_BIN="${FM_CREW_STATE_BIN:-$_FM_CLASSIFY_LIB_DIR/fm-crew-state.sh}" +# fm_run_timed, the shared hard bound the worktree write probe below puts around +# its one filesystem walk. bin/fm-timeout-lib.sh owns bounded execution for this +# repo, so nothing here re-derives the coreutils/BSD/perl selection. That library +# declares `set -u` for its own hygiene, which a sourced sibling must not impose on +# THIS library's consumers - several of them deliberately run without it - so the +# caller's setting is restored around the source. +case $- in *u*) _fm_classify_nounset=on ;; *) _fm_classify_nounset=off ;; esac +# shellcheck source=bin/fm-timeout-lib.sh +# shellcheck disable=SC1091 +. "$_FM_CLASSIFY_LIB_DIR/fm-timeout-lib.sh" +[ "$_fm_classify_nounset" = on ] || set +u +unset _fm_classify_nounset + # Captain-relevant status verbs. A status line carrying any of these is work # firstmate must see. Lines without these verbs are no-verb signals: the watcher # absorbs them only with positive provably-working evidence, while the daemon uses @@ -61,7 +76,7 @@ FM_CLASSIFY_CAPTAIN_RE_DEFAULT='done:|needs-decision:|blocked:|failed:|PR ready| # drift between the two consumers. FM_CLASSIFY_PAUSED_VERB overrides it. FM_CLASSIFY_PAUSED_VERB_DEFAULT='paused' -# Bounded re-surface cadence for a declared pause or a dead-agent captain hold. +# Bounded re-surface cadence for a declared pause or a verified captain hold. # Far longer than the wedge threshold (FM_STALE_ESCALATE_SECS, default 240s), it # avoids nagging a deliberate wait while ensuring a forgotten hold cannot rot # invisibly - it re-surfaces once for a recheck every window. One hour by default; @@ -73,7 +88,7 @@ FM_PAUSE_RESURFACE_SECS_DEFAULT=3600 # The resolution verb and durable-backlog-transfer verb that CLOSE a keyed # status decision opened by needs-decision or blocked. See status_open_decisions # below for the status-fold contract. The transfer verb is written only after -# fm-decision-hold.sh has verified the corresponding captain-held backlog item. +# fm-captain-hold.sh has verified the corresponding captain-held backlog item. FM_CLASSIFY_RESOLVE_VERB_DEFAULT='resolved' FM_CLASSIFY_CAPTAIN_HELD_VERB_DEFAULT='captain-held' @@ -131,19 +146,31 @@ status_is_paused() { # <status-line> [ "$verb" = "${FM_CLASSIFY_PAUSED_VERB:-$FM_CLASSIFY_PAUSED_VERB_DEFAULT}" ] } -# 0 if a status line declares either an external-wait pause or a verified -# captain-held transfer. -# Both declarations can intentionally leave an exited crew's endpoint idle, so -# the watcher applies its bounded pause cadence when agent death confirms that -# no live decision gate is being silenced. -status_is_paused_or_captain_held() { # <status-line> +# 0 if a status line's leading verb is the verified captain-held transfer verb. +# The same pure verb read as status_is_paused, and the discriminator a supervisor +# needs once a declared wait has already been recognized: the two declarations get +# the same bounded cadence, but they block on DIFFERENT humans, so a recheck that +# names an external dependency for a hold points the captain away from the fact +# that they are the one who can clear it. +status_is_captain_held() { # <status-line> local line=$1 verb - status_is_paused "$line" && return 0 [ -n "$line" ] || return 1 verb=$(status_line_verb "$line") [ "$verb" = "${FM_CLASSIFY_CAPTAIN_HELD_VERB:-$FM_CLASSIFY_CAPTAIN_HELD_VERB_DEFAULT}" ] } +# 0 if a status line declares either an external-wait pause or a verified +# captain-held transfer. +# Both declarations can intentionally leave a crew's endpoint idle, so both +# supervisors give them one cadence: the away-mode daemon defers the wedge and +# ages a pause marker instead, and the watcher applies its bounded pause cadence +# once pause_state_class has admitted the wait (fm-watch.sh owns which liveness +# evidence each kind of crew must supply for that). +status_is_paused_or_captain_held() { # <status-line> + local line=$1 + status_is_paused "$line" || status_is_captain_held "$line" +} + # --- durable keyed decisions ------------------------------------------------ # # The status stream is an append-only EVENT log. Reading it last-event-wins @@ -351,6 +378,75 @@ status_open_decisions() { # <status-file> printf '%s' "$open" } +# 0 when <key> has a record in a folded "<key>\t<verb>\t<note>" open set. +_fm_open_set_has() { # <open-set> <key> + case "$1" in + "$2"$'\t'*|*$'\n'"$2"$'\t'*) return 0 ;; + *) return 1 ;; + esac +} + +# The verb stored for <key> in a folded open set (empty when it has no record). +_fm_open_set_verb() { # <open-set> <key> + local line + while IFS= read -r line; do + case "$line" in + "$2"$'\t'*) line=${line#*$'\t'}; printf '%s' "${line%%$'\t'*}"; return 0 ;; + esac + done <<EOF +$1 +EOF + return 0 +} + +# The verb that last moved <key> in a status stream, which is what tells a +# consumer HOW the status side currently reads that key. Prints the opening verb +# (needs-decision or blocked) while the key is still open, the closing verb +# (resolved, or the captain-held durable-transfer verb) once it is closed, and +# nothing at all when no line in the stream ever stated a transition for it. +# +# The distinction between the two closing verbs is the whole point: a +# `captain-held` close is the VERIFIED handoff to a durable captain-held task +# (fm-captain-hold.sh complete writes it only after verifying that task), so the +# structured row staying open afterwards is correct. A `resolved` close claims +# the question is settled outright, so a structured row still open behind it is a +# contradiction between the two records - see fm-captain-hold.sh's `diverged`. +# +# Semantics are not re-derived here: every line goes through the same +# _fm_decision_fold_line rule the two folds use, and the reported verb is read +# off the transitions that rule produces. Only lines whose parsed key equals the +# requested one can move that key, so a caller-supplied key other than "default" +# lets the scan pre-filter the stream to lines carrying its token and stay cheap +# on a long log. +status_key_closing_verb() { # <status-file> <key> + local f=$1 want=$2 line resolve held open='' was verb='' stream + [ -f "$f" ] && [ -r "$f" ] && [ ! -L "$f" ] || return 0 + [ -n "$want" ] || return 0 + resolve=${FM_CLASSIFY_RESOLVE_VERB:-$FM_CLASSIFY_RESOLVE_VERB_DEFAULT} + held=${FM_CLASSIFY_CAPTAIN_HELD_VERB:-$FM_CLASSIFY_CAPTAIN_HELD_VERB_DEFAULT} + if [ "$want" = default ]; then + stream=$(cat "$f") || return 0 + else + stream=$(grep -F "[key=$want]" "$f") || stream='' + fi + [ -n "$stream" ] || return 0 + while IFS= read -r line || [ -n "$line" ]; do + was=0 + _fm_open_set_has "$open" "$want" && was=1 + open=$(_fm_decision_fold_line "$open" "$line" "$resolve" "$held") + if [ "$was" = 1 ] && ! _fm_open_set_has "$open" "$want"; then + verb=$(status_line_verb "$line") + fi + done <<EOF +$stream +EOF + if _fm_open_set_has "$open" "$want"; then + _fm_open_set_verb "$open" "$want" + return 0 + fi + printf '%s' "$verb" +} + # Fleet-wide wrapper around status_open_decisions: scans every task's status # log under <state> and prefixes each still-open decision with its owning task # id, so a per-wake or per-session surface can print the consolidated open set @@ -1133,6 +1229,95 @@ crew_is_paused() { # <id> [ "$(crew_absorb_class "$1")" = paused ] } +# Directories excluded from the worktree write probe below, and the depth it walks. +# The excluded set is everything a supervisor read or a package manager can write +# without the crew doing any work - .git first, so firstmate's own read-only git +# commands against the worktree can never make the probe self-fulfilling - plus the +# large generated trees that would make the walk expensive. Both are overridable so +# a home with an unusual layout can widen or narrow the probe. The list is a skip +# list, so clearing it skips nothing and widens the walk to the whole depth-bounded +# tree; it never disables the probe, which would quietly cost the wedge detector a +# liveness input on a home that meant to widen it. Defaulted with the plain form so +# an explicitly empty value stays empty: clearing the knob in the environment is the +# documented way to ask for that wider walk, and treating empty as unset would hand +# the default skip list back to exactly the home that asked for more coverage. +FM_WORKTREE_WRITE_PRUNE=${FM_WORKTREE_WRITE_PRUNE-'.git node_modules .venv venv __pycache__ .mypy_cache .pytest_cache .ruff_cache .tox target dist build .next .cache vendor'} +FM_WORKTREE_WRITE_MAXDEPTH=${FM_WORKTREE_WRITE_MAXDEPTH:-6} + +# Wall-clock seconds the probe's single walk may take. The walk runs synchronously +# inside the caller's poll loop at the exact moment an escalation would otherwise +# fire, and -xdev keeps it out of a nested mount but cannot help when the worktree +# root ITSELF sits on a hung network or container mount; unbounded, such a walk +# would wedge the very supervisor that exists to notice a wedge, stalling its +# heartbeat instead of escalating. Hitting the bound is a negative outcome like +# every other: it reads as no evidence, so the caller's escalation schedule is +# untouched and a stall that writes nothing still escalates on the existing +# schedule. A value that is not a positive integer is not a bound at all (`timeout +# 0` and the perl fallback's `alarm 0` both disable the deadline), so the default +# applies instead; the check lives at the point of use so an in-process override +# gets it too. +FM_WORKTREE_WRITE_TIMEOUT=${FM_WORKTREE_WRITE_TIMEOUT:-10} + +# 0 when some regular file under <id>'s recorded worktree is newer than +# <anchor-file>: positive evidence the crew is still producing work even though its +# rendered pane has gone quiet. This is the third liveness input the wedge detector +# has, after pane quietness and the run step, and it exists because neither of +# those can see a crew that is writing source, then tests, then documentation +# behind a static pane - the 2026-08-14 case of eight consecutive possible-wedge +# escalations against a crew that was demonstrably working the whole time. +# +# 1 for every other outcome, including an id with no recorded worktree, a worktree +# that is gone, a missing anchor, and a walk that fails or finds nothing. Absence of +# evidence therefore always leaves the caller's existing escalation schedule +# untouched, so a crew that writes nothing still escalates exactly as before. +# +# A kind=secondmate task records a provisioned firstmate home, not a code tree, and +# such a home runs its OWN supervision inside it: its state/ directory churns a +# watcher beacon, pane hashes, and heartbeats whether or not the mate is producing +# anything, so a walk there would report liveness for a mate that has done nothing. +# Those homes are excluded outright rather than by pruning "state", which would also +# hide a legitimate source directory of that name in an ordinary worktree. The +# exclusion is a negative outcome like any other, so an unproductive mate keeps +# escalating on the caller's unchanged schedule. +# +# The anchor is the caller's own idle-window timer file, whose mtime already marks +# when the quiet window opened, so `-newer` needs no clock arithmetic, no temp +# file, and no portable mtime-setting. Not a pure status-file read (see the header): +# one pruned, depth-bounded, wall-clock-bounded walk per call, which callers must +# reach only when they are otherwise about to escalate, never on every poll. A walk +# that outlives FM_WORKTREE_WRITE_TIMEOUT is killed and reported as no evidence, so +# a hung mount costs the escalation nothing but the bound. -xdev holds that walk to the +# worktree's own filesystem rather than descending into a nested network or container +# mount, so a write that lands only under such a mount is one more negative outcome. +crew_worktree_written_since() { # <id> <state> <anchor-file> + local id=$1 state=$2 anchor=$3 wt kind name hit bound + local -a names=() prune=() + [ -n "$id" ] || return 1 + [ -f "$anchor" ] || return 1 + wt=$(grep '^worktree=' "$state/$id.meta" 2>/dev/null | tail -1 | cut -d= -f2- || true) + [ -n "$wt" ] && [ -d "$wt" ] || return 1 + kind=$(grep '^kind=' "$state/$id.meta" 2>/dev/null | tail -1 | cut -d= -f2- || true) + [ "$kind" != secondmate ] || return 1 + if [ -e "$wt/.fm-secondmate-home" ] || [ -L "$wt/.fm-secondmate-home" ]; then + return 1 + fi + read -r -a names <<< "$FM_WORKTREE_WRITE_PRUNE" + for name in ${names[@]+"${names[@]}"}; do + [ "${#prune[@]}" -eq 0 ] || prune+=( -o ) + prune+=( -name "$name" ) + done + bound=$FM_WORKTREE_WRITE_TIMEOUT + case "$bound" in ''|*[!0-9]*|0) bound=10 ;; esac + if [ "${#prune[@]}" -gt 0 ]; then + hit=$(fm_run_timed "$bound" find "$wt" -xdev -maxdepth "$FM_WORKTREE_WRITE_MAXDEPTH" \ + \( "${prune[@]}" \) -prune -o -type f -newer "$anchor" -print -quit 2>/dev/null || true) + else + hit=$(fm_run_timed "$bound" find "$wt" -xdev -maxdepth "$FM_WORKTREE_WRITE_MAXDEPTH" \ + -type f -newer "$anchor" -print -quit 2>/dev/null || true) + fi + [ -n "$hit" ] +} + # 0 (benign/absorb) if EVERY task referenced by a no-verb "signal:" wake is provably # working; 1 (actionable/surface) if any is not, or no task can be resolved. Pass the # same space-separated file list as signal_reason_is_actionable. Files are mapped to diff --git a/bin/fm-claude-stop-autoarm.sh b/bin/fm-claude-stop-autoarm.sh index 806be1bfab8..89ce011f6bb 100755 --- a/bin/fm-claude-stop-autoarm.sh +++ b/bin/fm-claude-stop-autoarm.sh @@ -23,7 +23,10 @@ # - Single-flight: Claude does not dedupe async hooks, so a home-scoped owner # lock (state/.claude-autoarm.lock) admits exactly one owner; every other # concurrent firing exits 0 without translating, which keeps one event -# epoch on exactly one recovery turn. +# epoch on exactly one recovery turn. A lock left behind by a claim whose +# ledger outcome is already terminal, or whose recorded pid-identity no +# longer matches its live pid, is reclaimed once rather than deferred to +# forever (fm_autoarm_claim_abandoned in bin/fm-wake-lib.sh). # - Foreground arm: the owner runs bin/fm-watch-arm.sh in the FOREGROUND of # this hook-owned process tree (never shell &); Claude owns the process # group, so its timeout/session teardown kills arm and watcher together. @@ -135,7 +138,24 @@ fi # Claude runs one background process per firing with no dedupe. Exactly one # owner foregrounds the arm and translates its close; every other firing exits # 0 so one watcher cycle maps to at most one exit-2 rewake. -fm_lock_try_acquire "$OWNER_LOCK" || exit 0 +# +# A claim whose own ledger entry or recorded pid-identity proves its supervision +# decision already finished is abandoned, not in flight: deferring to it forever +# is what leaves a home unsupervised with no watcher and no lock +# (fm_autoarm_claim_abandoned in bin/fm-wake-lib.sh owns that proof and its +# race-free reclaim). Reclaim it once and retry; anything still genuinely +# deciding keeps the lock and this firing stays inert. +if ! fm_lock_try_acquire "$OWNER_LOCK"; then + fm_autoarm_release_abandoned "$STATE" || exit 0 + fm_lock_try_acquire "$OWNER_LOCK" || exit 0 +fi +# Record WHO this claim is before publishing the role both Stop participants read +# as ownership. A bare pid the operating system later hands to an unrelated live +# process is exactly what makes a killed claim look in flight forever, in the two +# shapes the ledger cannot settle: an entry still reading arming, and no entry at +# all. Best effort; a home whose identity cannot be recorded keeps the ledger-only +# boundary rather than losing its claim. +fm_autoarm_claim_record_identity "$STATE" || true if ! fm_lock_set_role "$OWNER_LOCK" autoarm; then fm_lock_release "$OWNER_LOCK" exit 0 diff --git a/bin/fm-composer-lib.sh b/bin/fm-composer-lib.sh index cb03c21d4ce..07b3b02fffb 100644 --- a/bin/fm-composer-lib.sh +++ b/bin/fm-composer-lib.sh @@ -63,7 +63,7 @@ # mode/model footer line. # separated - pi: content rows between two solid horizontal `─` rules, no # glyph and no side border. Provable only with a live agent -# identity reporting an idle/done/blocked pi (herdr `agent +# identity reporting an idle/done pi (herdr `agent # get`; the tmux foreground-process probe), because a blank # region between two transcript rules is otherwise exactly the # strict rule's unidentifiable blank row. @@ -1379,7 +1379,11 @@ _fm_composer_classify_bare_pi_overlap() { # <screen> <styled> <has-identity> <i # rule, now fleet-wide). A missing identity capability keeps the shape # unknown; an unfetched identity on an identity-capable backend asks the # adapter to probe (lazily) and re-call. Proven input remains pending for every -# live pi state, while only an idle/done/blocked pi proves an empty composer. +# live pi state, while only an idle/done pi proves an empty composer. A blocked +# pi is parked on an interactive prompt waiting for a human keystroke: its menu +# is drawn above the separator pair, so the composer region looks free while the +# keys would answer the prompt instead of composing (issue #2797). Structure +# cannot disprove that, so a blocked pi defers rather than claiming empty. _fm_composer_pi_verdict() { # <screen> <styled> <has_identity> <identity> local screen=$1 styled=$2 has_identity=$3 identity=$4 agent agent_status state if [ "$has_identity" != 1 ]; then @@ -1406,7 +1410,7 @@ _fm_composer_pi_verdict() { # <screen> <styled> <has_identity> <identity> return 0 fi case "$agent_status" in - idle|done|blocked) printf 'empty' ;; + idle|done) printf 'empty' ;; *) printf 'unknown' ;; esac } diff --git a/bin/fm-decision-hold.sh b/bin/fm-decision-hold.sh index 1e637de0151..c1a7a6c9f03 100755 --- a/bin/fm-decision-hold.sh +++ b/bin/fm-decision-hold.sh @@ -1,152 +1,37 @@ #!/usr/bin/env bash -# fm-decision-hold.sh - deterministic mechanics for durable captain decisions. +# fm-decision-hold.sh - transitional compatibility shim over bin/fm-captain-hold.sh. # -# The semantic policy is owned once by -# .agents/skills/decision-hold-lifecycle/SKILL.md. This script never reads report, -# visual-review, chat, or terminal prose to guess whether a decision exists. -# The invoking agent inventories unresolved decisions, assigns stable keys, and -# routes dependent work. This script supplies deterministic identities, creates -# and verifies structured tasks-axi captain holds, records completion attestation -# in the originating task's metadata, and requires a durable captain decision -# record before it closes or repairs a hold. +# The separate "decision" concept collapsed into the one primitive the captain +# cares about: a task held for the captain. bin/fm-captain-hold.sh owns every +# surviving behavior; this shim only maps the retired command surface onto it so +# in-flight work briefed before the collapse keeps working for one release, and +# it will be removed in the release after the collapse lands. # -# A hold identity is <origin-id>-decision-<decision-key>. Origin ids and decision -# keys must already be privacy-safe slugs. Repeating `hold` with the same identity -# is idempotent. A different decision key creates a different backlog identity. -# All backlog mutations run in the active FM_HOME, which keeps main-home and -# secondmate-home ownership aligned with the work that discovered the decision. -# -# Usage: -# fm-decision-hold.sh id <origin-id> <decision-key> -# fm-decision-hold.sh hold <origin-id> <decision-key> \ -# --title <title> --reason <reason> [--repo <repo>] -# fm-decision-hold.sh complete <origin-id> (--none | <decision-key>...) -# fm-decision-hold.sh verify <origin-id> -# fm-decision-hold.sh resolve <origin-id> <decision-key> \ -# --decision-file <path> --routed-to <task-id> [--routed-to <task-id>...] -# fm-decision-hold.sh answer <origin-id> <decision-key> --decision-file <path> -# fm-decision-hold.sh answers (<origin-id> | --any-origin) --source <provenance> (keyed answers on stdin) -# fm-decision-hold.sh bind <source-id> (<origin-id> | --any-origin) -# fm-decision-hold.sh unbind <source-id> -# fm-decision-hold.sh binding <source-id> -# fm-decision-hold.sh decline <origin-id> <decision-key> --decision-file <path> -# fm-decision-hold.sh repair <origin-id> <decision-key> --decision-file <path> -# -# `complete` is the shared investigation and visual-review completion gate. -# `--none` is an explicit semantic attestation that the just-reviewed surface has -# no unresolved captain decision. Later review passes may add keys; a live task's -# metadata inventory is unioned idempotently. A post-teardown visual review can -# complete against the surviving report and holds without recreating task state. -# `verify` is read-only and is called by scout teardown so teardown cannot erase a -# source before this gate has succeeded. -# -# `resolve`, `answer`, and `decline` close active holds; `repair` attests a hold -# already closed outside this script. All four paths require a non-empty captain -# decision file of at most 8192 bytes, record the same durable resolution block in -# the hold body, and store the decision digest plus routed identities so an exact -# retry is idempotent while a changed decision or, for `resolve`, routed set is -# rejected. New records include a `Resolution mode:` naming their path; older -# routed records remain valid. -# -# `resolve` is the routed path. It requires every --routed-to task to exist and to -# be blocked by the hold. It writes the captain decision and routed identities into -# the hold body, clears those dependency edges, and only then marks the hold Done. -# A failure before the final step leaves the captain hold open. -# -# `answer` is the answer-time closure path, the hold ledger's counterpart to -# `fm-send.sh --resolve-key`: it exists so the act that carries the captain's -# answer is the act that closes the hold, instead of leaving closure to a -# separate later call nobody is forced to make. It records the captain's answer -# on an actively held hold, records `(none)` as the routed identities because no -# follow-up work has been routed behind the hold yet, and closes it. It shares -# every guard `decline` has, including the refusal while any task is still -# blocked by the hold, so a decision whose follow-up work is already routed still -# goes through `resolve` and the routed-vs-unrouted distinction survives. It says -# only that the captain answered; `decline` still says the captain answered with -# no follow-up work at all. -# -# ONE KEYED-ANSWER INTAKE, FED BY EVERY CHANNEL. -# "A keyed answer closes its matching hold" is a single capability, owned here -# and nowhere else. `answers` is its channel-agnostic entry point: it reads -# `<decision-key>\t<answer>\t<label>` lines on stdin, maps each key to this -# origin's `<origin-id>-decision-<key>` hold, and closes it through the very same -# `answer` path above, so every guard applies identically no matter which channel -# the answer arrived on. With `--any-origin` in place of an origin id, each key -# is instead a FULL hold identity `<origin>-decision-<key>`, split at its first -# `-decision-`, so one source can carry answers for holds across origins - the -# aggregation a bearings board needs. An origin id that itself contains -# `-decision-` is outside any-origin resolution; bind such a source to its one -# origin instead. A key with no `-decision-` separator (a merge or dispatch -# instruction, for example) is reported as `skipped:` and feeds nothing, which -# keeps non-decision answers out of the hold ledger by construction. -# `--source` is provenance text recorded in the durable -# decision, never a behavior switch: this command has no per-channel branch and -# no knowledge of chat, review decks, or any transport. -# -# A channel's ONLY job is to turn whatever it received into those keyed lines and -# pipe them here. It must never map keys to holds, build decision records, decide -# resolve-versus-decline, or close a hold itself. A future channel needs no change -# here at all. -# -# The decision text is a pure function of (source, key, answer, label), which is -# what makes a replayed delivery an idempotent no-op rather than a rejected -# "different captain decision". A key whose hold is absent, already closed, or -# still blocking routed work is reported as `skipped:` and left for `resolve`; -# skipping is never forced closure, and the command exits nonzero when any key -# was skipped. -# -# `bind`, `unbind`, and `binding` record whether a captured-answer SOURCE belongs -# to one origin or uses the any-origin intake, for any channel whose answers arrive detached from the origin (a -# process-event source id, for example). The binding is a private record under -# `state/decision-bindings/`; a source with no binding feeds nothing, so this -# whole path is opt-in per source and an unbound source behaves as if it did not -# exist. `bind` deliberately does not require the source to exist yet, so a -# channel can be bound BEFORE it is armed and never produce an answer that has -# nowhere to go. `bind <source-id> --any-origin` records the any-origin marker -# instead of one origin; `binding` prints that marker verbatim and `answers` -# accepts it, so the process-event runner feeds an any-origin source through the -# same seam with no runner change. -# -# `decline` is the unrouted path for a decision the captain answered with no -# follow-up work. It takes no --routed-to task, records `(none)` as the routed -# identities, and closes an actively held hold. It refuses while any task is still -# blocked by the hold, because releasing routed work without recording it is -# `resolve`'s job. -# -# `repair` records the missing resolution block on a hold that was already closed -# outside this script, so `verify` stops failing on an origin whose decision was -# genuinely answered. It never reopens a hold, never clears a dependency edge, and -# refuses a hold that is still actively held, so an unanswered decision keeps -# blocking teardown until `resolve` or `decline` closes it with the captain's word. -# It also refuses an identity that does not carry surviving captain-hold -# provenance, so an ordinary captain-kind task cannot be repaired into a decision. +# Mapping (old -> new): +# id <origin> <key> -> prints the legacy <origin>-decision-<key> identity +# hold <origin> <key> --title --reason [--repo] +# -> hold <origin>-decision-<key> --origin <origin> ... +# complete <origin> (--none | <key>...) -> complete <origin> (--none | <origin>-decision-<key>...) +# verify <origin> -> verify <origin> +# resolve <origin> <key> --decision-file <f> --routed-to <id>... +# -> answer <origin>-decision-<key> with the routed ids +# appended to the decision text, then clear the +# recorded blocked-by edges through tasks-axi; an +# exact replay of a pre-collapse routed record reuses +# its historical digest and text before clearing edges +# answer|decline|repair <origin> <key> --decision-file <f> +# -> answer <origin>-decision-<key> --decision-file <f> +# answers (<origin> | --any-origin) --source <p> +# -> answers with the same positional (the intake resolves +# task ids first and legacy identities second) +# bind <source> (<origin> | --any-origin) -> bind <source> [<origin>] +# unbind | binding <source> -> unchanged set -eu SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" -STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" -DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" - -# shellcheck source=bin/fm-classify-lib.sh -# shellcheck disable=SC1091 -. "$SCRIPT_DIR/fm-classify-lib.sh" -# shellcheck source=bin/fm-tasks-axi-lib.sh -# shellcheck disable=SC1091 -. "$SCRIPT_DIR/fm-tasks-axi-lib.sh" -# shellcheck source=bin/fm-wake-lib.sh -# shellcheck disable=SC1091 -. "$SCRIPT_DIR/fm-wake-lib.sh" - -DECISION_META_LOCK= -DECISION_META_LOCK_HELD=0 -decision_hold_cleanup() { - if [ "$DECISION_META_LOCK_HELD" = 1 ]; then - fm_lock_release "$DECISION_META_LOCK" || true - DECISION_META_LOCK_HELD=0 - fi -} -trap decision_hold_cleanup EXIT +CAPTAIN_HOLD="$SCRIPT_DIR/fm-captain-hold.sh" usage() { awk ' @@ -162,400 +47,66 @@ fail() { } validate_slug() { # <label> <value> - local label=$1 value=$2 - case "$value" in - ''|*[!A-Za-z0-9._-]*) fail "$label must be a non-empty privacy-safe slug: $value" ;; - esac -} - -validate_one_line() { # <label> <value> - local label=$1 value=$2 - [ -n "$value" ] || fail "$label must not be empty" - case "$value" in - *$'\n'*|*$'\r'*) fail "$label must be one line" ;; + case "$2" in + ''|*[!A-Za-z0-9._-]*) fail "$1 must be a non-empty privacy-safe slug: $2" ;; esac } -sha256_text() { # <text> - if command -v shasum >/dev/null 2>&1; then - printf '%s' "$1" | shasum -a 256 | awk '{print $1}' - elif command -v sha256sum >/dev/null 2>&1; then - printf '%s' "$1" | sha256sum | awk '{print $1}' - else - fail "shasum or sha256sum is required" - fi -} - -hold_id() { # <origin-id> <decision-key> +compose() { # <origin> <key> validate_slug origin-id "$1" validate_slug decision-key "$2" - printf '%s-decision-%s\n' "$1" "$2" -} - -# The routed-identity token recorded when a close path routes no work. Slug -# validation rejects parentheses, so no real task identity can collide with it. -ROUTED_NONE='(none)' - -DECISION_TEXT='' -DECISION_DIGEST='' - -load_decision() { # <path>; sets DECISION_TEXT and DECISION_DIGEST - local path=$1 decision - [ -n "$path" ] || fail "--decision-file is required" - [ -f "$path" ] || fail "decision file does not exist: $path" - decision=$(cat "$path") - [ -n "$decision" ] || fail "decision file must not be empty" - [ "$(printf '%s' "$decision" | LC_ALL=C wc -c | tr -d ' ')" -le 8192 ] \ - || fail "decision file exceeds 8192 bytes" - DECISION_TEXT=$decision - DECISION_DIGEST=$(sha256_text "$decision") -} - -tasks_axi() { - (cd "$FM_HOME" && tasks-axi "$@") + printf '%s-decision-%s' "$1" "$2" } -require_tasks_axi() { - fm_tasks_axi_compatible || fail "compatible tasks-axi is required" - tasks-axi hold --help 2>&1 | grep -F -- '--kind captain' >/dev/null \ - || fail "tasks-axi does not expose the captain-hold contract" +task_show() { + (cd "$FM_HOME" && tasks-axi show "$1" --full) 2>/dev/null } -task_show() { # <id> - tasks_axi show "$1" --full 2>/dev/null -} - -show_field() { # <show-output> <field> +show_field() { local output=$1 field=$2 printf '%s\n' "$output" | sed -n "s/^ $field: //p" | head -1 } -origin_exists_here() { # <origin-id> - [ -f "$STATE/$1.meta" ] && return 0 - [ -f "$DATA/$1/report.md" ] && return 0 - task_show "$1" >/dev/null 2>&1 -} - -list_has_key() { # <comma-list> <key> - case ",$1," in - *",$2,"*) return 0 ;; - *) return 1 ;; - esac -} - -sorted_key_union() { # <comma-list> <newline-or-space-separated-new-keys> - local existing=$1 new=$2 - { - printf '%s\n' "$existing" | tr ',' '\n' - printf '%s\n' "$new" | tr ' ' '\n' - } | sed '/^$/d' | LC_ALL=C sort -u | paste -sd, - -} - -meta_value() { # <meta> <key> - grep "^$2=" "$1" 2>/dev/null | tail -1 | cut -d= -f2- || true -} - -origin_open_decisions() { # <origin-id> - local origin=$1 meta="$STATE/$1.meta" status_file="$STATE/$1.status" open kind last verb - open=$(status_open_decisions "$status_file") - [ -n "$open" ] || return 0 - [ -f "$meta" ] || { printf '%s' "$open"; return 0; } - kind=$(meta_value "$meta" kind) - [ -n "$kind" ] || kind=ship - if [ "$kind" != secondmate ]; then - last=$(last_status_line "$status_file") - verb=$(status_line_verb "$last") - case "$verb" in - done|failed) return 0 ;; - esac - fi - printf '%s' "$open" -} - -body_has_resolution_record() { # <hold-body> - case "$1" in - *"Resolution recorded by fm-decision-hold."*"Routed work:"*) return 0 ;; - esac - return 1 -} - -resolution_body() { # <mode> <routed-csv> [routed-task-id...] - local mode=$1 routed_csv=$2 body dep - shift 2 - # Command substitution strips the trailing newline, so restore it before the - # routed-work list to keep each entry on its own durable backlog line. - body=$(printf 'Resolution recorded by fm-decision-hold.\nDecision digest: %s\nRouted identities: %s\nResolution mode: %s\n\nCaptain decision:\n%s\n\nRouted work:' \ - "$DECISION_DIGEST" "$routed_csv" "$mode" "$DECISION_TEXT") - body="${body}"$'\n' - if [ "$#" -eq 0 ]; then - body="${body}${ROUTED_NONE}"$'\n' - else - for dep in "$@"; do - body="${body}- ${dep}"$'\n' - done - fi - printf '%s' "$body" -} - -# tasks-axi quotes multi-entry blocked_by as "a,b,c"; strip so edge ids match. -normalized_blocked_by() { # <show-output> +normalized_blocked_by() { local blocked blocked=$(show_field "$1" blocked_by | tr -d '[:space:]') blocked=${blocked#\"} blocked=${blocked%\"} + [ "$blocked" != - ] || blocked='' printf '%s' "$blocked" } -# Space-separated ids of live work still blocked by <hold-id>. The listing is only -# a cheap prefilter whose first field is always an unquoted id; every candidate is -# confirmed against its own authoritative record before it is reported. -tasks_blocked_by() { # <hold-id> - local id=$1 rows row candidate show found='' - rows=$(tasks_axi list --fields blocked_by) \ - || fail "could not read backlog work while checking what $id still blocks" - while IFS= read -r row; do - case "$row" in - *"$id"*) : ;; - *) continue ;; - esac - candidate=${row%%,*} - candidate=${candidate// /} - [ -n "$candidate" ] || continue - [ "$candidate" != "$id" ] || continue - case "$candidate" in - *[!A-Za-z0-9._-]*) continue ;; - esac - show=$(task_show "$candidate") || continue - list_has_key "$(normalized_blocked_by "$show")" "$id" || continue - found="${found}${found:+ }$candidate" - done <<EOF -$rows -EOF - printf '%s' "$found" -} - -verify_hold_active() { # <hold-id> - local id=$1 show state held kind hold_kind - show=$(task_show "$id") || fail "captain hold $id is absent from $FM_HOME/data/backlog.md" - state=$(show_field "$show" state) - held=$(show_field "$show" held) - kind=$(show_field "$show" kind) - hold_kind=$(show_field "$show" hold_kind) - [ "$state" = queued ] || fail "captain hold $id is not queued (state=$state)" - [ "$held" = yes ] || fail "captain hold $id is not active" - [ "$kind" = captain ] || fail "backlog item $id is not kind captain" - [ "$hold_kind" = captain ] || fail "backlog item $id is not held for the captain" -} - -verify_hold_resolved() { # <hold-id> - local id=$1 show state kind body - show=$(task_show "$id") || return 1 - state=$(show_field "$show" state) - kind=$(show_field "$show" kind) - body=$(show_field "$show" body) - [ "$state" = "done" ] || return 1 - [ "$kind" = captain ] || return 1 - body_has_resolution_record "$body" -} - -verify_hold_durable() { # <hold-id> - local id=$1 show state held kind hold_kind body - show=$(task_show "$id") || fail "captain decision $id is absent from $FM_HOME/data/backlog.md" - state=$(show_field "$show" state) - held=$(show_field "$show" held) - kind=$(show_field "$show" kind) - hold_kind=$(show_field "$show" hold_kind) - body=$(show_field "$show" body) - if [ "$state" = queued ] && [ "$held" = yes ] && [ "$kind" = captain ] && [ "$hold_kind" = captain ]; then - return 0 - fi - if [ "$state" = "done" ] && [ "$kind" = captain ] && body_has_resolution_record "$body"; then - return 0 - fi - fail "captain decision $id is neither actively held nor durably resolved" -} - -verify_resolution_identity() { - local id=$1 hold_body=$2 decision_digest=$3 routed_csv=$4 resolution_prefix resolution_fields recorded_digest recorded_routes - resolution_prefix='"Resolution recorded by fm-decision-hold.\nDecision digest: ' - case "$hold_body" in - "$resolution_prefix"*) resolution_fields=${hold_body#"$resolution_prefix"} ;; - *) fail "captain hold $id has no retry identity record" ;; - esac - case "$resolution_fields" in - *'\nRouted identities: '*'\n\nCaptain decision:'*) : ;; - *) fail "captain hold $id has an invalid retry identity record" ;; +list_has_key() { + case ",$1," in + *",$2,"*) return 0 ;; + *) return 1 ;; esac - recorded_digest=${resolution_fields%%\\n*} - resolution_fields=${resolution_fields#*\\nRouted identities: } - recorded_routes=${resolution_fields%%\\n*} - [ "$recorded_digest" = "$decision_digest" ] \ - || fail "captain hold $id records a different captain decision" - [ "$recorded_routes" = "$routed_csv" ] \ - || fail "captain hold $id records different routed work" -} - -command_id() { - [ "$#" -eq 2 ] || { usage >&2; exit 2; } - hold_id "$1" "$2" -} - -command_hold() { - local origin=${1:-} key=${2:-} title='' reason='' repo='' id show state kind existing_title body - [ "$#" -ge 2 ] || { usage >&2; exit 2; } - shift 2 - while [ "$#" -gt 0 ]; do - case "$1" in - --title) shift; title=${1:-} ;; - --reason) shift; reason=${1:-} ;; - --repo) shift; repo=${1:-} ;; - *) usage >&2; exit 2 ;; - esac - shift - done - validate_slug origin-id "$origin" - validate_slug decision-key "$key" - validate_one_line title "$title" - validate_one_line reason "$reason" - case "$reason" in *'('*|*')'*) fail "reason must not contain parentheses (tasks-axi hold contract)" ;; esac - require_tasks_axi - origin_exists_here "$origin" || fail "origin $origin is not owned by the active home $FM_HOME" - id=$(hold_id "$origin" "$key") - if show=$(task_show "$id"); then - state=$(show_field "$show" state) - kind=$(show_field "$show" kind) - existing_title=$(show_field "$show" title) - [ "$state" != "done" ] || fail "captain decision $id is already durably resolved; use a new decision key for a new decision" - [ "$kind" = captain ] || fail "existing backlog identity $id is not kind captain" - [ "$existing_title" = "$title" ] || fail "existing captain hold $id has a different title" - else - if [ -z "$repo" ] && [ -f "$STATE/$origin.meta" ]; then - repo=$(meta_value "$STATE/$origin.meta" project) - repo=${repo%/} - repo=${repo##*/} - fi - [ -n "$repo" ] || repo=firstmate - validate_one_line repo "$repo" - body=$(printf 'Origin: %s\nDecision key: %s\nState: awaiting captain decision.' "$origin" "$key") - tasks_axi add "$id" "$title" --kind captain --repo "$repo" --body "$body" >/dev/null \ - || fail "could not create captain decision item $id" - fi - tasks_axi hold "$id" --reason "$reason" --kind captain >/dev/null \ - || fail "could not activate captain hold $id" - verify_hold_active "$id" - printf '%s\n' "$id" } -command_complete() { - local origin=${1:-} meta previous='' supplied='' keys='' key status_file open raw_open key_seen=0 has_meta=0 - [ "$#" -ge 2 ] || { usage >&2; exit 2; } - validate_slug origin-id "$origin" - shift - meta="$STATE/$origin.meta" - [ -f "$meta" ] && has_meta=1 - if [ "$has_meta" = 1 ]; then - DECISION_META_LOCK=$(fm_meta_lock_path "$meta") || fail "could not resolve task metadata lock" - fm_lock_acquire_wait "$DECISION_META_LOCK" - DECISION_META_LOCK_HELD=1 - [ -f "$meta" ] || fail "task metadata disappeared while recording completion" - fi - require_tasks_axi - origin_exists_here "$origin" || fail "origin $origin is not owned by the active home $FM_HOME" - if [ "$#" -eq 1 ] && [ "$1" = --none ]; then - supplied='' +sha256_text() { + if command -v shasum >/dev/null 2>&1; then + printf '%s' "$1" | shasum -a 256 | awk '{print $1}' + elif command -v sha256sum >/dev/null 2>&1; then + printf '%s' "$1" | sha256sum | awk '{print $1}' else - while [ "$#" -gt 0 ]; do - [ "$1" != --none ] || fail "--none cannot be combined with decision keys" - validate_slug decision-key "$1" - supplied="${supplied}${supplied:+ }$1" - shift - done - fi - if [ "$has_meta" = 1 ]; then - previous=$(meta_value "$meta" decision_keys) - fi - keys=$(sorted_key_union "$previous" "$supplied") - if [ -n "$keys" ]; then - while IFS= read -r key; do - [ -n "$key" ] || continue - verify_hold_durable "$(hold_id "$origin" "$key")" - done <<EOF -$(printf '%s\n' "$keys" | tr ',' '\n') -EOF - fi - - status_file="$STATE/$origin.status" - raw_open=$(status_open_decisions "$status_file") - open=$(origin_open_decisions "$origin") - while IFS=$'\t' read -r key _verb _summary; do - [ -n "$key" ] || continue - list_has_key "$keys" "$key" \ - || fail "open structured decision $origin/$key has no captain-held inventory entry" - done <<EOF -$open -EOF - - if [ "$has_meta" = 1 ]; then - if [ "$(meta_value "$meta" decisions_reviewed)" != 1 ] || [ "$previous" != "$keys" ]; then - printf 'decisions_reviewed=1\ndecision_keys=%s\n' "$keys" >> "$meta" - fi - fm_lock_release "$DECISION_META_LOCK" - DECISION_META_LOCK_HELD=0 - - # Transfer any still-open status decision to its durable backlog owner so the - # live status fold does not duplicate the same Captain's Call item. - # The transfer line is this home's own bookkeeping close, written by the - # turn that just reviewed the decision, so it uses the guarded - # self-announced append (bin/fm-wake-lib.sh) and does not wake this same - # session; an append failure still fails this command loudly. - while IFS=$'\t' read -r key _verb _summary; do - [ -n "$key" ] || continue - list_has_key "$keys" "$key" || continue - transfer_rc=0 - fm_wake_status_append_self_announced "$STATE" "$status_file" \ - "captain-held [key=$key]: tracked by $(hold_id "$origin" "$key")" || transfer_rc=$? - [ "$transfer_rc" -ne 2 ] || fail "cannot append the captain-held transfer for $origin/$key" - key_seen=1 - done <<EOF -$raw_open -EOF + fail "shasum or sha256sum is required" fi - : "$key_seen" - printf 'complete: %s decision inventory reviewed%s\n' "$origin" "${keys:+ ($keys)}" } -command_verify() { - local origin=${1:-} meta reviewed keys key open - [ "$#" -eq 1 ] || { usage >&2; exit 2; } - validate_slug origin-id "$origin" - meta="$STATE/$origin.meta" - [ -f "$meta" ] || fail "origin metadata is absent: $meta" - require_tasks_axi - reviewed=$(meta_value "$meta" decisions_reviewed) - [ "$reviewed" = 1 ] || fail "origin $origin has no completed unresolved-decision inventory" - keys=$(meta_value "$meta" decision_keys) - if [ -n "$keys" ]; then - while IFS= read -r key; do - [ -n "$key" ] || continue - verify_hold_durable "$(hold_id "$origin" "$key")" - done <<EOF -$(printf '%s\n' "$keys" | tr ',' '\n') -EOF - fi - open=$(origin_open_decisions "$origin") - while IFS=$'\t' read -r key _verb _summary; do - [ -n "$key" ] || continue - list_has_key "$keys" "$key" \ - || fail "open structured decision $origin/$key is outside the reviewed inventory" - verify_hold_durable "$(hold_id "$origin" "$key")" - done <<EOF -$open -EOF - printf 'verified: %s unresolved-decision inventory\n' "$origin" +recorded_field() { + local rest=$1 label=$2 + case "$rest" in + *"$label: "*) rest=${rest#*"$label: "} ;; + *) return 1 ;; + esac + rest=${rest%%\\n*} + rest=${rest%%$'\n'*} + printf '%s' "$rest" } command_resolve() { - local origin=${1:-} key=${2:-} decision_file='' id='' body='' routed='' routed_csv='' dep show blocked state hold_show hold_body resolution_recorded=0 + local origin=${1:-} key=${2:-} decision_file='' routed='' routed_csv id dep tmp answer_file show state blocked hold_show hold_body + local resolution_recorded=0 legacy_replay=0 decision_text decision_digest recorded_digest recorded_routes [ "$#" -ge 2 ] || { usage >&2; exit 2; } shift 2 while [ "$#" -gt 0 ]; do @@ -566,336 +117,116 @@ command_resolve() { esac shift done - validate_slug origin-id "$origin" - validate_slug decision-key "$key" - load_decision "$decision_file" - [ -n "$routed" ] || fail "at least one --routed-to task is required; use decline when the captain's answer routes no work" + id=$(compose "$origin" "$key") + [ -n "$decision_file" ] || fail "--decision-file is required" + [ -f "$decision_file" ] || fail "decision file does not exist: $decision_file" + [ -n "$routed" ] || fail "at least one --routed-to task is required; use answer when the captain's answer routes no work" routed=$(printf '%s\n' "$routed" | tr ' ' '\n' | sed '/^$/d' | LC_ALL=C sort -u | paste -sd' ' -) - routed_csv=$(printf '%s\n' "$routed" | tr ' ' ',') - require_tasks_axi - id=$(hold_id "$origin" "$key") - if verify_hold_resolved "$id"; then - hold_show=$(task_show "$id") - hold_body=$(show_field "$hold_show" body) - verify_resolution_identity "$id" "$hold_body" "$DECISION_DIGEST" "$routed_csv" - printf 'resolved: %s\n' "$id" - return 0 - fi - verify_hold_active "$id" - hold_show=$(task_show "$id") + routed_csv=$(printf '%s' "$routed" | tr ' ' ',') + decision_text=$(cat "$decision_file") + [ -n "$decision_text" ] || fail "decision file must not be empty" + decision_digest=$(sha256_text "$decision_text") + hold_show=$(task_show "$id") || fail "captain decision $id does not exist in the active home" hold_body=$(show_field "$hold_show" body) case "$hold_body" in - *"Resolution recorded by fm-decision-hold."*) - verify_resolution_identity "$id" "$hold_body" "$DECISION_DIGEST" "$routed_csv" + *"Resolution recorded by fm-decision-hold."*"Routed identities: "*) + recorded_digest=$(recorded_field "$hold_body" "Decision digest" || true) + recorded_routes=$(recorded_field "$hold_body" "Routed identities" || true) + [ "$recorded_digest" = "$decision_digest" ] \ + || fail "captain decision $id records a different captain decision" + [ "$recorded_routes" = "$routed_csv" ] \ + || fail "captain decision $id records different routed work" + resolution_recorded=1 + legacy_replay=1 + ;; + *"Resolution recorded by fm-captain-hold."*) resolution_recorded=1 ;; esac - for dep in $routed; do show=$(task_show "$dep") || fail "routed task $dep does not exist in the active home" state=$(show_field "$show" state) [ "$state" != "done" ] || [ "$resolution_recorded" = 1 ] \ || fail "routed task $dep is already done" blocked=$(normalized_blocked_by "$show") - if ! list_has_key "$blocked" "$id"; then - case "$hold_body" in - *"Resolution recorded by fm-decision-hold."*"- $dep"*) : ;; - *) fail "routed task $dep is not durably blocked by $id" ;; - esac - fi + list_has_key "$blocked" "$id" || [ "$resolution_recorded" = 1 ] \ + || fail "routed task $dep is not durably blocked by $id" done - - # shellcheck disable=SC2086 # routed is a validated space-separated slug list. - body=$(resolution_body routed "$routed_csv" $routed) - tasks_axi update "$id" --body "$body" >/dev/null \ - || fail "could not record the captain decision on $id" + tmp=$(umask 077; mktemp "${TMPDIR:-/tmp}/fm-decision-hold-resolve.XXXXXX") \ + || fail "cannot stage the captain decision" + if ! { cat "$decision_file" && printf '\n\nRouted work:\n' \ + && printf '%s\n' "$routed" | tr ' ' '\n' | sed 's/^/- /'; } > "$tmp"; then + rm -f -- "$tmp" + fail "cannot stage the captain decision for $id" + fi + answer_file=$tmp + [ "$legacy_replay" = 0 ] || answer_file=$decision_file + if ! "$CAPTAIN_HOLD" answer "$id" --decision-file "$answer_file"; then + rm -f -- "$tmp" + exit 1 + fi + rm -f -- "$tmp" for dep in $routed; do show=$(task_show "$dep") || fail "routed task $dep disappeared before routing" if list_has_key "$(normalized_blocked_by "$show")" "$id"; then - tasks_axi unblock "$dep" --by "$id" >/dev/null \ + (cd "$FM_HOME" && tasks-axi unblock "$dep" --by "$id" >/dev/null) \ || fail "could not route the recorded decision to $dep" fi done - tasks_axi "done" "$id" >/dev/null || fail "could not close resolved captain hold $id" - verify_hold_resolved "$id" || fail "captain hold $id did not retain its durable resolution record" printf 'resolved: %s -> %s\n' "$id" "$routed" } -parse_decision_only_flags() { # <args...>; prints the --decision-file value - local decision_file='' - while [ "$#" -gt 0 ]; do - case "$1" in - --decision-file) shift; decision_file=${1:-} ;; - *) usage >&2; exit 2 ;; - esac - shift - done - printf '%s' "$decision_file" -} - -# The one unrouted close path, shared by `answer` and `decline`. They differ only -# in the resolution mode they record and the outcome word they print; every -# guard - the captain decision file, the active-hold requirement, the retry -# identity, and the refusal to release still-routed work - is identical, so -# neither can drift into a weaker close than the other. -close_unrouted_hold() { # <mode> <outcome-word> <origin-id> <decision-key> <flag-args...> - local mode=$1 outcome=$2 origin=$3 key=$4 decision_file id body hold_show hold_body state dependents - shift 4 - decision_file=$(parse_decision_only_flags "$@") || exit 2 +command_complete() { + local origin=${1:-} mapped='' + [ "$#" -ge 2 ] || { usage >&2; exit 2; } validate_slug origin-id "$origin" - validate_slug decision-key "$key" - load_decision "$decision_file" - require_tasks_axi - id=$(hold_id "$origin" "$key") - if verify_hold_resolved "$id"; then - hold_show=$(task_show "$id") - hold_body=$(show_field "$hold_show" body) - verify_resolution_identity "$id" "$hold_body" "$DECISION_DIGEST" "$ROUTED_NONE" - printf '%s: %s\n' "$outcome" "$id" - return 0 + shift + if [ "$#" -eq 1 ] && [ "$1" = --none ]; then + exec "$CAPTAIN_HOLD" complete "$origin" --none fi - hold_show=$(task_show "$id") || fail "captain hold $id is absent from $FM_HOME/data/backlog.md" - state=$(show_field "$hold_show" state) - [ "$state" != "done" ] \ - || fail "captain hold $id was closed outside fm-decision-hold; use repair to record the captain decision" - verify_hold_active "$id" - hold_body=$(show_field "$hold_show" body) - case "$hold_body" in - *"Resolution recorded by fm-decision-hold."*) - verify_resolution_identity "$id" "$hold_body" "$DECISION_DIGEST" "$ROUTED_NONE" - ;; - esac - dependents=$(tasks_blocked_by "$id") || exit 1 - [ -z "$dependents" ] \ - || fail "captain hold $id still blocks routed work ($dependents); use resolve to record that work" - body=$(resolution_body "$mode" "$ROUTED_NONE") - tasks_axi update "$id" --body "$body" >/dev/null \ - || fail "could not record the captain decision on $id" - tasks_axi "done" "$id" >/dev/null || fail "could not close $mode captain hold $id" - verify_hold_resolved "$id" || fail "captain hold $id did not retain its durable resolution record" - printf '%s: %s\n' "$outcome" "$id" + for key in "$@"; do + [ "$key" != --none ] || fail "--none cannot be combined with decision keys" + mapped="${mapped}${mapped:+ }$(compose "$origin" "$key")" + done + # shellcheck disable=SC2086 # mapped is a validated space-separated slug list. + exec "$CAPTAIN_HOLD" complete "$origin" $mapped } -command_answer() { +command_close() { # <origin> <key> <flag-args...> + local origin=${1:-} key=${2:-} id [ "$#" -ge 2 ] || { usage >&2; exit 2; } - close_unrouted_hold answered answered "$@" -} - -# --- the one keyed-answer intake, and the source bindings that feed it -------- - -BINDING_DIR="$STATE/decision-bindings" -BINDING_SCHEMA=fm-decision-binding.v1 - -# The any-origin binding marker. Slug validation rejects parentheses, so no real -# origin id can collide with it; `binding` prints it and `answers` accepts it, -# which is what lets the runner's feed seam carry an any-origin source unchanged. -BINDING_ANY='(any)' - -validate_source_id() { # <source-id> - validate_slug source-id "$1" - [ "${#1}" -le 64 ] || fail "source-id must be at most 64 characters: $1" -} - -binding_path() { printf '%s/%s.origin\n' "$BINDING_DIR" "$1"; } - -# The origin a captured-answer source belongs to, or empty when it is unbound. -# An unreadable or wrong-schema record is a hard error rather than a silent -# "unbound": feeding nothing is the safe direction only when it is a deliberate -# choice, never when it is a corrupted record. -read_binding() { # <source-id> - local path origin schema - path=$(binding_path "$1") - [ -e "$path" ] || return 0 - [ -f "$path" ] && [ ! -L "$path" ] || fail "decision binding is unsafe: $path" - schema=$(sed -n 's/^schema=//p' "$path" | head -1) - [ "$schema" = "$BINDING_SCHEMA" ] || fail "decision binding has an incompatible schema: $path" - origin=$(sed -n 's/^origin=//p' "$path" | head -1) - if [ "$origin" != "$BINDING_ANY" ]; then - case "$origin" in - ''|*[!A-Za-z0-9._-]*) fail "decision binding has an invalid origin id: $path" ;; - esac - fi - printf '%s\n' "$origin" -} - -command_bind() { - local source=${1:-} origin=${2:-} dest tmp - [ "$#" -eq 2 ] || { usage >&2; exit 2; } - validate_source_id "$source" - if [ "$origin" = --any-origin ]; then - origin=$BINDING_ANY - else - validate_slug origin-id "$origin" - fi - (umask 077; mkdir -p "$BINDING_DIR") || fail "cannot create $BINDING_DIR" - [ -d "$BINDING_DIR" ] && [ ! -L "$BINDING_DIR" ] || fail "decision binding dir is unsafe: $BINDING_DIR" - dest=$(binding_path "$source") - tmp=$(umask 077; mktemp "$BINDING_DIR/.origin.XXXXXX") || fail "cannot stage the decision binding" - if ! { printf 'schema=%s\norigin=%s\n' "$BINDING_SCHEMA" "$origin" > "$tmp" \ - && chmod 0600 "$tmp" && mv -f -- "$tmp" "$dest"; }; then - rm -f -- "$tmp" - fail "cannot record the decision binding for $source" - fi - printf 'bound: %s -> %s\n' "$source" "$origin" -} - -command_unbind() { - local source=${1:-} - [ "$#" -eq 1 ] || { usage >&2; exit 2; } - validate_source_id "$source" - rm -f -- "$(binding_path "$source")" - printf 'unbound: %s\n' "$source" -} - -command_binding() { - local source=${1:-} origin - [ "$#" -eq 1 ] || { usage >&2; exit 2; } - validate_source_id "$source" - origin=$(read_binding "$source") || exit 1 - [ -n "$origin" ] || return 1 - printf '%s\n' "$origin" -} - -# The durable captain decision one keyed answer records. Pure function of its -# inputs, so the same answer delivered twice is idempotent rather than a -# conflicting decision. -keyed_decision_text() { # <source> <key> <answer> <label> - printf 'Captain answered this decision through %s.\n' "$1" - printf 'Decision key: %s\n' "$2" - printf 'Answer: %s\n' "$3" - [ -z "$4" ] || printf 'Answer as shown to the captain: %s\n' "$4" -} - -sanitize_field() { # <text> - printf '%s' "$1" | tr '\n\r\t' ' ' | LC_ALL=C tr -d '\000-\037\177' | cut -c1-512 -} - -command_answers() { - local origin=${1:-} source='' any=0 key answer label hold k_origin k_key tmp err closed=0 skipped=0 reason - [ "$#" -ge 1 ] || { usage >&2; exit 2; } - shift + id=$(compose "$origin" "$key") + shift 2 + local decision_file='' while [ "$#" -gt 0 ]; do case "$1" in - --source) shift; source=${1:-} ;; + --decision-file) shift; decision_file=${1:-} ;; *) usage >&2; exit 2 ;; esac shift done - # `binding` prints the stored marker, so accept both the flag spelling a human - # types and the marker the runner pipes through unchanged. - if [ "$origin" = --any-origin ] || [ "$origin" = "$BINDING_ANY" ]; then - any=1 - origin=$BINDING_ANY - else - validate_slug origin-id "$origin" - fi - [ -n "$source" ] || fail "--source provenance is required so the durable decision records where the answer came from" - source=$(sanitize_field "$source") - require_tasks_axi - tmp=$(umask 077; mktemp "${TMPDIR:-/tmp}/fm-keyed-decision.XXXXXX") || fail "cannot stage the captain decision" - err=$(umask 077; mktemp "${TMPDIR:-/tmp}/fm-keyed-decision-err.XXXXXX") \ - || { rm -f -- "$tmp"; fail "cannot stage the captain decision diagnostics"; } - while IFS=$'\t' read -r key answer label; do - [ -n "${key:-}" ] || continue - case "$key" in *[!A-Za-z0-9._-]*) continue ;; esac - if [ "$any" = 1 ]; then - [ "${#key}" -le 128 ] || continue - else - [ "${#key}" -le 64 ] || continue - fi - answer=$(sanitize_field "${answer:-}") - [ -n "$answer" ] || continue - label=$(sanitize_field "${label:-}") - if [ "$any" = 1 ]; then - # Each key is a full hold identity; split at its FIRST -decision- so the - # origin half can never swallow a later separator inside the decision key. - case "$key" in - *-decision-*) : ;; - *) - printf 'skipped: %s (not a full hold identity)\n' "$key" - skipped=$((skipped + 1)) - continue - ;; - esac - k_origin=${key%%-decision-*} - k_key=${key#*-decision-} - hold=$key - else - k_origin=$origin - k_key=$key - hold="$origin-decision-$key" - fi - keyed_decision_text "$source" "$k_key" "$answer" "$label" > "$tmp" \ - || fail "cannot stage the captain decision for $hold" - if "$0" answer "$k_origin" "$k_key" --decision-file "$tmp" >/dev/null 2>"$err"; then - printf 'closed: %s\n' "$hold" - closed=$((closed + 1)) - else - reason=$(tr -d '\n' < "$err" | sed 's/^fm-decision-hold: //') - printf 'skipped: %s (%s)\n' "$hold" "$reason" - skipped=$((skipped + 1)) - fi - done - rm -f -- "$tmp" "$err" - printf 'answers: closed=%s skipped=%s origin=%s\n' "$closed" "$skipped" "$origin" - [ "$skipped" -eq 0 ] + exec "$CAPTAIN_HOLD" answer "$id" --decision-file "$decision_file" } -command_decline() { - [ "$#" -ge 2 ] || { usage >&2; exit 2; } - close_unrouted_hold declined declined "$@" -} - -command_repair() { - local origin=${1:-} key=${2:-} decision_file id body show state kind hold_kind hold_body +command_hold() { + local origin=${1:-} key=${2:-} id [ "$#" -ge 2 ] || { usage >&2; exit 2; } + id=$(compose "$origin" "$key") shift 2 - decision_file=$(parse_decision_only_flags "$@") || exit 2 - validate_slug origin-id "$origin" - validate_slug decision-key "$key" - load_decision "$decision_file" - require_tasks_axi - id=$(hold_id "$origin" "$key") - show=$(task_show "$id") || fail "captain decision $id is absent from $FM_HOME/data/backlog.md" - kind=$(show_field "$show" kind) - [ "$kind" = captain ] || fail "backlog item $id is not kind captain" - # tasks-axi keeps hold_kind after a close, so it is the surviving proof that - # this identity really was a captain hold rather than an ordinary captain-kind - # task that was never held for the captain at all. - hold_kind=$(show_field "$show" hold_kind) - [ "$hold_kind" = captain ] \ - || fail "backlog item $id was never held for the captain; repair records a captain decision only on a captain hold" - state=$(show_field "$show" state) - hold_body=$(show_field "$show" body) - if [ "$state" = "done" ] && body_has_resolution_record "$hold_body"; then - verify_resolution_identity "$id" "$hold_body" "$DECISION_DIGEST" "$ROUTED_NONE" - printf 'repaired: %s\n' "$id" - return 0 - fi - [ "$state" = "done" ] \ - || fail "captain hold $id is still open (state=$state); use resolve or decline to close it with the captain's decision" - body=$(resolution_body repaired "$ROUTED_NONE") - tasks_axi update "$id" --body "$body" >/dev/null \ - || fail "could not record the captain decision on $id" - show=$(task_show "$id") || fail "captain decision $id disappeared while recording the repair" - [ "$(show_field "$show" state)" = "done" ] || fail "repairing $id reopened a closed captain decision" - verify_hold_resolved "$id" || fail "captain hold $id did not retain its durable resolution record" - printf 'repaired: %s\n' "$id" + exec "$CAPTAIN_HOLD" hold "$id" --origin "$origin" "$@" } case "${1:-}" in - id) shift; command_id "$@" ;; + id) shift; [ "$#" -eq 2 ] || { usage >&2; exit 2; }; compose "$1" "$2"; printf '\n' ;; hold) shift; command_hold "$@" ;; complete) shift; command_complete "$@" ;; - verify) shift; command_verify "$@" ;; + verify) shift; exec "$CAPTAIN_HOLD" verify "$@" ;; resolve) shift; command_resolve "$@" ;; - answer) shift; command_answer "$@" ;; - answers) shift; command_answers "$@" ;; - bind) shift; command_bind "$@" ;; - unbind) shift; command_unbind "$@" ;; - binding) shift; command_binding "$@" ;; - decline) shift; command_decline "$@" ;; - repair) shift; command_repair "$@" ;; + answer|decline|repair) shift; command_close "$@" ;; + answers) shift; exec "$CAPTAIN_HOLD" answers "$@" ;; + bind) shift; exec "$CAPTAIN_HOLD" bind "$@" ;; + unbind) shift; exec "$CAPTAIN_HOLD" unbind "$@" ;; + binding) shift; exec "$CAPTAIN_HOLD" binding "$@" ;; -h|--help) usage ;; *) usage >&2; exit 2 ;; esac diff --git a/bin/fm-fleet-snapshot.sh b/bin/fm-fleet-snapshot.sh index bc7f1a3c479..c257d110be5 100755 --- a/bin/fm-fleet-snapshot.sh +++ b/bin/fm-fleet-snapshot.sh @@ -15,11 +15,21 @@ # data/backlog.md and cover In flight, Queued, and Done. # Canonical tasks-axi rows are structured; free-form non-empty lines in # those sections are preserved as unstructured records. -# Structured rows preserve captain-hold metadata such as hold_kind and -# hold_reason when tasks-axi emits it. They also carry normalized current_role, -# requires_child_metadata, blocked_by_ids, unresolved_blocker_ids, and -# captain_actionable fields. Repeated blocker tokens remain ordered; a blocker -# resolves only when its structured record is Done, and missing ids stay open. +# Structured rows preserve captain-hold metadata such as hold_kind, +# hold_reason, and hold_until when tasks-axi emits it. They also carry +# normalized current_role, requires_child_metadata, blocked_by_ids, +# unresolved_blocker_ids, captain_actionable, and deferred_marker fields. +# Repeated blocker tokens remain ordered; a blocker resolves only when its +# structured record is Done, and missing ids stay open. +# captain_actionable means "waiting on the captain now": queued, held for +# the captain, unblocked, and due (no hold_until, or hold_until at or +# before the observation date, matching tasks-axi's own date-gate rule). +# There is no separate decision type: any captain-held task is the same +# primitive, whatever kind its row carries. +# deferred_marker is a presentation hint only: the row's hold reason or +# body carries an explicit SUPERSEDED / NOT REQUIRED / DEFERRED marker. +# It never changes captain_actionable; renderers may use it to keep +# prose-deferred rows out of default views. # tasks[]: one row per state/<id>.meta, sorted by id. # current_state is parsed from bin/fm-crew-state.sh <id> and preserves # state, source, detail, and raw line separately. @@ -74,6 +84,14 @@ else || date +%s) fi case "$SNAPSHOT_EPOCH" in ''|*[!0-9]*) SNAPSHOT_EPOCH=$(date +%s) ;; esac +# The observation date gates captain-hold deferral: a `hold-until` date still in +# the future keeps a captain hold out of captain_actionable until it is due +# (tasks-axi's own contract: the hold is inactive on and after that date). +SNAPSHOT_TODAY=${SNAPSHOT_NOW%%T*} +case "$SNAPSHOT_TODAY" in + [0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]) : ;; + *) SNAPSHOT_TODAY=$(date -u +%Y-%m-%d) ;; +esac # Cross-home bounds are explicit so one broken or unexpectedly large home cannot # hang or explode the parent snapshot. @@ -151,8 +169,9 @@ validated registered-home handoff. It is local-only, skips nested secondmate aggregation, and marks inventory contradictions or unavailable child state invalid. Its invalidity object names the normalized failure kind and affected ids. Actionable tasks-axi captain holds appear as decisions_open and stay visible in -queued with hold_reason, hold_kind, and plural blocker fields for downstream -projections. A captain hold is actionable only when every blocker is Done. +queued with hold_reason, hold_kind, hold_until, deferred_marker, and plural +blocker fields for downstream projections. A captain hold is actionable only +when every blocker is Done and any hold-until date has arrived. Cross-home reads use FM_SNAPSHOT_SECONDMATES (default 20, 0 lifts the count bound), FM_SNAPSHOT_SECONDMATE_TIMEOUT, and FM_SNAPSHOT_SECONDMATE_MAX_BYTES. Terminal contradiction evidence uses @@ -258,7 +277,7 @@ backlog_json() { # [<backlog-path>] - defaults to this home's $BACKLOG fi # shellcheck disable=SC2094 - jq -Rn --arg path "$backlog" ' + jq -Rn --arg path "$backlog" --arg today "$SNAPSHOT_TODAY" ' def trim: gsub("^[[:space:]]+|[[:space:]]+$"; ""); def section_state: if . == "In flight" then "in_flight" @@ -277,7 +296,7 @@ backlog_json() { # [<backlog-path>] - defaults to this home's $BACKLOG def links($rest): [$rest | scan(url_pattern)]; def strip_trailing_metadata: reduce range(0; 20) as $_ (.; - sub("[[:space:]]*\\([[:space:]]*(?:(?:repo|kind|priority|hold|hold-kind):[[:space:]]*[^)]*|(?:since|merged|reported|done)[[:space:]]+[^)]*)[[:space:]]*\\)[[:space:]]*$"; "")); + sub("[[:space:]]*\\([[:space:]]*(?:(?:repo|kind|priority|hold|hold-kind|hold-until):[[:space:]]*[^)]*|(?:since|merged|reported|done)[[:space:]]+[^)]*)[[:space:]]*\\)[[:space:]]*$"; "")); def strip_title_artifacts: sub("[[:space:]]+-[[:space:]]+data/[^[:space:])]+/report\\.md$"; "") | sub("[[:space:]]+data/[^[:space:])]+/report\\.md$"; "") @@ -337,6 +356,7 @@ backlog_json() { # [<backlog-path>] - defaults to this home's $BACKLOG priority:metadata($rest; "priority"), hold_reason:metadata($rest; "hold"), hold_kind:metadata($rest; "hold-kind"), + hold_until:metadata($rest; "hold-until"), blocked_by:cap($rest; ".*blocked-by:[[:space:]]*(?<v>[^[:space:])]+).*"), blocked_by_ids:blocked_by_ids($rest), blocked_reason:blocked_reason($rest), @@ -393,8 +413,12 @@ backlog_json() { # [<backlog-path>] - defaults to this home's $BACKLOG else "done" end) | .requires_child_metadata = (.current_role == "worker") | .captain_actionable = - (.state == "queued" and .kind == "captain" and .hold_kind == "captain" - and .hold_reason != null and (.unresolved_blocker_ids | length) == 0) + (.state == "queued" and .hold_kind == "captain" + and .hold_reason != null and (.unresolved_blocker_ids | length) == 0 + and (.hold_until == null or .hold_until <= $today)) + | .deferred_marker = + ((((.hold_reason // "") + " " + (.body_excerpt // "")) + | test("SUPERSEDED|NOT REQUIRED|NOT-REQUIRED|DEFERRED"; "i"))) else . end) | del(.section,.order) ' < "$backlog" @@ -660,8 +684,10 @@ secondmate_home_summary_json() { # <backlog-json> <tasks-json> | ([ $queued_all[] | select(.captain_actionable == true) | {id,key:.id,verb:"captain-hold",summary:(.title | trunc(160)), - reason:(.hold_reason | trunc(160)),source:"backlog"} ]) as $captain_holds_all - | ([ $backlog.records[]? | select(.state == "done" and .structured and .kind != "captain") + reason:(.hold_reason | trunc(160)), + hold_until:(.hold_until // null), + deferred_marker:(.deferred_marker // false),source:"backlog"} ]) as $captain_holds_all + | ([ $backlog.records[]? | select(.state == "done" and .structured and .hold_kind != "captain") | {id:(.id | trunc(120)),title:(.title | trunc(120)), pr_url:((.pr_url // null) | if . == null then null else trunc(500) end), report_path:((.report_path // null) | if . == null then null else trunc(500) end), @@ -757,6 +783,8 @@ secondmate_home_summary_json() { # <backlog-json> <tasks-json> blocked_reason:((.blocked_reason // null) | if . == null then null else trunc(160) end), hold_reason:((.hold_reason // null) | if . == null then null else trunc(160) end), hold_kind:((.hold_kind // null) | if . == null then null else trunc(40) end), + hold_until:((.hold_until // null) | if . == null then null else trunc(40) end), + deferred_marker:(.deferred_marker // false), captain_actionable:(.captain_actionable // false), repo:((.repo // null) | if . == null then null else trunc(120) end), kind:((.kind // null) | if . == null then null else trunc(40) end)}][:$queued_n]), diff --git a/bin/fm-fleet-sync.sh b/bin/fm-fleet-sync.sh index d5c951e1a74..dd00be86baa 100755 --- a/bin/fm-fleet-sync.sh +++ b/bin/fm-fleet-sync.sh @@ -13,6 +13,11 @@ # stashed, or discarded. # Still skips (benignly) local-only/no-origin projects, missing remotes/branches, # and fetch failures. +# A candidate under projects/ must be the root of its own work tree: git discovery +# walks up, so a plain nested directory would otherwise resolve to the enclosing +# repository (the firstmate checkout) and be synced under that directory's label. +# Anything else is reported as "skipped: not a clone root" naming the repository +# that would have been touched. # Pruning never deletes the checked-out branch or a branch that still has a # worktree, so it cannot discard unlanded work; set FM_FLEET_PRUNE=0 to disable it. # When the fetch fails on an orphaned .git/packed-refs.lock (left by a ref rewrite @@ -300,10 +305,25 @@ sync_project() { echo "$label: skipped: not a directory" return 0 fi - if ! git -C "$PROJ" rev-parse --is-inside-work-tree >/dev/null 2>&1; then + # Git repository discovery walks UP from $PROJ, so a plain directory merely + # nested inside a repository - a worktree container left under projects/, say - + # resolves to the ENCLOSING repository, which in a firstmate home is the + # firstmate checkout itself. Every later `git -C "$PROJ"` would then read, prune + # and fast-forward that repository under this project's label, turning a routine + # refresh into an unrequested self-update reported as a project sync. Require + # $PROJ to be the root of its own work tree before any other git command runs. + proj_top=$(git -C "$PROJ" rev-parse --show-toplevel 2>/dev/null) || proj_top="" + if [ -z "$proj_top" ]; then echo "$label: skipped: not a git repo" return 0 fi + # Both sides are physical paths (git resolves --show-toplevel through symlinks), + # so a symlinked clone dir still compares equal to its own root. + proj_abs=$(cd "$PROJ" && pwd -P) || proj_abs="" + if [ "$proj_top" != "$proj_abs" ]; then + echo "$label: skipped: not a clone root (git would act on $proj_top)" + return 0 + fi mode_line=$("$FM_ROOT/bin/fm-project-mode.sh" "$label" 2>/dev/null || echo "no-mistakes off") mode=${mode_line%% *} if [ "$mode" = "local-only" ]; then diff --git a/bin/fm-inbox.sh b/bin/fm-inbox.sh new file mode 100755 index 00000000000..3f967fd80f2 --- /dev/null +++ b/bin/fm-inbox.sh @@ -0,0 +1,399 @@ +#!/usr/bin/env bash +# fm-inbox.sh - the captain's out-of-band capture surface. +# +# Solves three DIFFERENT problems with three different mechanisms, because they +# are not the same problem: +# +# note Queue an idea for firstmate while firstmate is mid-turn and cannot +# answer. Writes a durable record and appends ONE `check` wake, so the +# note survives a crash and is presented at firstmate's next drain. +# This is the only subcommand that touches firstmate's wake queue. +# say Same as `note`, but the body comes from spoken audio on stdin. +# Speech is an INPUT METHOD here, not an architecture: it transcribes +# and then takes exactly the `note` path. +# status Answer "what is happening" from durable records ONLY. Reads no +# network and appends NO wake, so it never interrupts work and is safe +# to run in a loop. +# ask Answer a side question with a one-shot model call that never touches +# firstmate, the backlog, or the wake queue. A side question is not +# fleet work and must not become fleet work. +# +# Usage: +# fm-inbox.sh note <text>... | fm-inbox.sh note - (body from stdin) +# fm-inbox.sh say [<file.wav>] (default: audio on stdin) +# fm-inbox.sh status +# fm-inbox.sh ask <question>... +# fm-inbox.sh list +# fm-inbox.sh drain [--ack <id>...] +# +# Configuration. A region, a model id and an AWS profile name somebody's account +# and somebody's choices, so this file carries no default for any of them. Each is +# read from the home's gitignored config/ directory, or from the matching +# environment variable, and the model-backed subcommands refuse with the path to +# write rather than reaching for a value that belongs to another home. That +# configuration is also the opt-in: `say` and `ask` are off until it exists. +# +# config/inbox-region FM_INBOX_REGION AWS region. required +# config/inbox-stt-model FM_INBOX_STT_MODEL speech-to-text model. required by say +# config/inbox-ask-model FM_INBOX_ASK_MODEL side-question model. required by ask +# config/inbox-profile FM_INBOX_PROFILE AWS profile. optional +# +# An absent profile means the call uses whatever credentials are already in the +# environment, which is also what FM_INBOX_PROFILE= (empty) forces. +# +# `note`, `status`, `list` and `drain` need NO configuration at all, because they +# make no model call. The voice handover depends on `note`, so it keeps working in +# a home that has configured nothing. +# +# Environment: +# FM_HOME operational home whose state/ and data/ are used. +# +# PRIVACY: `say` sends your audio and `ask` sends your question to Bedrock. +# `note`, `status`, `list` and `drain` make no network call at all. +# +# `note` is also the queueing half of the spoken interface: when the voice agent +# in bin/fm-voice-relay.py hands real work over to firstmate, it runs this +# subcommand rather than carrying a second queue of its own. Keep the `note` +# contract stable for that caller. `status` is the HUMAN view of the records; +# bin/fm_voice_records.py owns the scope-controlled machine view the voice agent +# reads, because the voice agent must be able to answer without record free text +# ever reaching a model. +set -euo pipefail + +# A non-interactive `ssh host fm-inbox.sh ...` does NOT get a login shell, so it +# does not get ~/.toolbox/bin on PATH. The AWS profile's credential_process is +# the bare word `ada`, so without this the model-backed subcommands fail with +# "[Errno 2] No such file or directory: 'ada'" while note/status still work. +# Verified: this is exactly what happens over SSH without the fix. +for _extra in "$HOME/.toolbox/bin" "$HOME/.local/bin"; do + case ":$PATH:" in + *":$_extra:"*) ;; + *) [ -d "$_extra" ] && PATH="$_extra:$PATH" ;; + esac +done +unset _extra +export PATH + +SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_ROOT="$(cd "$SELF_DIR/.." && pwd)" +FM_HOME="${FM_HOME:-$FM_ROOT}" +STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" +DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" +INBOX="$STATE/inbox" + +CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" + +die() { printf 'fm-inbox: %s\n' "$*" >&2; exit 1; } + +# First non-comment, non-blank line of a config file, or nothing. +read_setting() { # <file-name> + local path="$CONFIG/$1" line + [ -r "$path" ] || return 0 + while IFS= read -r line || [ -n "$line" ]; do + line=${line%%#*} + line=${line#"${line%%[![:space:]]*}"} + line=${line%"${line##*[![:space:]]}"} + [ -n "$line" ] || continue + printf '%s' "$line" + return 0 + done < "$path" +} + +# Refuse by naming the file to write. A model call that guessed at a region or an +# account would either fail confusingly or, worse, succeed against a stranger's. +require_setting() { # <file-name> <env-var> <what> + local value + value=$(read_setting "$1") + [ -n "$value" ] || die "no $3 is configured: write one line into $CONFIG/$1 or set $2" + printf '%s' "$value" +} + +REGION="${FM_INBOX_REGION:-}" +STT_MODEL="${FM_INBOX_STT_MODEL:-}" +ASK_MODEL="${FM_INBOX_ASK_MODEL:-}" +# Unset falls through to config; explicitly empty means "use ambient credentials". +PROFILE="${FM_INBOX_PROFILE-$(read_setting inbox-profile)}" + +# Resolved only by the subcommands that make a model call, so note, status, list +# and drain keep working in a home that has configured nothing. +need_region() { + [ -n "$REGION" ] || REGION=$(require_setting inbox-region FM_INBOX_REGION "AWS region") +} + +need_stt_model() { + need_region + [ -n "$STT_MODEL" ] || STT_MODEL=$(require_setting inbox-stt-model \ + FM_INBOX_STT_MODEL "speech-to-text model") +} + +need_ask_model() { + need_region + [ -n "$ASK_MODEL" ] || ASK_MODEL=$(require_setting inbox-ask-model \ + FM_INBOX_ASK_MODEL "side-question model") +} + +need() { command -v "$1" >/dev/null 2>&1 || die "required command not found: $1"; } + +# The profile's credential_process (`ada`) costs a MEASURED ~1030ms on every +# single call, which is about half the wall time of `say` and `ask`. If real +# credentials are already in the environment, skip --profile entirely and let the +# ambient ones win. Set FM_INBOX_PROFILE= (empty) to force that even without env +# credentials present. +aws_call() { + if [ -z "$PROFILE" ] || [ -n "${AWS_ACCESS_KEY_ID:-}" ]; then + aws --region "$REGION" "$@" + else + aws --profile "$PROFILE" --region "$REGION" "$@" + fi +} + +# ---------------------------------------------------------------- note + +# Append exactly one wake so firstmate picks the note up at its next drain. +# Failure to wake is NOT allowed to lose the note: the record is already on +# disk, so we report the wake failure and still exit non-zero loudly. +wake_for() { + local id=$1 summary=$2 lib="$FM_ROOT/bin/fm-wake-lib.sh" + if [ ! -r "$lib" ]; then + printf 'fm-inbox: note saved but NOT announced (missing %s)\n' "$lib" >&2 + return 1 + fi + # shellcheck source=/dev/null + FM_ROOT_OVERRIDE="$FM_ROOT" FM_HOME="$FM_HOME" STATE="$STATE" . "$lib" + fm_wake_append check "inbox:$id" "check: captain inbox note $id - $summary" +} + +queue_note() { + local source=$1 body=$2 extra=${3:-} + [ -n "${body//[[:space:]]/}" ] || die "refusing to queue an empty note" + mkdir -p "$INBOX" + + local tmp id summary + tmp=$(mktemp "$INBOX/.staging-XXXXXX") + { + printf 'id=PENDING\n' + printf 'at=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" + printf 'source=%s\n' "$source" + [ -z "$extra" ] || printf '%s\n' "$extra" + printf -- '--\n' + printf '%s\n' "$body" + } >"$tmp" + + id="$(date +%s)-$(basename "$tmp" | sed 's/^\.staging-//')" + # Rewrite the id line now that we know it, then publish atomically. + sed -i "s/^id=PENDING$/id=$id/" "$tmp" + mv "$tmp" "$INBOX/$id.note" + + # One-line summary for the wake payload; the full body stays in the file. + summary=$(printf '%s' "$body" | tr '\n\t' ' ' | cut -c1-100) + printf 'queued %s\n' "$id" + printf ' %s\n' "$summary" + if wake_for "$id" "$summary"; then + printf ' firstmate will pick this up at its next check.\n' + else + die "note $id is saved at $INBOX/$id.note but firstmate was NOT woken" + fi +} + +cmd_note() { + local body + if [ "$#" -eq 0 ]; then + die "usage: fm-inbox.sh note <text>... (or: note - to read stdin)" + elif [ "$1" = "-" ]; then + body=$(cat) + else + body="$*" + fi + queue_note text "$body" +} + +# ---------------------------------------------------------------- say + +cmd_say() { + # Before the tool checks, so an unconfigured home is told what to configure + # rather than what to install for a call it is not yet allowed to make. + need_stt_model + need aws + need python3 + need base64 + + local src wav raw transcript + raw=$(mktemp /tmp/fm-inbox-audio-XXXXXX) + wav=$(mktemp /tmp/fm-inbox-wav-XXXXXX.wav) + # shellcheck disable=SC2064 + trap "rm -f '$raw' '$wav' '$wav.json'" EXIT + + if [ "$#" -ge 1 ] && [ "$1" != "-" ]; then + src=$1 + [ -r "$src" ] || die "cannot read audio file: $src" + cat "$src" >"$raw" + else + cat >"$raw" + fi + [ -s "$raw" ] || die "no audio received on stdin" + + # Accept a real WAV as-is; wrap headerless 16kHz mono s16le PCM if that is + # what arrived. Anything else is rejected rather than silently mistranscribed. + python3 - "$raw" "$wav" <<'PY' +import sys, wave +src, dst = sys.argv[1], sys.argv[2] +data = open(src, 'rb').read() +if data[:4] == b'RIFF': + open(dst, 'wb').write(data) + sys.stderr.write("fm-inbox: input is WAV, passing through\n") +elif data[:4] in (b'OggS', b'fLaC') or data[:3] == b'ID3': + sys.exit("fm-inbox: got Ogg/FLAC/MP3; re-encode to WAV first") +else: + if len(data) % 2: + data = data[:-1] + w = wave.open(dst, 'wb') + w.setnchannels(1); w.setsampwidth(2); w.setframerate(16000) + w.writeframes(data); w.close() + sys.stderr.write("fm-inbox: input looked like raw PCM, wrapped as 16kHz mono WAV\n") +PY + + local secs + secs=$(python3 -c " +import wave,sys +w=wave.open('$wav'); print(round(w.getnframes()/w.getframerate(),2))") + printf 'fm-inbox: %ss of audio, transcribing with %s in %s\n' "$secs" "$STT_MODEL" "$REGION" >&2 + + python3 - "$wav" "$wav.json" <<'PY' +import base64, json, sys +b = base64.b64encode(open(sys.argv[1], 'rb').read()).decode() +json.dump([{"role": "user", "content": [ + {"audio": {"format": "wav", "source": {"bytes": b}}}, + {"text": "Transcribe the speech exactly. Output only the transcript, nothing else."}, +]}], open(sys.argv[2], 'w')) +PY + + transcript=$(aws_call bedrock-runtime converse \ + --model-id "$STT_MODEL" \ + --messages "file://$wav.json" \ + --inference-config '{"maxTokens":600,"temperature":0}' \ + --query 'output.message.content[0].text' --output text) \ + || die "transcription failed" + + [ -n "${transcript//[[:space:]]/}" ] || die "transcription came back empty" + printf 'fm-inbox: heard: %s\n' "$transcript" >&2 + queue_note voice "$transcript" "transcript_model=$STT_MODEL +audio_seconds=$secs" +} + +# ---------------------------------------------------------------- status + +cmd_status() { + local pending=0 + [ -d "$INBOX" ] && pending=$(find "$INBOX" -maxdepth 1 -name '*.note' 2>/dev/null | wc -l | tr -d ' ') + + printf '=== firstmate status (read-only, no wake sent) ===\n' + printf 'home %s\n' "$FM_HOME" + printf 'time %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" + printf 'inbox %s note(s) waiting for firstmate\n' "$pending" + + if [ -f "$DATA/backlog.md" ]; then + printf '\n--- in flight ---\n' + awk '/^## In flight/{f=1;next} /^## /{f=0} f && /^- \[/{print}' \ + "$DATA/backlog.md" | sed 's/^- \[ \] / /' | cut -c1-150 + else + printf '\n(no backlog at %s)\n' "$DATA/backlog.md" + fi + + local any=0 + for m in "$STATE"/*.meta; do + [ -e "$m" ] || break + if [ "$any" -eq 0 ]; then printf '\n--- workers ---\n'; any=1; fi + local id kind mode last + id=$(basename "$m" .meta) + kind=$(sed -n 's/^kind=//p' "$m" | head -1) + mode=$(sed -n 's/^mode=//p' "$m" | head -1) + last="" + [ -f "$STATE/$id.status" ] && last=$(tail -1 "$STATE/$id.status" 2>/dev/null | cut -c1-100) + printf ' %-42s %-6s %-10s %s\n' "$id" "${kind:-?}" "${mode:--}" "${last:-(no events yet)}" + done + [ "$any" -eq 1 ] || printf '\n(no workers on deck)\n' + + printf '\nNote: the last event line is history, not current state.\n' +} + +# ---------------------------------------------------------------- ask + +cmd_ask() { + [ "$#" -gt 0 ] || die "usage: fm-inbox.sh ask <question>..." + need_ask_model + need aws + need python3 + local q="$*" msg + msg=$(mktemp /tmp/fm-inbox-ask-XXXXXX.json) + # shellcheck disable=SC2064 + trap "rm -f '$msg'" EXIT + + Q="$q" python3 - "$msg" <<'PY' +import json, os, sys +json.dump([{"role": "user", "content": [{"text": os.environ["Q"]}]}], + open(sys.argv[1], 'w')) +PY + + aws_call bedrock-runtime converse \ + --model-id "$ASK_MODEL" \ + --messages "file://$msg" \ + --system '[{"text":"You are a terse engineering assistant answering a side question. Be direct and concrete. No preamble. If you are not sure, say so."}]' \ + --inference-config '{"maxTokens":700,"temperature":0.2}' \ + --query 'output.message.content[0].text' --output text \ + || die "ask failed" +} + +# ---------------------------------------------------------------- list / drain + +cmd_list() { + [ -d "$INBOX" ] || { printf '(inbox empty)\n'; return 0; } + local any=0 + for f in "$INBOX"/*.note; do + [ -e "$f" ] || break + any=1 + printf '%s\n' "$(basename "$f" .note)" + sed -n '/^--$/,$p' "$f" | tail -n +2 | sed 's/^/ /' + done + [ "$any" -eq 1 ] || printf '(inbox empty)\n' +} + +cmd_drain() { + if [ "${1:-}" = "--ack" ]; then + shift + [ "$#" -gt 0 ] || die "usage: fm-inbox.sh drain --ack <id>..." + mkdir -p "$INBOX/handled" + local id + for id in "$@"; do + if [ -f "$INBOX/$id.note" ]; then + mv "$INBOX/$id.note" "$INBOX/handled/$id.note" + printf 'acked %s\n' "$id" + else + printf 'already-acked %s\n' "$id" + fi + done + return 0 + fi + cmd_list + printf '\nAck with: fm-inbox.sh drain --ack <id>...\n' +} + +# ---------------------------------------------------------------- dispatch + +case "${1:-}" in + note) shift; cmd_note "$@" ;; + say) shift; cmd_say "$@" ;; + status) shift; cmd_status ;; + ask) shift; cmd_ask "$@" ;; + list) shift; cmd_list ;; + drain) shift; cmd_drain "$@" ;; + ''|-h|--help|help) + # The whole header block, found rather than counted: everything after the + # shebang up to the first line that is not a comment. A fixed line range + # silently truncates this help the next time the header grows, and the last + # thing to fall off the end is the PRIVACY paragraph, which is the one place + # a new operator is told which subcommands send anything off this host. + awk 'NR == 1 { next } + /^#/ { sub(/^# ?/, ""); print; next } + { exit }' "${BASH_SOURCE[0]}" ;; + *) die "unknown subcommand: $1 (try --help)" ;; +esac diff --git a/bin/fm-lint-workflows.sh b/bin/fm-lint-workflows.sh index 0e2d7b07e19..41883d10012 100755 --- a/bin/fm-lint-workflows.sh +++ b/bin/fm-lint-workflows.sh @@ -108,15 +108,15 @@ else fi if ! command -v actionlint >/dev/null 2>&1; then - printf 'fm-lint-workflows.sh: actionlint not found; install actionlint %s for CI parity.\n' \ + printf 'fm-lint-workflows.sh: actionlint not found; install actionlint %s with bin/fm-install-actionlint.sh <destination-directory> and put that directory on PATH.\n' \ "$REQUIRED_ACTIONLINT" >&2 - exit 127 + exit 1 fi ACTIONLINT_BIN=$(command -v actionlint) resolved=$("$ACTIONLINT_BIN" -version | awk 'NR==1 {print; exit}') printf 'fm-lint-workflows.sh: actionlint %s (pinned %s)\n' "$resolved" "$REQUIRED_ACTIONLINT" >&2 if [ "$resolved" != "$REQUIRED_ACTIONLINT" ]; then - printf 'fm-lint-workflows.sh: actionlint %s required for CI parity, found %s. Install %s.\n' \ + printf 'fm-lint-workflows.sh: actionlint %s required for CI parity, found %s. Install %s with bin/fm-install-actionlint.sh <destination-directory>.\n' \ "$REQUIRED_ACTIONLINT" "$resolved" "$REQUIRED_ACTIONLINT" >&2 exit 1 fi diff --git a/bin/fm-lint.sh b/bin/fm-lint.sh index d848a2ac83f..53a3f0cff89 100755 --- a/bin/fm-lint.sh +++ b/bin/fm-lint.sh @@ -230,9 +230,9 @@ if [ "$LIST_FILES" -eq 1 ]; then fi if ! command -v shellcheck >/dev/null 2>&1; then - printf 'fm-lint.sh: ShellCheck not found; install ShellCheck %s for CI parity.\n' \ + printf 'fm-lint.sh: ShellCheck not found; install ShellCheck %s with bin/fm-install-shellcheck.sh <destination-directory> and put that directory on PATH.\n' \ "$REQUIRED_SHELLCHECK" >&2 - exit 127 + exit 1 fi unset SHELLCHECK_OPTS SHELLCHECK_BIN=$(command -v shellcheck) @@ -243,7 +243,7 @@ fi resolved=$("$SHELLCHECK_BIN" --version | awk '/^version:/ {print $2; exit}') printf 'fm-lint.sh: ShellCheck %s (pinned %s)\n' "$resolved" "$REQUIRED_SHELLCHECK" >&2 if [ "$resolved" != "$REQUIRED_SHELLCHECK" ]; then - printf 'fm-lint.sh: ShellCheck %s required for CI parity, found %s. Install %s.\n' \ + printf 'fm-lint.sh: ShellCheck %s required for CI parity, found %s. Install %s with bin/fm-install-shellcheck.sh <destination-directory>.\n' \ "$REQUIRED_SHELLCHECK" "$resolved" "$REQUIRED_SHELLCHECK" >&2 exit 1 fi diff --git a/bin/fm-pr-check.sh b/bin/fm-pr-check.sh index 96cb14dc938..dea5e34e7b9 100755 --- a/bin/fm-pr-check.sh +++ b/bin/fm-pr-check.sh @@ -71,6 +71,8 @@ fi # bin/fm-teardown.sh reads the head from the forge at teardown rather than from # metadata and falls back to its provider-agnostic content check, and # bin/fm-review-diff.sh resolves the head from the remote when none is recorded. +# bin/fm-pr-merge.sh reads a GitLab head live at merge time for the same reason, +# and treats a recorded value that disagrees as stale rather than authoritative. WT=$(grep '^worktree=' "$META" | tail -1 | cut -d= -f2- || true) PR_HEAD= if [ "$PROVIDER" = github ] && [ -n "$WT" ] && [ -d "$WT" ] && command -v gh >/dev/null 2>&1; then diff --git a/bin/fm-pr-lib.sh b/bin/fm-pr-lib.sh index b70d8468894..b8ea9eb8fd8 100755 --- a/bin/fm-pr-lib.sh +++ b/bin/fm-pr-lib.sh @@ -163,8 +163,8 @@ fm_pr_gitlab_path_valid() { # # FM_PR_OWNER and FM_PR_REPO are additionally set for github because # bin/fm-pr-merge.sh addresses GitHub by owner/repository. A gitlab URL leaves -# them empty; teaching the merge path about GitLab is a separate change, and -# until then it refuses a GitLab URL rather than merging anything. +# them empty, and that path addresses the project by FM_PR_HOST and FM_PR_PATH +# instead, so a merge request on any instance resolves without a hardcoded host. fm_pr_url_parse() { local raw=${1-} pattern host path local LC_ALL=C diff --git a/bin/fm-pr-merge.sh b/bin/fm-pr-merge.sh index 8226798a673..9afd4e4acfa 100755 --- a/bin/fm-pr-merge.sh +++ b/bin/fm-pr-merge.sh @@ -1,13 +1,33 @@ #!/usr/bin/env bash -# Merge a task's PR after recording pr= and any available pr_head= through +# Merge a task's PR or MR after recording pr= and any available pr_head= through # bin/fm-pr-check.sh, so teardown can verify landed work after squash merges. -# The full canonical GitHub PR URL is parsed by bin/fm-pr-lib.sh and the derived -# owner/repository and PR number are passed to gh-axi as separate arguments. +# The full canonical URL is parsed by bin/fm-pr-lib.sh. A GitHub pull request is +# addressed through gh-axi by the derived owner and repository; a GitLab merge +# request is addressed through glab by the project URL rebuilt from the parsed +# host and path, so any instance works and no host is hardcoded. # -# Merge method defaults to --squash when the caller passes none of --squash, -# --merge, --rebase, or --method after the optional -- separator. Extra args -# must not include --repo or -R because the repository comes only from the URL. -# Usage: fm-pr-merge.sh <task-id> <pr-url> [-- <extra gh-axi pr merge args>] +# Merge method on GitHub defaults to --squash when the caller passes none of +# --squash, --merge, --rebase, or --method after the optional -- separator. +# GitLab adds no method flag at all: its merge method is the project's own +# setting, which the merge API applies, and imposing squash there would override +# that convention rather than mirror the GitHub default. +# +# A GitLab merge is refused unless every pre-merge condition holds, each read +# live at merge time rather than taken from recorded metadata: the merge request +# is open, detailed_merge_status is mergeable, has_conflicts is false, +# blocking_discussions_resolved is true, and the head pipeline succeeded at the +# exact current head commit. Every failing condition is reported, not just the +# first. The verified head is then passed to glab as --sha, so a push that lands +# between that read and the merge fails the merge instead of landing commits +# nothing verified. A recorded pr_head that disagrees with the live head is +# reported rather than trusted, because a rebase moves the head and leaves the +# recorded value stale. Reading that state needs glab and jq, and either one +# absent stops the merge before any state is recorded. +# +# Extra args must not include --repo or -R in any form, including a bundled +# short-option cluster such as -yR, because the repository comes only from the +# URL, nor --sha on GitLab because the head comes only from the live read. +# Usage: fm-pr-merge.sh <task-id> <pr-url> [-- <extra forge merge args>] set -eu SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -24,18 +44,18 @@ if [ "$#" -lt 2 ]; then fi ID=$1 RAW_URL=$2 -# bin/fm-pr-lib.sh parses GitLab merge request URLs so the watcher can follow -# them, but this path still addresses only GitHub by owner/repository. The -# provider check holds that refusal exactly as it was until merge parity lands. -if ! fm_pr_task_id_valid "$ID" || ! fm_pr_url_parse "$RAW_URL" \ - || [ "$FM_PR_PROVIDER" != github ]; then +if ! fm_pr_task_id_valid "$ID" || ! fm_pr_url_parse "$RAW_URL"; then echo "error: invalid PR merge request" >&2 exit 2 fi URL=$FM_PR_URL +PROVIDER=$FM_PR_PROVIDER PR_OWNER=$FM_PR_OWNER PR_REPO=$FM_PR_REPO PR_NUMBER=$FM_PR_NUMBER +# glab resolves the instance from the project URL passed to -R, so the host is +# rebuilt from the parsed identity rather than read from any ambient default. +PROJECT_URL="https://$FM_PR_HOST/$FM_PR_PATH" shift 2 [ "${1:-}" = "--" ] && shift @@ -53,7 +73,14 @@ reject_repo_overrides() { local arg for arg in "$@"; do case "$arg" in - --repo|--repo=*|-R|-R?*) + --repo|--repo=*) + echo "error: extra merge arguments must not override the repository" >&2 + return 1 + ;; + --*) ;; + # A single-dash argument is a short-option cluster, which both CLIs expand + # one character at a time, so -yR carries --repo exactly as a bare -R does. + -*R*) echo "error: extra merge arguments must not override the repository" >&2 return 1 ;; @@ -61,7 +88,20 @@ reject_repo_overrides() { done } +reject_head_overrides() { + local arg + for arg in "$@"; do + case "$arg" in + --sha|--sha=*) + echo "error: extra merge arguments must not override the head commit" >&2 + return 1 + ;; + esac + done +} + reject_repo_overrides "$@" || exit 1 +[ "$PROVIDER" != gitlab ] || reject_head_overrides "$@" || exit 1 # Task-derived paths are constructed only after the canonical ID validation. META="$STATE/$ID.meta" @@ -70,15 +110,154 @@ if [ ! -f "$META" ] || [ -L "$META" ]; then exit 1 fi +# Reading the merge request state needs both tools. Report them together and +# before anything is recorded, so a missing tool is a named prerequisite rather +# than a merge that is armed and then refused for an unexplained reason. +GITLAB_MISSING= +if [ "$PROVIDER" = gitlab ]; then + command -v glab >/dev/null 2>&1 || GITLAB_MISSING="glab" + if ! command -v jq >/dev/null 2>&1; then + GITLAB_MISSING="${GITLAB_MISSING:+$GITLAB_MISSING and }jq" + fi + if [ -n "$GITLAB_MISSING" ]; then + echo "error: merging a GitLab merge request requires $GITLAB_MISSING on PATH" >&2 + exit 1 + fi +fi + +# The recorded head is read before bin/fm-pr-check.sh rewrites the metadata, +# because that script re-records pr= and drops a pr_head= it cannot resolve. +RECORDED_HEAD= +if [ "$PROVIDER" = gitlab ]; then + RECORDED_HEAD=$(grep '^pr_head=' "$META" | tail -1 | cut -d= -f2- || true) +fi + "$SCRIPT_DIR/fm-pr-check.sh" "$ID" "$URL" grep -qxF "pr=$URL" "$META" || { echo "error: PR metadata recording failed" >&2 exit 1 } -merge_args=() -if ! caller_has_merge_method "$@"; then - merge_args=(--squash) -fi +# Pre-merge conditions for a GitLab merge request, read from one live view of +# the merge request. Sets FM_PR_MERGE_HEAD to the verified head on success and +# returns non-zero after reporting every condition that failed. +FM_PR_MERGE_HEAD= +gitlab_verify_mergeable() { + local json fields line + local total=0 named=0 refusals='' + local state='' detail='' conflicts='' discussions='' + local live_head='' pipeline_sha='' pipeline_status='' + + # GITLAB_HOST is set to the same host the project URL already carries, so the + # instance is taken from the parsed URL by both signals and never from the + # operator's configured default. + if ! json=$(GITLAB_HOST="$FM_PR_HOST" glab mr view "$PR_NUMBER" -R "$PROJECT_URL" -F json 2>/dev/null) \ + || [ -z "$json" ]; then + echo "error: could not read the GitLab merge request state before merging" >&2 + return 1 + fi + # One named field per line. The names keep a trailing empty value readable + # after command substitution strips blank lines, and an absent or null field + # becomes an empty string or the literal "null", neither of which satisfies any + # check below, so an unreadable field refuses the merge instead of passing it. + if ! fields=$(printf '%s' "$json" | jq -r ' + if type == "object" then + "state=" + ((.state // "") | tostring), + "detail=" + ((.detailed_merge_status // "") | tostring), + "conflicts=" + (.has_conflicts | tostring), + "discussions=" + (.blocking_discussions_resolved | tostring), + "head=" + ((.sha // "") | tostring), + "pipeline_sha=" + ((.head_pipeline.sha // "") | tostring), + "pipeline_status=" + ((.head_pipeline.status // "") | tostring) + else + error("merge request payload is not an object") + end' 2>/dev/null); then + echo "error: could not read the GitLab merge request state before merging" >&2 + return 1 + fi + while IFS= read -r line; do + total=$((total + 1)) + case "$line" in + state=*) state=${line#state=} ;; + detail=*) detail=${line#detail=} ;; + conflicts=*) conflicts=${line#conflicts=} ;; + discussions=*) discussions=${line#discussions=} ;; + head=*) live_head=${line#head=} ;; + pipeline_sha=*) pipeline_sha=${line#pipeline_sha=} ;; + pipeline_status=*) pipeline_status=${line#pipeline_status=} ;; + *) continue ;; + esac + named=$((named + 1)) + done <<FIELDS +$fields +FIELDS + # Every field named exactly once and no unnamed line: a value carrying a + # newline would split into a line no name matches, so it is refused here + # rather than silently truncated into a value a check could accept. + if [ "$named" -ne 7 ] || [ "$total" -ne 7 ]; then + echo "error: could not read the GitLab merge request state before merging" >&2 + return 1 + fi + + if ! fm_pr_head_valid "$live_head"; then + echo "error: could not read the GitLab merge request head commit before merging" >&2 + return 1 + fi + # A rebase moves the head and leaves the recorded value behind, so the + # disagreement is reported and the live head is what gets verified and merged. + if [ -n "$RECORDED_HEAD" ] && [ "$RECORDED_HEAD" != "$live_head" ]; then + printf 'notice: recorded head %s disagrees with the live head %s; verifying the live head\n' \ + "$RECORDED_HEAD" "$live_head" >&2 + fi + + [ "$state" = opened ] \ + || refusals="$refusals - state is \"${state:-unreadable}\", not open +" + [ "$detail" = mergeable ] \ + || refusals="$refusals - detailed_merge_status is \"${detail:-unreadable}\", not mergeable +" + [ "$conflicts" = false ] \ + || refusals="$refusals - has_conflicts is \"${conflicts:-unreadable}\", not false +" + [ "$discussions" = true ] \ + || refusals="$refusals - blocking_discussions_resolved is \"${discussions:-unreadable}\", not true +" + [ "$pipeline_status" = success ] \ + || refusals="$refusals - the head pipeline status is \"${pipeline_status:-none}\", not success +" + [ "$pipeline_sha" = "$live_head" ] \ + || refusals="$refusals - the head pipeline ran at \"${pipeline_sha:-none}\", not at the current head $live_head +" + + if [ -n "$refusals" ]; then + printf 'error: refusing to merge %s\n' "$URL" >&2 + printf '%s' "$refusals" >&2 + return 1 + fi + printf 'verified: %s is open and mergeable, with a successful pipeline at head %s\n' \ + "$URL" "$live_head" >&2 + FM_PR_MERGE_HEAD=$live_head +} -gh-axi pr merge "$PR_NUMBER" --repo "$PR_OWNER/$PR_REPO" "${merge_args[@]+"${merge_args[@]}"}" "$@" +case "$PROVIDER" in + github) + merge_args=() + if ! caller_has_merge_method "$@"; then + merge_args=(--squash) + fi + gh-axi pr merge "$PR_NUMBER" --repo "$PR_OWNER/$PR_REPO" "${merge_args[@]+"${merge_args[@]}"}" "$@" + ;; + gitlab) + gitlab_verify_mergeable || exit 1 + # --sha binds the merge to the head this run verified, so a push that lands + # in between is refused by GitLab instead of merged unverified. --yes only + # skips the interactive confirmation, which no supervised run can answer; + # the conditions above are what authorize the merge. + GITLAB_HOST="$FM_PR_HOST" glab mr merge "$PR_NUMBER" -R "$PROJECT_URL" \ + --sha "$FM_PR_MERGE_HEAD" --yes "$@" + ;; + *) + echo "error: invalid PR merge request" >&2 + exit 2 + ;; +esac diff --git a/bin/fm-procevent-lavish.sh b/bin/fm-procevent-lavish.sh index fb5eee0686d..2a73281ee6c 100755 --- a/bin/fm-procevent-lavish.sh +++ b/bin/fm-procevent-lavish.sh @@ -8,9 +8,14 @@ # fm-procevent-lavish.sh answers <result-file> # fm-procevent-lavish.sh source-id <artifact.html> # fm-procevent-lavish.sh retire <artifact.html> +# fm-procevent-lavish.sh poll <artifact.html> # # classify Print the lifecycle state a handler should act on: feedback, ended, # waiting, missing, or unknown. +# poll The registered listener command `arm` publishes, not a command to +# run in a conversational turn. It runs the published blocking poll +# and prints its response verbatim, absorbing only the one exact +# transient interruption described below. # terminal Exit 0 when the captured result means this Lavish source will never # produce another result, so the runner may retire it; any other exit # keeps it armed. This is the generic adapter contract bin/fm-procevent.sh @@ -23,11 +28,11 @@ # # `answers` is this adapter's half of the generic keyed-answer contract in # bin/fm-procevent.sh. It reports what the captain actually chose, as -# `<decision-key>\t<answer>\t<label>` lines, and stops there. It maps nothing to a -# hold, records no decision, and closes nothing: a captain answer is not special to -# Lavish, so every rule about what a keyed answer DOES belongs to the one intake in -# bin/fm-decision-hold.sh, which the runner feeds. A Lavish review is just an -# ephemeral discussion format that happens to carry answers. +# `<task-id>\t<answer>\t<label>` lines, and stops there. It maps nothing to a +# task, records no decision, and closes nothing: a captain answer is not special +# to Lavish, so every rule about what a keyed answer DOES belongs to the one +# intake in bin/fm-captain-hold.sh, which the runner feeds. A Lavish review is +# just an ephemeral discussion format that happens to carry answers. # # Only rows tagged `choice` are read. A freeform captain message is prose that may # contain anything, and must never be able to forge a decision key. @@ -39,6 +44,23 @@ # server-side events. It adds no periodic discovery, no timer fallback, and no # dependency on any unreleased capability. # +# BOUNDED QUIET RETRY, owned here and nowhere else. A live listener can be cut +# short by the server with exactly this two-line response while the session's +# marks remain available: +# +# error: Lavish Editor poll response was interrupted +# code: SERVER_ERROR +# +# That is an internal retry, not news, so registering the raw poll made the +# generic runner capture it and wake the whole fleet. `poll` therefore re-runs +# the published poll up to POLL_RETRY_LIMIT times for that exact response, with +# POLL_RETRY_DELAY_DEFAULT seconds between attempts. The match is exact and +# deliberately narrow: real feedback, ended and missing sessions, any other +# SERVER_ERROR, and the same interruption still standing after the bound is +# spent are all printed straight through and captured normally. The retry is a +# Lavish fact, so the generic runner in bin/fm-procevent.sh stays +# adapter-agnostic and learns nothing about it. +# # LOSS LIMITATION, stated plainly. The published poll destructively clears # feedback before returning it. A result lost after that clearing and before the # runner reads the process output is unrecoverable, and no Firstmate wrapper can @@ -59,7 +81,7 @@ FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" . "$SCRIPT_DIR/fm-procevent-lib.sh" die() { printf 'error: %s\n' "$1" >&2; exit 1; } -usage() { sed -n '2,47p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 2; } +usage() { sed -n '2,69p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 2; } # Canonical identity is physical, not the path string: Lavish itself keys a # session on the realpath of the artifact, so two names for one file are one @@ -83,11 +105,16 @@ cmd_arm() { [ -n "$artifact" ] || usage [ "$#" -eq 1 ] || usage command -v lavish-axi >/dev/null 2>&1 || die "lavish-axi is not installed" + poll_retry_delay >/dev/null id=$(cmd_source_id "$artifact") || exit 1 real=$(perl -MCwd=realpath -e '$p = realpath($ARGV[0]); defined($p) or exit 1; print "$p\n"' "$artifact" 2>/dev/null) \ || die "cannot resolve the artifact path: $artifact" - # The plain blocking form: no --timeout-ms, so completion is a server event. - "$SCRIPT_DIR/fm-procevent.sh" register lavish "$id" -- lavish-axi poll "$real" || exit 1 + # This adapter's own listener command, which runs the plain blocking form with + # no --timeout-ms so completion is a server event, and absorbs only the exact + # transient interruption. Registering raw poll output is what let that + # interruption reach the runner as a captured result. + "$SCRIPT_DIR/fm-procevent.sh" register lavish "$id" \ + -- "$SCRIPT_DIR/fm-procevent-lavish.sh" poll "$real" || exit 1 printf 'armed: %s\n' "$id" printf 'artifact: %s\n' "$real" } @@ -99,6 +126,124 @@ cmd_retire() { "$SCRIPT_DIR/fm-procevent.sh" retire "$id" } +# The bounded quiet retry described in the header. The bound is a constant +# because it is a property of the transient response, not an operator choice; +# only the delay takes an override, so a test can exercise the real bound +# without waiting it out. +POLL_RETRY_LIMIT=12 +POLL_RETRY_DELAY_DEFAULT=5 +POLL_RETRY_DELAY_MAX=60 + +# Exit 0 only for the exact two-line interruption, and nothing else. The whole +# response must be those two lines with those exact bytes: whitespace variants, +# a longer response that merely opens with them, and any other SERVER_ERROR are +# genuine errors this adapter must never swallow. +poll_response_filter() { # <response-file> + perl -e ' + use strict; + use warnings; + my ($stage) = @ARGV; + my $expected = "error: Lavish Editor poll response was interrupted\ncode: SERVER_ERROR\n"; + open my $staged, ">", $stage or exit 2; + binmode STDIN; + binmode STDOUT; + binmode $staged; + my ($candidate, $streaming) = ("", 0); + sub write_all { + my ($handle, $bytes) = @_; + my $offset = 0; + while ($offset < length $bytes) { + my $written = syswrite $handle, $bytes, length($bytes) - $offset, $offset; + exit 2 unless defined $written; + $offset += $written; + } + } + while (1) { + my $count = sysread STDIN, my $chunk, 65536; + exit 2 unless defined $count; + last if $count == 0; + if ($streaming) { + write_all(*STDOUT, $chunk); + next; + } + my $room = length($expected) + 1 - length($candidate); + my $take = length($chunk) < $room ? length($chunk) : $room; + my $prefix = substr($chunk, 0, $take); + $candidate .= $prefix; + write_all($staged, $prefix); + my $matches_prefix = length($candidate) <= length($expected) + && substr($expected, 0, length($candidate)) eq $candidate; + if (!$matches_prefix) { + write_all(*STDOUT, $candidate); + write_all(*STDOUT, substr($chunk, $take)); + $streaming = 1; + } + } + exit 10 if !$streaming && $candidate eq $expected; + write_all(*STDOUT, $candidate) unless $streaming; + ' "$1" +} + +# Seconds between retries. FM_LAVISH_POLL_RETRY_DELAY is a bounded test +# override; a malformed or out-of-range value is refused rather than quietly +# rounded, because silently changing a retry cadence is how a bound stops +# meaning anything. +poll_retry_delay() { + local delay=${FM_LAVISH_POLL_RETRY_DELAY-} + if [ -z "$delay" ]; then + printf '%s\n' "$POLL_RETRY_DELAY_DEFAULT" + return 0 + fi + case "$delay" in + *[!0-9]*) die "FM_LAVISH_POLL_RETRY_DELAY must be whole seconds from 0 to $POLL_RETRY_DELAY_MAX: $delay" ;; + esac + [ "$delay" -le "$POLL_RETRY_DELAY_MAX" ] \ + || die "FM_LAVISH_POLL_RETRY_DELAY must be whole seconds from 0 to $POLL_RETRY_DELAY_MAX: $delay" + printf '%s\n' "$delay" +} + +cmd_poll() { + local artifact=${1-} delay attempt=0 response cleanup_command rc filter_rc + local pipeline_status + [ -n "$artifact" ] || usage + [ "$#" -eq 1 ] || usage + command -v lavish-axi >/dev/null 2>&1 || die "lavish-axi is not installed" + delay=$(poll_retry_delay) || exit 1 + response=$(mktemp "${TMPDIR:-/tmp}/fm-lavish-poll.XXXXXX") || die "cannot stage the poll response" + printf -v cleanup_command 'rm -f -- %q' "$response" + # shellcheck disable=SC2064 # $cleanup_command must expand now, while the staged path is still set. + trap "$cleanup_command" EXIT + # Retirement stops this listener by signalling its process group, and bash runs + # no EXIT trap for an uncaught signal, so each one cleans up the staged + # response and then re-raises itself with the default disposition, leaving the + # process dying exactly as the runner expects. + local signal + for signal in INT TERM HUP; do + # shellcheck disable=SC2064 # Same reason: expand now, while both are set. + trap "$cleanup_command; trap - $signal; kill -$signal $$" "$signal" + done + while :; do + lavish-axi poll "$artifact" | poll_response_filter "$response" + pipeline_status=("${PIPESTATUS[@]}") + rc=${pipeline_status[0]} + filter_rc=${pipeline_status[1]} + case "$filter_rc" in + 0) break ;; + 10) + if [ "$attempt" -lt "$POLL_RETRY_LIMIT" ]; then + attempt=$((attempt + 1)) + sleep "$delay" + else + cat -- "$response" + break + fi + ;; + *) die "cannot classify the poll response" ;; + esac + done + return "$rc" +} + # Read one field of the response's leading `session:` block. Those fields are # INDENTED, so each is read as the first indented match inside that block rather # than an anchored whole-line match; anchoring on "^status:" silently never @@ -158,8 +303,9 @@ cmd_terminal() { return 1 } -# Print `key<TAB>answer<TAB>label` for every structured choice the captain -# submitted in a captured result. The published response frames queued feedback as +# Print `key<TAB>answer<TAB>label[<TAB>mode]` for every structured choice the +# captain submitted in a captured result; the optional mode column relays the +# card's declared close mode (`done` or `release`) to the keyed-answer intake. The published response frames queued feedback as # a `prompts[N]{field,...}:` header followed by exactly N indented CSV rows whose # quoted fields carry JSON-style escapes, so this reads the declared field ORDER # rather than assuming a fixed column, and takes only rows whose `tag` field is @@ -167,14 +313,14 @@ cmd_terminal() { # source of decision keys. A row that does not carry both a slug-shaped `question` # and an `answer` inside its `Context data:` block is skipped, so a deck that does # not key its forms by decision key simply yields nothing. -# The question cap is 128 so a FULL hold identity (<origin>-decision-<key>) fits -# for an any-origin bound deck such as the bearings board; the security property -# is the slug SHAPE, which is unchanged. +# The question cap is 128 so any task id fits, including the long legacy +# `<origin>-decision-<key>` identities pre-collapse decks still carry; the +# security property is the slug SHAPE, which is unchanged. cmd_answers() { local file=${1-} [ -n "$file" ] || usage [ -f "$file" ] && [ ! -L "$file" ] || die "result file does not exist: $file" - perl -e ' + perl -MJSON::PP -e ' use strict; use warnings; my ($path) = @ARGV; open my $fh, "<", $path or exit 1; @@ -213,11 +359,17 @@ cmd_answers() { my $prompt = $f{prompt}; next unless defined $prompt && $prompt =~ /Context data:\s*(\{.*\})/s; my $ctx = $1; - next unless $ctx =~ /"question"\s*:\s*"((?:[^"\\]|\\.)*)"/; - my $key = $1; - next unless $ctx =~ /"answer"\s*:\s*"((?:[^"\\]|\\.)*)"/; - my $answer = $1; - $_ =~ s/\\(.)/$1/g for ($key, $answer); + my $data = eval { decode_json($ctx) }; + next unless ref($data) eq "HASH"; + my $key = $data->{question}; + my $answer = $data->{answer}; + next if !defined($key) || ref($key) || !defined($answer) || ref($answer); + my $mode = ""; + if (exists $data->{close}) { + next if !defined($data->{close}) || ref($data->{close}) + || ($data->{close} ne "done" && $data->{close} ne "release"); + $mode = $data->{close}; + } next unless $key =~ /\A[A-Za-z0-9._-]{1,128}\z/; next unless length $answer && length($answer) <= 512; my $label = defined $f{text} ? $f{text} : ""; @@ -226,7 +378,7 @@ cmd_answers() { # A re-answered form appears again later in the queue; the last submission wins. if (defined $seen{$key}) { $out[$seen{$key}] = undef } $seen{$key} = scalar @out; - push @out, "$key\t$answer\t$label"; + push @out, length $mode ? "$key\t$answer\t$label\t$mode" : "$key\t$answer\t$label"; } print "$_\n" for grep { defined } @out; ' "$file" @@ -235,6 +387,7 @@ cmd_answers() { case "${1-}" in arm) shift; cmd_arm "$@" ;; retire) shift; cmd_retire "$@" ;; + poll) shift; cmd_poll "$@" ;; source-id) shift; cmd_source_id "$@" ;; classify) shift; cmd_classify "$@" ;; terminal) shift; cmd_terminal "$@" ;; diff --git a/bin/fm-procevent.sh b/bin/fm-procevent.sh index 1061859e3ce..c26a2402224 100755 --- a/bin/fm-procevent.sh +++ b/bin/fm-procevent.sh @@ -77,10 +77,10 @@ # # Keyed captain answers are adapter-owned through one more seam of the same kind, # and this runner still decides nothing about them. Some sources carry the -# captain's answer to a durable decision. What such an answer MEANS is owned once, -# by bin/fm-decision-hold.sh's keyed-answer intake, and reaching it must not -# depend on an agent remembering. So after capture, a source that has been bound -# to a decision origin has its result passed to +# captain's answer to a captain-held task. What such an answer MEANS is owned +# once, by bin/fm-captain-hold.sh's keyed-answer intake, and reaching it must not +# depend on an agent remembering. So after capture, a bound source +# has its result passed to # `bin/fm-procevent-<adapter>.sh answers <result-file>`, and whatever that prints # is piped straight into that one intake. The adapter reports only what the # captain chose; the intake owns every rule about what happens next. This runner @@ -181,11 +181,11 @@ feed_keyed_answers() { # <adapter> <source-id> <result-file> local adapter=$1 id=$2 result=$3 script origin seq script=$(adapter_script "$adapter") [ -f "$script" ] && [ ! -L "$script" ] || return 1 - origin=$("$SCRIPT_DIR/fm-decision-hold.sh" binding "$id" 2>/dev/null) || return 1 + origin=$("$SCRIPT_DIR/fm-captain-hold.sh" binding "$id" 2>/dev/null) || return 1 [ -n "$origin" ] || return 1 seq=$(fm_procevent_result_sequence "$result") || return 1 "$script" answers "$result" 2>/dev/null \ - | "$SCRIPT_DIR/fm-decision-hold.sh" answers "$origin" \ + | "$SCRIPT_DIR/fm-captain-hold.sh" answers "$origin" \ --source "the captured result $id sequence $seq" >/dev/null 2>&1 } @@ -710,7 +710,7 @@ cmd_retire() { # A retired source produces no further answer, so drop any decision binding it # carried. Generic and idempotent: the binding owner is asked to forget this # source id, and an unbound source is unaffected. - "$SCRIPT_DIR/fm-decision-hold.sh" unbind "$id" >/dev/null 2>&1 || true + "$SCRIPT_DIR/fm-captain-hold.sh" unbind "$id" >/dev/null 2>&1 || true printf 'retired: %s\n' "$id" } diff --git a/bin/fm-project-mode.sh b/bin/fm-project-mode.sh index 6a97ce2dfed..3046202f23f 100755 --- a/bin/fm-project-mode.sh +++ b/bin/fm-project-mode.sh @@ -26,9 +26,8 @@ # Mechanical output maps it to its most rigorous leg, # no-mistakes, so sync, seeding, and init treat such a # project as the remote-backed pipeline project it is. -# yolo (orthogonal) = when on, firstmate may make routine approval decisions itself. -# AGENTS.md section 7 is the single owner of authority exceptions, including -# ask-user contract expansion and stronger captain boundaries. +# yolo (orthogonal) = merge authority only: when on, firstmate merges green, +# in-scope work itself (AGENTS.md section 7). # # --raw prints the registered annotation unmapped, so a caller that must tell a # conditional policy apart from a flat mode sees "no-mistakes-prod-only" itself. diff --git a/bin/fm-promote.sh b/bin/fm-promote.sh index 0ed1fd06161..51d74eca45b 100755 --- a/bin/fm-promote.sh +++ b/bin/fm-promote.sh @@ -24,6 +24,12 @@ STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" . "$SCRIPT_DIR/fm-pr-lib.sh" # shellcheck source=bin/fm-wake-lib.sh . "$SCRIPT_DIR/fm-wake-lib.sh" +# shellcheck source=bin/fm-public-followup-lib.sh +. "$SCRIPT_DIR/fm-public-followup-lib.sh" +# shellcheck source=bin/fm-secondmate-parent-lib.sh +. "$SCRIPT_DIR/fm-secondmate-parent-lib.sh" +# shellcheck source=bin/fm-secondmate-registry-lib.sh +. "$SCRIPT_DIR/fm-secondmate-registry-lib.sh" MODE= YOLO= @@ -58,7 +64,7 @@ done exit 1 } [ "$YOLO_SET" -eq 1 ] || { - echo "error: promotion requires --yolo <on|off>; it is this task's routine approval authority, not a project lookup" >&2 + echo "error: promotion requires --yolo <on|off>; it is this task's merge authority, not a project lookup" >&2 exit 1 } case "$MODE" in @@ -123,3 +129,105 @@ META_LOCK_HELD=0 HOME_Q=$(printf '%q' "$FM_HOME") echo "promoted $ID to ship mode=$MODE yolo=$YOLO (teardown protection restored)" echo "next: FM_HOME=$HOME_Q bin/fm-send.sh fm-$ID '<ship instructions for mode=$MODE: review scratch state with git status and git log; reset to a clean default-branch base; carry over only intended fix changes; create branch fm/$ID; implement; report done>'" + +promote_print_rechain_hint() { + local consent_home=$1 work_home=$2 task_id=$3 id prefix + prefix= + [ "$consent_home" = "$FM_HOME" ] || prefix="FM_HOME=$(printf '%q' "$consent_home") " + while IFS= read -r id; do + [ -n "$id" ] || continue + [ "$(fm_pf_registry_get "$consent_home/state" "$id" state)" = delivered ] || continue + echo "next: ${prefix}bin/fm-public-followup.sh rechain <new-obligation-id> --from $id --work-home $work_home --work-id $task_id --expected pr-merged" + done <<EOF +$(fm_pf_registry_ids_for_work "$consent_home/state" "$work_home" "$task_id") +EOF +} + +promote_canonical_home() { + local home=$1 + case "$home" in /*) ;; *) return 1 ;; esac + CDPATH='' cd -- "$home" 2>/dev/null && pwd -P +} + +promote_resolve_primary_home() { + local parent=$1 child=$2 mate_id=$3 parent_meta registry meta_home + fm_pf_home_id_valid "secondmate:$mate_id" || return 1 + parent=$(promote_canonical_home "$parent") || return 1 + child=$(promote_canonical_home "$child") || return 1 + [ "$parent" != "$child" ] || return 1 + parent_meta="$parent/state/$mate_id.meta" + [ -f "$parent_meta" ] && [ ! -L "$parent_meta" ] || return 1 + [ "$(fmx_meta_get "$parent_meta" kind)" = secondmate ] || return 1 + meta_home=$(fmx_meta_get "$parent_meta" home) + meta_home=$(CDPATH='' cd -- "$meta_home" 2>/dev/null && pwd -P) || return 1 + [ "$meta_home" = "$child" ] || return 1 + registry="$parent/data/secondmates.md" + secondmate_registry_validate_bindings "$registry" secondmate_registry_path_key \ + "$mate_id" "$child" || return 1 + printf '%s\n' "$parent" +} + +promote_warn_parent_unresolved() { + echo "warning: could not resolve the consent-holding parent home for secondmate $1; promotion succeeded, but any open public loop must be inspected and rechained from the parent." >&2 +} + +if [ -f "$FM_HOME/.fm-secondmate-home" ]; then + PROMOTE_MATE_ID=$(sed -n '1p' "$FM_HOME/.fm-secondmate-home" 2>/dev/null || true) + PROMOTE_PARENT_RECORD=absent + PROMOTE_PARENT_ROUTE= + PROMOTE_DURABLE_PARENT= + if [ -e "$FM_HOME/.fm-secondmate-parent" ] || [ -L "$FM_HOME/.fm-secondmate-parent" ]; then + PROMOTE_PARENT_RECORD=invalid + if fm_secondmate_parent_record_parse "$FM_HOME/.fm-secondmate-parent"; then + PROMOTE_PARENT_RECORD=valid + PROMOTE_PARENT_ROUTE=$FM_SECONDMATE_PARENT_ROUTE + PROMOTE_DURABLE_PARENT=$FM_SECONDMATE_PARENT_HOME + fi + fi + if [ "$PROMOTE_PARENT_RECORD" = invalid ]; then + promote_warn_parent_unresolved "$PROMOTE_MATE_ID" + elif [ "$PROMOTE_PARENT_ROUTE" = local ]; then + PROMOTE_PARENT_CANDIDATE=${FM_PUBLIC_FOLLOWUP_PRIMARY_HOME:-$PROMOTE_DURABLE_PARENT} + PROMOTE_PARENT_BINDINGS_MATCH=1 + if [ -n "${FM_PUBLIC_FOLLOWUP_PRIMARY_HOME:-}" ]; then + PROMOTE_LIVE_PARENT=$(promote_canonical_home "$FM_PUBLIC_FOLLOWUP_PRIMARY_HOME") \ + || PROMOTE_PARENT_BINDINGS_MATCH=0 + PROMOTE_RECORDED_PARENT=$(promote_canonical_home "$PROMOTE_DURABLE_PARENT") \ + || PROMOTE_PARENT_BINDINGS_MATCH=0 + if [ "$PROMOTE_PARENT_BINDINGS_MATCH" = 1 ] \ + && [ "$PROMOTE_LIVE_PARENT" != "$PROMOTE_RECORDED_PARENT" ]; then + PROMOTE_PARENT_BINDINGS_MATCH=0 + fi + fi + if [ "$PROMOTE_PARENT_BINDINGS_MATCH" = 1 ] \ + && PROMOTE_PARENT=$(promote_resolve_primary_home \ + "$PROMOTE_PARENT_CANDIDATE" "$FM_HOME" "$PROMOTE_MATE_ID"); then + if fm_pf_relay_active "$PROMOTE_PARENT"; then + promote_print_rechain_hint "$PROMOTE_PARENT" "secondmate:$PROMOTE_MATE_ID" "$ID" + fi + else + promote_warn_parent_unresolved "$PROMOTE_MATE_ID" + fi + elif [ "$PROMOTE_PARENT_ROUTE" = remote ]; then + PROMOTE_HOME_ENV_TOKEN= + if [ -f "$FM_HOME/.env" ]; then + PROMOTE_HOME_ENV_TOKEN=$(fmx_env_get FMX_PAIRING_TOKEN "$FM_HOME/.env") + fi + if [ -n "$PROMOTE_HOME_ENV_TOKEN" ]; then + promote_warn_parent_unresolved "$PROMOTE_MATE_ID" + fi + elif [ -n "${FM_PUBLIC_FOLLOWUP_PRIMARY_HOME:-}" ]; then + if fm_pf_relay_active "$FM_PUBLIC_FOLLOWUP_PRIMARY_HOME"; then + if PROMOTE_PARENT=$(promote_resolve_primary_home \ + "$FM_PUBLIC_FOLLOWUP_PRIMARY_HOME" "$FM_HOME" "$PROMOTE_MATE_ID"); then + promote_print_rechain_hint "$PROMOTE_PARENT" "secondmate:$PROMOTE_MATE_ID" "$ID" + else + promote_warn_parent_unresolved "$PROMOTE_MATE_ID" + fi + fi + elif fm_pf_relay_active "$FM_HOME"; then + promote_warn_parent_unresolved "$PROMOTE_MATE_ID" + fi +elif fm_pf_relay_active "$FM_HOME"; then + promote_print_rechain_hint "$FM_HOME" main "$ID" +fi diff --git a/bin/fm-public-followup-lib.sh b/bin/fm-public-followup-lib.sh index dc7153d53cf..20ebd372d7e 100644 --- a/bin/fm-public-followup-lib.sh +++ b/bin/fm-public-followup-lib.sh @@ -5,9 +5,10 @@ # Firstmate promises a public final reply when a myfirstmate relay mention (X or # Discord) asks for work. `tasks-axi public-followup` is the sole owner of that # typed obligation and its state machine; state/x-context/ is the sole owner of -# the private full request context. This library owns only the small Firstmate -# side: the activation gate, the private per-home transport directories, and the -# deterministic terminal-event identity. +# the private full request context. This library owns Firstmate's activation +# gate, private per-home transport paths, retained-loop state and locking +# helpers, follow-up window classification, and deterministic terminal-event +# identity. # # Sourced, never executed. No side effects on source (it creates nothing), which # is what keeps a relay-disabled home free of public-followup artifacts. @@ -21,17 +22,26 @@ # [ -f ] test and nothing else runs. # 2. fm_pf_has_registrations O(1) presence check on the registry created # / fm_pf_has_events only by the relay path (fm-public-followup.sh -# register). Relay-enabled homes with no -# public commitments stop here, so no -# tasks-axi call and no backlog scan happens. +# / fm_pf_has_open_loops register). Open loops ARE registrations: +# a delivered final keeps the record, so this +# same check is the fail-loud session-start +# gate. Relay-enabled homes with no public +# loops stop here, so no tasks-axi call and +# no backlog scan happens. # # Private transport layout, all under <home>/state/public-followup (mode 0700, -# created only by `fm-public-followup.sh register`): -# registry/<obligation-id> registration record: the bounded public-safe -# binding (obligation, relation, work ref, -# generation, platform, request id). Presence hint -# and reverse work->obligation index only; the -# obligation itself always remains tasks-axi truth. +# initialized by `fm-public-followup.sh register` and extended only by these +# public-followup commands): +# registry/<obligation-id> registration record: the bounded private binding +# (obligation, relation, work ref and canonical +# secondmate path, generation, platform, request id) +# plus the loop fields that survive delivery (state, +# delivered_at, followup_expires_at, +# request_context_b64). Presence means the public +# loop is still open. Delivery +# stamps state=delivered; only `retire` removes the +# record. The obligation itself always remains +# tasks-axi truth. # events/<event-id>.json inbound typed terminal events awaiting # reconciliation, one file per event id. # consumed/<event-id> idempotency ledger: an accepted event id is never @@ -43,6 +53,10 @@ # surfaced last surfaced pending-event signature, so the # existing relay poll wakes once per new event set # instead of every cycle. +# retired/<obligation-id> private retirement receipt containing the bounded +# reason and timestamp recorded before the registry +# entry is removed; its presence prevents replayed +# registration from reopening the closed loop. # # Event identity is DERIVED, never random: fm_pf_event_id hashes the canonical # identity tuple, so re-emitting the same terminal result produces the same @@ -91,6 +105,13 @@ fm_pf_registry_dir() { printf '%s\n' "$1/$FM_PF_DIRNAME/registry"; } fm_pf_events_dir() { printf '%s\n' "$1/$FM_PF_DIRNAME/events"; } fm_pf_consumed_dir() { printf '%s\n' "$1/$FM_PF_DIRNAME/consumed"; } fm_pf_rejected_dir() { printf '%s\n' "$1/$FM_PF_DIRNAME/rejected"; } +fm_pf_retired_dir() { printf '%s\n' "$1/$FM_PF_DIRNAME/retired"; } + +fm_pf_retirement_receipt_exists() { + local file + file="$(fm_pf_retired_dir "$1")/$2" + [ -f "$file" ] && [ ! -L "$file" ] +} # fm_pf_dir_has_entry <dir>: 0 when <dir> is a real directory holding at least # one non-dot entry. Stops at the first hit, so cost does not grow with the @@ -107,6 +128,11 @@ fm_pf_dir_has_entry() { fm_pf_has_registrations() { fm_pf_dir_has_entry "$(fm_pf_registry_dir "$1")"; } fm_pf_has_events() { fm_pf_dir_has_entry "$(fm_pf_events_dir "$1")"; } +# Every retained registration is an open public loop (owed or delivered). Same +# O(1) directory presence check as fm_pf_has_registrations; the name is the +# post-retention semantic so callers do not treat "a reply is owed" as the +# only reason a record exists. +fm_pf_has_open_loops() { fm_pf_has_registrations "$1"; } # fm_pf_active <home> <state>: both gates, in order. The single predicate every # caller outside the relay path should use before doing any public-followup work. @@ -224,6 +250,131 @@ $(fm_pf_registry_ids "$state") EOF } +# fm_pf_now_epoch: wall clock as epoch seconds. FMX_NOW_OVERRIDE pins it for +# tests, matching bin/fm-x-lib.sh. +fm_pf_now_epoch() { + printf '%s\n' "${FMX_NOW_OVERRIDE:-$(date +%s)}" +} + +# fm_pf_now_rfc3339: UTC timestamp for delivered_at and similar stamps. +fm_pf_now_rfc3339() { + local epoch + epoch=$(fm_pf_now_epoch) + date -u -r "$epoch" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \ + || date -u -d "@$epoch" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \ + || date -u +%Y-%m-%dT%H:%M:%SZ +} + +# fm_pf_rfc3339_to_epoch <rfc3339>: parse a Zulu timestamp. Empty on failure. +fm_pf_rfc3339_to_epoch() { + local ts=$1 + [ -n "$ts" ] || return 1 + date -u -j -f '%Y-%m-%dT%H:%M:%SZ' "$ts" +%s 2>/dev/null \ + || date -u -d "$ts" +%s 2>/dev/null \ + || return 1 +} + +# fm_pf_followup_window_class <rfc3339>: ok, closing (<48h), expired, or unknown. +fm_pf_followup_window_class() { + local ts=$1 exp now + exp=$(fm_pf_rfc3339_to_epoch "$ts") || { printf 'unknown\n'; return 0; } + now=$(fm_pf_now_epoch) + if [ "$now" -ge "$exp" ]; then + printf 'expired\n' + elif [ $((exp - now)) -lt 172800 ]; then + printf 'closing\n' + else + printf 'ok\n' + fi +} + +# fm_pf_b64_encode: stdin to a single-line base64 payload (no wrapping). +fm_pf_b64_encode() { + base64 2>/dev/null | tr -d '\n\r' +} + +# fm_pf_b64_decode: stdin (single-line or wrapped base64) to bytes on stdout. +fm_pf_b64_decode() { + local data + data=$(cat) + printf '%s\n' "$data" | base64 -d 2>/dev/null \ + || printf '%s\n' "$data" | base64 -D 2>/dev/null +} + +# fm_pf_registry_loop_state <state> <id>: open or delivered. A pre-change +# record with no state= is treated as open so live homes never crash. +fm_pf_registry_loop_state() { + local v + v=$(fm_pf_registry_get "$1" "$2" state) + case "$v" in + delivered) printf 'delivered\n' ;; + *) printf 'open\n' ;; + esac +} + +# fm_pf_registry_rechainable <state> <id>: 0 when request_context_b64 is present. +fm_pf_registry_rechainable() { + local ctx + ctx=$(fm_pf_registry_get "$1" "$2" request_context_b64) + [ -n "$ctx" ] +} + +# fm_pf_has_delivered_open_loops <state>: 0 when any retained record is +# state=delivered (an open loop with nothing owed). Pre-change records have no +# state= and are treated as still-owed, not delivered. +fm_pf_has_delivered_open_loops() { + local state=$1 id + while IFS= read -r id; do + [ -n "$id" ] || continue + [ "$(fm_pf_registry_get "$state" "$id" state)" = delivered ] || continue + return 0 + done <<EOF +$(fm_pf_registry_ids "$state") +EOF + return 1 +} + +fm_pf_registry_lock_path() { + printf '%s/.registry-%s.lock\n' "$(fm_pf_root "$1")" "$2" +} + +fm_pf_registry_lock_acquire() { + local state=$1 id=$2 + fm_pf_slug_valid "$id" || return 1 + fmx_private_artifact_dir_prepare "$(fm_pf_root "$state")" >/dev/null || return 1 + if ! command -v fm_lock_acquire_wait >/dev/null 2>&1; then + # shellcheck source=bin/fm-wake-lib.sh + . "$_FM_PF_LIB_DIR/fm-wake-lib.sh" + fi + fm_lock_acquire_wait "$(fm_pf_registry_lock_path "$state" "$id")" +} + +fm_pf_registry_lock_release() { + fm_lock_release "$(fm_pf_registry_lock_path "$1" "$2")" +} + +# fm_pf_registry_stamp_delivered <state> <id> <rfc3339>: rewrite one record +# with state=delivered and delivered_at, keeping every other field. The record +# stays; only retire removes it. +fm_pf_registry_stamp_delivered() { + local state=$1 id=$2 delivered_at=$3 file rest rc=0 + fm_pf_slug_valid "$id" || return 1 + [ -n "$delivered_at" ] || return 1 + fm_pf_registry_lock_acquire "$state" "$id" || return 1 + file="$(fm_pf_registry_dir "$state")/$id" + if [ -f "$file" ] && [ ! -L "$file" ]; then + rest=$(grep -v -E '^(state|delivered_at|delivered_obligation)=' "$file" 2>/dev/null || true) + printf '%s\nstate=delivered\ndelivered_at=%s\ndelivered_obligation=%s\n' \ + "$rest" "$delivered_at" "$id" \ + | fmx_private_artifact_publish_stdin "$(fm_pf_registry_dir "$state")" "$id" 600 \ + || rc=$? + else + rc=3 + fi + fm_pf_registry_lock_release "$state" "$id" + return "$rc" +} + # --- pending-event signature ------------------------------------------------ # Consumed by the sourcing scripts, not by this library. diff --git a/bin/fm-public-followup.sh b/bin/fm-public-followup.sh index aa754d9e646..dda567cca17 100755 --- a/bin/fm-public-followup.sh +++ b/bin/fm-public-followup.sh @@ -27,7 +27,8 @@ # Usage: # fm-public-followup.sh active # Silent gate probe. Exit 0 when this home has live public-followup work -# worth looking at, 1 otherwise. Safe to call unconditionally. +# worth looking at, including a delivered open loop, 1 otherwise. Safe to +# call unconditionally. # # fm-public-followup.sh register <obligation-id> --relation <relation-id> # --work-home <main|secondmate:<id>> --work-id <task-id> --generation <n> @@ -42,7 +43,8 @@ # fm-public-followup.sh brief <obligation-id> # Print the exact fm-public-followup-emit.sh command line the bound worker # must run when its work reaches the promised terminal outcome, so the -# binding is copied into a brief instead of hand-assembled. +# binding is copied into a brief instead of hand-assembled. The +# --deliverable flags name the obligation's actual required keys. # # fm-public-followup.sh consume # Drain every pending typed terminal event: validate its derived identity, @@ -54,43 +56,60 @@ # replay are no-ops. # # fm-public-followup.sh pending -# One bounded public-safe line per unresolved commitment, for the session -# start digest. Prunes registrations whose obligation is already closed. -# Silent when nothing is unresolved. +# One bounded public-safe line per open public loop, for the session +# start digest. Unresolved commitments print as "unresolved" (a reply is +# still owed). Delivered or settled registrations print as "open-loop" +# (the thread is still open with nothing owed). Registrations are never +# pruned here; only `retire` removes one. Silent when nothing is open. # # fm-public-followup.sh deliver <obligation-id> [--text-file <path>] -# Post the final public reply into the ORIGINAL thread and close the -# obligation. Uses the stored platform and opaque context binding, so the -# destination is never guessed. Without --text-file the accepted terminal -# event's bounded public-safe outcome is reused exactly, which keeps the -# common path deterministic. The sequence is begin-delivery with the -# payload hash, post, then record the posted receipt or a typed error. -# A validated receipt also clears any bound legacy X link before the -# registration is removed. -# An already-posted obligation is an idempotent success without another -# post; an obligation left in delivery-posting by a crash is REFUSED -# rather than posted again. +# Post the final public reply into the ORIGINAL thread. Uses the stored +# platform and opaque context binding, so the destination is never guessed. +# Without --text-file the accepted terminal event's bounded public-safe +# outcome is reused exactly, which keeps the common path deterministic. +# The sequence is begin-delivery with the payload hash, post, then record +# the posted receipt or a typed error. A validated receipt also clears any +# bound legacy X link, then stamps the registration state=delivered. Delivery +# does not close the public loop; `retire` is the only close. Prints a +# disposition line so the loop is handed on with `rechain` or closed +# explicitly. An already-posted obligation is an idempotent success +# without another post; an obligation left in delivery-posting by a crash +# is REFUSED rather than posted again. # # fm-public-followup.sh record-posted <obligation-id> --attempt <n> --chunks <n> -# Close an obligation whose post is known to have landed on exactly +# Record an obligation whose post is known to have landed on exactly # attempt <n> with exactly <n> messages, without posting anything. This is # the late-receipt path: use it when a post succeeded but its receipt was -# lost, never to paper over an unknown outcome. +# lost, never to paper over an unknown outcome. Stamps the registration +# delivered; does not remove it. # # fm-public-followup.sh guard-work <work-home-id> <work-id> # Exit 3 when this home has an unresolved public commitment bound to that # exact work, printing one line per blocking obligation. Exit 0 otherwise. # Cleanup paths call this so bound work is never treated as finished while -# its public promise is still open. +# its public promise is still open. A delivered registration is not a +# block: that work's reply already landed. # -# fm-public-followup.sh retire <obligation-id> [--force] -# Drop the registration once its obligation is closed. --force is the -# explicit discard-approved escape hatch for an unresolved or missing -# obligation. +# fm-public-followup.sh rechain <new-obligation-id> --from <delivered-id> +# --work-home <main|secondmate:<id>> --work-id <task-id> +# --expected <pr-merged|report-ready|local-main> +# [--deliverable-key <k>]... +# Hand a delivered public loop on to follow-on work against the same +# thread. Decodes the retained request context, creates and binds a fresh +# promised-final obligation, registers it, retires the source with reason +# "handed on to <new-id>", and prints `brief` for the new obligation. +# Refuses unless the source is state=delivered, the follow-up window is +# still open, and the relay is active. A pre-change record without +# request_context_b64 is un-rechainable. # -# Requires jq and a compatible tasks-axi for registration, reconciliation, -# delivery, cleanup guards, and retirement; `active` and `brief` only inspect -# local state. +# fm-public-followup.sh retire <obligation-id> --reason "<why the loop is done>" [--force] +# The only close. Drops the registration after recording --reason. +# --force is the explicit discard-approved escape hatch for an unresolved +# or missing obligation. --reason is required. +# +# Requires jq and a compatible tasks-axi for registration, briefs, +# reconciliation, delivery, cleanup guards, and retirement; only `active` +# inspects local state alone. # FM_PF_RETRY_BACKOFF_SECS (default 900) sets the next-attempt time recorded with # a retryable delivery error. set -u @@ -110,7 +129,7 @@ RETRY_BACKOFF=${FM_PF_RETRY_BACKOFF_SECS:-900} case "$RETRY_BACKOFF" in ''|*[!0-9]*) RETRY_BACKOFF=900 ;; esac usage() { - echo "usage: fm-public-followup.sh <active|register|brief|consume|pending|deliver|record-posted|guard-work|retire> [args]" >&2 + echo "usage: fm-public-followup.sh <active|register|brief|consume|pending|deliver|record-posted|guard-work|rechain|retire> [args]" >&2 } # The header comment IS the help text, so the two can never drift apart. @@ -119,12 +138,40 @@ help() { sed -n '2,/^set -u$/p' "$0" | sed '$d; s/^# \{0,1\}//'; } die() { printf 'fm-public-followup: %s\n' "$1" >&2; exit "${2:-2}"; } PF_TEMP_FILES=() -pf_cleanup_temp_files() { +PF_REGISTRY_LOCK_IDS=() +pf_registry_lock_held() { + local wanted=$1 held + for held in "${PF_REGISTRY_LOCK_IDS[@]}"; do + [ "$held" = "$wanted" ] && return 0 + done + return 1 +} +pf_registry_lock_acquire() { + local id=$1 + pf_registry_lock_held "$id" && return 0 + fm_pf_registry_lock_acquire "$STATE" "$id" || return 1 + PF_REGISTRY_LOCK_IDS+=("$id") +} +pf_registry_lock_release() { + local id=$1 held + local -a remaining=() + pf_registry_lock_held "$id" || return 0 + fm_pf_registry_lock_release "$STATE" "$id" + for held in "${PF_REGISTRY_LOCK_IDS[@]}"; do + [ "$held" = "$id" ] || remaining+=("$held") + done + PF_REGISTRY_LOCK_IDS=("${remaining[@]}") +} +pf_cleanup() { + local i + for ((i=${#PF_REGISTRY_LOCK_IDS[@]}-1; i>=0; i--)); do + fm_pf_registry_lock_release "$STATE" "${PF_REGISTRY_LOCK_IDS[$i]}" 2>/dev/null || true + done [ "${#PF_TEMP_FILES[@]}" -eq 0 ] || rm -f -- "${PF_TEMP_FILES[@]}" } -trap pf_cleanup_temp_files EXIT +trap pf_cleanup EXIT -now_rfc3339() { date -u +%Y-%m-%dT%H:%M:%SZ; } +now_rfc3339() { fm_pf_now_rfc3339; } # next_attempt_rfc3339: the retry time recorded with a retryable delivery error. # BSD and GNU date disagree on the flag, so try both and print nothing when @@ -143,7 +190,7 @@ require_tools() { } # Every tasks-axi call runs from the home whose backlog owns the obligation, the -# same convention bin/fm-decision-hold.sh uses for typed backlog state. +# same convention bin/fm-captain-hold.sh uses for typed backlog state. tx() { (cd "$FM_HOME" && tasks-axi "$@"); } # obligation_json <id>: the complete typed obligation payload on stdout, empty @@ -233,17 +280,48 @@ cmd_register() { [ -n "$request" ] || request=$(pf_field "$payload" '.public_followup.request.request_id') [ -z "$request" ] || fm_pf_slug_valid "$request" || die "unsafe request id: $request" - local mkdir_target + local followup_expires_at request_json request_context_b64 work_home_path + followup_expires_at=$(pf_field "$payload" '.public_followup.request.followup_expires_at') + request_json=$(printf '%s' "$payload" | jq -c '.public_followup.request // empty' 2>/dev/null || true) + request_context_b64= + if [ -n "$request_json" ]; then + request_context_b64=$(printf '%s' "$request_json" | fm_pf_b64_encode) + fi + work_home_path= + case "$work_home" in + secondmate:*) + work_home_path=$(public_followup_secondmate_home "${work_home#secondmate:}" 2>/dev/null || true) + case "$work_home_path" in + *$'\n'*|*$'\r'*) work_home_path= ;; + esac + ;; + esac + + local mkdir_target registry_state retired_file for mkdir_target in "$(fm_pf_registry_dir "$STATE")" "$(fm_pf_events_dir "$STATE")" \ "$(fm_pf_consumed_dir "$STATE")" "$(fm_pf_rejected_dir "$STATE")"; do fmx_private_artifact_dir_prepare "$mkdir_target" >/dev/null \ || die "could not prepare $mkdir_target" 1 done - printf 'obligation_id=%s\nrelation_id=%s\nwork_home=%s\nwork_id=%s\ngeneration=%s\nplatform=%s\nrequest_id=%s\n' \ - "$id" "$relation" "$work_home" "$work_id" "$generation" "$platform" "$request" \ + pf_registry_lock_acquire "$id" \ + || die "could not lock registration '$id'" 1 + retired_file="$(fm_pf_retired_dir "$STATE")/$id" + if [ -e "$retired_file" ] || [ -L "$retired_file" ]; then + die "public loop '$id' has already been retired and cannot be registered again" 1 + fi + registry_state=$(fm_pf_registry_loop_state "$STATE" "$id") + if [ "$registry_state" = delivered ]; then + pf_registry_lock_release "$id" + printf 'already registered %s state=delivered\n' "$id" + return 0 + fi + printf 'obligation_id=%s\nrelation_id=%s\nwork_home=%s\nwork_home_path=%s\nwork_id=%s\ngeneration=%s\nplatform=%s\nrequest_id=%s\nstate=open\nfollowup_expires_at=%s\nrequest_context_b64=%s\n' \ + "$id" "$relation" "$work_home" "$work_home_path" "$work_id" "$generation" "$platform" "$request" \ + "$followup_expires_at" "$request_context_b64" \ | fmx_private_artifact_publish_stdin "$(fm_pf_registry_dir "$STATE")" "$id" 600 \ || die "could not write the registration record" 1 + pf_registry_lock_release "$id" printf 'registered %s %s/%s generation=%s platform=%s\n' \ "$id" "$work_home" "$work_id" "$generation" "${platform:-unknown}" @@ -252,7 +330,7 @@ cmd_register() { # --- subcommand: brief ------------------------------------------------------ cmd_brief() { - local id=${1:-} relation work_home work_id generation + local id=${1:-} relation work_home work_id generation payload outcome keys key deliverable_flags [ -n "$id" ] || { usage; exit 2; } fm_pf_slug_valid "$id" || die "unsafe obligation id: $id" fm_pf_relay_active "$FM_HOME" || die "the relay is not active for this home" 1 @@ -264,6 +342,29 @@ cmd_brief() { work_id=$(fm_pf_registry_get "$STATE" "$id" work_id) generation=$(fm_pf_registry_get "$STATE" "$id" generation) + require_tools + payload=$(obligation_json "$id") \ + || die "could not read public-followup obligation '$id' through tasks-axi" 1 + [ -n "$payload" ] \ + || die "public-followup obligation '$id' is missing from tasks-axi" 1 + outcome=$(pf_field "$payload" '.public_followup.expected_final.type') + [ -n "$outcome" ] \ + || die "public-followup obligation '$id' has no expected final type" 1 + keys=$(printf '%s' "$payload" \ + | jq -er '.public_followup.expected_final.required_deliverables + | select(type == "array" and length > 0 + and (map(type == "string" and test("^[a-z0-9_]+$")) | all)) + | .[]' 2>/dev/null) \ + || die "public-followup obligation '$id' has no readable required deliverable keys" 1 + deliverable_flags= + while IFS= read -r key; do + [ -n "$key" ] || continue + deliverable_flags="${deliverable_flags} --deliverable ${key}=<value> \\ +" + done <<EOF +$keys +EOF + cat <<EOF When this work reaches its promised terminal outcome, report it as typed data (never as a sentence for someone to parse) by running exactly: @@ -275,9 +376,8 @@ When this work reaches its promised terminal outcome, report it as typed data --source-home $work_home \\ --work-id $work_id \\ --generation $generation \\ - --outcome <pr-merged|report-ready|local-main|failed> \\ - --deliverable <key>=<value> \\ - --outcome-text '<one bounded public-safe sentence>' + --outcome $outcome \\ +${deliverable_flags} --outcome-text '<one bounded public-safe sentence>' Do not post anything publicly yourself and do not look for the public thread: the home above owns the reply. @@ -426,10 +526,52 @@ cmd_consume() { # --- subcommand: pending ---------------------------------------------------- +# print_open_loop <id> <payload>: the session-start line for a public loop that +# is still open after delivery (or whose obligation has left the backlog). +print_window_escalation() { + local expires=$1 window + window=$(fm_pf_followup_window_class "$expires") + case "$window" in + expired) + printf ' DEADLINE: thread can no longer be reached (window closed %s); this needs a captain decision\n' \ + "${expires:-unknown}" + ;; + closing) + printf ' DEADLINE: window closes %s (under 48 hours)\n' "${expires:-unknown}" + ;; + esac +} + +print_open_loop() { + local id=$1 payload=$2 request platform summary delivered expires ctx + request=$(fm_pf_registry_get "$STATE" "$id" request_id) + [ -n "$request" ] || request=$(pf_field "$payload" '.public_followup.request.request_id') + platform=$(fm_pf_registry_get "$STATE" "$id" platform) + [ -n "$platform" ] || platform=$(pf_field "$payload" '.public_followup.request.platform') + delivered=$(fm_pf_registry_get "$STATE" "$id" delivered_at) + expires=$(fm_pf_registry_get "$STATE" "$id" followup_expires_at) + [ -n "$expires" ] || expires=$(pf_field "$payload" '.public_followup.request.followup_expires_at') + summary=$(pf_field "$payload" '.public_followup.request.public_safe_summary' | fm_pf_clean_outcome_text) + if [ -z "$summary" ]; then + ctx=$(fm_pf_registry_get "$STATE" "$id" request_context_b64) + if [ -n "$ctx" ]; then + summary=$(printf '%s' "$ctx" | fm_pf_b64_decode | jq -r '.public_safe_summary // empty' 2>/dev/null | fm_pf_clean_outcome_text) + fi + fi + printf 'open-loop %s request=%s platform=%s\n' "$id" "${request:-unknown}" "${platform:-unknown}" + printf ' delivered=%s window-closes=%s\n' "${delivered:-unknown}" "${expires:-unknown}" + printf ' summary=%s\n' "$summary" + if ! fm_pf_registry_rechainable "$STATE" "$id"; then + printf ' unrechainable: pre-change registration lacks request_context_b64\n' + fi + print_window_escalation "$expires" + printf ' -> bind the follow-on with rechain, or close the loop with retire %s --reason ...\n' "$id" +} + cmd_pending() { gate_or_exit - local listing id payload delivery task_state summary platform request printed=0 + local listing id payload delivery task_state summary platform request expires printed=0 loop_state settled stamp_rc # An unreadable backlog with registrations present is exactly the silence this # whole path exists to prevent, so say so rather than printing nothing. if ! command -v jq >/dev/null 2>&1 || ! command -v tasks-axi >/dev/null 2>&1 \ @@ -460,26 +602,32 @@ cmd_pending() { [ -n "$id" ] || continue payload=$(printf '%s' "$listing" | jq -ce --arg id "$id" \ '(.public_followups // []) | map(select(.id == $id)) | .[0] // empty' 2>/dev/null) - if [ -z "$payload" ]; then - # The obligation is gone from the backlog (pruned after Done): the - # registration is stale bookkeeping, not evidence, so drop it. - if ! clear_public_followup_link "$id"; then - printf 'cannot clear the legacy X link for closed public commitment %s; registration retained for reconciliation\n' "$id" - printed=1 - continue - fi - rm -f -- "$(fm_pf_registry_dir "$STATE")/$id" 2>/dev/null || true - continue - fi + loop_state=$(fm_pf_registry_loop_state "$STATE" "$id") delivery=$(pf_field "$payload" '.public_followup.delivery.state') task_state=$(pf_field "$payload" '.state') - if [ "$task_state" = 'done' ] || [ "$delivery" = 'posted' ] || [ "$delivery" = 'waived' ]; then - if ! clear_public_followup_link "$id"; then - printf 'cannot clear the legacy X link for closed public commitment %s; registration retained for reconciliation\n' "$id" - printed=1 - continue + settled=0 + if [ -z "$payload" ] || [ "$task_state" = 'done' ] \ + || [ "$delivery" = 'posted' ] || [ "$delivery" = 'waived' ] \ + || [ "$loop_state" = delivered ]; then + settled=1 + fi + if [ "$settled" -eq 1 ]; then + if [ "$loop_state" != delivered ]; then + stamp_rc=0 + fm_pf_registry_stamp_delivered "$STATE" "$id" "$(now_rfc3339)" || stamp_rc=$? + if [ "$stamp_rc" -eq 3 ] && fm_pf_retirement_receipt_exists "$STATE" "$id"; then + continue + fi + [ "$stamp_rc" -eq 0 ] \ + || die "could not stamp settled registration '$id' as delivered" 1 fi - rm -f -- "$(fm_pf_registry_dir "$STATE")/$id" 2>/dev/null || true + # Keep the registration. Clearing a leftover legacy link is best-effort + # and never the close; only retire removes the record. + if public_followup_registration_valid "$id"; then + clear_public_followup_link "$id" >/dev/null 2>&1 || true + fi + print_open_loop "$id" "$payload" + printed=1 continue fi summary=$(pf_field "$payload" '.public_followup.request.public_safe_summary' | fm_pf_clean_outcome_text) @@ -487,6 +635,12 @@ cmd_pending() { request=$(pf_field "$payload" '.public_followup.request.request_id') printf 'unresolved %s state=%s platform=%s request=%s summary=%s\n' \ "$id" "${delivery:-unknown}" "${platform:-unknown}" "${request:-unknown}" "$summary" + expires=$(fm_pf_registry_get "$STATE" "$id" followup_expires_at) + [ -n "$expires" ] || expires=$(pf_field "$payload" '.public_followup.request.followup_expires_at') + print_window_escalation "$expires" + if ! fm_pf_registry_rechainable "$STATE" "$id"; then + printf ' unrechainable: pre-change registration lacks request_context_b64\n' + fi printed=1 done <<EOF $(fm_pf_registry_ids "$STATE") @@ -518,24 +672,33 @@ public_followup_registration_valid() { } public_followup_secondmate_home() { - local id=$1 meta home marker + local id=$1 include_absent=${2:-} meta_home registry_home home marker fm_pf_home_id_valid "secondmate:$id" || return 1 - meta="$STATE/$id.meta" - home=$(fmx_meta_get "$meta" home) - if [ -z "$home" ] && [ -f "$DATA/secondmates.md" ] && [ ! -L "$DATA/secondmates.md" ]; then - home=$(secondmate_registry_field "$DATA/secondmates.md" "$id" home || true) - fi - [ -n "$home" ] || return 1 - case "$home" in /*) ;; *) return 1 ;; esac - home=$(CDPATH='' cd -- "$home" 2>/dev/null && pwd -P) || return 1 - [ -f "$home/.fm-secondmate-home" ] && [ ! -L "$home/.fm-secondmate-home" ] || return 1 + meta_home=$(fmx_meta_get "$STATE/$id.meta" home) + registry_home= + if [ -f "$DATA/secondmates.md" ] && [ ! -L "$DATA/secondmates.md" ]; then + registry_home=$(secondmate_registry_field "$DATA/secondmates.md" "$id" home || true) + fi + if [ -n "$meta_home" ] && [ -n "$registry_home" ] && [ "$meta_home" != "$registry_home" ]; then + return 2 + fi + home=${meta_home:-$registry_home} + [ -n "$home" ] || return 4 + case "$home" in /*) ;; *) return 2 ;; esac + if [ ! -e "$home" ]; then + [ ! -L "$home" ] || return 2 + [ "$include_absent" = include-absent ] && printf '%s\n' "$home" + return 3 + fi + home=$(CDPATH='' cd -- "$home" 2>/dev/null && pwd -P) || return 2 + [ -f "$home/.fm-secondmate-home" ] && [ ! -L "$home/.fm-secondmate-home" ] || return 2 marker=$(sed -n '1p' "$home/.fm-secondmate-home" 2>/dev/null) - [ "$marker" = "$id" ] || return 1 + [ "$marker" = "$id" ] || return 2 printf '%s\n' "$home" } clear_public_followup_link() { - local id=$1 work_home work_id home state + local id=$1 work_home work_home_path work_id home state rc public_followup_registration_valid "$id" || return 1 work_home=$(fm_pf_registry_get "$STATE" "$id" work_home) work_id=$(fm_pf_registry_get "$STATE" "$id" work_id) @@ -546,7 +709,22 @@ clear_public_followup_link() { state=$STATE ;; secondmate:*) - home=$(public_followup_secondmate_home "${work_home#secondmate:}") || return 1 + work_home_path=$(fm_pf_registry_get "$STATE" "$id" work_home_path) + case "$work_home_path" in /*) ;; *) return 1 ;; esac + case "$work_home_path" in *$'\n'*|*$'\r'*) return 1 ;; esac + rc=0 + home=$(public_followup_secondmate_home "${work_home#secondmate:}" include-absent) || rc=$? + if [ "$rc" -eq 3 ]; then + [ "$home" = "$work_home_path" ] || return 1 + [ ! -e "$work_home_path" ] && [ ! -L "$work_home_path" ] || return 1 + return 0 + fi + if [ "$rc" -eq 4 ]; then + [ ! -e "$work_home_path" ] && [ ! -L "$work_home_path" ] || return 1 + return 0 + fi + [ "$rc" -eq 0 ] || return 1 + [ "$home" = "$work_home_path" ] || return 1 state="$home/state" ;; *) return 1 ;; @@ -619,6 +797,24 @@ record_posted() { return "$rc" } +# Delivery keeps the registration. Stamp it delivered and tell the caller the +# public loop is still open. +mark_loop_delivered() { + local id=$1 rc=0 + fm_pf_registry_stamp_delivered "$STATE" "$id" "$(now_rfc3339)" || rc=$? + case "$rc" in + 0) return 0 ;; + 3) return 3 ;; + *) die "could not stamp registration '$id' as delivered after the public reply landed" 1 ;; + esac +} + +print_loop_open_disposition() { + local id=$1 request=$2 + printf "thread %s is still OPEN: hand it on with 'rechain ...' or close it with 'retire %s --reason ...'\n" \ + "${request:-unknown}" "$id" +} + cmd_deliver() { local id=${1:-} text_file= [ -n "$id" ] || { usage; exit 2; } @@ -637,6 +833,7 @@ cmd_deliver() { require_tools local payload delivery attempt request platform text tmp_text hash chunks rc receipt receipt_fields receipt_dry_run link_status + local loop_retained=0 payload=$(obligation_json "$id") || die "could not read the backlog through tasks-axi" 1 [ -n "$payload" ] || die "no public-followup obligation '$id' in this home's backlog" 1 @@ -661,8 +858,9 @@ cmd_deliver() { *) die "obligation '$id' is already $delivery, but its registration is missing or invalid and the legacy X link cannot be verified; reconcile it before any later terminal follow-up" 1 ;; esac fi - rm -f -- "$(fm_pf_registry_dir "$STATE")/$id" 2>/dev/null || true + if mark_loop_delivered "$id"; then loop_retained=1; fi printf 'already delivered %s state=%s\n' "$id" "$delivery" + [ "$loop_retained" -eq 0 ] || print_loop_open_disposition "$id" "$request" return 0 ;; ready|retry-due|context-blocked|unknown|partial) @@ -743,8 +941,9 @@ EOF if ! clear_public_followup_link "$id"; then die "the public reply for '$id' POSTED and its receipt was recorded, but its legacy X link could not be cleared; the registration was retained for reconciliation" 1 fi - rm -f -- "$(fm_pf_registry_dir "$STATE")/$id" 2>/dev/null || true + if mark_loop_delivered "$id"; then loop_retained=1; fi printf 'delivered %s request=%s platform=%s chunks=%s\n' "$id" "$request" "$platform" "$chunks" + [ "$loop_retained" -eq 0 ] || print_loop_open_disposition "$id" "$request" return 0 fi die "the public reply for '$id' POSTED but its receipt could not be recorded; close it with 'record-posted $id --attempt $attempt --chunks <exact-count>' before any retry, or the thread will get a second reply" 1 @@ -789,7 +988,7 @@ cmd_record_posted() { || die "public-followup registration for '$id' is missing or invalid; reconcile it before recording a receipt so any legacy X link can be cleared" 1 require_tools - local payload request platform + local payload request platform loop_retained=0 payload=$(obligation_json "$id") || die "could not read the backlog through tasks-axi" 1 [ -n "$payload" ] || die "no public-followup obligation '$id' in this home's backlog" 1 request=$(pf_field "$payload" '.public_followup.request.request_id') @@ -800,8 +999,9 @@ cmd_record_posted() { if ! clear_public_followup_link "$id"; then die "the receipt for '$id' was recorded, but its legacy X link could not be cleared; the registration was retained for reconciliation" 1 fi - rm -f -- "$(fm_pf_registry_dir "$STATE")/$id" 2>/dev/null || true + if mark_loop_delivered "$id"; then loop_retained=1; fi printf 'recorded %s attempt=%s request=%s\n' "$id" "$attempt" "$request" + [ "$loop_retained" -eq 0 ] || print_loop_open_disposition "$id" "$request" } # --- subcommand: guard-work ------------------------------------------------- @@ -847,22 +1047,229 @@ EOF [ "$blocked" -eq 0 ] || exit 3 } +# --- subcommand: rechain ---------------------------------------------------- + +rechain_default_deliverable_key() { + case "$1" in + pr-merged) printf 'pr_url\n' ;; + report-ready) printf 'report_path\n' ;; + *) return 1 ;; + esac +} + +cmd_rechain() { + local new_id=${1:-} from='' work_home='' work_id='' expected='' + local -a deliverable_keys=() + [ -n "$new_id" ] || { usage; exit 2; } + shift + while [ "$#" -gt 0 ]; do + case "$1" in + --from) shift; from=${1:-} ;; + --work-home) shift; work_home=${1:-} ;; + --work-id) shift; work_id=${1:-} ;; + --expected) shift; expected=${1:-} ;; + --deliverable-key) shift; deliverable_keys+=("${1:-}") ;; + *) die "unknown argument '$1'" ;; + esac + shift || true + done + + fm_pf_relay_active "$FM_HOME" \ + || die "this home has not opted into the myfirstmate relay, so it cannot own a public commitment" 1 + require_tools + fm_pf_slug_valid "$new_id" || die "unsafe obligation id: $new_id" + fm_pf_slug_valid "$from" || die "unsafe source obligation id: $from" + fm_pf_slug_valid "$work_id" || die "unsafe work id: $work_id" + fm_pf_home_id_valid "$work_home" \ + || die "work home must be 'main' or 'secondmate:<stable-id>', got '$work_home'" + case "$expected" in + pr-merged|report-ready|local-main) ;; + *) die "--expected must be pr-merged, report-ready, or local-main, got '$expected'" ;; + esac + [ "$new_id" != "$from" ] || die "the new obligation id must differ from --from" 2 + + pf_registry_lock_acquire "$from" \ + || die "could not lock source registration '$from' for rechain" 1 + local src_file loop_state expires window ctx rechain_to source_record first_claim=0 existing + src_file="$(fm_pf_registry_dir "$STATE")/$from" + [ -f "$src_file" ] && [ ! -L "$src_file" ] \ + || die "no registration for '$from' in this home" 1 + loop_state=$(fm_pf_registry_loop_state "$STATE" "$from") + [ "$loop_state" = delivered ] \ + || die "source '$from' is not state=delivered (got '$loop_state'); nothing to hand on until that final lands" 1 + fm_pf_registry_rechainable "$STATE" "$from" \ + || die "source '$from' is un-rechainable: a pre-change registration has no request_context_b64. Close it with retire --reason or reconstruct the request context by hand." 1 + + expires=$(fm_pf_registry_get "$STATE" "$from" followup_expires_at) + [ -n "$expires" ] || die "source '$from' has no followup_expires_at; the thread window cannot be checked" 1 + window=$(fm_pf_followup_window_class "$expires") + case "$window" in + ok|closing) ;; + expired) + die "followup_expires_at $expires is in the past: the thread can no longer be reached, so this loop cannot be closed publicly. This is a captain decision." 1 + ;; + *) + die "followup_expires_at $expires could not be parsed: the thread window cannot be checked, so this loop cannot be rechained" 1 + ;; + esac + + if [ "${#deliverable_keys[@]}" -eq 0 ]; then + local default_key + default_key=$(rechain_default_deliverable_key "$expected") \ + || die "--expected $expected needs --deliverable-key <k> (no default key)" + deliverable_keys+=("$default_key") + fi + local key + for key in "${deliverable_keys[@]}"; do + case "$key" in + ''|*[!a-z0-9_]*) die "deliverable key must be lowercase [a-z0-9_], got '$key'" ;; + esac + done + + # Claim the delivered baton before publishing its destination. The claim is + # retained if any later retirement step fails, so a retry may resume the same + # destination but can never fork this thread into a second obligation. + rechain_to=$(fm_pf_registry_get "$STATE" "$from" rechain_to) + if [ -n "$rechain_to" ] && [ "$rechain_to" != "$new_id" ]; then + die "source '$from' is already claimed by rechain destination '$rechain_to'; resume that destination" 1 + fi + if [ -z "$rechain_to" ]; then + existing=$(obligation_json "$new_id") \ + || die "could not check whether rechain destination '$new_id' is unused" 1 + [ -z "$existing" ] \ + || die "'$new_id' already exists and was not created by this rechain; choose another id" 1 + [ ! -e "$(fm_pf_registry_dir "$STATE")/$new_id" ] \ + && [ ! -L "$(fm_pf_registry_dir "$STATE")/$new_id" ] \ + && [ ! -e "$(fm_pf_retired_dir "$STATE")/$new_id" ] \ + && [ ! -L "$(fm_pf_retired_dir "$STATE")/$new_id" ] \ + || die "'$new_id' already has local public-loop state; choose another id" 1 + source_record=$(grep -v -E '^rechain_to=' "$src_file" 2>/dev/null) \ + || die "could not read source registration '$from' while claiming it" 1 + printf '%s\nrechain_to=%s\n' "$source_record" "$new_id" \ + | fmx_private_artifact_publish_stdin "$(fm_pf_registry_dir "$STATE")" "$from" 600 \ + || die "could not claim source registration '$from' for '$new_id'" 1 + first_claim=1 + fi + + local ctx_file expected_file relation_file keys_json project src_payload + ctx=$(fm_pf_registry_get "$STATE" "$from" request_context_b64) + ctx_file=$(mktemp "${TMPDIR:-/tmp}/fm-pf-rechain-ctx.XXXXXX") \ + || die "could not stage the retained request context" 1 + expected_file=$(mktemp "${TMPDIR:-/tmp}/fm-pf-rechain-exp.XXXXXX") \ + || die "could not stage the expected-final document" 1 + relation_file=$(mktemp "${TMPDIR:-/tmp}/fm-pf-rechain-rel.XXXXXX") \ + || die "could not stage the relation document" 1 + PF_TEMP_FILES+=("$ctx_file" "$expected_file" "$relation_file") + printf '%s' "$ctx" | fm_pf_b64_decode > "$ctx_file" \ + || die "could not decode request_context_b64 for '$from'" 1 + jq -e 'type == "object" and (.request_id | type == "string")' "$ctx_file" >/dev/null 2>&1 \ + || die "decoded request context for '$from' is not usable" 1 + + keys_json=$(printf '%s\n' "${deliverable_keys[@]}" | jq -R . | jq -s -c .) + project= + if src_payload=$(obligation_json "$from") && [ -n "$src_payload" ]; then + project=$(pf_field "$src_payload" '.public_followup.expected_final.project') + fi + if [ -n "$project" ]; then + jq -n --arg t "$expected" --arg p "$project" --argjson keys "$keys_json" \ + '{type:$t, project:$p, required_deliverables:$keys, completion_policy:"all-required"}' \ + > "$expected_file" + else + jq -n --arg t "$expected" --argjson keys "$keys_json" \ + '{type:$t, required_deliverables:$keys, completion_policy:"all-required"}' \ + > "$expected_file" + fi + jq -n --arg h "$work_home" --arg w "$work_id" \ + '{relation_id:"rel-1", work_ref:{home_id:$h, task_id:$w}, + role:"fulfills", required:true, generation:1}' > "$relation_file" + + local relation_count new_registry + if [ "$first_claim" -eq 1 ]; then + existing= + else + existing=$(obligation_json "$new_id") \ + || die "could not read the backlog through tasks-axi" 1 + fi + if [ -n "$existing" ]; then + printf '%s' "$existing" | jq -e \ + --slurpfile request "$ctx_file" --slurpfile expected "$expected_file" \ + --arg expires "$expires" \ + '.public_followup as $pf + | $pf.request == $request[0] + and $pf.purpose == "promised-final" + and $pf.expected_final == $expected[0] + and $pf.obligation_expires_at == $expires' >/dev/null 2>&1 \ + || die "'$new_id' already exists with different public-followup data; choose another id" 1 + else + tx public-followup add "$new_id" --request-context-file "$ctx_file" \ + --purpose promised-final --expected-final-file "$expected_file" \ + --expires-at "$expires" >/dev/null \ + || die "tasks-axi refused to add '$new_id' on the retained thread binding" 1 + existing=$(obligation_json "$new_id") \ + || die "added '$new_id' but could not read it back through tasks-axi; retry this same rechain command" 1 + fi + + relation_count=$(printf '%s' "$existing" \ + | jq -r '(.public_followup.work_relations // []) | length' 2>/dev/null) \ + || die "could not inspect work bindings for '$new_id'" 1 + if [ "$relation_count" -eq 0 ]; then + tx public-followup bind-work "$new_id" --relation-file "$relation_file" >/dev/null \ + || die "tasks-axi refused to bind '$new_id' to $work_home/$work_id; retry this same rechain command" 1 + else + printf '%s' "$existing" | jq -e --arg h "$work_home" --arg w "$work_id" \ + '(.public_followup.work_relations // []) as $relations + | ($relations | length) == 1 + and $relations[0].relation_id == "rel-1" + and $relations[0].work_ref.home_id == $h + and $relations[0].work_ref.task_id == $w + and $relations[0].role == "fulfills" + and $relations[0].required == true + and $relations[0].generation == 1' >/dev/null 2>&1 \ + || die "'$new_id' already has a different work binding; choose another id" 1 + fi + + new_registry="$(fm_pf_registry_dir "$STATE")/$new_id" + if [ -f "$new_registry" ] && [ ! -L "$new_registry" ]; then + [ "$(fm_pf_registry_get "$STATE" "$new_id" relation_id)" = rel-1 ] \ + && [ "$(fm_pf_registry_get "$STATE" "$new_id" work_home)" = "$work_home" ] \ + && [ "$(fm_pf_registry_get "$STATE" "$new_id" work_id)" = "$work_id" ] \ + && [ "$(fm_pf_registry_get "$STATE" "$new_id" generation)" = 1 ] \ + || die "registration '$new_id' already names different work; choose another id" 1 + else + cmd_register "$new_id" --relation rel-1 --work-home "$work_home" \ + --work-id "$work_id" --generation 1 >/dev/null \ + || die "could not register '$new_id'; retry this same rechain command" 1 + fi + + cmd_retire "$from" --reason "handed on to $new_id" \ + || die "registered '$new_id' but could not retire '$from'; both loops are open until '$from' is retired" 1 + + cmd_brief "$new_id" +} + # --- subcommand: retire ----------------------------------------------------- cmd_retire() { - local id=${1:-} force=0 payload delivery task_state + local id=${1:-} force=0 reason='' payload delivery task_state registry_file retired_dir retired_at + local retirement_rc=0 [ -n "$id" ] || { usage; exit 2; } shift while [ "$#" -gt 0 ]; do case "$1" in --force) force=1 ;; + --reason) shift; reason=${1:-} ;; *) die "unknown argument '$1'" ;; esac shift || true done fm_pf_slug_valid "$id" || die "unsafe obligation id: $id" fm_pf_relay_active "$FM_HOME" || exit 0 + [ -n "$reason" ] || die "retire requires --reason \"<why the public loop is done>\"" 2 + reason=$(printf '%s' "$reason" | fm_pf_clean_outcome_text) + [ -n "$reason" ] || die "retire requires --reason \"<why the public loop is done>\"" 2 require_tools + pf_registry_lock_acquire "$id" \ + || die "could not lock registration '$id' for retirement" 1 payload=$(obligation_json "$id") || die "could not read the backlog through tasks-axi" 1 if [ -n "$payload" ]; then @@ -879,8 +1286,24 @@ cmd_retire() { if ! clear_public_followup_link "$id"; then die "could not clear the legacy X link for '$id'; its registration was retained for reconciliation" 1 fi - rm -f -- "$(fm_pf_registry_dir "$STATE")/$id" 2>/dev/null || true - printf 'retired %s\n' "$id" + retired_dir=$(fm_pf_retired_dir "$STATE") + retired_at=$(now_rfc3339) + registry_file="$(fm_pf_registry_dir "$STATE")/$id" + printf 'reason=%s\nretired_at=%s\n' "$reason" "$retired_at" \ + | fmx_private_artifact_publish_stdin "$retired_dir" "$id" 600 \ + || retirement_rc=1 + if [ "$retirement_rc" -eq 0 ]; then + if ! rm -f -- "$registry_file" 2>/dev/null \ + || [ -e "$registry_file" ] || [ -L "$registry_file" ]; then + retirement_rc=2 + fi + fi + pf_registry_lock_release "$id" + case "$retirement_rc" in + 1) die "could not record the retirement reason for '$id'; the public loop remains open" 1 ;; + 2) die "could not remove registration for '$id'; the public loop remains open" 1 ;; + esac + printf 'retired %s reason=%s\n' "$id" "$reason" } # --- dispatch --------------------------------------------------------------- @@ -901,6 +1324,7 @@ case "$CMD" in deliver) cmd_deliver "$@" ;; record-posted) cmd_record_posted "$@" ;; guard-work) cmd_guard_work "$@" ;; + rechain) cmd_rechain "$@" ;; retire) cmd_retire "$@" ;; *) usage; exit 2 ;; esac diff --git a/bin/fm-push-transition-lib.sh b/bin/fm-push-transition-lib.sh index 5ee55fd3b42..19d0a142a90 100644 --- a/bin/fm-push-transition-lib.sh +++ b/bin/fm-push-transition-lib.sh @@ -130,8 +130,12 @@ handle_push_transition() { # <backend> <session> <record> [ -n "$pane_id" ] || { sleep 1; return; } window="$session:$pane_id" task=$(window_to_task "$window" "$STATE") - if status_is_paused "$(last_status_line "$STATE/$task.status")"; then - triage_log "absorbed push $to (declared pause, awaiting external): $window" + # A declared wait already names the human this transition would report: an + # external dependency, or the captain a verified hold transferred the work to. + # Either way the wait is durably recorded, so absorb the immediate escalation + # and leave the bounded re-surface to the watcher's own pause cadence. + if status_is_paused_or_captain_held "$(last_status_line "$STATE/$task.status")"; then + triage_log "absorbed push $to (declared wait, awaiting external or captain): $window" fm_backend_commit_transition "$backend" "$STATE" "$session" "$record" || exit 1 return fi diff --git a/bin/fm-remote-job-lib.sh b/bin/fm-remote-job-lib.sh index 73bffa54c70..25d7bb73b40 100755 --- a/bin/fm-remote-job-lib.sh +++ b/bin/fm-remote-job-lib.sh @@ -30,7 +30,10 @@ # PATH, HOME, FM_HOME, FM_ROOT_OVERRIDE, and FM_REMOTE_JOB_ACTIVE=1. The PATH # is intentionally filesystem-discovered rather than login-shell-derived: # ~/.local/bin; nvm, asdf, and mise shims/install bins; Nix; Homebrew; and the -# system tail. No shell startup files are evaluated. +# system tail. No shell startup files are evaluated. Each discovered set is +# appended in the shell's own sorted pathname-expansion order, so which install +# of a multi-version tool wins is fixed by this composition rather than by the +# order the filesystem happens to return. # # On macOS the worker is Firstmate's Aqua LaunchAgent # dev.firstmate.remote-job at ~/Library/LaunchAgents/dev.firstmate.remote-job.plist @@ -130,11 +133,18 @@ fm_remote_job_path_append_resolved_dir() { # <directory> fm_remote_job_path_append "$physical" } -fm_remote_job_append_glob_dirs() { # <glob whose matches are directories> - local pattern=$1 directory - while IFS= read -r directory; do +# Callers pass an already-expanded glob rather than the pattern, because only +# the shell's own pathname expansion sorts its matches: bash sorts +# glob_filename's result in pathexp.c, while `compgen -G` reaches the same +# glob_filename through pcomplete.c, which does not sort. On bash 3.2 (macOS +# /bin/bash) that handed back raw readdir order, so which install of a +# multi-version tool a remote job resolved depended on the filesystem instead +# of on this composition. +fm_remote_job_append_dirs() { # <expanded glob matches> + local directory + for directory in "$@"; do fm_remote_job_path_append_if_dir "$directory" - done < <(compgen -G "$pattern" || true) + done } fm_remote_job_nvm_default_selector() { # <account-home> @@ -217,11 +227,11 @@ fm_remote_job_compose_operator_path() { # <account-home> nvm_bin=$(fm_remote_job_nvm_selected_bin "$account_home" 2>/dev/null || true) [ -z "$nvm_bin" ] || fm_remote_job_path_append "$nvm_bin" fm_remote_job_path_append_if_dir "$account_home/.asdf/shims" - fm_remote_job_append_glob_dirs "$account_home/.asdf/installs/*/*/bin" + fm_remote_job_append_dirs "$account_home"/.asdf/installs/*/*/bin fm_remote_job_path_append_if_dir "$account_home/.local/share/mise/shims" fm_remote_job_path_append_if_dir "$account_home/.mise/shims" - fm_remote_job_append_glob_dirs "$account_home/.local/share/mise/installs/*/*/bin" - fm_remote_job_append_glob_dirs "$account_home/.mise/installs/*/*/bin" + fm_remote_job_append_dirs "$account_home"/.local/share/mise/installs/*/*/bin + fm_remote_job_append_dirs "$account_home"/.mise/installs/*/*/bin fm_remote_job_path_append_resolved_dir "$account_home/.nix-profile/bin" account_user=$(id -un 2>/dev/null || true) if [ -n "$account_user" ]; then diff --git a/bin/fm-remote-job-worker.sh b/bin/fm-remote-job-worker.sh index 2a49dd66947..edd123c859e 100755 --- a/bin/fm-remote-job-worker.sh +++ b/bin/fm-remote-job-worker.sh @@ -291,8 +291,17 @@ worker_stop_active_execution() { WORKER_ACTIVE_JOB= } +# Ignore, rather than restore the default disposition for, the signals this +# handler answers. A replacement stops a Linux worker by signalling its whole +# isolated group, and the supervisor in that group forwards a second stop signal +# to this same serving child, so a repeat is the normal case and not an +# exception. Restoring the default let that second signal kill the shutdown part +# way through, which left the ownership lock behind holding a half-written temp +# file that no later worker could clear, so every replacement then failed to +# report ready. A shutdown that hangs is still stopped: the caller escalates to +# KILL, which no disposition can block. worker_shutdown() { - trap - HUP INT TERM + trap '' HUP INT TERM worker_publish_quarantine || { worker_error "cannot guard worker ownership for shutdown" trap worker_shutdown HUP INT TERM diff --git a/bin/fm-send.sh b/bin/fm-send.sh index cc199c9b016..512df6245c7 100755 --- a/bin/fm-send.sh +++ b/bin/fm-send.sh @@ -74,19 +74,20 @@ # backend or remote transport. # # Chat is also a channel that carries keyed captain answers, so the same flag -# feeds bin/fm-decision-hold.sh's one keyed-answer intake for any key that names -# a durable decision hold on the target task. fm-send maps nothing to a hold and -# closes nothing itself; it hands the intake `<key>\t<answer>\t<label>` exactly -# as every other channel does, and the intake owns what that means. This is what -# lets an answer reach a decision that has already been transferred from the live -# status log to its durable hold, which the status ledger alone can no longer -# close. +# feeds bin/fm-captain-hold.sh's one keyed-answer intake for any key that names +# a captain-held task in this home - the key as a task id itself, or through +# the legacy `<task>-decision-<key>` identity for pre-collapse rows. fm-send +# closes nothing itself; it hands the intake `<task-id>\t<answer>\t<label>` +# exactly as every other channel does, and the intake owns what that means. +# This is what lets an answer reach a decision that has already been +# transferred from the live status log to its durable captain-held task, which +# the status ledger alone can no longer close. # # Each named key must therefore currently be open in ONE of the two ledgers: open # in this home's status log per status_open_decisions (bin/fm-classify-lib.sh), or -# an active captain hold for the target task. A key in neither is refused before -# sending, so a mistyped key cannot deliver an answer while silently orphaning the -# decision. A failed or unconfirmed send never closes a key (a remote +# a still-open captain-held task resolved as above. A key in neither is refused +# before sending, so a mistyped key cannot deliver an answer while silently +# orphaning the decision. A failed or unconfirmed send never closes a key (a remote # delivered-with-pending-confirmation outcome counts as delivered - see the # remote paragraph above); a # delivered answer whose closing append fails exits nonzero with the exact @@ -388,21 +389,32 @@ fi # silently leaves its decision open. RESOLVE_STATUS_FILE= # Which ledger each answered key belongs to. A key still open in the status log -# is owned by the status log: fm-decision-hold's `complete` closes that live copy -# at the moment it transfers a decision to its durable hold, so "still open in -# status" and "already a hold" are the two sides of one transfer, never both at -# once. Checking the hold only for keys the status log no longer owns also keeps -# the common path free of any backlog read. +# is owned by the status log: fm-captain-hold's `complete` closes that live copy +# at the moment it transfers a decision to its durable captain-held task, so +# "still open in status" and "already held" are the two sides of one transfer, +# never both at once. Checking the backlog only for keys the status log no +# longer owns also keeps the common path free of any backlog read. RESOLVE_STATUS_KEYS= RESOLVE_HOLD_KEYS= -fm_send_hold_is_active() { # <task-id> <decision-key> - local show +# Resolve a --resolve-key key that the status log no longer owns to the +# captain-held task that carries it: the key as a task id itself (the collapsed +# identity - a captain call IS a task held for the captain), then the legacy +# derived `<task>-decision-<key>` identity for pre-collapse rows. Answerable +# means not closed and still carrying the captain-hold annotations tasks-axi +# preserves even past a hold-until date. +fm_send_hold_resolved_id() { # <task-id> <decision-key> + local show id state hold_kind command -v tasks-axi >/dev/null 2>&1 || return 1 - show=$( (cd "$FM_HOME" && tasks-axi show "$1-decision-$2" --full) 2>/dev/null ) || return 1 - case "$show" in *"held: yes"*) : ;; *) return 1 ;; esac - case "$show" in *"hold_kind: captain"*) : ;; *) return 1 ;; esac - case "$show" in *"state: queued"*) return 0 ;; esac + for id in "$2" "$1-decision-$2"; do + show=$( (cd "$FM_HOME" && tasks-axi show "$id" --full) 2>/dev/null ) || continue + state=$(printf '%s\n' "$show" | sed -n 's/^ state: //p' | head -1) + hold_kind=$(printf '%s\n' "$show" | sed -n 's/^ hold_kind: //p' | head -1) + [ "$state" != "done" ] || continue + [ "$hold_kind" = captain ] || continue + printf '%s\n' "$id" + return 0 + done return 1 } @@ -430,13 +442,13 @@ if [ -n "$RESOLVE_KEYS" ]; then ;; esac # Not open in the status log. A decision already transferred to its durable - # hold is exactly this case, and it is answerable - just through the other - # ledger - so check there before refusing. - if fm_send_hold_is_active "$RESOLVE_TASK_ID" "$k"; then - RESOLVE_HOLD_KEYS="${RESOLVE_HOLD_KEYS}${RESOLVE_HOLD_KEYS:+ }$k" + # captain-held task is exactly this case, and it is answerable - just + # through the other ledger - so check there before refusing. + if resolved_hold_id=$(fm_send_hold_resolved_id "$RESOLVE_TASK_ID" "$k"); then + RESOLVE_HOLD_KEYS="${RESOLVE_HOLD_KEYS}${RESOLVE_HOLD_KEYS:+ }$resolved_hold_id" continue fi - echo "error: --resolve-key '$k': no open decision or blocker with that key in $RESOLVE_STATUS_FILE, and no active captain decision $RESOLVE_TASK_ID-decision-$k (already closed or mistyped). Re-check the OPEN DECISIONS listing, then resend without that key or with the right one; nothing was sent." >&2 + echo "error: --resolve-key '$k': no open decision or blocker with that key in $RESOLVE_STATUS_FILE, and no captain-held task '$k' or '$RESOLVE_TASK_ID-decision-$k' still open (already closed or mistyped). Re-check the OPEN DECISIONS listing, then resend without that key or with the right one; nothing was sent." >&2 exit 1 done fi @@ -463,9 +475,10 @@ fm_send_close_resolved_keys() { # <answer-text> done } -# Feed the answered hold keys to the ONE keyed-answer intake, as keyed lines, -# exactly the way every other channel does. fm-send decides nothing here: it does -# not map a key to a hold, build a decision record, or choose a close path. +# Feed the answered captain-held tasks to the ONE keyed-answer intake, as keyed +# lines, exactly the way every other channel does. fm-send decides nothing here: +# it does not build a decision record or choose a close path; the keys were +# already resolved to task ids above, so the intake needs no legacy origin. fm_send_feed_resolved_holds() { # <answer-text> local note=$1 k lines='' [ -n "$RESOLVE_HOLD_KEYS" ] || return 0 @@ -473,9 +486,9 @@ fm_send_feed_resolved_holds() { # <answer-text> for k in $RESOLVE_HOLD_KEYS; do lines="${lines}${k}"$'\t'"${note}"$'\t'$'\n' done - if ! printf '%s' "$lines" | "$SCRIPT_DIR/fm-decision-hold.sh" answers "$RESOLVE_TASK_ID" \ + if ! printf '%s' "$lines" | "$SCRIPT_DIR/fm-captain-hold.sh" answers \ --source "a firstmate answer sent to $RESOLVE_TASK_ID" >/dev/null 2>&1; then - echo "error: the answer was delivered to $T, but this captain decision could not be closed: ${RESOLVE_HOLD_KEYS}. Close it with fm-decision-hold.sh (answer, or resolve when it routes work) - do not resend the answer." >&2 + echo "error: the answer was delivered to $T, but this captain-held task could not be closed: ${RESOLVE_HOLD_KEYS}. Close it with fm-captain-hold.sh answer - do not resend the answer." >&2 return 1 fi } diff --git a/bin/fm-session-start.sh b/bin/fm-session-start.sh index ba9d5ccef3d..a0383e46810 100755 --- a/bin/fm-session-start.sh +++ b/bin/fm-session-start.sh @@ -846,11 +846,12 @@ if fm_pf_relay_active "$FM_HOME" \ && { fm_pf_has_registrations "$STATE" || fm_pf_has_events "$STATE"; }; then PUBLIC_FOLLOWUP=$("$SCRIPT_DIR/fm-public-followup.sh" pending 2>/dev/null) || PUBLIC_FOLLOWUP= if [ -n "$PUBLIC_FOLLOWUP" ]; then - subsection "Public commitments awaiting delivery" + subsection "Public commitments" printf '%s\n' "$PUBLIC_FOLLOWUP" - printf '\nEach line is a public reply this home still owes. Reconcile terminal results with\n' - printf '%s/bin/fm-public-followup.sh consume, then deliver a ready one with\n' "$FM_ROOT" - printf '%s/bin/fm-public-followup.sh deliver <id>. Load fmx-respond for the procedure.\n' "$FM_ROOT" + printf '\nEach line is a public loop this home still holds: a reply still owed, or an open loop with nothing owed.\n' + printf 'Reconcile terminal results with %s/bin/fm-public-followup.sh consume, then deliver a ready one with\n' "$FM_ROOT" + printf '%s/bin/fm-public-followup.sh deliver <id>. Hand a delivered loop on with rechain, or close it with\n' "$FM_ROOT" + printf '%s/bin/fm-public-followup.sh retire <id> --reason "...". Load fmx-respond for the procedure.\n' "$FM_ROOT" fi fi diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index fc7758ac70c..c37df23bcb1 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -374,7 +374,7 @@ else exit 1 } [ "$YOLO_SET" -eq 1 ] || { - echo "error: ship spawns require --yolo <on|off>; it is this task's routine approval authority, not a project lookup" >&2 + echo "error: ship spawns require --yolo <on|off>; it is this task's merge authority, not a project lookup" >&2 exit 1 } case "$MODE" in diff --git a/bin/fm-supervise-daemon.sh b/bin/fm-supervise-daemon.sh index ff123a7fafb..86bad52b44c 100755 --- a/bin/fm-supervise-daemon.sh +++ b/bin/fm-supervise-daemon.sh @@ -5,11 +5,11 @@ # durable wake after an actionable close, acknowledges only after routing, and # either SELF-HANDLES the routine majority in bash (no firstmate turn) or # ESCALATES a batched, distilled digest to the supervisor pane on -# captain-relevant events plus bounded declared-pause rechecks. This is the +# captain-relevant events plus bounded declared-wait rechecks. This is the # token-efficient replacement for the prior always-inject daemon: routine # signal/stale/heartbeat wakes cost zero firstmate context; only done/ # needs-decision/blocked/failed/persistent-wedge/check-output events and a -# declared-pause recheck reach the LLM, and even then as one pre-read digest per +# declared-wait recheck reach the LLM, and even then as one pre-read digest per # batch window. # # PRESENCE-GATING (the /afk contract). The daemon is the away-mode engine: it @@ -41,11 +41,13 @@ # drain and acknowledges it only after routing completes. # - Fail-safe-to-escalate: any wake the classifier cannot confidently mark # routine is escalated. -# - Bounded wedge latency: a stale pane without a declared external wait is -# escalated only after it has been idle for STALE_ESCALATE_SECS +# - Bounded wedge latency: a stale pane without a declared wait is escalated +# only after it has been idle for STALE_ESCALATE_SECS # (configurable), rechecked once. A wedged crewmate is therefore detected -# within STALE_ESCALATE_SECS + a tick, never lost. A declared pause instead -# gets its own longer PAUSE_RESURFACE_SECS recheck, never a wedge escalation. +# within STALE_ESCALATE_SECS + a tick, never lost. A declared wait - either a +# paused: external wait or a verified captain-held transfer, per +# fm-classify-lib.sh's combined predicate - instead gets its own longer +# PAUSE_RESURFACE_SECS recheck, never a wedge escalation. # Crewmates are autonomous, so a delayed stale response does not stall a # healthy crewmate's own progress. # Buffered escalation delivery also has a max-defer alarm: if a digest stays @@ -89,8 +91,9 @@ # kinds. # FM_STALE_ESCALATE_SECS idle seconds before a stale pane escalates # as a possible wedge (default 240) -# FM_PAUSE_RESURFACE_SECS idle seconds before a declared external wait -# re-surfaces as a recheck (default 3600) +# FM_PAUSE_RESURFACE_SECS idle seconds before a declared wait (external +# or captain-held) re-surfaces as a recheck +# (default 3600) # FM_ESCALATE_BATCH_SECS buffer window for batched escalation # digests; 0 = flush immediately (default 90) # FM_HEARTBEAT_SCAN_SECS cadence for the catch-all status scan @@ -372,12 +375,13 @@ classify_stale() { # <window> <state> local win=$1 state=$2 task last seen task=$(window_to_task "$win" "$state") last=$(last_status_line "$state/$task.status") - if [ -n "$last" ] && status_is_paused "$last"; then - # A DECLARED external-wait pause (fm-classify-lib.sh): an idle pane is EXPECTED, - # so this is not a wedge. The caller records a pause marker (long re-surface - # cadence in housekeeping) rather than a wedge stale marker. Cheap: reuses the - # status line already read, no fm-crew-state.sh call, mirroring the daemon's - # existing status-log classification. + if [ -n "$last" ] && status_is_paused_or_captain_held "$last"; then + # A DECLARED external-wait pause or a verified captain-held transfer + # (fm-classify-lib.sh owns which declarations qualify): an idle pane is + # EXPECTED, so this is not a wedge. The caller records a pause marker (long + # re-surface cadence in housekeeping) rather than a wedge stale marker. Cheap: + # reuses the status line already read, no fm-crew-state.sh call, mirroring the + # daemon's existing status-log classification. printf 'pause|paused (awaiting external), rechecked on a long cadence: %s' "$last" return fi @@ -446,10 +450,11 @@ stale_marker_remove() { # <window> <state> rm -f "$state/.subsuper-stale-$key" } -# Pause marker: state/.subsuper-paused-<key> holds the epoch a declared pause was -# first observed idle. Housekeeping ages it against PAUSE_RESURFACE_SECS (much -# longer than a wedge) and re-surfaces the pause once per window. Recording is -# create-if-absent so the timestamp is stable across a churny idle pane (many +# Pause marker: state/.subsuper-paused-<key> holds the epoch a declared wait (a +# paused: external wait or a verified captain-held transfer) was first observed +# idle. Housekeeping ages it against PAUSE_RESURFACE_SECS (much longer than a +# wedge) and re-surfaces the wait once per window. Recording is create-if-absent +# so the timestamp is stable across a churny idle pane (many # distinct stale hashes map to one marker), keeping the cadence hash-immune. pause_marker_record() { # <window> <state> - create if absent local win=$1 state=$2 key marker @@ -471,7 +476,8 @@ clear_pause_tracking() { # <window> <state> watcher_key=$(_stale_key "$win") rm -f "$state/.subsuper-paused-$key" "$state/.subsuper-stale-$key" \ "$state/.paused-$watcher_key" "$state/.paused-rechecked-$watcher_key" "$state/.paused-resurfaced-$watcher_key" \ - "$state/.stale-$watcher_key" "$state/.stale-since-$watcher_key" "$state/.wedge-escalations-$watcher_key" + "$state/.stale-$watcher_key" "$state/.stale-since-$watcher_key" "$state/.wedge-escalations-$watcher_key" \ + "$state/.writing-since-$watcher_key" "$state/.writing-resurfaced-$watcher_key" } reconcile_pause_tracking() { # <window> <state> <last-status-line> @@ -480,7 +486,7 @@ reconcile_pause_tracking() { # <window> <state> <last-status-line> key=$(_stale_key "$task") marker="$state/.subsuper-paused-$key" watcher_key=$(_stale_key "$win") - if status_is_paused "$last"; then + if status_is_paused_or_captain_held "$last"; then stale_marker_remove "$win" "$state" pause_marker_record "$win" "$state" elif [ -e "$marker" ] || [ -e "$state/.paused-$watcher_key" ]; then @@ -498,7 +504,7 @@ migrate_watcher_pause_markers() { # <state> key=$(_stale_key "$task") watcher_key=$(_stale_key "$win") last=$(last_status_line "$state/$task.status") - if status_is_paused "$last" || [ -e "$state/.subsuper-paused-$key" ] || [ -e "$state/.paused-$watcher_key" ]; then + if status_is_paused_or_captain_held "$last" || [ -e "$state/.subsuper-paused-$key" ] || [ -e "$state/.paused-$watcher_key" ]; then reconcile_pause_tracking "$win" "$state" "$last" fi done @@ -953,9 +959,10 @@ _oldest_line_age() { # <buf> -> seconds since the oldest buffered item first ar # Never silently defer forever. # 2) stale recheck: for each pending stale marker past STALE_ESCALATE_SECS, # re-peek the pane; still idle -> escalate (wedge); resumed -> clear marker. -# 2b) pause re-surface: for each declared-pause marker past PAUSE_RESURFACE_SECS, -# re-peek; busy/gone -> clear; still idle + still paused -> escalate a recheck -# digest and reset the window (repeating bounded re-surface, never a wedge). +# 2b) pause re-surface: for each declared-wait marker past PAUSE_RESURFACE_SECS, +# re-peek; busy/gone -> clear; still idle + still declaring the wait -> escalate +# a recheck digest naming which human the wait is on, and reset the window +# (repeating bounded re-surface, never a wedge). # 3) heartbeat scan: every HEARTBEAT_SCAN_SECS, grep state/*.status for a # captain-relevant line the per-wake classifier missed and escalate it. housekeeping() { # <state> @@ -1006,7 +1013,7 @@ housekeeping() { # <state> fi task=$(window_to_task "$win" "$state") last=$(last_status_line "$state/$task.status") - if [ -n "$last" ] && status_is_paused "$last"; then + if [ -n "$last" ] && status_is_paused_or_captain_held "$last"; then reconcile_pause_tracking "$win" "$state" "$last" continue fi @@ -1021,12 +1028,15 @@ housekeeping() { # <state> esac done - # (2b) pause re-surface recheck. A DECLARED external-wait pause idles by design, - # so it is rechecked on a much longer cadence than a wedge (PAUSE_RESURFACE_SECS) - # and never escalated as one - but it MUST re-surface, so a forgotten pause cannot - # rot invisibly. Past the window: busy (resumed) or gone -> drop; still idle and - # still declaring the pause -> escalate a recheck digest and reset the marker so - # the window repeats. + # (2b) pause re-surface recheck. A declared wait idles by design (fm-classify-lib.sh's + # status_is_paused_or_captain_held owns which declarations qualify), so it is + # rechecked on a much longer cadence than a wedge (PAUSE_RESURFACE_SECS) and never + # escalated as one - but it MUST re-surface, so neither a forgotten pause nor a + # forgotten captain hold can rot invisibly. Past the window: busy (resumed) or gone + # -> drop; still idle and still declaring the wait -> escalate a recheck digest and + # reset the marker so the window repeats. The digest names WHICH human the wait is + # on, because the captain is the one reading it: an external dependency for a + # paused: declaration, and the captain themself for a verified hold transfer. pause_secs=${FM_PAUSE_RESURFACE_SECS:-$FM_PAUSE_RESURFACE_SECS_DEFAULT} for marker in "$state"/.subsuper-paused-*; do [ -e "$marker" ] || continue @@ -1037,7 +1047,7 @@ housekeeping() { # <state> fi task=$(window_to_task "$win" "$state") last=$(last_status_line "$state/$task.status") - if [ -z "$last" ] || ! status_is_paused "$last"; then + if [ -z "$last" ] || ! status_is_paused_or_captain_held "$last"; then reconcile_pause_tracking "$win" "$state" "$last" continue fi @@ -1049,7 +1059,10 @@ housekeeping() { # <state> 2) rm -f "$marker" ;; *) last=$(last_status_line "$state/$task.status") - if [ -n "$last" ] && status_is_paused "$last"; then + if [ -n "$last" ] && status_is_captain_held "$last"; then + escalate_add "$state" "captain-held ${age}s (awaiting the captain, answer the held decision or release the hold): $win" + _now > "$marker" + elif [ -n "$last" ] && status_is_paused "$last"; then escalate_add "$state" "paused ${age}s (awaiting external, recheck whether the wait still holds): $win" _now > "$marker" else @@ -1236,10 +1249,10 @@ handle_wake() { # <reason> <state> [ "${FM_ESCALATE_BATCH_SECS:-$ESCALATE_BATCH_SECS_DEFAULT}" -le 0 ] && { escalate_flush "$state" || true; } ;; pause) - # Declared external-wait pause: record a pause marker (long re-surface - # cadence in housekeeping) and drop any wedge stale marker, so a pane that - # transitioned working->paused is not still wedge-aged. Only stale produces - # this action. + # Declared wait, an external-wait pause or a verified captain-held transfer: + # record a pause marker (long re-surface cadence in housekeeping) and drop any + # wedge stale marker, so a pane that transitioned working->declared-wait is not + # still wedge-aged. Only stale produces this action. if [ "$kind" = "stale" ]; then stale_marker_remove "$arg" "$state" pause_marker_record "$arg" "$state" diff --git a/bin/fm-teardown.sh b/bin/fm-teardown.sh index d152745d11d..791a609fcf7 100755 --- a/bin/fm-teardown.sh +++ b/bin/fm-teardown.sh @@ -2413,9 +2413,9 @@ if [ "$KIND" = scout ] && [ "$FORCE" != "--force" ]; then exit 1 fi if ! FM_HOME="$FM_HOME" FM_STATE_OVERRIDE="$STATE" FM_DATA_OVERRIDE="$DATA" \ - FM_CONFIG_OVERRIDE="$CONFIG" "$SCRIPT_DIR/fm-decision-hold.sh" verify "$ID" >/dev/null; then - echo "REFUSED: scout task $ID has not passed the unresolved-decision completion gate." >&2 - echo "Inventory its report and any visual review through bin/fm-decision-hold.sh before teardown." >&2 + FM_CONFIG_OVERRIDE="$CONFIG" "$SCRIPT_DIR/fm-captain-hold.sh" verify "$ID" >/dev/null; then + echo "REFUSED: scout task $ID has not passed the captain-call completion gate." >&2 + echo "Inventory its report and any visual review through bin/fm-captain-hold.sh before teardown." >&2 exit 1 fi fi @@ -2442,6 +2442,22 @@ if [ "$FORCE" != "--force" ] \ fi fi +# Non-blocking: a delivered public loop is not a teardown refusal (guard-work +# already passed), but tearing down a ship whose PR merged while a loop is still +# open with nothing owed is the moment the drop is detectable. +if [ "$KIND" = ship ] && [ -n "$PR_URL" ] \ + && [ -n "$PUBLIC_FOLLOWUP_STATE" ] \ + && [ "${PUBLIC_FOLLOWUP_RELAY_ACTIVE:-0}" = 1 ] \ + && fm_pf_has_delivered_open_loops "$PUBLIC_FOLLOWUP_STATE"; then + echo "warning: an open public loop with nothing owed is still recorded in the consent-holding home while cleaning up ship task $ID. Hand it on with bin/fm-public-followup.sh rechain or close it with retire --reason." >&2 +fi + +# Non-blocking: the legacy Relay link is not guarded as a refusal. +X_REQUEST=$(grep '^x_request=' "$META" 2>/dev/null | tail -1 | cut -d= -f2- || true) +if [ -n "$X_REQUEST" ]; then + echo "warning: task $ID still carries an unreconciled Relay request link ($X_REQUEST) on its task record." >&2 +fi + if [ "$BACKEND" = orca ] && [ "$KIND" != scout ] && [ "$KIND" != secondmate ] && [ "$FORCE" != "--force" ]; then if ! inspectable_git_worktree "$WT"; then echo "REFUSED: Orca ship task $ID has no inspectable git worktree at ${WT:-<missing>}." >&2 diff --git a/bin/fm-test-isolation-proof.sh b/bin/fm-test-isolation-proof.sh index 4aceb1a1041..137aff8b268 100755 --- a/bin/fm-test-isolation-proof.sh +++ b/bin/fm-test-isolation-proof.sh @@ -153,11 +153,11 @@ list_parallel_candidates() { tests/fm-arm-pretool-check.test.sh tests/fm-backend-herdr.test.sh tests/fm-brief.test.sh +tests/fm-captain-hold-lifecycle.test.sh tests/fm-cd-pretool-check.test.sh tests/fm-composer-ghost.test.sh tests/fm-composer-lib.test.sh tests/fm-crew-state.test.sh -tests/fm-decision-hold-lifecycle.test.sh tests/fm-ensure-agents-md.test.sh tests/fm-grok-harness.test.sh tests/fm-herdr-lab.test.sh diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh index 8a5a310c1ec..69bd7ff978c 100755 --- a/bin/fm-test-run.sh +++ b/bin/fm-test-run.sh @@ -138,10 +138,11 @@ family_for_basename() { fm-calm-pi-extension.test.sh|fm-cd-pretool-check.test.sh|\ fm-classify-decision-key.test.sh|\ fm-composer-ghost.test.sh|fm-composer-lib.test.sh|\ - fm-crew-state.test.sh|fm-decision-hold-lifecycle.test.sh|\ + fm-crew-state.test.sh|fm-captain-hold-lifecycle.test.sh|\ fm-documentation-audiences.test.sh|fm-ensure-agents-md.test.sh|fm-grok-harness.test.sh|\ fm-kimi-harness.test.sh|fm-muse-harness.test.sh|fm-herdr-lab.test.sh|fm-lint.test.sh|\ fm-lint-workflows.test.sh|\ + fm-no-mistakes-attestation.test.sh|\ fm-operational-input.test.sh|fm-pi-primary-types.test.sh|\ fm-send-popup-settle.test.sh|fm-send-settle.test.sh|\ fm-subagent-pretool-check.test.sh|\ @@ -155,7 +156,9 @@ family_for_basename() { fm-session-lock-ancestry.test.sh|fm-cursor-primary.test.sh|\ fm-supervision-events.test.sh|fm-turnend-guard.test.sh|fm-wake-daemon-lifecycle-e2e.test.sh|\ fm-wake-drain-unread-status.test.sh|\ - fm-wake-queue.test.sh|fm-watch-arm.test.sh|fm-watch-checkpoint.test.sh|fm-watch-triage.test.sh|\ + fm-tool-update-check.test.sh|\ + fm-wake-queue.test.sh|fm-watch-arm.test.sh|fm-watch-checkpoint.test.sh|fm-watch-recovery-loop.test.sh|\ + fm-watch-triage.test.sh|\ fm-watcher-lock.test.sh|fm-inactive-reconcile.test.sh) printf '%s\n' watcher-wake-lock ;; @@ -281,11 +284,11 @@ list_proven_isolated() { tests/fm-arm-pretool-check.test.sh tests/fm-backend-herdr.test.sh tests/fm-brief.test.sh +tests/fm-captain-hold-lifecycle.test.sh tests/fm-cd-pretool-check.test.sh tests/fm-composer-ghost.test.sh tests/fm-composer-lib.test.sh tests/fm-crew-state.test.sh -tests/fm-decision-hold-lifecycle.test.sh tests/fm-ensure-agents-md.test.sh tests/fm-grok-harness.test.sh tests/fm-herdr-lab.test.sh @@ -312,7 +315,7 @@ list_portable_parallel_1() { cat <<'EOF' tests/fm-x-mode.test.sh tests/fm-cd-pretool-check.test.sh -tests/fm-decision-hold-lifecycle.test.sh +tests/fm-captain-hold-lifecycle.test.sh tests/fm-test-run.test.sh tests/fm-composer-ghost.test.sh tests/fm-grok-harness.test.sh @@ -379,77 +382,124 @@ list_portable_serial() { # procedure. portable_serial_weight_hints() { cat <<'EOF' -tests/fm-afk-inject-e2e.test.sh 34019 -tests/fm-afk-pi-herdr-return-e2e.test.sh 42 -tests/fm-afk-return.test.sh 1105 -tests/fm-ask-user-authority.test.sh 68 -tests/fm-backend-cmux-smoke.test.sh 29 -tests/fm-backend-cmux.test.sh 2349 +tests/fm-afk-inject-e2e.test.sh 35900 +tests/fm-afk-pi-herdr-return-e2e.test.sh 66 +tests/fm-afk-return.test.sh 3974 +tests/fm-ask-user-authority.test.sh 83 +tests/fm-backend-cmux-smoke.test.sh 30 +tests/fm-backend-cmux.test.sh 3351 tests/fm-backend-herdr-focus-flash-e2e.test.sh 21 -tests/fm-backend-orca.test.sh 12041 -tests/fm-backend-tmux-smoke.test.sh 314 -tests/fm-backend-zellij-smoke.test.sh 21 -tests/fm-backend-zellij.test.sh 4225 -tests/fm-backend.test.sh 16370 -tests/fm-backlog-handoff.test.sh 2786 -tests/fm-bearings-snapshot.test.sh 60103 -tests/fm-bootstrap.test.sh 21912 -tests/fm-busy-adapter-wiring.test.sh 13962 -tests/fm-busy-state.test.sh 607 -tests/fm-calm-pi-extension.test.sh 203 -tests/fm-claude-stop-autoarm-live-e2e.test.sh 19 -tests/fm-claude-stop-autoarm.test.sh 60521 +tests/fm-backend-orca.test.sh 14681 +tests/fm-backend-tmux-smoke.test.sh 361 +tests/fm-backend-zellij-smoke.test.sh 22 +tests/fm-backend-zellij.test.sh 8297 +tests/fm-backend.test.sh 17169 +tests/fm-backlog-handoff.test.sh 4157 +tests/fm-bearings-board.test.sh 3385 +tests/fm-bearings-snapshot.test.sh 68659 +tests/fm-bootstrap.test.sh 38417 +tests/fm-busy-adapter-wiring.test.sh 14880 +tests/fm-busy-state.test.sh 714 +tests/fm-calm-pi-extension.test.sh 464 +tests/fm-classify-decision-key.test.sh 928 +tests/fm-claude-stop-autoarm-live-e2e.test.sh 30 +tests/fm-claude-stop-autoarm.test.sh 60633 +tests/fm-cmux-claude-composer-live-e2e.test.sh 20 tests/fm-codex-continuity-live-e2e.test.sh 19 -tests/fm-daemon.test.sh 15140 -tests/fm-documentation-audiences.test.sh 572 -tests/fm-fleet-snapshot-view.test.sh 5902 -tests/fm-fleet-sync.test.sh 16417 -tests/fm-gate-refuse.test.sh 2839 -tests/fm-gitignore-config.test.sh 28 -tests/fm-gotmp.test.sh 308 +tests/fm-composer-matrix-live-e2e.test.sh 21 +tests/fm-control-relaunch.test.sh 31881 +tests/fm-control.test.sh 36712 +tests/fm-cursor-harness.test.sh 30071 +tests/fm-cursor-primary-live-e2e.test.sh 20 +tests/fm-cursor-primary.test.sh 52324 +tests/fm-daemon.test.sh 25834 +tests/fm-documentation-audiences.test.sh 642 +tests/fm-fleet-snapshot-view.test.sh 6995 +tests/fm-fleet-sync.test.sh 20194 +tests/fm-gate-refuse.test.sh 4071 +tests/fm-gitignore-config.test.sh 63 +tests/fm-gotmp.test.sh 762 tests/fm-grok-continuity-live-e2e.test.sh 19 -tests/fm-grok-stop-live-e2e.test.sh 19 -tests/fm-guard-stale-banner.test.sh 2917 -tests/fm-herdr-session-cleanup.test.sh 4802 -tests/fm-kimi-harness.test.sh 12590 -tests/fm-opencode-primary-live-e2e.test.sh 18 -tests/fm-operational-input.test.sh 184 -tests/fm-pending-reply.test.sh 7328 -tests/fm-pi-primary-live-e2e.test.sh 19 -tests/fm-pi-watch-extension.test.sh 16386 -tests/fm-pr-check-security.test.sh 199573 -tests/fm-procevent.test.sh 42789 -tests/fm-public-followup.test.sh 23365 -tests/fm-quota-array-dispatch-live-e2e.test.sh 19 -tests/fm-secondmate-harness.test.sh 87895 -tests/fm-secondmate-lifecycle-e2e.test.sh 4929 -tests/fm-secondmate-liveness.test.sh 12553 -tests/fm-secondmate-safety.test.sh 24432 -tests/fm-secondmate-sync.test.sh 12289 -tests/fm-send-secondmate-marker-herdr-e2e.test.sh 27 -tests/fm-send-secondmate-marker.test.sh 2136 -tests/fm-session-start.test.sh 37289 -tests/fm-sessionstart-nudge.test.sh 264 -tests/fm-sessionstart-instruction-refresh-live-e2e.test.sh 19 -tests/fm-shared-captain-inheritance.test.sh 3506 -tests/fm-spawn-dispatch-profile.test.sh 41351 -tests/fm-spawn-worktree-settle.test.sh 4598 -tests/fm-startup-memory-budget.test.sh 4260 -tests/fm-subagent-pretool-check.test.sh 901 -tests/fm-supervision-events.test.sh 413 -tests/fm-tangle-guard.test.sh 7230 -tests/fm-teardown-endpoint-safety.test.sh 1073 -tests/fm-teardown.test.sh 23237 -tests/fm-test-isolation-proof.test.sh 326 -tests/fm-turnend-guard.test.sh 5986 -tests/fm-update.test.sh 1894 -tests/fm-vendor-auth-probe.test.sh 42796 -tests/fm-wake-daemon-lifecycle-e2e.test.sh 4284 -tests/fm-wake-drain-unread-status.test.sh 4000 -tests/fm-wake-queue.test.sh 22787 -tests/fm-watch-checkpoint.test.sh 3943 -tests/fm-watch-triage.test.sh 113051 -tests/fm-watcher-lock.test.sh 98342 +tests/fm-grok-stop-live-e2e.test.sh 21 +tests/fm-guard-stale-banner.test.sh 11280 +tests/fm-harness-liveness-drift-live-e2e.test.sh 19 +tests/fm-herdr-session-cleanup.test.sh 14120 +tests/fm-herdr-submit-confirm-live-e2e.test.sh 20 +tests/fm-herdr-version-floor-live-e2e.test.sh 20 +tests/fm-inactive-reconcile.test.sh 41671 +tests/fm-kimi-harness.test.sh 15092 +tests/fm-lint-workflows.test.sh 744 +tests/fm-muse-harness.test.sh 27414 +tests/fm-muse-signals-live-e2e.test.sh 21 +tests/fm-no-mistakes-attestation.test.sh 276 +tests/fm-on.test.sh 8602 +tests/fm-opencode-primary-live-e2e.test.sh 22 +tests/fm-operational-input.test.sh 246 +tests/fm-peek-remote.test.sh 848 +tests/fm-pending-reply.test.sh 19488 +tests/fm-pi-primary-live-e2e.test.sh 41 +tests/fm-pi-watch-extension.test.sh 17979 +tests/fm-pr-check-security.test.sh 250417 +tests/fm-procevent-when.test.sh 15249 +tests/fm-procevent.test.sh 53142 +tests/fm-project-origin.test.sh 105 +tests/fm-public-followup.test.sh 36301 +tests/fm-quota-array-dispatch-live-e2e.test.sh 18 +tests/fm-remote-backlog-handoff.test.sh 20389 +tests/fm-remote-doctor.test.sh 4705 +tests/fm-remote-entrypoint.test.sh 98 +tests/fm-remote-job-orphan-reap.test.sh 2903 +tests/fm-remote-job.test.sh 48068 +tests/fm-remote-reply.test.sh 40906 +tests/fm-remote-secondmate-lifecycle-e2e.test.sh 170240 +tests/fm-remote-secondmate-parent-binding.test.sh 13064 +tests/fm-remote-secondmate-trace-context.test.sh 39927 +tests/fm-secondmate-harness.test.sh 123471 +tests/fm-secondmate-lifecycle-e2e.test.sh 6539 +tests/fm-secondmate-liveness.test.sh 16365 +tests/fm-secondmate-safety.test.sh 49011 +tests/fm-secondmate-sync.test.sh 29236 +tests/fm-send-remote-delivery.test.sh 4892 +tests/fm-send-resolve-key.test.sh 13450 +tests/fm-send-secondmate-marker-herdr-e2e.test.sh 45 +tests/fm-send-secondmate-marker.test.sh 4439 +tests/fm-session-lock-ancestry.test.sh 1205 +tests/fm-session-start.test.sh 144836 +tests/fm-sessionstart-hook-live-e2e.test.sh 21 +tests/fm-sessionstart-instruction-refresh-live-e2e.test.sh 21 +tests/fm-sessionstart-nudge.test.sh 26684 +tests/fm-shared-captain-inheritance.test.sh 10672 +tests/fm-spawn-dispatch-profile.test.sh 57765 +tests/fm-spawn-pool-base-freshen.test.sh 13257 +tests/fm-spawn-worktree-settle.test.sh 4828 +tests/fm-startup-memory-budget.test.sh 6550 +tests/fm-startup-network.test.sh 48888 +tests/fm-stow-cascade.test.sh 2986 +tests/fm-subagent-pretool-check.test.sh 1066 +tests/fm-supervision-events.test.sh 1431 +tests/fm-tangle-guard.test.sh 8364 +tests/fm-task-delivery.test.sh 2414 +tests/fm-teardown-endpoint-safety.test.sh 7295 +tests/fm-teardown.test.sh 87400 +tests/fm-test-fixture-cleanup.test.sh 532 +tests/fm-test-isolation-proof.test.sh 451 +tests/fm-tmux-agent-liveness.test.sh 4065 +tests/fm-tool-update-check.test.sh 12846 +tests/fm-trace-context-lib.test.sh 194 +tests/fm-trace-context-spawn.test.sh 35325 +tests/fm-turnend-guard.test.sh 34915 +tests/fm-update.test.sh 5280 +tests/fm-vendor-auth-probe.test.sh 43243 +tests/fm-wake-daemon-lifecycle-e2e.test.sh 6219 +tests/fm-wake-drain-open-decisions-cursor.test.sh 17357 +tests/fm-wake-drain-open-decisions.test.sh 11300 +tests/fm-wake-drain-unread-status.test.sh 25214 +tests/fm-wake-queue.test.sh 30887 +tests/fm-watch-arm.test.sh 53598 +tests/fm-watch-checkpoint.test.sh 5293 +tests/fm-watch-recovery-loop.test.sh 58721 +tests/fm-watch-triage.test.sh 142409 +tests/fm-watcher-lock.test.sh 54364 EOF } @@ -923,12 +973,13 @@ families_for_changed_path() { ;; bin/fm-timeout-lib.sh) # The shared hard bound: session start's runtime bound, the fleet/bearings - # snapshots, the vendor auth probe, and the stow cascade's per-home step - # all depend on it. + # snapshots, the vendor auth probe, the stow cascade's per-home step, and + # the wedge detector's worktree write probe all depend on it. printf '%s\n' session-bootstrap printf '%s\n' snapshot-bearings printf '%s\n' pure-contract-unit printf '%s\n' secondmate + printf '%s\n' watcher-wake-lock ;; bin/fm-pr-*|bin/fm-merge-local.sh|bin/fm-teardown.sh|bin/fm-review-diff.sh|\ bin/fm-x-*|bin/fm-check*) @@ -974,7 +1025,7 @@ families_for_changed_path() { bin/fm-lint.sh|bin/fm-lint-workflows.sh|bin/fm-install-shellcheck.sh|\ bin/fm-install-actionlint.sh|\ bin/fm-brief.sh|bin/fm-ensure-agents-md.sh|bin/fm-crew-state.sh|\ - bin/fm-decision-hold.sh|bin/fm-supervision*|bin/fm-transition-lib.sh|\ + bin/fm-captain-hold.sh|bin/fm-decision-hold.sh|bin/fm-supervision*|bin/fm-transition-lib.sh|\ bin/fm-tmux-lib.sh|bin/fm-marker-lib.sh|bin/fm-operational-input.sh|bin/fm-tasks-axi-lib.sh|\ bin/fm-vendor-auth-probe.sh|\ bin/fm-primary-scope-lib.sh|bin/fm-project-mode.sh|bin/fm-promote.sh|\ diff --git a/bin/fm-tool-update-check.sh b/bin/fm-tool-update-check.sh new file mode 100755 index 00000000000..bbaf7d25245 --- /dev/null +++ b/bin/fm-tool-update-check.sh @@ -0,0 +1,898 @@ +#!/usr/bin/env bash +# fm-tool-update-check.sh - report watched tooling that has an update available, +# and tooling whose update is installed but not in effect. +# +# Usage: +# fm-tool-update-check.sh [check] +# fm-tool-update-check.sh arm +# fm-tool-update-check.sh disarm +# fm-tool-update-check.sh --help +# +# `check` prints one line when something needs attention and prints nothing at +# all otherwise, so it composes with the existing watcher state-check contract +# instead of needing a schedule of its own. `arm` writes +# state/tool-updates.check.sh and binds its bytes with fm-check-register.sh, so +# the watcher dispatches it on its normal FM_CHECK_INTERVAL cadence and turns +# its one line into a `check:` wake. `disarm` removes the shim, its trust +# binding, and the report record. +# +# Two conditions are reported, and they are deliberately distinct: +# +# "<tool> update available" a newer version exists at the update source. +# "<tool> update not in effect" a newer copy is installed on this host, but +# PATH still resolves an older one. +# +# The second condition is the reason this script exists. A tool that +# self-installs into ~/.local/bin while a version manager keeps its own older +# copy earlier on PATH looks fully up to date to anything that asks only "is a +# newer version published". So PATH skew is measured, never inferred: every +# executable copy on PATH is asked for its own version, and those answers are +# compared. A directory name is never read as a version, because a version +# manager's "latest" directory can hold an older build. A copy that will not +# report a version is reported as a check failure rather than assumed current. +# +# What this script never does: it reports, and it repairs nothing. It does not +# install, update, uninstall, reorder PATH, or touch any version manager's +# configuration, and it never fetches into a watched git repository. Every git +# probe is read-only (rev-parse, symbolic-ref, ls-remote, cat-file, merge-base, +# rev-list), so a watched project is never mutated. +# +# The watched tools live in config/watched-tools.json, which is local and +# gitignored, and is never propagated to another home. Adding a tool is a config +# edit, never a code change. docs/configuration.md owns that schema. +# +# Probing costs real time, so `check` runs its probes at most once per +# FM_TOOL_UPDATE_INTERVAL (default 900, 0 disables the gate, otherwise 60..86400) +# and stays silent in between. Each probe is bounded by +# FM_TOOL_UPDATE_PROBE_SECS (default 5, valid 1..30) and a whole sweep by +# FM_TOOL_UPDATE_BUDGET_SECS (default 20, valid 1..120). +# +# The sweep has to finish inside the watcher's own per check bound, because a run +# the watcher kills prints nothing and writes no record, so it would repeat that +# silence on every poll. That coupling is enforced rather than assumed: a budget +# larger than FM_CHECK_TIMEOUT (default 30, read from this check's own +# environment because the watcher runs it as a direct child) allows is cut down +# to what fits, and the cut is reported in the report line so the operator sees +# it. A budget that cannot be read as a whole number from 1 to 120 is still +# refused outright. +# +# The report record state/.tool-updates is written only when a sweep runs to its +# end, and it carries the whole finding set the last report was made from, +# uncut, so the same pending update is reported once rather than on every poll +# while a new finding that lands past the one-line cut is still news. A sweep +# killed part way through leaves no record and is retried, instead of +# suppressing its finding. +set -u +export LC_ALL=C +# A watched git remote must never stop to ask for credentials; an unauthenticated +# probe has to fail inside its bound instead of waiting for an answer. +export GIT_TERMINAL_PROMPT=0 + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}}" +STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" +CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}/watched-tools.json" +RECORD="$STATE/.tool-updates" +CHECK_ID=tool-updates +CHECK_SHIM="$STATE/$CHECK_ID.check.sh" +CHECK_TRUST="$STATE/$CHECK_ID.check-trust" +REGISTER_BIN="$SCRIPT_DIR/fm-check-register.sh" +RECORD_SCHEMA=fm-tool-updates-v1 +# Wider than the digest default because one finding names two absolute paths and +# their two versions, and several tools can report in the same sweep. +MAX_LINE=1000 + +# shellcheck source=bin/fm-timeout-lib.sh +. "$SCRIPT_DIR/fm-timeout-lib.sh" +# shellcheck source=bin/fm-pr-lib.sh +. "$SCRIPT_DIR/fm-pr-lib.sh" +# shellcheck source=bin/fm-line-cap-lib.sh +. "$SCRIPT_DIR/fm-line-cap-lib.sh" +# shellcheck source=bin/fm-check-lib.sh +. "$SCRIPT_DIR/fm-check-lib.sh" + +usage() { + cat <<'EOF' +Usage: + fm-tool-update-check.sh [check] report watched tools needing attention (silent when current) + fm-tool-update-check.sh arm write and register state/tool-updates.check.sh + fm-tool-update-check.sh disarm remove the check shim, its trust binding, and the record + fm-tool-update-check.sh --help print this help + +Watched tools are read from config/watched-tools.json (local, gitignored). +See docs/configuration.md for the schema and docs/examples/watched-tools.json for a starting point. +EOF +} + +die_usage() { + printf 'fm-tool-update-check: %s\n' "$1" >&2 + usage >&2 + exit 2 +} + +INTERVAL=${FM_TOOL_UPDATE_INTERVAL:-900} +case "$INTERVAL" in + ''|*[!0-9]*) + printf 'fm-tool-update-check: FM_TOOL_UPDATE_INTERVAL must be 0 or a whole number from 60 to 86400\n' >&2 + exit 2 + ;; +esac +if [ "$INTERVAL" -ne 0 ] && { [ "$INTERVAL" -lt 60 ] || [ "$INTERVAL" -gt 86400 ]; }; then + printf 'fm-tool-update-check: FM_TOOL_UPDATE_INTERVAL must be 0 or a whole number from 60 to 86400\n' >&2 + exit 2 +fi + +PROBE_SECS=${FM_TOOL_UPDATE_PROBE_SECS:-5} +case "$PROBE_SECS" in + ''|*[!0-9]*|0) + printf 'fm-tool-update-check: FM_TOOL_UPDATE_PROBE_SECS must be a whole number from 1 to 30\n' >&2 + exit 2 + ;; +esac +if [ "$PROBE_SECS" -gt 30 ]; then + printf 'fm-tool-update-check: FM_TOOL_UPDATE_PROBE_SECS must be a whole number from 1 to 30\n' >&2 + exit 2 +fi + +BUDGET_SECS=${FM_TOOL_UPDATE_BUDGET_SECS:-20} +case "$BUDGET_SECS" in + ''|*[!0-9]*|0) + printf 'fm-tool-update-check: FM_TOOL_UPDATE_BUDGET_SECS must be a whole number from 1 to 120\n' >&2 + exit 2 + ;; +esac +if [ "$BUDGET_SECS" -gt 120 ]; then + printf 'fm-tool-update-check: FM_TOOL_UPDATE_BUDGET_SECS must be a whole number from 1 to 120\n' >&2 + exit 2 +fi + +# The smallest bound a probe can be given, because fm_run_timed treats a +# non-positive bound as no bound. +PROBE_MIN_SECS=1 +# Both clocks here count whole seconds, so a probe can start when the arithmetic +# says a second is left while almost none of it really is, and it still gets a +# full bound. +CLOCK_ROUNDING_SECS=1 +# fm_run_timed asks its runner for -k 1, so a probe that does not stop on TERM is +# only killed a second after its bound. +KILL_GRACE_SECS=1 + +# The watcher's per check bound, read from this check's own environment. The +# watcher runs the check as a direct child, so an operator who raised it is seen +# here too, and when it is unset both sides resolve the same default. +CHECK_TIMEOUT=${FM_CHECK_TIMEOUT:-30} +case "$CHECK_TIMEOUT" in + ''|*[!0-9]*|0) CHECK_TIMEOUT=30 ;; +esac +# The last probe of a sweep can end this far past the deadline, so that is what +# the budget has to leave the watcher's own bound. +BUDGET_MAX=$((CHECK_TIMEOUT - PROBE_MIN_SECS - CLOCK_ROUNDING_SECS - KILL_GRACE_SECS)) +[ "$BUDGET_MAX" -ge 1 ] || BUDGET_MAX=1 +# Cut rather than refuse. A refusal is reported once and then suppressed by the +# no-nag gate, which leaves the detector dead and quiet, and a check that goes +# silent is worse than a check that reports something awkward. +BUDGET_CUT_FROM= +if [ "$BUDGET_SECS" -gt "$BUDGET_MAX" ]; then + BUDGET_CUT_FROM=$BUDGET_SECS + BUDGET_SECS=$BUDGET_MAX +fi + +# --- small helpers ---------------------------------------------------------- + +# The record epoch is overridable so a test can drive the cadence gate; the +# sweep budget always uses real time so a frozen epoch cannot disable it. +record_epoch_now() { + case "${FM_TOOL_UPDATE_NOW:-}" in + ''|*[!0-9]*) date +%s ;; + *) printf '%s\n' "$FM_TOOL_UPDATE_NOW" ;; + esac +} + +real_epoch() { date +%s; } + +FINDINGS= +DEADLINE=0 +INCOMPLETE_REPORTED=0 + +# Each finding is flattened to a single line here, because the whole report must +# stay one line for the wake record. +emit() { + local text + text=$(printf '%s' "$1" | tr '\t\r\n' ' ') + if [ -z "$FINDINGS" ]; then + FINDINGS=$text + else + FINDINGS="$FINDINGS; $text" + fi +} + +budget_exhausted() { + [ "$(real_epoch)" -ge "$DEADLINE" ] +} + +# True while the sweep budget still has room for another probe. When it does not, +# it records once which tool the sweep did not finish, so a sweep that cannot +# finish says so rather than being killed by the watcher with nothing printed. +budget_allows() { + local name=$1 + budget_exhausted || return 0 + if [ "$INCOMPLETE_REPORTED" -eq 0 ]; then + INCOMPLETE_REPORTED=1 + emit "check incomplete: the time budget ran out before $name" + fi + return 1 +} + +# The bound for one probe: the probe bound, cut down to whatever the sweep +# budget has left, so no probe can run past the end of the sweep. Never below +# PROBE_MIN_SECS, because fm_run_timed treats a non-positive bound as no bound. +probe_bound() { + local left + left=$((DEADLINE - $(real_epoch))) + if [ "$left" -lt "$PROBE_MIN_SECS" ]; then + printf '%s\n' "$PROBE_MIN_SECS" + elif [ "$left" -lt "$PROBE_SECS" ]; then + printf '%s\n' "$left" + else + printf '%s\n' "$PROBE_SECS" + fi +} + +# First dotted number in the text, so "herdr 0.8.2" and "v1.46.0" both work. +parse_version() { + printf '%s' "$1" | grep -oE '[0-9]+(\.[0-9]+)+' | head -n 1 +} + +# version_newer <a> <b>: true when version a is numerically newer than b. +version_newer() { + local a=$1 b=$2 i left right + local -a ap bp + IFS=. read -r -a ap <<< "$a" + IFS=. read -r -a bp <<< "$b" + i=0 + while [ "$i" -lt "${#ap[@]}" ] || [ "$i" -lt "${#bp[@]}" ]; do + left=$((10#${ap[i]:-0})) + right=$((10#${bp[i]:-0})) + if [ "$left" -gt "$right" ]; then + return 0 + elif [ "$left" -lt "$right" ]; then + return 1 + fi + i=$((i + 1)) + done + return 1 +} + +commit_phrase() { + if [ "$1" = 1 ]; then + printf '1 commit\n' + else + printf '%s commits\n' "$1" + fi +} + +# --- watched tool registry -------------------------------------------------- + +CONFIG_PROBLEM= + +# jq can check that an announce_pattern is a non-empty single-line string, but +# only grep can say whether it compiles as an extended regular expression. A +# pattern grep refuses would silently disable that tool's update source, which is +# the exact failure this script exists to prevent. +announce_pattern_usable() { + local pattern=$1 status + printf '%s' '' | grep -qE -- "$pattern" 2>/dev/null + status=$? + [ "$status" -le 1 ] +} + +# Deliberately separate from config_validate, and asked only by arm. Arming is a +# deliberate operator action that should fail loudly, but a sweep must not treat +# one tool's unusable pattern as a reason to stop watching every other tool: that +# would let a one character typo turn the PATH skew detector off. So `check` +# reports this per tool instead, in command_findings. +config_announce_patterns_usable() { + local name announce + while IFS=$FIELD_SEP read -r name _ _ announce _; do + [ -n "$announce" ] || continue + if ! announce_pattern_usable "$announce"; then + CONFIG_PROBLEM="tool $name announce_pattern is not a usable extended regular expression" + return 1 + fi + done < <(config_records) + return 0 +} + +config_validate() { + local problem status + if ! command -v jq >/dev/null 2>&1; then + CONFIG_PROBLEM='jq is required to read the watched tool registry' + return 1 + fi + problem=$(jq -r ' + def tool_problem($t): + if ($t | type) != "object" then "every entry in tools must be an object" + elif ($t.name | type) != "string" or ($t.name | length) == 0 then "every tool needs a non-empty name" + elif ($t.name | test("^[A-Za-z0-9._+-]+$") | not) then "tool name \($t.name) may use only letters, digits, dot, underscore, plus, and dash" + elif ($t | has("command") | not) and ($t | has("git") | not) then "tool \($t.name) needs command, git, or both" + elif ($t | has("command")) and (($t.command | type) != "string" or ($t.command | test("^[A-Za-z0-9._+-]+$") | not)) then "tool \($t.name) command must be a bare executable name" + elif ($t | has("version_args")) and (($t.version_args | type) != "array" or ($t.version_args | length) == 0) then "tool \($t.name) version_args must be a non-empty array" + elif ($t | has("version_args")) and ([$t.version_args[] | select((type != "string") or (test("^[A-Za-z0-9._=+/:-]+$") | not))] | length) > 0 then "tool \($t.name) version_args must be simple flag strings without spaces" + elif ($t | has("announce_pattern")) and (($t.announce_pattern | type) != "string" or ($t.announce_pattern | length) == 0 or ($t.announce_pattern | test("[[:cntrl:]]"))) then "tool \($t.name) announce_pattern must be a non-empty single-line string" + elif ($t | has("announce_pattern")) and (($t | has("command")) | not) then "tool \($t.name) announce_pattern needs command" + elif ($t | has("announce_args")) and (($t.announce_args | type) != "array" or ($t.announce_args | length) == 0) then "tool \($t.name) announce_args must be a non-empty array" + elif ($t | has("announce_args")) and ([$t.announce_args[] | select((type != "string") or (test("^[A-Za-z0-9._=+/:-]+$") | not))] | length) > 0 then "tool \($t.name) announce_args must be simple flag strings without spaces" + elif ($t | has("announce_args")) and (($t | has("announce_pattern")) | not) then "tool \($t.name) announce_args needs announce_pattern" + elif ($t | has("git")) and (($t.git | type) != "object") then "tool \($t.name) git must be an object" + elif ($t | has("git")) and (($t.git.repo | type) != "string" or ($t.git.repo | startswith("/") | not) or ($t.git.repo | test("[[:cntrl:]]"))) then "tool \($t.name) git.repo must be an absolute path on one line" + elif ($t | has("git")) and ($t.git | has("remote")) and (($t.git.remote | type) != "string" or ($t.git.remote | test("^[A-Za-z0-9._-]+$") | not)) then "tool \($t.name) git.remote must be a simple remote name" + elif ($t | has("git")) and ($t.git | has("branch")) and (($t.git.branch | type) != "string" or ($t.git.branch | test("^[A-Za-z0-9._/-]+$") | not)) then "tool \($t.name) git.branch must be a simple branch name" + else empty + end; + def problems: + if type != "object" then ["the top level must be an object"] + elif (.tools | type) != "array" then ["tools must be an array"] + elif (.tools | length) == 0 then ["tools must list at least one tool"] + else + [.tools[] | tool_problem(.)] + + (if ([.tools[].name] | unique | length) != (.tools | length) then ["tool names must be unique"] else [] end) + end; + problems | .[0] // "ok" + ' "$CONFIG" 2>/dev/null) + status=$? + if [ "$status" -ne 0 ] || [ -z "$problem" ]; then + CONFIG_PROBLEM='the watched tool registry is not valid JSON' + return 1 + fi + if [ "$problem" != ok ]; then + CONFIG_PROBLEM=$problem + return 1 + fi + CONFIG_PROBLEM= + return 0 +} + +# One record per tool, in config order. Fields are joined with the unit +# separator rather than a tab, because tab is IFS whitespace and `read` would +# collapse the empty fields that an optional key leaves behind. +FIELD_SEP=$(printf '\037') + +config_records() { + jq -r ' + .tools[] | [ + .name, + (.command // ""), + ((.version_args // ["--version"]) | join(" ")), + (.announce_pattern // ""), + ((.announce_args // .version_args // ["--version"]) | join(" ")), + (.git.repo // ""), + (.git.remote // "origin"), + (.git.branch // "") + ] | join("\u001f") + ' "$CONFIG" 2>/dev/null +} + +# --- PATH probes ------------------------------------------------------------ + +# Every executable copy of <command> on PATH, in PATH order, deduplicated by +# device and inode so one copy reached through two PATH entries is not read as +# two installs. +path_hits() { + local command_name=$1 dir candidate identity seen='' + while IFS= read -r dir; do + [ -n "$dir" ] || continue + candidate="$dir/$command_name" + [ -f "$candidate" ] && [ -x "$candidate" ] || continue + identity=$(fm_pr_file_identity "$candidate" 2>/dev/null) || identity= + [ -n "$identity" ] || identity=$candidate + case " $seen " in + *" $identity "*) continue ;; + esac + seen="$seen $identity" + printf '%s\n' "$candidate" + done < <(printf '%s\n' "$PATH" | tr ':' '\n') +} + +# Ask one copy for its own version. Combined output, because tools answer on +# either stream, and no-mistakes announces its update on stderr. +probe_output() { + local path=$1 + shift + fm_run_timed "$(probe_bound)" "$path" "$@" 2>&1 +} + +command_findings() { + local name=$1 command_name=$2 args_joined=$3 announce=$4 announce_args=$5 + local hit out version matched announce_out status + local resolved_path='' resolved_version='' resolved_out='' + local best_path='' best_version='' unreadable='' hits='' + + # This tool's announcement source is dead if its pattern cannot be used, which + # is reported here, for this tool alone, so the rest of the sweep still runs. + if [ -n "$announce" ] && ! announce_pattern_usable "$announce"; then + emit "$name check failed: announce_pattern is not a usable extended regular expression" + announce= + fi + + hits=$(path_hits "$command_name") + if [ -z "$hits" ]; then + emit "$name check failed: $command_name is not on PATH" + return 0 + fi + + while IFS= read -r hit; do + [ -n "$hit" ] || continue + if budget_exhausted; then + emit "$name check failed: the time budget ran out before every copy answered" + break + fi + # shellcheck disable=SC2086 # deliberate split on validated space-free tokens + out=$(probe_output "$hit" $args_joined) + version=$(parse_version "$out") + if [ -z "$resolved_path" ]; then + resolved_path=$hit + resolved_version=$version + resolved_out=$out + fi + if [ -z "$version" ]; then + [ -n "$unreadable" ] || unreadable=$hit + continue + fi + if [ -z "$best_version" ] || version_newer "$version" "$best_version"; then + best_version=$version + best_path=$hit + fi + done <<EOF +$hits +EOF + + if [ -n "$announce" ] && [ -n "$resolved_path" ]; then + # A tool does not have to announce its update on the command that reports its + # version: no-mistakes prints its version for --version but announces a new + # release on its other commands. So announce_args may name a second command, + # and it is asked of the copy PATH actually resolves. + announce_out=$resolved_out + if [ "$announce_args" != "$args_joined" ]; then + if budget_exhausted; then + # The version probe's output cannot carry the announcement, so searching + # it would present a source that was never asked as a clean result. + emit "$name check failed: the time budget ran out before the update announcement was checked" + announce_out= + else + # shellcheck disable=SC2086 # deliberate split on validated space-free tokens + announce_out=$(probe_output "$resolved_path" $announce_args) + status=$? + if [ "$status" -eq 124 ]; then + # A source that was asked and never answered is not a source that had + # nothing to say. The one that answers with nothing stays silent below. + emit "$name check failed: $resolved_path did not answer when asked for its update announcement" + announce_out= + fi + fi + fi + if [ -n "$announce_out" ]; then + # Not a pipeline, so grep's own status is still readable here: a pattern + # grep cannot use is a check failure, never read as nothing to announce. + matched=$(grep -oE -- "$announce" <<< "$announce_out" 2>/dev/null) + status=$? + if [ "$status" -gt 1 ]; then + emit "$name check failed: announce_pattern is not a usable extended regular expression" + elif [ -n "$matched" ]; then + emit "$name update available: $(printf '%s\n' "$matched" | head -n 1)" + fi + fi + fi + + if [ -z "$resolved_version" ]; then + # No copy was probed at all when the path is empty, and the budget report + # already covers that, so do not blame a copy that was never asked. + [ -z "$resolved_path" ] || emit "$name check failed: $resolved_path did not report a version" + return 0 + fi + + if [ -n "$best_version" ] && [ "$best_path" != "$resolved_path" ] \ + && version_newer "$best_version" "$resolved_version"; then + emit "$name update not in effect: PATH resolves $resolved_version at $resolved_path but $best_version is installed at $best_path" + fi + + if [ -n "$unreadable" ]; then + emit "$name check failed: $unreadable did not report a version" + fi + return 0 +} + +# --- git probes ------------------------------------------------------------- + +# A probe the sweep budget can no longer afford is never issued, and says so with +# a status of its own rather than a git status, so no caller can read it as an +# answer. Neither git nor the bounded runner uses this value. +GIT_PROBE_NOT_ISSUED=3 + +# One bounded read-only git probe. The budget check lives here rather than in the +# callers, so no probe can be issued past the sweep deadline whatever a caller +# does, and the budget only has to leave room for the one probe that was already +# running when the deadline passed. +git_probe() { + local repo=$1 + shift + budget_exhausted && return "$GIT_PROBE_NOT_ISSUED" + fm_run_timed "$(probe_bound)" git -C "$repo" "$@" +} + +# The single place that reads a probe status as no answer at all, so every probe +# reports an unanswered read the same way instead of taking it for the answer no. +git_probe_answered() { + local status=$1 name=$2 subject=$3 question=$4 + case "$status" in + "$GIT_PROBE_NOT_ISSUED") + emit "$name check failed: the time budget ran out before $subject was asked $question" + return 1 + ;; + 124) + emit "$name check failed: $subject did not answer $question" + return 1 + ;; + esac + return 0 +} + +# Read-only throughout: nothing here writes to the watched repository. This is the +# one tool kind that issues several probes in a row, two of them over the network, +# and each of them goes through git_probe, which owns both the bound and the +# budget check, so the sweep cannot outrun its deadline here. +git_findings() { + local name=$1 repo=$2 remote=$3 branch=$4 + local status remote_sha local_sha local_label count short symref + + if ! command -v git >/dev/null 2>&1; then + emit "$name check failed: git is not installed" + return 0 + fi + if [ ! -d "$repo" ]; then + emit "$name check failed: $repo is not a directory" + return 0 + fi + budget_allows "$name" || return 0 + git_probe "$repo" rev-parse --git-dir >/dev/null 2>&1 + status=$? + git_probe_answered "$status" "$name" "$repo" "whether it is a git repository" || return 0 + if [ "$status" -ne 0 ]; then + emit "$name check failed: $repo is not a git repository" + return 0 + fi + + if [ -z "$branch" ]; then + branch=$(git_probe "$repo" symbolic-ref --short "refs/remotes/$remote/HEAD" 2>/dev/null) + git_probe_answered "$?" "$name" "$repo" "which branch it records for $remote" || return 0 + branch=${branch#"$remote/"} + fi + if [ -z "$branch" ]; then + # A clone made with --single-branch, or one that never ran remote set-head, + # has no local record of the remote's default branch. Ask the remote itself + # rather than reporting a check failure the operator cannot act on. + symref=$(git_probe "$repo" ls-remote --symref "$remote" HEAD 2>/dev/null) + git_probe_answered "$?" "$name" "$remote" "which branch it uses by default" || return 0 + branch=$(printf '%s\n' "$symref" \ + | awk '$1 == "ref:" { sub(/^refs\/heads\//, "", $2); print $2; exit }') + fi + if [ -z "$branch" ]; then + emit "$name check failed: cannot resolve the default branch of $remote in $repo" + return 0 + fi + + remote_sha=$(git_probe "$repo" ls-remote "$remote" "refs/heads/$branch" 2>/dev/null) + status=$? + git_probe_answered "$status" "$name" "$remote" "where $branch points" || return 0 + if [ "$status" -ne 0 ]; then + # The probe itself failed, so nothing at all is known about the branch. An + # offline host and a deleted branch are different problems, and reporting a + # missing branch here would name a cause that was never established. + emit "$name check failed: $remote could not be reached or read from $repo" + return 0 + fi + remote_sha=$(printf '%s\n' "$remote_sha" | awk 'NR == 1 { print $1 }') + if [ -z "$remote_sha" ]; then + emit "$name check failed: $remote has no branch $branch" + return 0 + fi + + # Each probe below is bounded, so a non-zero status means either the answer no + # or no answer at all. They are kept apart: reading a bound that was hit as an + # answer would report an update this check never established. + local_sha=$(git_probe "$repo" rev-parse --verify --quiet "refs/heads/$branch" 2>/dev/null) + git_probe_answered "$?" "$name" "$repo" "where $branch points" || return 0 + if [ -n "$local_sha" ]; then + local_label="local $branch" + else + local_sha=$(git_probe "$repo" rev-parse --verify --quiet HEAD 2>/dev/null) + git_probe_answered "$?" "$name" "$repo" "where HEAD points" || return 0 + if [ -z "$local_sha" ]; then + emit "$name check failed: $repo has no commit to compare" + return 0 + fi + local_label='local HEAD' + fi + + [ "$local_sha" != "$remote_sha" ] || return 0 + + short=$(printf '%s' "$remote_sha" | cut -c1-12) + + git_probe "$repo" cat-file -e "$remote_sha^{commit}" 2>/dev/null + status=$? + git_probe_answered "$status" "$name" "$repo" "whether it already has $short" || return 0 + if [ "$status" -eq 0 ]; then + # The local copy may be ahead of, or diverged from, the remote branch; only + # commits it does not have yet are an available update. + git_probe "$repo" merge-base --is-ancestor "$remote_sha" "$local_sha" 2>/dev/null + status=$? + git_probe_answered "$status" "$name" "$repo" "how its history compares with $remote/$branch" || return 0 + [ "$status" -ne 0 ] || return 0 + count=$(git_probe "$repo" rev-list --count "$local_sha..$remote_sha" 2>/dev/null) + git_probe_answered "$?" "$name" "$repo" "how many commits it is behind $remote/$branch" || return 0 + case "$count" in + ''|*[!0-9]*|0) count= ;; + esac + if [ -n "$count" ]; then + emit "$name update available: $local_label is $(commit_phrase "$count") behind $remote/$branch" + return 0 + fi + fi + + emit "$name update available: $remote/$branch is at $short which this copy does not have" + return 0 +} + +# --- report record ---------------------------------------------------------- + +RECORD_EPOCH=0 +RECORD_REPORTED= + +record_read() { + local line first=1 + RECORD_EPOCH=0 + RECORD_REPORTED= + [ -f "$RECORD" ] || return 0 + while IFS= read -r line; do + if [ "$first" = 1 ]; then + first=0 + [ "$line" = "$RECORD_SCHEMA" ] || return 0 + continue + fi + case "$line" in + epoch=*) + line=${line#epoch=} + case "$line" in + ''|*[!0-9]*) RECORD_EPOCH=0 ;; + *) RECORD_EPOCH=$line ;; + esac + ;; + reported=*) RECORD_REPORTED=${line#reported=} ;; + esac + done < "$RECORD" + return 0 +} + +record_write() { + local reported=$1 tmp + tmp=$(mktemp "$RECORD.XXXXXX" 2>/dev/null) || return 1 + chmod 0600 "$tmp" 2>/dev/null || { rm -f -- "$tmp"; return 1; } + { + printf '%s\n' "$RECORD_SCHEMA" + printf 'epoch=%s\n' "$(record_epoch_now)" + printf 'reported=%s\n' "$reported" + } > "$tmp" || { rm -f -- "$tmp"; return 1; } + mv -f -- "$tmp" "$RECORD" || { rm -f -- "$tmp"; return 1; } + return 0 +} + +# --- actions ---------------------------------------------------------------- + +action_check() { + local name command_name args_joined announce announce_args repo remote branch + local line now + + [ -f "$CONFIG" ] || return 0 + + record_read + now=$(record_epoch_now) + if [ "$INTERVAL" -ne 0 ] && [ "$RECORD_EPOCH" -gt 0 ] \ + && [ "$now" -ge "$RECORD_EPOCH" ] && [ $((now - RECORD_EPOCH)) -lt "$INTERVAL" ]; then + return 0 + fi + + DEADLINE=$(($(real_epoch) + BUDGET_SECS)) + + if [ -n "$BUDGET_CUT_FROM" ]; then + emit "sweep budget ${BUDGET_CUT_FROM}s cut to ${BUDGET_SECS}s to stay inside the watcher check timeout of ${CHECK_TIMEOUT}s" + fi + + if ! config_validate; then + emit "watched tool registry: $CONFIG_PROBLEM" + else + while IFS=$FIELD_SEP read -r name command_name args_joined announce announce_args repo remote branch; do + [ -n "$name" ] || continue + budget_allows "$name" || break + [ -z "$command_name" ] || command_findings "$name" "$command_name" "$args_joined" "$announce" "$announce_args" + [ -z "$repo" ] || git_findings "$name" "$repo" "$remote" "$branch" + done < <(config_records) + fi + + line= + if [ -n "$FINDINGS" ]; then + # Capped through the shared cut so an over-long report carries the same + # visible truncation marker the digests use, instead of ending mid-finding + # as if that were all of it. + fm_cap_line_var "tool updates: $FINDINGS" "$MAX_LINE" + line=$FM_LINE_CAP_LINE + fi + + # The cut line is what gets printed, but the whole finding set is what decides + # whether this is news, because a finding that lands past the cut leaves the + # printed line unchanged and would otherwise be suppressed for good. + # + # Report before recording, so a record that cannot be written costs a repeated + # report rather than a lost one. + if [ -n "$line" ] && [ "$FINDINGS" != "$RECORD_REPORTED" ]; then + printf '%s\n' "$line" + fi + record_write "$FINDINGS" || true + return 0 +} + +# The home is embedded already resolved, because the watcher runs the shim from +# its own working directory and a relative spelling would send the check to a +# different home, or to none at all. +shim_content() { + local home=$1 + printf '%s\n' \ + '#!/usr/bin/env bash' \ + '# Auto-generated by fm-tool-update-check.sh - watched tool update poll shim.' \ + '# The watcher validates these bytes, then dispatches the trusted check script.' \ + "export FM_HOME=$(printf '%q' "$home")" \ + "exec $(printf '%q' "$SCRIPT_DIR/fm-tool-update-check.sh") check" +} + +# Write the shim the way this repo writes its other trusted check shim: the +# guards run before anything is written, so a symlink at the shim path is +# refused instead of followed, and the bytes arrive by rename so the watcher +# never reads a half-written shim and rejects it as unauthenticated. +SHIM_WRITE_TMP= + +shim_write() { + local want=$1 device tmp + [ -d "$STATE" ] && [ ! -L "$STATE" ] || return 1 + device=$(fm_pr_file_device "$STATE") || return 1 + [ -n "$device" ] || return 1 + fm_pr_regular_destination_on_device_or_absent "$CHECK_SHIM" "$device" || return 1 + if [ -e "$CHECK_SHIM" ] && [ "$(fm_pr_file_mode "$CHECK_SHIM")" = 700 ] \ + && [ "$(cat "$CHECK_SHIM" 2>/dev/null)" = "$want" ]; then + return 0 + fi + tmp=$(umask 077; mktemp "$STATE/.fm-tool-updates-check.XXXXXX" 2>/dev/null) || return 1 + SHIM_WRITE_TMP=$tmp + if ! printf '%s\n' "$want" > "$tmp" \ + || ! chmod 0700 "$tmp" \ + || ! fm_pr_private_file_valid "$tmp" 700 "$device"; then + rm -f -- "$tmp" + SHIM_WRITE_TMP= + return 1 + fi + if ! fm_pr_regular_destination_on_device_or_absent "$CHECK_SHIM" "$device" \ + || ! mv -f -- "$tmp" "$CHECK_SHIM"; then + rm -f -- "$tmp" + SHIM_WRITE_TMP= + return 1 + fi + SHIM_WRITE_TMP= + fm_pr_private_file_valid "$CHECK_SHIM" 700 "$device" +} + +# Keep a byte copy of a shim that is already in place, so a failed arm can put +# back the shim a working home was already using rather than an equivalent +# rewrite. The trust binding is over the bytes, so a rewrite would satisfy it +# too, but a home that was armed stays armed with what it had. +shim_backup() { + local device tmp + device=$(fm_pr_file_device "$STATE") || return 1 + [ -n "$device" ] || return 1 + tmp=$(umask 077; mktemp "$STATE/.fm-tool-updates-check.XXXXXX" 2>/dev/null) || return 1 + if ! cat "$CHECK_SHIM" > "$tmp" 2>/dev/null \ + || ! chmod 0700 "$tmp" \ + || ! fm_pr_private_file_valid "$tmp" 700 "$device"; then + rm -f -- "$tmp" + return 1 + fi + printf '%s\n' "$tmp" +} + +ARM_BACKUP= + +# An unregistered shim is not inert: the watcher rejects it on every cycle and +# wakes firstmate about unauthenticated state checks. So the one rule after a +# failed or interrupted arm is that the home never holds a shim without a +# matching trust binding. The shim a working home had is put back and kept only +# when it is still bound; otherwise the shim goes, so the home is plainly not +# armed and the failure is the only thing the operator has to act on. +arm_rollback() { + [ -z "$SHIM_WRITE_TMP" ] || rm -f -- "$SHIM_WRITE_TMP" + SHIM_WRITE_TMP= + if [ -n "$ARM_BACKUP" ]; then + mv -f -- "$ARM_BACKUP" "$CHECK_SHIM" 2>/dev/null || rm -f -- "$ARM_BACKUP" + ARM_BACKUP= + if fm_custom_check_registered "$STATE" "$CHECK_ID"; then + return 0 + fi + fi + rm -f -- "$CHECK_SHIM" +} + +# shellcheck disable=SC2329 # Registered by action_arm's signal trap. +arm_interrupted() { + arm_rollback + printf 'fm-tool-update-check: arming was interrupted, so state/%s.check.sh is not armed\n' "$CHECK_ID" >&2 + exit 1 +} + +action_arm() { + local want home + if [ ! -f "$CONFIG" ]; then + printf 'fm-tool-update-check: no watched tool registry at %s\n' "$CONFIG" >&2 + return 1 + fi + if ! config_validate || ! config_announce_patterns_usable; then + printf 'fm-tool-update-check: %s (%s)\n' "$CONFIG_PROBLEM" "$CONFIG" >&2 + return 1 + fi + mkdir -p "$STATE" || return 1 + case "$FM_HOME" in + /*) home=$FM_HOME ;; + *) + home=$(CDPATH='' cd -- "$FM_HOME" 2>/dev/null && pwd -P) || { + printf 'fm-tool-update-check: cannot resolve FM_HOME %s\n' "$FM_HOME" >&2 + return 1 + } + ;; + esac + want=$(shim_content "$home") + ARM_BACKUP= + if [ -f "$CHECK_SHIM" ] && [ ! -L "$CHECK_SHIM" ]; then + ARM_BACKUP=$(shim_backup) || { + printf 'fm-tool-update-check: could not save the existing %s\n' "$CHECK_SHIM" >&2 + return 1 + } + fi + # The shim exists unbound from the rename until the register returns, so a + # signal in that window rolls back the same way a failure does. + trap arm_interrupted HUP INT TERM + if ! shim_write "$want"; then + trap - HUP INT TERM + arm_rollback + printf 'fm-tool-update-check: could not write %s\n' "$CHECK_SHIM" >&2 + return 1 + fi + if ! FM_HOME="$home" "$REGISTER_BIN" "$CHECK_ID" >/dev/null; then + trap - HUP INT TERM + arm_rollback + printf 'fm-tool-update-check: could not register %s\n' "$CHECK_SHIM" >&2 + return 1 + fi + trap - HUP INT TERM + [ -z "$ARM_BACKUP" ] || rm -f -- "$ARM_BACKUP" + ARM_BACKUP= + printf 'armed: state/%s.check.sh\n' "$CHECK_ID" + return 0 +} + +action_disarm() { + rm -f -- "$CHECK_SHIM" "$CHECK_TRUST" "$RECORD" + printf 'disarmed: state/%s.check.sh\n' "$CHECK_ID" + return 0 +} + +case "${1:-check}" in + check) action_check ;; + arm) action_arm ;; + disarm) action_disarm ;; + -h|--help) usage ;; + *) die_usage "unknown action: $1" ;; +esac diff --git a/bin/fm-turnend-guard.sh b/bin/fm-turnend-guard.sh index f3b4285511c..43e70457060 100755 --- a/bin/fm-turnend-guard.sh +++ b/bin/fm-turnend-guard.sh @@ -52,7 +52,9 @@ # 1. a live identity-matched watcher with a fresh beacon allows immediately; # 2. otherwise wait briefly (FM_CLAUDE_AUTOARM_SYNC_WAIT_MS, default 800ms) # for the auto-arm to claim this home (state/.claude-autoarm.lock owner -# alive) or to record a fresh actionable exit-2 outcome +# alive, with a supervision decision still open rather than a claim its own +# ledger entry or recorded pid-identity already settles as finished) or to +# record a fresh actionable exit-2 outcome # (state/.claude-autoarm-epoch) for this event epoch - either proof allows # without consuming a continuation, so one event epoch yields exactly one recovery turn; # the first fresh exhausted-failure epoch preserves the bounded progression, @@ -259,7 +261,15 @@ autoarm_owns_recovery() { fm_watcher_healthy "$STATE" "$WATCH" "$GRACE" "$FM_HOME" && return 0 pid=$(cat "$OWNER_LOCK/pid" 2>/dev/null || true) role=$(fm_lock_role "$OWNER_LOCK" 2>/dev/null || true) - if fm_pid_alive "$pid" && [ "$role" = autoarm ]; then + # A live auto-arm owner is only evidence of ownership while its supervision + # decision is still open. Once its own ledger entry records a terminal outcome, + # or its recorded pid-identity stops matching the pid holding the lock, the lock + # is abandoned, and treating it as ownership is what let a dead watcher go + # unnoticed for turn after turn. Fall through instead: the outcome cases below + # still cover a claim that finished moments ago, so a genuine handoff is not + # duplicated, while a stale one now reaches the block. + if fm_pid_alive "$pid" && [ "$role" = autoarm ] \ + && ! fm_autoarm_claim_abandoned "$STATE"; then [ ! -e "$FAILURE_NOTICE" ] || budget_account_current_epoch || true return 0 fi @@ -298,10 +308,18 @@ terminal_fail_open() { if ! fm_lock_try_acquire "$OWNER_LOCK"; then pid=$(cat "$OWNER_LOCK/pid" 2>/dev/null || true) role=$(fm_lock_role "$OWNER_LOCK" 2>/dev/null || true) - if fm_pid_alive "$pid" && [ "$role" = autoarm ]; then + # Same abandonment test as autoarm_owns_recovery: a claim whose ledger entry + # is already terminal, or whose recorded pid-identity no longer matches the + # live pid, is not a concurrent owner to step aside for. Stepping aside for one + # here allows the stop silently, and the episode's one attended alarm would + # never fire, so clear the abandoned claim and let this decision finish + # instead. Failing to clear it re-blocks rather than allowing. + if fm_pid_alive "$pid" && [ "$role" = autoarm ] \ + && ! fm_autoarm_claim_abandoned "$STATE"; then return 2 fi - return 1 + fm_autoarm_release_abandoned "$STATE" || return 1 + fm_lock_try_acquire "$OWNER_LOCK" || return 1 fi if ! fm_lock_set_role "$OWNER_LOCK" terminal-check; then fm_lock_release "$OWNER_LOCK" diff --git a/bin/fm-voice-client.py b/bin/fm-voice-client.py new file mode 100755 index 00000000000..9f9f9510ac8 --- /dev/null +++ b/bin/fm-voice-client.py @@ -0,0 +1,1373 @@ +#!/usr/bin/env python3 +"""fm-voice-client.py - the captain's laptop end of the spoken interface. + +Captures audio on the laptop, streams it over the SSH connection the captain +already has to the desktop, plays back the spoken reply, and reports how long +the round trip took. The desktop holds the Bedrock session and the AWS +credentials; this client needs neither. It needs Python and a microphone. + +WHAT IS VERIFIED AND WHAT IS NOT. Read this before trusting a number from it. + + Verified on the desktop: the frame protocol, the SSH transport, the relay + handshake, turn sequencing, the reply audio arriving intact, and the timing + arithmetic. All of that was exercised with --in-file and --out-file, which + replace the microphone and the speaker with files and leave everything else + alone. + + NOT verified, and cannot be from here: the audio DEVICES. The desktop this was + written on has neither a microphone nor a speaker, and no worker can reach the + captain's laptop. The sounddevice calls below are written from its documented + interface and have never been run against a real device. Treat the first live + run as the test. + + Verified, and worth telling apart from the devices: the speaker's own byte + ACCOUNTING, which is the arithmetic deciding which turn a chunk of reply audio + is credited to and which turn's first-audio clock it stamps. That is plain + logic rather than device work, so it is exercised against a stub stream with + the callback driven by hand. Nothing in that says how a real output device + behaves. + +TWO KINDS OF LISTENING, one of them built. --listen push-to-talk is the default +and the only mode that runs: the captain says when they are talking, the model is +only paid for that audio, and nothing is streamed while they are thinking. + +--listen open-mic is accepted as a setting and REFUSES at startup. Streaming +continuously needs something to decide when the captain stopped speaking, and +this client has no end-of-speech detection: it would open a turn, stream audio +forever and never mark a boundary, so the relay would keep appending to a session +that had already answered. That detection belongs with session continuity across +turns, which is step three of the design. The setting stays here so that turning +it on later is a small change rather than a new flag, and refusing is honest +where half a mode would not be. + +Copy this file and fm_voice_frame.py to the laptop; they are the only two files +it needs and both are standard library only, apart from sounddevice for the +audio devices. + +Usage: + fm-voice-client.py --host <sshhost> [options] + fm-voice-client.py --local [options] (relay as a child, no SSH) + +Options: + --host <name> SSH destination of the desktop holding the relay. + --local run the relay as a local child process instead. This is + how the relay path is measured without a laptop. + --relay <path> path to fm-voice-relay.py on the desktop, or set + $FM_VOICE_RELAY. Required: this file carries no default, + because one operator's home directory is not a path to + hand anybody else. + --relay-python <path> interpreter that has aws-sdk-bedrock-runtime installed. + default $FM_VOICE_PYTHON or python3 + --relay-arg <arg> extra argument for the relay, repeatable. Write it + joined with an equals sign, --relay-arg=--scope + --relay-arg=counts, or the leading dashes are read as + options of this client instead. + --listen <mode> push-to-talk, the default and the only mode that runs. + open-mic is accepted and refuses; see above. + --runs <n> turns to take in one session. default 1 + --talk-seconds <sec> capture for this long instead of waiting on a keypress. + --in-file <file.pcm> raw 16 kHz mono 16-bit input instead of the microphone. + --out-file <file.pcm> write reply audio here instead of playing it. + --input-device <id> sounddevice input device. + --output-device <id> sounddevice output device. + --timeout <sec> how long to wait for a reply. default 30 + --no-wait-for-reply open the next turn without waiting for the previous + answer to finish. The model treats that as being + interrupted and stops instead of answering, so this + exists to reproduce the trap, not to use. + --gap-seconds <sec> quiet beat after an answer finishes. default 0.5 + --verbose log the session to stderr. + +One JSON record per turn goes to stdout; everything human goes to stderr, so +`fm-voice-client.py --host desktop --runs 5 > runs.jsonl` gives measurements and +a readable session at the same time. +""" + +import argparse +import json +import os +import queue +import subprocess +import sys +import threading +import time +import traceback + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +import fm_voice_frame as frame # noqa: E402 + +IN_RATE = 16000 +OUT_RATE = 24000 +# 100 ms at each rate. The uplink chunk matches what the relay and the earlier +# prototype work measured with; changing it changes the numbers. +CHUNK = 3200 +OUT_BLOCK = 2400 + +PUSH_TO_TALK = "push-to-talk" +OPEN_MIC = "open-mic" +LISTEN_MODES = (PUSH_TO_TALK, OPEN_MIC) + +# Anything the relay's login shell prints on stdout ahead of the first frame is +# discarded, up to this much. Past it, the stream is not a relay. +MAX_PREAMBLE = 8192 + +# The two ends of a turn, queued rather than written, for the reason _sender +# gives: everything the uplink carries has to stay in the order it happened in. +START = object() +END = object() + + +class DeviceError(Exception): + """A microphone or speaker could not be opened, said in one line.""" + + +def log(enabled, message): + if enabled: + sys.stderr.write("client: {}\n".format(message)) + sys.stderr.flush() + + +def say(message): + sys.stderr.write("{}\n".format(message)) + sys.stderr.flush() + + +# --------------------------------------------------------------------- transport + + +def sync_magic(stream, verbose=False): + """Discard anything ahead of the relay's magic preamble. + + `ssh host command` runs the command through the captain's login shell, so a + shell startup file that prints a banner lands in front of the first frame. + Skipping to the preamble turns that from a baffling protocol error into a + warning naming the offending text. + """ + seen = bytearray() + while True: + byte = stream.read(1) + if not byte: + raise frame.FrameError( + "the relay closed the connection before it said hello; run the " + "relay command by hand over SSH to see its error") + seen += byte + if seen.endswith(frame.MAGIC): + junk = bytes(seen[: -len(frame.MAGIC)]) + if junk: + say("client: discarded {} bytes your login shell printed before " + "the relay started: {!r}".format(len(junk), junk[:200])) + log(verbose, "relay handshake found") + return + if len(seen) > MAX_PREAMBLE: + raise frame.FrameError( + "no relay handshake in the first {} bytes; the command on the " + "far end is not fm-voice-relay.py --serve".format(MAX_PREAMBLE)) + + +def relay_command(options): + """Return the argv that starts the relay, locally or over SSH.""" + remote = [options.relay_python, options.relay, "--serve"] + remote += list(options.relay_arg or []) + if options.verbose: + remote.append("--verbose") + if options.local: + return remote + # -T because a pty would rewrite bytes in the audio stream, which is the + # single most confusing way this could fail. + return ["ssh", "-T", options.host] + remote + + +class Uplink: + """Serialise every frame the client sends, from whichever thread sends it.""" + + def __init__(self, stream): + self._writer = frame.Writer(stream) + self._lock = threading.Lock() + + def send(self, kind, payload=b""): + with self._lock: + self._writer.send(kind, payload) + + +# ---------------------------------------------------------------------- playback + + +class FilePlayback: + """Write reply audio to a file. This is the path that can be verified here. + + Every chunk carries the turn it belongs to, and turn_reset names the turn + being measured. A chunk from a turn that has already been recorded is still + written, because it is the tail of an answer the captain is still listening + to, but it stamps no clock and is counted toward nobody: attributed to the + turn that happens to be open, it would hand that turn a first-audio figure + measured from somebody else's reply and report it answered when it was not. + """ + + def __init__(self, path): + self._handle = open(path, "wb") + # Two locks, and which one covers what is the point of them. _lock is the + # per-turn accounting, and turn_reset takes it while the client holds its + # own turn lock, so nothing slow may ever be done under it. _handle_lock + # covers the file itself, so a write and a close cannot overlap. The + # ordering is always _handle_lock then _lock and never the reverse. + # + # What close() needing _handle_lock costs: the exit is now only as bounded + # as one write to --out-file, so on a hung or full filesystem the five + # second downlink join in Client.close no longer bounds it. The wedged relay + # that join was written for is unaffected, being another process while this + # write is local. That cost belongs to the filed teardown-ordering work, + # whose other half is the same five second join being shorter than the ten + # seconds the relay may spend draining its own reply stream, which is why + # audio can arrive after the output is released at all. + self._handle_lock = threading.Lock() + self._lock = threading.Lock() + self.first_played = None + self.device_latency = None + self.turn_bytes = 0 + # Chunks dropped because they arrived after the file was released. Read by + # --verbose only; see write() for why it is not an outcome input. + self.discarded = 0 + self._turn = None + self._closed = False + + def write(self, pcm, turn): + # A chunk arriving after close is DISCARDED rather than raising. close() + # joins the downlink at five seconds while the relay teardown it waits on + # can take up to ten, so reply audio still in flight when the file is + # released is an expected and benign race, and erroring on it reported this + # end's own teardown as a fault through the frame-handling guard, on a + # session that worked. Discard is the honest semantic for it, and after + # this any fault line printed during teardown is a real one. + # + # Counted, because a write after close OUTSIDE teardown is a logic bug and + # a silent no-op would hide it. Counted and nothing more: discarded bytes + # stamp no clock, are credited to no turn, and so reach neither answered, + # first_audio_s nor the exit code, which are decided from turn_bytes. + # + # The turn comparison, the stamp and the count are one decision and are + # made together under _lock. The file write is not: it blocks, and holding + # the lock turn_reset needs across it would stall the whole client behind + # the filesystem. One writer keeps the file in order without that. + with self._handle_lock: + if self._closed: + with self._lock: + self.discarded += 1 + return + with self._lock: + mine = turn == self._turn + if mine and self.first_played is None: + self.first_played = time.monotonic() + if mine: + self.turn_bytes += len(pcm) + self._handle.write(pcm) + + def turn_reset(self, turn): + with self._lock: + self._turn = turn + self.first_played = None + self.turn_bytes = 0 + + def drain(self, timeout=5): + del timeout + + def close(self): + with self._handle_lock: + self._closed = True + self._handle.close() + + +class SpeakerPlayback: + """Play reply audio through the laptop speaker. + + The DEVICE is UNVERIFIED: written from the sounddevice interface and never run + against a real one, because the machine this was built on has no speaker, so + the first live run is its test. The byte ACCOUNTING below is covered, against + a stub stream with the callback driven by hand, and covering it says nothing + about how a real device behaves. + + The timestamp is taken when the audio is handed to the device callback, which + is the last moment this process can see. The device's own output buffer sits + after that, so its reported latency is included in the turn record rather + than pretended away. + + That timestamp is why the turn a chunk belongs to has to travel with the + chunk rather than being checked before the write: the moment that matters + happens in the callback, later than the frame arriving, and the gap between + the two is the honest content of the figure. So the buffer remembers how many + of its leading bytes belong to turns already recorded, and the first audio of + the turn being measured is the first byte past them. Ordering makes that a + count rather than a per-chunk tag: the downlink hands chunks over in arrival + order on one thread and a turn number never goes backwards, so a chunk from an + earlier turn can never queue behind one from a later turn. + """ + + def __init__(self, device=None): + import sounddevice # noqa: PLC0415 + self._buffer = bytearray() + self._lock = threading.Lock() + self.first_played = None + self.turn_bytes = 0 + # The same diagnostic the file path keeps, for the same reason. A counter + # that can only ever read zero is indistinguishable from one that measured + # zero, and this is the path the captain will actually use, so the write + # after close that the counter exists to catch has to be visible here too. + self.discarded = 0 + self._turn = None + self._earlier = 0 + self._closed = False + self._stream = sounddevice.RawOutputStream( + samplerate=OUT_RATE, channels=1, dtype="int16", + blocksize=OUT_BLOCK, device=device, latency="low", + callback=self._callback) + self._stream.start() + self.device_latency = getattr(self._stream, "latency", None) + + def _callback(self, outdata, frames_wanted, time_info, status): + del time_info, status + want = frames_wanted * 2 + with self._lock: + take = min(want, len(self._buffer)) + chunk = bytes(self._buffer[:take]) + del self._buffer[:take] + spent = min(self._earlier, take) + self._earlier -= spent + if take > spent and self.first_played is None: + self.first_played = time.monotonic() + outdata[:take] = chunk + if take < want: + outdata[take:want] = b"\x00" * (want - take) + + def write(self, pcm, turn): + with self._lock: + # close() stops the stream, and after that no callback drains the + # buffer, so a chunk arriving here was never going to be heard however + # it is stored. Discarded and counted rather than queued and credited + # to the turn, which is what the file path does: queued, it is a + # measurement the captain never heard, and silent, the write after + # close outside teardown that this counts for would be invisible on the + # one path they use. Counted and nothing more, so it stamps no clock + # and reaches neither answered, first_audio_s nor the exit code. + if self._closed: + self.discarded += 1 + return + if turn == self._turn: + self.turn_bytes += len(pcm) + else: + self._earlier += len(pcm) + self._buffer += pcm + + def turn_reset(self, turn): + with self._lock: + self._turn = turn + self.first_played = None + self.turn_bytes = 0 + # Whatever is still queued was spoken for an earlier turn. Counted as + # this turn's, the previous answer's undrained tail would stamp this + # turn's first audio the instant the device next asked for a block. + self._earlier = len(self._buffer) + + def drain(self, timeout=30): + """Wait for the buffered reply to finish, so the process does not cut it off.""" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + with self._lock: + if not self._buffer: + break + time.sleep(0.05) + time.sleep(0.2) + + def close(self): + # Marked before the stream is stopped and not while the lock is held: the + # device callback takes this lock, and stop() waits for a callback already + # running, so holding it across the stop is a deadlock. Marking first + # instead leaves no instant where the stream is gone and a write still + # queues for it. + with self._lock: + self._closed = True + try: + self._stream.stop() + self._stream.close() + except Exception: # noqa: BLE001 + pass + + +# ----------------------------------------------------------------------- capture + + +class FileCapture: + """Stream a PCM file as if it were the microphone, paced at real time. + + Paced deliberately: a file pushed as fast as the socket accepts it measures + the socket rather than the conversation. + """ + + def __init__(self, path): + with open(path, "rb") as handle: + self._pcm = handle.read() + self.seconds = round(len(self._pcm) / float(IN_RATE * 2), 3) + self.device_latency = None + self._q = None + self._talking = None + self._done = threading.Event() + + def start(self, out_q, talking): + self._q = out_q + self._talking = talking + + def begin_turn(self): + """Start feeding the file. One pass per turn, from the top each time.""" + self._done.clear() + + def run(): + for at in range(0, len(self._pcm), CHUNK): + if not self._talking.is_set(): + return + self._q.put(self._pcm[at:at + CHUNK]) + time.sleep(CHUNK / float(IN_RATE * 2)) + self._done.set() + + threading.Thread(target=run, daemon=True).start() + + def wait_exhausted(self, timeout): + return self._done.wait(timeout) + + def close(self): + pass + + +class MicCapture: + """Capture from the laptop microphone. + + UNVERIFIED: written from the sounddevice interface and never run against a + real device. The stream stays open for the whole session and the gate decides + what is sent, so push to talk costs no device setup per turn and the model is + only paid for audio while the gate is open. + """ + + def __init__(self, device=None): + import sounddevice # noqa: PLC0415 + self.seconds = None + self._q = None + self._talking = None + self._stream = sounddevice.RawInputStream( + samplerate=IN_RATE, channels=1, dtype="int16", + blocksize=CHUNK // 2, device=device, latency="low", + callback=self._callback) + self._stream.start() + self.device_latency = getattr(self._stream, "latency", None) + + def _callback(self, indata, frames_read, time_info, status): + del frames_read, time_info, status + if self._talking is not None and self._talking.is_set(): + self._q.put(bytes(indata)) + + def start(self, out_q, talking): + self._q = out_q + self._talking = talking + + def begin_turn(self): + """Nothing to do: the device stream is already open and the gate decides.""" + + def wait_exhausted(self, timeout): + del timeout + return False + + def close(self): + try: + self._stream.stop() + self._stream.close() + except Exception: # noqa: BLE001 + pass + + +# ------------------------------------------------------------------- audio setup + + +def open_file_end(flag, path, build): + """Open a file-backed end of the audio, naming the path and the flag for it. + + The file ends are the ones this host can run, and they are what every figure + in docs/voice-relay.md was measured with, so their refusal is the one most + likely to be read. It stays an OSError, which main prints as it stands, and it + names the path and the flag that chose it. Reporting a mistyped path as a + device failure would send the reader to the device flags instead of to the + path. + """ + try: + return build() + except OSError as exc: + raise OSError("could not open {}, given as {}: {}".format( + path, flag, exc)) + + +def open_device_end(flag, build): + """Open a device-backed end of the audio, or refuse in one line with a next step. + + sounddevice raises its own error types and is an optional import, so neither + shape reaches main as an OSError on its own and a traceback is what the + captain would otherwise get. Whether this refusal ever fires, and what a real + device says when it does, is unverified for the reason the module docstring + gives. + """ + try: + return build() + except Exception as exc: # noqa: BLE001 + raise DeviceError( + "could not open the audio device ({}: {}). Name another one with {}, " + "or run without a device using --in-file and --out-file".format( + type(exc).__name__, exc, flag)) + + +# ------------------------------------------------------------------------ client + + +class Client: + """One relay connection and the turns taken over it.""" + + def __init__(self, options): + self.options = options + self.verbose = options.verbose + self.proc = None + self.reader = None + self.uplink = None + self.playback = None + self.capture = None + self.down_thread = None + self.up_q = queue.Queue() + self.talking = threading.Event() + self.ready = threading.Event() + self.reply_done = threading.Event() + self.closed = threading.Event() + # Set the moment this end asks the relay to stop. It is the only thing + # that tells an expected goodbye from the relay stopping on its own, + # because the frame is the same one either way, and reading a clean end as + # a fault would train the captain to ignore the line that means it. + self.quitting = threading.Event() + self.ready_notice = {} + self.turn = {} + # Which turn self.turn is. A frame is read on one thread and applied on + # another, so a reply that arrives late, or a notice whose handling is + # descheduled, can be applied after the turn it belongs to has already + # been recorded and the next one opened. Without an identity to compare, + # that reply lands on the wrong turn: it names a fault that turn never + # had, releases it before its own answer, and stamps its first and last + # audio, which are the figures this whole tool exists to report. The + # downlink takes a copy of this when a frame arrives and applies nothing + # once it no longer matches. + self.turn_id = 0 + # What run() tells the captain when no further turn can be taken. Every + # path that makes the connection unusable names itself here, so the line + # about the runs that were lost restates the cause that was recorded + # rather than asserting one; a line naming the wrong cause sends them + # looking where the fault is not. The default only covers a closure with + # no path at all behind it, which nothing here can currently produce. + self.closed_because = "the connection closed" + self.lock = threading.Lock() + + # ------------------------------------------------------------------ lifecycle + + def open(self): + """Start the relay, the audio devices and the two frame threads. + + A startup that refuses part way through releases whatever it already + started, including on the SystemExit _wait_ready raises: a started + PortAudio stream left open at interpreter shutdown is a known hang on + macOS, which is the laptop this runs on. Whether it releases them + correctly against a real device is not something this host can show, for + the reason the module docstring gives. + """ + try: + self._start() + except BaseException: + self.close() + raise + + def _start(self): + argv = relay_command(self.options) + log(self.verbose, "starting relay: {}".format(" ".join(argv))) + self.proc = subprocess.Popen( + argv, stdin=subprocess.PIPE, stdout=subprocess.PIPE) + sync_magic(self.proc.stdout, self.verbose) + self.reader = frame.Reader(self.proc.stdout) + self.uplink = Uplink(self.proc.stdin) + + if self.options.out_file: + self.playback = open_file_end( + "--out-file", self.options.out_file, + lambda: FilePlayback(self.options.out_file)) + else: + self.playback = open_device_end( + "--output-device", + lambda: SpeakerPlayback(self.options.output_device)) + + if self.options.in_file: + self.capture = open_file_end( + "--in-file", self.options.in_file, + lambda: FileCapture(self.options.in_file)) + else: + self.capture = open_device_end( + "--input-device", + lambda: MicCapture(self.options.input_device)) + self.capture.start(self.up_q, self.talking) + + self.down_thread = threading.Thread(target=self._downlink, daemon=True) + self.down_thread.start() + threading.Thread(target=self._sender, daemon=True).start() + + self._wait_ready() + notice = self.ready_notice + say("client: relay ready, {} in {}, read scope {}, connected in {}s".format( + notice.get("model", "?"), notice.get("region", "?"), + notice.get("read_scope", "?"), notice.get("connect_seconds", "?"))) + + def _wait_ready(self): + """Wait for the relay's ready notice, or for the connection to close first. + + A relay that dies after the handshake is the likely first-run failure: + the Bedrock SDK is imported inside the model session, so a forgotten + --relay-python exits the relay after the handshake and before ready. Its + own one-line error is already on the captain's terminal, because stderr is + inherited rather than piped, so waiting out the full timeout after that + just leaves them watching nothing. + """ + deadline = time.monotonic() + self.options.timeout + while not self.ready.is_set(): + if self.closed.is_set(): + raise SystemExit( + "fm-voice-client: the relay closed the connection before it " + "was ready; run the relay command by hand over SSH to see " + "its error") + if time.monotonic() >= deadline: + raise SystemExit( + "fm-voice-client: the relay never reported ready; run it by " + "hand over SSH to see why") + self.ready.wait(0.2) + + def _quietly(self, what, action): + """Run one cleanup step without letting it mask why we are cleaning up.""" + try: + action() + except Exception as exc: # noqa: BLE001 + log(self.verbose, "{} did not close cleanly: {}: {}".format( + what, type(exc).__name__, exc)) + + def close(self): + # Every step is guarded and every field is checked, because close() also + # runs from a startup that refused part way through, where the later + # fields are still None and the original refusal is the message worth + # keeping. + if self.uplink is not None: + # Before the frame, so the goodbye that answers it is read as the + # answer to a question this end asked rather than as the relay + # stopping on its own. + self.quitting.set() + self._quietly("the uplink", lambda: self.uplink.send(frame.QUIT)) + # Before the devices are released, so the reply the goodbye above answers + # has somewhere to land, and bounded so a wedged relay cannot hold the + # exit. The bound is shorter than the relay's own teardown, so audio can + # still arrive after the output is released; the playback discards that + # rather than raising, which is what keeps a fault line meaning a fault. + if self.down_thread is not None: + self.down_thread.join(timeout=5) + if self.capture is not None: + self._quietly("the microphone", self.capture.close) + if self.playback is not None: + self._quietly("the speaker", self.playback.drain) + self._quietly("the speaker", self.playback.close) + if self.proc is not None: + try: + self.proc.stdin.close() + except Exception: # noqa: BLE001 + pass + try: + self.proc.wait(timeout=10) + except subprocess.TimeoutExpired: + self.proc.kill() + # Said last, because the relay exiting above is what stops the audio still + # in flight, and through _quietly like every other step here: a playback + # that cannot answer for its count must not replace the refusal that + # brought us into close() in the first place. + self._quietly("the discard count", self._say_dropped) + + def _say_dropped(self): + """Report reply audio the output was no longer open to take. + + A count read at teardown, which should normally be zero. This has one + caller and it is the last statement of close(), so the count is only ever + reported at the end of a session; the tripwire is still worth keeping, + because a close() added anywhere else would be counted here too. Both + output paths count it rather than only the file one. Diagnostic only: it + names nothing in the record and decides no exit code. + + Read straight off the playback rather than through a default, so a playback + that cannot answer is a failure rather than a zero indistinguishable from + having measured none. The None check is close()'s own, for the startup that + refused before there was an output at all. + """ + if self.playback is None: + return + dropped = self.playback.discarded + if dropped: + log(self.verbose, + "discarded {} reply audio chunk(s) that arrived after the output " + "was released".format(dropped)) + + # -------------------------------------------------------------------- threads + + def _sender(self): + """Own the whole uplink, so nothing on it can be sent out of order. + + Every frame a turn consists of goes through this one queue, talk start + included. Sending the start from the turn thread instead cost a turn: a + turn that ends with no answer to wait for - a failed turn, or the model + finishing with the session - returns as soon as it is told, while the last + chunk and the talk end may still be here. The next talk start would then + overtake them, the relay would open a fresh session and apply the previous + turn's talk end to it, and the captain's entire next question was dropped + as audio arriving with no turn open. It answered a question nobody had + finished asking. + + A closed connection is a dead uplink for talk start and talk end just as + much as for audio, so all three are sent through the same guard. Sending + the control frames outside it cost the rest of the session: the write + raised, this thread died with a traceback, and every later turn queued + frames nobody was left to send, so it waited out the full timeout with + no answer instead of reporting the lost connection the downlink had + already seen. + """ + while True: + item = self.up_q.get() + if item is None: + return + if item is START: + kind, payload = frame.TALK_START, b"" + elif item is END: + with self.lock: + self.turn["wire_end"] = time.monotonic() + kind, payload = frame.TALK_END, b"" + else: + kind, payload = frame.AUDIO, item + try: + self.uplink.send(kind, payload) + except (BrokenPipeError, OSError): + return + + def _unfinished(self, subject): + """Name a fault that landed on an open turn, in the words that turn earned. + + A relay dies mid-turn in two shapes and they are not the same fault. With + no reply audio yet, the turn went unanswered. With some already played, + the captain heard the start of an answer and the rest was cut off, so the + turn WAS answered and first_audio_s is a real measurement of when: saying + nothing arrived would contradict the answered field two lines below it in + the same record, and a reader who believes the wrong one goes looking in + the wrong place. + + Read off the same count answered is read off, so the two cannot disagree + about one turn whatever the timing. + """ + if self.playback.turn_bytes > 0: + return "{} before the reply finished".format(subject) + return "{} before this turn was answered".format(subject) + + def _downlink(self): + # Why the loop stopped, for a turn that was still waiting for its reply + # when it did. Neither of the quiet exits below raises, and they are + # different faults, so each names itself rather than leaving the tail to + # guess or to say nothing. + why = None + # And what run() says about the runs that were lost to it. Separate from + # the reason above because they are different statements: that one is why + # this turn has no answer, this one is why there will be no more turns. + cause = None + while True: + try: + got = self.reader.read() + except (frame.FrameError, OSError) as exc: + say("client: connection lost: {}".format(exc)) + # Recorded as well as said, because the turn record is what a + # latency figure is read from later and stderr is not. A dropped + # connection that only says answered: false is indistinguishable + # there from a turn the model declined to answer. setdefault + # because a relay that named the failure first said it better. + # + # closed is set in this same critical section, not left to the + # tail below, because take_turn decides whether another turn can + # be opened by reading it under this lock. Naming the failure + # first and announcing the closure afterwards left a window where + # the connection was known gone and no reader could tell. + # + # The reply_done test is the one the quiet close paths below + # already apply, and it is here for the same reason: a reason + # belongs to a turn that has not had its answer yet. Without it a + # fault landing in the gap between reply_end arriving and the + # record being copied named a turn that was fully answered, and + # since a recorded reason exits non-zero that failed a session + # which had delivered everything asked of it. An end of stream and + # a reset differ only in what the kernel handed us, so they must + # not produce two different exit codes for one relay death. + # + # WHAT THE TEST MAKES INVISIBLE, because it is a real cost rather + # than none: a relay failure arriving after the FINAL turn's reply + # was already complete now records no reason and exits 0. The relay + # puts every audioOutput chunk and the reply_end mark on one + # ordered queue, so by the time this end sets reply_done every byte + # of that answer has already reached the playback, and a fault + # after it cannot have cost the captain any part of what they were + # given. What it can still cost is a LATER turn, and that is + # reported with no per-turn reason at all by the remaining-runs + # check, which exits non-zero whenever the connection is known gone + # with runs still to take. A relay dying at that instant is also + # indistinguishable from the same relay dying a moment later during + # this end's own teardown, which this client already treats as + # benign. The bound: take_turn clears reply_done in the same + # critical section as the closure mark, so the blind spot is + # exactly "after this turn's reply completed" and never "during a + # turn". + # + # closed_because and the closure mark stay outside it, so the + # session still knows the connection went and still says so. + with self.lock: + if not self.reply_done.is_set(): + self.turn.setdefault( + "failed", "{}: {}".format( + self._unfinished("the connection was lost"), + exc)) + self.closed_because = "the connection was lost" + self.closed.set() + break + if got is None: + if not self.quitting.is_set(): + say("client: the connection ended") + # The subject only. Whether it ended before the turn was answered + # or partway through the answer is decided by _unfinished at the + # tail, where the audio count is read. + why = "the connection ended" + cause = "the connection ended" + break + kind, payload = got + # Which turn this frame belongs to, taken the moment it arrives. Every + # write below applies only while it is still that turn; see turn_id. + with self.lock: + arrived_in = self.turn_id + try: + if kind == frame.AUDIO: + with self.lock: + if arrived_in == self.turn_id: + now = time.monotonic() + self.turn.setdefault("first_frame", now) + self.turn["last_frame"] = now + # Played whichever turn it belongs to, and told which that is. + # Late audio is the tail of an answer the captain is still + # listening to, so dropping it would cut them off, but three + # figures are read off what this call does - first_played, the + # reply's own duration and whether the turn was answered at all + # - and a stale chunk credited to the turn now open reports an + # unanswered turn as answered, which is an exit code of zero on + # a session that lost one. + self.playback.write(payload, arrived_in) + elif kind == frame.TEXT: + obj = frame.decode_json(payload) + text = (obj.get("text") or "").strip() + if text and not text.startswith("{"): + who = "you" if obj.get("role") == "USER" else "assistant" + say(" {}: {}".format(who, text)) + elif kind == frame.NOTICE: + obj = frame.decode_json(payload) + event = obj.get("event", "") + if event == "ready": + self.ready_notice = obj + self.ready.set() + elif event == "queued": + say(" handed to the first mate: {}".format( + obj.get("request", ""))) + with self.lock: + if arrived_in == self.turn_id: + self.turn["queued"] = obj.get("note_id", "") + elif event == "interrupted": + with self.lock: + if arrived_in == self.turn_id: + self.turn["interrupted"] = True + log(self.verbose, "the model treated this turn as an " + "interruption of its own speech") + elif event == "turn-failed": + # The relay is still there and the next talk key gets a + # new session, so this ends the turn rather than the run. + say("client: the relay could not finish that turn: {}" + .format(obj.get("error", ""))) + # Named and released in one critical section, so no turn + # can be released without also being told why. The + # reply_done test is the read path's, for the reason given + # there: a relay whose model stream broke in the gap after + # this turn's answer completed has cost this turn nothing, + # and naming it here would fail a session that answered. + # The release stays outside the test, so a failure arriving + # while the turn is still waiting still ends its wait. + with self.lock: + if arrived_in == self.turn_id: + if not self.reply_done.is_set(): + self.turn["failed"] = obj.get("error", "") + self.reply_done.set() + elif event == "session-ended": + say("client: the relay ended the session") + # An ordinary session end is not a turn failure at the + # relay, and the next talk key still gets a working one. A + # turn released by it nevertheless has no answer, and + # relay_error is where the reason for that is read from + # later, so it carries what the captain was just told. The + # reply_done test and the setdefault are the tail's, for + # the tail's reasons. + with self.lock: + if arrived_in == self.turn_id: + if not self.reply_done.is_set(): + self.turn.setdefault( + "failed", + self._unfinished( + "the relay ended the session")) + self.reply_done.set() + else: + log(self.verbose, "notice {}".format(obj)) + elif kind == frame.MARK: + obj = frame.decode_json(payload) + with self.lock: + if arrived_in == self.turn_id: + self.turn.setdefault( + "marks", {})[obj.get("mark", "?")] = \ + obj.get("since_talk_end") + self.turn["tool_calls"] = obj.get("tool_calls", 0) + if obj.get("mark") == "reply_end": + self.reply_done.set() + elif kind == frame.BYE: + # The same frame ends a session this end asked to end and a + # relay that stopped on its own, so the frame says nothing on + # its own and whether we asked is the whole discriminator. + # Both speak, because a session that ended should say so, and + # neither borrows the other's words: a line that also appears + # when everything worked is a line the captain learns to skip, + # and then the one that means trouble is invisible too. + if self.quitting.is_set(): + say("client: the relay signed off") + cause = "the relay signed off after being asked to stop" + else: + say("client: the relay stopped without being asked to") + why = "the relay stopped" + cause = "the relay stopped without being asked to" + break + except Exception as exc: # noqa: BLE001 + # A fault on THIS end, handling a reply that did arrive: the + # speaker or the output file refusing the audio, or a payload that + # is not the JSON the wire format promises. Caught as a class + # rather than as a list, because this handling code can raise + # something nobody listed, and the failure being removed here is + # this thread dying silently: closed and reply_done then stay + # unset, and every remaining run opens a turn, waits out the whole + # timeout and is recorded unanswered with no reason at all, so one + # fault costs the session instead of one turn. + # + # Deliberately not worded as a lost connection. The connection is + # fine and naming it would send the captain to the wrong end. + fault = ("this end could not handle the relay's reply: {}: {}" + .format(type(exc).__name__, exc)) + say("client: {}".format(fault)) + # The one line is for the captain and the record; the traceback is + # for whoever has to find the bug behind it. Before this guard + # existed the thread died and threading.excepthook printed one, so + # a programming error in here would otherwise be strictly harder to + # locate than it used to be. Terminal path, so this prints once per + # session at worst, and the record keeps the one-line reason + # because that field is machine read. + sys.stderr.write(traceback.format_exc()) + sys.stderr.flush() + with self.lock: + self.turn.setdefault("failed", fault) + self.closed_because = fault + self.closed.set() + break + # Under the turn lock for the same reason the failure above is: a clean + # end of file and a goodbye leave the connection just as unusable as a + # dropped one, and take_turn reads this under that lock to decide whether + # a turn can still be opened. Already set on the failure path; setting an + # event twice costs nothing. + # + # A turn still waiting for its reply is named in the same critical + # section, and before the event that releases it, so the turn reading the + # record finds the reason rather than racing it. reply_done is the test: + # take_turn clears it under this lock when it opens a turn and it is set + # at every other moment, so an answered turn whose connection then ends + # cleanly keeps its record and stays reason-free. setdefault, because a + # relay that named the failure first said it more precisely than this end + # can infer it. + with self.lock: + if why is not None and not self.reply_done.is_set(): + self.turn.setdefault("failed", self._unfinished(why)) + if cause is not None: + self.closed_because = cause + self.closed.set() + self.reply_done.set() + + # ---------------------------------------------------------------------- turns + + def take_turn(self, index): + """Run one turn and return its record, or None if the connection is gone. + + The check and the reset share one critical section with the downlink's + closure mark on purpose. The wait between turns is seconds long and is + where a relay that dies between questions dies, so the run loop cannot + decide to open another turn by reading a flag the downlink sets after it + records the failure: between those two writes the connection is already + gone and the loop cannot see it. It then cleared the failure the downlink + had recorded, sent talk-start into a dead pipe, and came back after the + whole reply timeout as answered: false with relay_error: null - a lost + connection wearing the shape of a turn the model declined, in the file + docs/voice-relay.md computes its published latency spread from. + """ + with self.lock: + if self.closed.is_set(): + return None + self.turn = {} + # Advanced here, with the reset it names, so a frame still being + # handled from the previous turn can tell that its turn is over. + self.turn_id += 1 + # In the same critical section as the closure mark, because this + # event is how the downlink tells a turn waiting for a reply from the + # space between turns. Cleared outside the lock it leaves a window + # where the connection has already gone, the downlink has read the + # event as nobody waiting and named nothing, and this turn then waits + # out its whole timeout to be recorded with no reason at all. + self.reply_done.clear() + # In the same critical section, and named with the same identity the + # frames carry, so there is no instant where the turn has advanced and + # the playback is still counting audio toward the turn before it. + self.playback.turn_reset(self.turn_id) + self.up_q.put(START) + + # Unreachable while parse_args refuses open-mic, and kept so that turning + # the mode on later is a small change. It is still missing the turn + # boundary: it opens the gate and nothing ever closes it, so no talk end + # is ever sent. Do not lift the refusal without adding that first. + if self.options.listen == OPEN_MIC: + release = None + self.talking.set() + self.capture.begin_turn() + say("client: open microphone, run {}. Speak when you like.".format(index)) + else: + release = self._push_to_talk(index) + + deadline = self.options.timeout + if not self.reply_done.wait(timeout=deadline): + say("client: no reply within {}s".format(deadline)) + self._wait_audio_quiet(deadline) + + with self.lock: + turn = dict(self.turn) + marks = turn.get("marks", {}) + played = self.playback.first_played + reply_bytes = self.playback.turn_bytes + first_frame = turn.get("first_frame") + + def since(at): + if release is None or at is None: + return None + return round(at - release, 3) + + record = { + "run": index, + "listen": self.options.listen, + "transport": "local" if self.options.local else "ssh", + "host": None if self.options.local else self.options.host, + "input": self.options.in_file or "microphone", + "output": self.options.out_file or "speaker", + "model": self.ready_notice.get("model"), + "region": self.ready_notice.get("region"), + "read_scope": self.ready_notice.get("read_scope"), + "connect_seconds": self.ready_notice.get("connect_seconds"), + "tool_calls": turn.get("tool_calls", 0), + "queued_note": turn.get("queued"), + "interrupted": bool(turn.get("interrupted")), + # Why a turn has no answer, when either end knows: the relay names a + # failed turn, and this end names a connection that went during one. + # A results file that only says answered: false invites the reader to + # average an infrastructure failure into a latency figure. + "relay_error": turn.get("failed"), + # The number this build exists to produce: the captain stopped + # talking, and this many seconds later sound came out. + "first_audio_s": since(played if played is not None else first_frame), + "first_frame_s": since(first_frame), + "first_played_s": since(played), + "last_frame_s": since(turn.get("last_frame")), + "uplink_drain_s": since(turn.get("wire_end")), + "device_output_latency_s": self.playback.device_latency, + "device_input_latency_s": self.capture.device_latency, + "relay_marks_since_talk_end": marks, + # This turn's own audio, counted by the playback rather than by + # subtracting a byte total it shares with every other turn. A total + # cannot tell a reply from the previous reply's tail arriving late, and + # counting that tail here reports a turn nobody answered as answered. + "reply_audio_seconds": round( + reply_bytes / float(OUT_RATE * 2), 3), + "answered": reply_bytes > 0, + } + if release is None: + record["first_audio_note"] = ( + "An open microphone has no local end of speech, so the model's " + "own detector is the only clock. Read " + "relay_marks_since_talk_end instead.") + elif not self.options.out_file: + record["first_audio_note"] = ( + "Measured to the moment audio was handed to the output device. " + "The device's own buffer, reported as " + "device_output_latency_s, comes after that.") + else: + record["first_audio_note"] = ( + "Measured to the moment reply audio reached this process. There " + "is no speaker in this configuration, so no playback latency is " + "included.") + return record + + def _wait_audio_quiet(self, deadline): + """Wait for the reply audio to stop arriving before reading the turn. + + Measured, the last audio frame and END_TURN land within about ten + milliseconds of each other, audio first, so this almost always returns + at once. It is here because the count of reply audio is what the + no-overlap wait below depends on, and a turn that ends any other way, + such as the session closing, would otherwise be counted short. + """ + limit = time.monotonic() + deadline + while time.monotonic() < limit: + with self.lock: + last = self.turn.get("last_frame") + if last is None: + return + if time.monotonic() - last >= self.options.audio_idle: + return + time.sleep(0.05) + + def _push_to_talk(self, index): + """Open the gate, close it, and return the moment the captain stopped. + + That instant, not the moment the last byte reaches the wire, is what the + captain experiences as the end of their own speech. Every headline number + is measured from it, and uplink_drain_s reports the difference so a slow + connection stays visible rather than hiding inside the total. + """ + seconds = self.options.talk_seconds + if seconds is None and not self.options.in_file: + try: + input("\nrun {}: press Enter, speak, then press Enter again.".format( + index)) + except EOFError: + raise SystemExit( + "fm-voice-client: no keyboard on this input. Use " + "--talk-seconds or --in-file for an unattended run.") + + self.talking.set() + self.capture.begin_turn() + if seconds is not None: + say("client: run {}, capturing {}s.".format(index, seconds)) + time.sleep(seconds) + elif self.options.in_file: + self.capture.wait_exhausted(self.options.timeout) + else: + say(" listening. Enter to send.") + try: + input() + except EOFError: + pass + + self.talking.clear() + release = time.monotonic() + self.up_q.put(END) + log(self.verbose, "talk end queued") + return release + + def _let_reply_finish(self, record): + """Wait for the previous answer to finish before opening another turn. + + The model tracks its own speech, and audio arriving while it believes it + is still talking is an interruption: it emits an INTERRUPTED marker, and + the interrupted turn is then lost. It goes as far as calling the tool and + then produces no answer at all, which is the worst of both, so this is + not an inconvenience to be tolerated. + + The clock that matters runs from the END of generation, not the start. + The model streams a six second answer in about one second, and a turn + opened at first-frame plus six seconds was still interrupted, while + last-frame plus six seconds was not. So the wait is the reply's own + duration measured from the last frame, plus a beat. In conversation that + costs nothing: it is exactly the pause a captain takes anyway, because + they are listening to the answer. + + Barge-in is step three of the design, so until it is built a turn waits. + --no-wait-for-reply reproduces the trap deliberately. + """ + if not self.options.wait_for_reply: + return + self.playback.drain() + with self.lock: + last = self.turn.get("last_frame") + seconds = record.get("reply_audio_seconds") or 0 + if last is None or not seconds: + return + remaining = last + seconds + self.options.gap_seconds - time.monotonic() + if remaining > 0: + log(self.verbose, + "waiting {:.2f}s for the answer to finish".format(remaining)) + time.sleep(remaining) + + def _say_stopped(self, index): + """Name why no more turns can be taken, and which run was the first lost. + + The cause is whatever the path that closed the connection recorded, not an + assertion made here: a fault on this end leaves the connection open, and a + line blaming the connection for it sends the captain to the wrong end. + """ + with self.lock: + because = self.closed_because + say("client: {} before run {} of {}; it and the rest were not " + "taken".format(because, index, self.options.runs)) + + def run(self): + rc = 0 + for index in range(1, self.options.runs + 1): + record = self.take_turn(index) + # take_turn refusing is the one place a closed connection stops the + # session, so the outcome is the same wherever the connection went: + # nothing more can be taken over it, the runs the captain asked for + # were not, and the exit code says so, because a session that stops + # early while reporting success is read later as a complete + # measurement. A second check here, on a flag read before the turn + # rather than under the lock that guards it, is what let a lost + # connection through in the first place; and no record is printed for + # a turn that never opened, since an invented turn is the whole thing + # being kept out of runs.jsonl. + if record is None: + self._say_stopped(index) + rc = 1 + break + print(json.dumps(record)) + sys.stdout.flush() + # A named reason counts as well as an unanswered turn, and not only + # when a later run remains. A relay killed while speaking leaves a + # turn that was answered and a record that says why the answer stopped + # partway, and at the default of one run that turn cleared all three of + # the other paths to a non-zero code and reported the session a + # success. A results file whose own record names an infrastructure + # failure must not sit behind an exit code that says nothing happened. + if not record["answered"] or record["relay_error"]: + rc = 1 + if index < self.options.runs: + # Checked after the record and before the wait, because that wait + # is seconds long and exists only to avoid interrupting the model's + # own speech, which a relay that is already gone cannot be doing. + # Waiting it out here left the captain sitting through the last + # reply's whole spoken duration before being told the session had + # stopped. The exit code is still the unhappy one: the runs asked + # for were not taken, whatever the last one reported. + if self.closed.is_set(): + self._say_stopped(index + 1) + rc = 1 + break + self._let_reply_finish(record) + return rc + + +def device_selector(value): + """Return a sounddevice device: an index when the value is digits, a name otherwise. + + sounddevice reads an int as an index into its device list and a str as a + substring to match against device names, so an index left as text is looked + up as a device literally called "3" and raises. docs/voice-relay.md tells the + captain these flags take a name or an index, so both have to arrive typed. + """ + return int(value) if value.strip().isdigit() else value + + +def parse_args(argv): + parser = argparse.ArgumentParser( + prog="fm-voice-client.py", add_help=True, + description=__doc__.splitlines()[0]) + parser.add_argument("--host") + parser.add_argument("--local", action="store_true") + parser.add_argument("--relay", default=os.environ.get("FM_VOICE_RELAY"), + help="path to fm-voice-relay.py on the desktop; required, " + "and FM_VOICE_RELAY sets it for a whole shell") + parser.add_argument("--relay-python", + default=os.environ.get("FM_VOICE_PYTHON", "python3")) + parser.add_argument("--relay-arg", action="append") + parser.add_argument("--listen", choices=LISTEN_MODES, default=PUSH_TO_TALK, + help="push-to-talk is the default and the only mode that " + "runs; open-mic is accepted and refuses until " + "end-of-speech detection exists") + parser.add_argument("--runs", type=int, default=1) + parser.add_argument("--talk-seconds", type=float) + parser.add_argument("--in-file") + parser.add_argument("--out-file") + parser.add_argument("--input-device", type=device_selector) + parser.add_argument("--output-device", type=device_selector) + parser.add_argument("--timeout", type=float, default=30.0) + parser.add_argument("--wait-for-reply", action=argparse.BooleanOptionalAction, + default=True, + help="wait for each answer to finish being spoken before " + "opening the next turn (default on)") + parser.add_argument("--gap-seconds", type=float, default=0.5, + help="quiet beat after an answer finishes. default 0.5") + parser.add_argument("--audio-idle", type=float, default=0.4, + help="silence that counts as the reply having stopped " + "arriving. default 0.4") + parser.add_argument("--verbose", action="store_true") + options = parser.parse_args(argv) + if bool(options.host) == bool(options.local): + parser.error("give exactly one of --host <sshhost> or --local") + if not options.relay: + parser.error( + "say where the relay is: --relay <path to fm-voice-relay.py on the " + "desktop>, or set FM_VOICE_RELAY") + if options.runs < 1: + parser.error("--runs must be at least 1") + if options.listen == OPEN_MIC and options.in_file: + parser.error( + "--listen open-mic with --in-file would end the turn when the file " + "ran out, which is not what an open microphone does") + if options.listen == OPEN_MIC: + # Here rather than in open(), so nothing is spent: no ssh, no relay, no + # model session. See the module docstring on the two kinds of listening. + parser.error( + "--listen open-mic is not built yet: it needs end-of-speech " + "detection to know when a turn ended, which lands with session " + "continuity across turns, so it would stream forever and never end " + "a turn. Use the default --listen push-to-talk.") + return options + + +def main(argv): + options = parse_args(argv) + client = Client(options) + try: + client.open() + except SystemExit as exc: + # _wait_ready refuses this way and its message is already the whole + # story. open() has released what it started; this turns the refusal + # into the same one-line exit the rest of this file gives. + if exc.code not in (None, 0): + sys.stderr.write("{}\n".format(exc.code)) + return 2 + except (frame.FrameError, OSError, DeviceError) as exc: + sys.stderr.write("fm-voice-client: {}\n".format(exc)) + return 2 + except Exception as exc: # noqa: BLE001 + sys.stderr.write("fm-voice-client: could not start: {}: {}\n".format( + type(exc).__name__, exc)) + return 2 + try: + return client.run() + except KeyboardInterrupt: + say("client: stopping.") + return 130 + finally: + client.close() + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/bin/fm-voice-relay.py b/bin/fm-voice-relay.py new file mode 100755 index 00000000000..f6b61297754 --- /dev/null +++ b/bin/fm-voice-relay.py @@ -0,0 +1,1256 @@ +#!/usr/bin/env python3 +"""fm-voice-relay.py - hold the Nova Sonic session on this desktop, on behalf of the laptop. + +The captain talks into their laptop. The laptop captures audio and streams it +over the SSH connection it already has to this desktop. This relay holds the +Bedrock bidirectional session, answers the model's tool calls from firstmate's +records, and streams the spoken reply back down the same connection. AWS +credentials therefore stay on this desktop and never go near the laptop, which +is the whole reason for the shape. + +The voice agent this relay runs is NOT firstmate. It stands in front of +firstmate: it answers questions about the fleet from the records, and when the +captain asks for real work it says out loud that it is handing the request over +and then queues it. It never claims to have done the work. + +Modes: + --serve read fm_voice_frame frames on stdin, write them on stdout. + This is what the laptop client runs over SSH, and the + default when no mode is given. + --self-test FILE feed one raw 16 kHz PCM file into a session as if it had + arrived from the client, print the timings as JSON, exit. + This is the control measurement for the relay path, and it + needs no client, no SSH and no microphone. + +The two traps this code already avoids, both found the expensive way and both +measured rather than assumed: + + 1. completionEnd does not arrive on its own. The model holds the session open + waiting for more speech. The real "the reply is finished" signal is a + contentEnd carrying stopReason END_TURN. + 2. Audio with no trailing silence is truncated and never answered, even when + contentEnd follows immediately. A push-to-talk release supplies no trailing + silence at all, so this relay appends its own on talk end. --tail-ms sets + how much. Measured here, the tail is a content requirement and not a time + one: nothing was answered at 0 or 100 ms, everything was answered from + 200 ms up, and 200 through 800 ms all landed in the same spread because the + silence is sent unpaced. The 400 ms default is margin that costs nothing. + +Read scope, deny list and the handover queue all belong to bin/fm_voice_records.py. +bin/fm_voice_frame.py owns the wire contract between the two machines, and +docs/voice-relay.md is the operator-facing guide. + +CONFIGURATION. The region, the model and the AWS profile name somebody's account +and somebody's choices, so this file carries no default for them. Each is read +from the home's gitignored config/ directory, or from the matching environment +variable, and a missing one refuses with the path to write rather than reaching +for a value that belongs to another home. That configuration is also the opt-in: +an unconfigured home cannot start this relay at all. + + config/voice-region FM_VOICE_REGION Bedrock region. required + config/voice-model FM_VOICE_MODEL Nova Sonic model id. required + config/voice-profile FM_VOICE_PROFILE AWS profile. optional + config/voice-id FM_VOICE_ID output voice. default matthew + +An absent profile means the relay uses only credentials that are already in its +environment. An empty FM_VOICE_PROFILE, or an empty `--profile ""`, forces that +even when config/voice-profile exists. + +On choosing the model: the first-generation Nova Sonic model is marked legacy by +AWS and measured 25 percent slower on the tool-backed path, which is the path this +interface actually uses, so the figures in docs/voice-relay.md were taken against +the second generation, which that document names. + +Usage: + fm-voice-relay.py [--serve] [options] + fm-voice-relay.py --self-test <file.pcm> [options] + +Options: + --region <name> Bedrock region. default from config + --model <id> Nova Sonic model id. default from config + --profile <name> AWS profile. default from config + --voice <id> output voice. default matthew + --home <dir> firstmate home for records. default $FM_HOME or this repo + --scope <name> override the read scope for this run. + --tail-ms <int> silence appended on talk end. default 400 + --turn-timeout <sec> how long --self-test waits. default 40 + --verbose log the session to stderr. +""" + +import argparse +import asyncio +import base64 +import datetime +import json +import os +import queue +import subprocess +import sys +import threading +import time +import traceback +import uuid + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +import fm_voice_frame as frame # noqa: E402 +import fm_voice_records as records # noqa: E402 + +# A voice id names nobody and costs nothing to inherit, so this one has a +# default. The region, the model and the profile do not; see CONFIGURATION above. +VOICE = "matthew" +SETTINGS = { + "region": ("voice-region", "FM_VOICE_REGION", "Bedrock region"), + "model": ("voice-model", "FM_VOICE_MODEL", "Nova Sonic model id"), + "profile": ("voice-profile", "FM_VOICE_PROFILE", "AWS profile"), + "voice": ("voice-id", "FM_VOICE_ID", "output voice"), +} + +IN_RATE = 16000 +OUT_RATE = 24000 +# 3200 bytes is 100 ms at 16 kHz 16-bit mono, the chunk size earlier prototype +# work measured its timings with. Keeping it identical keeps those comparable. +CHUNK = 3200 +BYTES_PER_MS_IN = IN_RATE * 2 // 1000 + +# Push-to-talk supplies no trailing silence, and trap 2 above means a turn with +# none is never answered. 400 ms is the measured floor plus one chunk of margin; +# see docs/voice-relay.md for the runs behind it. +TAIL_MS = 400 + +SYSTEM_PROMPT = ( + "You are the captain's voice assistant. You are NOT the first mate, and you " + "must never claim to be. You stand in front of the first mate and you are " + "the captain's spoken way of reaching it.\n" + "\n" + "When the captain asks how things are going, what is in flight, what is " + "waiting on them, or whether anything is ready to review, call " + "get_fleet_status and answer from what it returns. Give counts and at most a " + "couple of names. Never invent a number, a name or a pull request. If the " + "tool says detail is withheld, say the detail is not available by voice.\n" + "\n" + "Call get_fleet_status every single time the captain asks, including when " + "they asked a moment ago. The records change while you are talking, and an " + "answer repeated from memory is a stale answer given confidently, which is " + "worse than a slow one.\n" + "\n" + "When the captain asks for actual work, anything that would change code, " + "open a pull request, investigate a bug, or start a job, you do not do it " + "and you do not pretend to. Say out loud that you are handing it to the " + "first mate, then call hand_over_to_firstmate with the captain's request in " + "their own words. Then confirm it is queued. Never say you have done, " + "started, fixed or built anything yourself.\n" + "\n" + "Speak in one or two short sentences. You are being listened to, not read." +) + +TOOLS = {"tools": [ + {"toolSpec": { + "name": "get_fleet_status", + "description": ( + "Read the first mate's durable records: how many jobs are in " + "flight, how many decisions are waiting on the captain, how many " + "pull requests are open, and the names of a few of them."), + "inputSchema": {"json": json.dumps( + {"type": "object", "properties": {}, "required": []})}, + }}, + {"toolSpec": { + "name": "hand_over_to_firstmate", + "description": ( + "Hand a request for real work to the first mate, which will pick it " + "up at its next check. Use this for anything you cannot answer from " + "the records. It queues the request and does not do the work."), + "inputSchema": {"json": json.dumps({ + "type": "object", + "properties": {"request": { + "type": "string", + "description": "The captain's request, in the captain's own words.", + }}, + "required": ["request"], + })}, + }}, +]} + + +def log(enabled, message): + if enabled: + sys.stderr.write("relay: {}\n".format(message)) + sys.stderr.flush() + + +def widen_path(): + """Put the toolbox directories on PATH, as bin/fm-inbox.sh does and for the same reason. + + `ssh host command` gets no login shell, so it gets no ~/.toolbox/bin. The + sandbox profile's credential_process is the bare word `ada`, so without this + the relay starts, connects to nothing, and reports a missing file. That is + the normal way this relay is launched, so it has to hold here. + """ + extra = [os.path.expanduser(p) for p in ("~/.toolbox/bin", "~/.local/bin")] + parts = os.environ.get("PATH", "").split(os.pathsep) + added = [p for p in extra if os.path.isdir(p) and p not in parts] + if added: + os.environ["PATH"] = os.pathsep.join(added + parts) + + +# A credential that states an expiry this interpreter cannot read. The +# credential itself is fine; only its deadline is unknown, and that is not the +# same thing as not having one. +EXPIRY_UNKNOWN = object() + +# Where a set of credentials came from. The difference matters to the cache: the +# profile can be asked again for fresher credentials, and the environment of an +# already-running process cannot. +FROM_ENVIRONMENT = "environment" +FROM_PROFILE = "profile" + + +class CredentialError(Exception): + """No usable AWS credentials, and the caller is told which door was tried. + + An ordinary exception rather than SystemExit, because credentials are now + resolved lazily and a refresh can therefore land in the middle of a turn. + SystemExit would walk straight through the turn boundary in + handle_uplink_frame and end the relay over one bad refresh, which is the + failure that boundary exists to absorb. + """ + + +def _expires_at(stamp): + """Return the expiry as epoch seconds, None when there is none, or EXPIRY_UNKNOWN. + + The two failure shapes mean opposite things and must not collapse into one. + No Expiration at all is a credential that does not expire. An Expiration + that will not parse, such as an offset written +0000 on an interpreter older + than 3.11, is a credential that does expire at a moment this process cannot + read, and treating that as "never" would cache it past its real deadline and + fail every session from then on. + """ + if not stamp: + return None + try: + when = datetime.datetime.fromisoformat(str(stamp).replace("Z", "+00:00")) + except ValueError: + return EXPIRY_UNKNOWN + if when.tzinfo is None: + when = when.replace(tzinfo=datetime.timezone.utc) + return when.timestamp() + + +def ambient_credentials(verbose=False, margin=0, only_source=False): + """Return (credentials, expiry) from the environment, or None if it has none to give. + + None means "ask the profile instead", and there are three ways to get it. + An environment with no key id at all is the ordinary ssh case. One carrying + a key id without a secret beside it is a half-set variable, which is a + mistake worth naming rather than a KeyError from inside a worker thread. + And one whose AWS_CREDENTIAL_EXPIRATION has passed, or passes within margin + seconds, is no longer usable: os.environ cannot get fresher values while + this process runs, so the only way forward is the profile. + + only_source says there is no profile to ask, which changes what a passed + deadline means. The environment is then the only place a credential can come + from, so a stale one is still the best answer available, and refusing it + would end a live conversation over something only the operator can refresh. + AWS says so itself if the credential really is dead. An environment with no + keys in it at all is a refusal either way. + + Temporary credentials with no stated deadline are reported as + EXPIRY_UNKNOWN rather than as eternal, because a session token always has a + deadline whether or not the shell that exported it said so. + """ + key = os.environ.get("AWS_ACCESS_KEY_ID") + if not key: + return None + secret = os.environ.get("AWS_SECRET_ACCESS_KEY") + if not secret: + log(verbose, "AWS_ACCESS_KEY_ID is set with no AWS_SECRET_ACCESS_KEY " + "beside it, so the environment is being ignored") + return None + token = os.environ.get("AWS_SESSION_TOKEN") + expires = _expires_at(os.environ.get("AWS_CREDENTIAL_EXPIRATION")) + if expires is None and token: + expires = EXPIRY_UNKNOWN + if (not only_source and expires not in (None, EXPIRY_UNKNOWN) + and time.time() + margin >= expires): + log(verbose, "the credentials in the environment have expired") + return None + log(verbose, "using credentials already in the environment") + return { + "aws_access_key_id": key, + "aws_secret_access_key": secret, + "aws_session_token": token, + }, expires + + +def profile_credentials(profile, verbose=False): + """Return (credentials, expiry) exported from an AWS profile, or refuse by name.""" + if not profile: + raise CredentialError( + "no credentials in the environment and no AWS profile configured: " + "write one into config/voice-profile, set FM_VOICE_PROFILE, or " + "export AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY") + log(verbose, "exporting credentials from profile {}".format(profile)) + widen_path() + done = subprocess.run( + ["aws", "configure", "export-credentials", "--profile", profile, + "--format", "process"], + # The relay's own stdin is the captain's audio in --serve mode. A child + # that read it would eat frames and desynchronise the uplink, so no + # child gets it. + stdin=subprocess.DEVNULL, + capture_output=True, text=True, timeout=60, check=False) + if done.returncode != 0: + raise CredentialError( + "could not get credentials for profile {}: {}".format( + profile, (done.stderr or done.stdout).strip())) + blob = json.loads(done.stdout) + return { + "aws_access_key_id": blob["AccessKeyId"], + "aws_secret_access_key": blob["SecretAccessKey"], + "aws_session_token": blob.get("SessionToken"), + }, _expires_at(blob.get("Expiration")) + + +def resolve_credentials(profile, verbose=False, margin=0, allow_ambient=True): + """Return (credentials, expiry, source), preferring the environment when allowed. + + The sandbox profile's credential_process costs about a second, so ambient + credentials win while they are usable. It also blocks the caller for that + second, so Credentials below owns when this runs and keeps it out of a turn. + The source is reported because only one of the two can be asked again for + something fresher, and the cache has to know which it is holding. + + A relay with no profile at all is a supported shape, so the environment gets + a second look when there is nothing to escalate to. Giving up on the only + source there is would turn "these credentials are getting old" into "this + relay is over", which is a worse answer than handing over keys that AWS can + refuse for itself. + """ + if allow_ambient: + ambient = ambient_credentials(verbose, margin) + if ambient is None and not profile: + ambient = ambient_credentials(verbose, margin, only_source=True) + if ambient is not None: + log(verbose, "keeping the credentials in the environment anyway: " + "there is no profile to fall back to") + if ambient is not None: + return ambient[0], ambient[1], FROM_ENVIRONMENT + try: + creds, expires = profile_credentials(profile, verbose) + except CredentialError: + # The profile was the escalation and it refused. Whatever the environment + # still holds is older than we would like, which is why the profile was + # asked at all, but it is a real answer and AWS refuses it for itself if + # it is dead. Ending the conversation instead would spend the captain's + # session on a preference. An environment with nothing in it re-raises. + ambient = ambient_credentials(verbose, margin, only_source=True) + if ambient is None: + raise + log(verbose, "the profile refused, so falling back to the credentials " + "still in the environment") + return ambient[0], ambient[1], FROM_ENVIRONMENT + return creds, expires, FROM_PROFILE + + +class Credentials: + """The relay's credentials, resolved once and shared by every session it opens. + + A session is rebuilt for every turn, on purpose and for a measured reason + (see renew), so resolving per session would charge the credential_process + second to each turn after the first. The relay resolves once at start and + every later session reuses that answer, so a reconnect costs a reconnect + and not a credential fetch. + + Credentials that carry an expiry are refreshed a few minutes ahead of it, + because a relay left running outlives them. One whose expiry cannot be read + is held for that same margin and no longer, so an unreadable deadline costs + an occasional resolution rather than every session after the deadline. The + margin is handed to the resolver as well, because credentials taken from the + environment cannot be refreshed in place and have to be abandoned for the + profile once they are that close to the end. + + That abandonment has to be remembered, not just decided. os.environ never + gets fresher values while this process runs, so re-reading it after giving up + on an ambient credential would hand back the same stale keys forever and the + bound above would be a bound in name only. Once an ambient answer is spent, + this asks the profile from then on. + + An ambient answer is only ever spent when there IS a profile to spend it on. + With no profile the environment is the only source, so the bound becomes a + re-read of it rather than an escalation: a relay configured that way keeps + answering, and whether the keys still work is between AWS and the operator + who exported them. + + Every resolution, the first one included, runs in a worker thread, so the + event loop keeps reading the captain's audio while it happens. + """ + + REFRESH_MARGIN = 300 + + def __init__(self, profile, verbose=False): + self.profile = profile + self.verbose = verbose + self._creds = None + self._expires = None + self._source = None + self._resolved = None + self._ambient_spent = False + self._lock = asyncio.Lock() + + def _usable(self): + if self._creds is None: + return False + if self._expires is EXPIRY_UNKNOWN: + return time.monotonic() - self._resolved < self.REFRESH_MARGIN + if self._expires is None: + return True + return time.time() + self.REFRESH_MARGIN < self._expires + + async def get(self): + async with self._lock: + if not self._usable(): + spend = self._source == FROM_ENVIRONMENT and bool(self.profile) + creds, expires, source = await asyncio.to_thread( + resolve_credentials, self.profile, self.verbose, + self.REFRESH_MARGIN, not (self._ambient_spent or spend)) + # Latched only now, and only if the profile is what answered. A + # profile that cannot answer raises out of the line above or is + # answered for by the environment, and latching either of those + # would abandon credentials this process is still holding on the + # strength of a source that just refused, turning one failed + # refresh into every later turn. + if spend and source == FROM_PROFILE: + log(self.verbose, "the credentials from the environment are " + "spent; asking the profile from now on") + self._ambient_spent = True + self._creds, self._expires, self._source = creds, expires, source + self._resolved = time.monotonic() + return dict(self._creds) + + +class Downlink: + """Write frames to the client from one dedicated thread. + + A blocking write to a stalled SSH channel must not stop the relay reading + the captain's audio or the model's output, and the moment a reply byte is + actually handed to the connection is the only honest place to timestamp it. + Both of those want the writes off the event loop, so they live here. + """ + + def __init__(self, stream): + self._stream = stream + self._queue = queue.Queue() + self._first_audio = None + self._lock = threading.Lock() + self._thread = threading.Thread(target=self._run, daemon=True) + self._thread.start() + + def _run(self): + writer = frame.Writer(self._stream) + while True: + item = self._queue.get() + if item is None: + return + kind, payload = item + try: + writer.send(kind, payload) + except (BrokenPipeError, ValueError, OSError): + return + if kind == frame.AUDIO: + with self._lock: + if self._first_audio is None: + self._first_audio = time.monotonic() + + def send(self, kind, payload=b""): + self._queue.put((kind, payload)) + + def send_json(self, kind, obj): + self.send(kind, json.dumps(obj, separators=(",", ":")).encode("utf-8")) + + def arm_turn(self): + """Forget the previous turn's first-audio mark.""" + with self._lock: + self._first_audio = None + + def first_audio(self): + with self._lock: + return self._first_audio + + def close(self): + self._queue.put(None) + self._thread.join(timeout=5) + + +class Session: + """One Nova Sonic bidirectional session, plus the turn bookkeeping around it.""" + + def __init__(self, options, down, credentials): + self.options = options + self.down = down + self.credentials = credentials + self.verbose = options.verbose + self.prompt = str(uuid.uuid4()) + self.stream = None + self.reader_task = None + self.audio_content = None + self.turn = {} + self.tool_calls = 0 + # Replies this session has finished. One is the most it should ever + # deliver; see serve() for why a second turn gets a new session. + self.replies = 0 + # Set when a call into the model raised, which makes this session spent + # whether or not it ever answered. fail_turn owns it. + self.failed = False + # Set while close() is deliberately tearing this session down, so the + # reader can tell a stream that went away because we ended it from one + # that went away on its own. + self.closing = False + # Which tools ran, in order. The handover boundary is the whole point of + # this relay, so "it called hand_over_to_firstmate and did not answer + # for firstmate" has to be evidence in the run record, not an inference + # from a count. + self.tool_names = [] + self.ended = asyncio.Event() + self.turn_done = asyncio.Event() + self.home = options.home or records.default_home() + self.scope = options.scope or records.read_scope(self.home) + self.root = os.path.dirname(os.path.abspath(__file__)) + + # ---------------------------------------------------------------- protocol + + def _event(self, obj): + from aws_sdk_bedrock_runtime.models import ( + BidirectionalInputPayloadPart, + InvokeModelWithBidirectionalStreamInputChunk) + return InvokeModelWithBidirectionalStreamInputChunk( + value=BidirectionalInputPayloadPart( + bytes_=json.dumps({"event": obj}).encode())) + + async def _send(self, obj): + await self.stream.input_stream.send(self._event(obj)) + + async def start(self): + from aws_sdk_bedrock_runtime.client import ( + AsyncBedrockRuntimeClient, + InvokeModelWithBidirectionalStreamOperationInput) + from aws_sdk_bedrock_runtime.config import AsyncBedrockRuntimeConfig + + creds = await self.credentials.get() + began = time.monotonic() + config = await AsyncBedrockRuntimeConfig.resolve( + endpoint_uri="https://bedrock-runtime.{}.amazonaws.com".format( + self.options.region), + region=self.options.region, **creds) + client = AsyncBedrockRuntimeClient(config=config) + self.stream = await client.invoke_model_with_bidirectional_stream( + InvokeModelWithBidirectionalStreamOperationInput( + model_id=self.options.model)) + self.connect_seconds = round(time.monotonic() - began, 3) + self.reader_task = asyncio.create_task(self._read_model()) + + await self._send({"sessionStart": {"inferenceConfiguration": { + "maxTokens": 512, "topP": 0.9, "temperature": 0.7}}}) + await self._send({"promptStart": { + "promptName": self.prompt, + "textOutputConfiguration": {"mediaType": "text/plain"}, + "audioOutputConfiguration": { + "mediaType": "audio/lpcm", "sampleRateHertz": OUT_RATE, + "sampleSizeBits": 16, "channelCount": 1, + "voiceId": self.options.voice, "encoding": "base64", + "audioType": "SPEECH"}, + "toolUseOutputConfiguration": {"mediaType": "application/json"}, + "toolConfiguration": TOOLS}}) + content = str(uuid.uuid4()) + await self._send({"contentStart": { + "promptName": self.prompt, "contentName": content, "type": "TEXT", + "interactive": True, "role": "SYSTEM", + "textInputConfiguration": {"mediaType": "text/plain"}}}) + await self._send({"textInput": { + "promptName": self.prompt, "contentName": content, + "content": SYSTEM_PROMPT}}) + await self._send({"contentEnd": { + "promptName": self.prompt, "contentName": content}}) + log(self.verbose, "session up in {}s, read scope {}".format( + self.connect_seconds, self.scope)) + + async def close(self): + self.closing = True + if self.stream is None: + return + try: + if self.audio_content: + await self._send({"contentEnd": { + "promptName": self.prompt, "contentName": self.audio_content}}) + self.audio_content = None + await self._send({"promptEnd": {"promptName": self.prompt}}) + await self._send({"sessionEnd": {}}) + await self.stream.input_stream.close() + except Exception as exc: # noqa: BLE001 + log(self.verbose, "close: {}: {}".format(type(exc).__name__, exc)) + if self.reader_task is not None: + try: + # gather collects a reader that died on its own instead of + # re-raising it here, the same way the sends above are absorbed. + # Awaiting a failed task raises on EVERY await, and close() is + # the first statement of renew and of serve's finally, so a + # close that re-raises is the difference between one failed turn + # and a relay that can never build another session or even say + # goodbye to the client. + await asyncio.wait_for( + asyncio.gather(self.reader_task, return_exceptions=True), + timeout=10) + except (asyncio.TimeoutError, asyncio.CancelledError): + pass + + # ------------------------------------------------------------------ uplink + + async def talk_start(self): + """Open an audio block for a new turn, if one is not already open.""" + if self.audio_content is not None: + return + self.audio_content = str(uuid.uuid4()) + self.turn = {"began": time.monotonic()} + self.tool_calls = 0 + self.tool_names = [] + self.turn_done.clear() + self.down.arm_turn() + await self._send({"contentStart": { + "promptName": self.prompt, "contentName": self.audio_content, + "type": "AUDIO", "interactive": True, "role": "USER", + "audioInputConfiguration": { + "mediaType": "audio/lpcm", "sampleRateHertz": IN_RATE, + "sampleSizeBits": 16, "channelCount": 1, + "audioType": "SPEECH", "encoding": "base64"}}}) + log(self.verbose, "talk start") + + async def audio(self, pcm): + """Forward captured audio, chunked the way the measurements were taken. + + Audio with no turn open is dropped rather than opening one. Both listen + modes send a talk start before any audio, so this never fires in ordinary + use, but the capture callback races the key release: a chunk already past + the gate check can reach the relay behind the talk end. Opening a block + for it would append the captain's stray tenth of a second to a session + that is already generating its reply, which is the unconditional barge-in + the per-turn reconnect exists to avoid, and it would leave that block open + so the next turn skipped its own reset and its first-audio mark. + """ + if self.audio_content is None: + log(self.verbose, "dropping {} bytes of audio that arrived with no " + "turn open".format(len(pcm))) + return + for at in range(0, len(pcm), CHUNK): + await self._send({"audioInput": { + "promptName": self.prompt, "contentName": self.audio_content, + "content": base64.b64encode(pcm[at:at + CHUNK]).decode()}}) + + async def talk_end(self): + """Close the turn: pad with silence, then close the audio block. + + The padding is trap 2: a clip with no trailing silence is truncated and + never answered. It is a CONTENT requirement rather than a time one. The + padding is sent unpaced, so measured against tail_ms 200 through 800 it + cost no wall clock at all; what it buys is the model deciding the + captain has stopped. 400 ms is therefore free margin above the 200 ms + floor where answers first appear. + + The clock is still taken before the padding, because that instant is + when the captain actually stopped talking and every number this build + reports has to be measured from there. + """ + if self.audio_content is None: + return + self.turn["talk_end"] = time.monotonic() + tail = self.options.tail_ms * BYTES_PER_MS_IN + if tail: + await self.audio(b"\x00" * tail) + await self._send({"contentEnd": { + "promptName": self.prompt, "contentName": self.audio_content}}) + self.audio_content = None + log(self.verbose, "talk end, {} ms of silence appended".format( + self.options.tail_ms)) + + # ---------------------------------------------------------------- downlink + + def _mark(self, name, at=None): + now = at if at is not None else time.monotonic() + self.turn.setdefault(name, now) + base = self.turn.get("talk_end") + if base is None: + return + self.down.send_json(frame.MARK, { + "mark": name, + "since_talk_end": round(now - base, 3), + "tool_calls": self.tool_calls, + }) + + # Every question worth asking about a session is a question about the order + # of these events and the stop reason on them, so --verbose prints that + # order. audioOutput and usageEvent are left out because they repeat many + # times per reply and bury everything else. + TRACE_SKIP = ("audioOutput", "usageEvent") + + def _trace(self, event): + for name, body in event.items(): + if name in self.TRACE_SKIP: + continue + detail = "" + if isinstance(body, dict): + bits = [(k, body.get(k)) for k in ("type", "role", "stopReason") + if body.get(k)] + detail = "".join(" {}={}".format(k, v) for k, v in bits) + log(True, "event {}{}".format(name, detail)) + + async def _read_model(self): + """Read the model's events until the stream ends or fails, and report which. + + Handling an event reaches back into the model, to answer a tool call, so + it can fail on its own rather than only the read can. Either way this + session is finished, and the finally below is the one thing that must + still happen: --self-test waits on turn_done for the length of a turn, + and the next talk key reads ended to decide whether this session can + still be used. Leaving them clear is what turned one dropped stream into + a relay that never answered again. + + The two ways out are not the same event and are not reported the same + way. A stream that simply ends is the end of a session and nothing more, + so it is named as that and not as a failure. A stream that raises, here + or under an event handler, is this turn failing, so it goes through + fail_turn and reaches the captain. + + Either way the client is told, once, because either way it is waiting on + a turn that is not coming and a notice is the only thing that releases it. + The end is announced HERE rather than from the serve loop because this is + the one moment it happens: the flag it sets stays set for every later + frame of the same key press, so a loop that announced it would say it ten + times a second while the captain was still speaking. + + Neither is a stream that went away because close() asked it to: renew + closes the old session on every single turn, so announcing that would put + a failure notice in front of the captain on every ordinary turn. + """ + broke = None + try: + while True: + try: + out = await self.stream.await_output() + result = await out[1].receive() + except Exception as exc: # noqa: BLE001 + log(self.verbose, "model stream dropped: {}: {}".format( + type(exc).__name__, exc)) + broke = exc + break + if result is None: + break + raw = result.value.bytes_ + if not raw: + continue + try: + event = json.loads(raw.decode()).get("event", {}) + except ValueError: + continue + try: + await self._handle(event) + except Exception as exc: # noqa: BLE001 + log(self.verbose, "handling {} failed: {}: {}".format( + ", ".join(event) or "an event", type(exc).__name__, exc)) + broke = exc + break + finally: + # Neither is said when the uplink has already named this turn: the + # frame that broke the model usually breaks the reader an instant + # later, and the captain hears about one turn once. + if not self.closing and not self.failed: + if broke is not None: + fail_turn(self, self.down, broke) + else: + self.down.send_json( + frame.NOTICE, {"event": "session-ended"}) + self.ended.set() + self.turn_done.set() + + async def _handle(self, event): + if self.verbose: + self._trace(event) + + if "userSpeechEnd" in event: + # Open microphone: the model's own detector, not a talk-end frame, + # is what ends the turn, so the clock starts here instead. + self.turn.setdefault("talk_end", time.monotonic()) + log(self.verbose, "model reports the captain stopped speaking") + + if "audioOutput" in event: + pcm = base64.b64decode(event["audioOutput"].get("content", "")) + if pcm: + if "first_audio" not in self.turn: + self._mark("first_audio") + self.down.send(frame.AUDIO, pcm) + + if "textOutput" in event: + text = event["textOutput"].get("content", "") + role = event["textOutput"].get("role", "") + if text: + self.down.send_json(frame.TEXT, {"role": role, "text": text}) + log(self.verbose, "{}: {}".format(role.lower(), text[:120])) + if '"interrupted"' in text and "true" in text: + # Informational only. Stopping playback mid-sentence is + # barge-in, which is step three of the design, not this build. + self.down.send_json(frame.NOTICE, {"event": "interrupted"}) + + if "toolUse" in event: + self._mark("tool_use") + self.tool_calls += 1 + self.tool_names.append(event["toolUse"].get("toolName", "")) + await self._run_tool(event["toolUse"]) + + if "contentEnd" in event: + stop = event["contentEnd"].get("stopReason") + if stop == "INTERRUPTED": + self.down.send_json(frame.NOTICE, {"event": "interrupted"}) + if stop == "END_TURN": + # Trap 1: this, not completionEnd, is the end of the reply. + self._mark("reply_end") + # first_audio above is stamped when the model event is decoded. + # The Downlink knows the later instant when that audio reached + # the connection, which is the one the captain hears, so it is + # reported too rather than measured and thrown away. It can only + # be read once the frame is out, hence here and not there. + wire = self.down.first_audio() + if wire is not None: + self._mark("first_audio_wire", wire) + self.replies += 1 + self.turn_done.set() + + # -------------------------------------------------------------------- tools + + async def _run_tool(self, call): + name = call.get("toolName", "") + use_id = call.get("toolUseId") + raw = call.get("content") or "{}" + try: + arguments = json.loads(raw) if isinstance(raw, str) else dict(raw) + except ValueError: + arguments = {} + log(self.verbose, "tool {} {}".format(name, arguments)) + + try: + if name == "get_fleet_status": + # Off the loop like the handover below it: the model is told to + # call this on every question, and its directory and file reads + # would otherwise stop the relay reading the captain's audio. + result = await asyncio.to_thread( + records.fleet_status, self.home, self.scope) + elif name == "hand_over_to_firstmate": + request = (arguments.get("request") or "").strip() + result = await asyncio.to_thread( + records.queue_request, request, self.home, self.root) + self.down.send_json(frame.NOTICE, { + "event": "queued", "request": request, + "note_id": result.get("note_id", "")}) + else: + result = {"error": "no such tool: {}".format(name)} + except records.RecordError as exc: + result = {"error": str(exc)} + except Exception as exc: # noqa: BLE001 + result = {"error": "{}: {}".format(type(exc).__name__, exc)} + + content = str(uuid.uuid4()) + await self._send({"contentStart": { + "promptName": self.prompt, "contentName": content, "type": "TOOL", + "interactive": False, "role": "TOOL", + "toolResultInputConfiguration": { + "toolUseId": use_id, "type": "TEXT", + "textInputConfiguration": {"mediaType": "text/plain"}}}}) + await self._send({"toolResult": { + "promptName": self.prompt, "contentName": content, + "content": json.dumps(result)}}) + await self._send({"contentEnd": { + "promptName": self.prompt, "contentName": content}}) + self._mark("tool_answered") + + +def fail_turn(session, down, exc): + """Mark a session spent and name this turn's failure to the client. + + One place, because both ends of the relay can break a turn and the captain + should not be able to tell which by whether they heard anything. Every part + of it is for a different reader. The mark is what the next talk key reads to + build a replacement instead of talking into a session that is already gone. + The notice is what the captain gets, and it is the only thing that releases a + client waiting for a reply, so a failure that is merely marked costs them + their whole timeout and leaves a record saying the turn went unanswered + without saying why. The reason on the turn is for --self-test, which has no + client to notice anything. + """ + reason = "{}: {}".format(type(exc).__name__, exc) + session.failed = True + session.turn["failed"] = reason + down.send_json(frame.NOTICE, {"event": "turn-failed", "error": reason}) + + +async def renew(session, options, down): + """Replace a session that has already answered once, and return the new one. + + MEASURED, and the reason this exists: a second user audio block in a session + that has already spoken is treated as barge-in, unconditionally. The model + raises INTERRUPTED the instant the block opens, and waiting does not help. + Six consecutive turns were tried with no wait, with a wait until the reply's + audio had all arrived, and with a wait of the reply's full spoken duration + after that; every one of those interrupted every second turn. Worse, an + interrupted turn that calls a tool is then lost outright: the model asks for + the tool, takes the result, and never answers. + + Reconnecting instead costs 0.02 seconds, measured, and it happens when the + captain presses the talk key rather than while they are waiting for a reply, + so it is invisible. What it gives up is conversational memory: each turn + starts fresh, so the captain cannot say "and what about that one". Carrying + context across turns means handling barge-in properly, which is step three of + the design, not this build. It also means the system prompt is sent once per + turn rather than once per session, which is the small cost of the trade. + """ + log(options.verbose, "renewing the session for a new turn") + await session.close() + fresh = Session(options, down, session.credentials) + try: + await fresh.start() + except BaseException: + # start() creates the reader task before it sends anything, so a + # reconnect that fails part way leaves a live task holding an open + # bidirectional stream. Nothing would ever close it, and it would keep + # writing into the shared Downlink, so each retry would strand one more. + await fresh.close() + raise + down.send_json(frame.NOTICE, { + "event": "renewed", "connect_seconds": fresh.connect_seconds}) + return fresh + + +async def read_uplink_frame(reader): + """Return the next (kind, payload) the client sent, or raise on a bad header. + + The header is checked before the payload is read, not after. A + desynchronised uplink offers a length of up to 4 GiB, and waiting for that + many bytes is a hang where the wire format promises a loud error, with the + captain sitting in front of a client that will never answer. + """ + head = await reader.readexactly(frame.HEADER.size) + kind, length = frame.HEADER.unpack(head) + frame.check_header(kind, length) + payload = await reader.readexactly(length) if length else b"" + return kind, payload + + +async def handle_uplink_frame(kind, payload, session, options, down): + """Act on one frame from the client. Returns (session to use next, keep serving). + + Every branch below reaches the model, and the model side fails on its own: + a reconnect can be throttled, a token can expire between turns, a stream can + drop. Because the relay rebuilds the session on every turn by design, one + such failure would otherwise leave the loop, end the relay with a traceback + on the stderr the client inherits, and cost the captain a whole session for + a single bad reconnect. Instead it is named in a notice and the session is + marked spent, so the next press of the talk key builds a new one and tries + again. A failure the model cannot recover from is named once per turn, which + is a captain who can hear what is wrong rather than a dead pipe. + + Once per TURN and not once per frame: the captain is still holding the talk + key when the failure lands, and the rest of that key press is another thirty + audio frames a second apart in tenths. Reporting each one would put ten + identical lines a second in front of the captain and keep calling into a + session that is already gone, so the remainder of a failed turn is dropped + where it arrives. + """ + if kind == frame.QUIT: + return session, False + if session.failed and kind != frame.TALK_START: + return session, True + try: + if kind == frame.TALK_START: + if session.failed or session.replies or session.ended.is_set(): + session = await renew(session, options, down) + await session.talk_start() + elif kind == frame.AUDIO: + await session.audio(payload) + elif kind == frame.TALK_END: + await session.talk_end() + else: + log(options.verbose, "ignoring uplink kind {!r}".format(kind)) + except Exception as exc: # noqa: BLE001 + log(options.verbose, "turn failed: {}: {}".format( + type(exc).__name__, exc)) + fail_turn(session, down, exc) + return session, True + + +async def serve(options): + """Relay frames between the client on stdin/stdout and the model sessions behind it. + + Three things end this, and nothing else does: the client's QUIT frame, the + client closing the connection, and an uplink that has stopped being a frame + stream. In particular a model session ending is not one of them. It happens + on its own, mid-conversation, and the next talk key builds a replacement + through the same path every ordinary turn already uses, at a measured cost of + 0.02 s. A renew that cannot be made is spoken to the captain by fail_turn, so + the loud failure is the one they get; ending the relay here would instead + leave them speaking a whole question into nothing. + """ + loop = asyncio.get_running_loop() + reader = asyncio.StreamReader() + await loop.connect_read_pipe( + lambda: asyncio.StreamReaderProtocol(reader), sys.stdin.buffer) + # Ahead of every frame, so a login shell that prints a banner on stdout + # cannot desynchronise the client. See fm_voice_frame.MAGIC. + sys.stdout.buffer.write(frame.MAGIC) + sys.stdout.buffer.flush() + down = Downlink(sys.stdout.buffer) + session = Session(options, down, Credentials(options.profile, options.verbose)) + await session.start() + down.send_json(frame.NOTICE, { + "event": "ready", "model": options.model, "region": options.region, + "read_scope": session.scope, "tail_ms": options.tail_ms, + "connect_seconds": session.connect_seconds}) + + status = 0 + # A fault the client cannot see for itself, held so the teardown can name it + # down the connection as well as on this stderr. Nothing is captured on the + # branch above it: there the client is the end that went away, and there is + # nobody left to tell. + reason = None + try: + while True: + kind, payload = await read_uplink_frame(reader) + session, serving = await handle_uplink_frame( + kind, payload, session, options, down) + if not serving: + break + except (asyncio.IncompleteReadError, ConnectionResetError): + log(options.verbose, "client closed the connection") + except frame.FrameError as exc: + sys.stderr.write( + "fm-voice-relay: the uplink is not a frame stream any more: {}\n" + .format(exc)) + reason = "{}: {}".format(type(exc).__name__, exc) + status = 2 + finally: + # On fail_turn's shape and before close(), which awaits the model stream + # and can be slow or raise. session.close() also sets closing, which + # silences the reader's own notice, so a goodbye on its own would leave + # the captain's turn record saying only that the turn went unanswered + # while the reason for it sat on a stderr no run file quotes. + if reason is not None: + down.send_json(frame.NOTICE, {"event": "turn-failed", + "error": reason}) + await session.close() + down.send(frame.BYE) + down.close() + return status + + +async def self_test(options): + """Feed one PCM file through a real session and report the timings.""" + with open(options.self_test, "rb") as handle: + pcm = handle.read() + + class Sink: + """Stands in for the client, counting reply audio and timing its arrival. + + There is no connection here and no writer thread: this stamps its arrival + inline, in the same coroutine that decoded the model event. So the wire + hand-off Downlink times on the --serve path does not exist in this mode, + and the record below reports no figure for it rather than reporting one + that would be zero because of how this stub is built. The first_audio + figure it does report is the model event, which is real in both modes. + """ + + def __init__(self): + self.first = None + self.bytes = 0 + self.heard = [] + self.said = [] + self.notices = [] + + def send(self, kind, payload=b""): + if kind == frame.AUDIO: + if self.first is None: + self.first = time.monotonic() + self.bytes += len(payload) + + def send_json(self, kind, obj): + # The transcript is the only way to check the two things that matter + # about a spoken answer: that the words were heard correctly, and + # that the agent handed real work over instead of claiming it. + if kind == frame.TEXT: + text = (obj.get("text") or "").strip() + if not text or text.startswith("{"): + return + if obj.get("role") == "USER": + self.heard.append(text) + elif obj.get("role") == "ASSISTANT": + self.said.append(text) + elif kind == frame.NOTICE: + self.notices.append(obj.get("event", "")) + + def arm_turn(self): + self.first = None + + def first_audio(self): + return self.first + + sink = Sink() + session = Session(options, sink, Credentials(options.profile, options.verbose)) + await session.start() + await session.talk_start() + # Paced at real time, because a file pushed as fast as the socket accepts it + # would measure the socket rather than the conversation. + for at in range(0, len(pcm), CHUNK): + await session.audio(pcm[at:at + CHUNK]) + await asyncio.sleep(CHUNK / (IN_RATE * 2.0)) + await session.talk_end() + try: + await asyncio.wait_for(session.turn_done.wait(), + timeout=options.turn_timeout) + except asyncio.TimeoutError: + session.turn["timeout"] = True + await session.close() + + base = session.turn.get("talk_end") + + def since(name): + at = session.turn.get(name) + if at is None or base is None: + return None + return round(at - base, 3) + + # A negative figure means the model started answering before this end of the + # stream said the turn was over, which happens when the clip handed in + # ALREADY ends in silence: the model's own endpoint detector fires part way + # through that silence while the file is still being streamed at real time. + # The reply is genuinely fast in that case but the number is meaningless, + # because it is measured from the wrong instant. Feed --self-test a clip that + # ends on speech and let --tail-ms add the silence. This is flagged rather + # than silently recorded, because a negative in a results file gets averaged + # into a report by someone who was not here. + early = [n for n in ("tool_use", "first_audio", "reply_end") + if (since(n) or 0) < 0] + if early: + sys.stderr.write( + "fm-voice-relay: {} came in before the end of the clip, so these " + "timings are measured from the wrong instant. The clip already ends " + "in silence; pass one that ends on speech and use --tail-ms.\n" + .format(", ".join(early))) + + print(json.dumps({ + "mode": "self-test", + "model": options.model, + "region": options.region, + "read_scope": session.scope, + "input_seconds": round(len(pcm) / float(IN_RATE * 2), 3), + "tail_ms": options.tail_ms, + "connect_seconds": session.connect_seconds, + "tool_calls": session.tool_calls, + "tool_names": session.tool_names, + "tool_use_s": since("tool_use"), + "first_audio_s": since("first_audio"), + "reply_end_s": since("reply_end"), + "reply_audio_seconds": round(sink.bytes / float(OUT_RATE * 2), 3), + "answered": sink.bytes > 0, + "timed_out": bool(session.turn.get("timeout")), + # Named the same as the client's turn record, and here for the same + # reason: a record that says only that the turn was not answered invites + # someone who was not here to average an infrastructure failure into a + # latency figure. + "relay_error": session.turn.get("failed"), + "clock_unusable": early, + "heard": " ".join(sink.heard), + "said": " ".join(sink.said), + "notices": sink.notices, + })) + return 0 if sink.bytes > 0 else 1 + + +def parse_args(argv): + parser = argparse.ArgumentParser( + prog="fm-voice-relay.py", add_help=True, + description=__doc__.splitlines()[0]) + parser.add_argument("--serve", action="store_true") + parser.add_argument("--self-test", metavar="FILE") + parser.add_argument("--region", + help="Bedrock region; required, from config/voice-region " + "or FM_VOICE_REGION when not given here") + parser.add_argument("--model", + help="Nova Sonic model id; required, from config/voice-model " + "or FM_VOICE_MODEL when not given here") + parser.add_argument("--profile", + help="AWS profile; optional, from config/voice-profile or " + "FM_VOICE_PROFILE, and empty means the credentials " + "already in the environment") + parser.add_argument("--voice", + help="output voice; from config/voice-id or FM_VOICE_ID, " + "default {}".format(VOICE)) + parser.add_argument("--home") + parser.add_argument("--scope", choices=records.SCOPES) + parser.add_argument("--tail-ms", type=int, default=TAIL_MS) + parser.add_argument("--turn-timeout", type=float, default=40.0, + help="seconds --self-test waits for a reply") + parser.add_argument("--verbose", action="store_true") + options = parser.parse_args(argv) + if options.tail_ms < 0: + parser.error("--tail-ms cannot be negative") + return options + + +def resolve_settings(options): + """Fill in what this home configures, refusing rather than guessing. + + Deliberately not part of parse_args: --help and the flags this file can + answer for itself must work in a home that has configured nothing, and only + a run that is about to reach Bedrock needs to know whose account it is. + """ + home = options.home or records.default_home() + options.home = home + if not options.region: + options.region = records.require_setting(home, *SETTINGS["region"]) + if not options.model: + options.model = records.require_setting(home, *SETTINGS["model"]) + if options.profile is None: + # Presence, not truthiness: an empty FM_VOICE_PROFILE is the captain + # saying "use the credentials I already have" and must not fall through + # to a configured profile, which is how fm-inbox.sh reads its own + # equivalent and what docs/configuration.md promises for both. An empty + # region or model is still nothing, so those keep falling through. + name, env = SETTINGS["profile"][:2] + chosen = os.environ.get(env) + if chosen is None: + chosen = records.read_setting(home, name) + options.profile = (chosen or "").strip() + if not options.voice: + options.voice = records.read_setting(home, *SETTINGS["voice"][:2]) or VOICE + return options + + +def main(argv): + options = parse_args(argv) + try: + resolve_settings(options) + if options.self_test: + return asyncio.run(self_test(options)) + return asyncio.run(serve(options)) or 0 + except (records.RecordError, CredentialError) as exc: + sys.stderr.write("fm-voice-relay: {}\n".format(exc)) + return 2 + except KeyboardInterrupt: + return 130 + except Exception as exc: # noqa: BLE001 + # The captain reads this stderr over SSH, so a failure that gets this + # far says what it was in one line. --verbose still gets the traceback, + # because whoever passed it is debugging rather than talking. + sys.stderr.write("fm-voice-relay: {}: {}\n".format( + type(exc).__name__, exc)) + if options.verbose: + traceback.print_exc() + return 2 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/bin/fm-wake-drain.sh b/bin/fm-wake-drain.sh index fcf46a55167..203765be80f 100755 --- a/bin/fm-wake-drain.sh +++ b/bin/fm-wake-drain.sh @@ -1,7 +1,8 @@ #!/usr/bin/env bash # Present durable watcher wake records, optionally acknowledge handled records, # annotate every unread line for validated signal status keys, surface unread -# informational status lines and OPEN DECISIONS, then assert liveness. +# informational status lines, OPEN DECISIONS, and captain-call record +# divergence, then assert liveness. # # Keep sequence-bound row consumption independent from generation-bound episode # retirement; docs/watcher-continuity.md owns the recovery contract. @@ -14,6 +15,8 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" . "$SCRIPT_DIR/fm-classify-lib.sh" # shellcheck source=bin/fm-line-cap-lib.sh . "$SCRIPT_DIR/fm-line-cap-lib.sh" +# shellcheck source=bin/fm-timeout-lib.sh +. "$SCRIPT_DIR/fm-timeout-lib.sh" DRAIN_TMP= DRAIN_LOCK_HELD=false @@ -174,6 +177,66 @@ EOF printf "OPEN DECISIONS: close one by answering it: bin/fm-send.sh <task> --resolve-key <key> '<answer>'\n" || return 1 } +# Print the RECORD DIVERGENCE section: every captain call whose two records +# contradict each other - the status log says a key was resolved outright while +# the task held for the captain is still open. Nothing here closes anything; the +# section exists because posting the resolution alone reads as complete on the +# status side, so the durable record can keep saying the captain owes an answer +# with no warning at all. bin/fm-captain-hold.sh's `diverged` owns which pairs +# count and why; this prints what it reports. +# +# Bounded and silent like OPEN DECISIONS above: nothing prints when the two +# records agree, which is the common case. If tasks-axi is unavailable, the +# guard cannot read the structured record and stays silent. A guard failure +# never changes the drain's exit status - a supervision turn must still present +# its wakes when the backlog tool is having a bad day. +print_record_divergence_section() { + local diverged task origin key title line shown=0 omitted=0 bound + local output='' used=0 bytes item_bytes=220 global_bytes=2000 + + # A non-positive bound is not a bound (bin/fm-timeout-lib.sh), so a bad + # override falls back to the default rather than disabling the deadline. + bound=${FM_DIVERGENCE_TIMEOUT:-20} + case "$bound" in ''|*[!0-9]*|0) bound=20 ;; esac + + # Bounded, because this runs at the top of every supervision turn: a backlog + # tool having a bad day must cost the drain a few seconds at worst, never the + # presentation of the wakes it exists to deliver. + diverged=$(fm_run_timed "$bound" "$SCRIPT_DIR/fm-captain-hold.sh" diverged 2>/dev/null) || return 0 + [ -n "$diverged" ] || return 0 + + while IFS=$(printf '\t') read -r task origin key title; do + [ -n "$task" ] || continue + line="$task [key=$key] reads resolved in $origin's status log but is still held for the captain" + [ -z "$title" ] || line="$line: $title" + fm_cap_line_var "$line" $((item_bytes - 1)) + line=$FM_LINE_CAP_LINE + bytes=$(( ${#line} + 1 )) + if [ $((used + bytes)) -gt "$global_bytes" ]; then + omitted=$((omitted + 1)) + continue + fi + output="$output$line +" + used=$((used + bytes)) + shown=$((shown + 1)) + done <<EOF +$diverged +EOF + + [ "$shown" -gt 0 ] || [ "$omitted" -gt 0 ] || return 0 + printf 'RECORD DIVERGENCE (answered in the status log, still held in the backlog - nothing was closed automatically):\n' || return 1 + printf '%s' "$output" || return 1 + if [ "$omitted" -gt 0 ]; then + printf 'RECORD DIVERGENCE: %d more omitted (byte cap)\n' "$omitted" || return 1 + fi + # Both directions, deliberately. The status resolution is not proof the + # captain ruled: a call can dissolve, or turn out to have been a question of + # fact. Reconcile with what actually happened - never by closing on the + # strength of this line. + printf 'RECORD DIVERGENCE: reconcile each one - record the captain'"'"'s own words with bin/fm-captain-hold.sh answer <task> --decision-file <path>, or re-open the status decision when that resolution was not the captain'"'"'s word.\n' || return 1 +} + print_status_sections() { local snapshot=${1:-} fully_presented=${2:-} acknowledged if [ -z "$snapshot" ]; then snapshot=$(status_presentation_snapshot "$STATE") || return 1; fi @@ -181,6 +244,7 @@ print_status_sections() { acknowledged=$(status_acknowledge_presented_snapshot "$STATE" "$snapshot" "$fully_presented") || return 1 print_unread_status_section "$snapshot" || return 1 print_open_decisions_section "$snapshot" || return 1 + print_record_divergence_section || return 1 status_commit_presentation_snapshot "$STATE" "$acknowledged" } @@ -271,7 +335,7 @@ if [ ! -s "$FM_WAKE_QUEUE" ]; then fm_recovery_marker_snapshot "$RECOVERY_MARKER" || true RECOVERY_MARKER_TOKEN=$FM_RECOVERY_MARKER_TOKEN case "$RECOVERY_MARKER_TOKEN" in - pending:downtime:*) + pending:downtime:*|announced:downtime:*) fm_recovery_marker_begin_handling "$RECOVERY_MARKER" || { echo "wake drain: decision recovery could not begin handling safely" >&2 exit 1 @@ -279,7 +343,7 @@ if [ ! -s "$FM_WAKE_QUEUE" ]; then RECOVERY_MARKER_TOKEN=$FM_RECOVERY_MARKER_TOKEN RECOVERY_ACK_REQUIRED=true ;; - pending:handling:*) RECOVERY_ACK_REQUIRED=true ;; + pending:handling:*|announced:handling:*) RECOVERY_ACK_REQUIRED=true ;; esac fm_lock_release "$FM_WAKE_QUEUE_LOCK" DRAIN_LOCK_HELD=false @@ -327,7 +391,7 @@ fi fm_recovery_marker_snapshot "$RECOVERY_MARKER" || exit 1 RECOVERY_MARKER_TOKEN=$FM_RECOVERY_MARKER_TOKEN case "$RECOVERY_MARKER_TOKEN" in - pending:*|acked:*) ;; + pending:*|announced:*|acked:*) ;; *) echo "wake drain: durable wakes have no recovery generation" >&2; exit 1 ;; esac fm_lock_release "$FM_WAKE_QUEUE_LOCK" diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index ff32d88196e..28249b661f3 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -476,6 +476,9 @@ fm_lock_recheck_stale_owner() { FM_RECOVERY_MARKER_TOKEN= FM_RECOVERY_MARKER_ACTION='none' +# Token grammar (one owner): <pending|announced|acked>:<handling|downtime>:<generation> +# docs/watcher-continuity.md owns the recovery-episode contract, including the +# once-per-generation announcement rule for unacknowledged downtime. fm_recovery_marker_read() { local marker=$1 line count FM_RECOVERY_MARKER_TOKEN= @@ -484,7 +487,7 @@ fm_recovery_marker_read() { [ "$count" = 1 ] || return 1 IFS= read -r line < "$marker" || return 1 case "$line" in - pending:handling:*|pending:downtime:*|acked:handling:*|acked:downtime:*) ;; + pending:handling:*|pending:downtime:*|announced:handling:*|announced:downtime:*|acked:handling:*|acked:downtime:*) ;; *) return 1 ;; esac case "${line##*:}" in @@ -498,11 +501,12 @@ _fm_atomic_replace() { } _fm_recovery_marker_write_locked() { - local marker=$1 kind=$2 generation=${3:-} tmp + local marker=$1 kind=$2 generation=${3:-} status=${4:-pending} tmp case "$kind" in handling|downtime) ;; *) return 1 ;; esac + case "$status" in pending|announced) ;; *) return 1 ;; esac tmp=$(mktemp "${marker}.tmp.XXXXXX") || return 1 [ -n "$generation" ] || generation="$(fm_current_pid).$(date +%s).${tmp##*.}" - if ! printf 'pending:%s:%s\n' "$kind" "$generation" > "$tmp" \ + if ! printf '%s:%s:%s\n' "$status" "$kind" "$generation" > "$tmp" \ || ! chmod 0600 "$tmp" \ || ! _fm_atomic_replace "$tmp" "$marker"; then rm -f -- "$tmp" @@ -510,11 +514,13 @@ _fm_recovery_marker_write_locked() { fi } -# Preserve a pending episode's generation across downtime republication so its -# outstanding acknowledgement remains usable; docs/watcher-continuity.md owns -# the recovery contract and sequence-safety rationale. +# Preserve a pending or announced episode's generation across downtime +# republication so its outstanding acknowledgement remains usable, and keep an +# already-announced generation announced so it cannot be re-presented until a +# new down stretch mints a new generation. +# docs/watcher-continuity.md owns the recovery contract and sequence-safety rationale. _fm_recovery_marker_publish() { - local marker=$1 kind=${2:-downtime} lock saved_token generation='' + local marker=$1 kind=${2:-downtime} lock saved_token generation='' status=pending case "$kind" in handling|downtime) ;; *) return 1 ;; esac lock="${marker}.lock" fm_lock_acquire_wait "$lock" || return 1 @@ -529,12 +535,19 @@ _fm_recovery_marker_publish() { saved_token=$FM_RECOVERY_MARKER_TOKEN if fm_recovery_marker_read "$marker"; then case "$FM_RECOVERY_MARKER_TOKEN" in - pending:handling:*|pending:downtime:*) generation=${FM_RECOVERY_MARKER_TOKEN##*:} ;; + pending:handling:*|pending:downtime:*) + generation=${FM_RECOVERY_MARKER_TOKEN##*:} + status=pending + ;; + announced:handling:*|announced:downtime:*) + generation=${FM_RECOVERY_MARKER_TOKEN##*:} + status=announced + ;; esac fi FM_RECOVERY_MARKER_TOKEN=$saved_token fi - if ! _fm_recovery_marker_write_locked "$marker" "$kind" "$generation"; then + if ! _fm_recovery_marker_write_locked "$marker" "$kind" "$generation" "$status"; then fm_lock_release "$lock" return 1 fi @@ -556,7 +569,7 @@ _fm_recovery_marker_begin_handling() { return 3 fi case "$line" in - pending:handling:*) ;; + pending:handling:*|announced:handling:*) ;; pending:downtime:*) if ! _fm_recovery_marker_write_locked "$marker" handling "$generation"; then fm_lock_release "$lock" @@ -564,6 +577,13 @@ _fm_recovery_marker_begin_handling() { fi FM_RECOVERY_MARKER_TOKEN="pending:handling:$generation" ;; + announced:downtime:*) + if ! _fm_recovery_marker_write_locked "$marker" handling "$generation" announced; then + fm_lock_release "$lock" + return 1 + fi + FM_RECOVERY_MARKER_TOKEN="announced:handling:$generation" + ;; *) fm_lock_release "$lock"; return 1 ;; esac fm_lock_release "$lock" @@ -590,8 +610,9 @@ _fm_recovery_marker_ack() { fi line=$FM_RECOVERY_MARKER_TOKEN case "$line" in - pending:*) line="acked:${line#pending:}" ;; + pending:*|announced:*) line="acked:${line#*:}" ;; acked:*) fm_lock_release "$lock"; return 0 ;; + *) fm_lock_release "$lock"; return 1 ;; esac tmp=$(mktemp "${marker}.tmp.XXXXXX") || { fm_lock_release "$lock"; return 1; } if ! printf '%s\n' "$line" > "$tmp" \ @@ -615,7 +636,7 @@ _fm_recovery_marker_arm_check() { fi if [ ! -e "$marker" ] && [ ! -L "$marker" ]; then if [ -s "$FM_WAKE_QUEUE" ]; then - if ! _fm_recovery_marker_write_locked "$marker" downtime; then + if ! _fm_recovery_marker_write_locked "$marker" downtime "" announced; then fm_lock_release "$lock" fm_lock_release "$FM_WAKE_QUEUE_LOCK" return 1 @@ -634,7 +655,7 @@ _fm_recovery_marker_arm_check() { return 1 } if ! mv -- "$marker" "$quarantine/marker" \ - || ! _fm_recovery_marker_write_locked "$marker" downtime; then + || ! _fm_recovery_marker_write_locked "$marker" downtime "" announced; then rmdir "$quarantine" 2>/dev/null || true fm_lock_release "$lock" fm_lock_release "$FM_WAKE_QUEUE_LOCK" @@ -647,16 +668,24 @@ _fm_recovery_marker_arm_check() { fi line=$FM_RECOVERY_MARKER_TOKEN case "$line" in - pending:handling:*) + pending:handling:*|announced:handling:*|announced:downtime:*) FM_RECOVERY_MARKER_ACTION='wait' fm_lock_release "$lock" fm_lock_release "$FM_WAKE_QUEUE_LOCK" return 0 ;; - pending:downtime:*) FM_RECOVERY_MARKER_ACTION='recover' ;; + pending:downtime:*) + if ! _fm_recovery_marker_write_locked "$marker" downtime "${line##*:}" announced; then + fm_lock_release "$lock" + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 + fi + FM_RECOVERY_MARKER_TOKEN="announced:downtime:${line##*:}" + FM_RECOVERY_MARKER_ACTION='recover' + ;; acked:*) if [ -s "$FM_WAKE_QUEUE" ]; then - if ! _fm_recovery_marker_write_locked "$marker" downtime; then + if ! _fm_recovery_marker_write_locked "$marker" downtime "" announced; then fm_lock_release "$lock" fm_lock_release "$FM_WAKE_QUEUE_LOCK" return 1 @@ -670,6 +699,29 @@ _fm_recovery_marker_arm_check() { fm_lock_release "$FM_WAKE_QUEUE_LOCK" } +# A non-successor watcher start after an announced-but-unacked episode is a new +# down stretch: mint a fresh pending generation so a still-open decision or +# buried note can be presented once more. Handling successors must not call +# this, because Option B re-arm is not a new down stretch. +_fm_recovery_marker_reopen_announced() { + local marker=$1 lock + lock="${marker}.lock" + fm_lock_acquire_wait "$lock" || return 1 + if ! fm_recovery_marker_read "$marker"; then + fm_lock_release "$lock" + return 0 + fi + case "$FM_RECOVERY_MARKER_TOKEN" in + announced:*) + if ! _fm_recovery_marker_write_locked "$marker" downtime ""; then + fm_lock_release "$lock" + return 1 + fi + ;; + esac + fm_lock_release "$lock" +} + fm_recovery_transition() { local marker=$1 action=$2 target=${3:-} value=${4:-} case "$action" in @@ -682,6 +734,9 @@ fm_recovery_transition() { arm-check) _fm_recovery_marker_arm_check "$marker" ;; + reopen-announced) + _fm_recovery_marker_reopen_announced "$marker" + ;; release-lock) [ -n "$target" ] || return 1 _fm_recovery_marker_publish "$marker" "${value:-downtime}" || return 1 @@ -723,6 +778,10 @@ fm_recovery_marker_arm_check() { fm_recovery_transition "$1" arm-check } +fm_recovery_marker_reopen_announced() { + fm_recovery_transition "$1" reopen-announced +} + fm_lock_try_acquire() { local lockdir=$1 pid steal cur rc steal_owner primary_owner FM_LOCK_HELD_PID= @@ -923,6 +982,141 @@ fm_failure_episode_reset() { return 0 } +# --- Claude Stop auto-arm claim abandonment ---------------------------------- +# Both Stop-event participants (bin/fm-claude-stop-autoarm.sh and +# bin/fm-turnend-guard.sh --claude) stand down for whoever holds the auto-arm's +# single-flight owner lock, on the premise that a live holder is still deciding +# supervision. A holder that has already FINISHED that decision but never +# released the lock turns the courtesy into indefinite silence: every later +# async firing exits at the lock, the epoch ledger freezes at its last outcome, +# and each following turn end allows a blind stop while nothing re-arms the +# watcher. Observed 2026-08-14: one delivered rewake, then a beacon that went +# 40 minutes without a beat, no watcher lock at all, two workers in flight, and +# both of their reports unread until an operator drained the queue by hand. +# +# One abandonment proof is the ledger, not pid liveness, because both ways a +# finished claim keeps a live pid - reuse of the recorded pid, and a hook still +# blocked writing its rewake banner - look alive: +# +# 1. the owner lock exists and carries the auto-arm role, +# 2. its recorded pid is numeric, +# 3. the ledger's owner_pid is exactly that pid, and +# 4. the ledger's outcome is present and is not "arming". +# +# Condition 3 is what makes reclaiming race-free. A fresh claimant creates the +# lock BEFORE it writes "arming", so until it does the ledger still names the +# PREVIOUS owner and the two pids cannot match; a just-started claim is never +# mistaken for an abandoned one. Condition 4 treats "arming" as in progress no +# matter how old, because the owner foregrounds fm-watch-arm.sh for the whole +# watcher cycle, which legitimately runs for hours. +# +# The ledger alone cannot prove every abandonment, though: an entry still reading +# "arming", or no entry at all, says nothing about a recorded pid the operating +# system has since handed to an unrelated live process - the same lapse, reached +# when a session teardown kills a claim's whole process group before it can record +# any outcome or run its release trap. So the claim also records the pid-identity +# every other supervision lock in this repo records (fm_pid_identity above, used by +# state/.watch.lock, the supervise-daemon lock, and the AFK launch lock), and a +# recorded identity that no longer matches the live pid is abandonment on its own, +# whatever the ledger says. That identity is written BEFORE the auto-arm role is +# published, and every participant requires that role first, so a claim that is +# genuinely mid-flight is never read as identity-less. A claim carrying no recorded +# identity at all (an older build, a hand-edited lock) keeps exactly the +# ledger-only reasoning above, and an identity that cannot be recomputed for the +# live pid proves nothing either way, so it falls through to the ledger too. +_fm_autoarm_epoch_field() { # <epoch-file> <field> + local file=$1 field=$2 tok + local -a toks=() + [ -r "$file" ] || return 1 + # 2> before <: a failed input redirection reports through whatever stderr is + # current when it runs, so the suppression has to be established first. + IFS=' ' read -r -a toks 2>/dev/null < "$file" || return 1 + for tok in ${toks[@]+"${toks[@]}"}; do + case "$tok" in + "$field="?*) printf '%s\n' "${tok#*=}"; return 0 ;; + esac + done + return 1 +} + +# Record the claiming process's pid-identity inside the auto-arm owner lock, the +# way every other supervision lock in this repo records it. Best effort by design: +# a platform where fm_pid_identity cannot answer keeps the ledger-only reasoning +# rather than losing the claim, and a record that cannot be completed leaves NO +# identity file behind, so a partial write can never read as a mismatch against +# its own live owner. Call it before publishing the auto-arm role. +fm_autoarm_claim_record_identity() { # <state-dir> + local state=$1 lock pid held identity back + lock="$state/.claude-autoarm.lock" + # Resolve the pid into a variable FIRST: expanding ${BASHPID:-$$} inside the + # command substitution below would resolve it in that subshell, recording the + # identity of a process that exits immediately and leaving every later reader + # with a permanent mismatch against the real owner. + pid=${BASHPID:-$$} + # The identity must describe the pid the lock publishes, so record it only for a + # lock this process actually holds (the same ownership test as fm_lock_set_role). + held=$(cat "$lock/pid" 2>/dev/null || true) + [ "$held" = "$pid" ] || return 1 + identity=$(fm_pid_identity "$pid" 2>/dev/null) || return 1 + [ -n "$identity" ] || return 1 + if ! printf '%s\n' "$identity" > "$lock/pid-identity" 2>/dev/null; then + rm -f "$lock/pid-identity" 2>/dev/null || true + return 1 + fi + back=$(cat "$lock/pid-identity" 2>/dev/null || true) + if [ "$back" != "$identity" ]; then + rm -f "$lock/pid-identity" 2>/dev/null || true + return 1 + fi + return 0 +} + +fm_autoarm_claim_abandoned() { # <state-dir> + local state=$1 epoch lock role pid owner outcome recorded current + lock="$state/.claude-autoarm.lock" + epoch="$state/.claude-autoarm-epoch" + [ -e "$lock" ] || [ -L "$lock" ] || return 1 + role=$(fm_lock_role "$lock") + [ "$role" = autoarm ] || return 1 + pid=$(cat "$lock/pid" 2>/dev/null || true) + case "$pid" in + ''|*[!0-9]*) return 1 ;; + esac + recorded=$(cat "$lock/pid-identity" 2>/dev/null || true) + if [ -n "$recorded" ] && current=$(fm_pid_identity "$pid" 2>/dev/null) \ + && [ -n "$current" ] && [ "$current" != "$recorded" ]; then + return 0 + fi + owner=$(_fm_autoarm_epoch_field "$epoch" owner_pid) || return 1 + [ "$owner" = "$pid" ] || return 1 + outcome=$(_fm_autoarm_epoch_field "$epoch" outcome) || return 1 + case "$outcome" in + ''|arming) return 1 ;; + esac + return 0 +} + +# Remove a proven-abandoned auto-arm claim so the next claimant can arm. +# The proof is re-verified while holding the lock's steal mutex, which is the +# same serialization fm_lock_try_acquire uses for stale-owner reclaim: while it +# is held no other process can publish the primary lock, so the window between +# proving abandonment and removing the lock cannot swallow a genuine new claim. +fm_autoarm_release_abandoned() { # <state-dir> + local state=$1 lock steal + lock="$state/.claude-autoarm.lock" + steal="$lock.steal" + fm_autoarm_claim_abandoned "$state" || return 1 + fm_lock_try_acquire "$steal" || return 1 + if ! fm_autoarm_claim_abandoned "$state"; then + fm_lock_release "$steal" + return 1 + fi + fm_lock_remove_path "$lock" || true + fm_lock_release "$steal" + [ -e "$lock" ] || [ -L "$lock" ] || return 0 + return 1 +} + fm_wake_clean_field() { LC_ALL=C tr '\t\r\n' ' ' } diff --git a/bin/fm-watch-arm.sh b/bin/fm-watch-arm.sh index 5ba132401af..d134f519402 100755 --- a/bin/fm-watch-arm.sh +++ b/bin/fm-watch-arm.sh @@ -376,7 +376,7 @@ handling_successor_generation() { [ -n "${FM_WATCH_PREDECESSOR_ARM_PID:-}" ] || return 0 fm_recovery_marker_snapshot "$STATE/.watcher-down" || return 1 case "$FM_RECOVERY_MARKER_TOKEN" in - pending:downtime:*|pending:handling:*) printf '%s' "${FM_RECOVERY_MARKER_TOKEN##*:}" ;; + pending:downtime:*|pending:handling:*|announced:downtime:*|announced:handling:*) printf '%s' "${FM_RECOVERY_MARKER_TOKEN##*:}" ;; acked:*|'') ;; *) return 1 ;; esac diff --git a/bin/fm-watch.sh b/bin/fm-watch.sh index a3f78fcc335..d1d59d3ceb5 100755 --- a/bin/fm-watch.sh +++ b/bin/fm-watch.sh @@ -6,9 +6,10 @@ # is absorbed only when the crew shows POSITIVE evidence it is still working (an # actively-running no-mistakes step, or a backend busy signal), and surfaced # otherwise, so a crew that finishes (or stops and waits) without a current -# working signal is never silently swallowed. A declared external-wait pause is -# the separate idle absorb case and re-surfaces only on its long bounded cadence, -# although its initial no-verb status signal still surfaces in normal mode. +# working signal is never silently swallowed. A declared wait, either a paused: +# external wait or a verified captain-held transfer, is the separate idle absorb +# case and re-surfaces only on its long bounded cadence, although its initial +# no-verb status signal still surfaces in normal mode. # While state/.afk exists, the daemon owns triage and this watcher queues and exits # on every wake. Printed reason lines: # signal: <file>... status/turn-end signals, surfaced when a listed status @@ -19,9 +20,11 @@ # run-step or busy pane outranks even a captain-relevant log # line, since the crew's own log gets no new entry once # firstmate hands it to a no-mistakes validation. A declared -# external-wait pause is absorbed instead with its own long -# re-surface cadence, never as a wedge. Only when neither -# absorb class applies does the log's last line decide: +# external-wait pause or verified captain-held transfer is +# absorbed instead with its own long re-surface cadence, +# never as a wedge, and that recheck reason names which +# human the wait is on. Only when neither absorb class +# applies does the log's last line decide: # terminal (captain-relevant) or non-terminal (no verb), # both surfaced at once. A provably-working stale past the # wedge threshold also surfaces, with an "escalation N" @@ -30,7 +33,14 @@ # also carries a "demand-deep-inspection" marker so the # wake payload itself, not just repetition, forces a # closer look instead of another routine supervision -# resume. Unless afk is active. A genuinely busy pane +# resume. Unless afk is active. A pane whose own task +# worktree was written during the quiet window is +# deferred rather than escalated (wedge_defer_writing), +# because files appearing there are liveness the pane and +# the run step cannot show; that deferral still +# re-surfaces once per PAUSE_RESURFACE_SECS, and a pane +# that writes nothing keeps the unchanged schedule. +# A genuinely busy pane # (window_is_busy true) is exempt from the above, but # only up to BUSY_TURN_MAX_SECS with no completed turn # (state/<id>.turn-ended, or the spawn record before any @@ -167,7 +177,9 @@ BUSY_TURN_MAX_SECS=${FM_BUSY_TURN_MAX_SECS:-3600} # A crew that declared a pause is idling on a known external wait, so its stale # pane is absorbed rather than wedge-escalated. # A captain-held or paused crew whose agent has confidently exited uses the same -# bounded cadence, while a live or ambiguously read agent still surfaces once. +# bounded cadence, while a live or ambiguously read agent still surfaces once; a +# secondmate earns the cadence on its declaration alone, because its endpoint +# liveness is deliberately never read (pause_state_class owns that split). # These cases re-surface once for a recheck every PAUSE_RESURFACE_SECS - far # longer than the wedge threshold, but finite so a forgotten hold cannot rot invisibly. PAUSE_RESURFACE_SECS=${FM_PAUSE_RESURFACE_SECS:-$FM_PAUSE_RESURFACE_SECS_DEFAULT} @@ -254,6 +266,20 @@ window_label() { [ -n "$task" ] && printf 'fm-%s' "$task" } +# The ONE derivation of a window's per-window marker key: `:`, `/` and `.` become +# `_` so a window name is usable as a filename suffix. Every per-window file the +# watcher keeps is named by it (.hash-, .count-, .stale-, .stale-since-, +# .wedge-escalations-, .paused-*, .writing-*), and live homes hold those markers on +# disk under the current format, so the format lives here alone: a second copy is +# how a future change to it silently orphans a window's markers instead of clearing +# them. The helpers below take the derived key rather than re-deriving it, so one +# poll of one window derives it once. +window_key() { # <window> + local key=${1//:/_} + key=${key//\//_} + printf '%s' "${key//./_}" +} + recorded_windows() { local meta w seen= for meta in "$STATE"/*.meta; do @@ -280,6 +306,57 @@ recorded_windows() { # below). FM_WEDGE_DEMAND_INSPECT_COUNT=${FM_WEDGE_DEMAND_INSPECT_COUNT:-3} +# One bounded re-surface for a pane the watcher is deliberately absorbing, so no +# absorb can rot invisibly. <age> is how long the current absorb has held and +# <throttle> is the per-window marker whose mtime records the last re-surface, so +# once past PAUSE_RESURFACE_SECS the pane wakes once per window rather than every +# poll. Shared by the declared-pause absorb and the worktree-write deferral so the +# two cadences cannot drift apart; each caller owns its own marker and reason. +# Returns without waking while either the absorb or the throttle is inside the +# window; wake() itself exits the cycle, exactly as it does inline. +resurface_absorbed() { # <window> <throttle-marker> <age> <reason> + local win=$1 throttle=$2 age=$3 reason=$4 + [ "$age" -ge "$PAUSE_RESURFACE_SECS" ] || return 0 + [ "$(age_of "$throttle")" -ge "$PAUSE_RESURFACE_SECS" ] || return 0 # 999999 when no prior re-surface + fm_wake_append stale "$win" "$reason" || exit 1 + date +%s > "$throttle" + wake "$reason" +} + +# Defer ONE wedge escalation for a pane that went quiet while its own task +# worktree is demonstrably still being written (crew_worktree_written_since in +# fm-classify-lib.sh). The pane and the run step both say nothing is happening; +# the worktree says otherwise, and files appearing in it is the harder signal to +# fake, so the escalation is deferred rather than fired. Deliberately a DEFERRAL, +# not a cancellation: the idle timer restarts, so the next window probes again, +# and a .writing-since-<key> marker ages the whole deferral chain so the pane +# still re-surfaces once every PAUSE_RESURFACE_SECS through the shared +# resurface_absorbed above - literally the same bounded cadence a declared pause +# uses, throttled by its own .writing-resurfaced-<key> marker - and a crew whose +# worktree churns without real progress cannot stay invisible. The escalation +# counter is left alone: it is neither advanced (this is not an escalation) nor +# reset (a later genuine escalation must still carry the demand-deep-inspection +# history it had already earned). +wedge_defer_writing() { # <window> <since-file> <triage-label> <idle-age> + local win=$1 since_file=$2 label=$3 age=$4 key wsf wage + key=$(window_key "$win") + wsf="$STATE/.writing-since-$key" + [ -e "$wsf" ] || date +%s > "$wsf" + wage=$(age_of "$wsf") + date +%s > "$since_file" + resurface_absorbed "$win" "$STATE/.writing-resurfaced-$key" "$wage" \ + "stale: $win (idle ${age}s, writing its worktree for ${wage}s, rechecked on a long cadence not a wedge; confirm the writes are real progress)" + triage_log "absorbed $label (worktree written since the idle window opened, idle ${age}s): $win" +} + +# Drop a window's write-deferral chain wherever its stale bookkeeping resets, so +# the bounded re-surface cadence is measured from the CURRENT quiet stretch and a +# long-finished one cannot make the next deferral resurface immediately. +clear_write_tracking() { # <window-key> + local key=$1 + rm -f "$STATE/.writing-since-$key" "$STATE/.writing-resurfaced-$key" +} + # Repeat-poll wedge-timer bookkeeping for an already-classified stale hash # absorbed as provably-working - repairs a missing/corrupt timer (self-heals a # watcher restart between recording the hash and recording the timer), or @@ -288,17 +365,25 @@ FM_WEDGE_DEMAND_INSPECT_COUNT=${FM_WEDGE_DEMAND_INSPECT_COUNT:-3} # both places a hash can be absorbed this way: the plain non-terminal path, # and the stale_is_terminal-overridden path (a captain-relevant status-log # line that an active run/busy pane outranked). -wedge_timer_check() { # <window> <since-file> <triage-label> <escalation-count-file> - local win=$1 since_file=$2 label=$3 escalation_file=$4 since age n reason +# The worktree write probe runs ONLY here, inside the at-threshold branch that is +# about to escalate: at most one bounded walk per window per STALE_ESCALATE_SECS, +# never per poll. +wedge_timer_check() { # <window> <since-file> <triage-label> <escalation-count-file> <task> + local win=$1 since_file=$2 label=$3 escalation_file=$4 task=$5 since age n reason since=$(cat "$since_file" 2>/dev/null || true) case "$since" in ''|*[!0-9]*) date +%s > "$since_file" + clear_write_tracking "$(window_key "$win")" triage_log "absorbed $label timer reset: $win" ;; *) age=$(( $(date +%s) - since )) if [ "$age" -ge "$STALE_ESCALATE_SECS" ]; then + if crew_worktree_written_since "$task" "$STATE" "$since_file"; then + wedge_defer_writing "$win" "$since_file" "$label" "$age" + return 0 + fi n=$(( $(cat "$escalation_file" 2>/dev/null || echo 0) + 1 )) echo "$n" > "$escalation_file" reason="stale: $win (idle ${age}s, possible wedge, escalation $n)" @@ -307,6 +392,7 @@ wedge_timer_check() { # <window> <since-file> <triage-label> <escalation-count- fi fm_wake_append stale "$win" "$reason" || exit 1 rm -f "$since_file" + clear_write_tracking "$(window_key "$win")" wake "$reason" fi ;; @@ -333,28 +419,35 @@ busy_turn_over_age() { # <task> # cheap: it NEVER re-reads crew state. The re-surface age is anchored on the # status file mtime, not a per-hash marker, so a churny idle pane (a ticking # clock, a token counter) cannot keep resetting the cadence the way a hash-tied -# timer would. A .paused-resurfaced-<key> throttle marker records the last -# re-surface epoch so, once past the window, it fires once per window rather than -# every poll. Advances the stale suppressor to <hash> and flags the key paused. +# timer would. The bounded re-surface itself is the shared resurface_absorbed +# above, throttled by this window's own .paused-resurfaced-<key> marker. Advances +# the stale suppressor to <hash> and flags the key paused. +# +# The recheck names WHICH human the declared wait is on, because that is the whole +# point of a recheck the captain reads: an external dependency for paused:, and the +# captain themself for a verified hold. Only the captain-held verb takes the second +# wording; a caller that reached the bounded cadence off pause tracking alone, with +# no declaring verb left on the log, keeps the external-wait wording it always had. handle_paused_stale() { # <window> <task> <hash> - local win=$1 task=$2 h=$3 key statusf mtime age rf rf_age reason - key=$(printf '%s' "$win" | tr ':/.' '___') + local win=$1 task=$2 h=$3 key statusf mtime age detail reason + key=$(window_key "$win") printf '%s' "$h" > "$STATE/.stale-$key" : > "$STATE/.paused-$key" rm -f "$STATE/.stale-since-$key" "$STATE/.wedge-escalations-$key" + clear_write_tracking "$key" statusf="$STATE/$task.status" mtime=$(stat_mtime "$statusf") case "$mtime" in ''|*[!0-9]*) mtime=$(date +%s) ;; esac age=$(( $(date +%s) - mtime )) - rf="$STATE/.paused-resurfaced-$key" - rf_age=$(age_of "$rf") # 999999 when no prior re-surface - if [ "$age" -ge "$PAUSE_RESURFACE_SECS" ] && [ "$rf_age" -ge "$PAUSE_RESURFACE_SECS" ]; then - reason="stale: $win (paused ${age}s, awaiting external - declared pause, rechecked on a long cadence not a wedge; confirm the wait still holds)" - fm_wake_append stale "$win" "$reason" || exit 1 - date +%s > "$rf" - wake "$reason" + if status_is_captain_held "$(last_status_line "$statusf")"; then + detail="captain-held, awaiting the captain" + reason="captain-held ${age}s, awaiting the captain - verified hold transfer, rechecked on a long cadence not a wedge; answer the held decision or release the hold" + else + detail="paused, awaiting external" + reason="paused ${age}s, awaiting external - declared pause, rechecked on a long cadence not a wedge; confirm the wait still holds" fi - triage_log "absorbed stale (paused, awaiting external, age ${age}s): $win" + resurface_absorbed "$win" "$STATE/.paused-resurfaced-$key" "$age" "stale: $win ($reason)" + triage_log "absorbed stale ($detail, age ${age}s): $win" } # Apply the busy-pane completed-turn bound to a window whose bound has already @@ -377,35 +470,29 @@ busy_turn_bound_check() { # <window> <task> <hash> <since-file> <escalation-fil handle_paused_stale "$win" "$task" "$h" return 0 fi - wedge_timer_check "$win" "$since_file" "busy (no completed turn)" "$escalation_file" + wedge_timer_check "$win" "$since_file" "busy (no completed turn)" "$escalation_file" "$task" return 1 } -clear_pause_state() { # <window> - local win=$1 key - key=${win//:/_} - key=${key//\//_} - key=${key//./_} +clear_pause_state() { # <window-key> + local key=$1 rm -f "$STATE/.paused-$key" "$STATE/.paused-rechecked-$key" "$STATE/.paused-resurfaced-$key" } -clear_pause_tracking() { # <window> - local win=$1 key - key=${win//:/_} - key=${key//\//_} - key=${key//./_} - clear_pause_state "$win" +clear_pause_tracking() { # <window-key> + local key=$1 + clear_pause_state "$key" + clear_write_tracking "$key" rm -f "$STATE/.stale-$key" "$STATE/.stale-since-$key" "$STATE/.wedge-escalations-$key" } # Reconcile a declared pause or captain-held status with authoritative crew state. -# Only a confidently dead ordinary crew may recover paused classification after -# fm-crew-state has fallen back to stopped or unknown. +# After fm-crew-state has fallen back to stopped or unknown, paused classification is +# recovered only for a confidently dead ordinary crew, or for a secondmate, whose +# endpoint liveness this function deliberately never reads. pause_state_class() { # <window> <task> - local win=$1 task=$2 key last recheck_file class agent_alive - key=${win//:/_} - key=${key//\//_} - key=${key//./_} + local win=$1 task=$2 key last recheck_file class agent_alive kind + key=$(window_key "$win") last=$(last_status_line "$STATE/$task.status") recheck_file="$STATE/.paused-rechecked-$key" if ! status_is_paused_or_captain_held "$last"; then @@ -413,8 +500,12 @@ pause_state_class() { # <window> <task> crew_absorb_class "$task" return fi + # Read once past the declared-wait gate and reused by both liveness gates below, + # so a mate's stale poll costs one metadata scan rather than one per gate, and the + # far more common no-declaration path above still costs none. + kind=$(window_kind "$win") if [ -e "$STATE/.paused-$key" ] && [ "$(age_of "$recheck_file")" -lt "$STALE_ESCALATE_SECS" ]; then - if [ "$(window_kind "$win")" != secondmate ]; then + if [ "$kind" != secondmate ]; then agent_alive=$(fm_backend_agent_alive "$(window_backend "$win")" "$win" 2>/dev/null) || agent_alive=unknown if [ "$agent_alive" != dead ]; then rm -f "$recheck_file" @@ -431,7 +522,7 @@ pause_state_class() { # <window> <task> printf 'working' return fi - if [ "$(window_kind "$win")" != secondmate ]; then + if [ "$kind" != secondmate ]; then agent_alive=$(fm_backend_agent_alive "$(window_backend "$win")" "$win" 2>/dev/null) || agent_alive=unknown if [ "$agent_alive" != dead ]; then rm -f "$recheck_file" @@ -439,7 +530,15 @@ pause_state_class() { # <window> <task> return fi fi - [ "$class" = none ] && [ "${agent_alive:-unknown}" = dead ] && class=paused + # Recover paused classification for a declared wait that authoritative crew state + # could not name. Reaching here already proves the only two admissible cases: an + # ordinary crew whose agent the gate above confirmed dead, so no live decision gate + # is being silenced, or a secondmate, whose endpoint liveness is deliberately never + # read and so cannot supply that confirmation. Without the mate case a mate's + # captain hold - which has no current-state mapping and so arrives as `none` - + # would be silenced by every caller rather than taking the bounded re-surface + # cadence, and a forgotten hold would rot invisibly. + [ "$class" = none ] && class=paused case "$class" in paused) date +%s > "$recheck_file" ;; *) rm -f "$recheck_file" ;; @@ -449,10 +548,11 @@ pause_state_class() { # <window> <task> surface_nonterminal_stale() { # <window> <hash> local win=$1 h=$2 key task last - key=$(printf '%s' "$win" | tr ':/.' '___') + key=$(window_key "$win") fm_wake_append stale "$win" "stale: $win" || exit 1 printf '%s' "$h" > "$STATE/.stale-$key" rm -f "$STATE/.stale-since-$key" + clear_write_tracking "$key" task=$(window_to_task "$win" "$STATE") last=$(last_status_line "$STATE/$task.status") if status_is_paused_or_captain_held "$last"; then @@ -774,6 +874,12 @@ WATCHER_RECOVERY_PENDING=0 if [ -n "${FM_LOCK_RECOVERED_PID:-}" ]; then WATCHER_RECOVERY_PENDING=1 fi +if [ "${FM_WATCH_HANDLING_SUCCESSOR:-0}" != 1 ]; then + if ! fm_recovery_marker_reopen_announced "$WATCHER_DOWNTIME_MARKER"; then + echo "watcher: recovery state could not be reopened safely; retaining stale lock evidence" >&2 + exit 1 + fi +fi if ! fm_recovery_marker_arm_check "$WATCHER_DOWNTIME_MARKER"; then echo "watcher: recovery state could not be consumed safely; retaining stale lock evidence" >&2 exit 1 @@ -828,6 +934,12 @@ if ! fm_pr_poll_retirement_recover_all "$STATE" "$SCRIPT_DIR/fm-pr-poll.sh"; the fi resurface_after_downtime() { + # Handling successors already have a predecessor-delivered wake on the way. + # Re-announcing from this cycle is what turned a lost handshake into an + # unbounded recovery loop; stay in the poll loop and supervise instead. + if [ "${FM_WATCH_HANDLING_SUCCESSOR:-0}" = 1 ]; then + return 0 + fi if [ "$WATCHER_RECOVERY_PENDING" -ne 1 ]; then if ! fm_recovery_marker_arm_check "$WATCHER_DOWNTIME_MARKER"; then echo "watcher: recovery state could not be consumed safely" >&2 @@ -838,21 +950,6 @@ resurface_after_downtime() { wake "check: rearm-resurface" } -if [ "${FM_WATCH_HANDLING_SUCCESSOR:-0}" = 1 ]; then - touch "$STATE/.last-watcher-beat" - handling_wait=0 - while [ "$handling_wait" -lt 600 ]; do - fm_recovery_marker_snapshot "$WATCHER_DOWNTIME_MARKER" || true - case "$FM_RECOVERY_MARKER_TOKEN" in - pending:downtime:*) ;; - *) break ;; - esac - sleep 0.05 - handling_wait=$((handling_wait + 1)) - done - [ "$handling_wait" -lt 600 ] || WATCHER_RECOVERY_PENDING=1 -fi - while :; do # Self-eviction: if the singleton lock no longer names this process, a second # watcher has taken over (e.g. a transient duplicate from a racy arm). Stand @@ -1034,19 +1131,23 @@ EOF while IFS= read -r w; do kind=$(window_kind "$w") task=$(window_to_task "$w" "$STATE") - key=${w//:/_} - key=${key//\//_} - key=${key//./_} + key=$(window_key "$w") last=$(last_status_line "$STATE/$task.status") if ! status_is_paused_or_captain_held "$last" && [ -e "$STATE/.paused-$key" ]; then - clear_pause_tracking "$w" + clear_pause_tracking "$key" fi - if [ "$kind" = secondmate ] && ! status_is_paused "$last"; then + # An idle secondmate endpoint is healthy by design, so a mate is admitted to + # the pane-stale path ONLY to serve a declared wait's bounded re-surface - + # the same declarations pause_state_class reconciles below, which is why this + # gate reads the shared predicate rather than the pause verb alone. Narrowing + # it to `paused` would leave a mate's captain hold rotting invisibly: the + # clear above already spares its pause tracking, but nothing would ever + # re-surface it. + if [ "$kind" = secondmate ] && ! status_is_paused_or_captain_held "$last"; then continue fi tail40=$(fm_backend_capture "$(window_backend "$w")" "$w" 40 "$(window_label "$w")" 2>/dev/null) || continue h=$(printf '%s' "$tail40" | hash_pane) - key=$(printf '%s' "$w" | tr ':/.' '___') hf="$STATE/.hash-$key" cf="$STATE/.count-$key" sf="$STATE/.stale-$key" @@ -1069,7 +1170,7 @@ EOF if [ "$kind" = secondmate ]; then case "$(pause_state_class "$w" "$task")" in paused) handle_paused_stale "$w" "$task" "$h" ;; - *) clear_pause_tracking "$w" ;; + *) clear_pause_tracking "$key" ;; esac elif afk_present; then # Daemon owns triage: one-shot per distinct stale hash, as before. @@ -1097,11 +1198,13 @@ EOF if crew_is_provably_working "$(window_to_task "$w" "$STATE")"; then printf '%s' "$h" > "$sf" date +%s > "$ssf" + clear_write_tracking "$key" triage_log "absorbed stale (provably working, overriding a stale captain-relevant status): $w" else fm_wake_append stale "$w" "stale: $w" || exit 1 printf '%s' "$h" > "$sf" rm -f "$ssf" + clear_write_tracking "$key" mark_surfaced "$STATE/$(window_to_task "$w" "$STATE").status" wake "stale: $w" fi @@ -1110,7 +1213,7 @@ EOF # wedge timer is running for it) - keep treating it that way # without re-reading the crew state every poll, and without # letting the still-captain-relevant log line re-surface it. - wedge_timer_check "$w" "$ssf" "stale (overridden terminal status)" "$ewf" + wedge_timer_check "$w" "$ssf" "stale (overridden terminal status)" "$ewf" "$task" fi # else: already surfaced as genuinely terminal on a prior poll of # this same hash - nothing left to do (matches the original, @@ -1122,10 +1225,10 @@ EOF # - working: an actively-running pipeline legitimately sits on a static # pane (e.g. waiting on CI), so absorb and start the wedge timer so a # genuinely frozen run still escalates past STALE_ESCALATE_SECS; - # - paused: the crew declared an external wait, or a declared pause or - # captain hold is paired with a confidently dead agent, so absorb on - # the long PAUSE_RESURFACE_SECS cadence instead of wedge-escalating; - # - none: no running pipeline, no exact busy verdict, no declared pause. + # - paused: a declared wait pause_state_class admits (its header owns which + # liveness evidence each kind of crew must supply), so absorb on the long + # PAUSE_RESURFACE_SECS cadence instead of wedge-escalating; + # - none: no running pipeline, no exact busy verdict, no admitted declared wait. # Surface immediately so firstmate inspects the inconclusive state # (it may be done via an interactive menu that wrote no done: status, # waiting on a decision, or wedged) instead of leaving the finish to @@ -1134,7 +1237,7 @@ EOF task=$(window_to_task "$w" "$STATE") case "$(pause_state_class "$w" "$task")" in working) - clear_pause_tracking "$w" + clear_pause_tracking "$key" printf '%s' "$h" > "$sf" date +%s > "$ssf" triage_log "absorbed non-terminal stale (provably working): $w" @@ -1151,14 +1254,14 @@ EOF if [ -e "$pf" ] || status_is_paused_or_captain_held "$(last_status_line "$STATE/$task.status")"; then case "$(pause_state_class "$w" "$task")" in paused) handle_paused_stale "$w" "$task" "$h" ;; - working) clear_pause_state "$w" + working) clear_pause_state "$key" printf '%s' "$h" > "$sf" - wedge_timer_check "$w" "$ssf" "non-terminal stale (provably working after a declared pause)" "$ewf" + wedge_timer_check "$w" "$ssf" "non-terminal stale (provably working after a declared pause)" "$ewf" "$task" triage_log "absorbed non-terminal stale (provably working): $w" ;; *) handle_paused_stale "$w" "$task" "$h" ;; esac else - wedge_timer_check "$w" "$ssf" "non-terminal stale" "$ewf" + wedge_timer_check "$w" "$ssf" "non-terminal stale" "$ewf" "$task" fi fi fi @@ -1172,13 +1275,14 @@ EOF busy_turn_bound_check "$w" "$task" "$h" "$ssf" "$ewf" && paused_bound=0 else rm -f "$ssf" "$ewf" + clear_write_tracking "$key" fi # A busy pane normally means real work resumed, so stale pause bookkeeping # is cleared - but not in the same poll the declared-pause cadence just # recorded it, or the re-surface throttle it depends on would be erased and # the pause would re-surface every poll instead of once per long cadence. if [ "$paused_bound" -ne 0 ] && [ -e "$pf" ] && { [ "$n" -ge 2 ] || ! status_is_paused_or_captain_held "$(last_status_line "$STATE/$(window_to_task "$w" "$STATE").status")"; }; then - clear_pause_tracking "$w" + clear_pause_tracking "$key" fi fi else @@ -1189,17 +1293,18 @@ EOF busy_turn_bound_check "$w" "$task" "$h" "$ssf" "$ewf" && paused_bound=0 else rm -f "$ssf" "$ewf" + clear_write_tracking "$key" fi task=$(window_to_task "$w" "$STATE") if ! afk_present && status_is_paused_or_captain_held "$(last_status_line "$STATE/$task.status")" && [ "$busy_now" -ne 0 ]; then case "$(pause_state_class "$w" "$task")" in paused) handle_paused_stale "$w" "$task" "$h" ;; - *) clear_pause_tracking "$w" ;; + *) clear_pause_tracking "$key" ;; esac elif [ "$paused_bound" -ne 0 ] && [ -e "$pf" ]; then # Same rule as the stable-hash branch: never clear pause bookkeeping the # declared-pause cadence recorded on this very poll. - clear_pause_tracking "$w" + clear_pause_tracking "$key" fi fi done < <(recorded_windows) diff --git a/bin/fm_voice_frame.py b/bin/fm_voice_frame.py new file mode 100644 index 00000000000..d512fc4f3a9 --- /dev/null +++ b/bin/fm_voice_frame.py @@ -0,0 +1,166 @@ +"""fm_voice_frame.py - the wire format between the voice client and the relay. + +The client and the relay share one bidirectional byte stream: an SSH exec +channel, where the client's stdout is the relay's stdin and the relay's stdout +is the client's stdin. Audio and control therefore travel together and need +framing. A frame is a 1 byte kind, a 4 byte unsigned big-endian payload length, +then exactly that many payload bytes. + +Kinds the client sends up to the relay: + S talk start, empty payload + A captured audio, 16000 Hz mono signed 16-bit little-endian + E talk end, empty payload + Q quit, empty payload + +Kinds the relay sends down to the client: + A reply audio, 24000 Hz mono signed 16-bit little-endian + T JSON {"role": ..., "text": ...}, one transcript line + V JSON {"event": ..., ...}, a notice such as a queued request or a failed turn + M JSON {"mark": ..., "since_talk_end": ..., "tool_calls": ...}, one relay-side + timing mark. since_talk_end is seconds from the moment the captain stopped + talking, which is the instant every figure in this build is measured from. + The marks the relay sends are tool_use, first_audio, first_audio_wire, + tool_answered and reply_end; bin/fm-voice-relay.py owns what each means. + B bye, empty payload + +Audio is raw PCM rather than base64 because base64 belongs to the Bedrock +event protocol, not to this hop, and the extra third of the bytes would sit +inside the latency this build exists to measure. + +This module is the owner of the contract above and of the sample rates; +docs/voice-relay.md is the operator-facing guide and points here for the format. +This module is copied to the laptop beside fm-voice-client.py, so it imports +nothing outside the standard library. +""" + +import json +import struct + +HEADER = struct.Struct(">cI") + +# The relay writes this once before its first frame and the client discards +# everything ahead of it. `ssh host command` runs the command through the login +# shell, so a shell startup file that prints to stdout would otherwise land in +# front of the first frame and desynchronise the stream, which reads as a +# baffling protocol error rather than as the chatty shell it is. +MAGIC = b"FMVOICE1" + +# One second of 24000 Hz 16-bit mono is 48000 bytes, so this ceiling is far +# above any real chunk while still rejecting a desynchronised stream early. +MAX_PAYLOAD = 1 << 20 + +TALK_START = b"S" +AUDIO = b"A" +TALK_END = b"E" +QUIT = b"Q" +TEXT = b"T" +NOTICE = b"V" +MARK = b"M" +BYE = b"B" + +KINDS = (TALK_START, AUDIO, TALK_END, QUIT, TEXT, NOTICE, MARK, BYE) + + +class FrameError(Exception): + """A frame could not be encoded or decoded.""" + + +def check_header(kind, length): + """Raise FrameError unless a decoded header is one this format allows. + + Both directions of the stream decode headers, and audio that happens to + look like one must be rejected identically wherever that happens, so the + rules live here rather than beside each decoder. + """ + if kind not in KINDS: + raise FrameError("unknown frame kind: {!r}".format(kind)) + if length > MAX_PAYLOAD: + raise FrameError("payload of {} bytes exceeds the {} byte limit".format( + length, MAX_PAYLOAD)) + + +def encode(kind, payload=b""): + """Return the wire bytes for one frame.""" + check_header(kind, len(payload)) + return HEADER.pack(kind, len(payload)) + payload + + +def encode_json(kind, obj): + """Return the wire bytes for one frame carrying a compact JSON payload.""" + return encode(kind, json.dumps(obj, separators=(",", ":")).encode("utf-8")) + + +def decode_json(payload): + """Return the object in a JSON frame payload.""" + try: + return json.loads(payload.decode("utf-8")) + except (UnicodeDecodeError, ValueError) as exc: + raise FrameError("payload is not JSON: {}".format(exc)) + + +class Reader: + """Read frames from a blocking binary stream. + + read() returns a (kind, payload) pair, or None once the peer has closed + the stream cleanly between frames. A stream that ends part way through a + frame raises FrameError, because a truncated frame is a real fault and + silently treating it as end of input would hide a dropped connection. + """ + + def __init__(self, stream): + self._stream = stream + + def _exact(self, count, what): + """Return exactly count bytes, or None if the stream ended before any. + + Ending part way through raises rather than returning None, because the + two are not the same fault and only the caller reading a header can + treat nothing-at-all as end of input. A partial header returned as None + would be read as a clean close, and a dropped connection would be + recorded as a turn the model simply did not answer. + """ + parts = [] + have = 0 + while have < count: + chunk = self._stream.read(count - have) + if not chunk: + if have: + raise FrameError( + "stream ended after {} of the {} bytes of a {}".format( + have, count, what)) + return None + parts.append(chunk) + have += len(chunk) + return b"".join(parts) + + def read(self): + head = self._exact(HEADER.size, "frame header") + if head is None: + return None + kind, length = HEADER.unpack(head) + check_header(kind, length) + if length == 0: + return kind, b"" + payload = self._exact(length, "payload") + if payload is None: + raise FrameError("stream ended inside a {} byte payload".format(length)) + return kind, payload + + +class Writer: + """Write frames to a blocking binary stream, flushing each one. + + Every frame is flushed because a buffered reply frame is indistinguishable + from a slow model, and this build exists to measure the difference. + """ + + def __init__(self, stream): + self._stream = stream + + def send(self, kind, payload=b""): + self._stream.write(encode(kind, payload)) + self._stream.flush() + + def send_json(self, kind, obj): + self._stream.write(encode_json(kind, obj)) + self._stream.flush() diff --git a/bin/fm_voice_records.py b/bin/fm_voice_records.py new file mode 100755 index 00000000000..55064f938f4 --- /dev/null +++ b/bin/fm_voice_records.py @@ -0,0 +1,574 @@ +#!/usr/bin/env python3 +"""fm_voice_records.py - what the voice agent is allowed to know, and how it hands work over. + +The voice agent answers status questions from firstmate's durable records and +queues everything else. This module owns both halves, because both halves are +where a mistake is expensive: one sends the captain's records to a model in +another region, and the other writes to firstmate's wake queue. + +WHAT IS NEVER READ. Two whole classes of record are excluded at every scope, +not filtered at the end: + + Done history, because a spoken "what is happening" answer is about open work, + and the finished items are where old engagements accumulate. + Free-form note bodies under a task, because they are long, they are written + for a reader with the whole file in front of them, and they are where + commercial detail gets quoted. + +Only open task lines and this home's own runtime records are ever assembled. +That is a confidentiality boundary as much as a brevity one. Verified on the +captain's live records on 2026-08-21: every occurrence of the one engagement +identifier those records contain sits in Done history or a note body, so +nothing in a full status answer named a customer. tests/fm-voice-relay.test.sh +holds that boundary as an executable check, so widening the reader later fails +the test rather than quietly widening what is sent. + +Runtime records outlive the work they describe: a task keeps its state/<id>.meta +until teardown removes it, which happens separately from marking the item done. +Two readings here treat that differently, on purpose. + + Pull requests, the count and the list, cover OPEN ids only. They name work, and + they feed the deny decision, which needs an open item to take a title from. A + finished task's pull request is therefore not counted and not named, and that + lost count is a deliberate cost: the alternative names finished work and puts + it out of reach of the deny list, which has no title to match without an open + item to take it from. + + The worker count and the state histogram cover every live runtime record, + finished ids included, because a task with a meta file still on disk is still + on deck and still needs tearing down. That is the question those two figures + answer, and it is the same meaning bin/fm-inbox.sh gives "workers" in the human + rendering. Neither can carry record free text: one is an integer, and the + other's keys are the state verb folded through the closed set below. + +READ SCOPE. config/voice-read-scope selects what a status answer may contain: + + counts (the default, and the value used when the file is absent) + Counts, states and one basis note, with no record free text assembled at + all. Safe by construction rather than by filtering: the agent can say how + much is waiting without saying what it is. This is the default because a + home that has configured nothing has granted nothing, and sending task + identifiers, titles and pull request links to a model in another region is + not something to inherit from somebody else's settings file. + + full + Counts plus the identifiers, titles and pull request links of open work. + A home widens to this by writing `full` into config/voice-read-scope, + which is the access being granted deliberately by the captain whose + records they are. + +DENY LIST. config/voice-read-deny holds anything that must never leave this +host even in full scope: one plain case-insensitive substring per line, `#` +starts a comment, blank lines ignored. Substrings rather than regular +expressions, because a confidentiality list is the wrong place for a pattern +that can match more or less than it looks like it matches. Each open item is +matched once, against its identifier, its title, its tag values and its pull +request link together, and a match is then withheld from every list it could +have appeared in and reduced to a withheld count. One decision per item rather +than one per list, because an item named in any list is an item that left this +host. The agent still says how much is waiting without saying what it is. The +file is optional and an absent file means an empty list; it exists so that a +future open task carrying a customer name can be excluded in one line rather +than by turning the whole feature down. + +WORKER STATE. This module reports the last recorded event verb, which is +history rather than a live check, and labels it that way in its own output so +the model cannot present it as current truth. bin/fm-crew-state.sh remains the +owner of real current-state reconciliation and is far too slow for a spoken +answer. The verb is folded through the closed STATE_VERBS vocabulary below, and +anything outside it becomes "note": a status line is free text, and this verb is +the only thing derived from a record that a counts-scope answer says out loud. + +bin/fm-inbox.sh `status` is the human rendering of the same records and stays +the owner of that. This module exists because a spoken answer needs a machine +shape and a read scope that the human rendering has no reason to carry. + +Usage: + fm_voice_records.py status [--home <dir>] [--scope counts|full] + fm_voice_records.py queue <text>... [--home <dir>] + +Both subcommands print JSON, which is exactly what the relay hands to the model +as a tool result, so the shell form is the same interface the relay uses. +""" + +import argparse +import json +import os +import re +import subprocess +import sys + +SCOPE_COUNTS = "counts" +SCOPE_FULL = "full" +SCOPES = (SCOPE_FULL, SCOPE_COUNTS) +SCOPE_DEFAULT = SCOPE_COUNTS + +BASIS = "Last recorded event, which is history and not a live check." + +# A spoken answer names a few things and gives a count for the rest. Every row +# sent is input tokens the model reads before it starts speaking, and this whole +# build exists to keep that delay honest, so the lists are capped rather than +# complete. A complete list is a screen, not a sentence. +DETAIL_LIMIT = 5 + +ITEM = re.compile(r"^- \[(?P<done>[ x])\] (?P<id>\S+) - (?P<rest>.*)$") +TAG = re.compile(r"\((?P<key>[a-z-]+): (?P<value>[^)]*)\)") +# (since 2026-08-21) and (done 2026-08-21) carry no colon, so the tag pattern +# leaves them in the title. A date read aloud in the middle of a sentence is +# noise, so they come out too. +DATE_TAG = re.compile(r"\((?:since|done) [0-9-]+\)") + +# The only backlog sections this module will parse. Done history is skipped +# before a line is even split, so widening the answer cannot reach it by +# accident. See "WHAT IS NEVER READ" above. +READ_SECTIONS = ("in flight", "queued") + +# The states a worker is asked to report, and the two more that close a decision. +# bin/fm-brief.sh states the first six to every crewmate and bin/fm-classify-lib.sh +# owns resolved and captain-held; this module only recognises them. +# +# A CLOSED set, not a shape. A status line is free text appended by a crewmate, +# and the verb taken off the front of it is the one record-derived string that +# reaches a counts-scope answer, where there are no titles or links for a deny +# list to filter. So an unrecognised token is reported as a note instead of being +# spoken, exactly as a malformed one already was; otherwise a crewmate writing +# "acmecorp-migration: waiting on their review" would put that word in front of a +# model in another region, with nothing in config/voice-read-deny able to stop it. +STATE_VERBS = ("working", "needs-decision", "blocked", "paused", "done", + "failed", "resolved", "captain-held") +NOTE_VERB = "note" + +# Enough tail to hold the last line of a status log. These logs are append-only +# and grow for the life of a task, while every spoken question reads one per +# worker, so the read is bounded and seeks rather than scanning from the top. +STATUS_TAIL_BYTES = 8192 + + +class RecordError(Exception): + """The records or the read-scope configuration cannot be used as asked.""" + + +def default_home(): + """Return the operational home, matching bin/fm-inbox.sh's resolution.""" + env = os.environ.get("FM_HOME") + if env: + return env + return os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + + +def state_dir(home): + """Return the runtime state directory, resolved as bin/fm-inbox.sh resolves it. + + fm-inbox.sh reads ${FM_STATE_OVERRIDE:-$FM_HOME/state}, and the handover + below queues through fm-inbox.sh with the ambient environment. A reader that + ignored the override would count notes in one directory while the queue wrote + them to another, so the agent would tell the captain their request was queued + and then, asked what is waiting, report nothing. + """ + override = os.environ.get("FM_STATE_OVERRIDE") + if override: + return override + return os.path.join(home, "state") + + +def data_dir(home): + """Return the durable records directory, the other half of the same pair. + + Every script that sets FM_DATA_OVERRIDE for a child sets FM_STATE_OVERRIDE + beside it, so resolving one and not the other would answer one question from + two different homes: counts of workers and notes from the overridden state + directory, counts of in-flight and queued work from the home's own backlog. + A spliced answer is worse than a wrong one, because nothing about it looks + wrong. + """ + override = os.environ.get("FM_DATA_OVERRIDE") + if override: + return override + return os.path.join(home, "data") + + +def config_dir(home): + """Return the configuration directory, honouring the repo-wide override.""" + override = os.environ.get("FM_CONFIG_OVERRIDE") + if override: + return override + return os.path.join(home, "config") + + +def _read_config(home, name): + path = os.path.join(config_dir(home), name) + try: + with open(path, encoding="utf-8") as handle: + return handle.read() + except FileNotFoundError: + return None + + +def read_setting(home, name, env=None): + """Return a one-line setting from the environment or this home's config, else None. + + The values this feature needs, an AWS profile and a region and a model id, + name somebody's account and somebody's choices. They belong to the home that + runs the relay rather than to the repository, so they are read from gitignored + config/ with an environment override and never carry a tracked default. + """ + if env: + value = (os.environ.get(env) or "").strip() + if value: + return value + raw = _read_config(home, name) + if raw is None: + return None + for line in raw.splitlines(): + text = line.split("#", 1)[0].strip() + if text: + return text + return None + + +def require_setting(home, name, env, what): + """Return a setting, or refuse naming the file to write and the variable to set.""" + value = read_setting(home, name, env) + if value is None: + raise RecordError( + "no {} is configured: write one line into {} or set {}".format( + what, os.path.join(config_dir(home), name), env)) + return value + + +def read_scope(home): + """Return the configured read scope, defaulting to the narrowest one.""" + raw = _read_config(home, "voice-read-scope") + if raw is None: + return SCOPE_DEFAULT + value = raw.strip() + if not value: + return SCOPE_DEFAULT + if value not in SCOPES: + raise RecordError( + "config/voice-read-scope says {!r}; it must be one of {}".format( + value, " or ".join(SCOPES))) + return value + + +def deny_list(home): + """Return the deny substrings; an absent file means an empty list.""" + raw = _read_config(home, "voice-read-deny") + if raw is None: + return [] + out = [] + for line in raw.splitlines(): + text = line.split("#", 1)[0].strip() + if text: + out.append(text.lower()) + return out + + +def _denied(denies, *fields): + haystack = " ".join(f for f in fields if f).lower() + return any(needle in haystack for needle in denies) + + +def _parse_backlog(path): + """Return (section, item) pairs for every task line in the backlog.""" + items = [] + section = "" + try: + with open(path, encoding="utf-8") as handle: + lines = handle.read().splitlines() + except FileNotFoundError: + return items + for line in lines: + if line.startswith("## "): + section = line[3:].strip().lower() + continue + if section not in READ_SECTIONS: + continue + match = ITEM.match(line) + if not match: + continue + rest = match.group("rest") + tags = {m.group("key"): m.group("value") for m in TAG.finditer(rest)} + title = re.sub(r"\s+", " ", DATE_TAG.sub("", TAG.sub("", rest))).strip() + items.append({ + "section": section, + "id": match.group("id"), + "title": title, + "done": match.group("done") == "x", + "tags": tags, + }) + return items + + +def _last_event(state_dir, task_id): + """Return (verb, line) from the last status event, or (None, None). + + The verb is what precedes the first ':' and the first '[', whichever comes + first, which is what status_line_verb in bin/fm-classify-lib.sh does and + that remains the owner of the format. The bracket matters: status metadata + sits between the verb and the colon, as in "done [token]: shipped it" and + "needs-decision [key=api-shape]: which shape". A line carrying no colon is + not a status line, and any token outside STATE_VERBS is reported as a note + rather than spoken aloud as a state. + + Only the tail of the log is read; see STATUS_TAIL_BYTES. + """ + path = os.path.join(state_dir, task_id + ".status") + try: + with open(path, "rb") as handle: + handle.seek(0, os.SEEK_END) + size = handle.tell() + handle.seek(max(0, size - STATUS_TAIL_BYTES)) + window = handle.read() + except OSError: + return None, None + lines = [text.strip() for text in + window.decode("utf-8", errors="replace").splitlines() if text.strip()] + if not lines: + return None, None + line = lines[-1] + verb = NOTE_VERB + if ":" in line: + verb = line.split(":", 1)[0].split("[", 1)[0].strip().lower() + if verb not in STATE_VERBS: + verb = NOTE_VERB + return verb, line + + +def _workers(state_dir): + """Return one record per task with runtime metadata in this home.""" + out = [] + try: + names = sorted(n for n in os.listdir(state_dir) if n.endswith(".meta")) + except OSError: + return out + for name in names: + task_id = name[: -len(".meta")] + meta = {} + try: + with open(os.path.join(state_dir, name), encoding="utf-8") as handle: + for line in handle: + if "=" in line: + key, value = line.rstrip("\n").split("=", 1) + meta[key] = value + except OSError: + continue + verb, line = _last_event(state_dir, task_id) + out.append({ + "id": task_id, + "kind": meta.get("kind", ""), + "mode": meta.get("mode", ""), + "pr": meta.get("pr", ""), + "verb": verb or "no events yet", + "line": line or "", + }) + return out + + +def fleet_status(home=None, scope=None): + """Return the status answer the voice agent is allowed to give.""" + home = home or default_home() + scope = scope or read_scope(home) + if scope not in SCOPES: + raise RecordError("unknown read scope: {!r}".format(scope)) + denies = deny_list(home) + + state = state_dir(home) + workers = _workers(state) + items = _parse_backlog(os.path.join(data_dir(home), "backlog.md")) + + open_items = [i for i in items if not i["done"]] + in_flight = [i for i in open_items if i["section"] == "in flight"] + queued = [i for i in open_items if i["section"] == "queued"] + # "What is waiting on me" is the union of decisions filed for the captain + # and anything explicitly held for them. The two overlap but neither + # contains the other, because a decision can be filed before it is held. + held_for_captain = [ + i for i in open_items + if i["tags"].get("hold-kind") == "captain" + or i["tags"].get("kind") == "captain" + ] + # OPEN work only. _workers lists every state/*.meta in the home, and a task + # keeps its meta after it is marked done until teardown removes it, so taking + # every worker with a pull request would count and name finished tasks. That + # breaks the promise at the top of this file twice over: it reads finished + # work, and the deny decision below cannot reach those items, because their + # ids have no open item to supply a title, so a captain substring matching a + # title would silently fail for exactly them. Losing the count of a pull + # request on a task already marked done is the accepted cost. + open_ids = {i["id"] for i in open_items} + with_pr = [w for w in workers if w["pr"] and w["id"] in open_ids] + + inbox = os.path.join(state, "inbox") + try: + waiting = len([n for n in os.listdir(inbox) if n.endswith(".note")]) + except OSError: + waiting = 0 + + states = {} + for worker in workers: + states[worker["verb"]] = states.get(worker["verb"], 0) + 1 + + answer = { + "scope": scope, + "basis": BASIS, + "workers_on_deck": len(workers), + "worker_states": states, + "in_flight": len(in_flight), + "queued": len(queued), + "awaiting_captain": len(held_for_captain), + "open_pull_requests": len(with_pr), + "captain_notes_waiting": waiting, + } + if scope == SCOPE_COUNTS: + answer["detail"] = ( + "Identifiers, titles and pull request links are withheld at this " + "read scope. Say that the detail is not available by voice rather " + "than guessing at it.") + return answer + + by_id = {w["id"]: w for w in workers} + + # ONE deny decision per item, taken over everything known about that item + # before any list is built, and then shared by every list it could appear + # in. The lists overlap by design: a task can be in flight, waiting on the + # captain and carrying a pull request at once. Deciding per list, from the + # fields that list happens to use, would withhold an item from one list and + # name it in another, which is not a narrower answer but a leak with a + # reassuring count beside it. It also makes the count what it says it is, + # distinct items rather than refusals. + # + # The fields come from every OPEN item, not only the ones a list iterates. A + # queued item that is not held for the captain still reaches the answer + # through its pull request link, and assembling its fields only where a list + # walks past it is how a title match gets missed on exactly that item. What + # is COUNTED is narrower: an item that no list could have named is not + # something the captain is having withheld. + known = {} + for item in open_items: + known.setdefault(item["id"], item) + nameable = ({i["id"] for i in in_flight} | {i["id"] for i in held_for_captain} + | {w["id"] for w in with_pr}) + + withheld_ids = set() + for item_id in nameable: + item = known.get(item_id) + worker = by_id.get(item_id) + fields = [item_id] + if item is not None: + fields.append(item["title"]) + fields.extend(item["tags"].values()) + if worker is not None: + fields.append(worker["pr"]) + if _denied(denies, *fields): + withheld_ids.add(item_id) + + def keep(item_id): + return item_id not in withheld_ids + + detail_in_flight = [] + for item in in_flight: + if not keep(item["id"]): + continue + worker = by_id.get(item["id"]) + detail_in_flight.append({ + "id": item["id"], + "title": item["title"], + # The state word only, never the raw event line. The agent speaks to + # the captain and must not read internal record text aloud. + "state": worker["verb"] if worker else "not started", + }) + + detail_captain = [] + for item in held_for_captain: + if not keep(item["id"]): + continue + detail_captain.append({"id": item["id"], "title": item["title"]}) + + detail_prs = [] + for worker in with_pr: + if not keep(worker["id"]): + continue + detail_prs.append({"id": worker["id"], "url": worker["pr"]}) + + def capped(rows, key): + answer[key] = rows[:DETAIL_LIMIT] + if len(rows) > DETAIL_LIMIT: + answer[key + "_not_listed"] = len(rows) - DETAIL_LIMIT + + capped(detail_in_flight, "in_flight_detail") + capped(detail_captain, "awaiting_captain_detail") + capped(detail_prs, "pull_request_detail") + answer["withheld_as_confidential"] = len(withheld_ids) + answer["detail"] = ( + "The lists name at most {} items each; the counts above are the whole " + "picture. Give the captain the counts and a couple of names, not every " + "row.".format(DETAIL_LIMIT)) + return answer + + +def queue_request(text, home=None, root=None): + """Hand real work to firstmate through bin/fm-inbox.sh note.""" + home = home or default_home() + root = root or os.path.dirname(os.path.abspath(__file__)) + body = (text or "").strip() + if not body: + raise RecordError("refusing to queue an empty request") + inbox = os.path.join(root, "fm-inbox.sh") + if not os.access(inbox, os.X_OK): + raise RecordError("cannot run {}".format(inbox)) + env = dict(os.environ, FM_HOME=home) + done = subprocess.run( + [inbox, "note", body], + # The relay's stdin is the captain's audio when this runs under + # --serve, and fm-inbox.sh reads a body from stdin for an argument of + # "-", so no child of the relay is given that stream to consume. + stdin=subprocess.DEVNULL, + env=env, capture_output=True, text=True, timeout=30, check=False) + if done.returncode != 0: + raise RecordError("fm-inbox.sh note failed: {}".format( + (done.stderr or done.stdout).strip())) + note_id = "" + for line in done.stdout.splitlines(): + if line.startswith("queued "): + note_id = line.split(None, 1)[1].strip() + break + return { + "queued": True, + "note_id": note_id, + "queued_text": body, + "handover": "Firstmate now owns this request and will pick it up at " + "its next check. You did not do the work yourself.", + } + + +def main(argv): + parser = argparse.ArgumentParser( + prog="fm_voice_records.py", description=__doc__.splitlines()[0], + formatter_class=argparse.RawDescriptionHelpFormatter) + sub = parser.add_subparsers(dest="command", required=True) + + status = sub.add_parser("status", help="print the allowed status answer") + status.add_argument("--home") + status.add_argument("--scope", choices=SCOPES) + + queue = sub.add_parser("queue", help="hand a request to firstmate") + queue.add_argument("text", nargs="+") + queue.add_argument("--home") + + args = parser.parse_args(argv) + try: + if args.command == "status": + result = fleet_status(home=args.home, scope=args.scope) + else: + result = queue_request(" ".join(args.text), home=args.home) + except RecordError as exc: + sys.stderr.write("fm_voice_records: {}\n".format(exc)) + return 2 + json.dump(result, sys.stdout, indent=2, sort_keys=True) + sys.stdout.write("\n") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/docs/architecture.md b/docs/architecture.md index a1d7d77753f..0f1cf8dd9ac 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -9,9 +9,13 @@ firstmate's always-loaded operating contract and routing index for conditional p ## Event-driven supervision A zero-token bash watcher (`bin/fm-watch.sh`) sleeps on the fleet, classifies detected wakes in bash, and wakes the first mate only when something is actionable. -Actionable wakes include captain-relevant status signals, no-verb signals whose crew is not provably working, authenticated check output such as PR merge polling or a Relay mention, stale panes whose crew is not provably working whether their status log looks terminal or non-terminal, provably-working stale panes that persist past `FM_STALE_ESCALATE_SECS`, declared external waits that remain paused past `FM_PAUSE_RESURFACE_SECS`, and heartbeat backstop hits. +Actionable wakes include captain-relevant status signals, no-verb signals whose crew is not provably working, authenticated check output such as PR merge polling or a Relay mention, stale panes whose crew is not provably working whether their status log looks terminal or non-terminal, provably-working stale panes that persist past `FM_STALE_ESCALATE_SECS` without their own task worktree being written, declared external waits and verified captain-held transfers that remain declared past `FM_PAUSE_RESURFACE_SECS`, and heartbeat backstop hits. Repeated provably-working stale escalations on the same unchanged pane add an escalation count to the wake reason and, at `FM_WEDGE_DEMAND_INSPECT_COUNT`, a `demand-deep-inspection` marker. -A busy pane is otherwise exempt from staleness, but only until its latest `state/<id>.turn-ended` marker reaches `FM_BUSY_TURN_MAX_SECS`, or its `state/<id>.meta` spawn record reaches that age before any turn completes; past that bound it is routed through the same wedge escalation, with the identical reason, escalation count, and `demand-deep-inspection` marker, for inspection only - never an automatic interrupt, signal, or restart. +A pane holding a file newer than the start of its own quiet window, anywhere in the worktree recorded for that task, is deferred instead of escalated, because a crew writing source, then tests, then documentation behind a static pane is liveness that neither pane quietness nor the run step can show. +That deferral re-surfaces on the same `FM_PAUSE_RESURFACE_SECS` cadence as a declared wait, with a reason naming the write evidence rather than a wedge, and it is bounded to one pruned, depth-bounded, wall-clock-bounded walk (`FM_WORKTREE_WRITE_PRUNE`, `FM_WORKTREE_WRITE_MAXDEPTH`, `FM_WORKTREE_WRITE_TIMEOUT`) taken only in the branch that was about to escalate, never on every poll. +Every absence of write evidence, including a missing worktree record, a torn-down worktree, a walk that outlives its wall-clock bound on a hung mount, and a failed walk, leaves the existing escalation schedule untouched, so a crew that writes nothing still escalates exactly as before. +A secondmate is never probed at all, because the worktree recorded for it is a provisioned firstmate home whose own supervision keeps writing inside it whether or not the mate produces anything, so its panes keep escalating on the unchanged schedule. +A busy pane is otherwise exempt from staleness, but only until its latest `state/<id>.turn-ended` marker reaches `FM_BUSY_TURN_MAX_SECS`, or its `state/<id>.meta` spawn record reaches that age before any turn completes; past that bound it is routed through the same wedge escalation, with the identical reason, escalation count, worktree-write deferral, and `demand-deep-inspection` marker, for inspection only - never an automatic interrupt, signal, or restart. A crew that declared an external wait (`paused:`) or a verified captain-held transfer is the one exception to that bound: its busy verdict supplies liveness while identifying the long-running foreground call as the declared wait, so it takes the bounded `FM_PAUSE_RESURFACE_SECS` recheck instead of a wedge escalation. Lifting the declaration restores the unchanged busy-pane wedge path, while a pane that is no longer busy returns to the existing idle declared-wait classification. Those actionable wakes are written to a durable local queue (`state/.wake-queue`) only after generation-bound recovery evidence is published, so an interrupted watcher or handling turn can be recovered without losing the queue record. @@ -21,9 +25,9 @@ A concurrent replacement remains armed, every non-merged or invalid observation `bin/fm-pr-lib.sh` owns the receipt format and strict identity mechanics, while `bin/fm-watch.sh` owns queue-before-retirement ordering. No-verb wakes, such as `working:` notes and bare turn-ended signals, are benign only when `bin/fm-crew-state.sh` reports positive evidence that the crew is still working: an actively running no-mistakes step attributed to that crew's current code, or an exact busy verdict from the semantic busy-state contract. A `kind=secondmate` task's status signal is the parent-directed reply stream and is never absorbed as provably working; only its bare turn-ended signal retains the ordinary absorb rule. -A crew that declares `paused:` for a known external wait is separately absorbed while idle and re-surfaced only on the longer pause cadence, rather than being treated as a possible wedge. +A crew that declares `paused:` for a known external wait, or carries a verified `captain-held` transfer, is separately absorbed while idle and re-surfaced only on the longer pause cadence, rather than being treated as a possible wedge. For an ordinary crew that has stopped, the normal-mode watcher first surfaces one stale wake, then applies that same cadence to an unchanged `paused:` or durable `captain-held` endpoint only when the backend confidently reports its agent dead. -Live or inconclusive liveness remains fail-open at that initial surface, and the secondmate idle-endpoint exemption is unchanged. +Live or inconclusive liveness remains fail-open at that initial surface, and a secondmate's endpoint liveness is still never read at all; a mate is admitted to that same cadence only to serve a declared wait's bounded re-surface, so a forgotten pause or captain hold on a mate cannot rot invisibly. Its initial normal-mode status signal still surfaces through the no-verb path, while away mode self-handles that routine signal and owns the later recheck. Fresh stale panes use the same current-state read before trusting the status log, so an active run or a proven busy worker outranks an old captain-relevant status-log line left behind before validation. No-change heartbeats are also benign. @@ -33,11 +37,12 @@ A secondmate retains a durable receipt for its idempotent report through the est Absorbed wakes advance their suppression markers, log to `state/.watch-triage.log`, and keep the watcher blocking without a queue record or LLM turn. Each `fm-wake-drain.sh` presentation runs the same liveness guard as the supervision scripts, so a lapsed watcher chain surfaces even on a turn that only handles queued wakes. Routine watcher polling, supervision no-ops, elapsed waiting time, and absorbed benign wakes stay silent. -A declared external wait trades that silence for one bounded recheck per pause window, so a forgotten pause cannot remain invisible indefinitely. +A declared external wait or verified captain-held transfer trades that silence for one bounded recheck per pause window, naming which human the wait is on, so neither a forgotten pause nor a forgotten hold can remain invisible indefinitely. Crew status files are append-only wake-event logs, not current-state fields. Because of that, a per-wake read of only the latest line can bury an earlier still-open `needs-decision`/`blocked` under later unrelated appends; `fm-wake-drain.sh` prints a separate, fleet-wide OPEN DECISIONS section on every presentation (including the empty-queue path session-start relies on), built through `fm-classify-lib.sh`'s cursor-backed incremental scan using the authoritative `status_open_decisions` fold semantics so the buried decision keeps surfacing until it is explicitly resolved while each presentation folds only new status-log appends. The drain coordinates that fold and its annotations through a locked fleet-wide snapshot whose `.status-presentation-cursor` manifest records each status file's identity and last-presented byte offset. A queued signal annotation prints every status line still unread at that cursor, while the fleet-wide UNREAD STATUS section prints `note:` lines and reserved-key pending-reply resolutions once even on an empty-queue drain because those verbs never enter the OPEN DECISIONS fold. +A third bounded section, RECORD DIVERGENCE, prints on the same drains for the opposite failure: the status fold went quiet on a key that the durable captain-held task still shows as open, so the status side reads as complete while the two records contradict each other; `bin/fm-captain-hold.sh diverged` decides what counts and closes nothing, and `docs/captain-hold-lifecycle.md` owns the mechanism. A failed read, output, or concurrent-replacement check prevents the snapshot cursor from advancing across uncertain bytes, and teardown retires a task's manifest row before that task ID can be reused. The explicit resolution is written by the actor that answers, not the busy worker: `fm-send`'s `--resolve-key` appends the closing `resolved` line to this home's own copy of the ledger at answer time, which covers crewmates, local secondmates, and remote secondmates identically because a remote mate's escalations reach that local copy through the parent-replies ingest and only the answer message itself crosses the transport. This home's answerer close, pending-reply escalation close, and captain-held transfer use the provenance-guarded append owned by `bin/fm-wake-lib.sh`, so they advance the watcher marker only across their own bytes when all earlier bytes were already announced; pending or interleaved foreign bytes fail toward an ordinary wake. @@ -87,11 +92,12 @@ On every verified primary harness, tracked hook integration gives the primary se The guard covers the main primary and genuinely marked secondmate homes, exempts child crewmate/scout worktrees, is loop-safe per harness, and is documented in [turnend-guard.md](turnend-guard.md). A presence-gated sub-supervisor (`bin/fm-supervise-daemon.sh`) extends this for walk-away supervision: the `/afk` skill starts it through the tracked foreground helper `bin/fm-afk-start.sh`, after which the watcher reverts to daemon-managed one-shot mode and the daemon self-handles routine wakes in bash. -The watcher and daemon share `bin/fm-classify-lib.sh` for captain-relevant status verbs, declared-external-wait vocabulary, and status-scan primitives. +The watcher and daemon share `bin/fm-classify-lib.sh` for captain-relevant status verbs, declared-wait vocabulary (a `paused:` external wait and a verified `captain-held` transfer alike, through one combined predicate), and status-scan primitives. Terminal verbs remain captain-relevant, while a nonterminal progress verb cannot become terminal merely because its prose contains a legacy free-text token such as `merged`; bare legacy free-text lines remain compatible. -The always-on watcher also uses that library's absorb classification on no-verb signals and first-sighting stale panes before status-log terminality is trusted, while the daemon maintains distinct wedge and declared-pause recheck cadences. +The always-on watcher also uses that library's absorb classification on no-verb signals and first-sighting stale panes before status-log terminality is trusted, while the daemon maintains distinct wedge and declared-wait recheck cadences. In away mode, seen-status dedupe does not clear possible-wedge aging for nonterminal progress, so housekeeping still re-escalates an unchanged idle pane at the configured bound. -The daemon escalates captain-relevant events, plus a bounded recheck for a declared pause that remains idle, as one batched, single-line digest using the canonical `away-supervisor` kind from `bin/fm-operational-input.sh` so firstmate can distinguish it structurally from real messages. +Away-mode housekeeping has no worktree-write deferral of its own, so while `state/.afk` exists a quiet crew that is writing its own worktree still escalates as a possible wedge at that bound. +The daemon escalates captain-relevant events, plus a bounded recheck for a declared pause or a verified captain-held transfer that remains idle, naming which human that wait is on, as one batched, single-line digest using the canonical `away-supervisor` kind from `bin/fm-operational-input.sh` so firstmate can distinguish it structurally from real messages. Its supervisor injection path supports tmux and herdr panes, with `FM_SUPERVISOR_BACKEND` and `FM_SUPERVISOR_TARGET` resolved independently from the task-spawn backend. Pane existence, busy checks, composer checks, capture, and verified submit route through `bin/fm-backend.sh`: tmux keeps the same submit core used by the tmux send backend, while herdr uses native agent-state submit confirmation on idle baselines, a composer empty fallback when native stays idle, and a pre-Enter rendered-footer transition when that baseline is unavailable. The retries-exhausted queued-Enter decision is owned by `fm_composer_queued_enter_verdict` in `bin/fm-composer-lib.sh`; tmux and herdr provide only their backend-specific busy signals. @@ -239,15 +245,19 @@ The `data/secondmates.md` line contract is owned by the [`secondmate-provisionin ## Delivery modes are explicit per task `no-mistakes` tasks run the full validation pipeline, `direct-PR` tasks open PRs without that pipeline, and `local-only` tasks stay local until firstmate performs an approved fast-forward merge. -Each task's mode and `yolo` posture are firstmate's decision at intake and are passed explicitly to `bin/fm-brief.sh`, `bin/fm-spawn.sh`, and `bin/fm-promote.sh`, which refuse a ship task that does not carry them. +Each task's mode and `yolo` merge posture are firstmate's decision at intake. +The mode is passed explicitly to `bin/fm-brief.sh`, and both values are passed explicitly to `bin/fm-spawn.sh` and `bin/fm-promote.sh`; each command refuses to guess the values it consumes. A ship brief records its mode as a fixed machine-readable line and the spawn refuses to launch on a different one, so the worker's instructions and the recorded task delivery cannot diverge. -`data/projects.md` records each project's standing posture and optional `+yolo` flag as the captain's default and as context for that decision, including the conditional `no-mistakes-prod-only` policy; a ship spawn that drops below the registered rigor prints a deviation notice and continues. +`data/projects.md` records each project's standing posture and optional `+yolo` merge flag as the captain's default and as context for that decision, including the conditional `no-mistakes-prod-only` policy; a ship spawn that drops below the registered rigor prints a deviation notice and continues. `bin/fm-project-mode.sh` remains the one registry parser for the mechanical consumers that have no task in hand: fleet sync's `local-only` skip and home seeding's refusal and no-mistakes initialization. When a selected delivery path calls for a diff, `bin/fm-review-diff.sh` refreshes the authoritative base and, when task meta records `pr=`, always fetches and compares against `refs/pull/<n>/head` by default (recorded `pr_head=` is only an offline fallback) before falling back to the local branch with a warning. Where a no-mistakes pipeline stores evidence in the repo, it publishes that PR-viewable validation evidence to an orphan evidence branch that shares no history with code branches, so it never enters the crew branch or the default branch. This repo uses that setting, and its own `.no-mistakes/` directory remains local state that stays gitignored and is rejected by CI if tracked; [`configuration.md`](configuration.md) owns the setting. -PR-based task merges go through `bin/fm-pr-merge.sh`, which records `pr=` and any available `pr_head=` through `bin/fm-pr-check.sh` before calling `gh-axi pr merge`. -The helper requires a full `https://github.com/<owner>/<repo>/pull/<n>` URL, invokes `gh-axi pr merge <n> --repo <owner>/<repo>`, defaults to `--squash`, preserves explicit merge-method flags, and rejects malformed URLs or repo override flags before recording merge state; a well-formed GitLab merge request URL (see [docs/gitlab-merge-watch.md](gitlab-merge-watch.md)) is refused too, explicitly, rather than sent to the wrong forge. +PR-based task merges go through `bin/fm-pr-merge.sh`, which records `pr=` and any available `pr_head=` through `bin/fm-pr-check.sh` before calling the forge CLI. +The helper requires a full canonical URL and rejects malformed URLs or repo override flags before recording merge state. +A `https://github.com/<owner>/<repo>/pull/<n>` URL invokes `gh-axi pr merge <n> --repo <owner>/<repo>`, defaults to `--squash`, and preserves explicit merge-method flags. +A `https://<host>/<path>/-/merge_requests/<n>` URL (see [docs/gitlab-merge-watch.md](gitlab-merge-watch.md)) invokes `glab mr merge <n> -R https://<host>/<path>`, so the instance comes from the URL, and adds no merge-method flag because the project's own merge method applies. +That path merges only after one live read of the merge request confirms it is open, mergeable, conflict-free, with blocking discussions resolved and a successful pipeline at the current head, and it binds the merge to that verified head; recorded metadata is never the authority for those conditions because a rebase leaves it stale. Teardown is fail-closed for ship worktrees: dirty worktrees refuse, and committed work must be landed before the worktree is returned. [`bin/fm-teardown.sh`](../bin/fm-teardown.sh)'s header owns the landed-work proofs, PR-discovery fallback, and stale-lock recovery procedure. @@ -286,7 +296,7 @@ The mechanism boundary is deliberately narrow. `tasks-axi` owns the obligation state machine and is the only thing that validates a terminal result's source home, work id, generation, schema, outcome, and deliverables. `state/x-context/` remains the only owner of the private full request context. `bin/fm-x-reply.sh` remains the only thing that posts. -`bin/fm-public-followup.sh` composes those three and adds nothing of its own beyond the activation gate, a private terminal-event inbox, and the idempotent delivery sequence. +`bin/fm-public-followup.sh` composes those three and adds the activation gate, a private terminal-event inbox, the idempotent delivery sequence, and retained-loop disposition: delivery stamps the registration delivered, `rechain` hands its thread binding to one follow-on obligation, and `retire` is the only close. Work routed to another home reports a *typed* terminal result through `bin/fm-public-followup-emit.sh`; firstmate never recovers the source home, work id, outcome, or deliverables by parsing a free-form `done:` sentence, and the child never learns the thread. Because a terminal event's id is derived from its identity tuple rather than generated, duplicate reports and restart replay converge without coordination. Reconciliation rides the existing relay poll and the session-start digest instead of a new watcher, daemon, or timer, and both are gated on the same `.env` activation contract so a home that never opted into the relay executes none of it. diff --git a/docs/captain-hold-lifecycle.md b/docs/captain-hold-lifecycle.md new file mode 100644 index 00000000000..cb8d5cea29a --- /dev/null +++ b/docs/captain-hold-lifecycle.md @@ -0,0 +1,95 @@ +# Captain-hold lifecycle mechanism + +The normative policy is owned by `.agents/skills/captain-hold-lifecycle/SKILL.md` and is not restated here. +This document records the deterministic mechanism, structured surfaces, compatibility contract, and privacy-safe regression evidence. + +## Mechanism + +A decision is not a separate thing in this system: it is an ordinary backlog task held for the captain, and the task id is the identity every surface and channel uses. +`bin/fm-captain-hold.sh` is the only lifecycle command layered on that primitive. +The command runs tasks-axi in the active `FM_HOME`, so the existing backlog remains the only durable work database and a secondmate-owned captain call stays in the secondmate home. +It never reads report bodies, review artifacts, terminal output, or chat. + +The `hold` subcommand places an existing task under an active captain hold, or creates the task when nothing exists to hold, then verifies the hold through `tasks-axi hold <id> --reason <reason> --kind captain`. +Repeats are idempotent, a closed task is refused rather than reopened, and `--until` stores the captain's own deferral date through tasks-axi's date gate. + +The `answer` subcommand records the captain's exact words and closes the call in the same act. +It requires a non-empty captain decision file of at most 8192 bytes, writes a resolution block carrying the decision digest and a `Resolution mode:` at the top of the task body (the previous body is preserved below the block and archived through tasks-axi `--archive-body`), then runs `tasks-axi done` - or `tasks-axi unhold` under `--release`, so a captain-gated work item resumes instead of closing. +An exact retry is idempotent only when the requested close mode matches the newest record; a drifted answer or mode mismatch is rejected, while a re-held task accepts a new answer as a new record on top. +On a task closed outside the script, `answer` records the missing block only when the captain-hold annotations tasks-axi preserves through a close prove the captain owned it, and it verifies the task stays closed. +A hold whose `--until` date has passed keeps those annotations while tasks-axi reports it no longer held, so an expired deferral remains answerable. + +The `complete` subcommand unions the reviewed captain-held task ids into `decision_keys=` and appends `decisions_reviewed=1` while originating task metadata is live. +A post-teardown visual review can complete against the surviving report and durable tasks without recreating volatile task metadata. +It accepts `--none` as an explicit semantic inventory result, refused while the origin still has a lifecycle-open keyed status decision, and verifies every listed task against tasks-axi before recording completion. +With a non-empty inventory it appends a `captain-held [key=<key>]: tracked by <inventory>` transfer event for every still-open keyed status decision, which `bin/fm-classify-lib.sh` recognizes as closing the live status copy without claiming that the captain has answered it. + +Scout teardown calls the read-only `verify` subcommand after checking for the report and before removing any source state. +`verify` requires the recorded attestation, requires every recorded inventory entry to still be durable (actively captain-held, or carrying a recorded answer), and fails on any keyed status decision that opened after the last `complete`, which makes re-running `complete` the repair. +The `--force` path remains the explicit captain-approved discard escape hatch. + +## Answer-time closure + +"A keyed answer closes its matching captain-held task" is one capability with one owner. +`answers` is its channel-agnostic entry point: it reads `<task-id>\t<answer>\t<label>[\t<mode>]` lines and closes each named task through the same `answer` path, so every guard applies identically no matter which channel the answer arrived on. +The optional mode column carries a card-declared close: `done` (default) completes the task and `release` lifts the hold so held work resumes; any other value is skipped. +A key that names no task, names a task that is not captain-held, or names a task already closed is reported as `skipped:` and feeds nothing; a replay whose answer and requested close mode match the newest record is an idempotent `closed:`, while a mode mismatch is skipped; and the command exits nonzero when any key was skipped. +`--source` is provenance text recorded in the durable decision, never a behavior switch, and the command carries no per-channel branch. + +`bind`, `unbind`, and `binding` record that a captured-answer source feeds this intake, as a private record under `state/decision-bindings/`; an unbound source feeds nothing, so the path is opt-in per source, and `bind` deliberately does not require the source to exist yet. + +Two channels feed that one intake today, and both are ordinary callers rather than special cases. +`bin/fm-send.sh --resolve-key` is the chat channel: its status-log close is unchanged for a key the status log still owns, and a key the status log no longer owns is resolved to a still-open captain-held task - the key as a task id, then the legacy derived identity - and fed as one keyed line. +`bin/fm-procevent.sh` is the captured-result channel: after capture, a bound source has its result passed to `bin/fm-procevent-<adapter>.sh answers <result-file>` and whatever that prints is piped into the intake, so any adapter with an `answers` command works and the runner names no adapter, parses no result, and carries no decision rule. +`bin/fm-procevent-lavish.sh answers` is one such adapter command; it reads only rows tagged `choice`, relays a card's declared close mode, and can never let freeform captain prose forge a task id or a mode. + +## Structured read surfaces + +`bin/fm-fleet-snapshot.sh` parses canonical tasks-axi `(hold: ...)`, `(hold-kind: ...)`, and `(hold-until: ...)` metadata alongside existing backlog fields. +It resolves every repeated `blocked-by:` edge against structured Done records, keeps missing blockers unresolved, and classifies a captain hold as `captain_actionable` - waiting on the captain now - only when it is queued, unblocked, and due, whatever kind its row carries. +It also emits a presentation-only `deferred_marker` when a hold's reason or body carries an explicit SUPERSEDED / NOT REQUIRED / DEFERRED marker. +Its secondmate-home summary classifies an actionable captain hold as `captain_decision` and preserves blocked or deferred captain holds as queued work in the owning home. + +`bin/fm-bearings-snapshot.sh` projects actionable captain holds into `decisions_open` and leaves blocked captain holds in ordinary queued gates. +A date-deferred captain hold renders as a gate with its `until <date>:` reason; a prose-deferred one leaves the default views with an `omitted[]` disclosure, revealed by `--all-decisions` / `--all-queued`. +Recently Landed excludes a record that closed while still held for the captain (surviving `hold-kind: captain` on a Done row), so answered questions do not masquerade as shipped work; a work item released before completion keeps no hold annotations and lands normally. +The projection remains read-only and does not inspect historical prose beyond the canonical snapshot's marker. + +## Record divergence + +A captain call can have two records, and closing one does not close the other. +A `resolved [key=...]` line closes the status-log fold; the structured captain-held task closes only through `answer`. +Until this guard existed, closing on the status side alone left no trace of the disagreement: the fold went quiet, the durable record kept saying the captain owed an answer, and nothing warned. + +`bin/fm-captain-hold.sh diverged` is the read-only report of that state, and `bin/fm-wake-drain.sh` prints it as a bounded `RECORD DIVERGENCE` section beside OPEN DECISIONS on every drain. +It flags exactly one condition: a task still open and still carrying the captain-hold annotations, whose key was closed on the status side by the resolve verb, resolved through the collapsed identity (the key is the task id) or the legacy derived one. +It closes nothing, ever - a captain call closed wrongly leaves review entirely, so both reconciliation directions stay human-owned and the printed hint names both. + +Three states are deliberately not divergence. +A `captain-held [key=...]` close is the verified transfer `complete` writes, so the structured row staying open behind it is correct; `bin/fm-classify-lib.sh`'s `status_key_closing_verb` is what keeps the two closing verbs distinguishable. +A still-open keyed status decision belongs to the OPEN DECISIONS fold. +And the absence of a routed work item is legitimate rather than incomplete - when the decision is the deliverable there is nothing to route - so routed work is no part of the test. + +Cost stays flat: one `tasks-axi list`, one key scan per status log, and the precise per-key fold only for a key that already names a still-open task. +The comparison is refused unless the status directory is the active home's own, since tasks-axi reads that home's backlog and a mismatch would report one home's logs against another's tasks. +If tasks-axi is unavailable or its listing cannot be parsed, the guard cannot read the structured record and prints nothing. + +## Compatibility with pre-collapse installs + +Older installs created derived `<origin>-decision-<key>` identities through the retired `bin/fm-decision-hold.sh`. +Those rows are already plain task ids, so they render, answer, verify, and close through the collapsed surfaces with no data migration. +Three legacy inputs are resolved in place: a `decision_keys=` metadata entry that names no task resolves through `<origin>-decision-<entry>`; a channel key that names no task resolves the same way when the source's binding carries a concrete legacy origin; and resolution records written by the old script are recognized wherever a record is read. +The shim recognizes an exact replay of a pre-collapse routed resolution by its historical answer digest and routed ids, then finishes any still-recorded dependency-edge cleanup without rewriting the old decision text. +`bin/fm-decision-hold.sh` itself remains for one release as a thin command-mapping shim over `bin/fm-captain-hold.sh`, so in-flight work briefed before the collapse keeps working; its header owns the exact mapping. + +## Verification record + +Verification date: 2026-08-21. + +The focused end-to-end regression suite is `tests/fm-captain-hold-lifecycle.test.sh`, using only synthetic `sample` identities and decision text. +It proves: the reconstructed silent-divergence case is signalled - a status resolution over a still-open captain-held task reaches both `diverged` and the drain's `RECORD DIVERGENCE` section, under the collapsed and the legacy identity alike, while the backlog task, its hold, and the status log all survive the report unchanged and the printed hint names both reconciliation directions; the false-signal boundary holds - a captain call with no routed work item, a verified `captain-held` transfer, a still-open status decision, an already answered call, and an ordinary task whose keyed question was answered all stay silent; a report-only unresolved captain call refuses `--none` completion before teardown can erase the source; non-forced scout teardown always requires the durable inventory verification; the recorded-answer guard (a bare `tasks-axi done` close fails `verify` until `answer` records the captain's word, and an ordinary finished task cannot be dressed up as an answered call); answer-time closure through a bound channel with task-id keys, including the `release` close mode, mode-matched replay idempotence, and the refusal of drifted, mode-mismatched, absent, unheld, and already-closed keys; the chat channel reaching the same intake; deferral through `--until` leaving `captain_actionable` false until due; and every legacy path (composed identities through the shim, pre-collapse `decision_keys=` metadata, routed-resolution replay, and a concrete-origin binding). + +`tests/fm-classify-decision-key.test.sh` pins `status_key_closing_verb` itself: it separates a resolution from the durable-transfer close and from a still-open key, reports the last real transition across re-openings and both key positions, and treats a prose mention as no transition. + +Projection regressions live in `tests/fm-fleet-snapshot-view.test.sh` (hold-until parsing, the due gate, kind-independent captain actionability, deferred_marker, title stripping) and `tests/fm-bearings-snapshot.test.sh` (Captain's Call membership, the dated-gate rendering, prose-deferral suppression with disclosure, and the landed exclusion by surviving captain-hold annotations). +The exact commands and their summarized outputs are recorded in the shipping PR's evidence; run the four suites above plus `tests/fm-send-resolve-key.test.sh`, `tests/fm-bearings-board.test.sh`, and `bin/fm-lint.sh` to refresh this record. diff --git a/docs/configuration.md b/docs/configuration.md index e315147d86b..db41a1d1440 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -170,6 +170,16 @@ An inherited `data/captain-shared.md` counts in a secondmate's total but remains The internal [`/stow` skill](../.agents/skills/stow/SKILL.md) owns curation and its automatic secondmate cascade, which accounts every home against this same per-home allowance separately rather than against a fleet total. The helper's header owns exact parsing, publication, and report output mechanics. +## Stow pass horizon (config/stow-pass-horizon) + +`config/stow-pass-horizon` is an optional local, gitignored presence flag that opts this home in to the pass-count decay horizon in the internal [`/stow` skill](../.agents/skills/stow/SKILL.md). +Without it a `/stow` pass decays memory entries on their wall-clock horizons alone - 30 days for `aging`, 7 days for `perishable` - which is the default and unchanged behavior. +With it, an entry is also stale after 10 passes (`aging`) or 3 passes (`perishable`) that evaluated it without reinforcing it, whichever horizon it reaches first. +Opt in for a home that stows often enough that entries never sit unreinforced for a wall-clock horizon, so memory only grows against the startup-memory budget above; a home that stows rarely already exceeds its date horizon on a single pass and gains nothing. +The flag is per home and is not inherited by secondmate homes, because stow cadence is a property of the home doing the stowing. +Only the file's presence is read, so its contents are ignored; remove it to return to the default contract on the next pass. +The skill text owns the marker spelling, the tick order, and the reinforcement rule. + ## Secondmate routes (data/secondmates.md) Persistent secondmate routes live locally in `data/secondmates.md`. @@ -341,6 +351,66 @@ The locked bootstrap inheritance pass uses the same placement-specific behavior; That live discovery starts from `state/*.meta` records with `kind=secondmate`; `data/secondmates.md` only backfills `home=` for older or incomplete meta records. Skipped items, such as a destination checkout that does not yet gitignore the item, are visible warnings but not hard failures. +## Watched tool updates (config/watched-tools.json) + +`config/watched-tools.json` is an optional local, gitignored list of the tools this home depends on. +When it is present and the check is armed, [`bin/fm-tool-update-check.sh`](../bin/fm-tool-update-check.sh) reports two conditions, and keeps them deliberately distinct: + +- `<tool> update available` means a newer version exists at the tool's update source. +- `<tool> update not in effect` means a newer copy is already installed on this host, but `PATH` still resolves an older one. + +The second condition is the reason the check exists. +An update can install correctly and stay inert because an earlier `PATH` entry still holds an older copy, and a check that only asks whether a newer version is published reports that host as up to date. +The script therefore runs every copy of a watched command found on `PATH` and asks it for its own version, rather than trusting one lookup or reading a version out of a directory name. +It only reports; it never installs, updates, fetches, or changes `PATH`, a version manager, or any installed tool. + +This section is the single owner of the canonical schema. +`bin/fm-tool-update-check.sh` owns probe mechanics, cadence, and the report record. + +```json +{ + "tools": [ + { + "name": "<label used in the report>", + "command": "<optional bare executable name to find on PATH>", + "version_args": ["<optional args that make it print its version, default --version>"], + "announce_pattern": "<optional extended regex matching the tool's own update announcement>", + "announce_args": ["<optional args for the command that carries that announcement, default version_args>"], + "git": { + "repo": "<optional absolute path to a local clone>", + "remote": "<optional remote name, default origin>", + "branch": "<optional branch, default the remote's own default branch>" + } + } + ] +} +``` + +Each entry needs a `name` and at least one of `command` or `git`; an entry may carry both. +A `command` entry gives the `PATH` comparison above, and adding `announce_pattern` also reports the tool's own update announcement, which is how a tool that already reports its own updates is read rather than reimplemented. +A tool does not always announce a new release on the command that prints its version: `no-mistakes --version` prints only the version, while its other commands carry the announcement. +`announce_args` names the command to search for the announcement in that case, and it is asked only of the copy `PATH` resolves; without it the version probe's own output is searched. +An `announce_pattern` that is not a usable extended regular expression stops `arm`, and during a sweep it is reported as that one tool's own check failure so one broken pattern never stops the other watched tools from being checked. +A `git` entry reports how many commits the local clone is behind its remote branch, and stays silent when the clone is current or ahead. +An omitted `branch` uses the remote's default branch, taken from the clone's own record of it and otherwise asked of the remote directly, so a `--single-branch` clone still resolves. +Both probe kinds are read-only and bounded, and a probe that cannot answer is reported as a check failure rather than assumed current. +See [`docs/examples/watched-tools.json`](examples/watched-tools.json) for a starting point to copy into local `config/watched-tools.json`. + +Arm the check once per home with `bin/fm-tool-update-check.sh arm`. +That writes `state/tool-updates.check.sh` and binds its bytes with `bin/fm-check-register.sh`, so the existing watcher polls it on its normal cadence and turns its one line into a `check:` wake; no separate schedule is involved. +The armed check runs whenever that home has a watcher running, and arming alone does not make watcher supervision required, so a home with no in-flight work and no other reason to watch does not start a watcher just for this check. +`bin/fm-tool-update-check.sh disarm` removes the shim, its trust binding, and the report record. +The check prints nothing when everything is current, and `state/.tool-updates` records the findings the last report was made from so the same pending update is reported once instead of on every poll. +A changed or returning condition is reported again. +Adding, removing, or changing a watched tool is an edit to this file and needs no code change or re-arming. +This file is not inherited by secondmate homes, so each home watches the tools it actually depends on. + +`FM_TOOL_UPDATE_INTERVAL` (default 900 seconds, `0` to probe on every run) sets how often probes actually run, `FM_TOOL_UPDATE_PROBE_SECS` (default 5) bounds one probe, and `FM_TOOL_UPDATE_BUDGET_SECS` (default 20) bounds a whole sweep. +A sweep that runs out of budget says which tool it did not reach rather than reporting the rest as current. +The sweep must finish inside `FM_CHECK_TIMEOUT` (default 30), because a run the watcher kills prints nothing and records nothing and would then repeat that silence on every poll. +So a budget larger than that timeout allows is cut down to what fits instead of being refused, and the cut is reported in the report line. +A budget that is not a whole number from 1 to 120 is still refused outright. + ## Relay (.env) Relay lets a firstmate instance answer public mentions and act on normal reversible mention requests through firstmate's normal lifecycle. @@ -432,11 +502,12 @@ These paths need `jq` to build the JSON payload, but they run before token and n ### Promised public replies (state/public-followup) A relay request that spawns real work can leave firstmate owing a specific public reply in a specific thread. -That promise is a typed `kind=public-followup` obligation owned entirely by `tasks-axi public-followup`, with the full private request context staying in `state/x-context/`; firstmate keeps no parallel copy of either. -`bin/fm-public-followup.sh` is firstmate's side: it registers a commitment, reconciles typed terminal work results into it, and posts the final reply through `bin/fm-x-reply.sh --followup`. +That promise is a typed `kind=public-followup` obligation whose state machine is owned entirely by `tasks-axi public-followup`, while the full private conversation context stays only in `state/x-context/`. +Firstmate's bounded registration retains the obligation's public-safe request binding so a delivered loop can be rechained without the original inbox. +`bin/fm-public-followup.sh` is firstmate's side: it registers a commitment, reconciles typed terminal work results into it, posts the final reply through `bin/fm-x-reply.sh --followup`, and explicitly rechains or retires the retained loop. Run `bin/fm-public-followup.sh --help` for the exact subcommands and flags. -Registration is what creates this home's private transport under `state/public-followup/` (mode 0700): `registry/` for the bounded public-safe binding of each live commitment, `events/` for typed terminal results awaiting reconciliation, `consumed/` for the accepted-event ledger, `rejected/` for refusals kept with a one-line reason, and `surfaced` for the poll's last-surfaced signature. +Registration is what creates this home's private transport under `state/public-followup/` (mode 0700): `registry/` for the bounded private binding of each open public loop (the record survives delivery, stamped `state=delivered`, and is removed only by `retire`), `events/` for typed terminal results awaiting reconciliation, `consumed/` for the accepted-event ledger, `rejected/` for refusals kept with a one-line reason, `retired/` for the mode-0600 reason-and-time receipt written before removal, and `surfaced` for the poll's last-surfaced signature. The home that owns the commitment also owns the outward post, because only it holds the relay consent, the request context, and the opaque thread binding. Work routed elsewhere reports a typed terminal result with `bin/fm-public-followup-emit.sh` and never looks for the thread; that emitter refuses to write into a home with no registration for the named obligation. A terminal event's id is derived from its identity tuple, so a duplicate report, a retry, or a replay after restart resolves to the same event and changes nothing. @@ -446,15 +517,18 @@ A home without that token runs one file test and stops: no `tasks-axi` call, no Ordinary startup, polling, cleanup, and silent read-side subcommands also produce no output; commands that require an active relay report that configuration error after the same gate. A relay-enabled home with no registered commitment stops at an O(1) directory presence check, so the empty state costs no CLI call and adds no periodic scan. Unreconciled terminal results ride the existing 30-second relay poll rather than a new process or timer: `bin/fm-x-poll.sh` compares the pending-event signature against `surfaced` and wakes firstmate once per new result set. -The session-start digest separately prints an "Public commitments awaiting delivery" subsection from disk when, and only when, this home is relay-active and still owes a reply, so compaction and restart are non-events. +The session-start digest separately prints a "Public commitments" subsection from disk when, and only when, this home is relay-active and still holds an open public loop (a reply still owed, or a delivered loop with nothing owed), so compaction and restart are non-events. `bin/fm-teardown.sh` refuses to clean up a task while this home still owes a public reply for exactly that work, unless `--force` carries explicit discard approval. `FM_PF_RETRY_BACKOFF_SECS` (default 900) sets the next-attempt time recorded with a retryable delivery error. -See [verification/public-followup.md](verification/public-followup.md) for the current maintainer evidence behind the restart end-to-end and the relay-disabled zero-overhead guarantee. +See [verification/public-followup.md](verification/public-followup.md) for the current maintainer evidence behind restart recovery, retained-loop disposition, and the relay-disabled zero-overhead guarantee. ## Process-to-event sources (state/procevent) A long-polling external process is registered as a *source* through its adapter, whose header and `--help` own the commands and flags. `bin/fm-procevent.sh` owns the generic contract; `bin/fm-procevent-lavish.sh` is the first adapter and wraps only the currently published `lavish-axi poll` interface. +That adapter, and only that adapter, retries the one exact transient response a cut-short listener returns while its marks remain available (`error: Lavish Editor poll response was interrupted` with `code: SERVER_ERROR`), up to 12 times at 5 second intervals, so an internal retry never reaches the runner as a captured result. +Real feedback, ended and missing sessions, any other `SERVER_ERROR`, and that same interruption still standing once the bound is spent are all captured and announced normally; `FM_LAVISH_POLL_RETRY_DELAY` is a bounded 0 to 60 second test override for the interval only, and the runner itself stays adapter-agnostic. +An already-armed Lavish source keeps its registered listener command until it is retired and armed again, so re-arm a live board once to adopt this retry policy. The `when` adapter (`bin/fm-procevent-when.sh`) turns this channel into a condition->action primitive: it registers a deterministic condition and a deterministic action once, its blocking child polls the condition without waking firstmate, and a stable true fires the action at most once before one terminal outcome is durably captured and published as a wake that remains eligible for re-announcement until handled. The (condition, action) spec is stored privately under `state/when/` and hash-bound by a trust record the same way `bin/fm-check-register.sh` binds a custom check, while the spec separately binds the resolved action executable's bytes; a mutated or unregistered spec or a changed action executable is refused before the action runs. @@ -487,8 +561,8 @@ Announcement ordering is adapter-declared through `bin/fm-procevent-<adapter>.sh The remote-secondmate reply adapter declares itself self-announcing: a captured reply reaches its local status mirror and settles its correlated pending-reply expectation without any handler step, the mirrored status bytes are the single wake for one remote note through the same signal classification a local secondmate's append gets, a byte-identical replayed capture adds no bytes and stays quiet, and only a capture the adapter could not fully apply is published as a `check` wake, whose adapter handling remains idempotent. Keyed captain answers use one more seam of the same kind, and the runner still decides nothing about them. -Some sources carry the captain's answer to a durable decision, and what such an answer means is owned once by `bin/fm-decision-hold.sh`'s keyed-answer intake rather than by any channel. -A source bound with `bin/fm-decision-hold.sh bind` therefore has each captured result passed to `bin/fm-procevent-<adapter>.sh answers <result-file>`, and whatever that prints is piped straight into that intake. +Some sources carry the captain's answer to a captain-held task, and what such an answer means is owned once by `bin/fm-captain-hold.sh`'s keyed-answer intake rather than by any channel. +A source bound with `bin/fm-captain-hold.sh bind` therefore has each captured result passed to `bin/fm-procevent-<adapter>.sh answers <result-file>`, and whatever that prints is piped straight into that intake. A binding can select one decision origin or the script's cross-origin mode; the command header owns the exact forms and key interpretation. The adapter reports only what the captain chose; the intake owns every rule about what happens next, so the runner names no adapter, parses no result, and carries no decision rule, and a future source needs nothing here beyond an `answers` command and a binding. Feeding is independent of handling: it never acknowledges a result and never suppresses a wake, because recording the answer is transcription while acting on it is firstmate's judgement. @@ -522,6 +596,30 @@ The published `lavish-axi poll` clears feedback destructively before returning i Never describe this path as at-least-once, no-loss, or lossless. `docs/verification/process-event-sources.md` holds the measurements and `.agents/skills/process-event-sources/SKILL.md` owns the handling procedure. +## Spoken interface and captain inbox (config/voice-*, config/inbox-*) + +The spoken interface in [`docs/voice-relay.md`](voice-relay.md) and the model-backed subcommands of `bin/fm-inbox.sh` reach a paid API in a named account, so no region, model id or AWS profile is shipped as a tracked default. +Each is one line in a local, gitignored `config/` file, with an environment variable that overrides it for a single run, and a missing required value refuses with the path to write rather than falling back to a value that belongs to another home. +That configuration is the whole opt-in: an unconfigured home cannot start the relay and cannot run `fm-inbox.sh say` or `ask`, while `note`, `status`, `list` and `drain` need no configuration at all because they make no model call. +The voice handover depends on `note`, so it keeps working in a home that has configured nothing. + +| File | Environment | Holds | +| --- | --- | --- | +| `config/voice-region` | `FM_VOICE_REGION` | Bedrock region for the relay's bidirectional session, required by `bin/fm-voice-relay.py`. | +| `config/voice-model` | `FM_VOICE_MODEL` | Speech-to-speech model id, required by `bin/fm-voice-relay.py`. | +| `config/voice-profile` | `FM_VOICE_PROFILE` | AWS profile the relay exports credentials from; absent, or an explicitly empty variable, means it uses only credentials already in its environment. | +| `config/voice-id` | `FM_VOICE_ID` | Output voice id, optional, `matthew` when unset. | +| `config/voice-read-scope` | none | `counts` (the default, and what an absent file means) or `full`; see [`docs/voice-relay.md`](voice-relay.md) for what each scope may say. | +| `config/voice-read-deny` | none | One plain case-insensitive substring per line; a matching open item is withheld from every list and reduced to a count. | +| `config/inbox-region` | `FM_INBOX_REGION` | AWS region for `fm-inbox.sh say` and `ask`. | +| `config/inbox-stt-model` | `FM_INBOX_STT_MODEL` | Speech-to-text model id, required by `fm-inbox.sh say`. | +| `config/inbox-ask-model` | `FM_INBOX_ASK_MODEL` | Side-question model id, required by `fm-inbox.sh ask`. | +| `config/inbox-profile` | `FM_INBOX_PROFILE` | AWS profile for those two calls; absent, or an explicitly empty variable, means whatever credentials are already in the environment. | + +Each account, model and voice file above is read as its first line that is not blank and not a `#` comment, so a comment above the value is fine. +The two read files are parsed differently: `config/voice-read-scope` must hold the bare word and nothing but blank space around it, so a comment header there refuses instead of being skipped, while every line of `config/voice-read-deny` that is not blank and not a `#` comment is one more substring. +`FM_VOICE_RELAY` and `FM_VOICE_PYTHON` belong to the laptop rather than to a home, so they have no config file: `bin/fm-voice-client.py` requires the relay path as a flag or that variable and carries no default path. + ## Environment variables Runtime tuning via environment variables (defaults shown): @@ -556,6 +654,10 @@ FM_INACTIVE_RECONCILE_SECS=900 # 60..1800-second watcher cadence and inactivity FM_INACTIVE_RECONCILE_BUDGET_SECS=10 # 1..30-second scan deadline; wedged-scan kill backstop follows one second later FM_CHECK_INTERVAL=300 # seconds between slow checks (authenticated merge polls, custom checks, or Relay dispatch) FM_CHECK_TIMEOUT=30 # seconds allowed per slow check script +FM_TOOL_UPDATE_INTERVAL=900 # seconds between watched-tool probe sweeps; 0 probes on every run, other values must be 60..86400 +FM_TOOL_UPDATE_PROBE_SECS=5 # 1..30 seconds allowed for one version or git probe +FM_TOOL_UPDATE_BUDGET_SECS=20 # 1..120 seconds allowed for a whole watched-tool sweep; cut to fit FM_CHECK_TIMEOUT, and the cut is reported +FM_TOOL_UPDATE_NOW= # test override for the watched-tool sweep clock; the sweep budget still uses real time FM_PROCEVENT_MAX_OUTPUT_BYTES=1048576 # bound on one captured process-to-event result FM_PROCEVENT_CLAIM_ROOT= # machine-wide source claim root; default $XDG_STATE_HOME/firstmate/procevent-claims FM_WHEN_OUTPUT_TAIL_BYTES=8192 # bound on the command-output tail inside one condition->action outcome document @@ -596,8 +698,11 @@ FM_CAPTAIN_RE='done:|needs-decision:|blocked:|failed:|PR ready|checks green|read FM_CLASSIFY_PAUSED_VERB=paused # leading status verb for a declared external wait; excluded from FM_CAPTAIN_RE and distinct from blocked FM_STALE_ESCALATE_SECS=240 # idle seconds before a provably-working stale pane escalates; stale panes whose crew is not provably working surface immediately unless they declare the pause verb FM_BUSY_TURN_MAX_SECS=3600 # maximum age of a busy pane's latest state/<id>.turn-ended marker, or its state/<id>.meta spawn record before any turn completes, before the same wedge escalation used for a provably-working non-busy stale takes over; inspection-only, never an automatic interrupt or restart; a declared external wait or verified captain-held transfer takes the FM_PAUSE_RESURFACE_SECS recheck below instead -FM_PAUSE_RESURFACE_SECS=3600 # seconds before the watcher re-surfaces a declared external wait or verified captain-held transfer for a recheck, including a live busy pane past FM_BUSY_TURN_MAX_SECS; the away-mode daemon uses the same setting for declared external waits +FM_PAUSE_RESURFACE_SECS=3600 # seconds before the watcher re-surfaces a declared external wait or verified captain-held transfer for a recheck, including a live busy pane past FM_BUSY_TURN_MAX_SECS; the away-mode daemon uses the same setting for a declared external wait or verified captain-held transfer FM_WEDGE_DEMAND_INSPECT_COUNT=3 # consecutive provably-working stale escalations on the same unchanged pane before demand-deep-inspection is added +FM_WORKTREE_WRITE_PRUNE='.git node_modules .venv venv __pycache__ .mypy_cache .pytest_cache .ruff_cache .tox target dist build .next .cache vendor' # directory names the wedge detector's task-worktree write probe skips; the default keeps .git out so a supervisor's own read-only git command can never look like crew progress; set it to the empty string to prune nothing, which widens the probe to the whole depth-bounded tree rather than disabling it +FM_WORKTREE_WRITE_MAXDEPTH=6 # depth that same probe walks below the recorded worktree; it runs only at the moment a wedge escalation would otherwise fire, never on every poll; no probe knob applies to a secondmate, whose recorded worktree is a provisioned home the probe skips entirely +FM_WORKTREE_WRITE_TIMEOUT=10 # wall-clock seconds that one walk may take, so a worktree on a hung mount cannot stall the watcher poll that started it; hitting the bound reads as no write evidence, which leaves the escalation schedule exactly as it was; a value that is not a positive integer falls back to the default FM_WATCH_TRIAGE_LOG_MAX_BYTES=262144 # size cap for the watcher's absorbed-wake debug log FM_FLEET_SYNC_BOOTSTRAP_TIMEOUT= # optional seconds allowed for bootstrap's best-effort clone refresh; unset/blank defaults to max(20, 5 + 3 * origin-backed-project-count) FM_FLEET_PRUNE=1 # set to 0 to skip pruning local branches whose upstream is gone @@ -638,6 +743,17 @@ FM_CRASH_BACKOFF=60 # seconds to wait after crossing the crash th FM_CRASH_NORMAL_SLEEP=5 # seconds to wait after an isolated watcher crash FM_LOG_MAX_BYTES=1048576 # daemon log size that triggers trimming FM_LOG_KEEP_LINES=2000 # daemon log lines kept when trimming +# spoken interface and captain inbox; see "Spoken interface and captain inbox" above +FM_VOICE_REGION= # overrides config/voice-region for one relay run +FM_VOICE_MODEL= # overrides config/voice-model for one relay run +FM_VOICE_PROFILE= # overrides config/voice-profile; explicitly empty forces ambient credentials +FM_VOICE_ID= # overrides config/voice-id; matthew when neither is set +FM_VOICE_RELAY= # laptop-side path to bin/fm-voice-relay.py on the desktop; required by fm-voice-client.py unless --relay is passed +FM_VOICE_PYTHON=python3 # laptop-side interpreter used to start the relay over ssh +FM_INBOX_REGION= # overrides config/inbox-region for fm-inbox.sh say and ask +FM_INBOX_STT_MODEL= # overrides config/inbox-stt-model for fm-inbox.sh say +FM_INBOX_ASK_MODEL= # overrides config/inbox-ask-model for fm-inbox.sh ask +FM_INBOX_PROFILE= # overrides config/inbox-profile; explicitly empty forces ambient credentials ``` `fm-teardown.sh` retries only Git's `Unable to create '...index.lock': File exists` return failure up to `FM_TREEHOUSE_RETURN_LOCK_RETRIES` times. diff --git a/docs/decision-hold-lifecycle.md b/docs/decision-hold-lifecycle.md deleted file mode 100644 index 1bcb63dcee2..00000000000 --- a/docs/decision-hold-lifecycle.md +++ /dev/null @@ -1,164 +0,0 @@ -# Decision hold lifecycle mechanism - -The normative policy is owned by `.agents/skills/decision-hold-lifecycle/SKILL.md` and is not restated here. -This document records the deterministic mechanism, structured surfaces, and privacy-safe regression evidence. - -## Mechanism - -`bin/fm-decision-hold.sh` is the only lifecycle command for an investigation or visual review's unresolved captain decisions. -The command runs tasks-axi in the active `FM_HOME`, so the existing backlog remains the only durable work database and a secondmate-owned decision stays in the secondmate home. -It never reads report bodies, review artifacts, terminal output, or chat. - -The `hold` subcommand maps an originating work id and stable decision key to `<origin-id>-decision-<decision-key>`. -It creates a kind `captain` backlog item when absent and invokes `tasks-axi hold <id> --reason <reason> --kind captain` on every retry. -It rejects an identity collision, a changed title, and attempts to reopen an already resolved identity. - -The `complete` subcommand unions the reviewed keys into `decision_keys=` and appends `decisions_reviewed=1` while originating task metadata is live. -A post-teardown visual review can complete against the surviving report and durable holds without recreating volatile task metadata. -It accepts `--none` as an explicit semantic inventory result, not as inferred absence. -It verifies every listed identity against tasks-axi before recording completion. -For an open keyed status decision, it appends a `captain-held [key=<key>]: ...` transfer event only after the matching backlog hold is durable. -`bin/fm-classify-lib.sh` recognizes that transfer as closing the live status copy without claiming that the captain has answered it. - -Scout teardown calls the script's read-only `verify` subcommand after checking for the report and before removing any source state. -The `--force` path remains the explicit captain-approved discard escape hatch. - -The `resolve`, `answer`, and `decline` subcommands close active holds, while `repair` attests a hold already closed outside the script. -All four require a non-empty captain decision file and record the same resolution block in the hold body with the decision digest, routed identities, and a `Resolution mode:` naming the path. -An exact retry is idempotent, while a changed decision or, for `resolve`, a changed routed-task set is rejected. - -The `resolve` subcommand is the routed path and additionally requires at least one existing dependent task whose structured `blocked-by` edge points to the hold. -It clears each dependency edge through tasks-axi and marks the hold Done only after those writes succeed. -An exact retry can finish a partial routing operation, and a failed intermediate step leaves the hold open. - -The `answer` and `decline` subcommands share one unrouted close implementation and differ only in the `Resolution mode:` they record and the outcome word they print, so neither can drift into a weaker close than the other. -Both record `(none)` as the routed identities and refuse while any task in the same backlog is still blocked by the hold, because releasing routed work without recording it is `resolve`'s job. -Every candidate found in the listing prefilter is confirmed against its own structured record before the refusal is reported. -`answer` exists so the act carrying a captain answer can also be the act that closes its hold; `decline` continues to mean the stronger claim that the answer routes no follow-up work at all. - -The `repair` subcommand records the resolution block on a hold that was already closed outside the script, such as by a direct `tasks-axi done`, so an origin whose decision was genuinely answered stops failing `verify`. -It refuses a hold that is still actively held, never reopens a closed hold, and never clears a dependency edge, so an unanswered decision keeps blocking teardown until the captain's word closes it. -It also requires the identity to carry the captain-hold provenance that tasks-axi preserves through a close, so an ordinary captain-kind task that was never held cannot be repaired into a resolved decision. - -## Answer-time closure - -The live status-log decision ledger has always had answer-time closure through `bin/fm-send.sh --resolve-key`: answering a keyed decision closes it in the same act. -The durable hold ledger did not, so an answer could be captured, believed, and even implemented while its hold stayed open, and the captain could then be asked to re-answer a decision already on disk. - -"A keyed answer closes its matching hold" is now one capability with one owner. -`answers` is its channel-agnostic entry point: it reads a key, answer, and label on each input line and closes the matching hold through the same `answer` path, so every guard applies identically no matter which channel the answer arrived on. -For a single-origin intake the key is the decision key mapped under that bound origin; for the cross-origin intake it is the full hold identity, while keys that do not name a full decision hold feed nothing. -`--source` is provenance text recorded in the durable decision, never a behavior switch, and the command carries no per-channel branch and no knowledge of chat, review decks, or any transport. -A channel's only job is to turn whatever it received into those keyed lines and pipe them in; it never maps keys to holds, builds decision records, chooses between the close paths, or closes a hold itself. -The decision text is a pure function of source, key, answer, and label, which is what makes a replayed delivery an idempotent no-op rather than a rejected different decision. -A key whose hold is absent, already closed, or still blocking routed work is reported as skipped and left for `resolve`, and the command exits nonzero when any key was skipped. - -`bind`, `unbind`, and `binding` record whether a captured-answer source belongs to one origin or uses the cross-origin intake, for a channel whose answers arrive detached from the origin. -The binding is a private record under `state/decision-bindings/`, and a source with no binding feeds nothing, so the path is opt-in per source. -`bind` deliberately does not require the source to exist yet, so a channel can be bound before it is armed and never produce an answer that has nowhere to go. -The script header and `--help` own the exact cross-origin marker, identity split, limits, and refusal behavior. - -Two channels feed that one intake today, and both are ordinary callers rather than special cases. - -`bin/fm-send.sh --resolve-key` is the chat channel. -Its existing status-log close is unchanged for a key the status log still owns. -For a key the status log no longer owns it checks whether that key names an active captain hold on the target task, and feeds the answer as one keyed line if so, which is what lets chat answer a decision already transferred to its hold. -A key open in neither ledger is still refused before anything is sent. -Because `complete` closes the live status copy at the moment it transfers a decision to its hold, the two ledgers are the two sides of one transfer and never both own a key at once, so the common path still performs no backlog read. - -`bin/fm-procevent.sh` is the captured-result channel, and its wiring is generic. -After capture, a bound source has its result passed to `bin/fm-procevent-<adapter>.sh answers <result-file>` and whatever that prints is piped into the intake, so any adapter with an `answers` command works and the runner names no adapter, parses no result, and carries no decision rule. -Feeding is independent of handling: it never acknowledges a result and never suppresses a wake, so recording the captain's answer cannot retire the notification firstmate needs in order to act on it. -`bin/fm-procevent-lavish.sh answers` is one such adapter command; it reports the structured choices a review captured and stops there, reading only rows tagged `choice` so freeform captain prose can never forge a decision key. - -## Structured read surfaces - -`bin/fm-fleet-snapshot.sh` parses canonical tasks-axi `(hold: ...)` and `(hold-kind: captain)` metadata alongside existing backlog fields. -It resolves every repeated `blocked-by:` edge against structured Done records, keeps missing blockers unresolved, and classifies only an unblocked captain hold as actionable. -Its secondmate-home summary classifies an actionable captain hold as `captain_decision` and preserves blocked captain holds as queued work in the owning home. - -`bin/fm-bearings-snapshot.sh` projects actionable captain holds into `decisions_open` and leaves blocked captain holds in ordinary queued gates. -It excludes completed kind `captain` records from Recently Landed. -The projection remains read-only and does not inspect historical prose. - -## Verification record - -Verification date: 2026-07-14. -Additional quoted `blocked_by` regression verification date: 2026-07-17. -Plural blocker-readiness and mixed-home projection verification date: 2026-07-22. -Unrouted close-path verification date: 2026-08-13. -Answer-time closure verification date: 2026-08-16. -Cross-origin answer-time closure verification date: 2026-08-19. - -The focused end-to-end regression uses only synthetic `sample` identities and decision text. -It begins with a completed investigation and visual review whose genuine unresolved choice exists only in the report. -The initial Bearings snapshot correctly has no open decision, and the new teardown gate refuses to erase the source. -A later regression covers tasks-axi's quoted multi-entry `blocked_by` output so `resolve` matches the first, middle, and last ids and rejects a genuinely absent id. - -Three further regressions cover the close paths that route no work. -A declined decision closes with a recorded answer, satisfies `verify`, leaves Bearings' Captain's Call, and is refused while the hold still blocks routed work. -A hold closed by a direct `tasks-axi done` reproduces the shape that fails `verify` and blocks teardown, and `repair` with a captain decision file clears both. -An unanswered decision still blocks completion and teardown, and neither `decline` nor `repair` can close a hold that is still actively held or supply an answer with a missing or empty decision file. -`repair` also refuses a closed captain-kind task that was never held for the captain. - -Three answer-time closure regressions run against the published poll response shape, with synthetic `sample` identities. -A bound source whose origin exposes six holds captures one review carrying five structured choices plus one freeform message, and the runner feeds it through a fixture adapter that is not the review adapter at all, so what is proven is that any bound channel with an `answers` command gets closure rather than that one channel is wired specially. -Four holds whose answers route no work close, the one still blocking routed work is skipped and stays available to `resolve`, and the one whose key appears only inside the freeform prose never closes. -The capture is left unacknowledged throughout, so the wake firstmate needs in order to act on the answers is never retired. -A replayed delivery closes nothing new and is not rejected as a different decision, a source with no binding closes nothing at all, and the `answer` subcommand itself refuses an empty or missing decision file, an absent hold, and a drifted retry. -A separate regression drives the real `fm-send` over a stubbed transport to prove the chat channel reaches the same intake for a decision already transferred to its hold, which the status ledger alone can no longer close. -The cross-origin regression drives a bound source through the real runner and adapter interface, closes full-identity holds from different origins, and proves that over-limit, malformed, non-decision, routed-work, absent-hold, and replayed answers all fail or skip without weakening the existing guards. - -The final verification commands and their exact summarized outputs follow. - -```text -$ bash tests/fm-decision-hold-lifecycle.test.sh -ok - report-only unresolved decision is reproduced and completion refuses before loss -ok - non-forced scout teardown always requires durable inventory verification -ok - a declined decision closes with a recorded answer and no routed work -ok - a decision closed outside the script is repairable and then clears teardown -ok - an unanswered decision still blocks completion and resists both unrouted close paths -ok - captain holds are idempotent, distinct, teardown-safe, Bearings-visible, and durably routed before close -ok - completion and verification validate origins before constructing paths -ok - ended visual review follows the same decision-hold completion owner -ok - resolved findings and decision-like prose do not create false holds -ok - terminal single-owner stale status decisions do not block empty inventory -ok - main-home and secondmate-home captain holds remain correctly routed -ok - resolve matches first/middle/last in quoted blocked_by and rejects a genuinely absent id -ok - a bound channel's captured answers close their captain holds at answer time -ok - a channel source with no decision binding closes nothing -ok - an any-origin bound source closes full-identity holds across origins -ok - the answer path keeps every guard the unrouted close path already had -ok - the chat channel feeds the same keyed-answer intake a captured review does - -$ bash tests/fm-fleet-snapshot-view.test.sh -ok - backlog normalization preserves strict roles and resolves every blocker compatibly -ok - durable captain-held transfer closes the duplicate live status decision -ok - snapshot parses tasks-axi rows and respects operational overrides - -$ bash tests/fm-bearings-snapshot.test.sh -ok - a completed scout with decision-like report prose is a pointer, not pending -ok - an authoritative captain hold surfaces end-to-end -ok - action-free items (working/done/queued/landed) do not leak into Captain's Call -ok - main and secondmate captain actionability use the same blocker readiness - -$ bash tests/fm-send-resolve-key.test.sh -ok - fm-send --resolve-key: the answer send itself closes the open decision -ok - fm-send --resolve-key: a key that is not open refuses loudly before anything is sent -(13 assertions total; the status-log ledger's behavior is unchanged) - -$ bash tests/fm-brief.test.sh -ok - fm-brief.sh: investigation and visual-review completions load the shared decision policy - -$ bash tests/fm-teardown.test.sh -ok - the run abort and the leaked-process reap both complete before the destructive worktree return - -$ bin/fm-lint.sh -fm-lint.sh: ShellCheck 0.11.0 (pinned 0.11.0) - -$ bin/fm-doc-audience-check.sh -fm-doc-audience-check: ok surfaces=68 local_links=253 - -$ git diff --check -(no output) -``` diff --git a/docs/documentation-audiences.json b/docs/documentation-audiences.json index 4889467bd41..fb127a18a6c 100644 --- a/docs/documentation-audiences.json +++ b/docs/documentation-audiences.json @@ -128,6 +128,10 @@ "path": ".agents/skills/bootstrap-diagnostics/SKILL.md", "audience": "agent-runtime" }, + { + "path": ".agents/skills/captain-hold-lifecycle/SKILL.md", + "audience": "agent-runtime" + }, { "path": ".agents/skills/decision-hold-lifecycle/SKILL.md", "audience": "agent-runtime" @@ -184,6 +188,10 @@ "path": ".agents/skills/updatefirstmate/SKILL.md", "audience": "agent-runtime" }, + { + "path": ".greptile/rules.md", + "audience": "maintainer-architecture" + }, { "path": "AGENTS.md", "audience": "agent-runtime" @@ -245,7 +253,7 @@ "audience": "operator-current" }, { - "path": "docs/decision-hold-lifecycle.md", + "path": "docs/captain-hold-lifecycle.md", "audience": "maintainer-architecture" }, { @@ -256,6 +264,10 @@ "path": "docs/examples/crew-dispatch.json", "audience": "operator-example" }, + { + "path": "docs/examples/watched-tools.json", + "audience": "operator-example" + }, { "path": "docs/examples/wedge-alarm", "audience": "operator-example" @@ -368,6 +380,10 @@ "path": "docs/verification/trace-context.md", "audience": "maintainer-verification" }, + { + "path": "docs/voice-relay.md", + "audience": "operator-current" + }, { "path": "docs/watcher-continuity.md", "audience": "operator-current" diff --git a/docs/examples/watched-tools.json b/docs/examples/watched-tools.json new file mode 100644 index 00000000000..45d63a9d74d --- /dev/null +++ b/docs/examples/watched-tools.json @@ -0,0 +1,24 @@ +{ + "tools": [ + { + "name": "firstmate", + "git": { "repo": "/absolute/path/to/firstmate", "remote": "origin" } + }, + { + "name": "agents-on-the-go", + "git": { "repo": "/absolute/path/to/agents-on-the-go", "remote": "origin", "branch": "mainline" } + }, + { + "name": "herdr", + "command": "herdr", + "version_args": ["--version"] + }, + { + "name": "no-mistakes", + "command": "no-mistakes", + "version_args": ["--version"], + "announce_args": ["--help"], + "announce_pattern": "A new version of no-mistakes is available: [^ ]+ -> [^ ]+" + } + ] +} diff --git a/docs/fm-test-isolation-proof.json b/docs/fm-test-isolation-proof.json index ec605bf10f2..376ba99845a 100644 --- a/docs/fm-test-isolation-proof.json +++ b/docs/fm-test-isolation-proof.json @@ -1,36 +1,36 @@ { "concurrency": 4, - "finished_at": "2026-07-29T23:21:46Z", + "finished_at": "2026-08-21T00:45:57Z", "fm_test_run_jobs_enabled": false, "kind": "isolation-proof", "production_sharding_enabled": false, - "run_id": "fm-isolation-1785367157179-18165", + "run_id": "fm-isolation-1787273044622-10250", "scripts": [ - {"duration_ms": 46788, "exit": 0, "path": "tests/fm-arm-pretool-check.test.sh", "worker": 1}, - {"duration_ms": 48294, "exit": 0, "path": "tests/fm-backend-herdr.test.sh", "worker": 2}, - {"duration_ms": 2224, "exit": 0, "path": "tests/fm-brief.test.sh", "worker": 3}, - {"duration_ms": 34207, "exit": 0, "path": "tests/fm-cd-pretool-check.test.sh", "worker": 4}, - {"duration_ms": 9065, "exit": 0, "path": "tests/fm-composer-ghost.test.sh", "worker": 5}, - {"duration_ms": 64, "exit": 0, "path": "tests/fm-composer-lib.test.sh", "worker": 6}, - {"duration_ms": 25365, "exit": 0, "path": "tests/fm-crew-state.test.sh", "worker": 7}, - {"duration_ms": 30771, "exit": 0, "path": "tests/fm-decision-hold-lifecycle.test.sh", "worker": 8}, - {"duration_ms": 581, "exit": 0, "path": "tests/fm-ensure-agents-md.test.sh", "worker": 9}, - {"duration_ms": 6251, "exit": 0, "path": "tests/fm-grok-harness.test.sh", "worker": 10}, - {"duration_ms": 15422, "exit": 0, "path": "tests/fm-herdr-lab.test.sh", "worker": 11}, - {"duration_ms": 5237, "exit": 0, "path": "tests/fm-lint.test.sh", "worker": 12}, - {"duration_ms": 2945, "exit": 0, "path": "tests/fm-pi-primary-types.test.sh", "worker": 13}, - {"duration_ms": 8564, "exit": 0, "path": "tests/fm-pr-merge.test.sh", "worker": 14}, - {"duration_ms": 2875, "exit": 0, "path": "tests/fm-review-diff.test.sh", "worker": 15}, - {"duration_ms": 5644, "exit": 0, "path": "tests/fm-send-popup-settle.test.sh", "worker": 16}, - {"duration_ms": 2911, "exit": 0, "path": "tests/fm-send-settle.test.sh", "worker": 17}, - {"duration_ms": 2747, "exit": 0, "path": "tests/fm-send-strict.test.sh", "worker": 18}, - {"duration_ms": 855, "exit": 0, "path": "tests/fm-spawn-batch.test.sh", "worker": 19}, - {"duration_ms": 703, "exit": 0, "path": "tests/fm-supervision-instructions.test.sh", "worker": 20}, - {"duration_ms": 15674, "exit": 0, "path": "tests/fm-test-run.test.sh", "worker": 21}, - {"duration_ms": 4816, "exit": 0, "path": "tests/fm-tmux-submit-busy.test.sh", "worker": 22}, - {"duration_ms": 248, "exit": 0, "path": "tests/fm-transition-lib.test.sh", "worker": 23}, - {"duration_ms": 52939, "exit": 0, "path": "tests/fm-x-mode.test.sh", "worker": 24} + {"duration_ms": 27529, "exit": 0, "path": "tests/fm-arm-pretool-check.test.sh", "worker": 1}, + {"duration_ms": 45356, "exit": 0, "path": "tests/fm-backend-herdr.test.sh", "worker": 2}, + {"duration_ms": 1315, "exit": 0, "path": "tests/fm-brief.test.sh", "worker": 3}, + {"duration_ms": 35095, "exit": 0, "path": "tests/fm-captain-hold-lifecycle.test.sh", "worker": 4}, + {"duration_ms": 16582, "exit": 0, "path": "tests/fm-cd-pretool-check.test.sh", "worker": 5}, + {"duration_ms": 5569, "exit": 0, "path": "tests/fm-composer-ghost.test.sh", "worker": 6}, + {"duration_ms": 3544, "exit": 0, "path": "tests/fm-composer-lib.test.sh", "worker": 7}, + {"duration_ms": 17558, "exit": 0, "path": "tests/fm-crew-state.test.sh", "worker": 8}, + {"duration_ms": 513, "exit": 0, "path": "tests/fm-ensure-agents-md.test.sh", "worker": 9}, + {"duration_ms": 6768, "exit": 0, "path": "tests/fm-grok-harness.test.sh", "worker": 10}, + {"duration_ms": 9562, "exit": 0, "path": "tests/fm-herdr-lab.test.sh", "worker": 11}, + {"duration_ms": 9766, "exit": 0, "path": "tests/fm-lint.test.sh", "worker": 12}, + {"duration_ms": 598, "exit": 0, "path": "tests/fm-pi-primary-types.test.sh", "worker": 13}, + {"duration_ms": 6290, "exit": 0, "path": "tests/fm-pr-merge.test.sh", "worker": 14}, + {"duration_ms": 2166, "exit": 0, "path": "tests/fm-review-diff.test.sh", "worker": 15}, + {"duration_ms": 4563, "exit": 0, "path": "tests/fm-send-popup-settle.test.sh", "worker": 16}, + {"duration_ms": 2753, "exit": 0, "path": "tests/fm-send-settle.test.sh", "worker": 17}, + {"duration_ms": 3025, "exit": 0, "path": "tests/fm-send-strict.test.sh", "worker": 18}, + {"duration_ms": 975, "exit": 0, "path": "tests/fm-spawn-batch.test.sh", "worker": 19}, + {"duration_ms": 331, "exit": 0, "path": "tests/fm-supervision-instructions.test.sh", "worker": 20}, + {"duration_ms": 20922, "exit": 0, "path": "tests/fm-test-run.test.sh", "worker": 21}, + {"duration_ms": 4021, "exit": 0, "path": "tests/fm-tmux-submit-busy.test.sh", "worker": 22}, + {"duration_ms": 99, "exit": 0, "path": "tests/fm-transition-lib.test.sh", "worker": 23}, + {"duration_ms": 35415, "exit": 0, "path": "tests/fm-x-mode.test.sh", "worker": 24} ], - "started_at": "2026-07-29T23:19:17Z", - "summary": {"duration_ms": 149010, "failed": 0, "total": 24} + "started_at": "2026-08-21T00:44:04Z", + "summary": {"duration_ms": 113278, "failed": 0, "total": 24} } diff --git a/docs/fm-test-isolation-proof.md b/docs/fm-test-isolation-proof.md index 716dca73a56..3ee9b18f3b1 100644 --- a/docs/fm-test-isolation-proof.md +++ b/docs/fm-test-isolation-proof.md @@ -6,30 +6,30 @@ This record is the concurrent isolation proof for the portable parallel candidat ## Verification -- Date: 2026-07-29 -- Command: `bin/fm-test-isolation-proof.sh --jobs 4 --json /tmp/fm-source-content-test-cleanup-r1-isolation.json` -- Result: `FM_ISOLATION_SUMMARY total=24 failed=0 concurrency=4 duration_ms=149010` +- Date: 2026-08-20 +- Command: `bin/fm-test-isolation-proof.sh --jobs 4 --json /tmp/fm-isolation-proof.json` +- Result: `FM_ISOLATION_SUMMARY total=24 failed=0 concurrency=4 duration_ms=113278` | Field | Value | |---|---| -| `run_id` | `fm-isolation-1785367157179-18165` | -| `started_at` | `2026-07-29T23:19:17Z` | -| `finished_at` | `2026-07-29T23:21:46Z` | +| `run_id` | `fm-isolation-1787273044622-10250` | +| `started_at` | `2026-08-21T00:44:04Z` | +| `finished_at` | `2026-08-21T00:45:57Z` | | concurrency | 4 | | candidates | 24 | | failed | 0 | -| wall duration | 149010 ms | +| wall duration | 113278 ms | ## Candidate set - `tests/fm-arm-pretool-check.test.sh` - `tests/fm-backend-herdr.test.sh` - `tests/fm-brief.test.sh` +- `tests/fm-captain-hold-lifecycle.test.sh` - `tests/fm-cd-pretool-check.test.sh` - `tests/fm-composer-ghost.test.sh` - `tests/fm-composer-lib.test.sh` - `tests/fm-crew-state.test.sh` -- `tests/fm-decision-hold-lifecycle.test.sh` - `tests/fm-ensure-agents-md.test.sh` - `tests/fm-grok-harness.test.sh` - `tests/fm-herdr-lab.test.sh` @@ -51,30 +51,30 @@ This record is the concurrent isolation proof for the portable parallel candidat | duration_ms | exit | worker | script | |---:|---:|---:|---| -| 52939 | 0 | 24 | `tests/fm-x-mode.test.sh` | -| 48294 | 0 | 2 | `tests/fm-backend-herdr.test.sh` | -| 46788 | 0 | 1 | `tests/fm-arm-pretool-check.test.sh` | -| 34207 | 0 | 4 | `tests/fm-cd-pretool-check.test.sh` | -| 30771 | 0 | 8 | `tests/fm-decision-hold-lifecycle.test.sh` | -| 25365 | 0 | 7 | `tests/fm-crew-state.test.sh` | -| 15674 | 0 | 21 | `tests/fm-test-run.test.sh` | -| 15422 | 0 | 11 | `tests/fm-herdr-lab.test.sh` | -| 9065 | 0 | 5 | `tests/fm-composer-ghost.test.sh` | -| 8564 | 0 | 14 | `tests/fm-pr-merge.test.sh` | -| 6251 | 0 | 10 | `tests/fm-grok-harness.test.sh` | -| 5644 | 0 | 16 | `tests/fm-send-popup-settle.test.sh` | -| 5237 | 0 | 12 | `tests/fm-lint.test.sh` | -| 4816 | 0 | 22 | `tests/fm-tmux-submit-busy.test.sh` | -| 2945 | 0 | 13 | `tests/fm-pi-primary-types.test.sh` | -| 2911 | 0 | 17 | `tests/fm-send-settle.test.sh` | -| 2875 | 0 | 15 | `tests/fm-review-diff.test.sh` | -| 2747 | 0 | 18 | `tests/fm-send-strict.test.sh` | -| 2224 | 0 | 3 | `tests/fm-brief.test.sh` | -| 855 | 0 | 19 | `tests/fm-spawn-batch.test.sh` | -| 703 | 0 | 20 | `tests/fm-supervision-instructions.test.sh` | -| 581 | 0 | 9 | `tests/fm-ensure-agents-md.test.sh` | -| 248 | 0 | 23 | `tests/fm-transition-lib.test.sh` | -| 64 | 0 | 6 | `tests/fm-composer-lib.test.sh` | +| 45356 | 0 | 2 | `tests/fm-backend-herdr.test.sh` | +| 35415 | 0 | 24 | `tests/fm-x-mode.test.sh` | +| 35095 | 0 | 4 | `tests/fm-captain-hold-lifecycle.test.sh` | +| 27529 | 0 | 1 | `tests/fm-arm-pretool-check.test.sh` | +| 20922 | 0 | 21 | `tests/fm-test-run.test.sh` | +| 17558 | 0 | 8 | `tests/fm-crew-state.test.sh` | +| 16582 | 0 | 5 | `tests/fm-cd-pretool-check.test.sh` | +| 9766 | 0 | 12 | `tests/fm-lint.test.sh` | +| 9562 | 0 | 11 | `tests/fm-herdr-lab.test.sh` | +| 6768 | 0 | 10 | `tests/fm-grok-harness.test.sh` | +| 6290 | 0 | 14 | `tests/fm-pr-merge.test.sh` | +| 5569 | 0 | 6 | `tests/fm-composer-ghost.test.sh` | +| 4563 | 0 | 16 | `tests/fm-send-popup-settle.test.sh` | +| 4021 | 0 | 22 | `tests/fm-tmux-submit-busy.test.sh` | +| 3544 | 0 | 7 | `tests/fm-composer-lib.test.sh` | +| 3025 | 0 | 18 | `tests/fm-send-strict.test.sh` | +| 2753 | 0 | 17 | `tests/fm-send-settle.test.sh` | +| 2166 | 0 | 15 | `tests/fm-review-diff.test.sh` | +| 1315 | 0 | 3 | `tests/fm-brief.test.sh` | +| 975 | 0 | 19 | `tests/fm-spawn-batch.test.sh` | +| 598 | 0 | 13 | `tests/fm-pi-primary-types.test.sh` | +| 513 | 0 | 9 | `tests/fm-ensure-agents-md.test.sh` | +| 331 | 0 | 20 | `tests/fm-supervision-instructions.test.sh` | +| 99 | 0 | 23 | `tests/fm-transition-lib.test.sh` | ## Scope diff --git a/docs/fm-test-portable-shards.md b/docs/fm-test-portable-shards.md index 5cf681a5019..89b0ef2c2df 100644 --- a/docs/fm-test-portable-shards.md +++ b/docs/fm-test-portable-shards.md @@ -5,35 +5,35 @@ ## Verification inputs -The current candidate timings came from the 2026-07-29 concurrent proof recorded in [fm-test-isolation-proof.md](fm-test-isolation-proof.md). +The current candidate timings came from the 2026-08-20 concurrent proof recorded in [fm-test-isolation-proof.md](fm-test-isolation-proof.md). The proof ran 24 candidates with four workers and no failures. | duration_ms | script | |---:|---| -| 52939 | `tests/fm-x-mode.test.sh` | -| 48294 | `tests/fm-backend-herdr.test.sh` | -| 46788 | `tests/fm-arm-pretool-check.test.sh` | -| 34207 | `tests/fm-cd-pretool-check.test.sh` | -| 30771 | `tests/fm-decision-hold-lifecycle.test.sh` | -| 25365 | `tests/fm-crew-state.test.sh` | -| 15674 | `tests/fm-test-run.test.sh` | -| 15422 | `tests/fm-herdr-lab.test.sh` | -| 9065 | `tests/fm-composer-ghost.test.sh` | -| 8564 | `tests/fm-pr-merge.test.sh` | -| 6251 | `tests/fm-grok-harness.test.sh` | -| 5644 | `tests/fm-send-popup-settle.test.sh` | -| 5237 | `tests/fm-lint.test.sh` | -| 4816 | `tests/fm-tmux-submit-busy.test.sh` | -| 2945 | `tests/fm-pi-primary-types.test.sh` | -| 2911 | `tests/fm-send-settle.test.sh` | -| 2875 | `tests/fm-review-diff.test.sh` | -| 2747 | `tests/fm-send-strict.test.sh` | -| 2224 | `tests/fm-brief.test.sh` | -| 855 | `tests/fm-spawn-batch.test.sh` | -| 703 | `tests/fm-supervision-instructions.test.sh` | -| 581 | `tests/fm-ensure-agents-md.test.sh` | -| 248 | `tests/fm-transition-lib.test.sh` | -| 64 | `tests/fm-composer-lib.test.sh` | +| 45356 | `tests/fm-backend-herdr.test.sh` | +| 35415 | `tests/fm-x-mode.test.sh` | +| 35095 | `tests/fm-captain-hold-lifecycle.test.sh` | +| 27529 | `tests/fm-arm-pretool-check.test.sh` | +| 20922 | `tests/fm-test-run.test.sh` | +| 17558 | `tests/fm-crew-state.test.sh` | +| 16582 | `tests/fm-cd-pretool-check.test.sh` | +| 9766 | `tests/fm-lint.test.sh` | +| 9562 | `tests/fm-herdr-lab.test.sh` | +| 6768 | `tests/fm-grok-harness.test.sh` | +| 6290 | `tests/fm-pr-merge.test.sh` | +| 5569 | `tests/fm-composer-ghost.test.sh` | +| 4563 | `tests/fm-send-popup-settle.test.sh` | +| 4021 | `tests/fm-tmux-submit-busy.test.sh` | +| 3544 | `tests/fm-composer-lib.test.sh` | +| 3025 | `tests/fm-send-strict.test.sh` | +| 2753 | `tests/fm-send-settle.test.sh` | +| 2166 | `tests/fm-review-diff.test.sh` | +| 1315 | `tests/fm-brief.test.sh` | +| 975 | `tests/fm-spawn-batch.test.sh` | +| 598 | `tests/fm-pi-primary-types.test.sh` | +| 513 | `tests/fm-ensure-agents-md.test.sh` | +| 331 | `tests/fm-supervision-instructions.test.sh` | +| 99 | `tests/fm-transition-lib.test.sh` | ## Parallel lanes @@ -41,9 +41,9 @@ The two parallel lanes use longest-processing-time assignment from those measure | Lane | Script count | Estimated duration | |---|---:|---:| -| `portable-parallel-1` | 11 | 162436 ms (~162.4 s) | -| `portable-parallel-2` | 13 | 162754 ms (~162.8 s) | -| imbalance | | 318 ms | +| `portable-parallel-1` | 11 | 134295 ms (~134.3 s) | +| `portable-parallel-2` | 13 | 126020 ms (~126.0 s) | +| imbalance | | 8275 ms | `bin/fm-test-run.sh` contains the exact ordered memberships in `list_portable_parallel_1` and `list_portable_parallel_2`. @@ -64,19 +64,23 @@ Each shard is still strictly serial in itself, and separate runners mean no two `.github/workflows/ci.yml` derives the same `n` from `strategy.job-total` rather than a literal, so changing the shard count in either file without the other fails the lane loudly instead of leaving part of the required suite unrun. Assignment is longest-processing-time bin packing over per-script duration hints embedded in `bin/fm-test-run.sh`. -The hints came from that run's `fm-test-timing-portable-serial` artifact on 2026-08-02, where the lane ran 69 scripts in 1143762 ms of serial work. +The hints came from the `fm-test-timing-portable-serial-*` artifacts of green CI run [32491999845](https://github.com/kunchenguid/firstmate/actions/runs/32491999845) on 2026-08-21, where the lane ran 116 scripts in 2541548 ms of serial work. +`tests/fm-tool-update-check.test.sh` did not exist on that run, so its 12846 ms hint comes from the shard 3 artifact of run [32461816719](https://github.com/kunchenguid/firstmate/actions/runs/32461816719), which is the first run that measured it. +`tests/fm-voice-relay.test.sh` and this fork's `tests/fm-claude-settings-live-e2e.test.sh` are unmeasured and carry the default weight, so the serial lane is 120 scripts against that run's 116. A script with no hint gets the conservative `PORTABLE_SERIAL_DEFAULT_WEIGHT_MS` default. Hints only affect balance: the coverage guard keeps the partition complete and disjoint whatever they say, so a stale hint costs a slower shard rather than lost coverage. +Balance is still worth keeping current, because enough unmeasured scripts let one shard carry more than twice another shard's real work and reach the job cap while another runner sits idle. +Refresh the hints whenever the serial lane gains scripts, rather than waiting for a shard to time out. | Lane | Script count | Estimated duration | |---|---:|---:| -| `portable-serial-1of4` | 15 | 285945 ms (~285.9 s) | -| `portable-serial-2of4` | 18 | 285944 ms (~285.9 s) | -| `portable-serial-3of4` | 17 | 285929 ms (~285.9 s) | -| `portable-serial-4of4` | 19 | 285944 ms (~285.9 s) | -| imbalance | | 16 ms | +| `portable-serial-1of4` | 29 | 648666 ms (~648.7 s) | +| `portable-serial-2of4` | 30 | 648662 ms (~648.7 s) | +| `portable-serial-3of4` | 31 | 648679 ms (~648.7 s) | +| `portable-serial-4of4` | 30 | 648663 ms (~648.7 s) | +| imbalance | | 17 ms | -The single longest script, `tests/fm-pr-check-security.test.sh` at 199573 ms, is the floor for any shard count. +The single longest script, `tests/fm-pr-check-security.test.sh` at 250417 ms, is the floor for any shard count. Refresh the hints by downloading the per-shard timing artifacts from a green CI run, replacing the `portable_serial_weight_hints` table in `bin/fm-test-run.sh` with the measured `path`/`duration_ms` pairs, and updating the table above: @@ -108,7 +112,7 @@ Portable shards, each portable serial shard, and the Herdr lane upload runner-ge | Lane | Bound | Rationale | |---|---|---| | portable parallel 1/2 | job `timeout-minutes: 10` | The measured shard sums are about three minutes and the timeout is a hang tripwire. | -| portable serial 1-4 | job `timeout-minutes: 15` | Each balanced shard is about five minutes, leaving roughly 3x hang-tripwire margin. | +| portable serial 1-4 | job `timeout-minutes: 20` | Each balanced shard is about eleven minutes of measured script time, leaving roughly 2x hang-tripwire margin for job setup and runner-speed spread. | | Herdr | family-run step `timeout-minutes: 20`; job `timeout-minutes: 75` backstop | Healthy runs finish around 7 minutes, so the step bound is the hang tripwire (cleanup and timing artifacts still upload) while the job cap stays a last-resort backstop. | Timeouts are hang tripwires rather than expected healthy durations. diff --git a/docs/gitlab-merge-watch.md b/docs/gitlab-merge-watch.md index 0540ed296d1..79dc138e1f6 100644 --- a/docs/gitlab-merge-watch.md +++ b/docs/gitlab-merge-watch.md @@ -1,7 +1,8 @@ -# GitLab merge request watch verification +# GitLab merge request watch and merge verification -Empirical record for the merge watch on GitLab, alongside the existing GitHub watch. -Every command below was run on 2026-07-21 and its output is reproduced exactly. +Empirical record for the merge watch and the merge path on GitLab, alongside the existing GitHub ones. +Every command through "Upgrade path from an existing armed watch" was run on 2026-07-21; "Merging a merge request" was run on 2026-08-22. +Every output is reproduced exactly. ## Versions @@ -13,6 +14,21 @@ $ bash --version | head -1 GNU bash, version 5.3.9(1)-release (x86_64-pc-linux-gnu) ``` +The merge evidence dated 2026-08-22 was collected on a different host, on: + +``` +$ glab --version +glab 1.82.0-<local build tag> (<local build commit>) + +$ jq --version +jq-1.8.1 + +$ bash --version | head -1 +GNU bash, version 5.2.15(1)-release (x86_64-amazon-linux-gnu) +``` + +That `glab` is a locally built 1.82.0; only its build tag and commit are elided, because they name a private build rather than a released version. + ## The evidence project All live evidence here reads <https://gitlab.com/KarotKris/gitlab-merge-watch-fixture>, a public project that exists only to be this evidence. @@ -190,11 +206,92 @@ merged No armed watch is lost by upgrading. -## What this change does not cover +## Merging a merge request + +`bin/fm-pr-merge.sh` now merges a GitLab merge request through the same recording and the same guards a GitHub pull request gets. +Every run below used a throwaway `FM_HOME`, so no live task record was touched, and a `glab` wrapper that refused any `merge` subcommand outright, so no merge could reach the forge even if a check were wrong. +That wrapper is why the open fixture merge request could be used as evidence at all: it is `mergeable` with discussions resolved, so the pipeline conditions are the only thing between it and a real merge. + +Merging needs `glab` for the read and `jq` to parse it, and either one absent refuses before anything is recorded: + +``` +$ PATH="$noglab" fm-pr-merge.sh e5 https://gitlab.com/KarotKris/gitlab-merge-watch-fixture/-/merge_requests/2 +error: merging a GitLab merge request requires glab on PATH +$ echo $? +1 +$ PATH="$nojq" fm-pr-merge.sh e6 https://gitlab.com/KarotKris/gitlab-merge-watch-fixture/-/merge_requests/2 +error: merging a GitLab merge request requires jq on PATH +$ echo $? +1 +``` + +Neither refusal armed a poll or recorded a `pr=`, so a missing tool leaves no half-prepared merge behind. + +`jq` is not one of firstmate's common tools, which is why the watch poll reads glab's field output instead. +The merge path cannot do the same: `detailed_merge_status`, `has_conflicts`, `blocking_discussions_resolved`, and the head pipeline appear only in glab's JSON. +The poll's silence on a missing tool is safe because silence means "not merged yet"; a merge cannot be silent about it, so the requirement is reported rather than assumed. + +The merged half of the fixture is refused, and every failing condition is listed rather than just the first: + +``` +$ fm-pr-merge.sh e1 https://gitlab.com/KarotKris/gitlab-merge-watch-fixture/-/merge_requests/1 +armed: state/e1.check.sh +error: refusing to merge https://gitlab.com/KarotKris/gitlab-merge-watch-fixture/-/merge_requests/1 + - state is "merged", not open + - detailed_merge_status is "not_open", not mergeable + - the head pipeline status is "none", not success + - the head pipeline ran at "none", not at the current head 33762fcf6777c8d993220d25fb541e56c48081b9 +$ echo $? +1 +``` + +The open half is `mergeable`, conflict-free, and has its discussions resolved, so only the pipeline conditions refuse it. +The fixture runs no CI, so its `head_pipeline` is `null`, which is reported as `none` rather than treated as nothing to check: + +``` +$ fm-pr-merge.sh e2 https://gitlab.com/KarotKris/gitlab-merge-watch-fixture/-/merge_requests/2 +armed: state/e2.check.sh +error: refusing to merge https://gitlab.com/KarotKris/gitlab-merge-watch-fixture/-/merge_requests/2 + - the head pipeline status is "none", not success + - the head pipeline ran at "none", not at the current head 66b8a6777bea5e291d7fa2fc20c42ad7686f6bc8 +$ echo $? +1 +``` + +A project that runs no pipeline at all therefore cannot merge through this path. +That is the intended reading of the requirement rather than an oversight: a successful pipeline at the head is a condition, and "there is no pipeline" does not satisfy it. + +Both refusals came after `pr=` was recorded and the merge poll was armed, exactly as a failing `gh-axi pr merge` does on the GitHub side, so a refusal still leaves the audit trail and the watch in place. + +A recorded `pr_head=` that no longer matches the live head is reported, and the live head is what gets verified. +The stale value below was written into the task record by hand, because a GitLab task never records one on its own: + +``` +$ fm-pr-merge.sh e4 https://gitlab.com/KarotKris/gitlab-merge-watch-fixture/-/merge_requests/2 +armed: state/e4.check.sh +notice: recorded head 1111111111111111111111111111111111111111 disagrees with the live head 66b8a6777bea5e291d7fa2fc20c42ad7686f6bc8; verifying the live head +error: refusing to merge https://gitlab.com/KarotKris/gitlab-merge-watch-fixture/-/merge_requests/2 + - the head pipeline status is "none", not success + - the head pipeline ran at "none", not at the current head 66b8a6777bea5e291d7fa2fc20c42ad7686f6bc8 +``` + +The remaining refusal conditions, and the merge itself, are covered by `tests/fm-pr-merge.test.sh` against fixtures. +The conflict, unresolved-discussion, and running-pipeline conditions were additionally exercised against real merge requests on a private instance; those runs cannot be reproduced here, so their identifiers stay out of this record. +The merge itself is not exercised against any live merge request, in either direction: `glab mr merge` has no dry run, so a live success path would mean merging someone's work to produce evidence. + +## Why the head is read live and bound to the merge + +The verified head is passed to `glab mr merge --sha`, so GitLab refuses the merge if the source branch moved between the read and the merge. +Without it, a push landing in that window would merge commits nothing verified. + +`--yes` is passed for the same reason the watch poll needs no terminal: an unattended run cannot answer a confirmation prompt, and a wedged prompt is worse than a refusal. +It skips only that prompt; the conditions above are what authorize the merge. + +## Why a recorded head is not the authority -`bin/fm-pr-merge.sh` still addresses GitHub only, by owner and repository. -It refuses a GitLab merge request URL rather than sending it to the wrong forge, so merging a merge request stays a deliberate manual step until merge parity lands separately. +`bin/fm-pr-check.sh` records `pr_head=` only for GitHub, where `gh` exposes the head commit as a selectable field. +It is optional by design, and the other consumers already treat it that way: `bin/fm-teardown.sh` reads the head from the forge at teardown and falls back to its provider-agnostic content check, and `bin/fm-review-diff.sh` resolves the head from the remote when none is recorded. -A GitLab task records no `pr_head=`. -`gh` exposes the head commit as a selectable field, while plain `glab` exposes it only inside its JSON output, which would need a JSON processor firstmate does not require. -Both consumers already treat it as optional: `bin/fm-teardown.sh` reads the head from the forge at teardown rather than from metadata and falls back to its provider-agnostic content check, and `bin/fm-review-diff.sh` resolves the head from the remote when none is recorded. +The merge path does not record one either, and deliberately does not depend on one. +A rebase moves the head and leaves any recorded value stale, so a merge decided from metadata can verify a commit that no longer exists. +Reading the head live at merge time, reporting a recorded value that disagrees, and binding the merge to what was actually verified is what closes that gap. diff --git a/docs/herdr-backend.md b/docs/herdr-backend.md index 22f9d967aaf..a8ee9556774 100644 --- a/docs/herdr-backend.md +++ b/docs/herdr-backend.md @@ -238,7 +238,8 @@ A human-blocked permission dialog has no busy banner and still surfaces. ## Composer and injection safety Herdr has no direct cursor-row primitive. -The adapter is a thin capture: it hands a bounded ANSI tail plus Herdr's capability facts to the fleet-wide classifier in `bin/fm-composer-lib.sh`, which owns every shape - bordered boxes, bare agent-glyph rows (including muse's `⟩`, which the adapter's retired local pattern silently omitted), opencode's left bar, and the Pi separator region this adapter pioneered, admitted only when native `agent get` identity is exactly Pi and state is idle, done, or blocked. +The adapter is a thin capture: it hands a bounded ANSI tail plus Herdr's capability facts to the fleet-wide classifier in `bin/fm-composer-lib.sh`, which owns every shape - bordered boxes, bare agent-glyph rows (including muse's `⟩`, which the adapter's retired local pattern silently omitted), opencode's left bar, and the Pi separator region this adapter pioneered, admitted only when native `agent get` identity is exactly Pi and state is idle or done. +A blocked Pi is parked on an interactive prompt, so its blank composer region is a menu's and not a free composer's; that state defers instead of proving emptiness. A working Pi, pending middle row, missing identity, incomplete separator pair, or over-tall candidate remains unknown or pending. Identity stays a lazy second read, consulted only when a separator pair could change the verdict. @@ -275,7 +276,7 @@ Mid-session secondmate liveness is not implemented because idle secondmates are Protocol 16 can subscribe to `pane.agent_status_changed` over one bounded Unix-socket reader. `bin/fm-transition-lib.sh` owns the backend-neutral transition vocabulary and policy. The Herdr adapter subscribes before reconciling current levels, buffers edges during reconciliation, and returns fresh blocked transitions for this home's panes. -The watcher maps the pane back to the task and skips secondmate endpoints and declared `paused:` waits. +The watcher maps the pane back to the task and skips secondmate endpoints, declared `paused:` waits, and verified `captain-held` transfers, because a declared wait already names the human the fast escalation would report and is left to the watcher's own bounded pause cadence. The push path only shortens latency. Polling runs every cycle and remains the permanent fallback when protocol 16, the event schema, Python, connection, subscription, or repeated reader execution is unavailable. diff --git a/docs/scripts.md b/docs/scripts.md index 72a509c6006..3359c32e6c8 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -26,7 +26,8 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-remote-doctor.sh` | Check, and with `--fix` repair, one remote account's second-mate readiness (remote job worker, Herdr, Aqua launch agents, PATH, and required tools) | | `fm-backlog-handoff.sh` | Validate and delegate queued backlog-item moves into a secondmate home | | `fm-backlog-receive.sh` | Idempotently ingest one confined remote handoff outbox through tasks-axi | -| `fm-decision-hold.sh` | Create, verify, complete, close, and repair durable captain-held decisions | +| `fm-captain-hold.sh` | Hold tasks for the captain, record the captain's answers, gate investigation completion, and report record divergence between the status log and the backlog | +| `fm-decision-hold.sh` | One-release compatibility shim mapping the retired decision commands onto fm-captain-hold.sh | | `fm-brief.sh` | Scaffold ship (explicit `--mode`), scout, secondmate-charter, and Herdr-lab briefs | | `fm-herdr-lab.sh` | Provision and guardedly operate an isolated, never-default Herdr lab session | | `fm-install-herdr.sh` | Install CI's exact-version Herdr pin with official asset URL, SHA-256, and protocol checks | @@ -90,7 +91,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-tasks-axi-lib.sh` | Shared backlog-backend selector and `tasks-axi` compatibility probe | | `fm-quota-axi-lib.sh` | Shared `quota-axi` compatibility floor for the bootstrap diagnostic | | `fm-vendor-auth-probe.sh`| Run one hard-bounded, non-destructive authentication probe of a named vendor CLI and report the fact | -| `fm-wake-drain.sh` | Present durable watcher wakes, unread informational status lines, and OPEN DECISIONS, consume acknowledged rows through their sequence, retire only the matching recovery generation, then assert supervision health | +| `fm-wake-drain.sh` | Present durable watcher wakes, unread informational status lines, OPEN DECISIONS, and captain-call RECORD DIVERGENCE, consume acknowledged rows through their sequence, retire only the matching recovery generation, then assert supervision health | | `fm-wake-lib.sh` | Shared durable wake queue, recovery generations, portable locks, and watcher identity/health helpers | | `fm-classify-lib.sh` | Shared wake-classification vocabulary, durable keyed-decision folds and scans, and unread informational status-line selection | | `fm-send.sh` | Send one verified literal line or supported key through the target's recorded backend | @@ -102,11 +103,12 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-peek.sh` | Print a bounded tail of a crewmate endpoint | | `fm-check-register.sh` | Bind an intentional custom watcher check to its current bytes | | `fm-check-lib.sh` | Validate custom-check registrations and prepare private execution snapshots | +| `fm-tool-update-check.sh` | Report watched tooling with an update available, and updates installed but left inert by PATH order | | `fm-pr-lib.sh` | Own canonical task and PR validation plus private atomic PR-poll publication and identity-bound retirement | | `fm-pr-poll.sh` | Provide the byte-static watcher program for validated PR/MR-poll sidecars | | `fm-pr-check-migrate.sh` | Quarantine older task polls without execution and rebuild only canonical polls | | `fm-pr-check.sh` | Record validated `pr=` and `pr_head=` values, then atomically arm a static merge poll | -| `fm-pr-merge.sh` | Record PR metadata, then merge a task's canonical full GitHub URL | +| `fm-pr-merge.sh` | Record PR metadata, then merge a task's canonical full GitHub or GitLab URL | | `fm-promote.sh` | Promote a scout task in place to a protected ship task with an explicit delivery mode | | `fm-teardown.sh` | Fail-closed teardown: return landed ship worktrees, require completed scout deliverables, retire secondmate homes | | `fm-harness.sh` | Detect the running harness and resolve crew or secondmate harness, model, and effort | @@ -117,6 +119,11 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-x-dismiss.sh` | Dismiss a skipped Relay mention at the relay without replying | | `fm-x-link.sh` | Link a spawned task to its originating Relay mention in task meta | | `fm-x-followup.sh` | Detect, post, and cap completion follow-ups for a Relay-linked task | -| `fm-public-followup-lib.sh` | Shared relay-activation gate, O(1) presence checks, and private transport paths for promised public replies | -| `fm-public-followup.sh` | Reconcile typed terminal work results into a public commitment and deliver its final reply once | +| `fm-public-followup-lib.sh` | Shared Relay gate, open-loop registry state, expiry classification, locking, and private transport paths | +| `fm-public-followup.sh` | Reconcile and deliver typed public commitments, then rechain or explicitly retire their retained loops | | `fm-public-followup-emit.sh` | Report one typed terminal work result into the home that owes the public reply | +| `fm-inbox.sh` | The captain's out-of-band capture surface: queue a note, dictate one, read status, ask a side question | +| `fm-voice-relay.py` | Hold the spoken conversation on this host, answer from the records, and hand real work to `fm-inbox.sh` ([voice-relay.md](voice-relay.md)) | +| `fm-voice-client.py` | The laptop end of the spoken interface: capture, playback, and turn timing over SSH; audio devices unverified | +| `fm_voice_frame.py` | The wire format both machines share, copied to the laptop beside the client | +| `fm_voice_records.py` | What a spoken answer may read, and the handover that queues real work | diff --git a/docs/turnend-guard.md b/docs/turnend-guard.md index 3620230f833..de9b5ed922e 100644 --- a/docs/turnend-guard.md +++ b/docs/turnend-guard.md @@ -72,7 +72,14 @@ In the default Codex mode, a true value lets the second stop finish after one fo Claude runs the guard with `--claude`, which ignores `stop_hook_active` and cooperates with the Stop-owned auto-arm. Claude Code sets `stop_hook_active=true` on every stop after any stop-hook continuation, including `asyncRewake` rewakes, which re-opened the 2026-07-21 blind window under the default one-shot behavior. -The Claude mode waits up to `FM_CLAUDE_AUTOARM_SYNC_WAIT_MS` (default 800 milliseconds) and allows the stop when the watcher is healthy, `state/.claude-autoarm.lock` has a live `autoarm` role owner whose eventual failure must exit 2, or `state/.claude-autoarm-epoch` contains a fresh actionable rewake owned by this event epoch. +The Claude mode waits up to `FM_CLAUDE_AUTOARM_SYNC_WAIT_MS` (default 800 milliseconds) and allows the stop when the watcher is healthy, `state/.claude-autoarm.lock` has a live `autoarm` role owner whose supervision decision is still open and whose eventual failure must exit 2, or `state/.claude-autoarm-epoch` contains a fresh actionable rewake owned by this event epoch. +A live owner counts as that proof only while its decision is open, which the ledger settles: an entry naming that owner's own pid with any outcome other than `arming` means the claim already finished, so the lock is abandoned rather than in flight. +The guard then stops reading it as recovery under way, the terminal check clears it instead of stepping aside for it, and the next Stop-owned firing reclaims it and arms rather than deferring. +Without that boundary a cycle that armed, delivered one rewake, and exited left both Stop participants deferring to its leftover lock indefinitely, so on 2026-08-14 a home with two tasks in flight and a beacon 40 minutes cold ended every turn blind until an operator intervened. +An `arming` entry stays in flight however old it is, because the owner foregrounds the arm for the whole watcher cycle. +The shapes the ledger cannot settle are settled by identity instead: the claim records the same `pid-identity` file every other supervision lock records, before it publishes its `autoarm` role, so a recorded identity that no longer matches the pid holding the lock proves abandonment on its own even while the entry still reads `arming` or no ledger entry exists at all. +That covers a claim whose process group was killed before it could record any outcome and whose pid the operating system later handed to an unrelated live process. +A claim carrying no recorded identity keeps the ledger-only boundary, and a failed reclaim re-blocks rather than allowing a blind stop. Fresh `failed` and `failed-suppressed` outcomes enter or advance the failure progression instead of acting as unconditional recovery proof. The auto-arm itself rechecks the healthy watcher predicate and retries a bounded number of times before reporting a genuine failure. The first fresh exhausted-failure epoch preserves its handoff without consuming a blocked-stop count, while later fresh failed epochs advance the same monotonic progression instead of resetting it. @@ -146,7 +153,7 @@ That warning uses `bin/fm-supervision-instructions.sh --repair-line`, so it alwa ## Regression coverage -`tests/fm-turnend-guard.test.sh` covers the predicate, main and secondmate primary scope, child-worktree exclusion, `FM_HOME` and `FM_STATE_OVERRIDE` precedence, the live-lock and fresh-beacon guard predicate, the cooperative `--claude` claim wait, monotonic failed-epoch progression, bounded attended fail-open, post-alarm continuation suppression, positive recovery reset, Pi logical-run latching, missing-`jq` behavior, all five primary registrations, Grok native and legacy selection, typed field precedence, malformed input, and exactly-one-path safety. +`tests/fm-turnend-guard.test.sh` covers the predicate, main and secondmate primary scope, child-worktree exclusion, `FM_HOME` and `FM_STATE_OVERRIDE` precedence, the live-lock and fresh-beacon guard predicate, the cooperative `--claude` claim wait, monotonic failed-epoch progression, bounded attended fail-open, post-alarm continuation suppression, positive recovery reset, the abandoned auto-arm claim cases that must block or clear instead of allowing a blind stop, Pi logical-run latching, missing-`jq` behavior, all five primary registrations, Grok native and legacy selection, typed field precedence, malformed input, and exactly-one-path safety. `tests/fm-guard-stale-banner.test.sh` covers the pull-guard predicate, including the persistent-model fresh-leftover-beacon negative control, the auto-arm model's healthy fresh-beacon-without-a-watcher case and stale-beacon alarm, and the extension model's live-watcher path, ownership-qualified fresh hand-off, held-lock failures, independently broken ownership signals, stale-beacon alarm, queued-wake warning, and Pi and pi-signed harness routing. It also covers true-reason banner wording and reason-keyed episode dedup surviving a beacon mtime change. `tests/fm-cursor-primary.test.sh` covers the Cursor park end to end over real processes with no harness installed: each tracked Claude-shaped entrypoint standing down on a Cursor payload, both follow-up sources, the bounded repair nag and its reset, the nested loop bounds, supersession, away-mode and lock-ownership inertness, child-worktree exclusion, and that the adapter never exits 2. diff --git a/docs/verification/muse.md b/docs/verification/muse.md index bc7ffe64ba0..11d7e3454b3 100644 --- a/docs/verification/muse.md +++ b/docs/verification/muse.md @@ -48,7 +48,7 @@ $ grep -nE 'muse-bin|exec ' launcher.sh `ps -o comm= -p <pid>` returns the full executable path, whose basename is `muse-bin-<version>`. That is why both `bin/fm-harness.sh` and `bin/backends/tmux.sh` match the anchored prefix `muse-bin-*` rather than an exact name, and why neither can rely on an install-path component: `~/.local/bin/muse-bin-<version>` contains no `muse` path component. -The Muse launch clears `CLAUDECODE`, `PI_CODING_AGENT`, `GROK_AGENT`, and `FM_PI_HARNESS` before the worker starts so foreign primary markers cannot override the versioned ancestry. +The Muse launch clears `CLAUDECODE`, `PI_CODING_AGENT`, `GROK_AGENT`, `FM_PI_HARNESS`, `CURSOR_AGENT`, and `CURSOR_INVOKED_AS` before the worker starts so foreign primary markers cannot override the versioned ancestry. [`runtime-backends.md`](runtime-backends.md#agent-liveness-name-sources) owns the resulting tmux liveness verdict and its relationship to the portable decoy regression. diff --git a/docs/verification/process-event-sources.md b/docs/verification/process-event-sources.md index 6570c79ac17..8102d057ac7 100644 --- a/docs/verification/process-event-sources.md +++ b/docs/verification/process-event-sources.md @@ -83,7 +83,7 @@ Exercised by `tests/fm-procevent.test.sh` against a fake blocking source whose c | proactive-delivery crash and drain boundaries | dotted and underscored source ids at the same sequence receive distinct markers; a concurrent drain cannot consume between queue revalidation and marker commit; failed output, failed marker commit, and a crash before marker commit leave replay available, while successful output still ends the actionable cycle and a crash after marker commit suppresses a duplicate | | adapter-owned terminal verdict | two fixture adapters - one that ends on any result, one with no terminal knowledge - decide the outcome alone: the first has its registration and claim retired automatically after one capture and is never restarted, the second stays armed | | adapter-owned application of a captured result | a remote-secondmate reply captured through the real relay in an isolated home reaches that secondmate's local status mirror, settles its correlated pending-reply expectation, re-arms the next cursor-anchored source, and is acknowledged, with no handler step or duplicate `check` wake; its new mirrored bytes remain visible to the watcher's signal gate, while a cursor-loss whole-log recapture that adds no bytes is acknowledged quietly; for an already-escalated request, the same path closes the exact decision so the open-decision fold clears and remains clear; a capture whose adapter application fails because local storage for a referenced remote document is obstructed is left unacknowledged and receives the fallback `check` wake, and the handler's own `handle` still applies it in full after storage recovers | -| generic keyed-answer feed | `tests/fm-decision-hold-lifecycle.test.sh` drives a single-origin bound source through the real runner with a fixture adapter, then drives a cross-origin source through the real runner and Lavish adapter interface, proving both reach the one keyed-answer intake: named holds close at capture time, routed-work and absent holds skip, malformed or over-limit keys feed nothing, replays are idempotent, an unbound source closes nothing, and capture remains independent of the handler wake. | +| generic keyed-answer feed | `tests/fm-captain-hold-lifecycle.test.sh` drives a bound source through the real runner with a fixture adapter that only prints keyed lines, proving any bound channel reaches the one keyed-answer intake: named captain-held tasks close at capture time, a card-declared release mode frees held work, keys naming no captain-held task skip, freeform prose forges nothing, matching answer-and-mode replays are idempotent while mode mismatches refuse, an unbound source closes nothing, and capture remains independent of the handler wake. | | terminal retirement preserves the result | the retired source's captured output, its announced event, its handled acknowledgement, and later explicit `retire` all still behave normally | | registration-generation retirement | an old terminal runner preserves a concurrently replaced registration and releases ownership so the replacement runs independently; injected registration-removal failure retains a terminal claim, performs no second poll, and completes idempotently once removal recovers | | one `Send & End`, one result | an armed Lavish source driven against a stand-in for the published poll, which delivers the final `session_ended` feedback once and empty ended sessions afterward, polls exactly once, captures exactly one result, publishes one distinct event, and retires itself | diff --git a/docs/verification/public-followup.md b/docs/verification/public-followup.md index 3bad5a605de..373bee35966 100644 --- a/docs/verification/public-followup.md +++ b/docs/verification/public-followup.md @@ -2,17 +2,18 @@ Audience: maintainer verification. -This record supports two active guarantees for promised public replies made through the myfirstmate relay: +This record supports three active guarantees for promised public replies made through the myfirstmate relay: 1. A promised final reply survives compaction and restart, reconciles from disk alone, and lands in the original thread exactly once. 2. A home that never opted into the relay pays nothing for any of it. +3. Delivering a final does not close the public loop: the registration is retained as `state=delivered` until `retire --reason`, session start surfaces an `open-loop` line, and `rechain` can bind follow-on work to the same thread. [`docs/configuration.md`](../configuration.md#promised-public-replies-statepublic-followup) owns the operator-facing contract, [`docs/architecture.md`](../architecture.md#optional-relay) owns the mechanism boundary, and `tasks-axi public-followup --help` owns the typed obligation schema. Task chronology and delivery evidence stay outside this record. ## Environment -Recorded 2026-07-30 on Darwin 25.5.0 (arm64) with GNU bash 5.3.9, tasks-axi 0.2.3, jq 1.8.1, and ShellCheck 0.11.0 (the version `bin/fm-lint.sh` pins). +Recorded 2026-08-21 on Darwin 25.5.0 (arm64) with GNU bash 5.3.9, tasks-axi 0.2.5, jq 1.8.1, and ShellCheck 0.11.0 (the version `bin/fm-lint.sh` pins). The relay is a fakebin `curl` in every case, so no public post is ever made; `tasks-axi` and `jq` are the real tools, because stubbing the obligation state machine would verify nothing. ## Restart end-to-end and regressions @@ -27,9 +28,25 @@ ok - restart end-to-end: typed result reconciles from disk and delivers one repl ok - duplicate terminal results, restart replay, and repeated delivery are all no-ops ok - wrong source, wrong work id, stale generation, malformed, unsupported deliverable, and forged identity are all refused ok - a relay transport failure is held as retryable with no false completion, and the retry posts once +ok - a dry-run records no public delivery and leaves the commitment retryable ok - a late success receipt closes the exact attempt with no second post, and a mismatched attempt is refused +ok - typed terminal cleanup clears the legacy link without posting ok - a delivery interrupted between post and receipt refuses to repost ok - a child home reports typed results but can never become the outward-post owner +ok - typed delivery refuses to post when its cleanup registration is missing +ok - marked secondmate teardown resolves its parent and fails closed when unavailable +ok - local seeding publishes durable parent state before its identity marker +ok - a lost launch-time parent binding is recovered from the durable local record +ok - a durable local parent record does not bypass a genuinely missing parent-side registration +ok - unknown durable parent fields remain forward-compatible +ok - conflicting live and durable parent bindings fail closed +ok - unsafe durable parent records fail closed before cleanup +ok - a NUL-bearing durable parent record fails closed before cleanup +ok - relay-disabled unmarked teardown runs no public-followup work +ok - a marked child proceeds without tasks-axi when its parent relay is disabled +ok - secondmate parent resolution matches the durable registry id literally +ok - traversal-shaped registrations are rejected before path construction or posting +ok - pending keeps registrations when tasks-axi returns malformed JSON ok - the retained private request context keeps the original thread deliverable after inbox cleanup ok - cleanup refuses while a public reply is owed and proceeds once it has landed ok - a relay-disabled home runs no tasks-axi call, prints nothing, and gains no artifact @@ -38,20 +55,39 @@ ok - a relay-exhausted follow-up binding is escalated rather than retried into t ok - the relay poll stays inert without a token, silent with no commitments, and surfaces a new result once ok - startup surfaces unresolved public commitments only in a relay home that owes one ok - typed public-followup records carry only public-safe summaries and deliverables +ok - dropped-baton regression: delivery retains the loop and pending prints open-loop +ok - CONTROL: the identical teardown REFUSES the moment a commitment is registered +ok - rechain posts the shipped follow-on into the same thread +ok - rechain resumes the same obligation after an interrupted bind +ok - concurrent rechains cannot fork one delivered source +ok - failed rechain retirement keeps the source claimed by one resumable destination +ok - registration replay preserves delivered and retired loop states +ok - redelivery does not report a retired loop as open +ok - retire closes delivered loops after secondmate home removal +ok - retire fails closed for an unbound existing secondmate +ok - retire fails closed when a secondmate ID is reassigned +ok - rechain refuses an unrelated existing destination +ok - pending skips a registration retired during settlement +ok - retire --reason closes the loop and drops the open-loop line +ok - retention creates no false teardown refusal and pending no longer prunes +ok - expiry escalation is pinned by FMX_NOW_OVERRIDE +ok - brief fails explicitly when typed deliverable keys are unavailable +ok - pre-change registrations are open loops and un-rechainable, never a crash +ok - teardown reports an unreconciled legacy Relay link +ok - secondmate promotion matches teardown parent resolution ``` -The first case is the end-to-end proof. +The restart case is the end-to-end proof of guarantee 1. It reproduces the stranded state first (work bound, no reconciled terminal result, delivery refused with "still waiting on its bound work" and zero posts), then has a secondmate-shaped child report a typed `pr-merged` result, deletes the drained inbox payload, reconciles from disk, and asserts exactly one `connector/followup` call carrying the original `request_id`, a validated `posted` receipt, and a Done obligation. -The existing Relay suite is unchanged by this work: +The dropped-baton case is the end-to-end proof of guarantee 3. +It delivers a `report-ready` promised-final, asserts the registration is retained and `pending` prints `open-loop`, then shows that an unbound follow-on ship is not teardown-refused (the one-variable control still refuses the moment a commitment is registered for that work). +`rechain` then binds a fresh `pr-merged` obligation onto the same request/thread, and a second follow-up carries the shipped text. +`retire --reason` records its private receipt before removal and is the only close; replayed registration cannot reopen that retired loop. +The concurrency and interrupted-bind cases verify that one delivered source cannot fork and that retry converges on the same destination obligation. +A pre-change on-disk record (no `state=`, no `request_context_b64`) is an open loop and un-rechainable rather than a crash. -```sh -bash tests/fm-x-mode.test.sh | grep -c '^ok -' -``` - -``` -103 -``` +The existing Relay mention suite (`tests/fm-x-mode.test.sh`) is unchanged by this work. ## Relay-disabled zero overhead @@ -66,10 +102,10 @@ for i in $(seq 1 1000); do fm_pf_relay_active "$HOME_DIR" || true; done ``` ``` -total_ns=69694000 per_call_us=69 +total_ns=22305959 per_call_us=22 ``` -Roughly 0.07 ms per session start, from a single `[ -f "$FM_HOME/.env" ]` test that returns false before anything else runs. +Roughly 0.02 ms per session start, from a single `[ -f "$FM_HOME/.env" ]` test that returns false before anything else runs. ## Compatibility axes reviewed @@ -79,4 +115,5 @@ The only supervision surfaces touched are the session-start digest, which `bin/f Runtime backends (tmux, herdr, zellij, orca, cmux): not applicable after inspection. No command here reads `state/<id>.meta`'s backend fields, resolves an endpoint, or captures a pane. -The one lifecycle integration is `bin/fm-teardown.sh`'s refusal, which runs before any backend command and keys only on the task id, so it behaves identically on every backend. +The lifecycle integrations are backlog-handoff warnings, promotion rechain hints, and `bin/fm-teardown.sh`'s owed-reply refusal plus non-blocking open-loop and legacy `x_request=` warnings. +They inspect home, task, parent-binding, and registration records rather than backend fields or endpoints, so they behave identically on every backend. diff --git a/docs/verification/supervision.md b/docs/verification/supervision.md index 119c365b5ae..f8270aae500 100644 --- a/docs/verification/supervision.md +++ b/docs/verification/supervision.md @@ -470,6 +470,21 @@ Observed guarantee: after ordinary `session_shutdown` for `/new`, `/resume`, and Stale prior-generation tool callbacks could not mutate the active child, repeated transitions kept exactly one live arm cycle, and terminal `quit` still refused late rearm. Plain Pi and pi-signed share the same tracked `.pi/extensions/fm-primary-pi-watch.ts` path, so both inherit the generation owner; other primary harnesses are not applicable because they do not use this Pi extension lifecycle. +The once-per-generation recovery bound and immediate handling-successor poll were verified on 2026-08-21 with the tracked Pi extension, real watcher processes, and an isolated home. +The regression forced handling confirmation to fail, observed one recovery follow-up across the former repeat window, confirmed the successor remained live, and then proved a separate handling successor durably queued a crew event within the bounded poll window. + +```sh +bin/fm-test-run.sh tests/fm-watch-recovery-loop.test.sh +``` + +Observed output: + +```text +ok - a resurfacing handling successor stays alive and supervises instead of going blind +ok - unacknowledged recovery is announced at most once per generation and the successor stays alive +FM_TEST_SUMMARY total=1 failed=0 skipped_gate=0 duration_ms=59357 +``` + Deterministic entry points: ```sh @@ -477,6 +492,7 @@ tests/fm-pi-watch-extension.test.sh tests/fm-pi-primary-types.test.sh tests/fm-watcher-lock.test.sh tests/fm-watch-arm.test.sh +tests/fm-watch-recovery-loop.test.sh tests/fm-wake-queue.test.sh tests/fm-subagent-pretool-check.test.sh tests/fm-claude-stop-autoarm.test.sh diff --git a/docs/voice-relay.md b/docs/voice-relay.md new file mode 100644 index 00000000000..4cf95ee1019 --- /dev/null +++ b/docs/voice-relay.md @@ -0,0 +1,295 @@ +# The spoken interface + +Talk to a voice agent that sits in front of the first mate. It answers questions +about what is happening from the first mate's own records, and when you ask for +real work it says so out loud and queues the request rather than pretending to +do it. + +This is step one of three: a spoken round trip that works. Interrupting the agent +mid-sentence and carrying context from one question to the next are step three, +and [what this build does not do](#what-this-build-does-not-do) is explicit about +where the edge is. + +## The shape + +Your laptop captures the audio and plays the reply. This desktop holds the +conversation with the model. Nothing in between needs AWS credentials on the +laptop, which is the whole reason for this shape. + +``` +laptop this desktop AWS +------ ------------ --- +microphone --> fm-voice-client.py --(ssh)--> fm-voice-relay.py --> Nova Sonic 2 +speaker <------------------------------------------------- (your region) + | + +--> the first mate's records (read) + +--> fm-inbox.sh note (queue real work) +``` + +The two ends share one bidirectional byte stream over an SSH exec channel, so +audio and control travel together and need framing. `bin/fm_voice_frame.py` is +the owner of that format and is the only file both machines run. + +The relay reads records and queues work. It never changes a project, and the +queueing half is `bin/fm-inbox.sh note`, the same surface the captain's own +out-of-band capture already uses, rather than a second queue. + +## What it costs in time + +Measured on 2026-08-21 against the reviewed relay code, `amazon.nova-2-sonic-v1:0` in `eu-north-1`, on a spoken question that makes the agent read the records before it can answer, which is the slowest ordinary case. +Six runs each, all six answered each way. + +| Path | First audio out, seconds | Median | +| --- | --- | --- | +| Direct from this desktop, no relay | 1.165 1.190 1.215 1.250 1.281 1.352 | 1.232 | +| Over the relay, real client and framing | 1.138 1.165 1.171 1.174 1.177 1.283 | 1.172 | + +The clock starts the instant the captain stops speaking and stops when the first byte of reply audio arrives. +An earlier measurement of the same question, on the same model and region and also reading the records, put the direct path at 1.164 seconds median over five runs, and this control reproduces it to within the noise floor below. +That measurement is not published here, so read it as corroboration rather than as something to open: the direct column stands as a control on its own, because it was taken in the same pass, on the same clip, model, region and read scope, with only the relay removed. + +**The relay's own cost is smaller than this measurement can resolve.** +The relay median lands below the direct control, which does not mean the relay is faster: two direct-control passes twenty minutes apart differ by 0.070 seconds of median, so that is the floor, and framing and the extra process hop are both under it. +The earlier measurement above independently agrees on that floor, spreading 0.087 seconds across its own five runs, and two measurements agreeing on the noise are worth more than one asserting it. +Read the two rows as the same number. + +The first pass, on the relay as first written, put it 0.22 seconds behind the control, and that gap read as framing, the process hop and the per-turn reconnect. +It was none of them, and the difference is worth keeping, because a wrong number invites a re-measurement while a wrong cause invites a fix to the wrong part of the relay. +Each relay run is six turns in one session, so a per-turn defect shows up as a step: that pass stepped from 1.229 on turn one to a 1.447 median across turns two to six, and the same step appeared independently on the talk-end-to-tool-request mark, 0.599 rising to 0.730. +The re-measured passes are flat, stepping 0.009 and 0.021. +The 0.22 seconds was the relay resolving AWS credentials again for every turn's session, which review found and fixed: `Credentials` in `bin/fm-voice-relay.py` resolves once, and every later session reuses that answer, so a reconnect costs a reconnect. +This is the second time credential resolution has dominated a voice path's latency on a host like this one, because earlier prototype work measured the local credential helper at about a second per call and found that fixed per-call overhead exceeded the model's own cost. +So it is the first thing to suspect when a spoken path is slower than the model, and it is worth checking that anything new doing per-turn work resolves credentials once rather than once per session. + +What the relay figure does NOT include, and could not be measured from here: + +- **The SSH hop itself.** + These runs drove the relay as a local child process, which is the identical relay command with only the `ssh -T <host>` prefix omitted, so the client, the framing, the uplink ordering, the relay, the records read and the handover are all real and only the SSH subprocess is absent. + Two facts bound what its absence can be hiding. + A constant transport cost cannot produce the turn-by-turn step that the credential defect produced, and the first pass, which did run over `ssh localhost`, put its own first turn 0.009 seconds above its own direct control. + Neither of those is a measurement of the SSH path on this code, and neither is offered as one. +- **Your laptop's round trip to this desktop.** + Add roughly your own round trip time: the audio goes up and the reply comes back, so it lands about once. +- **Microphone capture and speaker output latency.** + This desktop has no microphone and no speaker, so every measurement used audio files. + The client reports both device figures in its own output, so your first live run measures them rather than guessing. + +So your number is about 1.15 to 1.3 seconds plus your round trip time plus your audio devices. +It is worth saying plainly that this came in under the bottom of the 1.5 to 2.5 second estimate the relay shape was given before it was built. +The safer shape, with no credentials on the laptop, is not the slower one. + +## Setting up this desktop + +The model is only reachable over HTTP/2 bidirectional streaming, which the AWS +CLI cannot drive and `boto3` cannot either. It needs the experimental SDK, in a +virtual environment of its own: + +``` +python3 -m venv ~/.fm-voice-venv +~/.fm-voice-venv/bin/pip install aws-sdk-bedrock-runtime +``` + +Then tell this home which account and model to use. +The relay carries no default for any of these, because a region, a model id and an AWS profile name somebody's account and somebody's choices, and inheriting those from whoever wrote the code is not a sensible way to start talking to a paid API. +Each value is one line in your gitignored `config/` directory, and each has an environment variable that overrides it for a single run. + +| File | Environment | Holds | +| --- | --- | --- | +| `config/voice-region` | `FM_VOICE_REGION` | The Bedrock region to open the session in, required. | +| `config/voice-model` | `FM_VOICE_MODEL` | The Nova Sonic model id, required. | +| `config/voice-profile` | `FM_VOICE_PROFILE` | The AWS profile to export credentials from, optional: with no profile the relay uses only credentials that are already in its environment. | +| `config/voice-id` | `FM_VOICE_ID` | The output voice, optional and `matthew` when unset. | + +A missing required value refuses with the path to write, so an unconfigured home cannot start the relay by accident, and that configuration is the whole opt-in. +`docs/configuration.md` is the registry for these files. + +Check it end to end without a microphone, using a recorded question: + +``` +cd <your firstmate home> +~/.fm-voice-venv/bin/python bin/fm-voice-relay.py --self-test <clip.pcm> +``` + +The clip is headerless 16000 Hz mono signed 16-bit little-endian PCM and must end +on speech, not silence. It prints one JSON line: what it heard, what it said, how +long each stage took, whether it answered at all, and, in `relay_error`, what +broke when a turn broke rather than merely going unanswered, so an +infrastructure failure is not read as a slow answer. Feed it a clip that +already ends in silence and it will tell you the timings are measured from the +wrong instant rather than printing a number that looks fast. + +## Setting up the laptop + +**The audio devices are not verified.** No worker can reach the captain's laptop, so neither the microphone nor the speaker has ever been opened. +Treat the first live run as their test, and expect the device setup to be where it fails. +Everything around them is exercised with files. +That includes the speaker's own byte accounting, the arithmetic deciding which turn a chunk of reply audio is credited to and whose first-audio clock it stamps, which runs against a stub stream in the test suite. +Covering that arithmetic says nothing about how a real output device behaves. + +Copy the two files the laptop needs, and install the one dependency: + +``` +scp <desktop>:<firstmate home>/bin/fm-voice-client.py . +scp <desktop>:<firstmate home>/bin/fm_voice_frame.py . +python3 -m pip install sounddevice +``` + +`sounddevice` needs PortAudio, which on macOS is `brew install portaudio`. macOS +will ask for microphone permission for whichever terminal you run this from, once. + +Then talk: + +``` +python3 fm-voice-client.py --host <desktop> \ + --relay <firstmate home>/bin/fm-voice-relay.py \ + --relay-python ~/.fm-voice-venv/bin/python +``` + +The client has no built-in idea of where the relay lives on your desktop, so `--relay` is required and `FM_VOICE_RELAY` sets it once for a shell. + +Press Enter to start talking, press Enter again when you have finished. It prints +the timings for each turn as JSON on stdout and everything human on stderr, so +`--runs 5 > runs.jsonl` gives you your own spread to compare against the table +above. + +Every record carries `relay_error`, which is null when nothing broke and otherwise names what did. +Where this end is left to infer what happened, it tells the two mid-turn failures apart, because they are not the same fault: a turn that got no reply audio at all says the connection ended, or was lost, or the relay stopped, or the session ended, before that turn was answered, while a turn whose answer had already started playing says the same thing happened before the reply finished. +The second still reads `answered: true`, because sound did reach you and `first_audio_s` is a real measurement of when. +Two other shapes carry neither clause, so do not read the pair above as the whole list: a fault the relay names itself arrives as the relay's own words, which point at the desktop and are kept unaltered because it knows what this end can only guess at. +A reason opening `this end could not handle the relay's reply` is the one that points at your laptop instead, so a healthy relay is not where to look for it. + +The exit code is non-zero if any turn went unanswered, if any record carries a `relay_error`, or if the session stopped before it had taken the runs you asked for. +A truncated answer therefore fails the run rather than passing it, so a spread computed from `runs.jsonl` cannot quietly average an infrastructure failure into a latency figure. + +If the audio devices are not the ones you want, `--input-device` and `--output-device` take a name or an index. +Neither the client nor this guide can yet tell you which device it resolved, so an unexpected device is diagnosed by trying the other name or index rather than by reading a log line. +If it fails before any audio, add `--verbose` and look for the handshake: a chatty login shell on the desktop printing to stdout is the one failure that looks like a protocol error and is not. + +## What it may read + +An unconfigured home gets the narrow scope: counts of what is in flight, what is waiting on the captain and what is open for review, with no identifier, title or link assembled at all. +Widening that is one line the captain of those records writes into `config/voice-read-scope` themselves. +Two whole classes of record are excluded at every scope, and excluded by construction rather than filtered on the way out: + +- **Finished work in the backlog's done history**, because a spoken "what is + happening" answer is about open work, and old engagements accumulate there. +- **Free-form note bodies**, because they are written for someone with the whole + file in front of them, and they are where commercial detail gets quoted. + +Only open work and this home's own runtime records are ever assembled. +A task keeps its runtime record until teardown, so the count of workers on deck +and the states beside it still include one whose item is already done; both are a +number and a state word, never anything written in a record. +Verified against the captain's live records on 2026-08-21: every occurrence of +the one customer identifier those records contain sits in finished work or a note +body, so nothing a status answer can say names a customer. +`tests/fm-voice-relay.test.sh` holds that boundary as an executable check, so +widening the reader later fails a test instead of quietly widening what is sent. + +Two settings control it, both optional and both in `config/`: + +| File | Effect | +| --- | --- | +| `voice-read-scope` | `counts` (the default, and what an absent file means) sends counts only, with no record free text assembled at all. `full` sends counts plus the names, titles and pull request links of open work. | +| `voice-read-deny` | One plain case-insensitive substring per line; `#` comments. Each open item is matched once, against its identifier, its title, its tag values and its pull request link together, and a match is withheld from every list it could have appeared in and reduced to a count, so the agent still says how much is waiting without saying what it is. An absent file means an empty list. | + +`voice-read-deny` exists so that one future open item carrying a customer name +can be excluded in a single line rather than by turning the feature off. + +The wider scope is not free. Measured on 2026-08-21 on the same question, on the +relay as first written, so compare the two sides with each other rather than with +the table above: the wide answer is 2872 bytes against 445, and it costs both time +and consistency, at 1.348, 1.866 and 2.273 seconds against 1.351, 1.299 and 1.376. +If the spoken answer only ever needs to be "three jobs running, two decisions +waiting", `counts` is faster and steadier as well as narrower. + +An unreadable or misspelled `voice-read-scope` refuses rather than falling back +to the wider setting, because falling back would widen what is sent on the +strength of a typo. + +## Push to talk, and the setting that refuses + +Push to talk is the default: the microphone is closed until you ask for it. That +is `$0.0101` per minute against `$0.0151` for an open microphone, and it is the +setting nobody has decided yet, so this build does not choose the expensive one +on the captain's behalf. + +`--listen open-mic` exists as a setting and refuses at startup today. +An open microphone needs something to decide when you stopped speaking, and the client has no end-of-speech detection, so the mode would open a turn, stream audio forever and never mark a boundary, which leaves the relay appending to a session that has already answered. +That detection belongs with carrying context across turns, which is step three, so the flag refuses before it opens an SSH connection or spends anything rather than half working. +The setting stays where it is so that turning it on later is a small change rather than a new flag. + +## One turn per session, and what that gives up + +The relay reconnects to the model at the start of each turn. That is not +tidiness, it is a measured requirement. + +A second question inside a session that has already answered one is treated as an +interruption, unconditionally: the model raises it the instant the audio block +opens. Waiting does not help. Six consecutive turns were tried with no wait, with +a wait until all the reply audio had arrived, and with a wait of the reply's full +spoken duration on top of that. Every one interrupted every second turn. Worse, +an interrupted turn that needs to read the records is lost outright: the model +asks for the records, takes them, and then never answers at all. + +Reconnecting costs 0.02 seconds and happens while the captain is pressing the +talk key rather than while they are waiting for a reply, so it is invisible. With +it, six turns in a row all answered. + +The same path covers a session the model ends on its own, mid-conversation: that +costs the turn it was in and not the relay, and the next talk key builds a +replacement. Either way the client hears about it at once rather than waiting out +the whole reply timeout in silence. +A turn still waiting for its answer when either happens names why in its own `relay_error`, and [setting up the laptop](#setting-up-the-laptop) describes those reasons. + +**What it gives up is memory.** Every question starts fresh, so "and what about +that one" will not work. Carrying context across turns means handling +interruption properly, which is step three. + +## Two traps worth keeping + +Both cost real time to find the first time. The code comments own the detail; +these are the shapes. + +1. **The end of a reply is not the event that says the reply ended.** The obvious + completion event never arrives on its own. The real end is the content-end + event carrying an end-of-turn reason. +2. **A clip with no trailing silence is never answered.** The model truncates it + and waits forever. The relay appends 400 ms of silence. Measured, this is a + content requirement and not a timing one: 0 ms and 100 ms were never answered, + while 200, 300, 400 and 800 ms all answered inside the same spread, because the + padding is sent as fast as the socket takes it. 400 ms is free margin above the + floor where answers start. + +## What this build does not do + +- **Interrupting the agent mid-sentence.** Nova Sonic supports it, measured, on + both model versions, so the capability is there when it is wanted. The concrete + thing step three has to solve is the interruption finding above: today any + second question in a session is treated as an interruption, and an interrupted + turn that reads the records produces no answer at all. +- **Remembering the last question.** See above. +- **Doing any project work.** Real work is queued for the first mate and the + agent says so out loud. It has no tool that changes a project. + +## Cost + +`$0.00293` per exchange, derived from the first pass's token counts and session seconds, which is roughly a dollar for three hundred and forty questions. +The re-measured exchange is about a quarter of a second shorter, worth about `$0.00004` at the session rate below, so the figure is unchanged at the precision it is quoted to. +Push to talk is `$0.0101` per minute of session against `$0.0151` with an open microphone. + +Text in and out is materially dearer on this model version than the one it +replaces, so a long system prompt or a large record answer is a real cost as well +as a real delay. That is the second reason the reader caps its lists rather than +sending every row. + +## Owners + +| Concern | Owner | +| --- | --- | +| Wire format between the two machines | `bin/fm_voice_frame.py` | +| The relay, the model session, the tools | `bin/fm-voice-relay.py` | +| The laptop end, capture and playback | `bin/fm-voice-client.py` | +| What may be read, and queueing real work | `bin/fm_voice_records.py` | +| The queue the handover writes to | `bin/fm-inbox.sh` | +| The boundary as an executable check | `tests/fm-voice-relay.test.sh` | diff --git a/docs/watcher-continuity.md b/docs/watcher-continuity.md index 1a94ec0edef..cf458e2640d 100644 --- a/docs/watcher-continuity.md +++ b/docs/watcher-continuity.md @@ -23,6 +23,8 @@ While supervision is still needed and away mode remains inactive, an actionable ## Actionable wake ordering After an actionable Pi or OpenCode child close, the adapter starts and verifies one singleton successor before it delivers the original wake. +It confirms the handling handoff against that successor before scheduling the follow-up, retries once against the current generation and successor, and treats a failed confirmation as a restoration failure: it classifies the error, retires a successor that is no longer alive, and surfaces exactly one typed message. +A failed confirmation is never swallowed. It waits at most one readiness timeout per attempt, then sends TERM and waits a bounded retirement confirmation before the next lock-verified exponential retry. If the unready arm does not retire within that bound, the adapter keeps ownership, starts no overlapping retry, and delivers the typed fallback immediately. When that retained arm later closes, its actual close is classified as a new supervised event without replaying the earlier fallback. @@ -30,9 +32,9 @@ After the configured retry bound is exhausted, it delivers the original wake wit This is deliberate Option B ordering: the fleet is protected before the model handles the wake whenever restoration succeeds, but the model is never left blind when it does not. Claude's Stop hook starts the successor arm at the next Stop after the handling turn, rather than before notification as Pi and OpenCode do. -The durable wake queue preserves actionable events during the residual active-turn window, and the bounded turn-end guard enforces recovery at Stop when no watcher or auto-arm claim is present. -For every supported arm path, a successor that observes an accepted down stretch emits `check: rearm-resurface` through the ordinary durable handling path before settling into its live wait. -That recovery presentation includes all unacknowledged queue rows, the cursor-folded OPEN DECISIONS set, and still-unread informational status lines, so a still-open decision or a buried `note:` answer reappears even when recovery has no queue row of its own. +The durable wake queue preserves actionable events during the residual active-turn window, and the bounded turn-end guard enforces recovery at Stop when no watcher is live and no auto-arm claim is still deciding, so a leftover claim whose own decision already finished cannot suppress it ([`turnend-guard.md`](turnend-guard.md#harness-integrations) owns that boundary). +The recovery-episode contract below owns once-per-generation announcement. +A handling successor does not re-announce; it enters its poll loop immediately and keeps scanning signals, stale panes, and checks. The model no longer re-arms after ordinary wakes. No PreToolUse hook denies fleet commands based on watcher status. A genuine auto-arm failure describes the automatic mechanism as broken and never directs a routine manual background arm. @@ -46,7 +48,9 @@ The turn-end guard remains the final backstop rather than the normal continuity ## Recovery episode acknowledgement A recovery episode is one generation of `state/.watcher-down`, and it is retired only by the generation-bound acknowledgement the drain prints as `WAKE_ACK_REQUIRED`. -Every watcher close and every durable queue append publishes downtime, so a downtime republication of any pending episode reuses its generation instead of minting a new one. +An unacknowledged downtime generation is announced at most once: the first recovery marks that generation announced, and later arms wait until a new down stretch mints a new generation. +A non-successor watcher start after an announced-but-unacked episode is a new down stretch and mints a fresh generation so buried decisions still resurface once. +Every watcher close and every durable queue append publishes downtime, so a downtime republication of any pending episode reuses its generation instead of minting a new one, and an already-announced generation stays announced. That reuse keeps a watcher close inside the handling window from orphaning the acknowledgement already presented and trapping later arms in repeated recovery presentation. An acknowledgement carries two separable facts: queue-row consumption is bound to the monotonic `--ack-through` sequence, while only retiring the episode is bound to `--recovery-generation`. A generation mismatch therefore does not block consumption of rows through that sequence; it is a non-fatal result that names its own remedy - re-drain, then acknowledge the newer episode. @@ -78,11 +82,13 @@ Only the watcher process touches `state/.last-watcher-beat`; no helper process c `tests/fm-pi-watch-extension.test.sh` checks Pi's first-cycle-or-explicit-repair tool metadata and ownership-based redundant-call no-ops, then simulates actionable and empty child closes against the actual Pi and OpenCode close handlers, blocks prompt delivery to prove the successor launches first, verifies single-flight behavior, changes the session lock before close to prove ownership is rechecked, and hangs each successor arm to prove bounded fallback delivery includes the typed restoration failure. The same suite covers ordinary same-process session replacement for `/new`, `/resume`, and `/fork`, same-instance shutdown-plus-start, stale prior-generation callbacks, repeated transitions with exactly one live cycle, disappearance of the shutting-down refusal after a valid replacement activates, and terminal quit still refusing late rearm. `tests/fm-watch-arm.test.sh` covers durable queue replay, real remote parent-replies ingestion into the authoritative status log, decision-only OPEN DECISIONS recovery, interrupted handling replay, generation-bound acknowledgement, a persistent live successor after recovery, a watcher close inside the handling window that must leave the printed acknowledgement valid, and the self-healing moved-generation acknowledgement that consumes its handled rows and names its remedy. +`tests/fm-watch-recovery-loop.test.sh` covers the once-per-generation announcement bound with the real Pi extension against a refused handling handshake, and a handling successor that must surface a real crew event instead of going blind. `tests/fm-watcher-lock.test.sh` covers verified-successor attach, recovery publication before stale-lock removal, the typed self-eviction failure, bounded and successor-linked lifecycle rows, and a SIGSTOP counterfactual that distinguishes a live PID from a stale beacon before classifying termination. `tests/fm-subagent-pretool-check.test.sh` proves Claude retains only the non-status Bash seatbelts. `tests/fm-claude-stop-autoarm.test.sh` covers the auto-arm's scope, stale and live session owners, unchanged AFK and need boundaries, single-flight, bounded failure retries, benign live-watcher cycle ends, one-notice failure episodes, and exit-2 translation. +It also covers abandoned single-flight claims: a claim the ledger shows already finished, and one whose recorded pid-identity no longer matches its live pid while the ledger still reads arming or is absent entirely, are both reclaimed so a lapsed home re-arms, while an identity-matched claim still arming, one the ledger does not name, and the guard's own terminal check keep the gate closed ([`turnend-guard.md`](turnend-guard.md) owns that boundary). `FM_CLAUDE_LIVE_E2E=1 tests/fm-claude-stop-autoarm-live-e2e.test.sh` starts with the reproduced stale-lock state, runs session start first, completes two tokenless cycles, and checks the competing-live-owner negative control. -`tests/fm-turnend-guard.test.sh` covers the cooperative `--claude` guard, including monotonic failed-epoch progression, the integrated bounded fail-open, post-alarm continuation suppression, and positive recovery reset. +`tests/fm-turnend-guard.test.sh` covers the cooperative `--claude` guard, including monotonic failed-epoch progression, the integrated bounded fail-open, post-alarm continuation suppression, and positive recovery reset; [`turnend-guard.md`](turnend-guard.md#regression-coverage) lists that suite's full coverage, including the abandoned-claim cases. ## Active limits and verification diff --git a/skills/stow/SKILL.md b/skills/stow/SKILL.md index 95522b37ed5..b45ebe8fd9b 100644 --- a/skills/stow/SKILL.md +++ b/skills/stow/SKILL.md @@ -93,6 +93,8 @@ Markers are compact trailing HTML comments, deliberately cheap because marker by - `<!--a:YYYY-MM-DD-->` - an `aging` entry; the embedded date is its last-reinforced date. - `<!--p:YYYY-MM-DD-->` - a `perishable` entry; the embedded date is its last-reinforced date. +- `<!--a:YYYY-MM-DD/N-->` - only in a file whose header pointer opts in to the pass horizon below: either dated marker may carry `/N`, the number of passes that evaluated the entry without reinforcing it. + An absent `/N` means zero, so an entry you keep exercising costs no counter bytes at all, and a file that has not opted in never writes one. - `<!--P-->` - an explicitly `pinned` entry in a file whose default tier is not `pinned`. - `<!--g-->` - migration-only: an unconfirmed legacy entry that has consumed its one grace cycle, carrying no date because grace is not reinforcement. @@ -100,6 +102,7 @@ Markers are compact trailing HTML comments, deliberately cheap because marker by - The staging deploy needs the VPN profile active or the smoke test hangs. <!--a:2026-08-03--> - CI is red on the flaky auth test until the pinned runner image updates (tracked in TODO). <!--p:2026-07-20--> - Always run the schema linter before touching migrations. <!--P--> +- The staging seed script must run before the fixture import. <!--a:2026-07-28/6--> ``` The tier names say what this skill does with an entry: @@ -114,14 +117,21 @@ Rules: - Unless a file's own header pointer names a different default, a user-level memory file defaults to `pinned`, while a project memory file and `.stow-notes.md` default to `aging`. - An entry matching its file's `pinned` default carries no marker at all; every `aging` and `perishable` entry always carries its dated marker, whose letter names the tier, so a clock-carrying entry is never ambiguous with unmarked legacy material. - Marker and pointer bytes are part of the file's cost, so bookkeeping stays minimal by design. -- Every governed memory file this skill curates carries at most a one-line header pointer naming this skill as the scheme owner, such as `<!-- memory tiers: see the stow skill -->`, optionally naming that file's default tier when it deviates. - The tier semantics, marker spellings, and clocks live only in this skill and are never restated in a file header. +- Every governed memory file this skill curates carries at most a one-line header pointer naming this skill as the scheme owner, such as `<!-- memory tiers: see the stow skill -->`, optionally naming that file's default tier when it deviates and the pass horizon when that file opts in, as in `<!-- memory tiers: see the stow skill; pass horizon -->`. + The tier semantics, marker spellings, and clocks live only in this skill and are never restated in a file header, which names an option but never its numbers. During one-time migration, add the pointer even to a default-pinned file that contains only unmarked entries, so every governed file names its scheme owner. - Refresh an entry's last-reinforced date only on real evidence from the current session: the fact was used, confirmed, or re-derived. Mere presence in the file is not evidence, and re-reading memory is never reinforcement. +- The dates above are the default and only clock, and a file gets exactly them unless its header pointer opts in to the pass horizon. + Opt a file in where you stow often enough that the date clock never fires: admitting findings is a per-pass event, so an entry you keep exercising never sits unreinforced for 30 wall-clock days and the file only grows, while a project you stow rarely already passes its date horizon in a single pass and gains nothing. + Never add that opt-in on your own initiative; the user chooses it, one file at a time. +- While a file is opted in, an `aging` entry there is stale at whichever comes first - 10 passes that evaluated it without reinforcing it, or 30 days - and a `perishable` entry at whichever comes first - 3 unreinforced passes, or 7 days. + Increment the counter of every dated entry that pass did not reinforce before judging staleness, read a dated marker with no `/N` as counter zero so nothing needs migrating, and clear the counter only by refreshing the date on real evidence. + In a file that is not opted in, never write a counter and never read one that is already there; preserve any existing `/N` byte-for-byte instead of normalizing or removing it. - Re-confirm a stale `perishable` entry against its named condition: still open means refresh the date, while resolved, expired, or no longer checkable means archive it now. - Decay is evaluated only when this skill runs; nothing happens between passes, so an infrequently stowed project experiences the clocks at its stow interval. - Stale never means deleted: a stale entry moves to a `.stow-archive.md` in the source file's own directory, never loaded by any session, and its archive record includes the source filename, tier, reinforcement date when present, and a one-line reason. + Include the unreinforced-pass counter only when the pass horizon itself made the entry stale, using the exact reason `unreinforced <N>p`; omit the counter when the wall-clock horizon or any other reason caused archival, even if the active marker carried one. In a git worktree, verify that this archive path is not already tracked in the index before writing any archived fact there. If it is tracked, do not write to it and report that archival is blocked until the user chooses a safe destination. Otherwise add a `.stow-archive.md` line to a `.gitignore` file in the archive's directory, and never write archived facts into a git-tracked file. diff --git a/tests/fm-afk-return.test.sh b/tests/fm-afk-return.test.sh index 537b1bff977..c345e4f55e2 100755 --- a/tests/fm-afk-return.test.sh +++ b/tests/fm-afk-return.test.sh @@ -19,6 +19,9 @@ install_runner() { # <case-dir> cp "$ROOT/bin/fm-afk-return.sh" "$dir/bin/" cp "$ROOT/bin/fm-wake-lib.sh" "$dir/bin/" cp "$ROOT/bin/fm-classify-lib.sh" "$dir/bin/" + # fm-timeout-lib.sh: the shared hard bound fm-classify-lib.sh sources for the + # wedge detector's bounded worktree write probe. + cp "$ROOT/bin/fm-timeout-lib.sh" "$dir/bin/" cat > "$dir/bin/fm-afk-launch.sh" <<'SH' #!/usr/bin/env bash [ "${1:-}" = stop ] || exit 2 diff --git a/tests/fm-ask-user-authority.test.sh b/tests/fm-ask-user-authority.test.sh index 469eb92c2a2..7b6e185a00e 100644 --- a/tests/fm-ask-user-authority.test.sh +++ b/tests/fm-ask-user-authority.test.sh @@ -18,7 +18,7 @@ test_primary_and_secondmate_instruction_generation() { ship="$home/data/authority-worker/brief.md" assert_grep 'ask-user findings are never yours to answer' "$ship" \ "generated implementation brief lets the worker own an ask-user decision" - assert_grep "Firstmate applies the authority contract in its \`AGENTS.md\`" "$ship" \ + assert_grep "Firstmate applies \`ask-user-authority\` and obtains any required captain decision" "$ship" \ "generated implementation brief bypasses the primary authority owner" assert_grep "silently bypass firstmate's authority check and any required captain escalation" "$ship" \ "generated implementation brief permits silent ask-user auto-resolution" diff --git a/tests/fm-backend-herdr-presentation-e2e.test.sh b/tests/fm-backend-herdr-presentation-e2e.test.sh index 39b0e13b517..ac62abb4275 100755 --- a/tests/fm-backend-herdr-presentation-e2e.test.sh +++ b/tests/fm-backend-herdr-presentation-e2e.test.sh @@ -421,6 +421,15 @@ teardown_task() { # <id> <home> "$ROOT/bin/fm-teardown.sh" "$id" --force } +finish_concurrent_teardown() { # <id> <status> <stdout> <stderr> + local id=$1 status=$2 out=$3 err=$4 + [ "$status" -ne 0 ] || return 0 + grep -F "session presentation lock is contended" "$err" >/dev/null 2>&1 \ + || fail "projected teardown $id failed unexpectedly: $(cat "$err")" + teardown_task "$id" "$HOME_DIR" > "$out" 2> "$err" \ + || fail "projected teardown $id retry failed after presentation cleanup completed: $(cat "$err")" +} + normalize_meta() { # <meta> sed -E \ -e 's|^window=.*$|window=<herdr-container-id>|' \ @@ -886,8 +895,10 @@ teardown_task order-a "$HOME_DIR" > "$TMP_ROOT/order-a-teardown.out" 2> "$TMP_RO ORDER_A_TEARDOWN_PID=$! teardown_task order-b "$HOME_DIR" > "$TMP_ROOT/order-b-teardown.out" 2> "$TMP_ROOT/order-b-teardown.err" & ORDER_B_TEARDOWN_PID=$! -wait "$ORDER_A_TEARDOWN_PID" || fail "projected ordering fixture A teardown failed" -wait "$ORDER_B_TEARDOWN_PID" || fail "projected ordering fixture B teardown failed" +if wait "$ORDER_A_TEARDOWN_PID"; then ORDER_A_TEARDOWN_STATUS=0; else ORDER_A_TEARDOWN_STATUS=$?; fi +if wait "$ORDER_B_TEARDOWN_PID"; then ORDER_B_TEARDOWN_STATUS=0; else ORDER_B_TEARDOWN_STATUS=$?; fi +finish_concurrent_teardown order-a "$ORDER_A_TEARDOWN_STATUS" "$TMP_ROOT/order-a-teardown.out" "$TMP_ROOT/order-a-teardown.err" +finish_concurrent_teardown order-b "$ORDER_B_TEARDOWN_STATUS" "$TMP_ROOT/order-b-teardown.out" "$TMP_ROOT/order-b-teardown.err" assert_focus_is "$CAPTAIN_FOCUS" "concurrent projected teardowns" teardown_task order-fail "$HOME_DIR" > "$TMP_ROOT/order-fail-teardown.out" 2> "$TMP_ROOT/order-fail-teardown.err" \ || fail "projected ordering failure fixture teardown failed" @@ -928,8 +939,10 @@ for ROUND in 1 2 3; do WAVE_A_TEARDOWN_PID=$! teardown_task "focus-$ROUND-b" "$HOME_DIR" > "$TMP_ROOT/focus-$ROUND-b-teardown.out" 2> "$TMP_ROOT/focus-$ROUND-b-teardown.err" & WAVE_B_TEARDOWN_PID=$! - wait "$WAVE_A_TEARDOWN_PID" || fail "focus wave $ROUND teardown A failed" - wait "$WAVE_B_TEARDOWN_PID" || fail "focus wave $ROUND teardown B failed" + if wait "$WAVE_A_TEARDOWN_PID"; then WAVE_A_TEARDOWN_STATUS=0; else WAVE_A_TEARDOWN_STATUS=$?; fi + if wait "$WAVE_B_TEARDOWN_PID"; then WAVE_B_TEARDOWN_STATUS=0; else WAVE_B_TEARDOWN_STATUS=$?; fi + finish_concurrent_teardown "focus-$ROUND-a" "$WAVE_A_TEARDOWN_STATUS" "$TMP_ROOT/focus-$ROUND-a-teardown.out" "$TMP_ROOT/focus-$ROUND-a-teardown.err" + finish_concurrent_teardown "focus-$ROUND-b" "$WAVE_B_TEARDOWN_STATUS" "$TMP_ROOT/focus-$ROUND-b-teardown.out" "$TMP_ROOT/focus-$ROUND-b-teardown.err" assert_focus_is "$CAPTAIN_FOCUS" "focus wave $ROUND concurrent teardowns" WAVE_REMAINING=$(lab workspace list | jq -r '.result.workspaces[].label') [ "$WAVE_REMAINING" = $'firstmate\n2ndmate-alpha\n2ndmate-bravo' ] \ diff --git a/tests/fm-backend-herdr.test.sh b/tests/fm-backend-herdr.test.sh index dc1be58f9c5..23be2bd1a7d 100755 --- a/tests/fm-backend-herdr.test.sh +++ b/tests/fm-backend-herdr.test.sh @@ -3083,6 +3083,28 @@ test_composer_state_pi_separator_idle_is_empty() { pass "fm_backend_herdr_composer_state: a native idle Pi separator composer reads empty" } +# A pi worker parked on an interactive prompt (permission dialog, question +# menu, trust dialog) reports agent_status=blocked: it is waiting on a human +# keystroke. The menu is drawn ABOVE the separator pair, so the composer region +# itself is blank and structure alone looks like a free composer. Typing there +# does not compose a message - the menu consumes the keys and Enter selects the +# highlighted default, so the text is discarded and a decision nobody made is +# recorded (issue #2797). Every "is it safe to type here?" consumer reads this +# verdict: the away-mode injection guard (bin/fm-supervise-daemon.sh) and +# fm-send's pre-type refusal both proceed ONLY on an affirmative `empty`. +test_composer_state_pi_parked_prompt_is_not_empty() { + local dir log resp fb out + dir="$TMP_ROOT/composer-pi-parked-prompt"; mkdir -p "$dir/responses"; log="$dir/log"; resp="$dir/responses"; : > "$log" + printf 'Should I keep going?\n 1. Yes, continue\n\x1b[7m 2. Stop, do not act\x1b[0m\n\x1b[0m\x1b[38;2;129;162;190m─────────────────────────────────────────────────────\x1b[0m\n\x1b[0m\x1b[7m \x1b[0m \n\x1b[0m\x1b[38;2;129;162;190m─────────────────────────────────────────────────────\x1b[0m\n' > "$resp/1.out" + printf '{"result":{"agent":{"agent":"pi","agent_status":"blocked"}}}\n' > "$resp/2.out" + fb=$(make_herdr_fakebin "$dir") + out=$( PATH="$fb:$PATH" FM_HERDR_LOG="$log" FM_HERDR_RESPONSES="$resp" \ + bash -c '. "$0/bin/backends/herdr.sh"; fm_backend_herdr_composer_state lab:w1:p2' "$ROOT" ) + [ "$out" != empty ] \ + || fail "a pi pane parked on a prompt must not report an affirmatively empty composer, got '$out'" + pass "fm_backend_herdr_composer_state: a blocked pi pane parked on a prompt is not an empty composer" +} + test_composer_state_pi_separator_real_text_is_pending() { local dir log resp fb out dir="$TMP_ROOT/composer-pi-separated-pending"; mkdir -p "$dir/responses"; log="$dir/log"; resp="$dir/responses"; : > "$log" @@ -4520,6 +4542,7 @@ test_composer_state_real_text_is_pending test_composer_state_popup_placeholder_fill_is_pending test_composer_state_unknown_on_capture_failure test_composer_state_unknown_when_no_composer_row_found +test_composer_state_pi_parked_prompt_is_not_empty test_composer_state_pi_separator_idle_is_empty test_composer_state_pi_separator_real_text_is_pending test_composer_state_pi_incomplete_separator_below_stale_generic_is_unknown diff --git a/tests/fm-bearings-board.test.sh b/tests/fm-bearings-board.test.sh index 1bf22ae88b8..a0807d87a7e 100644 --- a/tests/fm-bearings-board.test.sh +++ b/tests/fm-bearings-board.test.sh @@ -200,7 +200,7 @@ test_build_injects_binds_then_arms() { out=$(run_board "$home" build "$data") || fail "a valid payload did not build" assert_contains "$out" "board: $board" "build did not report the board path: $out" assert_contains "$out" "served: $board" "build did not establish the Lavish session: $out" - assert_contains "$out" "(any-origin)" "build did not report the any-origin binding: $out" + assert_contains "$out" "bound: " "build did not report the answer binding: $out" assert_contains "$out" "armed: " "the first build did not arm the board source: $out" assert_present "$board" "build reported success without a board" diff --git a/tests/fm-bearings-snapshot.test.sh b/tests/fm-bearings-snapshot.test.sh index e955608fdb8..3ff7722216c 100755 --- a/tests/fm-bearings-snapshot.test.sh +++ b/tests/fm-bearings-snapshot.test.sh @@ -970,6 +970,52 @@ test_superseded_queued_item_dropped_by_default() { pass "superseded queued items are dropped by default and restored with --all-queued" } +# The collapsed captain-call contract: any due, unblocked captain-held task is +# Captain's Call whatever its kind; a date-deferred hold is a dated gate until +# due; a prose-deferred hold leaves the default views with a disclosure; and +# Recently Landed excludes only what closed while still held for the captain. +test_collapsed_captain_call_deferral_and_landed() { + local home fakebin json + home=$(make_home collapsed-call) + mkdir -p "$home/data" + cat > "$home/data/backlog.md" <<'EOF' +## In flight + +## Queued +- [ ] work-gate - Captain-gated ship work (repo: firstmate) (kind: ship) (hold: captain go needed) (hold-kind: captain) +- [ ] later-call - Deferred captain call (repo: firstmate) (kind: captain) (hold: revisit with the captain) (hold-kind: captain) (hold-until: 2026-08-01) +- [ ] due-call - Due captain call (repo: firstmate) (kind: captain) (hold: overdue captain choice) (hold-kind: captain) (hold-until: 2026-07-11) +- [ ] parked-call - Prose-parked captain call (repo: firstmate) (kind: ship) (hold: DEFERRED by captain revisit later) (hold-kind: captain) +- [ ] external-gate - Externally held work (repo: firstmate) (kind: ship) (hold: upstream release pending) (hold-kind: external) + +## Done +- [x] answered-call - Answered captain question (repo: firstmate) (kind: captain) (done 2026-07-10) (hold: captain choice pending) (hold-kind: captain) +- [x] shipped-work - Ordinary landed work (repo: firstmate) (kind: ship) (merged 2026-07-10) +EOF + fakebin=$(make_fakebin "$home") + json=$(run "$home" "$fakebin" --json) + printf '%s' "$json" | jq -e ' + (.decisions_open | any(.[]; .id == "work-gate")) + and (.decisions_open | any(.[]; .id == "due-call")) + and (.decisions_open | any(.[]; .id == "later-call") | not) + and (.decisions_open | any(.[]; .id == "parked-call") | not) + and (.decisions_open | any(.[]; .id == "external-gate") | not) + and (.gates | any(.[]; .id == "later-call" and (.reason | startswith("until 2026-08-01")))) + and (.gates | any(.[]; .id == "work-gate") | not) + and (.gates | any(.[]; .id == "parked-call") | not) + and (.gates | any(.[]; .id == "external-gate")) + and (.landed | any(.[]; .id == "shipped-work")) + and (.landed | any(.[]; .id == "answered-call") | not) + and (.omitted | any(.[]; .surface | startswith("captain holds marked deferred"))) + ' >/dev/null || fail "the collapsed captain-call projection is wrong: $json" + json=$(run "$home" "$fakebin" --json --all-decisions --all-queued) + printf '%s' "$json" | jq -e ' + (.decisions_open | any(.[]; .id == "parked-call")) + and (.gates | any(.[]; .id == "parked-call") | not) + ' >/dev/null || fail "--all-decisions must reveal the prose-deferred call: $json" + pass "captain-held tasks of any kind reach Captain's Call, deferral is honored, and landed excludes answered calls" +} + test_include_prs_is_the_only_fetch_path() { local home fakebin json home=$(make_home prs); write_fixture "$home" @@ -1932,6 +1978,7 @@ test_include_prs_is_the_only_fetch_path test_partial_github_failure_degrades test_perl_fallback_bounds_github_call test_section_caps_and_expansion_flags +test_collapsed_captain_call_deferral_and_landed test_pr_repository_cap_and_expansion test_per_repository_pr_cap_is_disclosed test_projection_and_toon_fail_closed diff --git a/tests/fm-brief.test.sh b/tests/fm-brief.test.sh index c5ee3d00f05..a4342d758f0 100755 --- a/tests/fm-brief.test.sh +++ b/tests/fm-brief.test.sh @@ -267,7 +267,7 @@ test_ship_mode_is_explicit_not_registry() { pass "fm-brief.sh: the explicit ship mode wins over the registered posture" } -# yolo is firstmate's approval authority and never reaches the worker, and a scout +# yolo is firstmate's merge authority and never reaches the worker, and a scout # or charter carries no delivery contract. Each must refuse rather than accept and # discard the flag, which would look recorded but change nothing. test_delivery_flags_are_refused_where_they_do_not_apply() { @@ -439,6 +439,41 @@ test_herdr_lab_omission_is_loud_for_ship_and_scout() { pass "fm-brief.sh: ship and scout scaffolds make omitted Herdr intent fail-visible" } +# Regression (issue #2575): AGENTS.md section 11 and this script's own help tell +# firstmate to replace EVERY `{TASK}` placeholder. The unguarded Herdr gate used +# to quote `{TASK}` in its own prose, so that documented global replace spliced +# the whole task body into the middle of the gate's sentence - silently +# destroying the one contract that exists precisely because the scaffold cannot +# see the task text. The placeholder must exist only at the genuine fill site, +# so the documented fill leaves the gate intact and the body appears once. +test_documented_global_replace_leaves_the_herdr_gate_intact() { + local home id brief kind count content filled body + home="$TMP_ROOT/task-fill-site-home" + mkdir -p "$home/data" + body='Restart the herdr session, then profile it' + for kind in ship scout; do + id="brief-fill-site-$kind" + if [ "$kind" = scout ]; then + FM_HOME="$home" "$ROOT/bin/fm-brief.sh" "$id" firstmate --scout >/dev/null 2>&1 + else + FM_HOME="$home" "$ROOT/bin/fm-brief.sh" "$id" firstmate --mode no-mistakes >/dev/null 2>&1 + fi + brief="$home/data/$id/brief.md" + assert_present "$brief" "$kind brief was not scaffolded" + count=$(grep -c -F '{TASK}' "$brief") + [ "$count" = 1 ] \ + || fail "$kind brief must carry exactly one {TASK} fill site, found $count" + content=$(cat "$brief") + filled=${content//'{TASK}'/$body} + count=$(printf '%s\n' "$filled" | grep -c -F "$body") + [ "$count" = 1 ] \ + || fail "$kind brief: the documented global {TASK} replace duplicated the task body $count times" + printf '%s\n' "$filled" | grep -qF 'this scaffold cannot inspect the task text' \ + || fail "$kind brief: the Herdr safety gate did not survive the documented global replace" + done + pass "fm-brief.sh: the documented {TASK} fill cannot corrupt the Herdr safety gate" +} + test_secondmate_no_projects_charter() { local home brief status home="$TMP_ROOT/no-projects-home" @@ -678,15 +713,15 @@ test_scout_and_secondmate_load_decision_hold_policy() { FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ "$ROOT/bin/fm-brief.sh" sample-investigation sample --scout >/dev/null 2>&1 scout="$home/data/sample-investigation/brief.md" - assert_grep "$ROOT/.agents/skills/decision-hold-lifecycle/SKILL.md" "$scout" \ - "scout brief did not load the unresolved-decision policy before done" + assert_grep "$ROOT/.agents/skills/captain-hold-lifecycle/SKILL.md" "$scout" \ + "scout brief did not load the captain-call policy before done" assert_grep "pass its shared completion gate for the report and any visual review" "$scout" \ "scout brief did not cross-reference visual-review completion" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" FM_SECONDMATE_CHARTER='sample reviews' \ "$ROOT/bin/fm-brief.sh" sample-mate --secondmate --no-projects >/dev/null 2>&1 charter="$home/data/sample-mate/brief.md" - assert_grep "load \`decision-hold-lifecycle\`" "$charter" \ - "secondmate charter did not load the shared decision policy for detailed investigations" + assert_grep "load \`captain-hold-lifecycle\`" "$charter" \ + "secondmate charter did not load the shared captain-call policy for detailed investigations" pass "fm-brief.sh: investigation and visual-review completions load the shared decision policy" } @@ -725,6 +760,7 @@ test_ship_project_memory_wording test_herdr_lab_contract_is_explicit_and_complete test_herdr_lab_contract_quotes_foreign_firstmate_path test_herdr_lab_omission_is_loud_for_ship_and_scout +test_documented_global_replace_leaves_the_herdr_gate_intact test_herdr_lab_contract_applies_to_scouts_but_not_secondmates test_secondmate_no_projects_charter test_secondmate_marked_request_reporting_contract diff --git a/tests/fm-busy-state.test.sh b/tests/fm-busy-state.test.sh index b86c0108bed..e295871fa20 100755 --- a/tests/fm-busy-state.test.sh +++ b/tests/fm-busy-state.test.sh @@ -107,6 +107,64 @@ test_retire_serializes_and_rejects_stale_gen() { pass "retire waits for the writer lock and cannot remove a new incarnation" } +# Regression for issue #2625: the writer lock's stale-lock branch resolved the +# lock's mtime with `stat -f %m ... || stat -c %Y ...`. On GNU coreutils `-f` is +# *filesystem* stat, so it consumes the format string as a path, complains on +# stderr, prints " File: ..." on stdout, and still exits 0 - the GNU form in the +# fallback never ran. The following `$((now - mtime))` then evaluated the word +# `File`, which under `set -u` aborted the writer with "File: unbound variable". +# fm-teardown.sh died there after returning the worktree, leaving state/<id>.meta +# and friends behind to generate stale wakes forever, and every re-run died +# identically because the abandoned lock directory was never broken. +# +# The stat and uname stubs make this deterministic on any host: the writer must +# take the Linux path and still break a provably stale lock. +test_stale_lock_broken_under_gnu_stat() { + local state gen fakebin real_uname out status + state=$(new_state_dir gnu-stat-lock) + gen=$("$EV" arm "$state" t1) + fakebin=$(fm_fakebin "$TMP_ROOT/gnu-stat-lock") + real_uname=$(command -v uname) + + # GNU coreutils semantics, self-contained so no real stat is consulted. + cat > "$fakebin/stat" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = -c ] && [ "${2:-}" = %Y ]; then + printf '%s\n' 1000000000 # long-abandoned lock + exit 0 +fi +if [ "${1:-}" = -f ]; then + echo "stat: cannot read file system information for '$2': No such file or directory" >&2 + shift 2 + printf ' File: "%s"\n' "${1:-}" + exit 0 +fi +exit 1 +SH + chmod +x "$fakebin/stat" + cat > "$fakebin/uname" <<SH +#!/usr/bin/env bash +if [ \$# -eq 0 ]; then printf 'Linux\n'; exit 0; fi +exec "$real_uname" "\$@" +SH + chmod +x "$fakebin/uname" + + mkdir "$state/t1.busy-state.lock" + out=$(PATH="$fakebin:$PATH" "$EV" retire "$state" t1 --gen "$gen" 2>&1) && status=0 || status=$? + case "$out" in + *'unbound variable'*) fail "the writer still dies on GNU stat output: $out" ;; + esac + [ "$status" = 0 ] || fail "retire did not break a provably stale writer lock: $out" + [ ! -e "$state/t1.busy-state" ] || fail "retire left the record behind" + [ ! -e "$state/t1.busy-gen" ] || fail "retire left the gen sidecar behind" + [ ! -e "$state/t1.busy-state.lock" ] || fail "retire left the stale lock behind" + + # Teardown must be able to run again over the same task without failing. + PATH="$fakebin:$PATH" "$EV" retire "$state" t1 --current-gen \ + || fail "a repeated retire over already-cleaned state was not idempotent" + pass "the writer breaks a stale lock instead of dying on GNU stat output" +} + test_retire_missing_sidecar_is_idempotent() { local state gen state=$(new_state_dir retire-missing) @@ -387,6 +445,7 @@ test_apply_current_gen_reset test_apply_unarmed_refused test_retire_serializes_and_rejects_stale_gen test_retire_missing_sidecar_is_idempotent +test_stale_lock_broken_under_gnu_stat test_stale_gen_event_rejected test_stale_gen_record_unknown test_missing_record_unknown_not_idle diff --git a/tests/fm-captain-hold-lifecycle.test.sh b/tests/fm-captain-hold-lifecycle.test.sh new file mode 100755 index 00000000000..d5f8d2a527f --- /dev/null +++ b/tests/fm-captain-hold-lifecycle.test.sh @@ -0,0 +1,1184 @@ +#!/usr/bin/env bash +# End-to-end tests for captain-held tasks: the one primitive behind "a decision +# is simply a task waiting on the captain", its completion gate, its recorded +# answers, the record-divergence guard over its two records, and the legacy +# compatibility for pre-collapse decision identities. +set -u + +# shellcheck source=tests/lib.sh +# shellcheck disable=SC1091 +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +TEARDOWN="$ROOT/bin/fm-teardown.sh" +BEARINGS="$ROOT/bin/fm-bearings-snapshot.sh" +TMP_ROOT=$(fm_test_tmproot fm-captain-hold) +TASKS_AXI_BIN=$(command -v tasks-axi || true) + +command -v jq >/dev/null 2>&1 || { echo "skip: jq not found"; exit 0; } +command -v tasks-axi >/dev/null 2>&1 || { echo "skip: tasks-axi not found"; exit 0; } + +make_home() { # <name> + local home="$TMP_ROOT/$1" fakebin + mkdir -p "$home/data" "$home/state" "$home/config" "$home/projects" + cp "$ROOT/.tasks.toml" "$home/.tasks.toml" + cat > "$home/data/backlog.md" <<'EOF' +## In flight + +## Queued + +## Done +EOF + fakebin=$(fm_fakebin "$home") + fm_fake_exit0 "$fakebin" tmux treehouse no-mistakes gh gh-axi + printf '%s\n' "$home" +} + +# The Lavish review adapter, run against this suite's isolated home. The +# machine-wide process-event claim root is redirected into the fixture so arming +# a review here can never contend with a real one on this machine. +run_lavish() { # <home> <command args...> + local home=$1 + shift + PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_PROCEVENT_CLAIM_ROOT="$home/procevent-claims" \ + "$ROOT/bin/fm-procevent-lavish.sh" "$@" +} + +run_bearings() { # <home> + local home=$1 + PATH="$home/fakebin:$PATH" FM_HOME="$home" FM_BEARINGS_NOW=2026-07-14T12:00:00Z \ + "$BEARINGS" --json +} + +run_teardown() { # <home> <id> + local home=$1 id=$2 + PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_CONFIG_OVERRIDE="$home/config" "$TEARDOWN" "$id" +} + +tasks_in() { # <home> <tasks-axi args...> + local home=$1 + shift + (cd "$home" && tasks-axi "$@") +} + +run_captain() { # <home> <command args...> + local home=$1 + shift + PATH="$home/fakebin:$PATH" REAL_TASKS_AXI="$TASKS_AXI_BIN" \ + FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_CONFIG_OVERRIDE="$home/config" "$ROOT/bin/fm-captain-hold.sh" "$@" +} + +# The retired command surface, kept for one release as a shim; in-flight +# pre-collapse work still drives the lifecycle through these spellings. +run_shim() { # <home> <command args...> + local home=$1 + shift + PATH="$home/fakebin:$PATH" REAL_TASKS_AXI="$TASKS_AXI_BIN" \ + FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_CONFIG_OVERRIDE="$home/config" "$ROOT/bin/fm-decision-hold.sh" "$@" +} + +write_origin_meta() { # <home> <id> [kind] + local home=$1 id=$2 kind=${3:-scout} + fm_write_meta "$home/state/$id.meta" \ + "window=firstmate:fm-$id" \ + "worktree=$home/projects/missing-$id" \ + "project=$home/projects/sample" \ + "harness=codex" \ + "kind=$kind" \ + "mode=$kind" +} + +# Reproduces the loss exactly with privacy-safe synthetic names: the investigation +# and visual review have ended, the only genuine unresolved captain call is report +# prose, no held backlog item or open status exists, and the authoritative +# Bearings view correctly omits it. Completion must now refuse before teardown can +# erase the source. +test_uninventoried_report_decision_refuses_completion() { + local home id json rc + home=$(make_home omitted-decision) + id=sample-route-review + mkdir -p "$home/data/$id" + cat > "$home/data/backlog.md" <<EOF +## In flight +- [ ] $id - Investigate sample routing (repo: sample) (kind: scout) (since 2026-07-14) + +## Queued + +## Done +EOF + write_origin_meta "$home" "$id" + printf 'done: report and visual review complete\n' > "$home/state/$id.status" + cat > "$home/data/$id/report.md" <<'EOF' +# Sample route review + +The evidence is complete. +The captain still needs to choose route north or route south before follow-up work starts. +EOF + + json=$(run_bearings "$home") || fail "Bearings failed for unresolved-call regression" + printf '%s' "$json" | jq -e ' + (.decisions_open | length) == 0 + and (.gates | length) == 0 + and (.reports | any(.id == "sample-route-review")) + ' >/dev/null || fail "the pre-policy omission shape was not reproduced: $json" + + set +e + run_teardown "$home" "$id" > "$home/teardown.out" 2> "$home/teardown.err" + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "completed investigation teardown erased a report-only unresolved captain call" + assert_present "$home/state/$id.meta" "refused completion must preserve investigation metadata" + assert_grep "REFUSED" "$home/teardown.err" "refusal must be explicit" + pass "report-only unresolved captain call is reproduced and completion refuses before loss" +} + +# The completion gate on the collapsed primitive: an origin with open keyed +# status decisions refuses --none, refuses an inventory naming absent tasks, +# attests a verified inventory of captain-held task ids, and transfers every +# still-open status decision to that durable inventory. +test_completion_gate_attests_and_transfers() { + local home id json open before after + home=$(make_home completion-gate) + id=sample-systems-review + mkdir -p "$home/data/$id" + tasks_in "$home" add "$id" "Investigate sample systems" --kind scout --repo sample --start >/dev/null \ + || fail "could not create investigation backlog fixture" + write_origin_meta "$home" "$id" + cat > "$home/state/$id.status" <<'EOF' +working: report drafted +needs-decision [key=route]: choose route north or route south +needs-decision [key=access]: choose open or restricted sample access +EOF + cat > "$home/data/$id/report.md" <<'EOF' +# Sample systems review + +Two choices remain unresolved: the route and the sample access level. +A separate recommendation is already resolved and requires no captain action. +EOF + + if run_captain "$home" complete "$id" --none > "$home/none.out" 2> "$home/none.err"; then + fail "--none attested while captain calls were still open in the status stream" + fi + assert_no_grep "decisions_reviewed=1" "$home/state/$id.meta" \ + "failed completion recorded a false completion attestation" + if run_captain "$home" complete "$id" sample-route-call > "$home/absent.out" 2> "$home/absent.err"; then + fail "completion accepted an inventory entry that names no task" + fi + + run_captain "$home" hold sample-route-call \ + --title "Choose route: north, south" --reason "captain route and access choices pending" \ + --repo sample --origin "$id" >/dev/null \ + || fail "could not register the captain-held task" + run_captain "$home" hold sample-route-call \ + --title "Choose route: north, south" --reason "captain route and access choices pending" \ + --repo sample >/dev/null \ + || fail "idempotent hold retry failed" + [ "$(grep -cE "^- \[ \] sample-route-call -" "$home/data/backlog.md")" = 1 ] \ + || fail "idempotent retry duplicated the captain-held task" + if run_captain "$home" hold sample-route-call --title "A different title" \ + --reason "captain route and access choices pending" > "$home/title.out" 2> "$home/title.err"; then + fail "hold accepted a changed title on an existing task" + fi + + FM_STATE_OVERRIDE="$home/state" bash -c ' + . "$1" + sig=$(fm_wake_signal_sig "$3") || exit 1 + printf "%s" "$sig" > "$(fm_wake_signal_seen_path "$2" "$3")" + ' _ "$ROOT/bin/fm-wake-lib.sh" "$home/state" "$home/state/$id.status" \ + || fail "could not prime the announced decision baseline" + run_captain "$home" complete "$id" sample-route-call >/dev/null \ + || fail "shared investigation completion gate failed" + FM_STATE_OVERRIDE="$home/state" bash -c ' + . "$1"; fm_wake_signal_seen_current "$2" "$3" + ' _ "$ROOT/bin/fm-wake-lib.sh" "$home/state" "$home/state/$id.status" \ + || fail "captain-held bookkeeping closes re-woke their own home" + assert_grep "decisions_reviewed=1" "$home/state/$id.meta" "completion attestation missing" + assert_grep "decision_keys=sample-route-call" "$home/state/$id.meta" "inventory was not recorded as task ids" + open=$(bash -c '. "$1"; status_open_decisions "$2"' _ \ + "$ROOT/bin/fm-classify-lib.sh" "$home/state/$id.status") + [ -z "$open" ] || fail "captain-held transfer did not close the live status decisions: $open" + grep -F 'captain-held [key=route]: tracked by sample-route-call' "$home/state/$id.status" >/dev/null \ + || fail "the transfer line does not name the tracking inventory" + + before=$(shasum -a 256 "$home/data/backlog.md" | awk '{print $1}') + json=$(run_bearings "$home") || fail "Bearings failed with a captain-held task" + after=$(shasum -a 256 "$home/data/backlog.md" | awk '{print $1}') + [ "$before" = "$after" ] || fail "Bearings mutated the authoritative backlog" + printf '%s' "$json" | jq -e ' + (.decisions_open | any(.id == "sample-route-call" and .verb == "captain-hold" and .owner == "(main)")) + and (.gates | any(.id == "sample-route-call") | not) + ' >/dev/null || fail "Bearings did not surface the captain-held task: $json" + + run_teardown "$home" "$id" >/dev/null 2> "$home/teardown.err" \ + || fail "reviewed investigation teardown failed: $(cat "$home/teardown.err")" + tasks_in "$home" "done" "$id" --report "data/$id/report.md" --keep 0 >/dev/null \ + || fail "could not archive completed investigation" + json=$(run_bearings "$home") || fail "Bearings failed after source teardown and archival" + printf '%s' "$json" | jq -e ' + (.decisions_open | any(.id == "sample-route-call" and .verb == "captain-hold")) + and (.in_flight | any(.id == "sample-systems-review") | not) + ' >/dev/null || fail "teardown or archival erased a captain-held task: $json" + pass "the completion gate attests captain-held inventory and transfers open status decisions" +} + +# The recorded-answer rule: answering closes with the captain's exact words, an +# exact retry is idempotent, a drifted retry is rejected, dependent work routed +# behind the answered task is released by the close, and the completion gate is +# satisfied only by a recorded answer. +test_answer_records_and_closes() { + local home id json show + home=$(make_home answer-close) + id=sample-guard-review + mkdir -p "$home/data/$id" + tasks_in "$home" add "$id" "Guard the answer path" --kind scout --repo sample --start >/dev/null \ + || fail "could not create the answer-guard origin" + write_origin_meta "$home" "$id" + printf 'done: report complete\n' > "$home/state/$id.status" + printf '# Guard review\n\nOne captain choice remains.\n' > "$home/data/$id/report.md" + run_captain "$home" hold sample-guard-call \ + --title "Choose the guard option" --reason "captain guard choice pending" --repo sample >/dev/null \ + || fail "could not register the captain-held task" + run_captain "$home" complete "$id" sample-guard-call >/dev/null \ + || fail "completion failed for the held inventory" + tasks_in "$home" add sample-guard-work "Apply the guard option" \ + --kind ship --repo sample --blocked-by sample-guard-call >/dev/null \ + || fail "could not route work behind the captain-held task" + + printf '' > "$home/empty.txt" + if run_captain "$home" answer sample-guard-call --decision-file "$home/empty.txt" \ + > "$home/empty-answer.out" 2> "$home/empty-answer.err"; then + fail "answer accepted an empty captain decision" + fi + if run_captain "$home" answer sample-guard-call > "$home/bare-answer.out" 2> "$home/bare-answer.err"; then + fail "answer accepted a close with no captain decision file at all" + fi + printf 'An answer the captain never gave.\n' > "$home/invented.txt" + if run_captain "$home" answer sample-absent-call --decision-file "$home/invented.txt" \ + > "$home/absent-answer.out" 2> "$home/absent-answer.err"; then + fail "answer invented a resolution for a task that does not exist" + fi + if run_captain "$home" answer sample-guard-work --decision-file "$home/invented.txt" \ + > "$home/unheld-answer.out" 2> "$home/unheld-answer.err"; then + fail "answer closed a task that is not held for the captain" + fi + show=$(tasks_in "$home" show sample-guard-call --full) + assert_contains "$show" "state: queued" "a refused answer closed the captain-held task" + assert_contains "$show" "held: yes" "a refused answer released the captain-held task" + + printf 'Captain chose the guard option.\n' > "$home/guard-decision.txt" + run_captain "$home" answer sample-guard-call --decision-file "$home/guard-decision.txt" >/dev/null \ + || fail "answer could not close the captain-held task" + show=$(tasks_in "$home" show sample-guard-call --full) + assert_contains "$show" "state: done" "an answered captain-held task did not close" + assert_contains "$show" "Resolution recorded by fm-captain-hold" "the answered task lost the decision record" + assert_contains "$show" "Resolution mode: answered" "the answered task did not record its close path" + assert_contains "$show" "Captain chose the guard option." \ + "the answered task did not record the captain decision text" + run_captain "$home" answer sample-guard-call --decision-file "$home/guard-decision.txt" >/dev/null \ + || fail "identical answer retry was not idempotent" + printf 'Captain chose something else entirely.\n' > "$home/drifted.txt" + if run_captain "$home" answer sample-guard-call --decision-file "$home/drifted.txt" \ + > "$home/drifted-answer.out" 2> "$home/drifted-answer.err"; then + fail "answer retry accepted a different captain decision" + fi + # The answered call releases the work routed behind it: a Done blocker reads + # as resolved everywhere. + show=$(tasks_in "$home" show sample-guard-work --full) + assert_contains "$show" "blocked: no" "the recorded answer did not release dependent work" + run_captain "$home" verify "$id" >/dev/null \ + || fail "an answered captain call did not satisfy the completion gate" + json=$(run_bearings "$home") || fail "Bearings failed after the answer" + printf '%s' "$json" | jq -e ' + (.decisions_open | any(.id == "sample-guard-call") | not) + and (.gates | any(.id == "sample-guard-call") | not) + and (.landed | any(.id == "sample-guard-call") | not) + ' >/dev/null || fail "an answered captain call still renders somewhere it should not: $json" + pass "answer records the captain's words, closes idempotently, and releases routed work" +} + +# --release lifts the hold instead of closing, preserving the work item's own +# body under the record; a re-held task later accepts a new answer. +test_release_frees_held_work() { + local home show out + home=$(make_home release-work) + tasks_in "$home" add sample-widget "Ship the sample widget" --kind ship --repo sample \ + --body 'The widget plan body. Literal escape: \n. Unicode: café.' >/dev/null \ + || fail "could not create the held work item" + run_captain "$home" hold sample-widget --reason "captain go needed before shipping" >/dev/null \ + || fail "could not hold the work item for the captain" + printf 'Go: ship it as planned.\n' > "$home/go.txt" + run_captain "$home" answer sample-widget --decision-file "$home/go.txt" --release >/dev/null \ + || fail "answer --release failed on the held work item" + show=$(tasks_in "$home" show sample-widget --full) + assert_contains "$show" "state: queued" "a released work item did not stay queued" + assert_contains "$show" "held: no" "a released work item kept its hold" + assert_contains "$show" "Resolution mode: released" "the release did not record its close path" + assert_contains "$show" "Go: ship it as planned." "the release lost the captain's words" + assert_contains "$show" "The widget plan body." "the release destroyed the work item body" + assert_contains "$show" 'Literal escape: \\n. Unicode: café.' \ + "the release corrupted escaped or Unicode body text" + run_captain "$home" answer sample-widget --decision-file "$home/go.txt" --release >/dev/null \ + || fail "identical release retry was not idempotent" + if run_captain "$home" answer sample-widget --decision-file "$home/go.txt" \ + > "$home/wrong-mode.out" 2> "$home/wrong-mode.err"; then + fail "a released answer replay without --release reported completion" + fi + assert_grep "mode released" "$home/wrong-mode.err" \ + "the mismatched replay did not name the recorded release mode" + show=$(tasks_in "$home" show sample-widget --full) + assert_contains "$show" "state: queued" "a mismatched release replay closed the work item" + assert_contains "$show" "held: no" "a mismatched release replay re-held the work item" + + tasks_in "$home" add sample-empty-label-widget "Ship without a display label" \ + --kind ship --repo sample >/dev/null + run_captain "$home" hold sample-empty-label-widget --reason "captain go needed" >/dev/null + out=$(printf 'sample-empty-label-widget\tgo\t\trelease\n' \ + | run_captain "$home" answers --source "empty-label release fixture") \ + || fail "an empty answer label shifted the release close mode" + assert_contains "$out" "closed: sample-empty-label-widget" \ + "the empty-label release was not accepted" + show=$(tasks_in "$home" show sample-empty-label-widget --full) + assert_contains "$show" "state: queued" "an empty-label release completed its work item" + assert_contains "$show" "held: no" "an empty-label release did not lift the hold" + assert_contains "$show" "Resolution mode: released" \ + "an empty-label release recorded the wrong close mode" + + # A NEW captain gate on the same task later takes a NEW answer. + run_captain "$home" hold sample-widget --reason "captain pricing call needed" >/dev/null \ + || fail "could not re-hold the released work item" + printf 'Price it at nine dollars.\n' > "$home/price.txt" + run_captain "$home" answer sample-widget --decision-file "$home/price.txt" --release >/dev/null \ + || fail "a re-held task refused a new answer" + show=$(tasks_in "$home" show sample-widget --full) + assert_contains "$show" "Price it at nine dollars." "the new answer was not recorded" + assert_contains "$show" "Go: ship it as planned." "the new answer erased the earlier record" + + tasks_in "$home" "done" sample-widget >/dev/null \ + || fail "could not complete the released work item normally" + if run_captain "$home" answer sample-widget --decision-file "$home/price.txt" \ + > "$home/closed-wrong-mode.out" 2> "$home/closed-wrong-mode.err"; then + fail "a completed release replay without --release reported an answer" + fi + assert_grep "mode released" "$home/closed-wrong-mode.err" \ + "the completed replay did not name the recorded release mode" + show=$(tasks_in "$home" show sample-widget --full) + assert_contains "$show" "state: done" "a refused completed replay changed task state" + pass "release frees held work with the captain's words recorded and the body preserved" +} + +# Deferral is a date, not a live card: hold --until keeps the task out of +# captain_actionable until due, tasks-axi's own date-gate expiry keeps the task +# answerable, and Bearings renders the wait as a dated gate. +test_deferral_leaves_captains_call_until_due() { + local home json snap show + home=$(make_home deferral) + run_captain "$home" hold sample-later-call --title "Revisit the sample plan" \ + --reason "captain deferred revisit later" --repo sample --until 2026-08-01 >/dev/null \ + || fail "could not register the deferred captain call" + run_captain "$home" hold sample-now-call --title "Decide the sample cut" \ + --reason "captain cut choice pending" --repo sample >/dev/null \ + || fail "could not register the live captain call" + if run_captain "$home" hold sample-bad-date --title "Bad date" \ + --reason "captain choice" --until 2026-8-1 > "$home/bad-date.out" 2> "$home/bad-date.err"; then + fail "hold accepted a malformed --until date" + fi + + snap=$(PATH="$home/fakebin:$PATH" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" \ + FM_DATA_OVERRIDE="$home/data" FM_CONFIG_OVERRIDE="$home/config" \ + FM_PROJECTS_OVERRIDE="$home/projects" FM_SNAPSHOT_NOW=2026-07-14T12:00:00Z \ + "$ROOT/bin/fm-fleet-snapshot.sh" --json) || fail "fleet snapshot failed" + printf '%s' "$snap" | jq -e ' + ([.backlog.records[] | select(.id == "sample-later-call")][0]) as $later + | ([.backlog.records[] | select(.id == "sample-now-call")][0]) as $now + | $later.captain_actionable == false and $later.hold_until == "2026-08-01" + and $now.captain_actionable == true and $now.hold_until == null + and ($later.title | contains("hold-until") | not) + ' >/dev/null || fail "the due gate or hold-until parsing is wrong: $snap" + + json=$(run_bearings "$home") || fail "Bearings failed with a deferred call" + printf '%s' "$json" | jq -e ' + (.decisions_open | any(.id == "sample-now-call")) + and (.decisions_open | any(.id == "sample-later-call") | not) + and (.gates | any(.id == "sample-later-call" and (.reason | startswith("until 2026-08-01")))) + ' >/dev/null || fail "the deferred call did not render as a dated gate: $json" + + # On its date the call is due again - and still answerable even though + # tasks-axi reports the expired hold as no longer held. + snap=$(PATH="$home/fakebin:$PATH" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" \ + FM_DATA_OVERRIDE="$home/data" FM_CONFIG_OVERRIDE="$home/config" \ + FM_PROJECTS_OVERRIDE="$home/projects" FM_SNAPSHOT_NOW=2026-08-01T12:00:00Z \ + "$ROOT/bin/fm-fleet-snapshot.sh" --json) || fail "fleet snapshot failed at the due date" + printf '%s' "$snap" | jq -e ' + [.backlog.records[] | select(.id == "sample-later-call")][0].captain_actionable == true + ' >/dev/null || fail "a due deferral did not resurface as captain-actionable" + show=$(tasks_in "$home" show sample-later-call --full) + assert_contains "$show" "hold_kind: captain" "the expired deferral lost its captain-hold annotations" + printf 'Answered on the due date.\n' > "$home/due.txt" + run_captain "$home" answer sample-later-call --decision-file "$home/due.txt" >/dev/null \ + || fail "an expired deferral was not answerable" + pass "a deferred captain call leaves the live Captain's Call until its date and stays answerable" +} + +# The recorded-answer guard survives an out-of-band close: a bare tasks-axi done +# fails verify until answer records the captain's word, and an ordinary finished +# task can never be dressed up as an answered captain call. +test_out_of_band_close_is_recordable() { + local home id show + home=$(make_home out-of-band) + id=sample-fullrun-review + mkdir -p "$home/data/$id" + tasks_in "$home" add "$id" "Investigate the sample full run" --kind scout --repo sample --start >/dev/null \ + || fail "could not create out-of-band origin" + write_origin_meta "$home" "$id" + printf 'done: report complete\n' > "$home/state/$id.status" + printf '# Sample full run review\n\nOne captain choice remains.\n' > "$home/data/$id/report.md" + run_captain "$home" hold sample-submission-call --title "Choose the sample submission" \ + --reason "captain submission choice pending" --repo sample --origin "$id" >/dev/null \ + || fail "could not register the captain-held task" + run_captain "$home" complete "$id" sample-submission-call >/dev/null \ + || fail "completion failed before the out-of-band close" + + tasks_in "$home" "done" sample-submission-call >/dev/null \ + || fail "could not reproduce the direct out-of-band close" + if run_captain "$home" verify "$id" > "$home/broken-verify.out" 2> "$home/broken-verify.err"; then + fail "verification passed a captain call closed with no recorded answer" + fi + if run_teardown "$home" "$id" > "$home/broken-teardown.out" 2> "$home/broken-teardown.err"; then + fail "teardown proceeded while a captain call had no recorded answer" + fi + assert_present "$home/state/$id.meta" "refused teardown removed investigation metadata" + + printf 'Declined: do not submit the sample full run upstream.\n' > "$home/submission.txt" + run_captain "$home" answer sample-submission-call --decision-file "$home/submission.txt" >/dev/null \ + || fail "answer could not record the missing captain decision on the closed task" + show=$(tasks_in "$home" show sample-submission-call --full) + assert_contains "$show" "state: done" "recording the answer reopened the closed task" + assert_contains "$show" "Resolution mode: repaired" "the retroactive record did not name its path" + assert_contains "$show" "Declined: do not submit the sample full run upstream." \ + "the retroactive record lost the captain decision text" + run_captain "$home" verify "$id" >/dev/null \ + || fail "the recorded answer did not satisfy the completion gate" + run_captain "$home" answer sample-submission-call --decision-file "$home/submission.txt" >/dev/null \ + || fail "identical retroactive retry was not idempotent" + printf 'A different answer entirely.\n' > "$home/drifted.txt" + if run_captain "$home" answer sample-submission-call --decision-file "$home/drifted.txt" \ + > "$home/drifted.out" 2> "$home/drifted.err"; then + fail "a drifted retry overwrote the recorded captain decision" + fi + run_teardown "$home" "$id" >/dev/null 2> "$home/teardown.err" \ + || fail "teardown still refused after the answer was recorded: $(cat "$home/teardown.err")" + + # An ordinary finished task was never the captain's item; recording an + # invented answer on it must be refused. + tasks_in "$home" add sample-ordinary-work "Ordinary finished work" --kind ship --repo sample >/dev/null + tasks_in "$home" "done" sample-ordinary-work >/dev/null + printf 'An answer the captain never gave.\n' > "$home/invented.txt" + if run_captain "$home" answer sample-ordinary-work --decision-file "$home/invented.txt" \ + > "$home/never-held.out" 2> "$home/never-held.err"; then + fail "an ordinary finished task was dressed up as an answered captain call" + fi + assert_grep "never held for the captain" "$home/never-held.err" \ + "the refusal must say the task carries no captain-hold provenance" + pass "an out-of-band close is recordable with the captain's word and nothing else" +} + +# A post-teardown visual review completes against the surviving report and +# durable tasks, with no volatile task metadata and no second decision database. +test_visual_review_uses_shared_completion_owner() { + local home id json + home=$(make_home visual-review) + id=sample-board-review + mkdir -p "$home/data/$id" + tasks_in "$home" add "$id" "Review the sample board" --kind scout --repo sample --start >/dev/null + write_origin_meta "$home" "$id" + printf 'done: investigation complete\n' > "$home/state/$id.status" + printf '# Sample board investigation\n\nThe initial findings need no captain choice.\n' > "$home/data/$id/report.md" + run_captain "$home" complete "$id" --none >/dev/null \ + || fail "initial investigation could not pass the shared completion owner" + run_teardown "$home" "$id" >/dev/null 2> "$home/visual-teardown.err" \ + || fail "completed investigation teardown failed: $(cat "$home/visual-teardown.err")" + tasks_in "$home" "done" "$id" --report "data/$id/report.md" --keep 0 >/dev/null + + mkdir -p "$home/.lavish" + printf '<html><body>Synthetic sample board</body></html>\n' > "$home/.lavish/sample-board.html" + run_captain "$home" hold sample-layout-call --title "Choose the sample layout" \ + --reason "captain layout choice pending" --repo sample --origin "$id" >/dev/null \ + || fail "post-teardown visual review could not use the shared hold owner" + run_captain "$home" complete "$id" sample-layout-call >/dev/null \ + || fail "post-teardown visual review could not use the shared completion owner" + json=$(run_bearings "$home") || fail "Bearings failed after the ended visual review" + printf '%s' "$json" | jq -e ' + .decisions_open | any(.id == "sample-layout-call" and .verb == "captain-hold") + ' >/dev/null || fail "ended visual review did not leave its durable Captain Call: $json" + [ ! -e "$home/data/visual-review-decisions.json" ] \ + || fail "visual review created a second decision database" + pass "ended visual review follows the same captain-hold completion owner" +} + +test_none_inventory_and_resolved_prose_do_not_create_holds() { + local home id json + home=$(make_home no-false-holds) + id=sample-resolved-review + mkdir -p "$home/data/$id" + tasks_in "$home" add "$id" "Review a resolved sample finding" --kind scout --repo sample --start >/dev/null + write_origin_meta "$home" "$id" + printf 'resolved [key=old-choice]: the sample choice was already recorded\ndone: report complete\n' \ + > "$home/state/$id.status" + cat > "$home/data/$id/report.md" <<'EOF' +# Resolved sample finding + +Decision record: the earlier choice is resolved. +The recommendation is informational and needs no captain action. +EOF + run_captain "$home" complete "$id" --none >/dev/null \ + || fail "explicit no-call inventory failed" + json=$(run_bearings "$home") || fail "Bearings failed for no-call inventory" + printf '%s' "$json" | jq -e ' + (.decisions_open | any(.id | startswith("sample-resolved-review")) | not) + ' >/dev/null || fail "resolved findings or decision-like prose created a false captain call: $json" + pass "resolved findings and decision-like prose do not create captain-held tasks" +} + +test_terminal_single_owner_status_decision_does_not_block_empty_inventory() { + local home id open secondmate + home=$(make_home stale-terminal-decision) + id=sample-terminal-review + mkdir -p "$home/data/$id" + tasks_in "$home" add "$id" "Review a terminal sample finding" --kind scout --repo sample --start >/dev/null + write_origin_meta "$home" "$id" + printf 'needs-decision [key=default]: choose route A or route B\ndone: report complete\n' \ + > "$home/state/$id.status" + printf '# Terminal sample review\n\nNo unresolved captain choice remains.\n' > "$home/data/$id/report.md" + open=$(bash -c '. "$1"; status_open_decisions "$2"' _ \ + "$ROOT/bin/fm-classify-lib.sh" "$home/state/$id.status") + assert_contains "$open" "default" "fixture must retain the raw stale status decision" + run_captain "$home" complete "$id" --none >/dev/null \ + || fail "terminal single-owner stale status decision blocked empty inventory completion" + run_captain "$home" verify "$id" >/dev/null \ + || fail "terminal single-owner stale status decision blocked inventory verification" + run_teardown "$home" "$id" >/dev/null 2> "$home/terminal-teardown.err" \ + || fail "terminal single-owner stale status decision blocked teardown: $(cat "$home/terminal-teardown.err")" + + secondmate=sample-secondmate + write_origin_meta "$home" "$secondmate" secondmate + printf 'needs-decision [key=route]: choose route A or route B\ndone: heartbeat complete\n' \ + > "$home/state/$secondmate.status" + if run_captain "$home" complete "$secondmate" --none \ + > "$home/secondmate-terminal.out" 2> "$home/secondmate-terminal.err"; then + fail "secondmate terminal status decision was incorrectly cleared" + fi + pass "terminal single-owner stale status decisions do not block empty inventory" +} + +test_secondmate_hold_stays_in_authoritative_home() { + local parent mate fakebin origin json + parent=$(make_home main-routing) + mate="$TMP_ROOT/sample-mate-home" + mkdir -p "$mate/data" "$mate/state" "$mate/config" "$mate/projects" "$mate/bin" + cp "$ROOT/.tasks.toml" "$mate/.tasks.toml" + printf '# Synthetic secondmate home\n' > "$mate/AGENTS.md" + printf 'sample-mate\n' > "$mate/.fm-secondmate-home" + cat > "$mate/data/backlog.md" <<'EOF' +## In flight + +## Queued + +## Done +EOF + fakebin=$(fm_fakebin "$mate") + fm_fake_exit0 "$fakebin" tmux treehouse no-mistakes gh gh-axi + origin=sample-mate-review + mkdir -p "$mate/data/$origin" + tasks_in "$mate" add "$origin" "Investigate secondmate sample" --kind scout --repo sample --start >/dev/null + write_origin_meta "$mate" "$origin" + printf 'done: report and visual review complete\n' > "$mate/state/$origin.status" + printf '# Sample secondmate review\n\nOne captain choice remains.\n' > "$mate/data/$origin/report.md" + run_captain "$mate" hold sample-release-call --title "Choose the sample release" \ + --reason "captain release choice pending" --repo sample --origin "$origin" >/dev/null \ + || fail "secondmate-owned hold creation failed" + run_captain "$mate" complete "$origin" sample-release-call >/dev/null \ + || fail "secondmate-owned completion failed" + run_teardown "$mate" "$origin" >/dev/null 2> "$mate/teardown.err" \ + || fail "secondmate investigation teardown failed: $(cat "$mate/teardown.err")" + tasks_in "$mate" "done" "$origin" --report "data/$origin/report.md" --keep 0 >/dev/null + + printf -- '- sample-mate - synthetic scope (home: %s; scope: sample reviews; projects: sample; added 2026-07-14)\n' \ + "$mate" > "$parent/data/secondmates.md" + fm_write_secondmate_meta "$parent/state/sample-mate.meta" "$mate" \ + "firstmate:fm-sample-mate" sample + json=$(run_bearings "$parent") || fail "parent Bearings could not read the secondmate captain call" + printf '%s' "$json" | jq -e ' + .decisions_open | any(.owner == "sample-mate" and .verb == "captain-hold" + and (.id | endswith("sample-release-call"))) + ' >/dev/null || fail "secondmate captain call did not surface with authoritative owner: $json" + assert_no_grep "sample-release-call" "$parent/data/backlog.md" "secondmate call leaked into the main backlog" + assert_grep "sample-release-call" "$mate/data/backlog.md" "secondmate call left its authoritative backlog" + pass "main-home and secondmate-home captain calls remain correctly routed" +} + +# The one keyed-answer intake, fed through the real process-event runner by a +# fixture channel that knows nothing about captain holds: task-id keys close at +# answer time, a card-declared release mode frees held work, freeform prose can +# forge nothing, and a replayed capture is idempotent. +test_bound_channel_answers_close_at_answer_time() { + local home id sid artifact result out show rc + home=$(make_home channel-answer-closure) + id=sample-eval-proposal + mkdir -p "$home/data/$id" + tasks_in "$home" add "$id" "Propose sample eval changes" --kind scout --repo sample --start >/dev/null \ + || fail "could not create the review origin" + write_origin_meta "$home" "$id" + printf 'done: proposal deck ready for the captain\n' > "$home/state/$id.status" + printf '# Sample eval proposal\n\nThree captain choices remain.\n' > "$home/data/$id/report.md" + run_captain "$home" hold sample-membership-call --title "Captain call: membership" \ + --reason "captain membership choice pending" --repo sample --origin "$id" >/dev/null + run_captain "$home" hold sample-headline-call --title "Captain call: headline" \ + --reason "captain headline choice pending" --repo sample --origin "$id" >/dev/null + run_captain "$home" hold sample-forged-call --title "Captain call: forged" \ + --reason "captain forged choice pending" --repo sample --origin "$id" >/dev/null + run_captain "$home" hold sample-invalid-close-call --title "Captain call: invalid close" \ + --reason "captain close mode validation pending" --repo sample --origin "$id" >/dev/null + tasks_in "$home" add sample-gated-work "Gated sample work" --kind ship --repo sample \ + --body 'Gated work plan.' >/dev/null + run_captain "$home" hold sample-gated-work --reason "captain go needed" >/dev/null + run_captain "$home" complete "$id" \ + sample-membership-call sample-headline-call sample-forged-call sample-invalid-close-call \ + sample-gated-work >/dev/null \ + || fail "completion failed for the deck's inventoried calls" + + artifact="$home/data/$id/review.html" + printf '<h1>Sample eval proposal</h1>\n' > "$artifact" + fm_fake_exit0 "$home/fakebin" lavish-axi + sid=$(run_lavish "$home" source-id "$artifact") || fail "could not derive the review source id" + run_captain "$home" bind "$sid" >/dev/null \ + || fail "could not bind the review source to the keyed-answer intake" + [ "$(run_captain "$home" binding "$sid")" = "(any)" ] \ + || fail "the recorded binding did not resolve to the collapsed marker" + run_lavish "$home" arm "$artifact" >/dev/null || fail "could not arm the review deck" + + result="$home/state/procevent-inbox/$sid.1.result" + mkdir -p "$home/state/procevent-inbox" + cat > "$result" <<'EOF' +session: + file: /review.html + status: feedback + session_ended: true + ended_by: user +prompts[6]{uid,prompt,selector,tag,text}: + "2","Membership: gold-only\n\nContext data:\n{\n \"question\": \"sample-membership-call\",\n \"answer\": \"gold-only\"\n}","section#call > form:nth-of-type(1)",choice,"Membership: gold-only" + "3","Headline: f1-when-fp-gold\n\nContext data:\n{\n \"question\": \"sample-headline-call\",\n \"answer\": \"f1-when-fp-gold\"\n}","section#call > form:nth-of-type(2)",choice,"Headline: f1-when-fp-gold" + "4","Gated work: go\n\nContext data:\n{\n \"question\": \"sample-gated-work\",\n \"answer\": \"go\",\n \"close\": \"release\"\n}","section#call > form:nth-of-type(3)",choice,"Gated work: go" + "5","Absent call: yes\n\nContext data:\n{\n \"question\": \"sample-nonexistent-call\",\n \"answer\": \"yes\"\n}","section#call > form:nth-of-type(4)",choice,"Absent call: yes" + "6","Invalid close: yes\n\nContext data:\n{\n \"question\": \"sample-invalid-close-call\",\n \"answer\": \"yes\",\n \"close\": \"drop\"\n}","section#call > form:nth-of-type(5)",choice,"Invalid close: yes" + "",get this fully implemented. Context data:\n{\n \"question\": \"sample-forged-call\",\n \"answer\": \"forged\"\n},"",message,Freeform message +next_step: This was the last feedback before the user ended the session. +EOF + printf 'lavish\n' > "$home/state/procevent-inbox/$sid.1.adapter" + + out=$(run_lavish "$home" answers "$result") || fail "could not read the captured answers" + assert_contains "$out" "sample-membership-call gold-only" "a structured choice was not read as an answer" + assert_contains "$out" "sample-gated-work go Gated work: go release" \ + "the card-declared release mode was not relayed" + assert_not_contains "$out" "sample-forged-call" \ + "a freeform captain message forged a task id from its own prose" + assert_not_contains "$out" "sample-invalid-close-call" \ + "an unsupported card close mode defaulted to completion" + + mkdir -p "$home/adapter-root/bin" + cat > "$home/adapter-root/bin/fm-procevent-fixturechan.sh" <<SH +#!/usr/bin/env bash +# Fixture channel: reports keyed captain answers and nothing else. +case "\${1-}" in + answers) exec "$ROOT/bin/fm-procevent-lavish.sh" answers "\${2-}" ;; +esac +exit 2 +SH + chmod +x "$home/adapter-root/bin/fm-procevent-fixturechan.sh" + run_captain "$home" bind fixture-src >/dev/null \ + || fail "could not bind the fixture channel" + PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$home/adapter-root" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_PROCEVENT_CLAIM_ROOT="$home/procevent-claims" \ + "$ROOT/bin/fm-procevent.sh" register fixturechan fixture-src -- cat "$result" >/dev/null \ + || fail "could not register the fixture channel source" + PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$home/adapter-root" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_PROCEVENT_CLAIM_ROOT="$home/procevent-claims" \ + "$ROOT/bin/fm-procevent.sh" start fixture-src >/dev/null 2>&1 + assert_absent "$home/state/procevent-inbox/fixture-src.1.handled" \ + "feeding a captain answer retired the notification firstmate still needs" + assert_present "$home/state/procevent-inbox/fixture-src.1.result" \ + "the fixture channel captured no result to feed" + + show=$(tasks_in "$home" show sample-membership-call --full) + assert_contains "$show" "state: done" "capturing the captain's answer left the membership call open" + assert_contains "$show" "Resolution mode: answered" "the membership call did not record its close path" + assert_contains "$show" "Answer: gold-only" "the closed call did not record the captain's actual answer" + show=$(tasks_in "$home" show sample-gated-work --full) + assert_contains "$show" "state: queued" "the released work item did not stay queued" + assert_contains "$show" "held: no" "the card-declared release did not lift the hold" + assert_contains "$show" "Resolution mode: released" "the released work did not record its close path" + assert_contains "$show" "Gated work plan." "the released work item lost its body" + show=$(tasks_in "$home" show sample-forged-call --full) + assert_contains "$show" "state: queued" "a forged key from freeform prose closed a captain call" + show=$(tasks_in "$home" show sample-invalid-close-call --full) + assert_contains "$show" "state: queued" "an unsupported card close mode closed a captain call" + assert_contains "$show" "held: yes" "an unsupported card close mode released a captain call" + + # Replaying the same capture is a no-op, not a rejected different decision. A + # run that could not close every answered key still reports nonzero. + set +e + out=$(run_lavish "$home" answers "$result" \ + | run_captain "$home" answers --source "the captured result fixture-src sequence 1" 2>&1) + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "a run that skipped a key reported success" + assert_contains "$out" "closed: sample-membership-call" \ + "replaying an identical capture was not idempotent: $out" + assert_contains "$out" "closed: sample-gated-work" \ + "replaying an identical released answer was not idempotent: $out" + assert_contains "$out" "skipped: sample-nonexistent-call" \ + "a key naming no task was not reported as skipped: $out" + + printf 'Captain answered the forged call directly.\n' > "$home/forged.txt" + run_captain "$home" answer sample-forged-call --decision-file "$home/forged.txt" >/dev/null \ + || fail "could not close the untouched call through the answer path" + printf 'Captain answered the invalid-close call directly.\n' > "$home/invalid-close.txt" + run_captain "$home" answer sample-invalid-close-call --decision-file "$home/invalid-close.txt" >/dev/null \ + || fail "could not close the invalid-close call through the answer path" + run_captain "$home" verify "$id" >/dev/null \ + || fail "answered calls did not satisfy the completion gate" + pass "a bound channel's captured answers close their captain-held tasks at answer time" +} + +# Answer-time closure is opt-in per source. A channel with no binding must behave +# exactly as it always did: capture, announce, close nothing. +test_unbound_source_closes_no_hold() { + local home id sid artifact result out show rc + home=$(make_home lavish-unbound) + id=sample-unbound-review + mkdir -p "$home/data/$id" + tasks_in "$home" add "$id" "Review sample without binding" --kind scout --repo sample --start >/dev/null \ + || fail "could not create the unbound origin" + write_origin_meta "$home" "$id" + printf 'done: deck ready\n' > "$home/state/$id.status" + printf '# Unbound review\n\nOne captain choice remains.\n' > "$home/data/$id/report.md" + run_captain "$home" hold sample-only-call --title "Captain call: only choice" \ + --reason "captain only choice pending" --repo sample --origin "$id" >/dev/null \ + || fail "could not register the unbound call" + + artifact="$home/data/$id/review.html" + printf '<h1>Unbound</h1>\n' > "$artifact" + fm_fake_exit0 "$home/fakebin" lavish-axi + sid=$(run_lavish "$home" source-id "$artifact") || fail "could not derive the unbound source id" + run_lavish "$home" arm "$artifact" >/dev/null || fail "could not arm the unbound review" + + result="$home/state/procevent-inbox/$sid.1.result" + mkdir -p "$home/state/procevent-inbox" + cat > "$result" <<'EOF' +session: + file: /review.html + status: feedback +prompts[1]{uid,prompt,selector,tag,text}: + "2","Only choice: yes\n\nContext data:\n{\n \"question\": \"sample-only-call\",\n \"answer\": \"yes\"\n}","form",choice,"Only choice: yes" +EOF + set +e + out=$(run_captain "$home" binding "$sid" 2>&1) + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "an unbound source reported a binding" + [ -z "$out" ] || fail "an unbound source printed a binding: $out" + show=$(tasks_in "$home" show sample-only-call --full) + assert_contains "$show" "state: queued" "an unbound review closed a captain call" + assert_contains "$show" "held: yes" "an unbound review released a captain call" + pass "a channel source with no decision binding closes nothing" +} + +# Everything a pre-collapse install already has keeps working: composed +# identities through the shim, short decision keys in recorded metadata, a +# concrete-origin binding, and the chat fallback for old rows. +test_legacy_identities_keep_working() { + local home id hold out show legacy_text legacy_digest old_hold + home=$(make_home legacy-compat) + id=sample-legacy-review + mkdir -p "$home/data/$id" + tasks_in "$home" add "$id" "Legacy-shaped review" --kind scout --repo sample --start >/dev/null + write_origin_meta "$home" "$id" + printf 'done: report complete\n' > "$home/state/$id.status" + printf '# Legacy review\n\nTwo captain choices remain.\n' > "$home/data/$id/report.md" + + hold=$(run_shim "$home" id "$id" pick-one) + [ "$hold" = "$id-decision-pick-one" ] || fail "the shim identity was not deterministic: $hold" + out=$(run_shim "$home" hold "$id" pick-one \ + --title "Pick one" --reason "captain choice pending" --repo sample) \ + || fail "the shim hold path failed" + [ "$out" = "$hold" ] || fail "the shim hold did not print the composed identity: $out" + run_shim "$home" hold "$id" keep-two \ + --title "Keep two" --reason "captain second choice pending" --repo sample >/dev/null \ + || fail "the shim second hold failed" + show=$(tasks_in "$home" show "$hold" --full) + assert_contains "$show" "hold_kind: captain" "the shim-created row is not a plain captain-held task" + + # A pre-collapse metadata attestation records SHORT keys; verify must resolve + # them through the legacy composed identity. + printf 'decisions_reviewed=1\ndecision_keys=keep-two,pick-one\n' >> "$home/state/$id.meta" + run_captain "$home" verify "$id" >/dev/null \ + || fail "legacy short-key metadata did not verify against composed identities" + + # The shim's routed close records the routed work inside the captain decision + # and clears the recorded edge. + tasks_in "$home" add sample-legacy-work "Apply the legacy choice" \ + --kind ship --repo sample --blocked-by "$hold" >/dev/null + tasks_in "$home" add sample-unrouted-work "Unrouted legacy work" \ + --kind ship --repo sample >/dev/null + printf 'Use route north.\n' > "$home/route.txt" + if run_shim "$home" resolve "$id" pick-one --decision-file "$home/route.txt" \ + --routed-to sample-missing-work > "$home/missing-route.out" 2> "$home/missing-route.err"; then + fail "the shim resolve accepted a missing routed task" + fi + if run_shim "$home" resolve "$id" pick-one --decision-file "$home/route.txt" \ + --routed-to sample-unrouted-work > "$home/unrouted.out" 2> "$home/unrouted.err"; then + fail "the shim resolve accepted work not blocked by the legacy decision" + fi + show=$(tasks_in "$home" show "$hold" --full) + assert_contains "$show" "state: queued" "invalid shim routing closed the legacy decision" + assert_not_contains "$show" "Resolution recorded" "invalid shim routing recorded an answer" + run_shim "$home" resolve "$id" pick-one --decision-file "$home/route.txt" \ + --routed-to sample-legacy-work >/dev/null \ + || fail "the shim resolve path failed" + show=$(tasks_in "$home" show "$hold" --full) + assert_contains "$show" "state: done" "the shim resolve did not close the row" + assert_contains "$show" "Use route north." "the shim resolve lost the captain decision" + assert_contains "$show" "- sample-legacy-work" "the shim resolve lost the routed identities" + show=$(tasks_in "$home" show sample-legacy-work --full) + assert_contains "$show" "blocked: no" "the shim resolve did not release the routed work" + + old_hold=$(run_shim "$home" hold "$id" old-route \ + --title "Old routed choice" --reason "captain old route pending" --repo sample) + tasks_in "$home" add sample-old-routed-work "Apply the old routed choice" \ + --kind ship --repo sample --blocked-by "$old_hold" >/dev/null + printf 'Use the historical route.\n' > "$home/old-route.txt" + legacy_text=$(cat "$home/old-route.txt") + if command -v shasum >/dev/null 2>&1; then + legacy_digest=$(printf '%s' "$legacy_text" | shasum -a 256 | awk '{print $1}') + else + legacy_digest=$(printf '%s' "$legacy_text" | sha256sum | awk '{print $1}') + fi + printf 'Resolution recorded by fm-decision-hold.\nDecision digest: %s\nRouted identities: sample-old-routed-work\nResolution mode: routed\n\nCaptain decision:\n%s\n\nRouted work:\n- sample-old-routed-work\n' \ + "$legacy_digest" "$legacy_text" > "$home/old-route-body.txt" + tasks_in "$home" update "$old_hold" --body-file "$home/old-route-body.txt" --archive-body >/dev/null + run_shim "$home" resolve "$id" old-route --decision-file "$home/old-route.txt" \ + --routed-to sample-old-routed-work >/dev/null \ + || fail "the shim did not replay a matching pre-collapse routed record" + show=$(tasks_in "$home" show "$old_hold" --full) + assert_contains "$show" "state: done" "the replayed legacy resolve did not close its hold" + show=$(tasks_in "$home" show sample-old-routed-work --full) + assert_contains "$show" "blocked_by: none" "the replayed legacy resolve did not clear its recorded edge" + + # The shim decline path maps onto the same recorded answer. + printf 'Declined: keep the current shape.\n' > "$home/decline.txt" + run_shim "$home" decline "$id" keep-two --decision-file "$home/decline.txt" >/dev/null \ + || fail "the shim decline path failed" + run_captain "$home" verify "$id" >/dev/null \ + || fail "shim-closed rows did not satisfy the completion gate" + + # A concrete-origin binding (a pre-collapse record) makes short channel keys + # resolve through the composed identity. + run_shim "$home" hold "$id" third-choice \ + --title "Third choice" --reason "captain third choice pending" --repo sample >/dev/null + run_shim "$home" bind legacy-src "$id" >/dev/null || fail "the shim bind path failed" + [ "$(run_captain "$home" binding legacy-src)" = "$id" ] \ + || fail "the concrete-origin binding was not preserved" + printf 'third-choice\toption b\t\n' \ + | run_captain "$home" answers "$(run_captain "$home" binding legacy-src)" \ + --source "legacy channel" >/dev/null \ + || fail "a short key did not resolve through the concrete-origin binding" + show=$(tasks_in "$home" show "$id-decision-third-choice" --full) + assert_contains "$show" "state: done" "the legacy-keyed answer did not close its row" + + run_shim "$home" hold "$id" fourth-choice \ + --title "Fourth choice" --reason "captain fourth choice pending" --repo sample >/dev/null + legacy_text=$(printf 'Captain answered this decision through legacy replay.\nDecision key: fourth-choice\nAnswer: option c\n') + if command -v shasum >/dev/null 2>&1; then + legacy_digest=$(printf '%s' "$legacy_text" | shasum -a 256 | awk '{print $1}') + else + legacy_digest=$(printf '%s' "$legacy_text" | sha256sum | awk '{print $1}') + fi + printf 'Resolution recorded by fm-decision-hold.\nDecision digest: %s\nRouted identities: none\nResolution mode: answered\n\nCaptain decision:\n%s\n' \ + "$legacy_digest" "$legacy_text" > "$home/legacy-body.txt" + tasks_in "$home" update "$id-decision-fourth-choice" --body-file "$home/legacy-body.txt" --archive-body >/dev/null + tasks_in "$home" "done" "$id-decision-fourth-choice" >/dev/null + out=$(printf 'fourth-choice\toption c\t\n' \ + | run_captain "$home" answers "$id" --source "legacy replay") \ + || fail "an identical pre-collapse keyed answer was not idempotent" + assert_contains "$out" "closed: $id-decision-fourth-choice" \ + "the pre-collapse keyed answer digest was treated as drift" + out=$(printf '%s-decision-fourth-choice\toption c\t\n' "$id" \ + | run_captain "$home" answers --source "legacy replay") \ + || fail "a full legacy task-id replay without an origin was not idempotent" + assert_contains "$out" "closed: $id-decision-fourth-choice" \ + "the origin-free legacy replay digest was treated as drift" + pass "legacy identities, metadata, bindings, and the shim keep working" +} + +# The intake is channel-agnostic, so chat must reach it the same way a captured +# review does - for a task-id key, and for a legacy composed identity. +test_chat_channel_feeds_the_same_keyed_answer_intake() { + local home id fb show + home=$(make_home chat-channel) + id=sample-chat-review + mkdir -p "$home/data/$id" + tasks_in "$home" add "$id" "Review sample chat routing" --kind scout --repo sample --start >/dev/null \ + || fail "could not create the chat-channel origin" + write_origin_meta "$home" "$id" ship + printf 'needs-decision [key=chat-choice]: pick option A or option B\n' > "$home/state/$id.status" + printf '# Chat review\n\nTwo captain choices remain.\n' > "$home/data/$id/report.md" + run_shim "$home" hold "$id" chat-choice \ + --title "Choose the sample chat option" --reason "captain chat choice pending" --repo sample >/dev/null \ + || fail "could not register the legacy chat row" + run_captain "$home" hold sample-chat-followup --title "Choose the chat follow-up" \ + --reason "captain follow-up choice pending" --repo sample >/dev/null \ + || fail "could not register the task-id chat call" + run_captain "$home" complete "$id" "$id-decision-chat-choice" sample-chat-followup >/dev/null \ + || fail "completion failed for the chat calls" + grep -F 'captain-held [key=chat-choice]' "$home/state/$id.status" >/dev/null \ + || fail "precondition: completion did not transfer the decision to its durable owner" + + fb="$home/fakebin" + cat > "$fb/tmux" <<'SH' +#!/usr/bin/env bash +set -u +case "${1:-}" in + send-keys) + shift + literal=0 + while [ $# -gt 0 ]; do + case "$1" in + -t) shift 2 ;; + -l) literal=1; shift ;; + *) break ;; + esac + done + if [ "$literal" = 1 ]; then + printf '%s' "${1:-}" >> "$FM_SEND_LOG" + fi + exit 0 ;; + display-message) + for a in "$@"; do case "$a" in *cursor_y*) printf '1\n'; exit 0 ;; esac; done + printf 'fakepane\n'; exit 0 ;; + capture-pane) printf '╭────╮\n│ │\n╰────╯\n'; exit 0 ;; + list-windows) exit 0 ;; +esac +exit 0 +SH + chmod +x "$fb/tmux" + + : > "$home/send.log" + env PATH="$fb:$PATH" FM_ROOT_OVERRIDE="$home" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_SEND_LOG="$home/send.log" FM_SEND_SETTLE=0 \ + "$ROOT/bin/fm-send.sh" "$id" --resolve-key chat-choice "go with option A" >/dev/null 2>&1 \ + || fail "an answer to a transferred legacy decision was refused by the chat channel" + assert_contains "$(cat "$home/send.log")" "go with option A" "the answer text never reached the worker" + show=$(tasks_in "$home" show "$id-decision-chat-choice" --full) + assert_contains "$show" "state: done" "a chat answer left the legacy row open" + assert_contains "$show" "Answer: go with option A" "the chat-answered row lost the captain answer" + + : > "$home/send.log" + env PATH="$fb:$PATH" FM_ROOT_OVERRIDE="$home" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_SEND_LOG="$home/send.log" FM_SEND_SETTLE=0 \ + "$ROOT/bin/fm-send.sh" "$id" --resolve-key sample-chat-followup "take the second option" >/dev/null 2>&1 \ + || fail "an answer keyed by a task id was refused by the chat channel" + show=$(tasks_in "$home" show sample-chat-followup --full) + assert_contains "$show" "state: done" "a chat answer left the task-id call open" + assert_contains "$show" "Resolution mode: answered" "the chat-answered call did not record its close path" + assert_contains "$show" "Answer: take the second option" "the chat-answered call lost the captain answer" + assert_contains "$show" "answer sent to $id" "the chat-answered call lost its channel provenance" + + if env PATH="$fb:$PATH" FM_ROOT_OVERRIDE="$home" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_SEND_LOG="$home/send.log" FM_SEND_SETTLE=0 \ + "$ROOT/bin/fm-send.sh" "$id" --resolve-key sample-chat-followup "again" \ + > "$home/closed-key.out" 2> "$home/closed-key.err"; then + fail "a key already closed in both ledgers was accepted" + fi + run_captain "$home" verify "$id" >/dev/null \ + || fail "chat-answered calls did not satisfy the completion gate" + pass "the chat channel feeds the same keyed-answer intake a captured review does" +} + +test_origin_slug_validation_precedes_path_construction() { + local home + home=$(make_home slug-validation) + if run_captain "$home" complete "../escape" --none > "$home/escape.out" 2> "$home/escape.err"; then + fail "complete accepted a path-escaping origin id" + fi + assert_grep "privacy-safe slug" "$home/escape.err" "the refusal must name the slug contract" + if run_captain "$home" verify "../escape" > "$home/escape-verify.out" 2> "$home/escape-verify.err"; then + fail "verify accepted a path-escaping origin id" + fi + if run_captain "$home" hold "bad id" --title "x" --reason "y" > "$home/bad-hold.out" 2> "$home/bad-hold.err"; then + fail "hold accepted an invalid task id" + fi + pass "completion and verification validate origins before constructing paths" +} + +# --- record divergence ------------------------------------------------------ + +run_drain() { # <home> + local home=$1 + PATH="$home/fakebin:$PATH" REAL_TASKS_AXI="$TASKS_AXI_BIN" \ + FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" \ + FM_DATA_OVERRIDE="$home/data" FM_CONFIG_OVERRIDE="$home/config" \ + "$ROOT/bin/fm-wake-drain.sh" 2>/dev/null +} + +# Reconstructs the 2026-08-06 loss with synthetic names: the answer was posted +# as a `resolved [key=...]` line and nothing else, so the status fold went quiet +# while the durable captain-held task stayed open and kept reading as if the +# captain had never spoken. Both identities that can carry a captain call must +# be caught - the collapsed one (the key IS the task id) and the legacy derived +# one a pre-collapse origin minted - and the report must reach the drain, which +# is where firstmate actually looks. +test_status_resolution_over_an_open_hold_is_signalled() { + local home id out drain + home=$(make_home divergence-signalled) + id=sample-route-review + tasks_in "$home" add "$id" "Investigate sample routing" --kind scout --repo sample --start >/dev/null \ + || fail "could not create the investigation fixture" + write_origin_meta "$home" "$id" + run_captain "$home" hold sample-route-call \ + --title "Choose route: north or south" --reason "captain route choice pending" \ + --repo sample --origin "$id" >/dev/null \ + || fail "could not register the collapsed-identity captain call" + run_captain "$home" hold "$id-decision-access" \ + --title "Open or restricted sample access" --reason "captain access choice pending" \ + --repo sample --origin "$id" >/dev/null \ + || fail "could not register the legacy-identity captain call" + cat > "$home/state/$id.status" <<'EOF' +working: report drafted +needs-decision [key=sample-route-call]: north or south +resolved [key=sample-route-call]: answered: north +needs-decision [key=access]: open or restricted sample access +resolved [key=access]: answered: restricted +done: report complete +EOF + + out=$(run_captain "$home" diverged) || fail "diverged failed on the reconstructed loss" + printf '%s\n' "$out" | grep -F "sample-route-call $id sample-route-call" >/dev/null \ + || fail "the collapsed-identity divergence was not signalled: $out" + printf '%s\n' "$out" | grep -F "$id-decision-access $id access" >/dev/null \ + || fail "the legacy-identity divergence was not signalled: $out" + + drain=$(run_drain "$home") || fail "the drain failed while reporting divergence" + printf '%s\n' "$drain" | grep -F 'RECORD DIVERGENCE' >/dev/null \ + || fail "the divergence never reached the drain: $drain" + printf '%s\n' "$drain" | grep -F 'sample-route-call [key=sample-route-call]' >/dev/null \ + || fail "the drain section omitted the collapsed-identity divergence: $drain" + printf '%s\n' "$drain" | grep -F "$id-decision-access [key=access]" >/dev/null \ + || fail "the drain section omitted the legacy-identity divergence: $drain" + + # It signals; it never closes. Both records must survive the report unchanged, + # because closing a captain call wrongly removes it from review entirely. + assert_grep "sample-route-call" "$home/data/backlog.md" "the report must not remove the captain-held task" + tasks_in "$home" show sample-route-call --full | grep -E '^ held: yes' >/dev/null \ + || fail "the report released or closed the captain-held task" + [ "$(grep -c '^resolved \[key=sample-route-call\]' "$home/state/$id.status")" = 1 ] \ + || fail "the report rewrote the status log" + + # And it names BOTH reconciliation directions. A status resolution is not proof + # the captain ruled: one of the real cases dissolved because its premise was + # false and another was a question of fact whose first reading was wrong, so + # the only safe instruction is "reconcile with what actually happened". + printf '%s\n' "$drain" | grep -F 'fm-captain-hold.sh answer' >/dev/null \ + || fail "the drain section does not say how to record the captain's answer: $drain" + printf '%s\n' "$drain" | grep -F 're-open the status decision' >/dev/null \ + || fail "the drain section does not offer the re-open direction: $drain" + pass "a status resolution over a still-open captain-held task is signalled, not closed" +} + +# The false-signal boundary, driven by the shapes that are genuinely fine. A +# captain call whose deliverable IS the decision has no routed work item at all, +# and that is legitimate: routed work must never be part of the test. Nor may a +# verified `captain-held` transfer, a still-open status decision, an already +# answered call, or an ordinary task that merely had a keyed question answered. +test_legitimate_holds_produce_no_divergence_signal() { + local home id out drain answer + home=$(make_home divergence-no-false-signal) + id=sample-systems-review + tasks_in "$home" add "$id" "Investigate sample systems" --kind scout --repo sample --start >/dev/null \ + || fail "could not create the investigation fixture" + write_origin_meta "$home" "$id" + + # (1) The decision IS the deliverable: held for the captain, nothing routed, + # no status line anywhere naming it. + run_captain "$home" hold sample-standalone-call \ + --title "Adopt the sample naming convention" --reason "captain call with no routed work" \ + --repo sample >/dev/null || fail "could not register the deliverable-is-the-decision call" + # (2) The verified transfer: still open structurally, closed on the status side + # by the captain-held verb command_complete writes. + run_captain "$home" hold sample-transfer-call \ + --title "Choose the sample retention window" --reason "captain retention choice pending" \ + --repo sample >/dev/null || fail "could not register the transferred call" + # (4) An already answered call whose status line reads resolved. + run_captain "$home" hold sample-answered-call \ + --title "Choose the sample export format" --reason "captain export choice pending" \ + --repo sample >/dev/null || fail "could not register the answered call" + answer="$home/answer.txt" + printf 'Export as CSV.\n' > "$answer" + run_captain "$home" answer sample-answered-call --decision-file "$answer" >/dev/null \ + || fail "could not record the captain answer fixture" + # (5) An ordinary in-flight work item that is not held for the captain. + tasks_in "$home" add sample-plain-work "Ordinary sample work" --kind ship --repo sample --start >/dev/null \ + || fail "could not create the ordinary work fixture" + + cat > "$home/state/$id.status" <<'EOF' +working: report drafted +needs-decision [key=sample-transfer-call]: choose the retention window +captain-held [key=sample-transfer-call]: tracked by sample-transfer-call +needs-decision [key=sample-open-call]: still open on both sides +needs-decision [key=sample-answered-call]: choose the export format +resolved [key=sample-answered-call]: answered: CSV +needs-decision [key=sample-plain-work]: worker question about the sample fixture +resolved [key=sample-plain-work]: answered: go ahead +EOF + # (3) A still-open status decision whose structured twin is also still open. + run_captain "$home" hold sample-open-call \ + --title "Choose the sample refresh cadence" --reason "captain cadence choice pending" \ + --repo sample >/dev/null || fail "could not register the still-open call" + + out=$(run_captain "$home" diverged) || fail "diverged failed on the legitimate shapes" + [ -z "$out" ] || fail "legitimate captain holds produced a false divergence signal: $out" + + drain=$(run_drain "$home") || fail "the drain failed on the legitimate shapes" + if printf '%s\n' "$drain" | grep -F 'RECORD DIVERGENCE' >/dev/null; then + fail "the drain printed a divergence section with nothing diverging: $drain" + fi + printf '%s\n' "$drain" | grep -F 'sample-open-call' >/dev/null \ + || fail "setup error: the still-open decision should still reach OPEN DECISIONS: $drain" + pass "a captain call with no routed work, a verified transfer, an open decision, and an answered call all stay silent" +} + +test_uninventoried_report_decision_refuses_completion +test_completion_gate_attests_and_transfers +test_answer_records_and_closes +test_release_frees_held_work +test_deferral_leaves_captains_call_until_due +test_out_of_band_close_is_recordable +test_visual_review_uses_shared_completion_owner +test_none_inventory_and_resolved_prose_do_not_create_holds +test_terminal_single_owner_status_decision_does_not_block_empty_inventory +test_secondmate_hold_stays_in_authoritative_home +test_bound_channel_answers_close_at_answer_time +test_unbound_source_closes_no_hold +test_legacy_identities_keep_working +test_chat_channel_feeds_the_same_keyed_answer_intake +test_origin_slug_validation_precedes_path_construction +test_status_resolution_over_an_open_hold_is_signalled +test_legitimate_holds_produce_no_divergence_signal diff --git a/tests/fm-classify-decision-key.test.sh b/tests/fm-classify-decision-key.test.sh index 57adb376dbb..8c4196a8a26 100755 --- a/tests/fm-classify-decision-key.test.sh +++ b/tests/fm-classify-decision-key.test.sh @@ -11,7 +11,10 @@ # verb, regardless of order or count. These tests drive the REAL # status_line_verb / status_open_decisions / status_open_decisions_incremental # functions over crafted status files and assert their folded output, never the -# fold's own source text. Cross-drain cursor persistence and the incremental +# fold's own source text. Also covers status_key_closing_verb, which reports how +# the status side currently reads one key so a consumer can tell a settled key +# from one handed to a durable captain-held task (bin/fm-captain-hold.sh +# diverged). Cross-drain cursor persistence and the incremental # cost bound live in tests/fm-wake-drain-open-decisions-cursor.test.sh; the # drain wiring lives in tests/fm-wake-drain-open-decisions.test.sh. set -u @@ -272,3 +275,66 @@ test_corr_only_tag_opens_as_default_like_a_bare_line test_key_only_before_colon_still_opens_no_regression test_blocked_and_resolved_are_tag_order_independent test_incremental_agrees_with_full_fold_across_appends + +# status_key_closing_verb reports HOW the status side currently reads one key, +# which is what lets a consumer tell a settled key from a key handed to a +# durable captain-held task. The two closing verbs must stay distinguishable: +# `resolved` claims the question is settled outright, while `captain-held` is +# the verified transfer to that task, so treating them alike would either lose +# the record-divergence signal or invent one on every correct transfer. +test_closing_verb_separates_resolution_from_durable_transfer() { + local dir f + dir=$(case_dir closing-verb) + f="$dir/a.status" + cat > "$f" <<'EOF' +working: started +needs-decision [key=route]: north or south +resolved [key=route]: answered: north +needs-decision [key=access]: open or restricted +captain-held [key=access]: tracked by sample-access-call +blocked [key=creds]: need the deploy token +done: everything else shipped +EOF + [ "$(status_key_closing_verb "$f" route)" = resolved ] \ + || fail "a resolved key did not report the resolve verb: '$(status_key_closing_verb "$f" route)'" + [ "$(status_key_closing_verb "$f" access)" = captain-held ] \ + || fail "a durable-transfer close reported the wrong verb: '$(status_key_closing_verb "$f" access)'" + [ "$(status_key_closing_verb "$f" creds)" = blocked ] \ + || fail "a still-open key must report its opening verb: '$(status_key_closing_verb "$f" creds)'" + [ -z "$(status_key_closing_verb "$f" never-mentioned)" ] \ + || fail "a key with no transition line reported a verb" + [ -z "$(status_key_closing_verb "$dir/absent.status" route)" ] \ + || fail "an absent status file reported a verb" + pass "status_key_closing_verb separates resolution, durable transfer, and still-open" +} + +# The reported verb is the LAST transition, read through the same fold rule as +# everything else: the colon-first key position counts, a re-opened key reports +# open again, and a prose mention is never a transition. +test_closing_verb_tracks_the_last_transition_in_both_positions() { + local dir f + dir=$(case_dir closing-verb-last) + f="$dir/a.status" + cat > "$f" <<'EOF' +needs-decision: [key=route] colon-first open +resolved: [key=route] colon-first close +EOF + [ "$(status_key_closing_verb "$f" route)" = resolved ] \ + || fail "a colon-first resolution was not seen: '$(status_key_closing_verb "$f" route)'" + + printf 'needs-decision [key=route]: re-opened after a bad answer\n' >> "$f" + [ "$(status_key_closing_verb "$f" route)" = needs-decision ] \ + || fail "a re-opened key still reported closed: '$(status_key_closing_verb "$f" route)'" + + printf 'resolved [key=route]: answered: south after all\n' >> "$f" + [ "$(status_key_closing_verb "$f" route)" = resolved ] \ + || fail "the last of several transitions was not reported: '$(status_key_closing_verb "$f" route)'" + + printf 'working: a later append that only mentions [key=route] as prose\n' >> "$f" + [ "$(status_key_closing_verb "$f" route)" = resolved ] \ + || fail "a prose mention changed the reported verb: '$(status_key_closing_verb "$f" route)'" + pass "status_key_closing_verb reports the last real transition, in either key position" +} + +test_closing_verb_separates_resolution_from_durable_transfer +test_closing_verb_tracks_the_last_transition_in_both_positions diff --git a/tests/fm-claude-stop-autoarm.test.sh b/tests/fm-claude-stop-autoarm.test.sh index 7015fc4995f..ff095c32912 100755 --- a/tests/fm-claude-stop-autoarm.test.sh +++ b/tests/fm-claude-stop-autoarm.test.sh @@ -534,6 +534,215 @@ test_single_flight_admits_exactly_one_owner() { pass "auto-arm: concurrent firings admit one owner and one rewake translation" } +# --- abandoned single-flight claim recovery ----------------------------------- +# The 2026-08-14 lapse: one cycle armed, beat its beacon, delivered a single +# rewake, and exited, leaving its owner lock behind with a live pid. The single +# flight gate then turned every later firing into exit 0, so with two tasks in +# flight and a beacon 40 minutes cold nothing re-armed and both workers' reports +# sat unread until an operator drained the queue by hand. The lock alone is not +# enough to prove that: the ledger naming that same pid with a finished outcome, +# or a recorded pid-identity the live pid no longer matches, is what distinguishes +# an abandoned claim from one still deciding. + +# Fabricate a held owner lock: <dir> <pid> <role>. Plain-dir shape on purpose - +# the hook must reclaim whatever a crashed or blocked owner left behind. +record_autoarm_owner() { + local dir=$1 pid=$2 role=${3:-autoarm} + mkdir -p "$dir/state/.claude-autoarm.lock" + printf '%s\n' "$pid" > "$dir/state/.claude-autoarm.lock/pid" + printf '%s\n' "$role" > "$dir/state/.claude-autoarm.lock/role" +} + +# Record the pid-identity a claim leaves inside its own lock: <dir> <pid>. The +# claim writes the identity of the process that took the lock, so passing a pid +# OTHER than the lock's own reproduces pid reuse - the recorded claimant is gone +# and an unrelated live process now answers to its number. +record_autoarm_owner_identity() { + local dir=$1 pid=$2 identity + identity=$(fm_test_pid_identity "$pid") || return 1 + [ -n "$identity" ] || return 1 + printf '%s\n' "$identity" > "$dir/state/.claude-autoarm.lock/pid-identity" +} + +# <dir> <epoch-seq> <owner-pid> <outcome>, aged well past any freshness window. +record_autoarm_epoch() { + local dir=$1 seq=$2 owner=$3 outcome=$4 + printf 'epoch=%s owner_pid=%s outcome=%s updated_at=1\n' "$seq" "$owner" "$outcome" \ + > "$dir/state/.claude-autoarm-epoch" + touch -t 202001010000 "$dir/state/.claude-autoarm-epoch" +} + +epoch_field() { + local dir=$1 field=$2 + sed -n "s/^.*[[:space:]]\{0,1\}$field=\([A-Za-z0-9_-]*\).*\$/\1/p" \ + "$dir/state/.claude-autoarm-epoch" 2>/dev/null || true +} + +test_abandoned_owner_claim_is_reclaimed_and_rearms() { + local dir out status pid + dir=$(make_primary_dir "$TMP_ROOT/abandoned-claim") + : > "$dir/state/task1.meta" + : > "$dir/state/task2.meta" + write_arm_fixture "$dir" actionable + sleep 60 & + pid=$! + record_autoarm_owner "$dir" "$pid" + record_autoarm_epoch "$dir" 464 "$pid" rewake + out=$(run_autoarm "$dir" 2>/dev/null); status=$? + kill "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + expect_code 2 "$status" "a claim whose ledger outcome is already terminal must be reclaimed, not deferred to forever" + [ -e "$dir/state/arm-ran" ] || fail "abandoned claim left the home unarmed with work in flight" + assert_contains "$out" "firstmate watcher wake" "the reclaimed cycle must still translate its wake" + [ "$(epoch_field "$dir" epoch)" -gt 464 ] || fail "reclaimed cycle did not advance the frozen ledger: $(epoch_field "$dir" epoch)" + [ "$(epoch_outcome "$dir")" = rewake ] || fail "reclaimed cycle did not record its own outcome: $(epoch_outcome "$dir")" + [ "$(epoch_field "$dir" owner_pid)" != "$pid" ] || fail "reclaimed ledger still names the abandoned owner" + assert_absent "$dir/state/.claude-autoarm.lock" "reclaimed cycle left an owner lock behind" + assert_absent "$dir/state/.claude-autoarm.lock.steal" "reclaim left its serialization mutex behind" + pass "auto-arm: an abandoned owner claim is reclaimed so a lapsed cycle re-arms" +} + +test_arming_claim_is_never_reclaimed() { + local dir out status pid + dir=$(make_primary_dir "$TMP_ROOT/arming-claim") + : > "$dir/state/task1.meta" + write_arm_fixture "$dir" actionable + sleep 60 & + pid=$! + record_autoarm_owner "$dir" "$pid" + # An owner foregrounds the arm for the whole watcher cycle, so "arming" is in + # progress no matter how old its ledger entry is. + record_autoarm_epoch "$dir" 464 "$pid" arming + out=$(run_autoarm "$dir" 2>/dev/null); status=$? + kill "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + expect_code 0 "$status" "a claim still arming must keep the single-flight gate closed" + [ -z "$out" ] || fail "deferring to an arming claim produced output: $out" + assert_absent "$dir/state/arm-ran" "an arming claim was stolen and double-armed" + [ "$(epoch_field "$dir" epoch)" = 464 ] || fail "deferred firing rewrote the arming ledger entry" + assert_present "$dir/state/.claude-autoarm.lock" "an arming claim lost its owner lock" + pass "auto-arm: an owner still arming is never reclaimed, however long the cycle runs" +} + +test_claim_not_named_by_the_ledger_is_never_reclaimed() { + local dir out status pid + dir=$(make_primary_dir "$TMP_ROOT/unnamed-claim") + : > "$dir/state/task1.meta" + write_arm_fixture "$dir" actionable + sleep 60 & + pid=$! + record_autoarm_owner "$dir" "$pid" + # A fresh claimant holds the lock before it writes "arming", so until it does + # the ledger still names the PREVIOUS owner. Requiring the two pids to match is + # what keeps that window from being mistaken for abandonment. + record_autoarm_epoch "$dir" 464 999 rewake + out=$(run_autoarm "$dir" 2>/dev/null); status=$? + kill "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + expect_code 0 "$status" "a live claim the ledger does not name is unproven and must be left alone" + [ -z "$out" ] || fail "deferring to an unnamed claim produced output: $out" + assert_absent "$dir/state/arm-ran" "a claim the ledger does not name was stolen and double-armed" + assert_present "$dir/state/.claude-autoarm.lock" "an unproven claim lost its owner lock" + pass "auto-arm: a live claim the ledger does not name is never reclaimed" +} + +# The same unrecoverable lapse, reached where the ledger cannot prove it: a session +# teardown kills the claim's whole process group before it records any outcome, so +# the entry still reads "arming" (in flight however old, by contract) while the +# recorded pid is later handed to an unrelated live process. Only the identity the +# claim recorded inside its own lock separates that from a real arm in progress. +test_pid_reused_arming_claim_is_reclaimed_and_rearms() { + local dir out status pid + dir=$(make_primary_dir "$TMP_ROOT/reused-pid-arming") + : > "$dir/state/task1.meta" + : > "$dir/state/task2.meta" + write_arm_fixture "$dir" actionable + sleep 60 & + pid=$! + record_autoarm_owner "$dir" "$pid" + record_autoarm_owner_identity "$dir" "$$" || fail "could not record a claim pid-identity" + record_autoarm_epoch "$dir" 464 "$pid" arming + out=$(run_autoarm "$dir" 2>/dev/null); status=$? + kill "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + expect_code 2 "$status" "a claim whose recorded identity no longer matches its live pid must be reclaimed, arming entry or not" + [ -e "$dir/state/arm-ran" ] || fail "a reused-pid claim left the home unarmed with work in flight" + assert_contains "$out" "firstmate watcher wake" "the reclaimed cycle must still translate its wake" + [ "$(epoch_field "$dir" epoch)" -gt 464 ] || fail "reclaimed cycle did not advance the frozen ledger: $(epoch_field "$dir" epoch)" + assert_absent "$dir/state/.claude-autoarm.lock" "reclaimed cycle left an owner lock behind" + assert_absent "$dir/state/.claude-autoarm.lock.steal" "reclaim left its serialization mutex behind" + pass "auto-arm: a claim whose pid was reused is reclaimed even while its ledger entry still reads arming" +} + +# The other ledger-blind shape: no ledger at all (a fresh or hand-cleared home) +# plus a reused pid. Without the recorded identity nothing proves abandonment, so +# every later firing exits at the lock and the home never re-arms. +test_pid_reused_claim_with_no_ledger_is_reclaimed_and_rearms() { + local dir out status pid + dir=$(make_primary_dir "$TMP_ROOT/reused-pid-no-ledger") + : > "$dir/state/task1.meta" + write_arm_fixture "$dir" actionable + sleep 60 & + pid=$! + record_autoarm_owner "$dir" "$pid" + record_autoarm_owner_identity "$dir" "$$" || fail "could not record a claim pid-identity" + assert_absent "$dir/state/.claude-autoarm-epoch" "this case must start with no ledger at all" + out=$(run_autoarm "$dir" 2>/dev/null); status=$? + kill "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + expect_code 2 "$status" "a reused-pid claim with no ledger to consult must still be reclaimed" + [ -e "$dir/state/arm-ran" ] || fail "a reused-pid claim with no ledger left the home unarmed" + assert_contains "$out" "firstmate watcher wake" "the reclaimed cycle must still translate its wake" + [ "$(epoch_outcome "$dir")" = rewake ] || fail "reclaimed cycle did not record its own outcome: $(epoch_outcome "$dir")" + assert_absent "$dir/state/.claude-autoarm.lock" "reclaimed cycle left an owner lock behind" + pass "auto-arm: a reused-pid claim is reclaimed even with no ledger entry to prove it" +} + +# The negative control for the identity leg: a claim whose recorded identity still +# matches the process holding the lock is genuinely in flight, so an arm that has +# legitimately been running for hours must keep the single-flight gate closed. +test_identity_matched_arming_claim_is_never_reclaimed() { + local dir out status pid + dir=$(make_primary_dir "$TMP_ROOT/identity-matched-arming") + : > "$dir/state/task1.meta" + write_arm_fixture "$dir" actionable + sleep 60 & + pid=$! + record_autoarm_owner "$dir" "$pid" + record_autoarm_owner_identity "$dir" "$pid" || fail "could not record a claim pid-identity" + record_autoarm_epoch "$dir" 464 "$pid" arming + out=$(run_autoarm "$dir" 2>/dev/null); status=$? + kill "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + expect_code 0 "$status" "an identity-matched claim still arming must keep the single-flight gate closed" + [ -z "$out" ] || fail "deferring to an identity-matched arming claim produced output: $out" + assert_absent "$dir/state/arm-ran" "an identity-matched arming claim was stolen and double-armed" + [ "$(epoch_field "$dir" epoch)" = 464 ] || fail "deferred firing rewrote the arming ledger entry" + assert_present "$dir/state/.claude-autoarm.lock" "an identity-matched arming claim lost its owner lock" + pass "auto-arm: an identity-matched owner still arming is never reclaimed" +} + +test_terminal_check_claim_is_never_reclaimed() { + local dir out status pid + dir=$(make_primary_dir "$TMP_ROOT/terminal-check-claim") + : > "$dir/state/task1.meta" + write_arm_fixture "$dir" actionable + sleep 60 & + pid=$! + # The synchronous guard takes the same lock under its own role while it decides + # the attended fail-open. Reclaiming that would race the guard's own decision. + record_autoarm_owner "$dir" "$pid" terminal-check + record_autoarm_epoch "$dir" 464 "$pid" failed-suppressed + out=$(run_autoarm "$dir" 2>/dev/null); status=$? + kill "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + expect_code 0 "$status" "the guard's own terminal-check claim must never be reclaimed by the arm hook" + [ -z "$out" ] || fail "deferring to a terminal-check claim produced output: $out" + assert_absent "$dir/state/arm-ran" "a terminal-check claim was stolen and double-armed" + assert_present "$dir/state/.claude-autoarm.lock" "a terminal-check claim lost its owner lock" + pass "auto-arm: the guard's terminal-check claim is never reclaimed" +} + test_need_vanished_mid_cycle_closes_quietly() { local dir out status dir=$(make_primary_dir "$TMP_ROOT/vanished") @@ -595,6 +804,13 @@ test_benign_cycle_end_with_live_watcher_is_silent test_positive_recovery_budget_contention_preserves_episode test_arms_for_x_mode_poll_need_without_inflight test_single_flight_admits_exactly_one_owner +test_abandoned_owner_claim_is_reclaimed_and_rearms +test_arming_claim_is_never_reclaimed +test_claim_not_named_by_the_ledger_is_never_reclaimed +test_pid_reused_arming_claim_is_reclaimed_and_rearms +test_pid_reused_claim_with_no_ledger_is_reclaimed_and_rearms +test_identity_matched_arming_claim_is_never_reclaimed +test_terminal_check_claim_is_never_reclaimed test_need_vanished_mid_cycle_closes_quietly test_afk_mid_cycle_suppresses_rewake test_active_in_marked_secondmate_home diff --git a/tests/fm-composer-lib.test.sh b/tests/fm-composer-lib.test.sh index e99c55ceb43..2d61ffda865 100755 --- a/tests/fm-composer-lib.test.sh +++ b/tests/fm-composer-lib.test.sh @@ -291,11 +291,12 @@ test_matrix_herdr_halfblock_rule_bounds_bare_wrap() { test_matrix_pi_separated_needs_identity() { # Real idle pi: a blank row between two solid rules. The blank row alone is # exactly what the strict rule refuses; only structure PLUS a live - # idle/done/blocked pi identity proves the composer (herdr's rule, now + # idle/done pi identity proves the composer (herdr's rule, now # fleet-wide; tmux supplies identity from its foreground-process probe). - local screen typed pi_idle pi_working none + local screen typed pi_idle pi_working pi_blocked none screen=$'transcript\n────────────────────────\n\n────────────────────────\n footer' pi_idle=$(printf 'pi\tidle'); pi_working=$(printf 'pi\tworking'); none=$(printf 'zsh\t') + pi_blocked=$(printf 'pi\tblocked') assert_screen "pi idle with identity" empty "$CAPS_STYLED" "$screen" '' "$pi_idle" assert_screen "pi idle on tmux with identity" empty "$CAPS_TMUX" "$screen" 2 "$pi_idle" assert_screen "pi idle on zellij" unknown "$CAPS_STYLED_NOID" "$screen" @@ -306,6 +307,10 @@ test_matrix_pi_separated_needs_identity() { assert_screen "pi pair without identity capability" unknown "$CAPS_PLAIN" "$screen" # A working pi cannot authorize injection into the blank region. assert_screen "working pi defers" unknown "$CAPS_STYLED" "$screen" '' "$pi_working" + # A pi parked on an interactive prompt reports `blocked`: it is waiting on a + # human keystroke, so the blank region is a menu's, not a free composer's. + # Typing there answers the prompt and the text is discarded (issue #2797). + assert_screen "blocked pi defers" unknown "$CAPS_STYLED" "$screen" '' "$pi_blocked" # The audit's live counterexample: a plain shell running sleep, cursor # parked on a blank line between two rules, NO pi process. The permissive # rule read this `empty`; identity+structure refuses it. diff --git a/tests/fm-daemon.test.sh b/tests/fm-daemon.test.sh index ad0925be4d4..f2b5201e4ca 100755 --- a/tests/fm-daemon.test.sh +++ b/tests/fm-daemon.test.sh @@ -220,6 +220,21 @@ test_stale_paused_classifies_pause() { pass "paused reasons with captain phrases remain pause-classified" } +# A verified captain-held transfer is the other declaration that leaves an idle pane +# EXPECTED, so it earns the same pause action as paused: rather than being aged as a +# wedge. The wait itself is already durable in the captain-held backlog task. +test_stale_captain_held_classifies_pause() { + local dir state out held_reason + dir=$(make_supercase stale-captain-held) + state="$dir/state" + held_reason='captain-held [key=route]: tracked by task-decision-route' + status_is_captain_relevant "$held_reason" && fail "a captain-held transfer line was treated as captain-relevant" + printf '%s\n' "$held_reason" > "$state/held-w9h.status" + out=$(FM_STATE_OVERRIDE="$state" classify_stale "sess:fm-held-w9h" "$state") + case "$out" in pause\|*) ;; *) fail "captain-held transfer did not classify as pause: $out" ;; esac + pass "a captain-held transfer classifies as pause, not as a wedge candidate" +} + # handle_wake on a paused stale records a pause marker, drops any pre-existing wedge # marker (so a working->paused pane is not still wedge-aged), and does NOT escalate # on the wake itself - the recheck is housekeeping's job on the long cadence. @@ -342,6 +357,7 @@ test_housekeeping_paused_resurfaces_and_resets() { PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$win" FM_FAKE_TMUX_CAPTURE="$pane" \ FM_STATE_OVERRIDE="$state" FM_PAUSE_RESURFACE_SECS=240 housekeeping "$state" grep -F "awaiting external" "$state/.subsuper-escalations" >/dev/null 2>&1 || fail "declared pause was not re-surfaced as an awaiting-external recheck" + grep -F "awaiting the captain" "$state/.subsuper-escalations" >/dev/null 2>&1 && fail "declared pause named the captain instead of its external dependency" grep -F "possible wedge" "$state/.subsuper-escalations" >/dev/null 2>&1 && fail "declared pause was mislabeled a possible wedge" [ -e "$state/.subsuper-paused-$key" ] || fail "pause marker cleared instead of reset for the next window" age=$(( $(date +%s) - $(cat "$state/.subsuper-paused-$key" 2>/dev/null || echo 0) )) @@ -349,6 +365,33 @@ test_housekeeping_paused_resurfaces_and_resets() { pass "housekeeping re-surfaces a stale declared pause on the long cadence and resets its window" } +# The other half of quieting a captain-held task: it must NOT be silenced outright. +# fm-classify-lib.sh's cadence comment is explicit that a forgotten hold cannot rot +# invisibly, so a held task re-surfaces on the same bounded window as a pause, with +# its marker reset so the window repeats instead of firing once. The digest the +# captain reads must also name the captain rather than an external dependency: the +# hold is waiting on the one person reading the digest, so borrowing the pause verb's +# awaiting-external wording would point them away from being the blocker. +test_housekeeping_captain_held_resurfaces_and_resets() { + local dir state fakebin win pane key age + dir=$(make_supercase captain-held-resurface) + state="$dir/state"; fakebin="$dir/fakebin" + win="sess:fm-held-w11h"; pane="$dir/pane.txt" + printf 'captain-held [key=route]: tracked by task-decision-route\n' > "$state/held-w11h.status" + printf 'idle prompt $\n' > "$pane" + key=$(printf '%s' "held-w11h" | tr ':/.' '___') + echo $(( $(date +%s) - 5000 )) > "$state/.subsuper-paused-$key" + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$win" FM_FAKE_TMUX_CAPTURE="$pane" \ + FM_STATE_OVERRIDE="$state" FM_PAUSE_RESURFACE_SECS=240 housekeeping "$state" + grep -F "awaiting the captain" "$state/.subsuper-escalations" >/dev/null 2>&1 || fail "a captain hold was silenced entirely instead of re-surfacing as a captain-owned recheck: $(cat "$state/.subsuper-escalations" 2>/dev/null || true)" + grep -F "awaiting external" "$state/.subsuper-escalations" >/dev/null 2>&1 && fail "a captain hold was re-surfaced as an external wait, hiding that the captain is the blocker" + grep -F "possible wedge" "$state/.subsuper-escalations" >/dev/null 2>&1 && fail "a captain hold was re-surfaced as a possible wedge" + [ -e "$state/.subsuper-paused-$key" ] || fail "captain-held marker cleared instead of reset for the next window" + age=$(( $(date +%s) - $(cat "$state/.subsuper-paused-$key" 2>/dev/null || echo 0) )) + [ "$age" -lt 60 ] || fail "captain-held marker was not reset to now on re-surface (age ${age}s)" + pass "housekeeping re-surfaces a forgotten captain hold on the long cadence and resets its window" +} + # A pause whose pane became busy again (the crew resumed) drops its marker without # escalating, exactly like a resumed wedge. test_housekeeping_paused_resumed_cleared() { @@ -390,6 +433,25 @@ test_housekeeping_paused_unpaused_cleared() { pass "housekeeping clears a paused marker once the crew is no longer declaring the pause" } +# Once the captain answers, the hold is no longer a declared wait: the resolved line +# takes over the last-line read, so the pause cadence must stop claiming the task +# rather than keep re-surfacing a settled decision. +test_housekeeping_captain_held_resolved_cleared() { + local dir state fakebin win pane key + dir=$(make_supercase captain-held-resolved) + state="$dir/state"; fakebin="$dir/fakebin" + win="sess:fm-held-w13h"; pane="$dir/pane.txt" + printf 'captain-held [key=route]: tracked by task-decision-route\nresolved [key=route]: captain chose the direct path\n' > "$state/held-w13h.status" + printf 'idle prompt $\n' > "$pane" + key=$(printf '%s' "held-w13h" | tr ':/.' '___') + echo $(( $(date +%s) - 5000 )) > "$state/.subsuper-paused-$key" + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$win" FM_FAKE_TMUX_CAPTURE="$pane" \ + FM_STATE_OVERRIDE="$state" FM_PAUSE_RESURFACE_SECS=240 housekeeping "$state" + [ -e "$state/.subsuper-paused-$key" ] && fail "an answered captain hold kept its pause marker" + [ ! -s "$state/.subsuper-escalations" ] || fail "an answered captain hold was re-surfaced as a declared wait" + pass "housekeeping clears the pause marker once a captain hold is answered" +} + test_housekeeping_stale_marker_transitions_to_pause() { local dir state fakebin win pane key dir=$(make_supercase stale-to-paused) @@ -406,6 +468,25 @@ test_housekeeping_stale_marker_transitions_to_pause() { pass "housekeeping moves an existing stale marker to pause before wedge escalation" } +# The quieting half for a captain hold. A finished task marked captain-held is idle by +# design, so an already-aged wedge marker converts to pause tracking on the next sweep +# instead of firing the possible-wedge escalation. +test_housekeeping_captain_held_stale_marker_transitions_to_pause() { + local dir state fakebin win pane key + dir=$(make_supercase stale-to-captain-held) + state="$dir/state"; fakebin="$dir/fakebin"; win="sess:fm-held-w14h"; pane="$dir/pane.txt" + printf 'captain-held [key=route]: tracked by task-decision-route\n' > "$state/held-w14h.status" + printf 'idle prompt $\n' > "$pane" + key=$(printf '%s' "held-w14h" | tr ':/.' '___') + echo $(( $(date +%s) - 5000 )) > "$state/.subsuper-stale-$key" + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$win" FM_FAKE_TMUX_CAPTURE="$pane" \ + FM_STATE_OVERRIDE="$state" FM_STALE_ESCALATE_SECS=240 housekeeping "$state" + [ -e "$state/.subsuper-paused-$key" ] || fail "a captain hold did not move its stale marker to pause tracking" + [ ! -e "$state/.subsuper-stale-$key" ] || fail "a captain hold remained wedge-aged" + [ ! -s "$state/.subsuper-escalations" ] || fail "a captain hold was escalated as a possible wedge" + pass "housekeeping moves a captain hold's existing stale marker to pause before wedge escalation" +} + test_housekeeping_pause_marker_transitions_to_clear() { local dir state fakebin win pane key dir=$(make_supercase paused-to-stale) @@ -1853,6 +1934,7 @@ test_stale_transient_self_records_marker test_stale_diagnostic_wedge_survives_busy_housekeeping test_stale_terminal_escalates test_stale_paused_classifies_pause +test_stale_captain_held_classifies_pause test_handle_wake_paused_records_pause_marker test_handle_wake_paused_signal_records_pause_marker test_handle_wake_terminal_signal_clears_pause_tracking @@ -1862,9 +1944,12 @@ test_housekeeping_seeds_pause_marker_from_status test_housekeeping_persistent_stale_escalates test_housekeeping_resumed_stale_cleared test_housekeeping_paused_resurfaces_and_resets +test_housekeeping_captain_held_resurfaces_and_resets test_housekeeping_paused_resumed_cleared test_housekeeping_paused_unpaused_cleared +test_housekeeping_captain_held_resolved_cleared test_housekeeping_stale_marker_transitions_to_pause +test_housekeeping_captain_held_stale_marker_transitions_to_pause test_housekeeping_pause_marker_transitions_to_clear test_housekeeping_herdr_persistent_stale_resolves_meta test_housekeeping_herdr_idle_busy_record_clears_stale diff --git a/tests/fm-decision-hold-lifecycle.test.sh b/tests/fm-decision-hold-lifecycle.test.sh deleted file mode 100755 index ad81510fb82..00000000000 --- a/tests/fm-decision-hold-lifecycle.test.sh +++ /dev/null @@ -1,1278 +0,0 @@ -#!/usr/bin/env bash -# End-to-end tests for durable captain-held decisions discovered by investigations -# and visual reviews. -set -u - -# shellcheck source=tests/lib.sh -# shellcheck disable=SC1091 -. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" - -TEARDOWN="$ROOT/bin/fm-teardown.sh" -BEARINGS="$ROOT/bin/fm-bearings-snapshot.sh" -TMP_ROOT=$(fm_test_tmproot fm-decision-hold) -TASKS_AXI_BIN=$(command -v tasks-axi || true) - -command -v jq >/dev/null 2>&1 || { echo "skip: jq not found"; exit 0; } -command -v tasks-axi >/dev/null 2>&1 || { echo "skip: tasks-axi not found"; exit 0; } - -make_home() { # <name> - local home="$TMP_ROOT/$1" fakebin - mkdir -p "$home/data" "$home/state" "$home/config" "$home/projects" - cp "$ROOT/.tasks.toml" "$home/.tasks.toml" - cat > "$home/data/backlog.md" <<'EOF' -## In flight - -## Queued - -## Done -EOF - fakebin=$(fm_fakebin "$home") - fm_fake_exit0 "$fakebin" tmux treehouse no-mistakes gh gh-axi - printf '%s\n' "$home" -} - -# The Lavish review adapter, run against this suite's isolated home. The -# machine-wide process-event claim root is redirected into the fixture so arming -# a review here can never contend with a real one on this machine. -run_lavish() { # <home> <command args...> - local home=$1 - shift - PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ - FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ - FM_PROCEVENT_CLAIM_ROOT="$home/procevent-claims" \ - "$ROOT/bin/fm-procevent-lavish.sh" "$@" -} - -run_procevent() { # <home> <command args...> - local home=$1 - shift - PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ - FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ - FM_PROCEVENT_CLAIM_ROOT="$home/procevent-claims" \ - "$ROOT/bin/fm-procevent.sh" "$@" -} - -run_bearings() { # <home> - local home=$1 - PATH="$home/fakebin:$PATH" FM_HOME="$home" FM_BEARINGS_NOW=2026-07-14T12:00:00Z \ - "$BEARINGS" --json -} - -run_teardown() { # <home> <id> - local home=$1 id=$2 - PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ - FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ - FM_CONFIG_OVERRIDE="$home/config" "$TEARDOWN" "$id" -} - -# Reproduces the loss exactly with privacy-safe synthetic names: the investigation -# and visual review have ended, the only genuine unresolved decision is report prose, -# no held backlog item or open status exists, and the authoritative Bearings view -# correctly omits it. Completion must now refuse before teardown can erase the source. -test_uninventoried_report_decision_refuses_completion() { - local home id json rc - home=$(make_home omitted-decision) - id=sample-route-review - mkdir -p "$home/data/$id" - cat > "$home/data/backlog.md" <<EOF -## In flight -- [ ] $id - Investigate sample routing (repo: sample) (kind: scout) (since 2026-07-14) - -## Queued - -## Done -EOF - fm_write_meta "$home/state/$id.meta" \ - "window=firstmate:fm-$id" \ - "worktree=$home/projects/missing-scratch" \ - "project=$home/projects/sample" \ - "harness=codex" \ - "kind=scout" \ - "mode=scout" - printf 'done: report and visual review complete\n' > "$home/state/$id.status" - cat > "$home/data/$id/report.md" <<'EOF' -# Sample route review - -The evidence is complete. -The captain still needs to choose route north or route south before follow-up work starts. -EOF - - json=$(run_bearings "$home") || fail "Bearings failed for unresolved-decision regression" - printf '%s' "$json" | jq -e ' - (.decisions_open | length) == 0 - and (.gates | length) == 0 - and (.reports | any(.id == "sample-route-review")) - ' >/dev/null || fail "the pre-policy omission shape was not reproduced: $json" - - set +e - run_teardown "$home" "$id" > "$home/teardown.out" 2> "$home/teardown.err" - rc=$? - set -e - [ "$rc" -ne 0 ] || fail "completed investigation teardown erased a report-only unresolved decision" - assert_present "$home/state/$id.meta" "refused completion must preserve investigation metadata" - assert_grep "REFUSED" "$home/teardown.err" "refusal must be explicit" - pass "report-only unresolved decision is reproduced and completion refuses before loss" -} - -tasks_in() { # <home> <tasks-axi args...> - local home=$1 - shift - (cd "$home" && tasks-axi "$@") -} - -run_decisions() { # <home> <command args...> - local home=$1 - shift - PATH="$home/fakebin:$PATH" REAL_TASKS_AXI="$TASKS_AXI_BIN" \ - FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ - FM_CONFIG_OVERRIDE="$home/config" "$ROOT/bin/fm-decision-hold.sh" "$@" -} - -write_origin_meta() { # <home> <id> [kind] - local home=$1 id=$2 kind=${3:-scout} - fm_write_meta "$home/state/$id.meta" \ - "window=firstmate:fm-$id" \ - "worktree=$home/projects/missing-$id" \ - "project=$home/projects/sample" \ - "harness=codex" \ - "kind=$kind" \ - "mode=$kind" -} - -test_structured_holds_survive_teardown_and_route_resolution() { - local home id route_hold access_hold before after json open show - home=$(make_home durable-lifecycle) - id=sample-systems-review - mkdir -p "$home/data/$id" - tasks_in "$home" add "$id" "Investigate sample systems" --kind scout --repo sample --start >/dev/null \ - || fail "could not create investigation backlog fixture" - write_origin_meta "$home" "$id" - cat > "$home/state/$id.status" <<'EOF' -needs-decision [key=route]: choose route north or route south -needs-decision [key=access]: choose open or restricted sample access -done: report and visual review complete -EOF - cat > "$home/data/$id/report.md" <<'EOF' -# Sample systems review - -Two choices remain unresolved: the route and the sample access level. -A separate recommendation is already resolved and requires no captain action. -EOF - - if run_decisions "$home" complete "$id" route access > "$home/early-complete.out" 2> "$home/early-complete.err"; then - fail "completion succeeded before unresolved decisions had captain holds" - fi - assert_no_grep "decisions_reviewed=1" "$home/state/$id.meta" \ - "failed completion recorded a false completion attestation" - - route_hold=$(run_decisions "$home" hold "$id" route \ - --title "Choose the sample route" --reason "captain route choice pending" --repo sample) \ - || fail "could not register route hold" - [ "$route_hold" = "$id-decision-route" ] || fail "route hold identity was not deterministic: $route_hold" - run_decisions "$home" hold "$id" route \ - --title "Choose the sample route" --reason "captain route choice pending" --repo sample >/dev/null \ - || fail "idempotent hold retry failed" - if run_decisions "$home" complete "$id" route access > "$home/partial-complete.out" 2> "$home/partial-complete.err"; then - fail "completion succeeded while one of two distinct decisions lacked a hold" - fi - access_hold=$(run_decisions "$home" hold "$id" access \ - --title "Choose the sample access level" --reason "captain access choice pending" --repo sample) \ - || fail "could not register access hold" - [ "$access_hold" = "$id-decision-access" ] || fail "access hold identity was not distinct: $access_hold" - [ "$(grep -cE "^- \[ \] $route_hold -" "$home/data/backlog.md")" = 1 ] \ - || fail "idempotent retry duplicated the route hold" - [ "$(grep -cE "^- \[ \] $access_hold -" "$home/data/backlog.md")" = 1 ] \ - || fail "second decision did not retain one distinct backlog identity" - - FM_STATE_OVERRIDE="$home/state" bash -c ' - . "$1" - sig=$(fm_wake_signal_sig "$3") || exit 1 - printf "%s" "$sig" > "$(fm_wake_signal_seen_path "$2" "$3")" - ' _ "$ROOT/bin/fm-wake-lib.sh" "$home/state" "$home/state/$id.status" \ - || fail "could not prime the announced decision baseline" - run_decisions "$home" complete "$id" route access >/dev/null \ - || fail "shared investigation completion gate failed" - FM_STATE_OVERRIDE="$home/state" bash -c ' - . "$1"; fm_wake_signal_seen_current "$2" "$3" - ' _ "$ROOT/bin/fm-wake-lib.sh" "$home/state" "$home/state/$id.status" \ - || fail "captain-held bookkeeping closes re-woke their own home" - assert_grep "decisions_reviewed=1" "$home/state/$id.meta" "completion attestation missing" - assert_grep "decision_keys=access,route" "$home/state/$id.meta" "decision inventory was not deterministic" - open=$(bash -c '. "$1"; status_open_decisions "$2"' _ \ - "$ROOT/bin/fm-classify-lib.sh" "$home/state/$id.status") - [ -z "$open" ] || fail "captain-held transfer did not close duplicate live status decisions: $open" - - before=$(shasum -a 256 "$home/data/backlog.md" | awk '{print $1}') - json=$(run_bearings "$home") || fail "Bearings failed with captain-held decisions" - after=$(shasum -a 256 "$home/data/backlog.md" | awk '{print $1}') - [ "$before" = "$after" ] || fail "Bearings mutated the authoritative backlog" - printf '%s' "$json" | jq -e --arg route "$route_hold" --arg access "$access_hold" ' - (.decisions_open | any(.id == $route and .verb == "captain-hold" and .owner == "(main)")) - and (.decisions_open | any(.id == $access and .verb == "captain-hold" and .owner == "(main)")) - and (.gates | any(.id == $route or .id == $access) | not) - ' >/dev/null || fail "Bearings did not surface structured captain holds: $json" - - run_teardown "$home" "$id" >/dev/null 2> "$home/teardown.err" \ - || fail "reviewed investigation teardown failed: $(cat "$home/teardown.err")" - tasks_in "$home" "done" "$id" --report "data/$id/report.md" --keep 0 >/dev/null \ - || fail "could not archive completed investigation" - ! grep -E "^- \[[ x]\] $id -" "$home/data/backlog.md" >/dev/null \ - || fail "origin remained in the live backlog after archival" - grep -E "^- \[x\] $id -" "$home/data/done-archive.md" >/dev/null \ - || fail "origin was not durably archived" - json=$(run_bearings "$home") || fail "Bearings failed after source teardown and archival" - printf '%s' "$json" | jq -e --arg route "$route_hold" --arg access "$access_hold" ' - (.decisions_open | any(.id == $route and .verb == "captain-hold")) - and (.decisions_open | any(.id == $access and .verb == "captain-hold")) - and (.in_flight | any(.id == "sample-systems-review") | not) - ' >/dev/null || fail "teardown or archival erased a captain-held decision: $json" - - tasks_in "$home" add sample-route-implementation "Apply the selected sample route" \ - --kind ship --repo sample >/dev/null \ - || fail "could not create dependent work fixture" - printf 'Use route north for the sample system.\n' > "$home/route-decision.txt" - if run_decisions "$home" resolve "$id" route --decision-file "$home/route-decision.txt" \ - --routed-to sample-route-implementation > "$home/early-resolve.out" 2> "$home/early-resolve.err"; then - fail "captain hold closed before dependent work had a durable routing edge" - fi - show=$(tasks_in "$home" show "$route_hold" --full) - assert_contains "$show" "state: queued" "failed routing attempt closed the hold" - assert_contains "$show" "held: yes" "failed routing attempt released the hold" - tasks_in "$home" block sample-route-implementation --by "$route_hold" >/dev/null \ - || fail "could not route dependent work behind the decision hold" - tasks_in "$home" add sample-route-followup "Check the selected sample route" \ - --kind ship --repo sample --blocked-by "$route_hold" >/dev/null \ - || fail "could not create second dependent work fixture" - cat > "$home/fakebin/tasks-axi" <<'EOF' -#!/usr/bin/env bash -if [ "${1:-}" = unblock ] && [ "${2:-}" = sample-route-implementation ] \ - && [ ! -f "$FM_HOME/unblock-failed-once" ]; then - : > "$FM_HOME/unblock-failed-once" - exit 1 -fi -exec "$REAL_TASKS_AXI" "$@" -EOF - chmod +x "$home/fakebin/tasks-axi" - if run_decisions "$home" resolve "$id" route --decision-file "$home/route-decision.txt" \ - --routed-to sample-route-implementation --routed-to sample-route-followup \ - > "$home/partial-route.out" 2> "$home/partial-route.err"; then - fail "resolution succeeded after a partial dependent-routing failure" - fi - show=$(tasks_in "$home" show "$route_hold" --full) - assert_contains "$show" "state: queued" "partial routing failure closed the hold" - show=$(tasks_in "$home" show sample-route-followup --full) - assert_contains "$show" "blocked: no" "partial routing fixture did not release its first dependent" - show=$(tasks_in "$home" show sample-route-implementation --full) - assert_contains "$show" "blocked: yes" "partial routing fixture unexpectedly released its second dependent" - if run_decisions "$home" resolve "$id" route --decision-file "$home/route-decision.txt" \ - --routed-to sample-route-followup > "$home/reduced-retry.out" 2> "$home/reduced-retry.err"; then - fail "partial resolution retry accepted a reduced routed task set" - fi - printf 'Use route south for the sample system.\n' > "$home/changed-route-decision.txt" - if run_decisions "$home" resolve "$id" route --decision-file "$home/changed-route-decision.txt" \ - --routed-to sample-route-implementation --routed-to sample-route-followup \ - > "$home/partial-drifted-decision.out" 2> "$home/partial-drifted-decision.err"; then - fail "partial resolution retry accepted a different captain decision" - fi - tasks_in "$home" "done" sample-route-followup >/dev/null \ - || fail "could not complete already-routed dependent work" - run_decisions "$home" resolve "$id" route --decision-file "$home/route-decision.txt" \ - --routed-to sample-route-implementation --routed-to sample-route-followup >/dev/null \ - || fail "could not resume and complete partial decision routing" - run_decisions "$home" resolve "$id" route --decision-file "$home/route-decision.txt" \ - --routed-to sample-route-implementation --routed-to sample-route-followup >/dev/null \ - || fail "identical resolution retry was not idempotent" - if run_decisions "$home" resolve "$id" route --decision-file "$home/changed-route-decision.txt" \ - --routed-to sample-route-implementation --routed-to sample-route-followup \ - > "$home/drifted-decision.out" 2> "$home/drifted-decision.err"; then - fail "resolution retry accepted a different captain decision" - fi - if run_decisions "$home" resolve "$id" route --decision-file "$home/route-decision.txt" \ - --routed-to sample-route-implementation \ - > "$home/drifted-routes.out" 2> "$home/drifted-routes.err"; then - fail "resolution retry accepted a different routed task set" - fi - show=$(tasks_in "$home" show "$route_hold" --full) - assert_contains "$show" "state: done" "resolved hold did not close" - assert_contains "$show" "Resolution recorded by fm-decision-hold" "resolved hold lost the decision record" - show=$(tasks_in "$home" show sample-route-implementation --full) - assert_contains "$show" "blocked: no" "recorded decision did not release dependent work" - json=$(run_bearings "$home") || fail "Bearings failed after decision resolution" - printf '%s' "$json" | jq -e --arg route "$route_hold" --arg access "$access_hold" ' - (.decisions_open | any(.id == $route) | not) - and (.decisions_open | any(.id == $access and .verb == "captain-hold")) - and (.gates | any(.id == "sample-route-implementation")) - and (.decisions_open | any(.id == "sample-systems-review") | not) - ' >/dev/null || fail "resolved or decision-like report prose produced a false hold: $json" - pass "captain holds are idempotent, distinct, teardown-safe, Bearings-visible, and durably routed before close" -} - -test_scout_teardown_always_requires_inventory_verification() { - local home id - home=$(make_home unconditional-teardown) - id=sample-absent-review - mkdir -p "$home/data/$id" - write_origin_meta "$home" "$id" - printf '# Sample absent review\n\nNo decision inventory was recorded.\n' > "$home/data/$id/report.md" - if run_teardown "$home" "$id" > "$home/absent-teardown.out" 2> "$home/absent-teardown.err"; then - fail "scout teardown skipped verification when its backlog task was absent" - fi - assert_present "$home/state/$id.meta" "refused absent-task teardown removed metadata" - - home=$(make_home unavailable-teardown) - id=sample-unavailable-review - mkdir -p "$home/data/$id" - write_origin_meta "$home" "$id" - printf '# Sample unavailable review\n\nNo decision inventory was recorded.\n' > "$home/data/$id/report.md" - cat > "$home/fakebin/tasks-axi" <<'EOF' -#!/usr/bin/env bash -exit 127 -EOF - chmod +x "$home/fakebin/tasks-axi" - if run_teardown "$home" "$id" > "$home/unavailable-teardown.out" 2> "$home/unavailable-teardown.err"; then - fail "scout teardown skipped verification when tasks-axi was unavailable" - fi - assert_present "$home/state/$id.meta" "refused unavailable-task teardown removed metadata" - pass "non-forced scout teardown always requires durable inventory verification" -} - -test_origin_slug_validation_precedes_path_construction() { - local home escaped - home=$(make_home origin-validation) - escaped="$home/escaped-origin.meta" - printf 'sentinel=unchanged\n' > "$escaped" - if run_decisions "$home" complete ../escaped-origin --none \ - > "$home/invalid-complete.out" 2> "$home/invalid-complete.err"; then - fail "completion accepted an origin path traversal" - fi - if run_decisions "$home" verify ../escaped-origin \ - > "$home/invalid-verify.out" 2> "$home/invalid-verify.err"; then - fail "verification accepted an origin path traversal" - fi - [ "$(cat "$escaped")" = "sentinel=unchanged" ] \ - || fail "invalid origin changed metadata outside the state directory" - pass "completion and verification validate origins before constructing paths" -} - -test_visual_review_uses_shared_completion_owner() { - local home id hold json - home=$(make_home visual-review) - id=sample-board-review - mkdir -p "$home/data/$id" - tasks_in "$home" add "$id" "Review the sample board" --kind scout --repo sample --start >/dev/null - write_origin_meta "$home" "$id" - printf 'done: investigation complete\n' > "$home/state/$id.status" - printf '# Sample board investigation\n\nThe initial findings need no captain choice.\n' > "$home/data/$id/report.md" - run_decisions "$home" complete "$id" --none >/dev/null \ - || fail "initial investigation could not pass the shared completion owner" - run_teardown "$home" "$id" >/dev/null 2> "$home/visual-teardown.err" \ - || fail "completed investigation teardown failed: $(cat "$home/visual-teardown.err")" - tasks_in "$home" "done" "$id" --report "data/$id/report.md" --keep 0 >/dev/null - - mkdir -p "$home/.lavish" - printf '<html><body>Synthetic sample board</body></html>\n' > "$home/.lavish/sample-board.html" - hold=$(run_decisions "$home" hold "$id" layout \ - --title "Choose the sample layout" --reason "captain layout choice pending" --repo sample) \ - || fail "post-teardown visual review could not use the shared hold owner" - run_decisions "$home" complete "$id" layout >/dev/null \ - || fail "post-teardown visual review could not use the shared completion owner" - [ "$hold" = "$id-decision-layout" ] || fail "visual review used a separate identity policy" - json=$(run_bearings "$home") || fail "Bearings failed after the ended visual review" - printf '%s' "$json" | jq -e --arg hold "$hold" ' - .decisions_open | any(.id == $hold and .verb == "captain-hold") - ' >/dev/null || fail "ended visual review did not leave its durable Captain Call: $json" - [ ! -e "$home/data/visual-review-decisions.json" ] \ - || fail "visual review created a second decision database" - pass "ended visual review follows the same decision-hold completion owner" -} - -test_none_inventory_and_resolved_prose_do_not_create_holds() { - local home id json - home=$(make_home no-false-holds) - id=sample-resolved-review - mkdir -p "$home/data/$id" - tasks_in "$home" add "$id" "Review a resolved sample finding" --kind scout --repo sample --start >/dev/null - write_origin_meta "$home" "$id" - printf 'resolved [key=old-choice]: the sample choice was already recorded\ndone: report complete\n' \ - > "$home/state/$id.status" - cat > "$home/data/$id/report.md" <<'EOF' -# Resolved sample finding - -Decision record: the earlier choice is resolved. -The recommendation is informational and needs no captain action. -EOF - run_decisions "$home" complete "$id" --none >/dev/null \ - || fail "explicit no-decision inventory failed" - json=$(run_bearings "$home") || fail "Bearings failed for no-decision inventory" - printf '%s' "$json" | jq -e ' - (.decisions_open | any(.id | startswith("sample-resolved-review")) | not) - ' >/dev/null || fail "resolved findings or decision-like prose created a false hold: $json" - pass "resolved findings and decision-like prose do not create false holds" -} - -test_terminal_single_owner_status_decision_does_not_block_empty_inventory() { - local home id open secondmate - home=$(make_home stale-terminal-decision) - id=sample-terminal-review - mkdir -p "$home/data/$id" - tasks_in "$home" add "$id" "Review a terminal sample finding" --kind scout --repo sample --start >/dev/null - write_origin_meta "$home" "$id" - printf 'needs-decision [key=default]: choose route A or route B\ndone: report complete\n' \ - > "$home/state/$id.status" - printf '# Terminal sample review\n\nNo unresolved captain choice remains.\n' > "$home/data/$id/report.md" - open=$(bash -c '. "$1"; status_open_decisions "$2"' _ \ - "$ROOT/bin/fm-classify-lib.sh" "$home/state/$id.status") - assert_contains "$open" "default" "fixture must retain the raw stale status decision" - run_decisions "$home" complete "$id" --none >/dev/null \ - || fail "terminal single-owner stale status decision blocked empty inventory completion" - run_decisions "$home" verify "$id" >/dev/null \ - || fail "terminal single-owner stale status decision blocked inventory verification" - run_teardown "$home" "$id" >/dev/null 2> "$home/terminal-teardown.err" \ - || fail "terminal single-owner stale status decision blocked teardown: $(cat "$home/terminal-teardown.err")" - - secondmate=sample-secondmate - write_origin_meta "$home" "$secondmate" secondmate - printf 'needs-decision [key=route]: choose route A or route B\ndone: heartbeat complete\n' \ - > "$home/state/$secondmate.status" - if run_decisions "$home" complete "$secondmate" --none \ - > "$home/secondmate-terminal.out" 2> "$home/secondmate-terminal.err"; then - fail "secondmate terminal status decision was incorrectly cleared" - fi - pass "terminal single-owner stale status decisions do not block empty inventory" -} - -test_secondmate_hold_stays_in_authoritative_home() { - local parent mate origin hold json - parent=$(make_home main-routing) - mate="$TMP_ROOT/sample-mate-home" - mkdir -p "$mate/data" "$mate/state" "$mate/config" "$mate/projects" "$mate/bin" - cp "$ROOT/.tasks.toml" "$mate/.tasks.toml" - printf '# Synthetic secondmate home\n' > "$mate/AGENTS.md" - printf 'sample-mate\n' > "$mate/.fm-secondmate-home" - cat > "$mate/data/backlog.md" <<'EOF' -## In flight - -## Queued - -## Done -EOF - fakebin=$(fm_fakebin "$mate") - fm_fake_exit0 "$fakebin" tmux treehouse no-mistakes gh gh-axi - origin=sample-mate-review - mkdir -p "$mate/data/$origin" - tasks_in "$mate" add "$origin" "Investigate secondmate sample" --kind scout --repo sample --start >/dev/null - write_origin_meta "$mate" "$origin" - printf 'done: report and visual review complete\n' > "$mate/state/$origin.status" - printf '# Sample secondmate review\n\nOne captain choice remains.\n' > "$mate/data/$origin/report.md" - hold=$(run_decisions "$mate" hold "$origin" release \ - --title "Choose the sample release" --reason "captain release choice pending" --repo sample) \ - || fail "secondmate-owned hold creation failed" - run_decisions "$mate" complete "$origin" release >/dev/null \ - || fail "secondmate-owned completion failed" - run_teardown "$mate" "$origin" >/dev/null 2> "$mate/teardown.err" \ - || fail "secondmate investigation teardown failed: $(cat "$mate/teardown.err")" - tasks_in "$mate" "done" "$origin" --report "data/$origin/report.md" --keep 0 >/dev/null - - printf -- '- sample-mate - synthetic scope (home: %s; scope: sample reviews; projects: sample; added 2026-07-14)\n' \ - "$mate" > "$parent/data/secondmates.md" - fm_write_secondmate_meta "$parent/state/sample-mate.meta" "$mate" \ - "firstmate:fm-sample-mate" sample - json=$(run_bearings "$parent") || fail "parent Bearings could not read secondmate hold" - printf '%s' "$json" | jq -e --arg hold "$hold" ' - .decisions_open | any(.owner == "sample-mate" and .verb == "captain-hold" and (.id | endswith($hold))) - ' >/dev/null || fail "secondmate captain hold did not surface with authoritative owner: $json" - assert_no_grep "$hold" "$parent/data/backlog.md" "secondmate hold leaked into the main backlog" - assert_grep "$hold" "$mate/data/backlog.md" "secondmate hold left its authoritative backlog" - pass "main-home and secondmate-home captain holds remain correctly routed" -} - -# tasks-axi quotes multi-entry blocked_by values as "a,b,c". resolve must strip -# those surrounding quotes before comma-boundary membership so the first and last -# list elements match, not only middle elements. -test_resolve_matches_quoted_blocked_by_edges() { - local home origin hold_first hold_mid hold_last hold_absent show - home=$(make_home quoted-blocked-by-edges) - origin=sample-quote-review - mkdir -p "$home/data/$origin" - tasks_in "$home" add "$origin" "Quoted blocked_by edge review" --kind scout --repo sample --start >/dev/null \ - || fail "could not create quote-edge origin" - write_origin_meta "$home" "$origin" - printf 'done: report complete\n' > "$home/state/$origin.status" - printf '# Quote edge review\n\nThree edge decisions and one absent control.\n' > "$home/data/$origin/report.md" - - hold_first=$(run_decisions "$home" hold "$origin" edge-first \ - --title "First edge decision" --reason "captain first pending" --repo sample) \ - || fail "could not register first-edge hold" - hold_mid=$(run_decisions "$home" hold "$origin" edge-mid \ - --title "Middle edge decision" --reason "captain mid pending" --repo sample) \ - || fail "could not register mid-edge hold" - hold_last=$(run_decisions "$home" hold "$origin" edge-last \ - --title "Last edge decision" --reason "captain last pending" --repo sample) \ - || fail "could not register last-edge hold" - hold_absent=$(run_decisions "$home" hold "$origin" edge-absent \ - --title "Absent edge decision" --reason "captain absent pending" --repo sample) \ - || fail "could not register absent-edge hold" - - tasks_in "$home" add pad-a "Pad A" --kind ship --repo sample >/dev/null \ - || fail "could not create pad-a blocker" - tasks_in "$home" add pad-b "Pad B" --kind ship --repo sample >/dev/null \ - || fail "could not create pad-b blocker" - - tasks_in "$home" add dep-first "Dep first position" --kind ship --repo sample >/dev/null \ - || fail "could not create first-position dependent" - tasks_in "$home" block dep-first --by "$hold_first" >/dev/null || fail "could not block dep-first by first hold" - tasks_in "$home" block dep-first --by pad-a >/dev/null || fail "could not block dep-first by pad-a" - tasks_in "$home" block dep-first --by pad-b >/dev/null || fail "could not block dep-first by pad-b" - show=$(tasks_in "$home" show dep-first --full) - assert_contains "$show" "blocked_by: \"$hold_first,pad-a,pad-b\"" \ - "first-position fixture must quote multi-entry blocked_by" - printf 'Decide first edge.\n' > "$home/d-first.txt" - if ! run_decisions "$home" resolve "$origin" edge-first --decision-file "$home/d-first.txt" \ - --routed-to dep-first > "$home/first.out" 2> "$home/first.err"; then - fail "resolve failed when hold id is FIRST in quoted blocked_by: $(cat "$home/first.err")" - fi - - tasks_in "$home" add dep-mid "Dep mid position" --kind ship --repo sample >/dev/null \ - || fail "could not create mid-position dependent" - tasks_in "$home" block dep-mid --by pad-a >/dev/null || fail "could not block dep-mid by pad-a" - tasks_in "$home" block dep-mid --by "$hold_mid" >/dev/null || fail "could not block dep-mid by mid hold" - tasks_in "$home" block dep-mid --by pad-b >/dev/null || fail "could not block dep-mid by pad-b" - show=$(tasks_in "$home" show dep-mid --full) - assert_contains "$show" "blocked_by: \"pad-a,$hold_mid,pad-b\"" \ - "middle-position fixture must quote multi-entry blocked_by" - printf 'Decide mid edge.\n' > "$home/d-mid.txt" - if ! run_decisions "$home" resolve "$origin" edge-mid --decision-file "$home/d-mid.txt" \ - --routed-to dep-mid > "$home/mid.out" 2> "$home/mid.err"; then - fail "resolve failed when hold id is MIDDLE in quoted blocked_by: $(cat "$home/mid.err")" - fi - - tasks_in "$home" add dep-last "Dep last position" --kind ship --repo sample >/dev/null \ - || fail "could not create last-position dependent" - tasks_in "$home" block dep-last --by pad-a >/dev/null || fail "could not block dep-last by pad-a" - tasks_in "$home" block dep-last --by pad-b >/dev/null || fail "could not block dep-last by pad-b" - tasks_in "$home" block dep-last --by "$hold_last" >/dev/null || fail "could not block dep-last by last hold" - show=$(tasks_in "$home" show dep-last --full) - assert_contains "$show" "blocked_by: \"pad-a,pad-b,$hold_last\"" \ - "last-position fixture must quote multi-entry blocked_by" - printf 'Decide last edge.\n' > "$home/d-last.txt" - if ! run_decisions "$home" resolve "$origin" edge-last --decision-file "$home/d-last.txt" \ - --routed-to dep-last > "$home/last.out" 2> "$home/last.err"; then - fail "resolve failed when hold id is LAST in quoted blocked_by: $(cat "$home/last.err")" - fi - - tasks_in "$home" add dep-absent "Dep absent control" --kind ship --repo sample >/dev/null \ - || fail "could not create absent-control dependent" - tasks_in "$home" block dep-absent --by pad-a >/dev/null || fail "could not block dep-absent by pad-a" - tasks_in "$home" block dep-absent --by pad-b >/dev/null || fail "could not block dep-absent by pad-b" - show=$(tasks_in "$home" show dep-absent --full) - assert_contains "$show" "blocked_by: \"pad-a,pad-b\"" \ - "absent-control fixture must quote multi-entry blocked_by without the hold id" - printf 'Decide absent edge.\n' > "$home/d-absent.txt" - if run_decisions "$home" resolve "$origin" edge-absent --decision-file "$home/d-absent.txt" \ - --routed-to dep-absent > "$home/absent.out" 2> "$home/absent.err"; then - fail "resolve succeeded when hold id is genuinely absent from blocked_by" - fi - assert_grep "not durably blocked by" "$home/absent.err" \ - "absent id must fail with durable-block error" - show=$(tasks_in "$home" show "$hold_absent" --full) - assert_contains "$show" "state: queued" "failed absent resolve must leave the hold open" - assert_contains "$show" "held: yes" "failed absent resolve must leave the hold held" - - pass "resolve matches first/middle/last in quoted blocked_by and rejects a genuinely absent id" -} - -# A captain who declines a held decision leaves no follow-up work to route, so the -# routed close path cannot express the answer. The unrouted close path must record -# that answer durably while still refusing to release work the hold blocks. -test_declined_decision_closes_without_routed_work() { - local home id hold routed_hold json show - home=$(make_home declined-decision) - id=sample-benchmark-review - mkdir -p "$home/data/$id" - tasks_in "$home" add "$id" "Investigate sample benchmarks" --kind scout --repo sample --start >/dev/null \ - || fail "could not create declined-decision origin" - write_origin_meta "$home" "$id" - printf 'done: report complete\n' > "$home/state/$id.status" - printf '# Sample benchmark review\n\nOne captain choice remains.\n' > "$home/data/$id/report.md" - hold=$(run_decisions "$home" hold "$id" half-run \ - --title "Choose the sample half run" --reason "captain half-run choice pending" --repo sample) \ - || fail "could not register the declinable hold" - run_decisions "$home" complete "$id" half-run >/dev/null \ - || fail "completion failed for the declinable hold" - - printf '' > "$home/empty-decision.txt" - if run_decisions "$home" decline "$id" half-run --decision-file "$home/empty-decision.txt" \ - > "$home/empty-decline.out" 2> "$home/empty-decline.err"; then - fail "decline accepted an empty captain decision" - fi - if run_decisions "$home" decline "$id" half-run > "$home/bare-decline.out" 2> "$home/bare-decline.err"; then - fail "decline accepted a close with no captain decision file at all" - fi - show=$(tasks_in "$home" show "$hold" --full) - assert_contains "$show" "state: queued" "a refused decline closed the hold" - assert_contains "$show" "held: yes" "a refused decline released the hold" - - printf 'Declined: do not run the sample half benchmark.\n' > "$home/half-run-decision.txt" - run_decisions "$home" decline "$id" half-run --decision-file "$home/half-run-decision.txt" >/dev/null \ - || fail "decline could not close a hold that routes no work" - show=$(tasks_in "$home" show "$hold" --full) - assert_contains "$show" "state: done" "declined hold did not close" - assert_contains "$show" "Resolution recorded by fm-decision-hold" "declined hold lost the decision record" - assert_contains "$show" "Resolution mode: declined" "declined hold did not record its close path" - assert_contains "$show" "Declined: do not run the sample half benchmark." \ - "declined hold did not record the captain decision text" - run_decisions "$home" verify "$id" >/dev/null \ - || fail "a declined decision did not satisfy the completion gate" - run_decisions "$home" decline "$id" half-run --decision-file "$home/half-run-decision.txt" >/dev/null \ - || fail "identical decline retry was not idempotent" - printf 'Declined for a different reason.\n' > "$home/drifted-decision.txt" - if run_decisions "$home" decline "$id" half-run --decision-file "$home/drifted-decision.txt" \ - > "$home/drifted-decline.out" 2> "$home/drifted-decline.err"; then - fail "decline retry accepted a different captain decision" - fi - json=$(run_bearings "$home") || fail "Bearings failed after a declined decision" - printf '%s' "$json" | jq -e --arg hold "$hold" ' - (.decisions_open | any(.id == $hold) | not) - ' >/dev/null || fail "a declined decision remained an open Captain's Call: $json" - - routed_hold=$(run_decisions "$home" hold "$id" upstream \ - --title "Choose the sample upstream target" --reason "captain upstream choice pending" --repo sample) \ - || fail "could not register the routed-work hold" - tasks_in "$home" add sample-upstream-work "Apply the sample upstream choice" \ - --kind ship --repo sample --blocked-by "$routed_hold" >/dev/null \ - || fail "could not route work behind the second hold" - if run_decisions "$home" decline "$id" upstream --decision-file "$home/half-run-decision.txt" \ - > "$home/routed-decline.out" 2> "$home/routed-decline.err"; then - fail "decline released work that was still routed behind the hold" - fi - assert_grep "still blocks routed work" "$home/routed-decline.err" \ - "decline must name the routed work it refuses to release" - show=$(tasks_in "$home" show "$routed_hold" --full) - assert_contains "$show" "state: queued" "refused routed decline closed the hold" - show=$(tasks_in "$home" show sample-upstream-work --full) - assert_contains "$show" "blocked: yes" "refused routed decline released dependent work" - if run_decisions "$home" resolve "$id" upstream --decision-file "$home/half-run-decision.txt" \ - > "$home/unrouted-resolve.out" 2> "$home/unrouted-resolve.err"; then - fail "the routed close path accepted a resolution with no routed work" - fi - pass "a declined decision closes with a recorded answer and no routed work" -} - -# The exact incident: two declined captain decisions were closed with a direct -# tasks-axi done, so the durable resolution attestation this gate reads was never -# written and the investigation could no longer be cleaned up. -test_out_of_band_close_is_repairable_before_teardown() { - local home id hold show - home=$(make_home out-of-band-close) - id=sample-fullrun-review - mkdir -p "$home/data/$id" - tasks_in "$home" add "$id" "Investigate the sample full run" --kind scout --repo sample --start >/dev/null \ - || fail "could not create out-of-band-close origin" - write_origin_meta "$home" "$id" - printf 'done: report complete\n' > "$home/state/$id.status" - printf '# Sample full run review\n\nOne captain choice remains.\n' > "$home/data/$id/report.md" - hold=$(run_decisions "$home" hold "$id" submission \ - --title "Choose the sample submission" --reason "captain submission choice pending" --repo sample) \ - || fail "could not register the out-of-band hold" - run_decisions "$home" complete "$id" submission >/dev/null \ - || fail "completion failed before the out-of-band close" - - tasks_in "$home" "done" "$hold" >/dev/null || fail "could not reproduce the direct out-of-band close" - show=$(tasks_in "$home" show "$hold" --full) - assert_contains "$show" "state: done" "the out-of-band close shape was not reproduced" - assert_no_grep "Resolution recorded by fm-decision-hold" "$home/data/backlog.md" \ - "the out-of-band close must leave no durable resolution record" - if run_decisions "$home" verify "$id" > "$home/broken-verify.out" 2> "$home/broken-verify.err"; then - fail "verification passed a captain decision closed with no recorded answer" - fi - if run_teardown "$home" "$id" > "$home/broken-teardown.out" 2> "$home/broken-teardown.err"; then - fail "teardown proceeded while a captain decision had no recorded answer" - fi - assert_present "$home/state/$id.meta" "refused teardown removed investigation metadata" - - if run_decisions "$home" repair "$id" submission > "$home/bare-repair.out" 2> "$home/bare-repair.err"; then - fail "repair recorded a resolution with no captain decision file" - fi - printf '' > "$home/empty-repair.txt" - if run_decisions "$home" repair "$id" submission --decision-file "$home/empty-repair.txt" \ - > "$home/empty-repair.out" 2> "$home/empty-repair.err"; then - fail "repair recorded a resolution from an empty captain decision file" - fi - if run_decisions "$home" verify "$id" > "$home/still-broken.out" 2> "$home/still-broken.err"; then - fail "a refused repair still satisfied the completion gate" - fi - - printf 'Declined: do not submit the sample full run upstream.\n' > "$home/submission-decision.txt" - run_decisions "$home" repair "$id" submission --decision-file "$home/submission-decision.txt" >/dev/null \ - || fail "repair could not record the missing durable resolution" - show=$(tasks_in "$home" show "$hold" --full) - assert_contains "$show" "state: done" "repair reopened a closed captain decision" - assert_contains "$show" "Resolution mode: repaired" "repair did not record its close path" - assert_contains "$show" "Declined: do not submit the sample full run upstream." \ - "repair did not record the captain decision text" - run_decisions "$home" verify "$id" >/dev/null \ - || fail "the repaired decision did not satisfy the completion gate" - run_decisions "$home" repair "$id" submission --decision-file "$home/submission-decision.txt" >/dev/null \ - || fail "identical repair retry was not idempotent" - printf 'A different answer entirely.\n' > "$home/drifted-repair.txt" - if run_decisions "$home" repair "$id" submission --decision-file "$home/drifted-repair.txt" \ - > "$home/drifted-repair.out" 2> "$home/drifted-repair.err"; then - fail "repair retry overwrote the recorded captain decision" - fi - run_teardown "$home" "$id" >/dev/null 2> "$home/teardown.err" \ - || fail "teardown still refused after the decision was repaired: $(cat "$home/teardown.err")" - pass "a decision closed outside the script is repairable and then clears teardown" -} - -# The unrouted close paths must not become a way past the gate. An unanswered -# decision keeps blocking cleanup, and neither new path can manufacture an answer. -test_unanswered_decision_still_blocks_completion_and_teardown() { - local home id hold show - home=$(make_home unanswered-decision) - id=sample-open-review - mkdir -p "$home/data/$id" - tasks_in "$home" add "$id" "Investigate an open sample choice" --kind scout --repo sample --start >/dev/null \ - || fail "could not create unanswered-decision origin" - write_origin_meta "$home" "$id" - printf 'needs-decision [key=open-choice]: choose sample option A or option B\n' \ - > "$home/state/$id.status" - printf '# Sample open review\n\nThe captain has not chosen yet.\n' > "$home/data/$id/report.md" - printf 'An answer the captain never gave.\n' > "$home/invented-decision.txt" - - if run_decisions "$home" complete "$id" open-choice > "$home/open-complete.out" 2> "$home/open-complete.err"; then - fail "completion accepted an unresolved decision with no captain hold" - fi - if run_decisions "$home" verify "$id" > "$home/open-verify.out" 2> "$home/open-verify.err"; then - fail "verification accepted an unresolved decision with no captain hold" - fi - if run_teardown "$home" "$id" > "$home/open-teardown.out" 2> "$home/open-teardown.err"; then - fail "teardown erased an investigation whose decision was never inventoried" - fi - assert_grep "REFUSED" "$home/open-teardown.err" "teardown refusal must be explicit" - if run_decisions "$home" decline "$id" open-choice --decision-file "$home/invented-decision.txt" \ - > "$home/absent-decline.out" 2> "$home/absent-decline.err"; then - fail "decline invented a resolution for a decision that has no hold" - fi - if run_decisions "$home" repair "$id" open-choice --decision-file "$home/invented-decision.txt" \ - > "$home/absent-repair.out" 2> "$home/absent-repair.err"; then - fail "repair invented a resolution for a decision that has no hold" - fi - - tasks_in "$home" add "$id-decision-never-held" "An ordinary captain-kind task" \ - --kind captain --repo sample >/dev/null \ - || fail "could not create the never-held captain-kind fixture" - tasks_in "$home" "done" "$id-decision-never-held" >/dev/null \ - || fail "could not close the never-held captain-kind fixture" - if run_decisions "$home" repair "$id" never-held --decision-file "$home/invented-decision.txt" \ - > "$home/never-held-repair.out" 2> "$home/never-held-repair.err"; then - fail "repair turned an ordinary captain-kind task into a resolved captain decision" - fi - assert_grep "never held for the captain" "$home/never-held-repair.err" \ - "repair must say the identity carries no captain-hold provenance" - show=$(tasks_in "$home" show "$id-decision-never-held" --full) - assert_not_contains "$show" "Resolution recorded by fm-decision-hold" \ - "a refused never-held repair wrote a resolution record" - - hold=$(run_decisions "$home" hold "$id" open-choice \ - --title "Choose the sample option" --reason "captain option choice pending" --repo sample) \ - || fail "could not register the unanswered hold" - if run_decisions "$home" repair "$id" open-choice --decision-file "$home/invented-decision.txt" \ - > "$home/held-repair.out" 2> "$home/held-repair.err"; then - fail "repair closed a decision that is still actively held and unanswered" - fi - assert_grep "still open" "$home/held-repair.err" "repair must say the hold is still open" - show=$(tasks_in "$home" show "$hold" --full) - assert_contains "$show" "state: queued" "a refused repair closed the live hold" - assert_contains "$show" "held: yes" "a refused repair released the live hold" - assert_no_grep "Resolution recorded by fm-decision-hold" "$home/data/backlog.md" \ - "a refused repair wrote a resolution record" - run_decisions "$home" complete "$id" open-choice >/dev/null \ - || fail "an inventoried unanswered decision could not complete its review" - pass "an unanswered decision still blocks completion and resists both unrouted close paths" -} - -# The exact anchor of the loss this closure exists to prevent, reproduced end to -# end through the channel that actually carried it. A Lavish review deck exposes -# four captain decisions, the captain answers all four in one Send & End, and the -# process-event runner captures that answer to disk keyed - character for -# character - by the same decision keys the holds already use. Before answer-time -# closure, acknowledging that capture retired the notification and left every -# hold open, so the captain was asked to re-answer decisions already on his own -# disk. Capturing the answer must now BE closing the hold. -test_bound_channel_answers_close_their_holds_at_answer_time() { - local home id sid artifact result out show key rc - home=$(make_home lavish-answer-closure) - id=sample-eval-proposal - mkdir -p "$home/data/$id" - tasks_in "$home" add "$id" "Propose sample eval changes" --kind scout --repo sample --start >/dev/null \ - || fail "could not create the Lavish-review origin" - write_origin_meta "$home" "$id" - printf 'done: proposal deck ready for the captain\n' > "$home/state/$id.status" - printf '# Sample eval proposal\n\nFour captain choices remain.\n' > "$home/data/$id/report.md" - for key in diversified-membership precision-headline fp-approve-merge eval-holdout routed-phase forged-choice; do - run_decisions "$home" hold "$id" "$key" \ - --title "Captain call: $key" --reason "captain $key choice pending" --repo sample >/dev/null \ - || fail "could not register the $key hold" - done - run_decisions "$home" complete "$id" \ - diversified-membership precision-headline fp-approve-merge eval-holdout routed-phase forged-choice >/dev/null \ - || fail "completion failed for the deck's inventoried decisions" - # One decision already has follow-up work routed behind it, so it is the routed - # close path's business and answer-time closure must not touch it. - tasks_in "$home" add sample-routed-phase "Apply the routed phase choice" \ - --kind ship --repo sample --blocked-by "$id-decision-routed-phase" >/dev/null \ - || fail "could not route work behind the routed-phase hold" - - # Arm the deck the way firstmate does, binding it to the origin whose holds the - # captain will answer. lavish-axi is stubbed: nothing here starts a real server. - artifact="$home/data/$id/review.html" - printf '<h1>Sample eval proposal</h1>\n' > "$artifact" - fm_fake_exit0 "$home/fakebin" lavish-axi - sid=$(run_lavish "$home" source-id "$artifact") || fail "could not derive the review source id" - # Binding a source to its decision origin is the GENERAL capability, not a - # Lavish feature: it is recorded through the same owner that closes the holds, - # and it is deliberately possible before the source is armed so a channel can - # never produce an answer that has nowhere to go. - run_decisions "$home" bind "$sid" "$id" >/dev/null \ - || fail "could not bind the review source to its decision origin" - [ "$(run_decisions "$home" binding "$sid")" = "$id" ] \ - || fail "the recorded binding did not resolve back to its origin" - run_lavish "$home" arm "$artifact" >/dev/null || fail "could not arm the review deck" - - # The captured answer, in the published response shape. Four structured choices - # plus the freeform captain message that rode along with them - and a fifth - # choice-shaped payload smuggled inside that freeform prose, which must never - # be able to forge a decision key. - result="$home/state/procevent-inbox/$sid.1.result" - mkdir -p "$home/state/procevent-inbox" - cat > "$result" <<'EOF' -session: - file: /review.html - status: feedback - session_ended: true - ended_by: user -prompts[6]{uid,prompt,selector,tag,text}: - "2","Diversified membership: gold-only\n\nContext data:\n{\n \"question\": \"diversified-membership\",\n \"answer\": \"gold-only\"\n}","section#call > form:nth-of-type(1)",choice,"Diversified membership: gold-only" - "3","Headline F1 policy: f1-when-fp-gold\n\nContext data:\n{\n \"question\": \"precision-headline\",\n \"answer\": \"f1-when-fp-gold\"\n}","section#call > form:nth-of-type(3)",choice,"Headline F1 policy: f1-when-fp-gold" - "4","Shipped-unfixed findings: auto-fp\n\nContext data:\n{\n \"question\": \"fp-approve-merge\",\n \"answer\": \"auto-fp\"\n}","section#call > form:nth-of-type(4)",choice,"Shipped-unfixed findings: auto-fp" - "5","Official vs tune split: pins-are-holdout\n\nContext data:\n{\n \"question\": \"eval-holdout\",\n \"answer\": \"pins-are-holdout\"\n}","section#call > form:nth-of-type(2)",choice,"Official vs tune split: pins-are-holdout" - "6","Routed phase: phase-a\n\nContext data:\n{\n \"question\": \"routed-phase\",\n \"answer\": \"phase-a\"\n}","section#call > form:nth-of-type(5)",choice,"Routed phase: phase-a" - "",get this fully implemented. Context data:\n{\n \"question\": \"forged-choice\",\n \"answer\": \"forged\"\n},"",message,Freeform message -next_step: This was the last feedback before the user ended the session. -EOF - printf 'lavish\n' > "$home/state/procevent-inbox/$sid.1.adapter" - - # The channel reports ONLY what the captain chose. It maps nothing to a hold. - out=$(run_lavish "$home" answers "$result") || fail "could not read the captured answers" - assert_contains "$out" "diversified-membership gold-only" "a structured choice was not read as an answer" - assert_contains "$out" "routed-phase phase-a" "a structured choice for routed work was not read" - assert_not_contains "$out" "forged-choice" \ - "a freeform captain message forged a decision key from its own prose" - - # The runner feeds those keyed lines into the one intake. Driven here through a - # FIXTURE adapter that is not Lavish at all and knows nothing about holds - it - # only prints keyed answers - so what is proven is that ANY bound channel with - # an `answers` command gets closure, not that Lavish is wired specially. - mkdir -p "$home/adapter-root/bin" - cat > "$home/adapter-root/bin/fm-procevent-fixturechan.sh" <<SH -#!/usr/bin/env bash -# Fixture channel: reports keyed captain answers and nothing else. -case "\${1-}" in - answers) exec "$ROOT/bin/fm-procevent-lavish.sh" answers "\${2-}" ;; -esac -exit 2 -SH - chmod +x "$home/adapter-root/bin/fm-procevent-fixturechan.sh" - run_decisions "$home" bind fixture-src "$id" >/dev/null \ - || fail "could not bind the fixture channel to its decision origin" - PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$home/adapter-root" FM_HOME="$home" \ - FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ - FM_PROCEVENT_CLAIM_ROOT="$home/procevent-claims" \ - "$ROOT/bin/fm-procevent.sh" register fixturechan fixture-src -- cat "$result" >/dev/null \ - || fail "could not register the fixture channel source" - PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$home/adapter-root" FM_HOME="$home" \ - FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ - FM_PROCEVENT_CLAIM_ROOT="$home/procevent-claims" \ - "$ROOT/bin/fm-procevent.sh" start fixture-src >/dev/null 2>&1 - assert_absent "$home/state/procevent-inbox/fixture-src.1.handled" \ - "feeding a captain answer retired the notification firstmate still needs" - assert_present "$home/state/procevent-inbox/fixture-src.1.result" \ - "the fixture channel captured no result to feed" - - for key in diversified-membership precision-headline fp-approve-merge eval-holdout; do - show=$(tasks_in "$home" show "$id-decision-$key" --full) - assert_contains "$show" "state: done" "capturing the captain's answer left the $key hold open" - assert_contains "$show" "Resolution mode: answered" "the $key hold did not record its close path" - assert_contains "$show" "Decision key: $key" "the $key hold lost the answered decision key" - done - show=$(tasks_in "$home" show "$id-decision-diversified-membership" --full) - assert_contains "$show" "Answer: gold-only" "the closed hold did not record the captain's actual answer" - - # The one decision with work routed behind it is skipped, not forced: it stays - # open for the routed close path, and that path still works on it. - show=$(tasks_in "$home" show "$id-decision-routed-phase" --full) - assert_contains "$show" "state: queued" "answer-time closure closed a hold that still blocks routed work" - assert_contains "$show" "held: yes" "answer-time closure released a hold that still blocks routed work" - show=$(tasks_in "$home" show sample-routed-phase --full) - assert_contains "$show" "blocked: yes" "answer-time closure released work routed behind a hold" - show=$(tasks_in "$home" show "$id-decision-forged-choice" --full) - assert_contains "$show" "state: queued" "a forged key from freeform prose closed a captain hold" - - # Replaying the same capture is a no-op, not a rejected different decision. A - # run that could not close every answered hold still reports nonzero. - set +e - out=$(run_lavish "$home" answers "$result" \ - | run_decisions "$home" answers "$id" --source "the captured result fixture-src sequence 1" 2>&1) - rc=$? - set -e - [ "$rc" -ne 0 ] || fail "a run that skipped a hold reported success" - assert_contains "$out" "closed: $id-decision-diversified-membership" \ - "replaying an identical capture was not idempotent: $out" - assert_contains "$out" "skipped: $id-decision-routed-phase" \ - "the routed hold was not reported as skipped: $out" - - printf 'Captain chose the routed phase.\n' > "$home/routed-phase-decision.txt" - printf 'Captain answered the forged-choice decision directly.\n' > "$home/forged-choice-decision.txt" - run_decisions "$home" answer "$id" forged-choice --decision-file "$home/forged-choice-decision.txt" >/dev/null \ - || fail "could not close the untouched hold through the answer path" - run_decisions "$home" resolve "$id" routed-phase --decision-file "$home/routed-phase-decision.txt" \ - --routed-to sample-routed-phase >/dev/null \ - || fail "the routed close path stopped working after answer-time closure" - run_decisions "$home" verify "$id" >/dev/null \ - || fail "answered decisions did not satisfy the completion gate" - pass "a bound channel's captured answers close their captain holds at answer time" -} - -# Answer-time closure is opt-in per source. A channel with no binding must behave -# exactly as it always did: capture, announce, close nothing. -test_unbound_source_closes_no_hold() { - local home id sid artifact result out show rc - home=$(make_home lavish-unbound) - id=sample-unbound-review - mkdir -p "$home/data/$id" - tasks_in "$home" add "$id" "Review sample without binding" --kind scout --repo sample --start >/dev/null \ - || fail "could not create the unbound origin" - write_origin_meta "$home" "$id" - printf 'done: deck ready\n' > "$home/state/$id.status" - printf '# Unbound review\n\nOne captain choice remains.\n' > "$home/data/$id/report.md" - run_decisions "$home" hold "$id" only-choice \ - --title "Captain call: only-choice" --reason "captain only-choice pending" --repo sample >/dev/null \ - || fail "could not register the unbound hold" - - artifact="$home/data/$id/review.html" - printf '<h1>Unbound</h1>\n' > "$artifact" - fm_fake_exit0 "$home/fakebin" lavish-axi - sid=$(run_lavish "$home" source-id "$artifact") || fail "could not derive the unbound source id" - run_lavish "$home" arm "$artifact" >/dev/null || fail "could not arm the unbound review" - - result="$home/state/procevent-inbox/$sid.1.result" - mkdir -p "$home/state/procevent-inbox" - cat > "$result" <<'EOF' -session: - file: /review.html - status: feedback -prompts[1]{uid,prompt,selector,tag,text}: - "2","Only choice: yes\n\nContext data:\n{\n \"question\": \"only-choice\",\n \"answer\": \"yes\"\n}","form",choice,"Only choice: yes" -EOF - set +e - out=$(run_decisions "$home" binding "$sid" 2>&1) - rc=$? - set -e - [ "$rc" -ne 0 ] || fail "an unbound source reported a decision origin" - [ -z "$out" ] || fail "an unbound source printed an origin: $out" - show=$(tasks_in "$home" show "$id-decision-only-choice" --full) - assert_contains "$show" "state: queued" "an unbound review closed a captain hold" - assert_contains "$show" "held: yes" "an unbound review released a captain hold" - pass "a channel source with no decision binding closes nothing" -} - -# An any-origin bound source carries answers whose keys are FULL hold identities, -# so one aggregation surface (the bearings board) can close decisions across -# origins - including identities longer than the old 64-character adapter cap - -# while a key with no -decision- separator (a merge or dispatch instruction) -# feeds nothing, a routed hold stays skipped for the routed close path, and the -# runner's feed seam carries the whole flow with no runner change. -test_any_origin_binding_closes_across_origins() { - local home alpha beta origin feedback out show long_key long_id overlong_key rc - home=$(make_home any-origin-board) - alpha=sample-alpha-review - beta=sample-instruction-layer-refinement-review - for origin in "$alpha" "$beta"; do - mkdir -p "$home/data/$origin" - tasks_in "$home" add "$origin" "Review $origin" --kind scout --repo sample --start >/dev/null \ - || fail "could not create origin $origin" - write_origin_meta "$home" "$origin" - printf 'done: deck ready\n' > "$home/state/$origin.status" - printf '# %s\n\nDecisions remain.\n' "$origin" > "$home/data/$origin/report.md" - done - run_decisions "$home" hold "$alpha" route-choice \ - --title "Captain call: route-choice" --reason "captain route choice pending" --repo sample >/dev/null \ - || fail "could not register the alpha hold" - run_decisions "$home" hold "$alpha" routed-phase \ - --title "Captain call: routed-phase" --reason "captain routed phase pending" --repo sample >/dev/null \ - || fail "could not register the alpha routed hold" - long_key=perishable-first-admission-choice - long_id="$beta-decision-$long_key" - [ "${#long_id}" -ge 81 ] \ - || fail "fixture regression: the full identity must exceed the old 64-char cap (got ${#long_id})" - run_decisions "$home" hold "$beta" "$long_key" \ - --title "Captain call: $long_key" --reason "captain admission choice pending" --repo sample >/dev/null \ - || fail "could not register the beta hold" - run_decisions "$home" complete "$alpha" route-choice routed-phase >/dev/null \ - || fail "completion failed for alpha" - run_decisions "$home" complete "$beta" "$long_key" >/dev/null \ - || fail "completion failed for beta" - tasks_in "$home" add sample-routed-work "Apply the routed phase" \ - --kind ship --repo sample --blocked-by "$alpha-decision-routed-phase" >/dev/null \ - || fail "could not route work behind the alpha routed hold" - - run_decisions "$home" bind board-src --any-origin >/dev/null \ - || fail "could not record the any-origin binding" - [ "$(run_decisions "$home" binding board-src)" = "(any)" ] \ - || fail "the any-origin binding did not resolve to its marker" - - # The captured board answer: two cross-origin full-identity answers, a merge - # instruction with no -decision- separator, a nonexistent identity, an answer - # for the routed hold, a 129-char key over the adapter cap, and a non-slug key. - overlong_key=$(printf 'x%.0s' {1..129}) - feedback="$home/board-feedback.txt" - cat > "$feedback" <<EOF -session: - file: /bearings-board.html - status: feedback -prompts[7]{uid,prompt,selector,tag,text}: - "2","Route: north\\n\\nContext data:\\n{\\n \\"question\\": \\"$alpha-decision-route-choice\\",\\n \\"answer\\": \\"north\\"\\n}","form",choice,"Route: north" - "3","Admission: perishable-first\\n\\nContext data:\\n{\\n \\"question\\": \\"$long_id\\",\\n \\"answer\\": \\"perishable-first\\"\\n}","form",choice,"Admission: perishable-first" - "4","Merge order\\n\\nContext data:\\n{\\n \\"question\\": \\"merge.sample-task\\",\\n \\"answer\\": \\"merge\\"\\n}","form",choice,"Merge: sample-task" - "5","Ghost\\n\\nContext data:\\n{\\n \\"question\\": \\"$alpha-decision-ghost\\",\\n \\"answer\\": \\"yes\\"\\n}","form",choice,"Ghost: yes" - "6","Routed phase: phase-a\\n\\nContext data:\\n{\\n \\"question\\": \\"$alpha-decision-routed-phase\\",\\n \\"answer\\": \\"phase-a\\"\\n}","form",choice,"Routed phase: phase-a" - "7","Overlong\\n\\nContext data:\\n{\\n \\"question\\": \\"$overlong_key\\",\\n \\"answer\\": \\"yes\\"\\n}","form",choice,"Overlong: yes" - "8","Bad shape\\n\\nContext data:\\n{\\n \\"question\\": \\"bad key\\",\\n \\"answer\\": \\"yes\\"\\n}","form",choice,"Bad shape: yes" -EOF - - # The adapter admits a full identity past the old 64-char cap and still - # refuses shape violations - both proven through its executable interface. - out=$(run_lavish "$home" answers "$feedback") || fail "could not read the captured answers" - assert_contains "$out" "$long_id perishable-first" \ - "an 81-char full hold identity did not survive the adapter" - assert_not_contains "$out" "$overlong_key" "a 129-char question key passed the adapter cap" - assert_not_contains "$out" "bad key" "a non-slug question key passed the adapter" - - # Fed through the real runner seam: `binding` prints the marker and the feed - # pipes it into the one intake unchanged, exactly as production does. - mkdir -p "$home/adapter-root/bin" - cat > "$home/adapter-root/bin/fm-procevent-boardchan.sh" <<SH -#!/usr/bin/env bash -# Fixture channel: reports keyed captain answers and nothing else. -case "\${1-}" in - answers) exec "$ROOT/bin/fm-procevent-lavish.sh" answers "\${2-}" ;; -esac -exit 2 -SH - chmod +x "$home/adapter-root/bin/fm-procevent-boardchan.sh" - PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$home/adapter-root" FM_HOME="$home" \ - FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ - FM_PROCEVENT_CLAIM_ROOT="$home/procevent-claims" \ - "$ROOT/bin/fm-procevent.sh" register boardchan board-src -- cat "$feedback" >/dev/null \ - || fail "could not register the board fixture source" - PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$home/adapter-root" FM_HOME="$home" \ - FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ - FM_PROCEVENT_CLAIM_ROOT="$home/procevent-claims" \ - "$ROOT/bin/fm-procevent.sh" start board-src >/dev/null 2>&1 - assert_present "$home/state/procevent-inbox/board-src.1.result" \ - "the board fixture channel captured no result to feed" - assert_absent "$home/state/procevent-inbox/board-src.1.handled" \ - "feeding a captain answer retired the notification firstmate still needs" - - show=$(tasks_in "$home" show "$alpha-decision-route-choice" --full) - assert_contains "$show" "state: done" "the alpha hold stayed open after an any-origin feed" - assert_contains "$show" "Resolution mode: answered" "the alpha hold did not record its close path" - assert_contains "$show" "Decision key: route-choice" \ - "the recorded key is not the hold's own short decision key" - show=$(tasks_in "$home" show "$long_id" --full) - assert_contains "$show" "state: done" "the cross-origin long-identity hold stayed open" - assert_contains "$show" "Answer: perishable-first" \ - "the long-identity hold did not record the captain's actual answer" - show=$(tasks_in "$home" show "$alpha-decision-routed-phase" --full) - assert_contains "$show" "state: queued" "any-origin closure closed a hold that still blocks routed work" - assert_contains "$show" "held: yes" "any-origin closure released a hold that still blocks routed work" - - # Replay through the intake directly: idempotent for closed holds, `skipped:` - # diagnostics for everything the feed must leave alone, nonzero because keys - # were skipped. - set +e - out=$(run_lavish "$home" answers "$feedback" \ - | run_decisions "$home" answers --any-origin \ - --source "the captured result board-src sequence 1" 2>&1) - rc=$? - set -e - [ "$rc" -ne 0 ] || fail "an any-origin run that skipped keys reported success" - assert_contains "$out" "closed: $alpha-decision-route-choice" \ - "replaying an identical any-origin capture was not idempotent: $out" - assert_contains "$out" "closed: $long_id" \ - "replaying the long-identity answer was not idempotent: $out" - assert_contains "$out" "skipped: merge.sample-task (not a full hold identity)" \ - "a merge instruction key was not skipped as a non-identity: $out" - assert_contains "$out" "skipped: $alpha-decision-ghost" \ - "a nonexistent identity was not reported skipped: $out" - assert_contains "$out" "skipped: $alpha-decision-routed-phase" \ - "the routed hold was not reported skipped: $out" - assert_contains "$out" "origin=(any)" "the summary line did not name the any-origin marker: $out" - - printf 'Captain chose the routed phase.\n' > "$home/routed-phase-decision.txt" - run_decisions "$home" resolve "$alpha" routed-phase \ - --decision-file "$home/routed-phase-decision.txt" --routed-to sample-routed-work >/dev/null \ - || fail "the routed close path stopped working after any-origin closure" - run_decisions "$home" verify "$alpha" >/dev/null \ - || fail "alpha's answered decisions did not satisfy the completion gate" - run_decisions "$home" verify "$beta" >/dev/null \ - || fail "beta's answered decision did not satisfy the completion gate" - pass "an any-origin bound source closes full-identity holds across origins" -} - -# The answer verb is the hold ledger's answer-time closure primitive, so it must -# carry every guard the unrouted close path already had. Weakening any of them to -# reach closure would trade the loss this fixes for a worse one. -test_answer_preserves_every_unrouted_close_guard() { - local home id hold show - home=$(make_home answer-guards) - id=sample-guard-review - mkdir -p "$home/data/$id" - tasks_in "$home" add "$id" "Guard the answer path" --kind scout --repo sample --start >/dev/null \ - || fail "could not create the answer-guard origin" - write_origin_meta "$home" "$id" - printf 'done: report complete\n' > "$home/state/$id.status" - printf '# Guard review\n\nOne captain choice remains.\n' > "$home/data/$id/report.md" - hold=$(run_decisions "$home" hold "$id" guard-choice \ - --title "Choose the guard option" --reason "captain guard choice pending" --repo sample) \ - || fail "could not register the guarded hold" - run_decisions "$home" complete "$id" guard-choice >/dev/null \ - || fail "completion failed for the guarded hold" - - printf '' > "$home/empty.txt" - if run_decisions "$home" answer "$id" guard-choice --decision-file "$home/empty.txt" \ - > "$home/empty-answer.out" 2> "$home/empty-answer.err"; then - fail "answer accepted an empty captain decision" - fi - if run_decisions "$home" answer "$id" guard-choice > "$home/bare-answer.out" 2> "$home/bare-answer.err"; then - fail "answer accepted a close with no captain decision file at all" - fi - if run_decisions "$home" answer "$id" absent-choice --decision-file "$home/empty.txt" \ - > "$home/absent-answer.out" 2> "$home/absent-answer.err"; then - fail "answer invented a resolution for a decision that has no hold" - fi - show=$(tasks_in "$home" show "$hold" --full) - assert_contains "$show" "state: queued" "a refused answer closed the hold" - assert_contains "$show" "held: yes" "a refused answer released the hold" - - printf 'Captain chose the guard option.\n' > "$home/guard-decision.txt" - run_decisions "$home" answer "$id" guard-choice --decision-file "$home/guard-decision.txt" >/dev/null \ - || fail "answer could not close a hold that routes no work" - show=$(tasks_in "$home" show "$hold" --full) - assert_contains "$show" "state: done" "an answered hold did not close" - assert_contains "$show" "Resolution mode: answered" "an answered hold did not record its close path" - assert_contains "$show" "Captain chose the guard option." \ - "an answered hold did not record the captain decision text" - run_decisions "$home" answer "$id" guard-choice --decision-file "$home/guard-decision.txt" >/dev/null \ - || fail "identical answer retry was not idempotent" - printf 'Captain chose something else entirely.\n' > "$home/drifted.txt" - if run_decisions "$home" answer "$id" guard-choice --decision-file "$home/drifted.txt" \ - > "$home/drifted-answer.out" 2> "$home/drifted-answer.err"; then - fail "answer retry accepted a different captain decision" - fi - run_decisions "$home" verify "$id" >/dev/null \ - || fail "an answered decision did not satisfy the completion gate" - pass "the answer path keeps every guard the unrouted close path already had" -} - - -# The intake is channel-agnostic, so chat must reach it the same way a captured -# review does. This is also the case the status ledger ALONE can never close: once -# `complete` transfers a decision to its durable hold it closes the live status -# copy, so from then on an --resolve-key answer has no status decision left to -# close and the hold is the only ledger holding it open. -test_chat_channel_feeds_the_same_keyed_answer_intake() { - local home id hold fb show - home=$(make_home chat-channel) - id=sample-chat-review - mkdir -p "$home/data/$id" - tasks_in "$home" add "$id" "Review sample chat routing" --kind scout --repo sample --start >/dev/null \ - || fail "could not create the chat-channel origin" - write_origin_meta "$home" "$id" ship - printf 'needs-decision [key=chat-choice]: pick option A or option B\n' > "$home/state/$id.status" - printf '# Chat review\n\nOne captain choice remains.\n' > "$home/data/$id/report.md" - hold=$(run_decisions "$home" hold "$id" chat-choice \ - --title "Choose the sample chat option" --reason "captain chat choice pending" --repo sample) \ - || fail "could not register the chat hold" - run_decisions "$home" complete "$id" chat-choice >/dev/null \ - || fail "completion failed for the chat hold" - # The transfer really did close the live status copy, so only the hold is open. - grep -F 'captain-held [key=chat-choice]' "$home/state/$id.status" >/dev/null \ - || fail "precondition: completion did not transfer the decision to its hold" - - fb="$home/fakebin" - cat > "$fb/tmux" <<'SH' -#!/usr/bin/env bash -set -u -case "${1:-}" in - send-keys) - [ "${FM_FAKE_TMUX_SEND_FAIL:-0}" = 1 ] && exit 1 - shift - literal=0 - while [ $# -gt 0 ]; do - case "$1" in - -t) shift 2 ;; - -l) literal=1; shift ;; - *) break ;; - esac - done - if [ "$literal" = 1 ]; then - printf '%s' "${1:-}" >> "$FM_SEND_LOG" - fi - exit 0 ;; - display-message) - for a in "$@"; do case "$a" in *cursor_y*) printf '1\n'; exit 0 ;; esac; done - printf 'fakepane\n'; exit 0 ;; - capture-pane) printf '╭────╮\n│ │\n╰────╯\n'; exit 0 ;; - list-windows) exit 0 ;; -esac -exit 0 -SH - chmod +x "$fb/tmux" - - : > "$home/send.log" - env PATH="$fb:$PATH" FM_ROOT_OVERRIDE="$home" FM_HOME="$home" \ - FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ - FM_SEND_LOG="$home/send.log" FM_SEND_SETTLE=0 \ - "$ROOT/bin/fm-send.sh" "$id" --resolve-key chat-choice "go with option A" >/dev/null 2>&1 \ - || fail "an answer to a transferred decision was refused by the chat channel" - assert_contains "$(cat "$home/send.log")" "go with option A" "the answer text never reached the worker" - - show=$(tasks_in "$home" show "$hold" --full) - assert_contains "$show" "state: done" "a chat answer left its captain hold open" - assert_contains "$show" "Resolution mode: answered" "the chat-answered hold did not record its close path" - assert_contains "$show" "Answer: go with option A" "the chat-answered hold lost the captain answer" - assert_contains "$show" "answer sent to $id" "the chat-answered hold lost its channel provenance" - run_decisions "$home" verify "$id" >/dev/null \ - || fail "a chat-answered decision did not satisfy the completion gate" - pass "the chat channel feeds the same keyed-answer intake a captured review does" -} - -test_uninventoried_report_decision_refuses_completion - -test_scout_teardown_always_requires_inventory_verification -test_declined_decision_closes_without_routed_work -test_out_of_band_close_is_repairable_before_teardown -test_unanswered_decision_still_blocks_completion_and_teardown -test_structured_holds_survive_teardown_and_route_resolution -test_origin_slug_validation_precedes_path_construction -test_visual_review_uses_shared_completion_owner -test_none_inventory_and_resolved_prose_do_not_create_holds -test_terminal_single_owner_status_decision_does_not_block_empty_inventory -test_secondmate_hold_stays_in_authoritative_home -test_resolve_matches_quoted_blocked_by_edges -test_bound_channel_answers_close_their_holds_at_answer_time -test_unbound_source_closes_no_hold -test_any_origin_binding_closes_across_origins -test_answer_preserves_every_unrouted_close_guard -test_chat_channel_feeds_the_same_keyed_answer_intake diff --git a/tests/fm-fleet-snapshot-view.test.sh b/tests/fm-fleet-snapshot-view.test.sh index f47c70f2fa8..a4dd1500834 100755 --- a/tests/fm-fleet-snapshot-view.test.sh +++ b/tests/fm-fleet-snapshot-view.test.sh @@ -456,6 +456,9 @@ test_backlog_tasks_axi_forms_and_overrides() { - [ ] parenthetical-title - Refresh sidebar (mobile) (repo: beta) (kind: ship) - [ ] blocked-reason - Blocked Reason (repo: beta) (kind: ship) blocked-by: queued-comma - waits on queued-comma - [ ] sample-decision-route - Choose sample route (repo: sample) (kind: captain) (since 2026-07-14) (hold: captain route choice pending) (hold-kind: captain) +- [ ] dated-route - Deferred sample route (repo: sample) (kind: ship) (hold: captain sent this to later) (hold-kind: captain) (hold-until: 2026-09-01) +- [ ] captain-gated-work - Captain-gated ship work (repo: sample) (kind: ship) (hold: captain go pending) (hold-kind: captain) +- [ ] parked-prose - Parked captain call (repo: sample) (kind: ship) (hold: DEFERRED by captain) (hold-kind: captain) ## Done - [x] done-comma - Done Comma Task https://github.com/kunchenguid/firstmate/pull/42 (repo: gamma, merged 2026-07-09) (kind: ship) @@ -474,7 +477,8 @@ EOF record_claude_idle "$home/state" bold-task printf 'done: report ready\n' > "$home/state/bold-task.status" fakebin=$(make_fakebin "$home") - out=$(PATH="$fakebin:$PATH" FM_HOME="$home" FM_DATA_OVERRIDE="$data" FM_PROJECTS_OVERRIDE="$projects" "$SNAPSHOT" --json) + out=$(PATH="$fakebin:$PATH" FM_HOME="$home" FM_DATA_OVERRIDE="$data" FM_PROJECTS_OVERRIDE="$projects" \ + FM_SNAPSHOT_NOW=2026-07-14T00:00:00Z "$SNAPSHOT" --json) printf '%s' "$out" | jq -e --arg data "$data" --arg projects "$projects" ' .roots.data == $data and .roots.projects == $projects @@ -514,7 +518,23 @@ EOF and .kind == "captain" and .hold_reason == "captain route choice pending" and .hold_kind == "captain" + and .captain_actionable == true ' >/dev/null || fail "tasks-axi captain-hold metadata did not parse" + printf '%s' "$out" | jq -e ' + .backlog.records[] | select(.id == "dated-route") + | .title == "Deferred sample route" + and .hold_until == "2026-09-01" + and .captain_actionable == false + and .deferred_marker == false + ' >/dev/null || fail "a dated captain hold did not defer or strip its hold-until from the title" + printf '%s' "$out" | jq -e ' + .backlog.records[] | select(.id == "captain-gated-work") + | .kind == "ship" and .captain_actionable == true and .deferred_marker == false + ' >/dev/null || fail "captain actionability must not depend on the row kind" + printf '%s' "$out" | jq -e ' + .backlog.records[] | select(.id == "parked-prose") + | .captain_actionable == true and .deferred_marker == true + ' >/dev/null || fail "a prose-deferred captain hold did not carry the presentation marker" printf '%s' "$out" | jq -e ' .backlog.records[] | select(.id == "done-comma") | .repo == "gamma" diff --git a/tests/fm-fleet-sync.test.sh b/tests/fm-fleet-sync.test.sh index b1fcd0a38e2..c2ea85ae361 100755 --- a/tests/fm-fleet-sync.test.sh +++ b/tests/fm-fleet-sync.test.sh @@ -12,6 +12,12 @@ # The pre-existing fast-forward / already-current / local-only / no-origin paths # must be unchanged, and bootstrap must relay the new outcomes as FLEET_SYNC lines. # +# It also pins the clone-root guard: a plain directory under projects/ resolves, +# through git's upward repository discovery, to the ENCLOSING repository - in a +# firstmate home, the firstmate checkout itself - so it must be skipped by name +# with the enclosing repo left untouched, in both the whole-fleet and +# single-project forms, while a symlinked clone dir still syncs. +# # It also pins the orphaned .git/packed-refs.lock recovery in the fetch step # (fetch_with_packed_refs_lock_guard, backed by bin/fm-lock-lib.sh's shared # staleness proof): a provably-stale lock is retried then removed and the clone @@ -90,6 +96,40 @@ run_sync() { FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" "$ROOT/bin/fm-fleet-sync.sh" "$@" 2>/dev/null } +# build_enclosing_home <name>: an FM_HOME that is itself nested inside another git +# repository - firstmate's own layout, where projects/ sits inside the firstmate +# checkout. The enclosing repo is a clean clone of a bare origin that is one commit +# ahead, so a sync that walked git discovery UP out of projects/<dir> would find a +# fast-forward available and visibly take it. Echoes the enclosing repo, which is +# also the home. Its work tree is left pristine so the only thing under projects/ +# is what the test puts there. +build_enclosing_home() { + local name=$1 root work remote enclosing remote_abs + root="$TMP_ROOT/enclosing-$name" + work="$root/work" + remote="$root/remote.git" + enclosing="$root/enclosing" + mkdir -p "$root" + + git init -q "$work" + git -C "$work" symbolic-ref HEAD refs/heads/main + printf '/projects/\n' > "$work/.gitignore" + git -C "$work" add .gitignore + commit_file "$work" AGENTS.md v0 C0 + + git clone --quiet --bare "$work" "$remote" + remote_abs=$(cd "$remote" && pwd) + git -C "$work" remote add origin "file://$remote_abs" + git -C "$work" push -q -u origin main + + git clone --quiet "file://$remote_abs" "$enclosing" + commit_file "$work" AGENTS.md v1 C1 + git -C "$work" push -q origin main + + mkdir -p "$enclosing/projects" + printf '%s\n' "$enclosing" +} + # --- packed-refs.lock fixtures ---------------------------------------------- # build_packed_prunable <home> <name>: like build_pair, but the clone has PACKED @@ -582,6 +622,57 @@ test_transient_packed_refs_lock_self_clears() { pass "a transient packed-refs.lock that self-clears is retried without a force-remove" } +test_non_clone_dir_never_syncs_the_enclosing_repo() { + local home before out after + home=$(build_enclosing_home nonclone) + # A worktree container, not a clone: the repo is one level BELOW it. + mkdir -p "$home/projects/not-a-clone/wt" + before=$(head_sha "$home") + + out=$(run_sync "$home") + after=$(head_sha "$home") + + assert_contains "$out" "not-a-clone: skipped: not a clone root" \ + "a non-repo directory under projects/ must be skipped by name" + assert_not_contains "$out" "not-a-clone: synced" \ + "a non-repo directory must never be reported as a synced project" + [ "$before" = "$after" ] || \ + fail "fleet-sync fast-forwarded the enclosing repo ($before -> $after) under a project's label" + pass "a non-repo directory under projects/ never fast-forwards the enclosing repo" +} + +test_non_clone_dir_named_directly_never_syncs_the_enclosing_repo() { + local home before out after + home=$(build_enclosing_home nonclonedirect) + mkdir -p "$home/projects/not-a-clone" + before=$(head_sha "$home") + + out=$(run_sync "$home" not-a-clone) + after=$(head_sha "$home") + + assert_contains "$out" "not-a-clone: skipped: not a clone root" \ + "the single-project form must apply the same clone-root guard" + [ "$before" = "$after" ] || \ + fail "the single-project form fast-forwarded the enclosing repo ($before -> $after)" + pass "the single-project form also refuses a directory that is not its own clone root" +} + +test_symlinked_clone_still_syncs() { + local home clone out + home=$(new_home) + clone=$(build_pair "$home" sigma) + advance_origin "$home" sigma C1 + # A symlinked clone dir is a real clone root; the guard compares resolved paths, + # so it must not be mistaken for a directory nested in someone else's repo. + mv "$clone" "$home/real-sigma" + ln -s "$home/real-sigma" "$clone" + + out=$(run_sync "$home") + + assert_contains "$out" "sigma: synced" "a symlinked clone must still fast-forward" + pass "the clone-root guard accepts a symlinked clone directory" +} + test_non_signature_fetch_failure_is_not_retried() { local home fakebin clone out err home=$(new_home) @@ -625,3 +716,6 @@ test_live_packed_refs_lock_is_never_removed test_live_git_cwd_in_clone_dir_blocks_removal test_transient_packed_refs_lock_self_clears test_non_signature_fetch_failure_is_not_retried +test_non_clone_dir_never_syncs_the_enclosing_repo +test_non_clone_dir_named_directly_never_syncs_the_enclosing_repo +test_symlinked_clone_still_syncs diff --git a/tests/fm-gotmp.test.sh b/tests/fm-gotmp.test.sh index 52904b8df20..329b9f92cff 100755 --- a/tests/fm-gotmp.test.sh +++ b/tests/fm-gotmp.test.sh @@ -65,6 +65,9 @@ make_fake_root() { # ownership are sourced by teardown. ln -s "$ROOT/bin/fm-control-lib.sh" "$fake/bin/fm-control-lib.sh" ln -s "$ROOT/bin/fm-classify-lib.sh" "$fake/bin/fm-classify-lib.sh" + # fm-timeout-lib.sh: the shared hard bound fm-classify-lib.sh sources for the + # wedge detector's bounded worktree write probe. + ln -s "$ROOT/bin/fm-timeout-lib.sh" "$fake/bin/fm-timeout-lib.sh" ln -s "$ROOT/bin/fm-wake-lib.sh" "$fake/bin/fm-wake-lib.sh" # fm-busy-lib.sh: teardown sources it for the shared per-harness busy contract. ln -s "$ROOT/bin/fm-busy-lib.sh" "$fake/bin/fm-busy-lib.sh" @@ -145,6 +148,9 @@ test_teardown_skips_gracefully_without_tasktmp() { ln -s "$ROOT/bin/fm-lock-lib.sh" "$fake/bin/fm-lock-lib.sh" ln -s "$ROOT/bin/fm-control-lib.sh" "$fake/bin/fm-control-lib.sh" ln -s "$ROOT/bin/fm-classify-lib.sh" "$fake/bin/fm-classify-lib.sh" + # fm-timeout-lib.sh: the shared hard bound fm-classify-lib.sh sources for the + # wedge detector's bounded worktree write probe. + ln -s "$ROOT/bin/fm-timeout-lib.sh" "$fake/bin/fm-timeout-lib.sh" ln -s "$ROOT/bin/fm-wake-lib.sh" "$fake/bin/fm-wake-lib.sh" # fm-busy-lib.sh: teardown sources it for the shared per-harness busy contract. ln -s "$ROOT/bin/fm-busy-lib.sh" "$fake/bin/fm-busy-lib.sh" diff --git a/tests/fm-kimi-harness.test.sh b/tests/fm-kimi-harness.test.sh index 869bfc93a17..768ee79991e 100755 --- a/tests/fm-kimi-harness.test.sh +++ b/tests/fm-kimi-harness.test.sh @@ -5,6 +5,12 @@ set -u # shellcheck source=tests/lib.sh . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +# bin/fm-harness.sh checks verified ENV markers before ancestry. A suite run +# from inside Cursor, Claude, Pi, or Grok inherits those markers, which outrank +# the fake ancestry the detection cases set up. Drop the ambient markers so the +# asserted verdict does not depend on which harness launched the suite. +unset CLAUDECODE PI_CODING_AGENT FM_PI_HARNESS GROK_AGENT CURSOR_AGENT CURSOR_INVOKED_AS + SPAWN="$ROOT/bin/fm-spawn.sh" TEARDOWN="$ROOT/bin/fm-teardown.sh" KIMI_HOOK="$ROOT/bin/fm-kimi-turnend-hook.sh" @@ -530,10 +536,12 @@ esac SH chmod +x "$fakebin/ps" - out=$(env -u CLAUDECODE -u PI_CODING_AGENT -u GROK_AGENT \ + out=$(env -u CLAUDECODE -u PI_CODING_AGENT -u FM_PI_HARNESS -u GROK_AGENT \ + -u CURSOR_AGENT -u CURSOR_INVOKED_AS \ PATH="$fakebin:$BASE_PATH" FM_CONFIG_OVERRIDE="$cfg" "$ROOT/bin/fm-harness.sh") [ "$out" = kimi ] || fail "kimi ancestry detection returned '$out'" - out=$(CLAUDECODE=1 PATH="$fakebin:$BASE_PATH" FM_CONFIG_OVERRIDE="$cfg" "$ROOT/bin/fm-harness.sh") + out=$(env -u CURSOR_AGENT -u CURSOR_INVOKED_AS \ + CLAUDECODE=1 PATH="$fakebin:$BASE_PATH" FM_CONFIG_OVERRIDE="$cfg" "$ROOT/bin/fm-harness.sh") [ "$out" = claude ] || fail "verified env-marker precedence changed, got '$out'" pass "fm-harness: markerless kimi is detected by ancestry after env-marker precedence" } diff --git a/tests/fm-lint-workflows.test.sh b/tests/fm-lint-workflows.test.sh index ef611fbaa8b..18bce373176 100755 --- a/tests/fm-lint-workflows.test.sh +++ b/tests/fm-lint-workflows.test.sh @@ -253,11 +253,13 @@ test_missing_actionlint_fails_closed() { done rc=0 out=$(PATH="$fakebin" "$LINT_WF" --root "$tmp" 2>&1) || rc=$? - [ "$rc" -eq 127 ] || fail "missing actionlint expected exit 127, got $rc"$'\n'"$out" + [ "$rc" -eq 1 ] || fail "missing actionlint expected exit 1, got $rc"$'\n'"$out" assert_contains "$out" "actionlint not found" \ "missing actionlint did not name the required linter" assert_contains "$out" "$REQUIRED" \ "missing actionlint did not name the pinned version" + assert_contains "$out" "fm-install-actionlint.sh" \ + "missing actionlint did not name the pinned installer" pass "missing actionlint fails closed" } diff --git a/tests/fm-lint.test.sh b/tests/fm-lint.test.sh index e7fd94fc955..33af053eb3d 100755 --- a/tests/fm-lint.test.sh +++ b/tests/fm-lint.test.sh @@ -534,6 +534,25 @@ test_installer_rejects_unsupported_platform() { pass "ShellCheck installer rejects an unsupported OS or architecture" } +test_missing_shellcheck_fails_closed() { + local tmp fakebin out rc tool + tmp=$(fm_test_tmproot fm-lint-noshellcheck) + fakebin=$(fm_fakebin "$tmp") + for tool in bash dirname; do + ln -s "$(command -v "$tool")" "$fakebin/$tool" + done + rc=0 + out=$(PATH="$fakebin" CI=true GITHUB_ACTIONS=true "$LINT" 2>&1) || rc=$? + [ "$rc" -eq 1 ] || fail "missing ShellCheck expected exit 1, got $rc"$'\n'"$out" + assert_contains "$out" "ShellCheck not found" \ + "missing ShellCheck did not name the required linter" + assert_contains "$out" "$REQUIRED" \ + "missing ShellCheck did not name the pinned version" + assert_contains "$out" "fm-install-shellcheck.sh" \ + "missing ShellCheck did not name the pinned installer" + pass "missing ShellCheck fails closed" +} + test_rejects_wrong_shellcheck_version() { # Version-independent: a fake shellcheck reporting a different version must be # refused before any lint, proving local and CI cannot silently diverge. @@ -866,6 +885,7 @@ test_installer_rejects_wrong_checksum test_installer_falls_back_to_shasum test_installer_prefers_sha256sum_over_shasum test_installer_rejects_unsupported_platform +test_missing_shellcheck_fails_closed test_rejects_wrong_shellcheck_version test_catches_a_real_lint_defect test_ignores_ambient_shellcheck_opts diff --git a/tests/fm-muse-harness.test.sh b/tests/fm-muse-harness.test.sh index ac077c9d8da..83a0747458b 100755 --- a/tests/fm-muse-harness.test.sh +++ b/tests/fm-muse-harness.test.sh @@ -13,6 +13,13 @@ set -u # shellcheck source=tests/lib.sh . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +# bin/fm-harness.sh checks verified ENV markers before ancestry. Muse is +# markerless, so an inherited Cursor/Claude/Pi/Grok marker would outrank the +# versioned muse-bin ancestor these detection cases launch. Drop the ambient +# markers so the asserted verdict does not depend on which harness launched +# the suite. +unset CLAUDECODE PI_CODING_AGENT FM_PI_HARNESS GROK_AGENT CURSOR_AGENT CURSOR_INVOKED_AS + SPAWN="$ROOT/bin/fm-spawn.sh" TEARDOWN="$ROOT/bin/fm-teardown.sh" HARNESS="$ROOT/bin/fm-harness.sh" @@ -154,9 +161,10 @@ run_muse_spawn() { # <home> <proj> <wt> <fakebin> <id> [extra args...] # string, so each case launches an actual renamed executable and asks # fm-harness.sh from a child of it. # -# The foreign env markers are cleared because muse is markerless and the marker -# layer deliberately outranks ancestry: with one retained, these cases would -# assert the marker's verdict instead of the ancestry match they exist to pin. +# The foreign env markers, including Cursor's, are cleared because muse is +# markerless and the marker layer deliberately outranks ancestry: with one +# retained, these cases would assert the marker's verdict instead of the +# ancestry match they exist to pin. # The command substitution around the probe is load-bearing: a bare `-c <cmd>` # lets the shell exec the probe in place, which REPLACES the muse-bin-* process # name the walk is supposed to find. Real muse keeps its TUI process alive and @@ -167,7 +175,8 @@ test_detects_versioned_process_ancestor() { mkdir -p "$dir" for bin in muse-bin-0.1.0-R708.1 muse-bin-9.9.9-RZZZ.9 muse; do cp "$(command -v bash)" "$dir/$bin" - out=$(env -u CLAUDECODE -u PI_CODING_AGENT -u GROK_AGENT \ + out=$(env -u CLAUDECODE -u PI_CODING_AGENT -u FM_PI_HARNESS -u GROK_AGENT \ + -u CURSOR_AGENT -u CURSOR_INVOKED_AS \ "$dir/$bin" -c "r=\$(\"$HARNESS\"); printf '%s' \"\$r\"") [ "$out" = muse ] || fail "fm-harness.sh under process '$bin' reported '$out', expected muse" done @@ -182,7 +191,8 @@ test_detection_is_anchored() { mkdir -p "$dir" for bin in musescore amuse notmuse-bin muse-binary muse-bind; do cp "$(command -v bash)" "$dir/$bin" - out=$(env -u CLAUDECODE -u PI_CODING_AGENT -u GROK_AGENT \ + out=$(env -u CLAUDECODE -u PI_CODING_AGENT -u FM_PI_HARNESS -u GROK_AGENT \ + -u CURSOR_AGENT -u CURSOR_INVOKED_AS \ "$dir/$bin" -c "r=\$(\"$HARNESS\"); printf '%s' \"\$r\"") [ "$out" != muse ] || fail "fm-harness.sh misdetected unrelated process '$bin' as muse" done @@ -197,6 +207,7 @@ $rec EOF result="$case_dir/harness-result" out=$(CLAUDECODE=1 PI_CODING_AGENT=true GROK_AGENT=1 FM_PI_HARNESS=pi-signed \ + CURSOR_AGENT=1 CURSOR_INVOKED_AS=cursor-agent \ FM_FAKE_EXECUTE_MUSE_LAUNCH=1 FM_FAKE_HARNESS_RESULT="$result" \ run_muse_spawn "$home" "$proj" "$wt" "$fakebin" "$id" --mode no-mistakes --yolo off) status=$? diff --git a/tests/fm-no-mistakes-attestation.test.sh b/tests/fm-no-mistakes-attestation.test.sh new file mode 100755 index 00000000000..714bece23f9 --- /dev/null +++ b/tests/fm-no-mistakes-attestation.test.sh @@ -0,0 +1,83 @@ +#!/usr/bin/env bash +# Behavior of the "PR must be raised via no-mistakes" gate in +# .github/workflows/no-mistakes-required.yml. +# +# The step script is the executable contract here: this test parses the +# workflow into a semantic model, pulls out that step's `run` body, and runs it +# as GitHub Actions would, with PR_BODY fixtures. +# +# Regression origin: PR #7 was rejected with "structured pipeline step +# attestation is missing or unparseable" even though its body carried a valid +# attestation. Its evidence section quoted the attestation comment first (the +# placeholder from this check's own error text), and the check read only the +# first occurrence. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +WORKFLOW="$ROOT/.github/workflows/no-mistakes-required.yml" +TMP_ROOT=$(fm_test_tmproot fm-no-mistakes-attestation) + +STEP="$TMP_ROOT/step.sh" +python3 - "$WORKFLOW" "$STEP" <<'PY' +import sys, yaml + +wf = yaml.safe_load(open(sys.argv[1])) +steps = wf["jobs"]["check"]["steps"] +run = [s["run"] for s in steps if s["name"] == "Verify no-mistakes signature in PR body"] +assert len(run) == 1, f"expected exactly one verify step, got {len(run)}" +open(sys.argv[2], "w").write(run[0]) +PY + +SIGNATURE='Updates from [git push no-mistakes](https://github.com/kunchenguid/no-mistakes)' +ATTESTATION='<!-- no-mistakes-pipeline-attestation:v1 {"head_sha":"deadbeef","steps":[{"step":"review","status":"completed"},{"step":"test","status":"completed"},{"step":"document","status":"completed"}]} -->' +PLACEHOLDER='<!-- no-mistakes-pipeline-attestation:v1 {"head_sha":"...","steps":[...]} -->' + +run_step() { + PR_BODY=$1 PR_AUTHOR=tester PR_NUMBER=7 bash "$STEP" >"$TMP_ROOT/out" 2>"$TMP_ROOT/err" +} + +# 1. A body that quotes the placeholder before its real attestation still passes. +if run_step "$SIGNATURE + +Evidence quoting the guidance: + + $PLACEHOLDER + +## Pipeline + +$ATTESTATION"; then + pass "quoted placeholder before the real attestation does not shadow it" +else + fail "quoted placeholder shadowed the real attestation: $(cat "$TMP_ROOT/err")" +fi + +# 2. No attestation at all is still rejected. +if run_step "$SIGNATURE"; then + fail "signature-only legacy body was accepted" +else + grep -q 'attestation is missing or unparseable' "$TMP_ROOT/err" \ + || fail "legacy body rejected without the >= 1.46.0 guidance" + pass "signature-only legacy body is rejected" +fi + +# 3. A required step that is not completed is still rejected. +if run_step "$SIGNATURE + +<!-- no-mistakes-pipeline-attestation:v1 {\"head_sha\":\"deadbeef\",\"steps\":[{\"step\":\"review\",\"status\":\"completed\"},{\"step\":\"test\",\"status\":\"skipped\"}]} -->"; then + fail "attestation with test=skipped was accepted" +else + grep -q 'test=skipped, document=missing' "$TMP_ROOT/err" \ + || fail "skipped step rejected without naming the incomplete steps: $(cat "$TMP_ROOT/err")" + pass "attestation with an incomplete required step is rejected" +fi + +# 4. A body with no signature at all is rejected as not raised via no-mistakes. +if run_step "just a hand-written PR body"; then + fail "unsigned body was accepted" +else + grep -q 'was not raised through no-mistakes' "$TMP_ROOT/err" \ + || fail "unsigned body rejected with the wrong error" + pass "body without the no-mistakes signature is rejected" +fi diff --git a/tests/fm-pi-watch-extension.test.sh b/tests/fm-pi-watch-extension.test.sh index 9e29adc793f..fb473ee0343 100755 --- a/tests/fm-pi-watch-extension.test.sh +++ b/tests/fm-pi-watch-extension.test.sh @@ -12,6 +12,18 @@ EXT="$ROOT/.pi/extensions/fm-primary-pi-watch.ts" # unrelated to plugin output, which the assertions intentionally require empty. export NODE_NO_WARNINGS=1 +# One owner for the readiness budget every unready-successor test below spends +# on purpose. Both plugins start a successor arm through a login shell and +# SIGTERM it when it stays silent past this budget, so the budget has to outlast +# a cold login-shell start. A successor killed before its first statement never +# appends its arm row and never installs the TERM trap these tests observe, so +# too small a budget reports a lost successor instead of the bounded recovery +# under test. A stock login shell already costs about 200ms on an idle +# workstation, and a loaded CI runner is slower, so keep an order of magnitude +# over that rather than a value that only holds locally. The wait loops in those +# tests are sized against this number. +ARM_READY_TIMEOUT_MS=2000 + install_pi_watch_extension_fixture() { local repo=$1 mkdir -p \ @@ -365,7 +377,7 @@ const pi = { }, sendUserMessage: async () => { rowsAtDelivery = existsSync(process.env.FM_ARM_LOG) - ? readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n").length + ? readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n").filter((row) => row.startsWith("arm=")).length : 0; deliveryStarted = true; await deliveryBlocked; @@ -383,22 +395,23 @@ for (let i = 0; i < 250; i += 1) { await new Promise((resolve) => setTimeout(resolve, 10)); } const rows = readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n"); -if (rows.length !== 2) throw new Error(`expected one successor arm, got ${rows.length}: ${rows.join(" | ")}`); +const armRows = rows.filter((row) => row.startsWith("arm=")); +if (armRows.length !== 2) throw new Error(`expected one successor arm, got ${armRows.length}: ${rows.join(" | ")}`); if (!deliveryStarted) throw new Error("wake delivery did not begin"); if (rowsAtDelivery !== 2) throw new Error(`wake delivery began before successor establishment (${rowsAtDelivery} arm rows)`); -if (!/predecessor=[0-9]+/.test(rows[1])) throw new Error(`successor did not receive predecessor identity: ${rows[1]}`); +if (!/predecessor=[0-9]+/.test(armRows[1])) throw new Error(`successor did not receive predecessor identity: ${armRows[1]}`); +if (!rows.some((row) => row.startsWith("confirmed generation=fixture-generation"))) { + throw new Error(`handling delivery was not confirmed before the follow-up: ${rows.join(" | ")}`); +} await new Promise((resolve) => setTimeout(resolve, 100)); const stableRows = readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n"); -if (stableRows.length !== 2) throw new Error(`delivery was confirmed before the prompt succeeded: ${stableRows.join(" | ")}`); -releaseDelivery(); -for (let i = 0; i < 100; i += 1) { - if (readFileSync(process.env.FM_ARM_LOG, "utf8").includes("confirmed generation=fixture-generation")) break; - await new Promise((resolve) => setTimeout(resolve, 10)); +if (stableRows.filter((row) => row.startsWith("arm=")).length !== 2) { + throw new Error(`blocked follow-up started extra arm work: ${stableRows.join(" | ")}`); } -const confirmedRows = readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n"); -if (confirmedRows.filter((row) => row.startsWith("confirmed ")).length !== 1) { - throw new Error(`successful prompt delivery was not confirmed exactly once: ${confirmedRows.join(" | ")}`); +if (stableRows.filter((row) => row.startsWith("confirmed ")).length !== 1) { + throw new Error(`successful prompt delivery was not confirmed exactly once: ${stableRows.join(" | ")}`); } +releaseDelivery(); writeFileSync(process.env.FM_STOP_FILE, "stop\n"); process.exit(0); EOF @@ -409,6 +422,79 @@ EOF pass "Pi actionable close starts one successor before wake delivery settles" } +test_pi_handling_delivery_failure_is_typed_once() { + local repo home plugin log stop out status + repo="$TMP_ROOT/pi-handling-fail-root" + home="$TMP_ROOT/pi-handling-fail-home" + log="$TMP_ROOT/pi-handling-fail.log" + stop="$TMP_ROOT/pi-handling-fail.stop" + mkdir -p "$repo/bin" "$home/state" "$home/config" + install_pi_watch_extension_fixture "$repo" + plugin="$repo/.pi/extensions/fm-primary-pi-watch.ts" + cat > "$repo/bin/fm-watch-arm.sh" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = --handling-delivered ]; then + printf 'refused generation=%s watcher=%s\n' "$2" "$4" >> "${FM_ARM_LOG:?}" + exit 1 +fi +printf 'arm=%s predecessor=%s\n' "$$" "${FM_WATCH_PREDECESSOR_ARM_PID:-none}" >> "${FM_ARM_LOG:?}" +count=$(grep -c '^arm=' "$FM_ARM_LOG") +if [ "$count" -eq 1 ]; then + printf 'watcher: started pid=%s (beacon fresh)\n' "$$" + printf 'signal: synthetic actionable close\n' + exit 0 +fi +printf 'watcher: started pid=%s (beacon fresh) recovery-generation=fixture-generation\n' "$$" +trap 'exit 0' TERM INT +while [ ! -e "$FM_STOP_FILE" ]; do sleep 0.02; done +SH + chmod +x "$repo/bin/fm-watch-arm.sh" + out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_STOP_FILE="$stop" node --input-type=module 2>&1 <<'EOF' +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +let tool = null; +let prompt = ""; +const pi = { + on() {}, + registerCommand() {}, + registerTool(candidate) { + if (candidate.name === "fm_watch_arm_pi") tool = candidate; + }, + sendUserMessage: async (message) => { + prompt += message; + }, +}; +writeFileSync(`${process.env.FM_HOME}/state/.lock`, `${process.pid}\n`); +const mod = await import(pathToFileURL(process.env.PLUGIN).href); +mod.default(pi); +await tool.execute("tool-call-handling-fail", {}, undefined, undefined, {}); +for (let i = 0; i < 250 && !prompt.includes("handling delivery confirmation was rejected"); i += 1) { + await new Promise((resolve) => setTimeout(resolve, 20)); +} +if (!prompt.includes("FIRSTMATE WATCHER WAKE")) throw new Error(`missing follow-up: ${prompt}`); +if (!prompt.includes("handling delivery confirmation was rejected")) { + throw new Error(`failed handshake was swallowed: ${prompt}`); +} +if ((prompt.match(/FIRSTMATE WATCHER WAKE/g) || []).length !== 1) { + throw new Error(`failed handshake was not a single typed message: ${prompt}`); +} +const rows = existsSync(process.env.FM_ARM_LOG) + ? readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n") + : []; +if (rows.filter((row) => row.startsWith("refused ")).length < 1) { + throw new Error(`handling-delivered was never attempted: ${rows.join(" | ")}`); +} +writeFileSync(process.env.FM_STOP_FILE, "stop\n"); +process.exit(0); +EOF +) + status=$? + expect_code 0 "$status" "Pi must surface a refused handling handshake as one typed failure" + [ -z "$out" ] || fail "Pi handling-delivery failure test printed output: $out" + pass "Pi refused handling handshake is classified and not swallowed" +} + test_pi_hung_successor_falls_back_to_typed_wake() { local repo home plugin log out status repo="$TMP_ROOT/pi-hung-successor-root" @@ -430,7 +516,7 @@ trap 'exit 0' TERM INT while :; do sleep 0.02; done SH chmod +x "$repo/bin/fm-watch-arm.sh" - out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_PI_ARM_READY_TIMEOUT_MS=250 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node --input-type=module 2>&1 <<'EOF' + out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_PI_ARM_READY_TIMEOUT_MS="$ARM_READY_TIMEOUT_MS" FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node --input-type=module 2>&1 <<'EOF' import { existsSync, readFileSync, writeFileSync } from "node:fs"; import { pathToFileURL } from "node:url"; @@ -454,7 +540,9 @@ writeFileSync(`${process.env.FM_HOME}/state/.lock`, `${process.pid}\n`); const mod = await import(pathToFileURL(process.env.PLUGIN).href); mod.default(pi); await tool.execute("tool-call-hung-successor", {}, undefined, undefined, {}); -for (let i = 0; i < 500 && !prompt; i += 1) { +// Three unready successors each cost the full readiness budget, so wait well +// past their sum. The wait ends as soon as the wake lands. +for (let i = 0; i < 1500 && !prompt; i += 1) { await new Promise((resolve) => setTimeout(resolve, 10)); } const rows = existsSync(process.env.FM_ARM_LOG) @@ -502,7 +590,7 @@ printf 'arm=%s\n' "$$" >> "${FM_ARM_LOG:?}" while [ ! -e "$FM_RELEASE_FILE" ]; do sleep 0.1; done SH chmod +x "$repo/bin/fm-watch-arm.sh" - out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_RELEASE_FILE="$release" FM_PI_ARM_READY_TIMEOUT_MS=250 FM_WATCH_ARM_RETIRE_TIMEOUT_MS=20 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node --input-type=module 2>&1 <<'EOF' + out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_RELEASE_FILE="$release" FM_PI_ARM_READY_TIMEOUT_MS="$ARM_READY_TIMEOUT_MS" FM_WATCH_ARM_RETIRE_TIMEOUT_MS=20 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node --input-type=module 2>&1 <<'EOF' import { existsSync, readFileSync, writeFileSync } from "node:fs"; import { pathToFileURL } from "node:url"; @@ -580,7 +668,7 @@ trap 'exit 0' TERM INT while [ ! -e "$FM_STOP_FILE" ]; do sleep 0.02; done SH chmod +x "$repo/bin/fm-watch-arm.sh" - out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_UNRETIRED_READY_FILE="$ready" FM_UNRETIRED_RETIRE_FILE="$retired" FM_RELEASE_FILE="$release" FM_STOP_FILE="$stop" FM_LATE_KIND="$kind" FM_PI_ARM_READY_TIMEOUT_MS=250 FM_WATCH_ARM_RETIRE_TIMEOUT_MS=20 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node --input-type=module 2>&1 <<'EOF' + out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_UNRETIRED_READY_FILE="$ready" FM_UNRETIRED_RETIRE_FILE="$retired" FM_RELEASE_FILE="$release" FM_STOP_FILE="$stop" FM_LATE_KIND="$kind" FM_PI_ARM_READY_TIMEOUT_MS="$ARM_READY_TIMEOUT_MS" FM_WATCH_ARM_RETIRE_TIMEOUT_MS=20 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node --input-type=module 2>&1 <<'EOF' import { existsSync, readFileSync, writeFileSync } from "node:fs"; import { pathToFileURL } from "node:url"; @@ -1338,7 +1426,16 @@ const hooks = await mod.FmPrimaryWatchArm({ const event = { event: { type: "session.idle", properties: { sessionID: "session-test" } } }; writeFileSync(`${process.env.FM_HOME}/state/.lock`, "999999\n"); await hooks.event(event); -await new Promise((resolve) => setTimeout(resolve, 120)); +// The hook starts its attempt without awaiting it, and the plugin answers a +// second attempt from the one already in flight. Join that attempt through the +// coordinator rather than waiting a fixed span: refusing an unowned lock walks +// git and ps probes that can outlast any such span, and the owned-lock event +// below would then be answered from the refusal instead of arming. +const refusal = await globalThis.__firstmateOpenCodeWatchArm.ensureArmed("session-test", client); +if (refusal !== "read-only") { + console.error(`expected a read-only refusal without the session lock, got ${refusal}`); + process.exit(1); +} if (existsSync(process.env.FM_ARM_LOG)) { console.error("watch arm ran without owning the session lock"); process.exit(1); @@ -1451,7 +1548,7 @@ const client = { session: { promptAsync: async () => { rowsAtPrompt = existsSync(process.env.FM_ARM_LOG) - ? readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n").length + ? readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n").filter((row) => row.startsWith("arm=")).length : 0; prompts += 1; await promptBlocked; @@ -1474,22 +1571,23 @@ for (let i = 0; i < 250; i += 1) { await new Promise((resolve) => setTimeout(resolve, 10)); } const rows = readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n"); -if (rows.length !== 2) throw new Error(`expected one successor arm, got ${rows.length}: ${rows.join(" | ")}`); +const armRows = rows.filter((row) => row.startsWith("arm=")); +if (armRows.length !== 2) throw new Error(`expected one successor arm, got ${armRows.length}: ${rows.join(" | ")}`); if (prompts !== 1) throw new Error(`expected one blocked wake prompt, got ${prompts}`); if (rowsAtPrompt !== 2) throw new Error(`wake prompt began before successor establishment (${rowsAtPrompt} arm rows)`); -if (!/predecessor=[0-9]+/.test(rows[1])) throw new Error(`successor did not receive predecessor identity: ${rows[1]}`); +if (!/predecessor=[0-9]+/.test(armRows[1])) throw new Error(`successor did not receive predecessor identity: ${armRows[1]}`); +if (!rows.some((row) => row.startsWith("confirmed generation=fixture-generation"))) { + throw new Error(`handling delivery was not confirmed before the follow-up: ${rows.join(" | ")}`); +} await new Promise((resolve) => setTimeout(resolve, 100)); const stableRows = readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n"); -if (stableRows.length !== 2) throw new Error(`delivery was confirmed before the prompt succeeded: ${stableRows.join(" | ")}`); -releasePrompt(); -for (let i = 0; i < 100; i += 1) { - if (readFileSync(process.env.FM_ARM_LOG, "utf8").includes("confirmed generation=fixture-generation")) break; - await new Promise((resolve) => setTimeout(resolve, 10)); +if (stableRows.filter((row) => row.startsWith("arm=")).length !== 2) { + throw new Error(`blocked follow-up started extra arm work: ${stableRows.join(" | ")}`); } -const confirmedRows = readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n"); -if (confirmedRows.filter((row) => row.startsWith("confirmed ")).length !== 1) { - throw new Error(`successful prompt delivery was not confirmed exactly once: ${confirmedRows.join(" | ")}`); +if (stableRows.filter((row) => row.startsWith("confirmed ")).length !== 1) { + throw new Error(`successful prompt delivery was not confirmed exactly once: ${stableRows.join(" | ")}`); } +releasePrompt(); writeFileSync(process.env.FM_STOP_FILE, "stop\n"); EOF ) @@ -1605,7 +1703,7 @@ trap 'exit 0' TERM INT while :; do sleep 0.02; done SH chmod +x "$repo/bin/fm-watch-arm.sh" - out=$(PLUGIN="$plugin" WORKTREE="$repo" FM_HOME="$home" FM_ARM_LOG="$log" FM_OPENCODE_ARM_READY_TIMEOUT_MS=250 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node 2>&1 <<'EOF' + out=$(PLUGIN="$plugin" WORKTREE="$repo" FM_HOME="$home" FM_ARM_LOG="$log" FM_OPENCODE_ARM_READY_TIMEOUT_MS="$ARM_READY_TIMEOUT_MS" FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node 2>&1 <<'EOF' import { existsSync, readFileSync, writeFileSync } from "node:fs"; import { pathToFileURL } from "node:url"; @@ -1629,7 +1727,9 @@ const hooks = await mod.FmPrimaryWatchArm({ }); writeFileSync(`${process.env.FM_HOME}/state/.lock`, `${process.pid}\n`); await hooks.event({ event: { type: "session.idle", properties: { sessionID: "session-test" } } }); -for (let i = 0; i < 500 && !prompt; i += 1) { +// Three unready successors each cost the full readiness budget, so wait well +// past their sum. The wait ends as soon as the wake lands. +for (let i = 0; i < 1500 && !prompt; i += 1) { await new Promise((resolve) => setTimeout(resolve, 10)); } const rows = existsSync(process.env.FM_ARM_LOG) @@ -1679,7 +1779,7 @@ printf 'arm=%s\n' "$$" >> "${FM_ARM_LOG:?}" while [ ! -e "$FM_RELEASE_FILE" ]; do sleep 0.1; done SH chmod +x "$repo/bin/fm-watch-arm.sh" - out=$(PLUGIN="$plugin" WORKTREE="$repo" FM_HOME="$home" FM_ARM_LOG="$log" FM_RELEASE_FILE="$release" FM_OPENCODE_ARM_READY_TIMEOUT_MS=250 FM_WATCH_ARM_RETIRE_TIMEOUT_MS=20 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node 2>&1 <<'EOF' + out=$(PLUGIN="$plugin" WORKTREE="$repo" FM_HOME="$home" FM_ARM_LOG="$log" FM_RELEASE_FILE="$release" FM_OPENCODE_ARM_READY_TIMEOUT_MS="$ARM_READY_TIMEOUT_MS" FM_WATCH_ARM_RETIRE_TIMEOUT_MS=20 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node 2>&1 <<'EOF' import { existsSync, readFileSync, writeFileSync } from "node:fs"; import { pathToFileURL } from "node:url"; @@ -1759,7 +1859,7 @@ trap 'exit 0' TERM INT while [ ! -e "$FM_STOP_FILE" ]; do sleep 0.02; done SH chmod +x "$repo/bin/fm-watch-arm.sh" - out=$(PLUGIN="$plugin" WORKTREE="$repo" FM_HOME="$home" FM_ARM_LOG="$log" FM_UNRETIRED_READY_FILE="$ready" FM_UNRETIRED_RETIRE_FILE="$retired" FM_RELEASE_FILE="$release" FM_STOP_FILE="$stop" FM_LATE_KIND="$kind" FM_OPENCODE_ARM_READY_TIMEOUT_MS=250 FM_WATCH_ARM_RETIRE_TIMEOUT_MS=20 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node 2>&1 <<'EOF' + out=$(PLUGIN="$plugin" WORKTREE="$repo" FM_HOME="$home" FM_ARM_LOG="$log" FM_UNRETIRED_READY_FILE="$ready" FM_UNRETIRED_RETIRE_FILE="$retired" FM_RELEASE_FILE="$release" FM_STOP_FILE="$stop" FM_LATE_KIND="$kind" FM_OPENCODE_ARM_READY_TIMEOUT_MS="$ARM_READY_TIMEOUT_MS" FM_WATCH_ARM_RETIRE_TIMEOUT_MS=20 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node 2>&1 <<'EOF' import { existsSync, readFileSync, writeFileSync } from "node:fs"; import { pathToFileURL } from "node:url"; @@ -2155,6 +2255,7 @@ test_pi_tool_returns_agent_tool_result test_pi_redundant_tool_call_is_owned_noop test_pi_scheduled_retry_call_is_owned_noop test_pi_actionable_close_starts_single_successor_before_delivery +test_pi_handling_delivery_failure_is_typed_once test_pi_hung_successor_falls_back_to_typed_wake test_pi_unretired_successor_falls_back_without_retry test_pi_late_unretired_close_resumes_supervision diff --git a/tests/fm-pr-check-security.test.sh b/tests/fm-pr-check-security.test.sh index 03c6ce688e3..f9795c3d361 100755 --- a/tests/fm-pr-check-security.test.sh +++ b/tests/fm-pr-check-security.test.sh @@ -26,6 +26,10 @@ REAL_MV=$(command -v mv) REAL_STAT=$(command -v stat) REAL_CHMOD=$(command -v chmod) REAL_BASENAME=$(command -v basename) +# The merge path reads a merge request's JSON with the real jq, and BASE_PATH is +# deliberately restricted, so a case that needs jq exposes this one rather than +# depending on the host keeping jq in one of those four directories. +REAL_JQ=$(command -v jq) || fail "these tests read glab's JSON with the real jq, which was not found" ack_watcher_cycle() { # <state> local state=$1 err sequence generation @@ -2886,15 +2890,27 @@ EOF esac [ ! -e "$state/task-b.check.sh" ] || fail "refused GitLab arming left a poll armed" - # The merge path still addresses GitHub only, so it refuses rather than - # sending a merge request to the wrong forge. + # The merge path addresses the forge the URL names, and never the other one. + # This fixture's glab answers with the field output the poll reads, so the + # merge's JSON read cannot be parsed, which must refuse rather than merge on a + # state it could not read. write_task_meta "$dir" task-c + : > "$dir/glab.log" + # The merge path needs jq before it reads anything, so this case supplies it + # and the refusal below is the unreadable state rather than a missing tool. + ln -sf "$REAL_JQ" "$dir/fakebin/jq" set +e - run_merge_entry "$dir" task-c "$url" >/dev/null 2>&1 + run_merge_entry "$dir" task-c "$url" >/dev/null 2> "$dir/merge-c.err" rc=$? set -e - [ "$rc" -eq 2 ] || fail "merge wrapper did not refuse a GitLab merge request URL" + [ "$rc" -ne 0 ] || fail "merge wrapper merged a GitLab merge request it could not read" + grep -qF 'could not read the GitLab merge request state before merging' "$dir/merge-c.err" \ + || fail "merge wrapper refused for some reason other than the state it could not read" [ ! -s "$dir/gh-axi.log" ] || fail "merge wrapper reached the GitHub CLI for a GitLab URL" + grep -qF "mr view 7 -R https://gitlab.example/group/subgroup/project" "$dir/glab.log" \ + || fail "merge wrapper did not read the merge request through glab at its own instance" + ! grep -qF ' mr merge ' "$dir/glab.log" \ + || fail "merge wrapper merged despite an unreadable merge request state" pass "GitLab merge requests are followed on any instance and never wake falsely" } diff --git a/tests/fm-pr-merge.test.sh b/tests/fm-pr-merge.test.sh index a064b6919bc..2367e8e5d22 100755 --- a/tests/fm-pr-merge.test.sh +++ b/tests/fm-pr-merge.test.sh @@ -13,7 +13,17 @@ # (e) PR URL is parsed to number + --repo for gh-axi (defaults to --squash) # (f) malformed PR URL fails fast without calling gh-axi # (g) explicit merge method is not overridden by the default --squash -# (h) repo override args fail fast because the repo comes from the URL +# (h) repo override args fail fast because the repo comes from the URL, +# including a bundled short-option cluster that carries -R +# (i) a GitLab MR URL resolves and merges through glab instead of erroring +# (j) glab is addressed by the host from the URL, never an assumed one +# (k) no merge method is imposed on GitLab, so the project's own one applies +# (l) each pre-merge condition refuses independently, and all of them report +# (m) a stale recorded pr_head= is reported and the live head is verified +# (n) an unreadable merge request state refuses rather than merging blind +# (o) glab or jq absent refuses before any state is recorded +# (p) --sha in extra GitLab args fails fast, and still forwards on GitHub +# (q) a GitLab refusal still leaves pr= recorded and the merge poll armed set -u # shellcheck source=tests/lib.sh @@ -22,6 +32,18 @@ fm_git_identity fmtest fmtest@example.invalid PR_MERGE="$ROOT/bin/fm-pr-merge.sh" TMP_ROOT=$(fm_test_tmproot fm-pr-merge-tests) +BASE_PATH=$PATH + +# The GitLab fixture. A placeholder host that resolves nowhere, and a namespace +# deeper than one group, because a GitLab project has no owner/repository pair. +MR_HOST=gitlab.example +MR_PATH=group/subgroup/project +MR_PROJECT_URL="https://$MR_HOST/$MR_PATH" +MR_URL="$MR_PROJECT_URL/-/merge_requests/7" +MR_HEAD=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa +MR_STALE_HEAD=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb + +JQ_BIN=$(command -v jq) || fail "these tests read glab's JSON with the real jq, which was not found" # Build a fresh sandbox for one test case: a state dir with a task meta and a # fakebin with a gh-axi mock that records how it was invoked. Echoes the case dir. @@ -84,11 +106,119 @@ SH chmod +x "$case_dir/fakebin/gh-axi" "$case_dir/fakebin/gh" } +# glab mock recording every invocation together with the GITLAB_HOST it was +# given, so a test can prove the instance came from the URL. `mr view` answers +# from the case's JSON payload; marker files in the case dir drive the failure +# modes, so no test has to leak environment into a shared runner. +add_glab_mock() { + local case_dir=$1 + cat > "$case_dir/fakebin/glab" <<'SH' +#!/usr/bin/env bash +printf 'GITLAB_HOST=%s %s\n' "${GITLAB_HOST-<unset>}" "$*" >> "$FM_TEST_GLAB_LOG" +case_dir=$(dirname "$FM_TEST_GLAB_JSON") +case "${1:-} ${2:-}" in + "mr view") + [ ! -e "$case_dir/glab-view-fails" ] || exit 1 + cat "$FM_TEST_GLAB_JSON" + exit 0 + ;; + "mr merge") + [ ! -e "$case_dir/glab-merge-fails" ] || { echo "error: mr merge failed" >&2 ; exit 1 ; } + exit 0 + ;; +esac +exit 0 +SH + chmod +x "$case_dir/fakebin/glab" + ln -sf "$JQ_BIN" "$case_dir/fakebin/jq" +} + +# write_mr_json <file> [<field>=<value> ...] +# A merge request payload that satisfies every pre-merge condition, with the +# named fields overridden so one case drives exactly one condition. Values are +# written into the JSON as-is, so a value may carry a JSON escape. +write_mr_json() { + local file=$1 kv key value + local state=opened detail=mergeable conflicts=false discussions=true + local head=$MR_HEAD pipeline_sha=$MR_HEAD pipeline_status=success pipeline=present + shift + for kv in "$@"; do + key=${kv%%=*} + value=${kv#*=} + case "$key" in + state) state=$value ;; + detail) detail=$value ;; + conflicts) conflicts=$value ;; + discussions) discussions=$value ;; + head) head=$value ;; + pipeline_sha) pipeline_sha=$value ;; + pipeline_status) pipeline_status=$value ;; + pipeline) pipeline=$value ;; + *) fail "write_mr_json: unknown field '$key'" ;; + esac + done + if [ "$pipeline" = present ]; then + pipeline=$(printf '{"sha":"%s","status":"%s"}' "$pipeline_sha" "$pipeline_status") + fi + printf '{"iid":7,"state":"%s","detailed_merge_status":"%s","has_conflicts":%s,' \ + "$state" "$detail" "$conflicts" > "$file" + printf '"blocking_discussions_resolved":%s,"sha":"%s","head_pipeline":%s}\n' \ + "$discussions" "$head" "$pipeline" >> "$file" +} + +# make_gitlab_case <name> [<field>=<value> ...]: a case dir with both forge +# mocks and a merge request payload. Echoes the case dir. +make_gitlab_case() { + local name=$1 case_dir + shift + case_dir=$(make_case "$name") + mkdir -p "$case_dir/wt" + add_gh_mocks "$case_dir" cccccccccccccccccccccccccccccccccccccccc + add_glab_mock "$case_dir" + : > "$case_dir/gh-axi.log" + : > "$case_dir/glab.log" + write_mr_json "$case_dir/mr.json" "$@" + printf '%s\n' "$case_dir" +} + +# mirror_path_without <dir> <tool> [<bindir> ...]: the whole search path +# re-exposed by symlink except one tool, because a real copy anywhere on PATH +# would prove nothing. The named bindirs are mirrored ahead of the search path, +# so the case's own mocks answer for every tool that is not the omitted one and +# the refusal names that tool alone whatever the host happens to have installed. +mirror_path_without() { + local dir=$1 omit=$2 search bindir entry name + shift 2 + mkdir -p "$dir" + search=$(printf '%s\n' "$@"; printf '%s\n' "$BASE_PATH" | tr ':' '\n') + while IFS= read -r bindir; do + [ -d "$bindir" ] || continue + for entry in "$bindir"/*; do + [ -e "$entry" ] || continue + name=${entry##*/} + [ "$name" = "$omit" ] && continue + [ -e "$dir/$name" ] || ln -s "$entry" "$dir/$name" 2>/dev/null + done + done <<EOF +$search +EOF + ! PATH="$dir" command -v "$omit" >/dev/null 2>&1 \ + || fail "the $omit-free search path still resolved $omit" +} + +# The merge line glab was asked to run, so a test asserts one exact invocation +# rather than a substring of the whole log. +glab_merge_line() { + grep -F ' mr merge ' "$1" || true +} + run_pr_merge() { local case_dir=$1 rc; shift FM_ROOT_OVERRIDE="$ROOT" \ FM_STATE_OVERRIDE="$case_dir/state" \ FM_TEST_GH_AXI_LOG="$case_dir/gh-axi.log" \ + FM_TEST_GLAB_LOG="$case_dir/glab.log" \ + FM_TEST_GLAB_JSON="$case_dir/mr.json" \ PATH="$case_dir/fakebin:$PATH" \ "$PR_MERGE" "$@" rc=$? @@ -187,15 +317,18 @@ test_malformed_url_refuses_before_merge() { : > "$case_dir/gh-axi.log" set +e - run_pr_merge "$case_dir" task-x1 'https://gitlab.com/example/repo/-/merge_requests/1' \ + # A near-miss GitLab URL: one namespace segment where a project needs at + # least two. A well-formed merge request URL is merged now, so the refusal + # has to be proven on a URL that genuinely does not parse. + run_pr_merge "$case_dir" task-x1 'https://gitlab.com/example/-/merge_requests/1' \ > "$case_dir/stdout" 2> "$case_dir/stderr" rc=$? set -e - expect_code 2 "$rc" "malformed-url: fm-pr-merge should refuse a non-GitHub PR URL" + expect_code 2 "$rc" "malformed-url: fm-pr-merge should refuse a malformed merge request URL" assert_grep 'error: invalid PR merge request' "$case_dir/stderr" \ "malformed-url: refusal was not fixed and non-probing" - assert_no_grep 'pr=https://gitlab.com/example/repo/-/merge_requests/1' "$case_dir/state/task-x1.meta" \ + assert_no_grep 'pr=https://gitlab.com/example/-/merge_requests/1' "$case_dir/state/task-x1.meta" \ "malformed-url: malformed PR URL was recorded in meta" assert_absent "$case_dir/state/task-x1.check.sh" \ "malformed-url: malformed PR URL armed a merge poll" @@ -256,6 +389,67 @@ test_repo_override_args_refuse_before_recording() { pass "fm-pr-merge refuses repo override args before recording state" } +# A bundled short-option cluster carries -R without ever being exactly -R, and +# both CLIs expand it one character at a time, so the guard has to read the +# whole cluster. On GitLab that redirect names an instance, not only a +# repository, so it must refuse before anything is recorded or read. +test_bundled_repo_override_args_refuse_before_recording() { + local case_dir rc + case_dir=$(make_case bundled-repo-override) + mkdir -p "$case_dir/wt" + add_gh_mocks "$case_dir" abababababababababababababababababababab + : > "$case_dir/gh-axi.log" + + set +e + run_pr_merge "$case_dir" task-x1 https://github.com/right/repo/pull/6 -- -dR wrong/repo \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 1 "$rc" "bundled-repo-override: fm-pr-merge should refuse a bundled repo override" + assert_grep 'extra merge arguments must not override the repository' "$case_dir/stderr" \ + "bundled-repo-override: refusal did not explain the repo override" + assert_no_grep 'pr=https://github.com/right/repo/pull/6' "$case_dir/state/task-x1.meta" \ + "bundled-repo-override: PR URL was recorded before rejecting the bundled repo override" + assert_absent "$case_dir/state/task-x1.check.sh" \ + "bundled-repo-override: a bundled repo override armed a merge poll" + assert_no_grep 'pr merge' "$case_dir/gh-axi.log" \ + "bundled-repo-override: gh-axi pr merge was invoked despite the bundled repo override" + + case_dir=$(make_gitlab_case bundled-repo-override-gitlab) + + set +e + run_pr_merge "$case_dir" task-x1 "$MR_URL" -- -yR https://other.example/g/p \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 1 "$rc" "bundled-repo-override-gitlab: fm-pr-merge should refuse a bundled instance override" + assert_grep 'extra merge arguments must not override the repository' "$case_dir/stderr" \ + "bundled-repo-override-gitlab: refusal did not explain the repo override" + assert_no_grep "pr=$MR_URL" "$case_dir/state/task-x1.meta" \ + "bundled-repo-override-gitlab: the URL was recorded before rejecting the bundled override" + assert_absent "$case_dir/state/task-x1.check.sh" \ + "bundled-repo-override-gitlab: a bundled override armed a merge poll" + [ ! -s "$case_dir/glab.log" ] \ + || fail "bundled-repo-override-gitlab: glab was invoked despite the bundled override" + + # Only a cluster carrying the repository flag is refused: every other short + # cluster is still the caller's business and still reaches the forge. + case_dir=$(make_case bundled-non-repo-cluster) + mkdir -p "$case_dir/wt" + add_gh_mocks "$case_dir" bcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbc + : > "$case_dir/gh-axi.log" + + run_pr_merge "$case_dir" task-x1 https://github.com/example/repo/pull/8 -- -d \ + > "$case_dir/stdout" 2> "$case_dir/stderr" \ + || fail "bundled-non-repo-cluster: fm-pr-merge refused a short flag that overrides nothing" + + grep -qxF 'pr merge 8 --repo example/repo --squash -d' "$case_dir/gh-axi.log" \ + || fail "bundled-non-repo-cluster: a short flag carrying no repository override was not forwarded" + pass "fm-pr-merge refuses a bundled short-option repo override and forwards other short flags" +} + test_explicit_merge_method_not_overridden() { local case_dir case_dir=$(make_case explicit-merge-method) @@ -301,6 +495,321 @@ test_parses_pr_url_for_gh_axi() { pass "fm-pr-merge parses a GitHub PR URL into gh-axi number and --repo arguments" } +test_gitlab_url_resolves_and_merges() { + local case_dir rc merge_line + case_dir=$(make_gitlab_case gitlab-merges) + + set +e + run_pr_merge "$case_dir" task-x1 "$MR_URL" \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 0 "$rc" "gitlab-merges: a well-formed merge request URL should merge, not error" + assert_grep "pr=$MR_URL" "$case_dir/state/task-x1.meta" \ + "gitlab-merges: pr= was not recorded before merging" + assert_grep "GITLAB_HOST=$MR_HOST mr view 7 -R $MR_PROJECT_URL -F json" "$case_dir/glab.log" \ + "gitlab-merges: the pre-merge state was not read from the project URL" + merge_line=$(glab_merge_line "$case_dir/glab.log") + [ "$merge_line" = "GITLAB_HOST=$MR_HOST mr merge 7 -R $MR_PROJECT_URL --sha $MR_HEAD --yes" ] \ + || fail "gitlab-merges: unexpected merge invocation: '$merge_line'" + assert_grep "successful pipeline at head $MR_HEAD" "$case_dir/stderr" \ + "gitlab-merges: the verified head was not reported" + [ ! -s "$case_dir/gh-axi.log" ] || fail "gitlab-merges: a merge request reached the GitHub CLI" + pass "fm-pr-merge merges a GitLab merge request through glab instead of refusing it" +} + +test_gitlab_host_comes_from_the_url() { + local case_dir rc host path project_url url + host=gl.self-hosted.example + path=deep/nested/group/project + project_url="https://$host/$path" + url="$project_url/-/merge_requests/31" + case_dir=$(make_gitlab_case gitlab-host-from-url) + + set +e + run_pr_merge "$case_dir" task-x1 "$url" \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 0 "$rc" "gitlab-host-from-url: a self-hosted merge request should merge" + assert_grep "GITLAB_HOST=$host mr view 31 -R $project_url -F json" "$case_dir/glab.log" \ + "gitlab-host-from-url: the read did not use the host from the URL" + assert_grep "GITLAB_HOST=$host mr merge 31 -R $project_url" "$case_dir/glab.log" \ + "gitlab-host-from-url: the merge did not use the host from the URL" + assert_no_grep 'gitlab.com' "$case_dir/glab.log" \ + "gitlab-host-from-url: a host was assumed instead of taken from the URL" + assert_no_grep '<unset>' "$case_dir/glab.log" \ + "gitlab-host-from-url: glab was left to resolve the instance from its own default" + pass "fm-pr-merge takes the GitLab instance from the URL rather than assuming one" +} + +test_gitlab_imposes_no_merge_method() { + local case_dir rc merge_line flag + case_dir=$(make_gitlab_case gitlab-no-method) + + set +e + run_pr_merge "$case_dir" task-x1 "$MR_URL" \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 0 "$rc" "gitlab-no-method: merge should succeed" + merge_line=$(glab_merge_line "$case_dir/glab.log") + for flag in --squash --rebase --merge --method; do + case "$merge_line" in + *"$flag"*) fail "gitlab-no-method: '$flag' was imposed on GitLab: '$merge_line'" ;; + esac + done + pass "fm-pr-merge imposes no merge method on GitLab, leaving the project's own one" +} + +test_gitlab_extra_args_forwarded() { + local case_dir rc merge_line + case_dir=$(make_gitlab_case gitlab-extra-args) + + set +e + run_pr_merge "$case_dir" task-x1 "$MR_URL" -- --remove-source-branch \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 0 "$rc" "gitlab-extra-args: merge should succeed" + merge_line=$(glab_merge_line "$case_dir/glab.log") + [ "$merge_line" = "GITLAB_HOST=$MR_HOST mr merge 7 -R $MR_PROJECT_URL --sha $MR_HEAD --yes --remove-source-branch" ] \ + || fail "gitlab-extra-args: extra glab flags were not forwarded: '$merge_line'" + pass "fm-pr-merge forwards extra flags to glab mr merge after the -- separator" +} + +test_gitlab_merge_failure_propagates() { + local case_dir rc + case_dir=$(make_gitlab_case gitlab-merge-fails) + : > "$case_dir/glab-merge-fails" + + set +e + run_pr_merge "$case_dir" task-x1 "$MR_URL" \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 1 "$rc" "gitlab-merge-fails: a failing glab merge should not report success" + assert_grep "pr=$MR_URL" "$case_dir/state/task-x1.meta" \ + "gitlab-merge-fails: pr= should already be recorded even though the merge failed" + pass "fm-pr-merge propagates a real glab merge failure without silently succeeding" +} + +# Each pre-merge condition, driven one at a time, so no condition can be +# carried by another. The refusal names that condition, no merge is attempted, +# and pr= is still recorded and the poll still armed exactly as the GitHub path +# leaves them when gh-axi itself fails. +test_gitlab_each_condition_refuses_independently() { + local case_dir rc name expected spec + set -- \ + "state|state=closed|state is \"closed\", not open" \ + "detail|detail=need_rebase|detailed_merge_status is \"need_rebase\", not mergeable" \ + "conflicts|conflicts=true|has_conflicts is \"true\", not false" \ + "discussions|discussions=false|blocking_discussions_resolved is \"false\", not true" \ + "pipeline-status|pipeline_status=failed|the head pipeline status is \"failed\", not success" \ + "pipeline-sha|pipeline_sha=$MR_STALE_HEAD|the head pipeline ran at \"$MR_STALE_HEAD\", not at the current head $MR_HEAD" \ + "no-pipeline|pipeline=null|the head pipeline status is \"none\", not success" + for spec in "$@"; do + name=${spec%%|*} + expected=${spec##*|} + spec=${spec#*|} + case_dir=$(make_gitlab_case "gitlab-refuse-$name" "${spec%%|*}") + + set +e + run_pr_merge "$case_dir" task-x1 "$MR_URL" \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 1 "$rc" "gitlab-refuse-$name: fm-pr-merge should refuse" + assert_grep "error: refusing to merge $MR_URL" "$case_dir/stderr" \ + "gitlab-refuse-$name: refusal did not name the merge request" + assert_grep "$expected" "$case_dir/stderr" \ + "gitlab-refuse-$name: refusal did not name the failing condition" + [ -z "$(glab_merge_line "$case_dir/glab.log")" ] \ + || fail "gitlab-refuse-$name: a merge was attempted despite the refusal" + assert_grep "pr=$MR_URL" "$case_dir/state/task-x1.meta" \ + "gitlab-refuse-$name: a refusal should still leave the recorded PR reference" + assert_present "$case_dir/state/task-x1.check.sh" \ + "gitlab-refuse-$name: a refusal should still leave the merge poll armed" + done + pass "fm-pr-merge refuses on each GitLab pre-merge condition independently" +} + +test_gitlab_reports_every_failing_condition() { + local case_dir rc expected + case_dir=$(make_gitlab_case gitlab-refuse-all \ + state=closed detail=conflict conflicts=true discussions=false \ + pipeline_status=failed "pipeline_sha=$MR_STALE_HEAD") + + set +e + run_pr_merge "$case_dir" task-x1 "$MR_URL" \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 1 "$rc" "gitlab-refuse-all: fm-pr-merge should refuse" + for expected in \ + 'state is "closed", not open' \ + 'detailed_merge_status is "conflict", not mergeable' \ + 'has_conflicts is "true", not false' \ + 'blocking_discussions_resolved is "false", not true' \ + 'the head pipeline status is "failed", not success' \ + "the head pipeline ran at \"$MR_STALE_HEAD\", not at the current head $MR_HEAD" + do + assert_grep "$expected" "$case_dir/stderr" \ + "gitlab-refuse-all: '$expected' was not reported" + done + pass "fm-pr-merge reports every failing GitLab condition, not only the first" +} + +test_gitlab_stale_recorded_head_is_reported() { + local case_dir rc merge_line + case_dir=$(make_gitlab_case gitlab-stale-head) + # The recorded head is what a rebase leaves behind. It is read before + # fm-pr-check.sh rewrites the metadata, which drops a head it cannot resolve + # for a GitLab task, so reading it afterwards would find nothing at all. + printf 'pr_head=%s\n' "$MR_STALE_HEAD" >> "$case_dir/state/task-x1.meta" + + set +e + run_pr_merge "$case_dir" task-x1 "$MR_URL" \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 0 "$rc" "gitlab-stale-head: the live head satisfies every condition, so it should merge" + assert_grep "recorded head $MR_STALE_HEAD disagrees with the live head $MR_HEAD" \ + "$case_dir/stderr" "gitlab-stale-head: the stale recorded head was trusted silently" + merge_line=$(glab_merge_line "$case_dir/glab.log") + case "$merge_line" in + *"--sha $MR_HEAD"*) : ;; + *) fail "gitlab-stale-head: the merge was not bound to the live head: '$merge_line'" ;; + esac + assert_no_grep "pr_head=$MR_STALE_HEAD" "$case_dir/state/task-x1.meta" \ + "gitlab-stale-head: the recording step no longer drops an unresolvable GitLab head" + pass "fm-pr-merge reports a stale recorded head and verifies the live one" +} + +test_gitlab_unreadable_state_refuses() { + local case_dir rc name + for name in view-fails not-an-object split-value; do + case_dir=$(make_gitlab_case "gitlab-unreadable-$name") + case "$name" in + view-fails) : > "$case_dir/glab-view-fails" ;; + not-an-object) printf '[]\n' > "$case_dir/mr.json" ;; + # A value carrying a newline splits into a line no field name matches, so + # it must refuse rather than be truncated into a value a check accepts. + split-value) write_mr_json "$case_dir/mr.json" 'state=opened\nnot-a-field' ;; + esac + + set +e + run_pr_merge "$case_dir" task-x1 "$MR_URL" \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 1 "$rc" "gitlab-unreadable-$name: fm-pr-merge should refuse" + assert_grep 'could not read the GitLab merge request state before merging' \ + "$case_dir/stderr" "gitlab-unreadable-$name: refusal did not name the unreadable state" + [ -z "$(glab_merge_line "$case_dir/glab.log")" ] \ + || fail "gitlab-unreadable-$name: a merge was attempted on an unreadable state" + done + pass "fm-pr-merge refuses an unreadable GitLab merge request state rather than merging blind" +} + +test_gitlab_invalid_head_refuses() { + local case_dir rc + case_dir=$(make_gitlab_case gitlab-invalid-head head=not-a-sha) + + set +e + run_pr_merge "$case_dir" task-x1 "$MR_URL" \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 1 "$rc" "gitlab-invalid-head: fm-pr-merge should refuse" + assert_grep 'could not read the GitLab merge request head commit before merging' \ + "$case_dir/stderr" "gitlab-invalid-head: refusal did not name the unreadable head" + [ -z "$(glab_merge_line "$case_dir/glab.log")" ] \ + || fail "gitlab-invalid-head: a merge was bound to a head that is not a commit" + pass "fm-pr-merge refuses a GitLab head commit it cannot validate" +} + +test_gitlab_missing_tool_refuses_before_recording() { + local case_dir rc tool other + for tool in glab jq; do + if [ "$tool" = glab ]; then other=jq; else other=glab; fi + case_dir=$(make_gitlab_case "gitlab-no-$tool") + mirror_path_without "$case_dir/no$tool" "$tool" "$case_dir/fakebin" + # One tool absent, the other still answered by this case's own mock, so the + # refusal names exactly one tool on a host that ships neither. + PATH="$case_dir/no$tool" command -v "$other" >/dev/null 2>&1 \ + || fail "gitlab-no-$tool: the $tool-free search path lost the $other mock as well" + + set +e + FM_ROOT_OVERRIDE="$ROOT" \ + FM_STATE_OVERRIDE="$case_dir/state" \ + FM_TEST_GH_AXI_LOG="$case_dir/gh-axi.log" \ + FM_TEST_GLAB_LOG="$case_dir/glab.log" \ + FM_TEST_GLAB_JSON="$case_dir/mr.json" \ + PATH="$case_dir/no$tool" \ + "$PR_MERGE" task-x1 "$MR_URL" > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 1 "$rc" "gitlab-no-$tool: fm-pr-merge should refuse" + assert_grep "error: merging a GitLab merge request requires $tool on PATH" \ + "$case_dir/stderr" "gitlab-no-$tool: refusal did not name the missing tool" + assert_no_grep "pr=$MR_URL" "$case_dir/state/task-x1.meta" \ + "gitlab-no-$tool: a PR reference was recorded despite the missing tool" + assert_absent "$case_dir/state/task-x1.check.sh" \ + "gitlab-no-$tool: a merge poll was armed despite the missing tool" + done + pass "fm-pr-merge refuses before recording anything when glab or jq is absent" +} + +test_gitlab_head_override_args_refuse_before_recording() { + local case_dir rc + case_dir=$(make_gitlab_case gitlab-head-override) + + set +e + run_pr_merge "$case_dir" task-x1 "$MR_URL" -- --sha "$MR_STALE_HEAD" \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 1 "$rc" "gitlab-head-override: fm-pr-merge should refuse a caller head override" + assert_grep 'extra merge arguments must not override the head commit' "$case_dir/stderr" \ + "gitlab-head-override: refusal did not explain the head override" + assert_no_grep "pr=$MR_URL" "$case_dir/state/task-x1.meta" \ + "gitlab-head-override: the URL was recorded before rejecting the head override" + assert_absent "$case_dir/state/task-x1.check.sh" \ + "gitlab-head-override: a head override armed a merge poll" + [ ! -s "$case_dir/glab.log" ] || fail "gitlab-head-override: glab was invoked despite the head override" + pass "fm-pr-merge refuses a GitLab head override before recording state" +} + +test_github_still_forwards_sha_arg() { + local case_dir + case_dir=$(make_case github-sha-arg) + mkdir -p "$case_dir/wt" + add_gh_mocks "$case_dir" dddddddddddddddddddddddddddddddddddddddd + : > "$case_dir/gh-axi.log" + + # --sha is rejected only where the head is firstmate's to determine. GitHub's + # extra args are the caller's business exactly as they were. + run_pr_merge "$case_dir" task-x1 https://github.com/example/repo/pull/44 -- --sha abc123 \ + > "$case_dir/stdout" 2> "$case_dir/stderr" || fail "github-sha-arg: fm-pr-merge failed" + + grep -qxF 'pr merge 44 --repo example/repo --squash --sha abc123' "$case_dir/gh-axi.log" \ + || fail "github-sha-arg: the GitHub path stopped forwarding a caller --sha" + pass "fm-pr-merge leaves GitHub extra-arg handling unchanged, including --sha" +} + test_records_pr_and_head_before_merging test_merge_failure_propagates_after_recording test_extra_merge_args_forwarded @@ -308,6 +817,20 @@ test_missing_meta_refuses_before_merge test_malformed_url_refuses_before_merge test_rejects_unsafe_url_segments_before_recording test_repo_override_args_refuse_before_recording +test_bundled_repo_override_args_refuse_before_recording test_explicit_merge_method_not_overridden test_method_equals_merge_method_not_overridden test_parses_pr_url_for_gh_axi +test_github_still_forwards_sha_arg +test_gitlab_url_resolves_and_merges +test_gitlab_host_comes_from_the_url +test_gitlab_imposes_no_merge_method +test_gitlab_extra_args_forwarded +test_gitlab_merge_failure_propagates +test_gitlab_each_condition_refuses_independently +test_gitlab_reports_every_failing_condition +test_gitlab_stale_recorded_head_is_reported +test_gitlab_unreadable_state_refuses +test_gitlab_invalid_head_refuses +test_gitlab_missing_tool_refuses_before_recording +test_gitlab_head_override_args_refuse_before_recording diff --git a/tests/fm-procevent.test.sh b/tests/fm-procevent.test.sh index 878f71ac81b..3b93b1c1c37 100755 --- a/tests/fm-procevent.test.sh +++ b/tests/fm-procevent.test.sh @@ -595,6 +595,196 @@ out=$(PATH="$LAVISH_BIN:$PATH" FM_HOME="$HLT" "$ROOT/bin/fm-procevent-lavish.sh" assert_contains "$out" "retired: $lavish_id" "explicit adapter retirement stays supported after automatic retirement" pass "one Send & End yields exactly one captured result, automatic retirement, and no recurring poll" +# --- end-user-aligned regression: a transient poll interruption is not news --- +# The dogfood defect: a live board listener can answer with exactly +# error: Lavish Editor poll response was interrupted +# code: SERVER_ERROR +# while the board's marks remain available. Firstmate registered raw poll output, +# so the generic runner captured that transient response and woke the whole fleet +# over what is really an internal retry. Every scenario below runs through the +# adapter's own arm command and the real runner, so registration, capture, and +# publication are exercised for real. +LAVISH_SCRIPTED_BIN=$(fm_fakebin "$TMP_ROOT/lavish-scripted-stub") +cat > "$LAVISH_SCRIPTED_BIN/lavish-axi" <<'SH' +#!/usr/bin/env bash +# Stand-in for `lavish-axi poll <file>`, scripted per scenario: LAVISH_SCRIPT +# names the response for each successive poll, one word per poll, and its last +# word repeats forever. `interrupt` is the exact transient response the server +# returns while the board's marks stay available. +n=$(cat "$LAVISH_COUNT" 2>/dev/null || echo 0) +n=$((n + 1)) +printf '%s\n' "$n" > "$LAVISH_COUNT" +read -r -a plan <<< "$LAVISH_SCRIPT" +i=$((n - 1)) +[ "$i" -ge "${#plan[@]}" ] && i=$((${#plan[@]} - 1)) +case "${plan[$i]}" in + interrupt) + printf 'error: Lavish Editor poll response was interrupted\ncode: SERVER_ERROR\n'; exit 1 ;; + near-interrupt) + printf 'error: Lavish Editor poll response was interrupted \ncode: SERVER_ERROR\n'; exit 1 ;; + other-server-error) + printf 'error: Lavish Editor session store is unavailable\ncode: SERVER_ERROR\n'; exit 1 ;; + feedback) + printf 'session:\n file: /board.html\n status: feedback\n session_ended: true\n ended_by: user\nfeedback[1]{text}:\n ship it\n' ;; + stream) + printf 'x%.0s' {1..4096} + printf 'ready\n' > "$LAVISH_STREAM_READY" + while [ ! -e "$LAVISH_STREAM_RELEASE" ]; do sleep 0.05; done + printf '\n' ;; +esac +SH +chmod +x "$LAVISH_SCRIPTED_BIN/lavish-axi" +export LAVISH_COUNT LAVISH_SCRIPT +# A bounded test override keeps the retry policy's real bound under test without +# making the suite wait out the production delay. +export FM_LAVISH_POLL_RETRY_DELAY=0 + +# Two interruptions, then the captain's real feedback: the retries are silent and +# only the feedback becomes a captured result and a check wake. +HRETRY="$TMP_ROOT/hretry"; new_home "$HRETRY" +RETRY_ART="$TMP_ROOT/retry-board.html" +printf '<h1>retry</h1>\n' > "$RETRY_ART" +retry_id=$("$ROOT/bin/fm-procevent-lavish.sh" source-id "$RETRY_ART") +PE_TRACKED+=("$HRETRY|$retry_id") +LAVISH_COUNT="$TMP_ROOT/retry-count"; LAVISH_SCRIPT="interrupt interrupt feedback" +PATH="$LAVISH_SCRIPTED_BIN:$PATH" FM_HOME="$HRETRY" \ + "$ROOT/bin/fm-procevent-lavish.sh" arm "$RETRY_ART" >/dev/null +PATH="$LAVISH_SCRIPTED_BIN:$PATH" pe "$HRETRY" reconcile >/dev/null +wait_for "$HRETRY/state/.wake-queue" || fail "feedback after interrupted polls produced no wake" +[ "$(cat "$LAVISH_COUNT")" = 3 ] \ + || fail "the interrupted listener was polled $(cat "$LAVISH_COUNT") times, not the two quiet retries plus the delivering poll" +[ "$(count_results "$HRETRY" "$retry_id")" = 1 ] \ + || fail "a retried interruption produced $(count_results "$HRETRY" "$retry_id") captured results instead of one" +[ "$(wake_payloads "$HRETRY" | sort -u | grep -c .)" = 1 ] \ + || fail "a retried interruption woke the fleet: $(wake_payloads "$HRETRY" | sort -u)" +assert_contains "$(wake_payloads "$HRETRY")" "procevent lavish $retry_id 1" \ + "feedback arriving after quiet retries is captured and announced" +assert_grep 'ship it' "$(first_result "$HRETRY" "$retry_id")" \ + "the announced result is the captain's feedback, not the interruption" +pass "a transient Lavish poll interruption is retried quietly and never announced" + +# Exhaustion is news: after the bounded retries the same exact response is +# captured and announced normally rather than being swallowed forever. +HEXH="$TMP_ROOT/hexh"; new_home "$HEXH" +EXH_ART="$TMP_ROOT/exhaust-board.html" +printf '<h1>exhaust</h1>\n' > "$EXH_ART" +exh_id=$("$ROOT/bin/fm-procevent-lavish.sh" source-id "$EXH_ART") +PE_TRACKED+=("$HEXH|$exh_id") +LAVISH_COUNT="$TMP_ROOT/exhaust-count"; LAVISH_SCRIPT="interrupt" +PATH="$LAVISH_SCRIPTED_BIN:$PATH" FM_HOME="$HEXH" \ + "$ROOT/bin/fm-procevent-lavish.sh" arm "$EXH_ART" >/dev/null +PATH="$LAVISH_SCRIPTED_BIN:$PATH" pe "$HEXH" start "$exh_id" >/dev/null +[ "$(cat "$LAVISH_COUNT")" = 13 ] \ + || fail "the retry bound polled $(cat "$LAVISH_COUNT") times, not the first poll plus 12 bounded retries" +[ "$(count_results "$HEXH" "$exh_id")" = 1 ] \ + || fail "exhaustion produced $(count_results "$HEXH" "$exh_id") captured results instead of one" +assert_contains "$(wake_payloads "$HEXH")" "procevent lavish $exh_id 1" \ + "the interruption that survives the bound is announced normally" +assert_grep 'poll response was interrupted' "$(first_result "$HEXH" "$exh_id")" \ + "the announced result is the exact interruption the server returned" +PATH="$LAVISH_SCRIPTED_BIN:$PATH" FM_HOME="$HEXH" \ + "$ROOT/bin/fm-procevent-lavish.sh" retire "$EXH_ART" >/dev/null +pass "an interruption that outlives the bounded retries is captured and announced" + +# A different SERVER_ERROR is a genuine error, never a retry: no fail-open drift +# from the one exact transient response this adapter owns. +HOTHER="$TMP_ROOT/hother"; new_home "$HOTHER" +OTHER_ART="$TMP_ROOT/other-board.html" +printf '<h1>other</h1>\n' > "$OTHER_ART" +other_id=$("$ROOT/bin/fm-procevent-lavish.sh" source-id "$OTHER_ART") +PE_TRACKED+=("$HOTHER|$other_id") +LAVISH_COUNT="$TMP_ROOT/other-count"; LAVISH_SCRIPT="other-server-error" +PATH="$LAVISH_SCRIPTED_BIN:$PATH" FM_HOME="$HOTHER" \ + "$ROOT/bin/fm-procevent-lavish.sh" arm "$OTHER_ART" >/dev/null +PATH="$LAVISH_SCRIPTED_BIN:$PATH" pe "$HOTHER" start "$other_id" >/dev/null +[ "$(cat "$LAVISH_COUNT")" = 1 ] \ + || fail "an unrelated SERVER_ERROR was retried $(cat "$LAVISH_COUNT") times instead of surfacing at once" +assert_contains "$(wake_payloads "$HOTHER")" "procevent lavish $other_id 1" \ + "an unrelated SERVER_ERROR is captured and announced immediately" +PATH="$LAVISH_SCRIPTED_BIN:$PATH" FM_HOME="$HOTHER" \ + "$ROOT/bin/fm-procevent-lavish.sh" retire "$OTHER_ART" >/dev/null +pass "only the exact interruption is retried; an unrelated SERVER_ERROR still surfaces" +unset FM_LAVISH_POLL_RETRY_DELAY + +# A whitespace variant is not the exact transient response and must surface on +# the first poll instead of drifting into the quiet retry policy. +HNEAR="$TMP_ROOT/hnear"; new_home "$HNEAR" +NEAR_ART="$TMP_ROOT/near-board.html" +printf '<h1>near</h1>\n' > "$NEAR_ART" +near_id=$("$ROOT/bin/fm-procevent-lavish.sh" source-id "$NEAR_ART") +PE_TRACKED+=("$HNEAR|$near_id") +LAVISH_COUNT="$TMP_ROOT/near-count"; LAVISH_SCRIPT="near-interrupt feedback" +PATH="$LAVISH_SCRIPTED_BIN:$PATH" FM_HOME="$HNEAR" FM_LAVISH_POLL_RETRY_DELAY=0 \ + "$ROOT/bin/fm-procevent-lavish.sh" arm "$NEAR_ART" >/dev/null +PATH="$LAVISH_SCRIPTED_BIN:$PATH" FM_HOME="$HNEAR" pe "$HNEAR" start "$near_id" >/dev/null +[ "$(cat "$LAVISH_COUNT")" = 1 ] \ + || fail "a near-match interruption was retried instead of surfacing on its first poll" +assert_contains "$(wake_payloads "$HNEAR")" "procevent lavish $near_id 1" \ + "a whitespace variant of the interruption is captured and announced immediately" +PATH="$LAVISH_SCRIPTED_BIN:$PATH" FM_HOME="$HNEAR" \ + "$ROOT/bin/fm-procevent-lavish.sh" retire "$NEAR_ART" >/dev/null +pass "only the literal two-line interruption enters the quiet retry policy" + +# The public arm boundary refuses invalid retry intervals before it publishes a +# source registration, rather than arming a listener that can only fail later. +HINVALID="$TMP_ROOT/hinvalid"; new_home "$HINVALID" +INVALID_ART="$TMP_ROOT/invalid-delay-board.html" +printf '<h1>invalid delay</h1>\n' > "$INVALID_ART" +invalid_id=$("$ROOT/bin/fm-procevent-lavish.sh" source-id "$INVALID_ART") +for invalid_delay in 61 invalid; do + invalid_status=0 + invalid_out=$(PATH="$LAVISH_SCRIPTED_BIN:$PATH" FM_HOME="$HINVALID" \ + FM_LAVISH_POLL_RETRY_DELAY="$invalid_delay" \ + "$ROOT/bin/fm-procevent-lavish.sh" arm "$INVALID_ART" 2>&1) || invalid_status=$? + [ "$invalid_status" -ne 0 ] \ + || fail "arm accepted invalid retry delay: $invalid_delay" + assert_contains "$invalid_out" "must be whole seconds from 0 to 60" \ + "arm explains the rejected retry delay" + assert_absent "$HINVALID/state/procevent/$invalid_id.source" \ + "arm publishes no source registration for an invalid retry delay" +done +pass "arm rejects malformed and out-of-range retry delays before registration" + +# Shell-safe cleanup must preserve a valid TMPDIR containing an apostrophe. +QUOTED_TMPDIR="$TMP_ROOT/poll's-stage" +mkdir -p "$QUOTED_TMPDIR" +LAVISH_COUNT="$TMP_ROOT/quoted-count"; LAVISH_SCRIPT="feedback" +PATH="$LAVISH_SCRIPTED_BIN:$PATH" TMPDIR="$QUOTED_TMPDIR" \ + "$ROOT/bin/fm-procevent-lavish.sh" poll "$NEAR_ART" >/dev/null +quoted_staged=("$QUOTED_TMPDIR"/fm-lavish-poll.*) +[ ! -e "${quoted_staged[0]}" ] \ + || fail "poll left its staged response behind in an apostrophe-containing TMPDIR" +pass "poll cleanup safely handles an apostrophe-containing TMPDIR" + +HSTREAM="$TMP_ROOT/hstream"; new_home "$HSTREAM" +STREAM_ART="$TMP_ROOT/stream-board.html" +STREAM_TMPDIR="$TMP_ROOT/stream-stage" +LAVISH_STREAM_READY="$TMP_ROOT/stream-ready" +LAVISH_STREAM_RELEASE="$TMP_ROOT/stream-release" +mkdir -p "$STREAM_TMPDIR" +printf '<h1>stream</h1>\n' > "$STREAM_ART" +stream_id=$("$ROOT/bin/fm-procevent-lavish.sh" source-id "$STREAM_ART") +PE_TRACKED+=("$HSTREAM|$stream_id") +LAVISH_COUNT="$TMP_ROOT/stream-count"; LAVISH_SCRIPT="stream" +PATH="$LAVISH_SCRIPTED_BIN:$PATH" FM_HOME="$HSTREAM" \ + "$ROOT/bin/fm-procevent-lavish.sh" arm "$STREAM_ART" >/dev/null +PATH="$LAVISH_SCRIPTED_BIN:$PATH" TMPDIR="$STREAM_TMPDIR" \ + LAVISH_STREAM_READY="$LAVISH_STREAM_READY" LAVISH_STREAM_RELEASE="$LAVISH_STREAM_RELEASE" \ + FM_PROCEVENT_MAX_OUTPUT_BYTES=100 pe "$HSTREAM" reconcile >/dev/null +wait_for "$LAVISH_STREAM_READY" || fail "streaming poll did not start" +stream_staged=("$STREAM_TMPDIR"/fm-lavish-poll.*) +[ -e "${stream_staged[0]}" ] || fail "streaming poll created no classifier staging file" +[ "$(wc -c < "${stream_staged[0]}" | tr -d ' ')" -le 100 ] \ + || fail "streaming poll exceeded its bounded classifier staging" +: > "$LAVISH_STREAM_RELEASE" +wait_for "$HSTREAM/state/.wake-queue" || fail "streaming poll produced no wake" +stream_result=$(first_result "$HSTREAM" "$stream_id" || true) +[ "$(wc -c < "$stream_result" | tr -d ' ')" -le 100 ] \ + || fail "streaming poll bypassed the runner output bound" +PATH="$LAVISH_SCRIPTED_BIN:$PATH" FM_HOME="$HSTREAM" \ + "$ROOT/bin/fm-procevent-lavish.sh" retire "$STREAM_ART" >/dev/null +pass "Lavish classification staging stays bounded while nonmatches stream" + # --- end-user-aligned regression: the exact drain-before-handling restart cut # Reproduces the confirmed defect through the public interface end to end: a # real blocking source completes, its result is captured and published, the diff --git a/tests/fm-public-followup.test.sh b/tests/fm-public-followup.test.sh index fe15e239e2e..69a054abee8 100755 --- a/tests/fm-public-followup.test.sh +++ b/tests/fm-public-followup.test.sh @@ -20,6 +20,7 @@ PF="$ROOT/bin/fm-public-followup.sh" EMIT="$ROOT/bin/fm-public-followup-emit.sh" POLL="$ROOT/bin/fm-x-poll.sh" TEARDOWN="$ROOT/bin/fm-teardown.sh" +PROMOTE="$ROOT/bin/fm-promote.sh" SESSION_START="$ROOT/bin/fm-session-start.sh" TMP_ROOT=$(fm_test_tmproot fm-public-followup) @@ -150,6 +151,34 @@ seed_commitment() { || fail "could not register the public commitment" } +# The pi-rearm shape: a report-ready promised-final bound to a secondmate. +seed_repro_commitment() { # <home> <obligation> <request> <work-home> <work-id> + local home=$1 obligation=$2 request=$3 work_home=$4 work_id=$5 + jq -n --arg r "$request" \ + '{request_id:$r, platform:"discord", + context_binding:{version:"ctx1", value:("ctx1_" + $r)}, + public_safe_summary:"reproduce a Pi recovery notification loop", + received_at:"2026-08-21T01:12:00Z", + followup_expires_at:"2026-08-28T01:12:00Z", + reservation_expires_at:"2026-08-28T01:12:00Z"}' > "$home/request.json" + jq -n '{type:"report-ready", project:"firstmate", + required_deliverables:["report_path"], completion_policy:"all-required"}' \ + > "$home/expected.json" + jq -n --arg h "$work_home" --arg w "$work_id" \ + '{relation_id:"rel-code", work_ref:{home_id:$h, task_id:$w}, + role:"fulfills", required:true, generation:1}' > "$home/relation.json" + tasks_in "$home" public-followup add "$obligation" --request-context-file "$home/request.json" \ + --purpose promised-final --expected-final-file "$home/expected.json" \ + --expires-at 2026-10-01T00:00:00Z >/dev/null || fail "add failed" + tasks_in "$home" public-followup bind-work "$obligation" --relation-file "$home/relation.json" >/dev/null \ + || fail "bind-work failed" + FM_HOME="$home" bash -c \ + ". '$ROOT/bin/fm-x-lib.sh'; fmx_context_registry_set '$home/state' '$request' discord 2000" \ + || fail "context retain failed" + run_pf "$home" register "$obligation" --relation rel-code --work-home "$work_home" \ + --work-id "$work_id" --generation 1 >/dev/null || fail "register failed" +} + emit_terminal() { # <child-run-dir> <owning-home> <obligation> <work-home> <work-id> [pr-url] [outcome] local owning=$2 obligation=$3 work_home=$4 work_id=$5 local pr=${6:-https://github.com/example/repo/pull/7} outcome=${7:-pr-merged} @@ -240,9 +269,9 @@ test_restart_e2e_delivers_exactly_once() { home=$(make_home restart-e2e) child=$(make_home restart-child relay-off) log="$home/curl.log"; : > "$log" - seed_commitment "$home" pf-restart req-restart discord secondmate:fmdev work-code-q1 printf '%s\n' fmdev > "$child/.fm-secondmate-home" fm_write_meta "$home/state/fmdev.meta" "kind=secondmate" "home=$child" + seed_commitment "$home" pf-restart req-restart discord secondmate:fmdev work-code-q1 fm_write_meta "$child/state/work-code-q1.meta" \ "x_request=req-restart" "x_request_ts=1700000000" "x_followups=1" @@ -533,9 +562,9 @@ test_outward_delivery_stays_with_the_owning_home() { owner=$(make_home owner) child=$(make_home child relay-off) log="$owner/curl.log"; : > "$log" - seed_commitment "$owner" pf-own req-own discord secondmate:child work-child printf '%s\n' child > "$child/.fm-secondmate-home" fm_write_meta "$owner/state/child.meta" "kind=secondmate" "home=$child" + seed_commitment "$owner" pf-own req-own discord secondmate:child work-child fm_write_meta "$child/state/work-child.meta" \ "x_request=req-own" "x_request_ts=1700000000" "x_followups=1" @@ -1165,6 +1194,14 @@ SH [ -z "$out" ] || fail "'$cmd' must print nothing in a relay-disabled home, got: $out" done + # New fields and the open-loop gate must not create work in a disabled home. + # shellcheck disable=SC1091 + . "$ROOT/bin/fm-public-followup-lib.sh" + fm_pf_has_open_loops "$home/state" \ + && fail "a relay-disabled home must not grow an open-loop registry" + fm_pf_has_delivered_open_loops "$home/state" \ + && fail "a relay-disabled home must not grow a delivered open-loop registry" + rc=0 PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ FM_STATE_OVERRIDE="$home/state" FAKE_TASKS_AXI_LOG="$tasks_log" "$PF" active || rc=$? @@ -1290,7 +1327,7 @@ test_session_start_surfaces_only_when_owed() { out=$(PATH="$on/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$on" \ FM_STATE_OVERRIDE="$on/state" FM_DATA_OVERRIDE="$on/data" \ FM_CONFIG_OVERRIDE="$on/config" "$SESSION_START" 2>&1) - assert_contains "$out" "Public commitments awaiting delivery" \ + assert_contains "$out" "Public commitments" \ "an unresolved commitment must be surfaced at startup" assert_contains "$out" "unresolved pf-start state=pending-work platform=discord" \ "the startup summary must be typed and actionable" @@ -1323,6 +1360,847 @@ test_typed_records_exclude_raw_public_material() { pass "typed public-followup records carry only public-safe summaries and deliverables" } +# --- 10. delivery does not close the public loop -------------------------------- + +test_dropped_baton_now_surfaces_open_loop() { + local parent child log + parent=$(make_home baton-parent) + child="$TMP_ROOT/baton-child" + FM_SECONDMATE_CHARTER='Baton repro charter.' FM_HOME="$parent" \ + "$ROOT/bin/fm-home-seed.sh" mate "$child" --no-projects >/dev/null || fail "seed failed" + child=$(cd "$child" && pwd -P) + make_fake_curl "$child" >/dev/null + fm_fake_exit0 "$child/fakebin" tmux treehouse no-mistakes gh gh-axi + log="$TMP_ROOT/curl.log"; : > "$log" + + seed_repro_commitment "$parent" public-final-pi-rearm-repro req-pirearm \ + secondmate:mate pi-rearm-loop-repro-s1 + fm_write_meta "$parent/state/mate.meta" "kind=secondmate" "home=$child" + + "$EMIT" --home "$parent" --obligation public-final-pi-rearm-repro --relation rel-code \ + --source-home secondmate:mate --work-id pi-rearm-loop-repro-s1 --generation 1 \ + --outcome report-ready --deliverable report_path=data/pi-rearm-loop-repro-s1/report.md \ + --outcome-text 'Reproduced the loop. A bounded fix is scoped and waiting on you, captain.' >/dev/null \ + || fail "emit failed" + FAKE_CURL_LOG="$log" run_pf "$parent" consume | grep -q '^ready ' || fail "consume not ready" + FAKE_CURL_LOG="$log" run_pf "$parent" deliver public-final-pi-rearm-repro >/dev/null || fail "deliver failed" + [ "$(followup_posts "$log")" = 1 ] || fail "expected exactly one closing post" + assert_present "$parent/state/public-followup/registry/public-final-pi-rearm-repro" \ + "delivery must retain the registration" + + fm_write_meta "$child/state/pi-rearm-loop-fix-r1.meta" \ + "window=firstmate:fm-pi-rearm-loop-fix-r1" "endpoint_task_id=pi-rearm-loop-fix-r1" \ + "worktree=$child" "project=$child" "kind=ship" "mode=local-only" + + PATH="$parent/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$parent" \ + FM_STATE_OVERRIDE="$parent/state" "$PF" guard-work secondmate:mate pi-rearm-loop-fix-r1 \ + || fail "guard-work unexpectedly blocked the unregistered follow-on" + + run_pf "$parent" pending > "$TMP_ROOT/pending.out" + grep -q '^open-loop public-final-pi-rearm-repro ' "$TMP_ROOT/pending.out" \ + || fail "pending must print the open-loop line after delivery" + grep -q 'request=req-pirearm' "$TMP_ROOT/pending.out" \ + || fail "the open-loop line must name the original request" + + PATH="$child/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$child" \ + FM_STATE_OVERRIDE="$child/state" FM_DATA_OVERRIDE="$child/data" FAKE_CURL_LOG="$log" \ + "$TEARDOWN" pi-rearm-loop-fix-r1 > "$TMP_ROOT/td.out" 2>&1 || true + case "$(cat "$TMP_ROOT/td.out")" in + *"still owes a public reply"*) fail "teardown unexpectedly guarded the unregistered follow-on" ;; + esac + [ "$(followup_posts "$log")" = 1 ] || fail "unexpected extra post" + pass "dropped-baton regression: delivery retains the loop and pending prints open-loop" +} + +test_control_registered_followon_is_guarded() { + local parent child + parent=$(make_home baton-control-parent) + child="$TMP_ROOT/baton-control-child" + FM_SECONDMATE_CHARTER='Baton control charter.' FM_HOME="$parent" \ + "$ROOT/bin/fm-home-seed.sh" mate "$child" --no-projects >/dev/null || fail "seed failed" + child=$(cd "$child" && pwd -P) + make_fake_curl "$child" >/dev/null + fm_fake_exit0 "$child/fakebin" tmux treehouse no-mistakes gh gh-axi + seed_repro_commitment "$parent" public-final-pi-rearm-ship req-pirearm2 \ + secondmate:mate pi-rearm-loop-fix-r1 + fm_write_meta "$parent/state/mate.meta" "kind=secondmate" "home=$child" + fm_write_meta "$child/state/pi-rearm-loop-fix-r1.meta" \ + "window=firstmate:fm-pi-rearm-loop-fix-r1" "endpoint_task_id=pi-rearm-loop-fix-r1" \ + "worktree=$child" "project=$child" "kind=ship" "mode=local-only" + PATH="$child/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$child" \ + FM_STATE_OVERRIDE="$child/state" FM_DATA_OVERRIDE="$child/data" \ + expect_failure "registered follow-on must be guarded" "$TEARDOWN" pi-rearm-loop-fix-r1 + assert_contains "$EXPECT_OUT" "still owes a public reply" "the guard fires only on presence" + pass "CONTROL: the identical teardown REFUSES the moment a commitment is registered" +} + +test_rechain_delivers_second_post_on_same_thread() { + local parent log out posts command command_log + parent=$(make_home rechain-parent) + log="$parent/curl.log"; : > "$log" + seed_repro_commitment "$parent" public-final-a req-rechain main scout-a + "$EMIT" --home "$parent" --obligation public-final-a --relation rel-code \ + --source-home main --work-id scout-a --generation 1 \ + --outcome report-ready --deliverable report_path=data/scout-a/report.md \ + --outcome-text 'Reproduced. A fix is waiting.' >/dev/null || fail "emit failed" + FAKE_CURL_LOG="$log" run_pf "$parent" consume >/dev/null || fail "consume failed" + FAKE_CURL_LOG="$log" run_pf "$parent" deliver public-final-a >/dev/null || fail "deliver failed" + [ "$(followup_posts "$log")" = 1 ] || fail "expected the investigation post" + + out=$(FAKE_CURL_LOG="$log" run_pf "$parent" rechain public-final-b --from public-final-a \ + --work-home main --work-id ship-b --expected pr-merged) \ + || fail "rechain failed: $out" + assert_contains "$out" "retired public-final-a reason=handed on to public-final-b" \ + "rechain must retire the source loop" + assert_contains "$out" "--deliverable pr_url=<value>" \ + "rechain brief must name the actual required deliverable key" + command_log="$parent/brief-command.args" + cat > "$parent/fakebin/record-emit" <<'SH' +#!/usr/bin/env bash +printf '%s\n' "$@" > "$RECORD_ARGS" +SH + chmod +x "$parent/fakebin/record-emit" + command=$(printf '%s\n' "$out" | awk ' + index($0, "/bin/fm-public-followup-emit.sh") { capture=1 } + capture { if ($0 == "") exit; print } + ') + assert_contains "$command" "--outcome-text" \ + "the exact rechain command must remain continuous through outcome text" + command=${command/"$ROOT/bin/fm-public-followup-emit.sh"/"$parent/fakebin/record-emit"} + command=${command//<value>/https://github.com/example/repo/pull/99} + RECORD_ARGS="$command_log" bash -c "$command" \ + || fail "the exact rechain command must execute after filling its deliverable value" + assert_grep '--deliverable' "$command_log" \ + "the executable rechain command must pass its deliverable option" + assert_grep '--outcome-text' "$command_log" \ + "the executable rechain command must pass its outcome text option" + assert_absent "$parent/state/public-followup/registry/public-final-a" \ + "the source registration must be gone after rechain" + assert_present "$parent/state/public-followup/registry/public-final-b" \ + "the follow-on registration must exist" + + "$EMIT" --home "$parent" --obligation public-final-b --relation rel-1 \ + --source-home main --work-id ship-b --generation 1 \ + --outcome pr-merged --deliverable pr_url=https://github.com/example/repo/pull/99 \ + --outcome-text 'Shipped: the Pi recovery loop is fixed.' >/dev/null || fail "follow-on emit failed" + FAKE_CURL_LOG="$log" run_pf "$parent" consume | grep -q '^ready public-final-b ' \ + || fail "follow-on consume not ready" + FAKE_CURL_LOG="$log" run_pf "$parent" deliver public-final-b >/dev/null || fail "follow-on deliver failed" + posts=$(followup_posts "$log") + [ "$posts" = 2 ] || fail "expected exactly two posts in the same thread, got $posts" + assert_grep 'Shipped: the Pi recovery loop is fixed.' "$log" \ + "the second post must carry the shipped text" + assert_grep '"request_id":"req-rechain"' "$log" \ + "both posts must target the original request" + pass "rechain posts the shipped follow-on into the same thread" +} + +test_rechain_resumes_after_partial_add() { + local home log real_tasks marker out count + home=$(make_home rechain-resume) + log="$home/curl.log"; : > "$log" + seed_repro_commitment "$home" public-final-resume-a req-resume main scout-resume + "$EMIT" --home "$home" --obligation public-final-resume-a --relation rel-code \ + --source-home main --work-id scout-resume --generation 1 \ + --outcome report-ready --deliverable report_path=data/scout-resume/report.md \ + --outcome-text 'Reproduced. A fix is waiting.' >/dev/null || fail "emit failed" + FAKE_CURL_LOG="$log" run_pf "$home" consume >/dev/null || fail "consume failed" + FAKE_CURL_LOG="$log" run_pf "$home" deliver public-final-resume-a >/dev/null \ + || fail "deliver failed" + + real_tasks=$(command -v tasks-axi) + marker="$home/rechain-bind-failed" + cat > "$home/fakebin/tasks-axi" <<'SH' +#!/usr/bin/env bash +if [ "$1" = public-followup ] && [ "$2" = bind-work ] \ + && [ "$3" = public-final-resume-b ] && [ ! -e "$RECHAIN_FAIL_MARKER" ]; then + : > "$RECHAIN_FAIL_MARKER" + exit 73 +fi +exec "$REAL_TASKS_AXI" "$@" +SH + chmod +x "$home/fakebin/tasks-axi" + + REAL_TASKS_AXI="$real_tasks" RECHAIN_FAIL_MARKER="$marker" \ + expect_failure "rechain must expose a resumable partial add" \ + run_pf "$home" rechain public-final-resume-b --from public-final-resume-a \ + --work-home main --work-id ship-resume --expected pr-merged + assert_contains "$EXPECT_OUT" "retry this same rechain command" \ + "a partial add must direct the caller to the resumable path" + count=$(tasks_in "$home" public-followup list --json \ + | jq '[.public_followups[] | select(.id == "public-final-resume-b")] | length') + [ "$count" = 1 ] || fail "the interrupted rechain must leave exactly one recoverable obligation" + assert_present "$home/state/public-followup/registry/public-final-resume-a" \ + "a partial rechain must retain the source loop" + assert_absent "$home/state/public-followup/registry/public-final-resume-b" \ + "a failed bind must not publish a registration" + + out=$(REAL_TASKS_AXI="$real_tasks" RECHAIN_FAIL_MARKER="$marker" \ + run_pf "$home" rechain public-final-resume-b --from public-final-resume-a \ + --work-home main --work-id ship-resume --expected pr-merged) \ + || fail "retrying the same rechain command must resume: $out" + assert_contains "$out" "retired public-final-resume-a" \ + "the resumed rechain must retire its source" + count=$(tasks_in "$home" public-followup list --json \ + | jq '[.public_followups[] | select(.id == "public-final-resume-b")] | length') + [ "$count" = 1 ] || fail "the resumed rechain must not duplicate the obligation" + assert_present "$home/state/public-followup/registry/public-final-resume-b" \ + "the resumed rechain must publish the destination registration" + assert_absent "$home/state/public-followup/registry/public-final-resume-a" \ + "the resumed rechain must close the source registration" + pass "rechain resumes the same obligation after an interrupted bind" +} + +test_rechain_claims_delivered_source_once() { + local home log pid_b pid_c rc_b=0 rc_c=0 registry_count + home=$(make_home rechain-claim) + log="$home/curl.log"; : > "$log" + seed_repro_commitment "$home" public-final-claim-a req-claim main scout-claim + "$EMIT" --home "$home" --obligation public-final-claim-a --relation rel-code \ + --source-home main --work-id scout-claim --generation 1 \ + --outcome report-ready --deliverable report_path=data/scout-claim/report.md \ + --outcome-text 'Reproduced. One follow-on may claim this thread.' >/dev/null || fail "emit failed" + FAKE_CURL_LOG="$log" run_pf "$home" consume >/dev/null || fail "consume failed" + FAKE_CURL_LOG="$log" run_pf "$home" deliver public-final-claim-a >/dev/null || fail "deliver failed" + + FMX_NOW_OVERRIDE=1787539200 run_pf "$home" rechain public-final-claim-b \ + --from public-final-claim-a --work-home main --work-id ship-claim-b \ + --expected pr-merged > "$home/rechain-b.out" 2>&1 & + pid_b=$! + FMX_NOW_OVERRIDE=1787539200 run_pf "$home" rechain public-final-claim-c \ + --from public-final-claim-a --work-home main --work-id ship-claim-c \ + --expected pr-merged > "$home/rechain-c.out" 2>&1 & + pid_c=$! + wait "$pid_b" || rc_b=$? + wait "$pid_c" || rc_c=$? + + if [ "$rc_b" -eq 0 ]; then + [ "$rc_c" -ne 0 ] || fail "two concurrent rechains must not both claim one source" + else + [ "$rc_c" -eq 0 ] || fail "exactly one concurrent rechain must succeed" + fi + registry_count=$(find "$home/state/public-followup/registry" -type f \ + \( -name 'public-final-claim-b' -o -name 'public-final-claim-c' \) | wc -l | tr -d ' ') + [ "$registry_count" = 1 ] || fail "one source must produce exactly one registered destination" + assert_absent "$home/state/public-followup/registry/public-final-claim-a" \ + "the successfully claimed source must be retired" + pass "concurrent rechains cannot fork one delivered source" +} + +test_failed_rechain_retirement_keeps_source_claimed() { + local home log registry_file out + home=$(make_home rechain-retire-failure) + log="$home/curl.log"; : > "$log" + seed_repro_commitment "$home" public-final-retire-a req-retire-failure main scout-retire + "$EMIT" --home "$home" --obligation public-final-retire-a --relation rel-code \ + --source-home main --work-id scout-retire --generation 1 \ + --outcome report-ready --deliverable report_path=data/scout-retire/report.md \ + --outcome-text 'Reproduced. A fix is waiting.' >/dev/null || fail "emit failed" + FAKE_CURL_LOG="$log" run_pf "$home" consume >/dev/null || fail "consume failed" + FAKE_CURL_LOG="$log" run_pf "$home" deliver public-final-retire-a >/dev/null \ + || fail "deliver failed" + + registry_file="$home/state/public-followup/registry/public-final-retire-a" + cat > "$home/fakebin/rm" <<EOF +#!/usr/bin/env bash +for arg in "\$@"; do + [ "\$arg" != '$registry_file' ] || exit 1 +done +exec /bin/rm "\$@" +EOF + chmod +x "$home/fakebin/rm" + expect_failure "rechain must surface a failed source retirement" \ + run_pf "$home" rechain public-final-retire-b --from public-final-retire-a \ + --work-home main --work-id ship-retire-b --expected pr-merged + assert_contains "$EXPECT_OUT" "public loop remains open" \ + "failed retirement must report that the source remains open" + + expect_failure "a failed retirement must not leave the source claimable by another destination" \ + run_pf "$home" rechain public-final-retire-c --from public-final-retire-a \ + --work-home main --work-id ship-retire-c --expected pr-merged + assert_contains "$EXPECT_OUT" "already claimed by rechain destination 'public-final-retire-b'" \ + "a second destination must be refused after the first destination is published" + assert_absent "$home/state/public-followup/registry/public-final-retire-c" \ + "a refused competing destination must not be registered" + + /bin/rm "$home/fakebin/rm" + out=$(run_pf "$home" rechain public-final-retire-b --from public-final-retire-a \ + --work-home main --work-id ship-retire-b --expected pr-merged) \ + || fail "the claimed destination must remain resumable: $out" + assert_contains "$out" "retired public-final-retire-a" \ + "resuming the claimed destination must finish source retirement" + assert_absent "$registry_file" "successful retry must retire the source" + assert_present "$home/state/public-followup/registry/public-final-retire-b" \ + "successful retry must retain the one claimed destination" + pass "failed rechain retirement keeps the source claimed by one resumable destination" +} + +test_registration_replay_preserves_delivery_and_retirement() { + local home log registry snapshot + home=$(make_home register-replay) + log="$home/curl.log"; : > "$log" + seed_commitment "$home" pf-register-replay req-register-replay discord main work-register-replay + jq -n '{relation_id:"rel-alternate", work_ref:{home_id:"main", task_id:"work-alternate"}, + role:"fulfills", required:false, generation:1}' > "$home/alternate-relation.json" + tasks_in "$home" public-followup bind-work pf-register-replay \ + --relation-file "$home/alternate-relation.json" >/dev/null \ + || fail "could not add the alternate valid work binding" + emit_terminal "$home" "$home" pf-register-replay main work-register-replay >/dev/null || fail "emit failed" + run_pf "$home" consume >/dev/null || fail "initial consume failed" + run_pf "$home" register pf-register-replay --relation rel-alternate --work-home main \ + --work-id work-alternate --generation 1 >/dev/null \ + || fail "could not register the alternate valid work binding" + "$EMIT" --home "$home" --obligation pf-register-replay --relation rel-alternate \ + --source-home main --work-id work-alternate --generation 1 --outcome pr-merged \ + --deliverable pr_url=https://github.com/example/repo/pull/8 \ + --outcome-text 'The alternate bound work also reached its terminal outcome.' >/dev/null \ + || fail "alternate emit failed" + run_pf "$home" consume >/dev/null || fail "alternate consume failed" + run_pf "$home" register pf-register-replay --relation rel-code --work-home main \ + --work-id work-register-replay --generation 1 >/dev/null \ + || fail "could not restore the original open registration" + FAKE_CURL_LOG="$log" run_pf "$home" deliver pf-register-replay >/dev/null || fail "deliver failed" + + run_pf "$home" register pf-register-replay --relation rel-code --work-home main \ + --work-id work-register-replay --generation 1 >/dev/null || fail "registration replay failed" + registry="$home/state/public-followup/registry/pf-register-replay" + grep -q '^state=delivered$' "$registry" \ + || fail "registration replay must not downgrade a delivered loop" + + snapshot="$home/registry-before-replay" + cp "$registry" "$snapshot" + run_pf "$home" register pf-register-replay --relation rel-alternate --work-home main \ + --work-id work-alternate --generation 1 --platform x --request req-alternate >/dev/null \ + || fail "delivered registration replay against another valid binding failed" + cmp -s "$snapshot" "$registry" \ + || fail "delivered registration replay must preserve the complete retained baton" + + run_pf "$home" retire pf-register-replay --reason "finished after replay" >/dev/null \ + || fail "retire after replay failed" + expect_failure "registration replay must not reopen a retired loop" \ + run_pf "$home" register pf-register-replay --relation rel-code --work-home main \ + --work-id work-register-replay --generation 1 + assert_contains "$EXPECT_OUT" "already been retired" \ + "a retirement receipt must make registration fail closed" + assert_absent "$home/state/public-followup/registry/pf-register-replay" \ + "registration replay must not recreate a retired loop" + pass "registration replay preserves delivered and retired loop states" +} + +test_redelivery_does_not_report_retired_loop_open() { + local home log out + home=$(make_home redelivery-retired) + log="$home/curl.log"; : > "$log" + seed_commitment "$home" pf-redelivery-retired req-redelivery-retired discord main work-redelivery-retired + emit_terminal "$home" "$home" pf-redelivery-retired main work-redelivery-retired >/dev/null \ + || fail "emit failed" + run_pf "$home" consume >/dev/null || fail "consume failed" + FAKE_CURL_LOG="$log" run_pf "$home" deliver pf-redelivery-retired >/dev/null \ + || fail "initial delivery failed" + run_pf "$home" retire pf-redelivery-retired --reason "thread finished" >/dev/null \ + || fail "retire failed" + + out=$(FAKE_CURL_LOG="$log" run_pf "$home" deliver pf-redelivery-retired) \ + || fail "idempotent redelivery failed" + assert_contains "$out" "already delivered pf-redelivery-retired" \ + "redelivery must remain idempotent" + case "$out" in + *"still OPEN"*) fail "redelivery must not report a retired loop as open: $out" ;; + esac + [ "$(grep -c '^url=.*connector/followup' "$log" || true)" -eq 1 ] \ + || fail "redelivery must not post a second public reply" + pass "redelivery does not report a retired loop as open" +} + +test_retire_after_secondmate_home_removal() { + local home child log out + home=$(make_home retire-removed-secondmate) + child="$home/removed-mate" + mkdir -p "$child/state" + printf 'mate\n' > "$child/.fm-secondmate-home" + fm_write_meta "$home/state/mate.meta" "kind=secondmate" "home=$child" + log="$home/curl.log"; : > "$log" + seed_repro_commitment "$home" pf-removed-mate req-removed-mate secondmate:mate scout-removed + "$EMIT" --home "$home" --obligation pf-removed-mate --relation rel-code \ + --source-home secondmate:mate --work-id scout-removed --generation 1 \ + --outcome report-ready --deliverable report_path=data/scout-removed/report.md \ + --outcome-text 'The removed child completed its investigation.' >/dev/null || fail "emit failed" + run_pf "$home" consume >/dev/null || fail "consume failed" + FAKE_CURL_LOG="$log" run_pf "$home" deliver pf-removed-mate >/dev/null || fail "deliver failed" + + rm -rf "$child" + rm -f "$home/state/mate.meta" + out=$(run_pf "$home" retire pf-removed-mate --reason "child home was torn down") \ + || fail "retire must accept an already-absent child legacy link: $out" + assert_contains "$out" "retired pf-removed-mate" \ + "retire must close a delivered loop after its secondmate home is removed" + assert_absent "$home/state/public-followup/registry/pf-removed-mate" \ + "retire must remove the registration after child teardown" + assert_present "$home/state/public-followup/retired/pf-removed-mate" \ + "retire must still record its receipt" + pass "retire closes delivered loops after secondmate home removal" +} + +test_retire_refuses_unbound_existing_secondmate() { + local home child log + home=$(make_home retire-unbound-secondmate) + child="$home/unbound-mate" + mkdir -p "$child/state" + child=$(cd "$child" && pwd -P) + printf 'mate\n' > "$child/.fm-secondmate-home" + fm_write_meta "$home/state/mate.meta" "kind=secondmate" "home=$child" + log="$home/curl.log"; : > "$log" + seed_repro_commitment "$home" pf-unbound-mate req-unbound-mate secondmate:mate scout-unbound + "$EMIT" --home "$home" --obligation pf-unbound-mate --relation rel-code \ + --source-home secondmate:mate --work-id scout-unbound --generation 1 \ + --outcome report-ready --deliverable report_path=data/scout-unbound/report.md \ + --outcome-text 'The unbound child completed its investigation.' >/dev/null || fail "emit failed" + run_pf "$home" consume >/dev/null || fail "consume failed" + FAKE_CURL_LOG="$log" run_pf "$home" deliver pf-unbound-mate >/dev/null || fail "deliver failed" + fm_write_meta "$child/state/scout-unbound.meta" "status=working" "x_request=req-unbound-mate" + assert_grep "work_home_path=$child" \ + "$home/state/public-followup/registry/pf-unbound-mate" \ + "registration must retain the canonical secondmate path" + rm -f "$home/state/mate.meta" + + expect_failure "retire must not assume an unbound existing child link is cleared" \ + run_pf "$home" retire pf-unbound-mate --reason "child binding disappeared" + assert_contains "$EXPECT_OUT" "could not clear the legacy X link" \ + "retire must report an unverifiable secondmate legacy link" + assert_present "$home/state/public-followup/registry/pf-unbound-mate" \ + "an unverifiable child link must retain the registration" + assert_absent "$home/state/public-followup/retired/pf-unbound-mate" \ + "an unverifiable child link must not create a retirement receipt" + assert_grep 'x_request=req-unbound-mate' "$child/state/scout-unbound.meta" \ + "failed retirement must preserve the unresolved legacy link" + pass "retire fails closed for an unbound existing secondmate" +} + +test_retire_refuses_reassigned_secondmate_home() { + local home original replacement log + home=$(make_home retire-reassigned-secondmate) + original="$home/original-mate" + replacement="$home/replacement-mate" + mkdir -p "$original/state" + printf 'mate\n' > "$original/.fm-secondmate-home" + fm_write_meta "$home/state/mate.meta" "kind=secondmate" "home=$original" + log="$home/curl.log"; : > "$log" + seed_repro_commitment "$home" pf-reassigned-mate req-reassigned-mate secondmate:mate scout-reassigned + "$EMIT" --home "$home" --obligation pf-reassigned-mate --relation rel-code \ + --source-home secondmate:mate --work-id scout-reassigned --generation 1 \ + --outcome report-ready --deliverable report_path=data/scout-reassigned/report.md \ + --outcome-text 'The original child completed its investigation.' >/dev/null || fail "emit failed" + run_pf "$home" consume >/dev/null || fail "consume failed" + FAKE_CURL_LOG="$log" run_pf "$home" deliver pf-reassigned-mate >/dev/null || fail "deliver failed" + + rm -rf "$original" + mkdir -p "$replacement/state" + replacement=$(cd "$replacement" && pwd -P) + printf 'mate\n' > "$replacement/.fm-secondmate-home" + fm_write_meta "$replacement/state/scout-reassigned.meta" \ + "status=working" "x_request=req-unrelated-replacement" + fm_write_meta "$home/state/mate.meta" "kind=secondmate" "home=$replacement" + + expect_failure "retire must not clear a reassigned secondmate home" \ + run_pf "$home" retire pf-reassigned-mate --reason "original child was removed" + assert_contains "$EXPECT_OUT" "could not clear the legacy X link" \ + "retirement must fail when the stable ID resolves to a different home" + assert_present "$home/state/public-followup/registry/pf-reassigned-mate" \ + "a reassigned child must retain the registration" + assert_absent "$home/state/public-followup/retired/pf-reassigned-mate" \ + "a reassigned child must not create a retirement receipt" + assert_grep 'x_request=req-unrelated-replacement' "$replacement/state/scout-reassigned.meta" \ + "failed retirement must preserve the replacement home's Relay link" + pass "retire fails closed when a secondmate ID is reassigned" +} + +test_rechain_refuses_unclaimed_existing_destination() { + local home log out + home=$(make_home rechain-existing-destination) + log="$home/curl.log"; : > "$log" + seed_repro_commitment "$home" public-final-existing-a req-existing main scout-existing + "$EMIT" --home "$home" --obligation public-final-existing-a --relation rel-code \ + --source-home main --work-id scout-existing --generation 1 \ + --outcome report-ready --deliverable report_path=data/scout-existing/report.md \ + --outcome-text 'Investigation complete.' >/dev/null || fail "emit failed" + run_pf "$home" consume >/dev/null || fail "consume failed" + FAKE_CURL_LOG="$log" run_pf "$home" deliver public-final-existing-a >/dev/null \ + || fail "deliver failed" + + jq -n '{type:"pr-merged", project:"firstmate", required_deliverables:["pr_url"], + completion_policy:"all-required"}' > "$home/collision-expected.json" + tasks_in "$home" public-followup add public-final-existing-b \ + --request-context-file "$home/request.json" --purpose promised-final \ + --expected-final-file "$home/collision-expected.json" \ + --expires-at 2026-08-28T01:12:00Z >/dev/null || fail "could not seed destination collision" + + expect_failure "a first rechain must not adopt an unrelated existing obligation" \ + run_pf "$home" rechain public-final-existing-b --from public-final-existing-a \ + --work-home main --work-id ship-existing --expected pr-merged + assert_contains "$EXPECT_OUT" "was not created by this rechain" \ + "the collision refusal must identify the unclaimed destination" + out=$(cat "$home/state/public-followup/registry/public-final-existing-a") + case "$out" in + *rechain_to=*) fail "a destination collision must not claim the source" ;; + esac + assert_absent "$home/state/public-followup/registry/public-final-existing-b" \ + "an unrelated obligation must not become a registered destination" + pass "rechain refuses an unrelated existing destination" +} + +test_pending_skips_concurrent_retirement() { + local home log real_tasks pending_pid locker_pid rc=0 + home=$(make_home pending-retirement-race) + log="$home/curl.log"; : > "$log" + seed_commitment "$home" pf-race req-race discord main work-race + emit_terminal "$home" "$home" pf-race main work-race >/dev/null || fail "race emit failed" + run_pf "$home" consume >/dev/null || fail "race consume failed" + FAKE_CURL_LOG="$log" run_pf "$home" deliver pf-race >/dev/null || fail "race deliver failed" + sed -e 's/^state=delivered$/state=open/' \ + -e '/^delivered_at=/d' -e '/^delivered_obligation=/d' \ + "$home/state/public-followup/registry/pf-race" > "$home/race-open" + mv "$home/race-open" "$home/state/public-followup/registry/pf-race" + chmod 600 "$home/state/public-followup/registry/pf-race" + + seed_commitment "$home" pf-race-other req-race-other discord main work-race-other + + FM_RACE_HOME="$home" FM_RACE_ROOT="$ROOT" bash -c ' + . "$FM_RACE_ROOT/bin/fm-public-followup-lib.sh" + fm_pf_registry_lock_acquire "$FM_RACE_HOME/state" pf-race || exit 1 + : > "$FM_RACE_HOME/lock-ready" + while [ ! -e "$FM_RACE_HOME/release-lock" ]; do sleep 0.02; done + sleep 0.1 + mkdir -p "$FM_RACE_HOME/state/public-followup/retired" + printf "reason=concurrent close\nretired_at=2026-08-01T00:00:00Z\n" \ + > "$FM_RACE_HOME/state/public-followup/retired/pf-race" + chmod 600 "$FM_RACE_HOME/state/public-followup/retired/pf-race" + rm -f "$FM_RACE_HOME/state/public-followup/registry/pf-race" + fm_pf_registry_lock_release "$FM_RACE_HOME/state" pf-race + ' & + locker_pid=$! + for _ in $(seq 1 100); do [ -e "$home/lock-ready" ] && break; sleep 0.02; done + [ -e "$home/lock-ready" ] || fail "race locker did not start" + + real_tasks=$(command -v tasks-axi) + cat > "$home/fakebin/tasks-axi" <<'SH' +#!/usr/bin/env bash +"$REAL_TASKS_AXI" "$@" +rc=$? +if [ "$1" = public-followup ] && [ "$2" = list ]; then + : > "$PENDING_LISTED" +fi +exit "$rc" +SH + chmod +x "$home/fakebin/tasks-axi" + REAL_TASKS_AXI="$real_tasks" PENDING_LISTED="$home/pending-listed" \ + run_pf "$home" pending > "$home/pending-race.out" 2>&1 & + pending_pid=$! + for _ in $(seq 1 100); do [ -e "$home/pending-listed" ] && break; sleep 0.02; done + [ -e "$home/pending-listed" ] || fail "pending did not snapshot the backlog" + : > "$home/release-lock" + wait "$locker_pid" || fail "race retirement failed" + wait "$pending_pid" || rc=$? + [ "$rc" -eq 0 ] || fail "pending aborted on concurrent retirement: $(cat "$home/pending-race.out")" + assert_grep 'unresolved pf-race-other ' "$home/pending-race.out" \ + "pending must continue surfacing unrelated loops after concurrent retirement: $(cat "$home/pending-race.out")" + pass "pending skips a registration retired during settlement" +} + +test_retire_reason_closes_the_open_loop() { + local home log out registry_file receipt_mode + home=$(make_home retire-reason) + log="$home/curl.log"; : > "$log" + seed_commitment "$home" pf-retire req-retire discord main work-retire + emit_terminal "$home" "$home" pf-retire main work-retire >/dev/null || fail "emit failed" + FAKE_CURL_LOG="$log" run_pf "$home" consume >/dev/null || fail "consume failed" + FAKE_CURL_LOG="$log" run_pf "$home" deliver pf-retire >/dev/null || fail "deliver failed" + run_pf "$home" pending | grep -q '^open-loop pf-retire ' \ + || fail "pending must show the delivered open loop" + expect_failure "retire without --reason must refuse" run_pf "$home" retire pf-retire + assert_contains "$EXPECT_OUT" "--reason" "the refusal must name the required reason" + assert_present "$home/state/public-followup/registry/pf-retire" \ + "a reason-less retire must keep the registration" + registry_file="$home/state/public-followup/registry/pf-retire" + cat > "$home/fakebin/rm" <<EOF +#!/usr/bin/env bash +for arg in "\$@"; do + [ "\$arg" != '$registry_file' ] || exit 1 +done +exec /bin/rm "\$@" +EOF + chmod +x "$home/fakebin/rm" + expect_failure "retire must report registration removal failure" \ + run_pf "$home" retire pf-retire --reason "the public loop is finished" + assert_contains "$EXPECT_OUT" "public loop remains open" \ + "retire removal failure must report the truthful loop state" + assert_present "$registry_file" \ + "retire removal failure must retain the registration" + /bin/rm "$home/fakebin/rm" + out=$(run_pf "$home" retire pf-retire --reason "the public loop is finished") \ + || fail "retire --reason failed" + assert_contains "$out" "retired pf-retire reason=the public loop is finished" \ + "retire must report the reason" + assert_present "$home/state/public-followup/retired/pf-retire" \ + "retire must persist a private receipt before removing the registration" + assert_grep 'reason=the public loop is finished' \ + "$home/state/public-followup/retired/pf-retire" \ + "the retirement receipt must preserve the reason" + assert_grep 'retired_at=' "$home/state/public-followup/retired/pf-retire" \ + "the retirement receipt must preserve its timestamp" + receipt_mode=$(stat -c %a "$home/state/public-followup/retired/pf-retire" 2>/dev/null \ + || stat -f %Lp "$home/state/public-followup/retired/pf-retire" 2>/dev/null) \ + || fail "could not inspect the retirement receipt mode" + [ "$receipt_mode" = 600 ] || fail "the retirement receipt must be private" + assert_absent "$home/state/public-followup/registry/pf-retire" \ + "retire is the only removal" + out=$(run_pf "$home" pending || true) + case "$out" in + *pf-retire*) fail "pending must stop listing a retired loop: $out" ;; + esac + pass "retire --reason closes the loop and drops the open-loop line" +} + +test_retention_creates_no_false_teardown_refusal() { + local home home2 rc out registry tmp + home=$(make_home retain-teardown) + seed_commitment "$home" pf-retain req-retain discord main ship-retain + fm_write_meta "$home/state/ship-retain.meta" \ + "window=firstmate:fm-ship-retain" \ + "worktree=$home/projects/gone" \ + "project=$home/projects/sample" \ + "harness=codex" \ + "kind=ship" \ + "mode=no-mistakes" + emit_terminal "$home" "$home" pf-retain main ship-retain >/dev/null || fail "emit failed" + run_pf "$home" consume >/dev/null || fail "consume failed" + FAKE_CURL_LOG="$home/curl.log" run_pf "$home" deliver pf-retain >/dev/null || fail "delivery failed" + run_pf "$home" guard-work main ship-retain \ + || fail "guard-work must pass for the work whose reply already landed" + assert_present "$home/state/public-followup/registry/pf-retain" \ + "the delivered registration must still be present" + rc=0 + PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_CONFIG_OVERRIDE="$home/config" "$TEARDOWN" ship-retain \ + > "$home/td.out" 2> "$home/td.err" || rc=$? + [ "$rc" -eq 0 ] || fail "teardown must proceed with a retained delivered registration (rc=$rc)" + case "$(cat "$home/td.err")" in + *"still owes a public reply"*) fail "retention must not create a false public-reply refusal" ;; + esac + # pending still sees the loop because teardown does not retire it + home2=$(make_home retain-pending) + seed_commitment "$home2" pf-keep req-keep discord main work-keep + emit_terminal "$home2" "$home2" pf-keep main work-keep >/dev/null || fail "emit2 failed" + run_pf "$home2" consume >/dev/null || fail "consume2 failed" + FAKE_CURL_LOG="$home2/curl.log" run_pf "$home2" deliver pf-keep >/dev/null || fail "deliver2 failed" + registry="$home2/state/public-followup/registry/pf-keep" + tmp="$registry.tmp" + grep -v -E '^(state|delivered_at|delivered_obligation)=' "$registry" > "$tmp" + printf 'state=open\n' >> "$tmp" + chmod 600 "$tmp" + mv "$tmp" "$registry" + out=$(run_pf "$home2" pending) + assert_contains "$out" "open-loop pf-keep" "pending must keep a delivered registration as an open loop" + assert_grep 'state=delivered' "$registry" \ + "pending must repair a settled registration left open after legacy cleanup failed" + pass "retention creates no false teardown refusal and pending no longer prunes" +} + +test_expiry_escalation_uses_now_override() { + local home out exp now_closing now_expired registry tmp + home=$(make_home expiry-window) + seed_repro_commitment "$home" pf-exp req-exp main work-exp + exp=$(date -u -j -f '%Y-%m-%dT%H:%M:%SZ' '2026-08-28T01:12:00Z' +%s 2>/dev/null) \ + || exp=$(date -u -d '2026-08-28T01:12:00Z' +%s) + now_closing=$((exp - 3600)) + now_expired=$((exp + 60)) + out=$(FMX_NOW_OVERRIDE="$now_expired" run_pf "$home" pending) + assert_contains "$out" "unresolved pf-exp" "an owed reply must remain listed after expiry" + assert_contains "$out" "can no longer be reached" \ + "an expired unresolved reply must escalate the unreachable thread" + assert_contains "$out" "captain decision" \ + "an expired unresolved reply must name the captain call" + "$EMIT" --home "$home" --obligation pf-exp --relation rel-code \ + --source-home main --work-id work-exp --generation 1 \ + --outcome report-ready --deliverable report_path=data/x/report.md \ + --outcome-text 'Reproduced.' >/dev/null || fail "emit failed" + run_pf "$home" consume >/dev/null || fail "consume failed" + FAKE_CURL_LOG="$home/curl.log" run_pf "$home" deliver pf-exp >/dev/null || fail "deliver failed" + out=$(FMX_NOW_OVERRIDE="$now_closing" run_pf "$home" pending) + assert_contains "$out" "open-loop pf-exp" "closing window must still list the loop" + assert_contains "$out" "DEADLINE:" "a window under 48 hours must escalate" + assert_contains "$out" "under 48 hours" "the closing wording must name the remaining window" + out=$(FMX_NOW_OVERRIDE="$now_expired" run_pf "$home" pending) + assert_contains "$out" "can no longer be reached" "a past expiry must name the unreachable thread" + assert_contains "$out" "captain decision" "a past expiry is a captain call" + FMX_NOW_OVERRIDE="$now_expired" expect_failure "rechain past expiry must refuse" \ + run_pf "$home" rechain pf-exp-next --from pf-exp --work-home main --work-id work-next --expected pr-merged + assert_contains "$EXPECT_OUT" "can no longer be reached" "rechain must name the closed window" + registry="$home/state/public-followup/registry/pf-exp" + tmp="$registry.tmp" + awk ' + /^followup_expires_at=/ { print "followup_expires_at=not-a-time"; next } + { print } + ' "$registry" > "$tmp" + chmod 600 "$tmp" + mv "$tmp" "$registry" + expect_failure "rechain with an unknown expiry window must refuse" \ + run_pf "$home" rechain pf-exp-next --from pf-exp --work-home main --work-id work-next --expected pr-merged + assert_contains "$EXPECT_OUT" "thread window cannot be checked" \ + "rechain must fail closed when its retained expiry cannot be parsed" + pass "expiry escalation is pinned by FMX_NOW_OVERRIDE" +} + +test_brief_fails_without_typed_deliverable_keys() { + local home real_tasks invalid + home=$(make_home brief-keys) + seed_commitment "$home" pf-brief req-brief discord main work-brief + real_tasks=$(command -v tasks-axi) + cat > "$home/fakebin/tasks-axi" <<'SH' +#!/usr/bin/env bash +if [ "${FAKE_INVALID_KEYS:-}" = unreadable ]; then + exit 69 +fi +"$REAL_TASKS_AXI" "$@" | jq --argjson invalid "$FAKE_INVALID_KEYS" ' + if .public_followups then + .public_followups |= map( + if .id == "pf-brief" then + .public_followup.expected_final.required_deliverables = $invalid + else . end) + else . end' +SH + chmod +x "$home/fakebin/tasks-axi" + REAL_TASKS_AXI="$real_tasks" FAKE_INVALID_KEYS=unreadable expect_failure \ + "brief must fail when typed deliverable keys cannot be read" \ + run_pf "$home" brief pf-brief + assert_contains "$EXPECT_OUT" "could not read public-followup obligation" \ + "brief must explain why it cannot produce executable instructions" + assert_not_contains "$EXPECT_OUT" "<key>=<value>" \ + "brief must never substitute a generic deliverable placeholder" + + for invalid in '["pr_url",7]' '["pr_url",""]' '["PR_URL"]' '["pr-url"]'; do + REAL_TASKS_AXI="$real_tasks" FAKE_INVALID_KEYS="$invalid" expect_failure \ + "brief must reject an invalid required deliverable array" \ + run_pf "$home" brief pf-brief + assert_contains "$EXPECT_OUT" "no readable required deliverable keys" \ + "brief must reject the complete contract when any key is invalid" + assert_not_contains "$EXPECT_OUT" "--deliverable pr_url=<value>" \ + "brief must not emit a partial contract from an invalid key array" + done + pass "brief fails explicitly when typed deliverable keys are unavailable" +} + +test_prechange_registration_is_open_and_unrechainable() { + local home file out + home=$(make_home prechange) + mkdir -p "$home/state/public-followup/registry" + chmod 700 "$home/state/public-followup" "$home/state/public-followup/registry" + file="$home/state/public-followup/registry/pf-legacy" + printf 'obligation_id=pf-legacy\nrelation_id=rel-code\nwork_home=main\nwork_id=work-legacy\ngeneration=1\nplatform=discord\nrequest_id=req-legacy\n' \ + > "$file" + chmod 600 "$file" + out=$(run_pf "$home" pending) || fail "pending must not crash on a pre-change registration" + assert_contains "$out" "open-loop pf-legacy" "a pre-change record is an open loop" + assert_contains "$out" "unrechainable" "a pre-change record must be reported un-rechainable" + expect_failure "rechain of a pre-change record must refuse" \ + run_pf "$home" rechain pf-new --from pf-legacy --work-home main --work-id work-next --expected pr-merged + case "$EXPECT_OUT" in + *un-rechainable*) ;; + *'not state=delivered'*) ;; + *) fail "rechain must refuse a pre-change record without crashing: $EXPECT_OUT" ;; + esac + printf 'state=delivered\ndelivered_at=2026-08-21T00:00:00Z\n' >> "$file" + expect_failure "delivered pre-change record without context is un-rechainable" \ + run_pf "$home" rechain pf-new --from pf-legacy --work-home main --work-id work-next --expected pr-merged + assert_contains "$EXPECT_OUT" "un-rechainable" "missing request_context_b64 must be named" + pass "pre-change registrations are open loops and un-rechainable, never a crash" +} + +test_x_request_teardown_warns_when_final_unposted() { + local home rc + home=$(make_home xreq-warn) + fm_write_meta "$home/state/linked-task.meta" \ + "window=firstmate:fm-linked-task" \ + "worktree=$home/projects/gone" \ + "project=$home/projects/sample" \ + "kind=ship" \ + "mode=local-only" \ + "x_request=req-legacy-final" + rc=0 + PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_CONFIG_OVERRIDE="$home/config" "$TEARDOWN" linked-task \ + > "$home/td.out" 2> "$home/td.err" || rc=$? + [ "$rc" -eq 0 ] || fail "legacy-link warning must not block teardown (rc=$rc)" + assert_grep "still carries an unreconciled Relay request link (req-legacy-final) on its task record" "$home/td.err" \ + "teardown must report the remaining link without claiming the post failed" + assert_no_grep "never posted" "$home/td.err" \ + "a remaining legacy link must not be treated as proof that no post landed" + pass "teardown reports an unreconciled legacy Relay link" +} + +test_secondmate_promotion_uses_teardown_parent_resolution() { + local parent stale child remote_child out + parent=$(make_home promote-parent) + stale=$(make_home promote-stale-parent) + child=$(make_home promote-child relay-off) + printf '%s\n' mate > "$child/.fm-secondmate-home" + printf 'schema=fm-secondmate-parent.v1\nroute=local\nparent_home=%s\n' \ + "$stale" > "$child/.fm-secondmate-parent" + printf -- '- mate - synthetic (home: %s; scope: synthetic; projects: ; added 2026-08-21)\n' \ + "$child" > "$parent/data/secondmates.md" + fm_write_meta "$parent/state/mate.meta" "kind=secondmate" "home=$child" + mkdir -p "$parent/state/public-followup/registry" "$stale/state/public-followup/registry" + printf 'obligation_id=pf-valid\nwork_home=secondmate:mate\nwork_id=promote-legacy\nstate=delivered\n' \ + > "$parent/state/public-followup/registry/pf-valid" + printf 'obligation_id=pf-stale\nwork_home=secondmate:mate\nwork_id=promote-conflict\nstate=delivered\n' \ + > "$stale/state/public-followup/registry/pf-stale" + chmod 600 "$parent/state/public-followup/registry/pf-valid" \ + "$stale/state/public-followup/registry/pf-stale" + + fm_write_meta "$child/state/promote-conflict.meta" \ + "window=firstmate:fm-promote-conflict" "kind=scout" + out=$(PATH="$child/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$child" \ + FM_STATE_OVERRIDE="$child/state" FM_PUBLIC_FOLLOWUP_PRIMARY_HOME="$parent" \ + "$PROMOTE" promote-conflict --mode local-only --yolo off 2>&1) \ + || fail "promotion must not block on conflicting parent bindings: $out" + assert_contains "$out" "promoted promote-conflict to ship" \ + "parent-resolution trouble must never refuse the kind flip" + assert_contains "$out" "could not resolve the consent-holding parent home" \ + "conflicting live and durable bindings must warn" + assert_not_contains "$out" "--from pf-stale" \ + "a stale durable parent must not produce a rechain hint" + + rm -f "$child/.fm-secondmate-parent" + fm_write_meta "$child/state/promote-legacy.meta" \ + "window=firstmate:fm-promote-legacy" "kind=scout" + out=$(PATH="$child/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$child" \ + FM_STATE_OVERRIDE="$child/state" FM_PUBLIC_FOLLOWUP_PRIMARY_HOME="$parent" \ + "$PROMOTE" promote-legacy --mode local-only --yolo off 2>&1) \ + || fail "legacy parent recovery must not block promotion: $out" + assert_contains "$out" "next: FM_HOME=" \ + "a recovered legacy parent must identify the consent-holding home" + assert_contains "$out" "--from pf-valid --work-home secondmate:mate --work-id promote-legacy" \ + "legacy parent recovery must print the rechain hint" + + remote_child=$(make_home promote-remote-child relay-off) + printf '%s\n' remote-mate > "$remote_child/.fm-secondmate-home" + printf 'schema=fm-secondmate-parent.v1\nroute=remote\nparent_host=remote.example\n' \ + > "$remote_child/.fm-secondmate-parent" + printf 'FMX_PAIRING_TOKEN=child-local-token\n' > "$remote_child/.env" + fm_write_meta "$remote_child/state/promote-remote.meta" \ + "window=firstmate:fm-promote-remote" "kind=scout" + out=$(PATH="$remote_child/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$remote_child" \ + FM_STATE_OVERRIDE="$remote_child/state" \ + "$PROMOTE" promote-remote --mode local-only --yolo off 2>&1) \ + || fail "a remote parent route must not block promotion: $out" + assert_contains "$out" "promoted promote-remote to ship" \ + "an unresolved remote parent must never refuse the kind flip" + assert_contains "$out" "could not resolve the consent-holding parent home" \ + "a remote route with a local Relay token must warn as teardown does" + pass "secondmate promotion matches teardown parent resolution" +} + test_outcome_text_is_bounded_without_corrupting_characters test_restart_e2e_delivers_exactly_once test_duplicate_event_and_replay_are_noops @@ -1355,3 +2233,23 @@ test_exhausted_binding_is_not_retried test_relay_poll_stays_inert_and_surfaces_once test_session_start_surfaces_only_when_owed test_typed_records_exclude_raw_public_material +test_dropped_baton_now_surfaces_open_loop +test_control_registered_followon_is_guarded +test_rechain_delivers_second_post_on_same_thread +test_rechain_resumes_after_partial_add +test_rechain_claims_delivered_source_once +test_failed_rechain_retirement_keeps_source_claimed +test_registration_replay_preserves_delivery_and_retirement +test_redelivery_does_not_report_retired_loop_open +test_retire_after_secondmate_home_removal +test_retire_refuses_unbound_existing_secondmate +test_retire_refuses_reassigned_secondmate_home +test_rechain_refuses_unclaimed_existing_destination +test_pending_skips_concurrent_retirement +test_retire_reason_closes_the_open_loop +test_retention_creates_no_false_teardown_refusal +test_expiry_escalation_uses_now_override +test_brief_fails_without_typed_deliverable_keys +test_prechange_registration_is_open_and_unrechainable +test_x_request_teardown_warns_when_final_unposted +test_secondmate_promotion_uses_teardown_parent_resolution diff --git a/tests/fm-remote-job.test.sh b/tests/fm-remote-job.test.sh index 82f1cf8ccea..436dfd44123 100755 --- a/tests/fm-remote-job.test.sh +++ b/tests/fm-remote-job.test.sh @@ -18,6 +18,7 @@ FAKE_PERL_LOG="$TMP_ROOT/perl.log" REAL_GIT=$(command -v git) OTHER_PID= RECOVERY_WORKER_PID= +REPEAT_WORKER_PID= mkdir -p "$REMOTE_ROOT/bin" "$REMOTE_HOME" "$ACCOUNT_HOME" "$RUNTIME_BIN" # worker.pid records the serving child, not its restart supervisor, so stopping # that pid alone leaves the supervisor to respawn - the leak @@ -25,6 +26,7 @@ mkdir -p "$REMOTE_ROOT/bin" "$REMOTE_HOME" "$ACCOUNT_HOME" "$RUNTIME_BIN" cleanup_remote_job_fixture() { [ -z "$OTHER_PID" ] || kill "$OTHER_PID" 2>/dev/null || true [ -z "$RECOVERY_WORKER_PID" ] || kill "$RECOVERY_WORKER_PID" 2>/dev/null || true + [ -z "$REPEAT_WORKER_PID" ] || kill "$REPEAT_WORKER_PID" 2>/dev/null || true if [ -f "$STATE_ROOT/worker.pid" ]; then fm_remote_job_stop_worker_tree "$(cat "$STATE_ROOT/worker.pid")" || true fi @@ -163,6 +165,25 @@ case ":$FM_REMOTE_JOB_OPERATOR_PATH:" in esac pass "operator PATH resolves the authorized Nix profile bin link" +# Which install of a multi-version tool a remote job resolves is decided by the +# order these directories land on PATH, so the composition has to be sorted +# rather than whatever order the filesystem returns. The fixture is created in +# a deliberately unsorted order, and the expectation is the shell's own +# pathname expansion - the mechanism the portable-PATH contract in +# tests/fm-on.test.sh reconstructs. +MISE_INSTALLS="$ACCOUNT_HOME/.local/share/mise/installs" +for TOOL_VERSION in node/26.7.0 node/8.1 node/26 bun/1.4 bun/1.3.14 python/3.12.7; do + mkdir -p "$MISE_INSTALLS/$TOOL_VERSION/bin" +done +fm_remote_job_compose_operator_path "$ACCOUNT_HOME" >/dev/null +MISE_COMPOSED=$(printf '%s\n' "$FM_REMOTE_JOB_OPERATOR_PATH" | tr ':' '\n' | grep -F "$MISE_INSTALLS/" || true) +MISE_EXPECTED=$(printf '%s\n' "$MISE_INSTALLS"/*/*/bin) +[ "$MISE_COMPOSED" = "$MISE_EXPECTED" ] \ + || fail "the composed operator PATH did not order tool installs like the shell's own expansion"$'\n'"expected: $MISE_EXPECTED"$'\n'"actual: $MISE_COMPOSED" +# This assertion detects the defect on bash 3.2 and 5.2, where compgen -G returns unsorted glob matches, but reads green on bash 5.3+ because glob sorting moved into the glob library so both mechanisms agree there. +rm -rf -- "$ACCOUNT_HOME/.local/share/mise" +pass "operator PATH orders discovered tool installs deterministically" + HOME="$ACCOUNT_HOME" PATH="$RUNTIME_BIN:/usr/bin:/bin:/usr/sbin:/sbin" FM_FAKE_PERL_LOG="$FAKE_PERL_LOG" \ FM_ROOT_OVERRIDE="$REMOTE_ROOT" FM_REMOTE_JOB_STATE_ROOT="$STATE_ROOT" \ FM_REMOTE_JOB_PLATFORM_OVERRIDE=Linux FM_REMOTE_JOB_TIMEOUT=5 \ @@ -621,4 +642,72 @@ wait "$RECOVERY_WORKER_PID" 2>/dev/null || true RECOVERY_WORKER_PID= pass "quarantine clears only after recorded execution has stopped" +# A replacement stops a Linux worker by signalling its whole isolated group, and +# the supervisor in that group forwards a second stop signal to the same serving +# child, so the serving child is always signalled more than once. Signal a small +# bounded burst and then keep signalling until it is gone: the first signal +# starts the shutdown and every later one lands inside it, the same way the group +# signal and the forwarded signal do. A shutdown that dies part way through +# leaves its ownership lock behind holding a half-written temp file no later +# worker can clear, and every replacement then fails to report ready. +# +# The burst is bounded and the follow-up signals are paced deliberately. An +# unpaced signal loop delivers hundreds of thousands of signals per second, +# which corrupts the signalled bash's own pending-trap bookkeeping ("warning: +# run_pending_traps: bad value in trap_list[15]") and then kills it part way +# through the shutdown with SIGTERM or SIGSEGV. That reports a shutdown defect +# this worker does not have. Ten back-to-back signals still all land inside the +# shutdown's first file operation, so the repeat this pins is unchanged: with +# the default disposition restored instead of ignored, the ownership lock is +# left behind every run. +REPEAT_HOME="$TMP_ROOT/repeat-signal-account" +REPEAT_STATE="$TMP_ROOT/repeat-signal-jobs" +mkdir -p "$REPEAT_HOME" +chmod 700 "$REPEAT_HOME" +HOME="$REPEAT_HOME" FM_ROOT_OVERRIDE="$REMOTE_ROOT" FM_REMOTE_JOB_STATE_ROOT="$REPEAT_STATE" \ + FM_REMOTE_JOB_PLATFORM_OVERRIDE=Linux "$REMOTE_ROOT/bin/fm-remote-job-worker.sh" --serve \ + > "$TMP_ROOT/repeat-signal.out" 2> "$TMP_ROOT/repeat-signal.err" & +REPEAT_WORKER_PID=$! +for _ in $(seq 1 300); do + [ -f "$REPEAT_STATE/worker.ready" ] && break + sleep 0.05 +done +assert_present "$REPEAT_STATE/worker.ready" "the repeated-signal worker did not become ready" +REPEAT_DEADLINE=$((SECONDS + 30)) +REPEAT_BURST=0 +while [ "$REPEAT_BURST" -lt 10 ]; do + kill -TERM "$REPEAT_WORKER_PID" 2>/dev/null || true + REPEAT_BURST=$((REPEAT_BURST + 1)) +done +while kill -0 "$REPEAT_WORKER_PID" 2>/dev/null && [ "$SECONDS" -lt "$REPEAT_DEADLINE" ]; do + kill -TERM "$REPEAT_WORKER_PID" 2>/dev/null || true + sleep 0.05 +done +if kill -0 "$REPEAT_WORKER_PID" 2>/dev/null; then + kill -KILL "$REPEAT_WORKER_PID" 2>/dev/null || true + wait "$REPEAT_WORKER_PID" 2>/dev/null || true + REPEAT_WORKER_PID= + fail "the repeatedly signalled worker never finished its shutdown" +fi +wait "$REPEAT_WORKER_PID" 2>/dev/null || true +REPEAT_WORKER_PID= +assert_absent "$REPEAT_STATE/worker.lock" \ + "a repeatedly signalled shutdown left its ownership lock behind" +assert_absent "$REPEAT_STATE/worker.ready" \ + "a repeatedly signalled shutdown left its readiness heartbeat behind" +HOME="$REPEAT_HOME" FM_ROOT_OVERRIDE="$REMOTE_ROOT" FM_REMOTE_JOB_STATE_ROOT="$REPEAT_STATE" \ + FM_REMOTE_JOB_PLATFORM_OVERRIDE=Linux "$REMOTE_ROOT/bin/fm-remote-job-worker.sh" --serve \ + >> "$TMP_ROOT/repeat-signal.out" 2>> "$TMP_ROOT/repeat-signal.err" & +REPEAT_WORKER_PID=$! +for _ in $(seq 1 600); do + [ -f "$REPEAT_STATE/worker.ready" ] && break + sleep 0.05 +done +assert_present "$REPEAT_STATE/worker.ready" \ + "the worker after a repeatedly signalled shutdown never reported ready" +kill -TERM "$REPEAT_WORKER_PID" +wait "$REPEAT_WORKER_PID" 2>/dev/null || true +REPEAT_WORKER_PID= +pass "a repeatedly signalled shutdown still releases ownership for the next worker" + echo "ALL TESTS PASSED" diff --git a/tests/fm-spawn-batch.test.sh b/tests/fm-spawn-batch.test.sh index d35ab2df7a8..1c6a550d4ac 100755 --- a/tests/fm-spawn-batch.test.sh +++ b/tests/fm-spawn-batch.test.sh @@ -125,7 +125,7 @@ test_batch_requires_the_shared_delivery_contract() { status=$? [ "$status" -ne 0 ] || fail "a ship batch without --yolo should exit non-zero" printf '%s\n' "$out" | grep -F 'ship spawns require --yolo' >/dev/null \ - || fail "batch refusal did not name the missing approval posture" + || fail "batch refusal did not name the missing merge posture" pass "batch dispatch requires the shared ship delivery contract before any pair runs" } diff --git a/tests/fm-supervision-events.test.sh b/tests/fm-supervision-events.test.sh index ca6c6839072..57bd63d4b9a 100755 --- a/tests/fm-supervision-events.test.sh +++ b/tests/fm-supervision-events.test.sh @@ -82,6 +82,19 @@ fi grep -q 'absorbed push' "$STATE_DIR/.watch-triage.log" 2>/dev/null || fail "the paused absorb should be logged to the triage log" pass "handle_push_transition: a declared-pause crew is absorbed (no fast wake), left to the poll loop's long cadence" +# --- handle_push_transition: absorb for a verified captain-held transfer ------- + +reset_state +fm_write_meta "$STATE_DIR/tk2h.meta" "window=default:wG:pQ" "backend=herdr" "kind=ship" +printf 'captain-held [key=route]: tracked by task-decision-route\n' > "$STATE_DIR/tk2h.status" +handle_push_transition herdr default "$(mkrec wG:pQ blocked)" +if [ -e "$STATE_DIR/.wake-queue" ] && grep -q 'stale' "$STATE_DIR/.wake-queue"; then + fail "a captain-held crew must NOT be fast-escalated: $(cat "$STATE_DIR/.wake-queue")" +fi +[ ! -s "$WAKE_LOG" ] || fail "a captain-held crew must not wake the supervisor from the event fast-path" +grep -q 'absorbed push' "$STATE_DIR/.watch-triage.log" 2>/dev/null || fail "the captain-held absorb should be logged to the triage log" +pass "handle_push_transition: a captain-held crew is absorbed (no fast wake), left to the poll loop's long cadence" + # --- event_wait_or_sleep: secondmate windows are excluded from the pane list -- reset_state diff --git a/tests/fm-task-delivery.test.sh b/tests/fm-task-delivery.test.sh index 34df6fe748a..bfe835b8416 100755 --- a/tests/fm-task-delivery.test.sh +++ b/tests/fm-task-delivery.test.sh @@ -220,7 +220,7 @@ test_promote_requires_and_records_the_delivery_contract() { out=$(FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" "$PROMOTE" promote-d1 --mode direct-PR 2>&1) status=$? [ "$status" -ne 0 ] || fail "promotion without --yolo should exit non-zero" - assert_contains "$out" "promotion requires --yolo" "promote refusal did not name the missing approval posture" + assert_contains "$out" "promotion requires --yolo" "promote refusal did not name the missing merge posture" out=$(FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" "$PROMOTE" promote-d1 --mode no-mistakes-prod-only --yolo off 2>&1) status=$? @@ -232,7 +232,7 @@ test_promote_requires_and_records_the_delivery_contract() { expect_code 0 "$status" "a promotion carrying both flags should succeed" assert_grep 'kind=ship' "$meta" "promotion did not restore ship teardown protection" assert_grep 'mode=direct-PR' "$meta" "promotion did not record the decided delivery mode" - assert_grep 'yolo=on' "$meta" "promotion did not record the decided approval posture" + assert_grep 'yolo=on' "$meta" "promotion did not record the decided merge posture" assert_contains "$out" "ship instructions for mode=direct-PR" "promotion hint did not carry the decided mode" [ "$(grep -c '^mode=' "$meta")" = 1 ] || fail "promotion left more than one mode= line in the task record" pass "fm-promote: promotion requires the delivery contract and records it exactly once" diff --git a/tests/fm-tool-update-check.test.sh b/tests/fm-tool-update-check.test.sh new file mode 100755 index 00000000000..2f8843a821d --- /dev/null +++ b/tests/fm-tool-update-check.test.sh @@ -0,0 +1,1039 @@ +#!/usr/bin/env bash +# Tests for fm-tool-update-check.sh, the watched tooling update report. +# +# The case that matters most is PATH skew: a tool that has already installed its +# newer copy, while PATH still resolves an older one. On 2026-08-20 that exact +# shape broke this fleet. Herdr self-installed 0.8.2 into ~/.local/bin, a version +# manager kept its own 0.8.0 earlier on PATH inside a directory named "latest", +# and every Herdr command then failed on a protocol mismatch. A check that only +# asks "is a newer version published" reports everything up to date and misses +# it, so test_path_skew_is_reported_from_every_copy reproduces the incident and +# asserts the report names the older copy PATH resolves AND the newer copy that +# is already installed. A single `command -v` lookup cannot know the second +# version, so that assertion fails against any build without real per-copy +# probing. +# +# The fixtures use a synthetic command name and their own temporary PATH +# directories, so no case ever probes, launches, or otherwise touches a tool +# actually installed on this host. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +CHECK="$ROOT/bin/fm-tool-update-check.sh" +CHECKPOINT="$ROOT/bin/fm-watch-checkpoint.sh" +TMP_ROOT=$(fm_test_tmproot fm-tool-update-check) + +# Exported here, at the top level, because git_fixture runs inside a command +# substitution and an export from that subshell never reaches the cases, which +# make fixture commits of their own. A host with no git identity configured +# would otherwise fail those commits and leave the fixture in a shape the case +# did not ask for. +fm_git_identity fmtest fmtest@example.invalid + +# The incident's tool, under a name that cannot exist on this host. +TOOL=herdr-fixture + +make_home() { + local name=$1 home + home="$TMP_ROOT/$name" + mkdir -p "$home/state" "$home/config" + printf '%s\n' "$home" +} + +# make_copy <dir> <command> <version-output>: an executable copy that answers +# --version with the given text and nothing else. +make_copy() { + local dir=$1 command_name=$2 text=$3 + mkdir -p "$dir" + cat > "$dir/$command_name" <<SH +#!/usr/bin/env bash +printf '%s\n' '$text' +SH + chmod 0755 "$dir/$command_name" +} + +# make_slow_copy <dir> <command> <seconds>: a copy that answers far too late, so a +# case can spend the sweep budget the way a hung tool would. +make_slow_copy() { + local dir=$1 command_name=$2 seconds=$3 + mkdir -p "$dir" + cat > "$dir/$command_name" <<SH +#!/usr/bin/env bash +sleep $seconds +printf 'herdr 0.8.2\n' +SH + chmod 0755 "$dir/$command_name" +} + +# make_counting_copy <dir> <command> <version-output> <log>: the same copy, which +# also appends one line to <log> every time it runs, so a case can assert how +# many times the check actually probed it. +make_counting_copy() { + local dir=$1 command_name=$2 text=$3 log=$4 + mkdir -p "$dir" + cat > "$dir/$command_name" <<SH +#!/usr/bin/env bash +printf 'probed\n' >> '$log' +printf '%s\n' '$text' +SH + chmod 0755 "$dir/$command_name" +} + +write_config() { + local home=$1 + shift + printf '%s\n' "$*" > "$home/config/watched-tools.json" +} + +# The fixture directories first, then the ambient PATH, which the check needs +# because it shells out to ordinary tools such as jq, git, date, grep, stat, and +# timeout. What keeps a tool installed on this host out of a fixture is the +# synthetic command name, not this PATH, so every case watches a command name +# that cannot exist here. +fixture_path() { + printf '%s:%s\n' "$1" "$PATH" +} + +# The watcher check timeout is pinned to its documented default here, because the +# sweep budget is cut to fit it and an operator's ambient value would otherwise +# add a report line to cases that mean to be silent. The one case that exercises +# the cut sets its own value. +run_check() { + local home=$1 path=$2 out=$3 + shift 3 + local status=0 + env FM_CHECK_TIMEOUT=30 "$@" FM_HOME="$home" PATH="$path" FM_TOOL_UPDATE_INTERVAL=0 "$CHECK" >"$out" 2>&1 || status=$? + expect_code 0 "$status" "check exit" +} + +# --- the regression this script exists for ---------------------------------- + +test_path_skew_is_reported_from_every_copy() { + local home stale fresh out report + home=$(make_home skew) + # The stale copy sits in a directory named "latest" on purpose: the incident's + # version manager did exactly that, so a directory name is no evidence at all. + stale="$TMP_ROOT/skew/mise/installs/herdr/latest/bin" + fresh="$TMP_ROOT/skew/local/bin" + make_copy "$stale" "$TOOL" 'herdr 0.8.0' + make_copy "$fresh" "$TOOL" 'herdr 0.8.2' + write_config "$home" "{\"tools\":[{\"name\":\"herdr\",\"command\":\"$TOOL\",\"version_args\":[\"--version\"]}]}" + out="$home/out.txt" + run_check "$home" "$(fixture_path "$stale:$fresh")" "$out" + report=$(cat "$out") + + assert_contains "$report" "herdr update not in effect" "PATH skew was not reported as an update that is not in effect" + # Both sides of the comparison must be named, and the newer one can only be + # known by asking a copy other than the one PATH resolves. + assert_contains "$report" "PATH resolves 0.8.0 at $stale/$TOOL" "the report does not name the older version PATH actually resolves" + assert_contains "$report" "0.8.2 is installed at $fresh/$TOOL" "the report does not name the newer installed copy, so no other PATH copy was asked for its version" + assert_not_contains "$report" "update available" "PATH skew must not be reported as a published update" + assert_contains "$report" "$(printf 'tool updates:')" "the report is missing its one-line prefix" + [ "$(wc -l < "$out")" = 1 ] || fail "the report must be exactly one line for the wake record" + pass "PATH skew is reported by asking every copy on PATH for its own version" +} + +test_newest_copy_first_on_path_is_silent() { + local home stale fresh out + # Control for the case above: the same two copies, the newer one resolved + # first, must produce no report at all. + home=$(make_home no-skew) + stale="$TMP_ROOT/no-skew/mise/installs/herdr/latest/bin" + fresh="$TMP_ROOT/no-skew/local/bin" + make_copy "$stale" "$TOOL" 'herdr 0.8.0' + make_copy "$fresh" "$TOOL" 'herdr 0.8.2' + write_config "$home" "{\"tools\":[{\"name\":\"herdr\",\"command\":\"$TOOL\"}]}" + out="$home/out.txt" + run_check "$home" "$(fixture_path "$fresh:$stale")" "$out" + [ ! -s "$out" ] || fail "check reported skew when PATH already resolves the newest copy: $(cat "$out")" + pass "no report when PATH already resolves the newest installed copy" +} + +test_identical_versions_are_silent() { + local home first second out + home=$(make_home same-version) + first="$TMP_ROOT/same-version/a/bin" + second="$TMP_ROOT/same-version/b/bin" + make_copy "$first" "$TOOL" 'herdr 0.8.2' + make_copy "$second" "$TOOL" 'herdr 0.8.2' + write_config "$home" "{\"tools\":[{\"name\":\"herdr\",\"command\":\"$TOOL\"}]}" + out="$home/out.txt" + run_check "$home" "$(fixture_path "$first:$second")" "$out" + [ ! -s "$out" ] || fail "two copies of the same version reported skew: $(cat "$out")" + pass "two copies of the same version are not skew" +} + +test_one_copy_reached_twice_is_probed_once() { + local home dir link out log probes + # A single install reachable through two PATH entries must not read as two + # installs, or a symlinked bin directory would report skew against itself. + # Silence alone does not prove that, because two answers of the same version + # are silent too, so count the probes: the one install must be asked once. + home=$(make_home one-copy) + dir="$TMP_ROOT/one-copy/real/bin" + link="$TMP_ROOT/one-copy/linked-bin" + log="$TMP_ROOT/one-copy/probes.log" + make_counting_copy "$dir" "$TOOL" 'herdr 0.8.2' "$log" + ln -s "$dir" "$link" + write_config "$home" "{\"tools\":[{\"name\":\"herdr\",\"command\":\"$TOOL\"}]}" + out="$home/out.txt" + : > "$log" + run_check "$home" "$(fixture_path "$dir:$link")" "$out" + [ ! -s "$out" ] || fail "one copy reached through two PATH entries reported a finding: $(cat "$out")" + probes=$(wc -l < "$log" | tr -d ' ') + [ "$probes" = 1 ] || fail "one install reached through two PATH entries was probed $probes times, so the two entries were not recognized as one install" + pass "one copy reached through two PATH entries is probed once as one install" +} + +test_unreadable_version_is_a_failure_not_a_pass() { + local home dir out report + # A copy that will not say what it is cannot be called current. + home=$(make_home mute) + dir="$TMP_ROOT/mute/bin" + make_copy "$dir" "$TOOL" 'no version here' + write_config "$home" "{\"tools\":[{\"name\":\"herdr\",\"command\":\"$TOOL\"}]}" + out="$home/out.txt" + run_check "$home" "$(fixture_path "$dir")" "$out" + report=$(cat "$out") + assert_contains "$report" "herdr check failed" "a copy that reports no version was treated as current" + assert_contains "$report" "$dir/$TOOL did not report a version" "the failing copy was not named" + pass "a copy that reports no version is a check failure, not a pass" +} + +test_missing_command_is_reported() { + local home out + home=$(make_home absent) + write_config "$home" '{"tools":[{"name":"herdr","command":"herdr-absent-fixture"}]}' + out="$home/out.txt" + run_check "$home" "$PATH" "$out" + assert_contains "$(cat "$out")" "herdr check failed: herdr-absent-fixture is not on PATH" "a watched command missing from PATH was not reported" + pass "a watched command missing from PATH is reported" +} + +# --- published updates ------------------------------------------------------ + +test_announced_update_is_reported_from_the_tool_itself() { + local home dir out report + # no-mistakes already announces its own update on stderr; read that rather + # than reimplementing its version lookup. + home=$(make_home announce) + dir="$TMP_ROOT/announce/bin" + mkdir -p "$dir" + cat > "$dir/no-mistakes-fixture" <<'SH' +#!/usr/bin/env bash +printf '1.46.0\n' +printf 'A new version of no-mistakes is available: v1.46.0 -> v1.47.0\n' >&2 +SH + chmod 0755 "$dir/no-mistakes-fixture" + write_config "$home" '{"tools":[{"name":"no-mistakes","command":"no-mistakes-fixture","announce_pattern":"A new version of no-mistakes is available: [^ ]+ -> [^ ]+"}]}' + out="$home/out.txt" + run_check "$home" "$(fixture_path "$dir")" "$out" + report=$(cat "$out") + assert_contains "$report" "no-mistakes update available: A new version of no-mistakes is available: v1.46.0 -> v1.47.0" "the tool's own update announcement was not reported" + assert_not_contains "$report" "not in effect" "a published update must not be reported as PATH skew" + pass "a tool's own update announcement is read from its output" +} + +test_announcement_is_read_from_a_second_command() { + local home dir out report quiet_home + # The real no-mistakes prints its version for --version but announces a new + # release only on its other commands, so the announcement has to be asked of a + # command of its own while the version probe keeps reporting the version. + home=$(make_home announce-args) + dir="$TMP_ROOT/announce-args/bin" + mkdir -p "$dir" + cat > "$dir/no-mistakes-fixture" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = "--version" ]; then + printf 'no-mistakes version v1.46.0\n' + exit 0 +fi +printf 'A new version of no-mistakes is available: v1.46.0 -> v1.53.0\n' >&2 +printf 'Usage: no-mistakes <command>\n' +SH + chmod 0755 "$dir/no-mistakes-fixture" + out="$home/out.txt" + + write_config "$home" '{"tools":[{"name":"no-mistakes","command":"no-mistakes-fixture","version_args":["--version"],"announce_args":["--help"],"announce_pattern":"A new version of no-mistakes is available: [^ ]+ -> [^ ]+"}]}' + run_check "$home" "$(fixture_path "$dir")" "$out" + report=$(cat "$out") + assert_contains "$report" "no-mistakes update available: A new version of no-mistakes is available: v1.46.0 -> v1.53.0" "the announcement was not read from the command that carries it" + assert_not_contains "$report" "check failed" "the version probe stopped reporting this copy's version" + + # Control: the same tool watched without announce_args sees only the version + # probe, which never carries the announcement, so the update is missed. + quiet_home=$(make_home announce-args-control) + write_config "$quiet_home" '{"tools":[{"name":"no-mistakes","command":"no-mistakes-fixture","version_args":["--version"],"announce_pattern":"A new version of no-mistakes is available: [^ ]+ -> [^ ]+"}]}' + run_check "$quiet_home" "$(fixture_path "$dir")" "$quiet_home/out.txt" + [ ! -s "$quiet_home/out.txt" ] || fail "the control home reported without a second command, so this test proves nothing: $(cat "$quiet_home/out.txt")" + pass "an announcement carried by another command is read from that command" +} + +test_unusable_announce_pattern_is_reported_not_read_as_silence() { + local home dir out status + # A pattern the search cannot use answers exactly like a tool with nothing to + # announce, which is the silently dead update source this check exists to + # prevent. It is reported as that tool's own check failure. + home=$(make_home bad-pattern) + dir="$TMP_ROOT/bad-pattern/bin" + make_copy "$dir" no-mistakes-fixture 'no-mistakes version v1.46.0' + write_config "$home" '{"tools":[{"name":"no-mistakes","command":"no-mistakes-fixture","announce_pattern":"A new version of no-mistakes is available: ([^ ]+ -> [^ ]+"}]}' + out="$home/out.txt" + run_check "$home" "$(fixture_path "$dir")" "$out" + assert_contains "$(cat "$out")" "no-mistakes check failed: announce_pattern is not a usable extended regular expression" "a pattern that cannot be used was read as nothing to announce" + + # Arming is a deliberate operator action, so the same registry refuses it + # rather than arming a check with a source that can never fire. + status=0 + FM_HOME="$home" "$CHECK" arm >/dev/null 2>&1 || status=$? + expect_code 1 "$status" "arm with an unusable announce_pattern exit" + assert_absent "$home/state/tool-updates.check.sh" "arm registered a check whose announcement source cannot fire" + pass "an announce_pattern that cannot be used is reported instead of read as silence" +} + +test_one_broken_pattern_does_not_blind_the_rest_of_the_sweep() { + local home stale fresh dir out report + # A one character typo in one tool's pattern must not turn off the detector for + # every other tool. The PATH skew below is the whole reason this check exists, + # so it has to be reported in the same sweep as the pattern problem. + home=$(make_home pattern-blind) + stale="$TMP_ROOT/pattern-blind/mise/installs/herdr/latest/bin" + fresh="$TMP_ROOT/pattern-blind/local/bin" + dir="$TMP_ROOT/pattern-blind/announce/bin" + make_copy "$stale" "$TOOL" 'herdr 0.8.0' + make_copy "$fresh" "$TOOL" 'herdr 0.8.2' + make_copy "$dir" no-mistakes-fixture 'no-mistakes version v1.46.0' + write_config "$home" "{\"tools\":[{\"name\":\"herdr\",\"command\":\"$TOOL\"},{\"name\":\"no-mistakes\",\"command\":\"no-mistakes-fixture\",\"announce_pattern\":\"A new version of no-mistakes is available: ([^ ]+ -> [^ ]+\"}]}" + out="$home/out.txt" + run_check "$home" "$(fixture_path "$stale:$fresh:$dir")" "$out" + report=$(cat "$out") + assert_contains "$report" "herdr update not in effect: PATH resolves 0.8.0 at $stale/$TOOL" "a broken pattern on another tool suppressed the PATH skew report" + assert_contains "$report" "no-mistakes check failed: announce_pattern is not a usable extended regular expression" "the tool whose pattern cannot be used was not named" + [ "$(wc -l < "$out")" = 1 ] || fail "the report must stay exactly one line" + pass "a broken pattern is reported for its own tool and the rest of the sweep still reports" +} + +test_an_unchecked_announcement_source_is_not_read_as_current() { + local home dir out report + # When the budget is gone the separate announcement command cannot run, and the + # version probe's output never carries the announcement. Searching that output + # anyway would present a source that was never asked as a clean result, which is + # the same silently dead source announce_args was added to close. + home=$(make_home announce-budget) + dir="$TMP_ROOT/announce-budget/bin" + mkdir -p "$dir" + cat > "$dir/no-mistakes-fixture" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = "--version" ]; then + printf 'no-mistakes version v1.46.0\n' + sleep 30 + exit 0 +fi +printf 'A new version of no-mistakes is available: v1.46.0 -> v1.53.0\n' >&2 +SH + chmod 0755 "$dir/no-mistakes-fixture" + write_config "$home" '{"tools":[{"name":"no-mistakes","command":"no-mistakes-fixture","version_args":["--version"],"announce_args":["--help"],"announce_pattern":"A new version of no-mistakes is available: [^ ]+ -> [^ ]+"}]}' + out="$home/out.txt" + run_check "$home" "$(fixture_path "$dir")" "$out" FM_TOOL_UPDATE_BUDGET_SECS=1 + report=$(cat "$out") + assert_contains "$report" "no-mistakes check failed: the time budget ran out before the update announcement was checked" "an announcement source that was never asked was not reported" + pass "an announcement source the budget could not reach is reported, not read as current" +} + +test_an_announcement_probe_that_does_not_answer_is_reported() { + local home dir out report + # no-mistakes learns about a new release from the network, so the command that + # carries the announcement is exactly the one that stalls on a flaky link. A + # source that was asked and never answered must not read as a clean sweep. + home=$(make_home announce-mute) + dir="$TMP_ROOT/announce-mute/bin" + mkdir -p "$dir" + cat > "$dir/no-mistakes-fixture" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = "--version" ]; then + printf 'no-mistakes version v1.46.0\n' + exit 0 +fi +sleep 30 +SH + chmod 0755 "$dir/no-mistakes-fixture" + write_config "$home" '{"tools":[{"name":"no-mistakes","command":"no-mistakes-fixture","version_args":["--version"],"announce_args":["--help"],"announce_pattern":"A new version of no-mistakes is available: [^ ]+ -> [^ ]+"}]}' + out="$home/out.txt" + run_check "$home" "$(fixture_path "$dir")" "$out" FM_TOOL_UPDATE_PROBE_SECS=1 + report=$(cat "$out") + assert_contains "$report" "no-mistakes check failed: $dir/no-mistakes-fixture did not answer when asked for its update announcement" "an announcement probe that never answered was read as a clean sweep" + pass "an announcement probe that does not answer is reported, not read as current" +} + +test_quiet_tool_with_announce_pattern_is_silent() { + local home dir out + home=$(make_home announce-quiet) + dir="$TMP_ROOT/announce-quiet/bin" + make_copy "$dir" no-mistakes-fixture '1.46.0' + write_config "$home" '{"tools":[{"name":"no-mistakes","command":"no-mistakes-fixture","announce_pattern":"A new version of no-mistakes is available: [^ ]+ -> [^ ]+"}]}' + out="$home/out.txt" + run_check "$home" "$(fixture_path "$dir")" "$out" + [ ! -s "$out" ] || fail "a tool announcing nothing produced a report: $(cat "$out")" + pass "a tool that announces nothing stays silent" +} + +# --- git sources ------------------------------------------------------------ + +# git_fixture <name>: a work repo whose origin branch is two commits ahead, +# with those commits already present locally so the count is exact. +git_fixture() { + local name=$1 bare work + bare="$TMP_ROOT/$name.git" + work="$TMP_ROOT/$name" + git init -q --bare --initial-branch=main "$bare" + git clone -q "$bare" "$work" 2>/dev/null + printf 'one\n' > "$work/f1" + git -C "$work" add f1 + git -C "$work" commit -qm one + printf 'two\n' > "$work/f2" + git -C "$work" add f2 + git -C "$work" commit -qm two + printf 'three\n' > "$work/f3" + git -C "$work" add f3 + git -C "$work" commit -qm three + git -C "$work" push -q origin main + git -C "$work" remote set-head origin main >/dev/null 2>&1 + printf '%s\n' "$work" +} + +test_commits_behind_origin_are_reported() { + local home work out head_before + home=$(make_home git-behind) + work=$(git_fixture git-behind-repo) + git -C "$work" reset -q --hard HEAD~2 + head_before=$(git -C "$work" rev-parse HEAD) + write_config "$home" "{\"tools\":[{\"name\":\"firstmate\",\"git\":{\"repo\":\"$work\",\"remote\":\"origin\",\"branch\":\"main\"}}]}" + out="$home/out.txt" + run_check "$home" "$PATH" "$out" + assert_contains "$(cat "$out")" "firstmate update available: local main is 2 commits behind origin/main" "commits behind the origin branch were not reported" + # The probe is read-only: the watched repository must be untouched. + [ "$(git -C "$work" rev-parse HEAD)" = "$head_before" ] || fail "the check moved the watched repository's HEAD" + git -C "$work" diff --quiet || fail "the check left changes in the watched repository" + pass "commits behind the origin branch are reported without touching the repository" +} + +test_default_branch_is_detected_when_branch_is_omitted() { + local home work out + home=$(make_home git-default) + work=$(git_fixture git-default-repo) + git -C "$work" reset -q --hard HEAD~1 + write_config "$home" "{\"tools\":[{\"name\":\"firstmate\",\"git\":{\"repo\":\"$work\"}}]}" + out="$home/out.txt" + run_check "$home" "$PATH" "$out" + assert_contains "$(cat "$out")" "firstmate update available: local main is 1 commit behind origin/main" "the default branch was not detected from the remote" + pass "an omitted branch is detected from the remote's default branch" +} + +test_default_branch_is_asked_of_the_remote_when_the_clone_has_no_record() { + local home work out + # A --single-branch clone, or one that never ran remote set-head, has no local + # refs/remotes/origin/HEAD. The remote still knows its default branch, so this + # must report the update rather than an unactionable check failure. + home=$(make_home git-symref) + work=$(git_fixture git-symref-repo) + git -C "$work" remote set-head origin --delete >/dev/null 2>&1 + git -C "$work" reset -q --hard HEAD~2 + # Ask git what it knows rather than looking for a loose ref file, which never + # exists under a non-loose ref backend and would make this vacuous there. + ! git -C "$work" symbolic-ref --quiet refs/remotes/origin/HEAD >/dev/null 2>&1 \ + || fail "the fixture still records the remote's default branch locally" + write_config "$home" "{\"tools\":[{\"name\":\"firstmate\",\"git\":{\"repo\":\"$work\"}}]}" + out="$home/out.txt" + run_check "$home" "$PATH" "$out" + assert_contains "$(cat "$out")" "firstmate update available: local main is 2 commits behind origin/main" "the default branch was not asked of the remote" + pass "the default branch is asked of the remote when the clone has no local record" +} + +test_current_and_ahead_repositories_are_silent() { + local home work out + home=$(make_home git-current) + work=$(git_fixture git-current-repo) + write_config "$home" "{\"tools\":[{\"name\":\"firstmate\",\"git\":{\"repo\":\"$work\",\"branch\":\"main\"}}]}" + out="$home/out.txt" + run_check "$home" "$PATH" "$out" + [ ! -s "$out" ] || fail "an up to date repository produced a report: $(cat "$out")" + + printf 'local only\n' > "$work/f4" + git -C "$work" add f4 + git -C "$work" commit -qm four + rm -f "$home/state/.tool-updates" + run_check "$home" "$PATH" "$out" + [ ! -s "$out" ] || fail "a repository ahead of its origin branch produced a report: $(cat "$out")" + pass "a repository that is current or ahead of its origin branch is silent" +} + +test_unusable_git_source_is_reported() { + local home out + home=$(make_home git-broken) + mkdir -p "$TMP_ROOT/git-broken/not-a-repo" + write_config "$home" "{\"tools\":[{\"name\":\"firstmate\",\"git\":{\"repo\":\"$TMP_ROOT/git-broken/not-a-repo\",\"branch\":\"main\"}}]}" + out="$home/out.txt" + run_check "$home" "$PATH" "$out" + assert_contains "$(cat "$out")" "firstmate check failed" "an unusable git source was not reported" + pass "an unusable git source is reported as a check failure" +} + +test_unreadable_remote_is_not_reported_as_a_missing_branch() { + local home work out report + # A remote that cannot be reached at all and a branch that was deleted are + # different problems with different repairs. Reporting the first as the second + # wakes firstmate with a diagnosis that is simply wrong, so the report must + # name only what the probe established. + home=$(make_home git-unreadable) + work=$(git_fixture git-unreadable-repo) + rm -rf "$TMP_ROOT/git-unreadable-repo.git" + write_config "$home" "{\"tools\":[{\"name\":\"firstmate\",\"git\":{\"repo\":\"$work\",\"remote\":\"origin\",\"branch\":\"main\"}}]}" + out="$home/out.txt" + run_check "$home" "$PATH" "$out" + report=$(cat "$out") + assert_contains "$report" "firstmate check failed" "a remote that could not be read was not reported" + assert_contains "$report" "origin could not be reached or read" "the report does not name the condition the probe actually found" + assert_not_contains "$report" "has no branch" "a remote that could not be read was reported as a deleted branch" + pass "a remote that cannot be read is reported as unreadable, not as a missing branch" +} + +test_missing_branch_on_a_readable_remote_is_still_reported() { + local home work out + # The other side of the case above: the remote answers, and it really does not + # have the watched branch, so that must still be reported as a missing branch. + home=$(make_home git-no-branch) + work=$(git_fixture git-no-branch-repo) + write_config "$home" "{\"tools\":[{\"name\":\"firstmate\",\"git\":{\"repo\":\"$work\",\"remote\":\"origin\",\"branch\":\"release\"}}]}" + out="$home/out.txt" + run_check "$home" "$PATH" "$out" + assert_contains "$(cat "$out")" "firstmate check failed: origin has no branch release" "a branch the remote does not have was not reported as missing" + pass "a branch a readable remote does not have is still reported as missing" +} + +test_git_probes_stop_when_the_sweep_budget_is_gone() { + local home work slow out report + # The git probes are the several-in-a-row case, two of them over the network, + # so they are the ones that can push a sweep past the watcher's own timeout and + # leave it killed with nothing printed at all. Here the tool's command probe + # spends the whole budget, so its git probes must not start: the sweep says + # which tool it did not finish instead of quietly running on. + home=$(make_home git-budget) + work=$(git_fixture git-budget-repo) + git -C "$work" reset -q --hard HEAD~2 + slow="$TMP_ROOT/git-budget/bin" + make_slow_copy "$slow" "$TOOL" 30 + write_config "$home" "{\"tools\":[{\"name\":\"firstmate\",\"command\":\"$TOOL\",\"git\":{\"repo\":\"$work\",\"remote\":\"origin\",\"branch\":\"main\"}}]}" + out="$home/out.txt" + run_check "$home" "$(fixture_path "$slow")" "$out" FM_TOOL_UPDATE_BUDGET_SECS=1 + report=$(cat "$out") + assert_contains "$report" "check incomplete: the time budget ran out before firstmate" "a sweep with no budget left did not say which tool it did not finish" + assert_not_contains "$report" "commits behind" "the git probes ran after the sweep budget was already gone" + pass "git probes stop and name their tool once the sweep budget is gone" +} + +test_a_git_probe_that_does_not_answer_is_not_an_update() { + local home work dir out report head_before + # The local git probes are bounded too, so a bound that is hit must not be read + # as the answer "this clone does not have that commit". This clone is ahead of + # its remote branch, which is silent when the probes answer, so any claim of an + # available update here was never established. + home=$(make_home git-mute) + work=$(git_fixture git-mute-repo) + printf 'local only\n' > "$work/f4" + git -C "$work" add f4 + git -C "$work" commit -qm four + head_before=$(git -C "$work" rev-parse HEAD) + + # A git that answers everything except the object query, which never answers. + dir="$TMP_ROOT/git-mute/bin" + mkdir -p "$dir" + cat > "$dir/git" <<SH +#!/usr/bin/env bash +for arg in "\$@"; do + if [ "\$arg" = cat-file ]; then + sleep 30 + exit 0 + fi +done +exec $(command -v git) "\$@" +SH + chmod 0755 "$dir/git" + + write_config "$home" "{\"tools\":[{\"name\":\"firstmate\",\"git\":{\"repo\":\"$work\",\"remote\":\"origin\",\"branch\":\"main\"}}]}" + out="$home/out.txt" + # The bound is wide enough that the earlier probes answer comfortably, so the + # only probe that can hit it is the object query the fixture stalls. Asserting + # that specific report keeps an unrelated timeout from passing this case for the + # wrong reason. + run_check "$home" "$(fixture_path "$dir")" "$out" FM_TOOL_UPDATE_PROBE_SECS=3 + report=$(cat "$out") + assert_not_contains "$report" "update available" "a probe that never answered was reported as an available update" + assert_contains "$report" "firstmate check failed: $work did not answer whether it already has" "the stalled object query was not the reported failure" + [ "$(git -C "$work" rev-parse HEAD)" = "$head_before" ] || fail "the check moved the watched repository's HEAD" + pass "a git probe that does not answer is reported as a failure, never as an update" +} + +test_a_stalled_repository_probe_is_not_reported_as_not_a_repository() { + local home work dir out report + # The very first git probe is bounded too. A clone on a stalled mount that + # never answers must be reported as not answering, not as not being a git + # repository, which is a diagnosis the probe never established. + home=$(make_home git-stall) + work=$(git_fixture git-stall-repo) + git -C "$work" reset -q --hard HEAD~2 + + dir="$TMP_ROOT/git-stall/bin" + mkdir -p "$dir" + cat > "$dir/git" <<SH +#!/usr/bin/env bash +for arg in "\$@"; do + if [ "\$arg" = rev-parse ]; then + sleep 30 + exit 0 + fi +done +exec $(command -v git) "\$@" +SH + chmod 0755 "$dir/git" + + write_config "$home" "{\"tools\":[{\"name\":\"firstmate\",\"git\":{\"repo\":\"$work\",\"remote\":\"origin\",\"branch\":\"main\"}}]}" + out="$home/out.txt" + run_check "$home" "$(fixture_path "$dir")" "$out" FM_TOOL_UPDATE_PROBE_SECS=1 + report=$(cat "$out") + assert_contains "$report" "firstmate check failed: $work did not answer whether it is a git repository" "a repository probe that never answered was not reported as such" + assert_not_contains "$report" "is not a git repository" "a repository probe that never answered was reported as not a repository" + pass "a stalled repository probe is reported as no answer, not as not a repository" +} + +# --- registry and reporting contract ---------------------------------------- + +test_absent_registry_is_silent() { + local home out + home=$(make_home no-config) + out="$home/out.txt" + run_check "$home" "$PATH" "$out" + [ ! -s "$out" ] || fail "check spoke without a watched tool registry: $(cat "$out")" + assert_absent "$home/state/.tool-updates" "check wrote a record without a registry" + pass "no watched tool registry means no output at all" +} + +test_malformed_registry_is_reported_not_ignored() { + local home out + home=$(make_home bad-config) + printf 'not json at all\n' > "$home/config/watched-tools.json" + out="$home/out.txt" + run_check "$home" "$PATH" "$out" + assert_contains "$(cat "$out")" "watched tool registry: the watched tool registry is not valid JSON" "a malformed registry was silently ignored" + + printf '%s\n' '{"tools":[{"name":"herdr"}]}' > "$home/config/watched-tools.json" + rm -f "$home/state/.tool-updates" + run_check "$home" "$PATH" "$out" + assert_contains "$(cat "$out")" "tool herdr needs command, git, or both" "a tool entry with no update source was accepted" + + printf '%s\n' '{"tools":[{"name":"herdr","command":"herdr; rm -rf /"}]}' > "$home/config/watched-tools.json" + rm -f "$home/state/.tool-updates" + run_check "$home" "$PATH" "$out" + assert_contains "$(cat "$out")" "command must be a bare executable name" "a command name with shell characters was accepted" + + printf '%s\n' '{"tools":[{"name":"herdr","command":"herdr","announce_args":["--help"]}]}' > "$home/config/watched-tools.json" + rm -f "$home/state/.tool-updates" + run_check "$home" "$PATH" "$out" + assert_contains "$(cat "$out")" "tool herdr announce_args needs announce_pattern" "a command to search with no pattern to search for was accepted" + pass "a malformed registry is reported instead of quietly skipped" +} + +test_findings_are_reported_once_until_they_change() { + local home stale fresh out path + home=$(make_home no-nag) + stale="$TMP_ROOT/no-nag/old/bin" + fresh="$TMP_ROOT/no-nag/new/bin" + make_copy "$stale" "$TOOL" 'herdr 0.8.0' + make_copy "$fresh" "$TOOL" 'herdr 0.8.2' + write_config "$home" "{\"tools\":[{\"name\":\"herdr\",\"command\":\"$TOOL\"}]}" + out="$home/out.txt" + path=$(fixture_path "$stale:$fresh") + + run_check "$home" "$path" "$out" + assert_contains "$(cat "$out")" "not in effect" "the first sweep did not report the pending update" + run_check "$home" "$path" "$out" + [ ! -s "$out" ] || fail "the same pending update was reported twice: $(cat "$out")" + + # A changed finding is news again. + make_copy "$fresh" "$TOOL" 'herdr 0.9.0' + run_check "$home" "$path" "$out" + assert_contains "$(cat "$out")" "0.9.0 is installed" "a changed finding was suppressed as a repeat" + + # Once the condition clears, the report clears with it, and a later return of + # the same condition is reported again. + make_copy "$stale" "$TOOL" 'herdr 0.9.0' + run_check "$home" "$path" "$out" + [ ! -s "$out" ] || fail "a cleared finding still produced a report: $(cat "$out")" + make_copy "$stale" "$TOOL" 'herdr 0.8.0' + run_check "$home" "$path" "$out" + assert_contains "$(cat "$out")" "PATH resolves 0.8.0" "a returning finding was not reported again" + pass "the same pending update is reported once, and a change is reported again" +} + +test_an_overlong_report_says_it_was_cut() { + local home out report i tools= + # Many watched tools can outgrow one line. The report must say it was cut + # rather than end mid-finding as if that were everything found. + home=$(make_home long) + for i in $(seq 1 30); do + [ -z "$tools" ] || tools="$tools," + tools="$tools{\"name\":\"absent-tool-$i\",\"command\":\"fm-absent-fixture-$i\"}" + done + write_config "$home" "{\"tools\":[$tools]}" + out="$home/out.txt" + run_check "$home" "$PATH" "$out" + report=$(cat "$out") + assert_contains "$report" "[truncated]" "an over-long report was cut without saying so" + [ "$(wc -l < "$out")" = 1 ] || fail "the cut report must still be exactly one line" + pass "an over-long report is cut with the shared truncation marker" +} + +test_a_finding_past_the_cut_is_still_reported() { + local home stale fresh out report i tools= + # Once a report is long enough to be cut, a new finding lands past the cut and + # leaves the printed line unchanged. It still has to count as news, or the PATH + # skew this check exists for would be suppressed for good on a busy home. + home=$(make_home past-cut) + stale="$TMP_ROOT/past-cut/mise/installs/herdr/latest/bin" + fresh="$TMP_ROOT/past-cut/local/bin" + make_copy "$stale" "$TOOL" 'herdr 0.8.0' + make_copy "$fresh" "$TOOL" 'herdr 0.8.2' + for i in $(seq 1 30); do + [ -z "$tools" ] || tools="$tools," + tools="$tools{\"name\":\"absent-tool-$i\",\"command\":\"fm-absent-fixture-$i\"}" + done + out="$home/out.txt" + write_config "$home" "{\"tools\":[$tools]}" + run_check "$home" "$(fixture_path "$stale:$fresh")" "$out" + assert_contains "$(cat "$out")" "[truncated]" "the first report was not long enough to be cut, so this case proves nothing" + + # The skew tool goes last, so its finding falls past the cut and the printed + # line is byte identical to the one the first sweep already recorded. + write_config "$home" "{\"tools\":[$tools,{\"name\":\"herdr\",\"command\":\"$TOOL\"}]}" + run_check "$home" "$(fixture_path "$stale:$fresh")" "$out" + report=$(cat "$out") + [ -n "$report" ] || fail "a finding past the cut produced no report at all, so it can never reach the watcher" + assert_contains "$report" "[truncated]" "the second report was not cut, so the finding was not past the cut" + pass "a finding that lands past the cut is still reported as news" +} + +test_probes_are_skipped_between_intervals() { + local home dir out status now + home=$(make_home cadence) + dir="$TMP_ROOT/cadence/bin" + make_copy "$dir" "$TOOL" 'herdr 0.8.2' + write_config "$home" "{\"tools\":[{\"name\":\"herdr\",\"command\":\"$TOOL\"}]}" + out="$home/out.txt" + now=1700000000 + + status=0 + FM_HOME="$home" PATH="$(fixture_path "$dir")" FM_CHECK_TIMEOUT=30 FM_TOOL_UPDATE_INTERVAL=900 FM_TOOL_UPDATE_NOW="$now" \ + "$CHECK" >"$out" 2>&1 || status=$? + expect_code 0 "$status" "first cadence run exit" + assert_grep 'fm-tool-updates-v1' "$home/state/.tool-updates" "the first run did not record its sweep" + + # A finding appears, but the interval has not elapsed, so no probe runs. + make_copy "$dir" "$TOOL" 'no version here' + status=0 + FM_HOME="$home" PATH="$(fixture_path "$dir")" FM_CHECK_TIMEOUT=30 FM_TOOL_UPDATE_INTERVAL=900 FM_TOOL_UPDATE_NOW="$((now + 300))" \ + "$CHECK" >"$out" 2>&1 || status=$? + expect_code 0 "$status" "gated cadence run exit" + [ ! -s "$out" ] || fail "a run inside the interval probed and spoke: $(cat "$out")" + + status=0 + FM_HOME="$home" PATH="$(fixture_path "$dir")" FM_CHECK_TIMEOUT=30 FM_TOOL_UPDATE_INTERVAL=900 FM_TOOL_UPDATE_NOW="$((now + 901))" \ + "$CHECK" >"$out" 2>&1 || status=$? + expect_code 0 "$status" "due cadence run exit" + assert_contains "$(cat "$out")" "did not report a version" "the run after the interval did not probe" + pass "probes run once per interval, not on every poll" +} + +test_an_oversized_budget_is_cut_to_fit_and_reported() { + local home stale fresh out report status + # A sweep budget larger than the watcher's own per check bound lets the watcher + # kill the run, which prints nothing and records nothing, so the same silence + # repeats on every poll. Cutting it keeps the detector alive, and the cut is + # reported so the operator can see the setting was not used as written. + home=$(make_home budget-cut) + stale="$TMP_ROOT/budget-cut/mise/installs/herdr/latest/bin" + fresh="$TMP_ROOT/budget-cut/local/bin" + make_copy "$stale" "$TOOL" 'herdr 0.8.0' + make_copy "$fresh" "$TOOL" 'herdr 0.8.2' + write_config "$home" "{\"tools\":[{\"name\":\"herdr\",\"command\":\"$TOOL\"}]}" + out="$home/out.txt" + status=0 + env FM_HOME="$home" PATH="$(fixture_path "$stale:$fresh")" FM_TOOL_UPDATE_INTERVAL=0 \ + FM_TOOL_UPDATE_BUDGET_SECS=60 FM_CHECK_TIMEOUT=30 "$CHECK" >"$out" 2>&1 || status=$? + expect_code 0 "$status" "oversized budget exit" + report=$(cat "$out") + # The cut leaves room for the whole-second rounding and the kill grace as well + # as one probe bound, so a cut sweep really does end before the watcher bound. + assert_contains "$report" "sweep budget 60s cut to 27s to stay inside the watcher check timeout of 30s" "a budget that cannot fit the watcher bound was not cut and reported" + assert_contains "$report" "herdr update not in effect" "the detector went quiet instead of sweeping with the cut budget" + + # The default budget of 20s fits the default bound, so it is used as written. + # The record is cleared first because the no-nag gate would otherwise suppress + # this run, whose bare skew line differs from the cut run's line above. + rm -f "$home/state/.tool-updates" + status=0 + env FM_HOME="$home" PATH="$(fixture_path "$stale:$fresh")" FM_TOOL_UPDATE_INTERVAL=0 \ + FM_CHECK_TIMEOUT=30 "$CHECK" >"$out" 2>&1 || status=$? + expect_code 0 "$status" "default budget exit" + report=$(cat "$out") + assert_not_contains "$report" "sweep budget" "the default budget was cut at the default watcher bound" + assert_contains "$report" "herdr update not in effect" "the sweep stopped reporting with the default budget" + + # A budget the watcher bound has room for is used as written. The cleared + # record keeps the no-nag gate from hiding this run's repeat of the same line. + rm -f "$home/state/.tool-updates" + status=0 + env FM_HOME="$home" PATH="$(fixture_path "$stale:$fresh")" FM_TOOL_UPDATE_INTERVAL=0 \ + FM_TOOL_UPDATE_BUDGET_SECS=60 FM_CHECK_TIMEOUT=120 "$CHECK" >"$out" 2>&1 || status=$? + expect_code 0 "$status" "fitting budget exit" + report=$(cat "$out") + assert_not_contains "$report" "sweep budget" "a budget that fits the watcher bound was cut anyway" + assert_contains "$report" "herdr update not in effect" "the sweep stopped reporting with a budget that fits" + pass "a budget that cannot fit the watcher bound is cut and reported, and the sweep keeps working" +} + +test_invalid_environment_and_action_refuse() { + local home status + home=$(make_home refuse) + status=0 + FM_HOME="$home" FM_TOOL_UPDATE_INTERVAL=5 "$CHECK" >/dev/null 2>&1 || status=$? + expect_code 2 "$status" "too-small interval exit" + status=0 + FM_HOME="$home" FM_TOOL_UPDATE_PROBE_SECS=0 "$CHECK" >/dev/null 2>&1 || status=$? + expect_code 2 "$status" "zero probe bound exit" + status=0 + FM_HOME="$home" FM_TOOL_UPDATE_BUDGET_SECS=999 "$CHECK" >/dev/null 2>&1 || status=$? + expect_code 2 "$status" "oversized budget exit" + status=0 + FM_HOME="$home" "$CHECK" sweep >/dev/null 2>&1 || status=$? + expect_code 2 "$status" "unknown action exit" + status=0 + FM_HOME="$home" "$CHECK" --help >/dev/null 2>&1 || status=$? + expect_code 0 "$status" "help exit" + pass "an out of range bound or unknown action refuses instead of guessing" +} + +# --- arming through the existing watcher contract ---------------------------- + +test_arm_registers_the_check_and_disarm_removes_it() { + local home dir status + home=$(make_home arm) + dir="$TMP_ROOT/arm/bin" + make_copy "$dir" "$TOOL" 'herdr 0.8.2' + status=0 + FM_HOME="$home" "$CHECK" arm >/dev/null 2>&1 || status=$? + expect_code 1 "$status" "arm without a registry exit" + assert_absent "$home/state/tool-updates.check.sh" "arm wrote a check shim without a registry" + + write_config "$home" "{\"tools\":[{\"name\":\"herdr\",\"command\":\"$TOOL\"}]}" + status=0 + FM_HOME="$home" "$CHECK" arm >/dev/null || status=$? + expect_code 0 "$status" "arm exit" + assert_present "$home/state/tool-updates.check.sh" "arm did not write the check shim" + assert_present "$home/state/tool-updates.check-trust" "arm did not register the check's bytes" + [ "$(stat -c %a "$home/state/tool-updates.check.sh" 2>/dev/null || stat -f %Lp "$home/state/tool-updates.check.sh")" = 700 ] \ + || fail "the check shim is not mode 700" + assert_grep 'fm-custom-check-v1' "$home/state/tool-updates.check-trust" "the trust binding has the wrong schema" + + # Arming twice must stay valid rather than invalidating its own binding. + FM_HOME="$home" "$CHECK" arm >/dev/null || fail "arming twice failed" + assert_grep 'fm-custom-check-v1' "$home/state/tool-updates.check-trust" "re-arming lost the trust binding" + + FM_HOME="$home" "$CHECK" disarm >/dev/null || fail "disarm failed" + assert_absent "$home/state/tool-updates.check.sh" "disarm left the check shim behind" + assert_absent "$home/state/tool-updates.check-trust" "disarm left the trust binding behind" + assert_absent "$home/state/.tool-updates" "disarm left the report record behind" + pass "arm registers a trusted check and disarm removes every trace" +} + +test_arm_refuses_a_symlink_at_the_shim_path() { + local home dir target mode status + # A stale or hostile symlink at the shim path must be refused rather than + # followed: following it would write the shim body into a file someone else + # owns and then make that file executable. + home=$(make_home arm-symlink) + dir="$TMP_ROOT/arm-symlink/bin" + make_copy "$dir" "$TOOL" 'herdr 0.8.2' + write_config "$home" "{\"tools\":[{\"name\":\"herdr\",\"command\":\"$TOOL\"}]}" + target="$TMP_ROOT/arm-symlink/not-the-shim.txt" + printf 'a file the shim must not touch\n' > "$target" + mode=$(stat -c %a "$target" 2>/dev/null || stat -f %Lp "$target") + ln -s "$target" "$home/state/tool-updates.check.sh" + + status=0 + FM_HOME="$home" "$CHECK" arm >/dev/null 2>&1 || status=$? + expect_code 1 "$status" "arm over a symlink exit" + [ "$(cat "$target")" = 'a file the shim must not touch' ] || fail "arm followed the symlink and overwrote its target" + [ "$(stat -c %a "$target" 2>/dev/null || stat -f %Lp "$target")" = "$mode" ] || fail "arm changed the mode of the symlink's target" + assert_absent "$home/state/tool-updates.check-trust" "arm registered a shim it refused to write" + pass "a symlink at the shim path is refused instead of followed" +} + +test_a_failed_registration_leaves_no_unregistered_shim() { + local home dir target stale_shim status + # An unregistered shim in state/ is not inert: the watcher rejects it every + # cycle and wakes firstmate about unauthenticated state checks until someone + # deletes it by hand. So a home that could not be armed has to come back to the + # state it was in, and arm still has to say it failed. + home=$(make_home arm-register-fail) + dir="$TMP_ROOT/arm-register-fail/bin" + make_copy "$dir" "$TOOL" 'herdr 0.8.2' + write_config "$home" "{\"tools\":[{\"name\":\"herdr\",\"command\":\"$TOOL\"}]}" + # A symlink at the trust path makes registration refuse, which is the shape any + # register failure has from arm's side. + target="$TMP_ROOT/arm-register-fail/not-the-trust.txt" + printf 'a file the trust binding must not touch\n' > "$target" + ln -s "$target" "$home/state/tool-updates.check-trust" + + status=0 + FM_HOME="$home" "$CHECK" arm >/dev/null 2>&1 || status=$? + expect_code 1 "$status" "arm with an unusable trust path exit" + assert_absent "$home/state/tool-updates.check.sh" "a failed registration left an unregistered check shim behind" + [ "$(cat "$target")" = 'a file the trust binding must not touch' ] || fail "arm wrote through the trust symlink" + + # A shim that was already there is only kept when its trust binding is still + # intact. Here the binding is unusable, so putting the old bytes back would + # leave exactly the unbound shim the watcher wakes about, and the home has to + # end plainly not armed instead. + stale_shim="$TMP_ROOT/arm-register-fail/shim-armed-earlier" + printf '#!/usr/bin/env bash\n# a shim armed earlier\nexit 0\n' > "$stale_shim" + cp "$stale_shim" "$home/state/tool-updates.check.sh" + chmod 0700 "$home/state/tool-updates.check.sh" + status=0 + FM_HOME="$home" "$CHECK" arm >/dev/null 2>&1 || status=$? + expect_code 1 "$status" "arm over an existing shim with an unusable trust path exit" + assert_absent "$home/state/tool-updates.check.sh" "a failed arm left a shim behind that no trust binding covers" + pass "a failed registration never leaves a shim without a matching trust binding" +} + +test_a_failed_rearm_leaves_no_shim_the_trust_binding_lost() { + local home dir fake status + # The register removes an existing trust binding when its own post-write check + # fails, so an arm that fails there would leave a home that WAS armed holding a + # shim with no binding, which the watcher rejects on every cycle. The home must + # end plainly not armed instead. + home=$(make_home arm-rearm-fail) + dir="$TMP_ROOT/arm-rearm-fail/bin" + make_copy "$dir" "$TOOL" 'herdr 0.8.2' + write_config "$home" "{\"tools\":[{\"name\":\"herdr\",\"command\":\"$TOOL\"}]}" + FM_HOME="$home" "$CHECK" arm >/dev/null || fail "the first arm failed" + assert_present "$home/state/tool-updates.check-trust" "the first arm did not bind the shim" + + # A hash tool that answers with nothing makes the register write a binding it + # then rejects, and it removes the old binding on the way out. + fake="$TMP_ROOT/arm-rearm-fail/fake-hash" + mkdir -p "$fake" + printf '#!/usr/bin/env bash\nexit 1\n' > "$fake/shasum" + printf '#!/usr/bin/env bash\nexit 1\n' > "$fake/sha256sum" + chmod 0755 "$fake/shasum" "$fake/sha256sum" + printf '#!/usr/bin/env bash\n# a shim armed earlier\nexit 0\n' > "$home/state/tool-updates.check.sh" + chmod 0700 "$home/state/tool-updates.check.sh" + + status=0 + env FM_HOME="$home" PATH="$(fixture_path "$fake")" "$CHECK" arm >/dev/null 2>&1 || status=$? + expect_code 1 "$status" "arm whose registration cannot hash exit" + assert_absent "$home/state/tool-updates.check.sh" "a failed re-arm left a shim behind after the trust binding was removed" + assert_absent "$home/state/tool-updates.check-trust" "the failed registration left a trust binding behind" + pass "a re-arm that loses the trust binding leaves no shim behind" +} + +test_arm_resolves_a_relative_home_into_the_shim() { + local home stale fresh out status + # The watcher runs the shim from its own working directory, so a relative home + # has to be resolved before it is persisted. Otherwise the shim reads whatever + # sits under the watcher's directory, finds no registry, and stays silent for + # good. + home=$(make_home arm-relative) + stale="$TMP_ROOT/arm-relative/mise/bin" + fresh="$TMP_ROOT/arm-relative/local/bin" + make_copy "$stale" "$TOOL" 'herdr 0.8.0' + make_copy "$fresh" "$TOOL" 'herdr 0.8.2' + write_config "$home" "{\"tools\":[{\"name\":\"herdr\",\"command\":\"$TOOL\"}]}" + + status=0 + (cd "$TMP_ROOT" && FM_HOME=arm-relative "$CHECK" arm >/dev/null 2>&1) || status=$? + expect_code 0 "$status" "arm with a relative home exit" + + out="$home/out.txt" + status=0 + (cd / && env -u FM_HOME PATH="$(fixture_path "$stale:$fresh")" FM_CHECK_TIMEOUT=30 FM_TOOL_UPDATE_INTERVAL=0 \ + "$home/state/tool-updates.check.sh" >"$out" 2>&1) || status=$? + expect_code 0 "$status" "shim run from another directory exit" + assert_contains "$(cat "$out")" "herdr update not in effect" "the shim read a different home than the one it was armed for" + pass "a relative home is resolved before it is persisted into the shim" +} + +test_armed_check_wakes_the_watcher_with_the_skew_report() { + local home stale fresh out err status + # End to end through the real watcher: the armed check must reach it as a + # `check:` wake carrying the same PATH skew line, with no new machinery. + home=$(make_home wake) + stale="$TMP_ROOT/wake/mise/installs/herdr/latest/bin" + fresh="$TMP_ROOT/wake/local/bin" + make_copy "$stale" "$TOOL" 'herdr 0.8.0' + make_copy "$fresh" "$TOOL" 'herdr 0.8.2' + write_config "$home" "{\"tools\":[{\"name\":\"herdr\",\"command\":\"$TOOL\"}]}" + printf '%s\n' fm-pr-check-migration-scan-v1 > "$home/state/.pr-check-migration-scan-v1" + printf '%s\n' fm-pr-check-migration-v1 > "$home/state/.pr-check-migration-v1" + chmod 0600 "$home/state/.pr-check-migration-scan-v1" "$home/state/.pr-check-migration-v1" + FM_HOME="$home" "$CHECK" arm >/dev/null || fail "could not arm the watched tool check" + + out="$home/out.txt" + err="$home/err.txt" + status=0 + env FM_HOME="$home" PATH="$(fixture_path "$stale:$fresh")" FM_CHECK_TIMEOUT=30 FM_TOOL_UPDATE_INTERVAL=0 \ + FM_POLL=1 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=1 \ + "$CHECKPOINT" --seconds 10 >"$out" 2>"$err" || status=$? + expect_code 0 "$status" "watcher checkpoint exit" + assert_contains "$(cat "$out")" "check:" "the armed check did not reach the watcher as a check wake" + assert_contains "$(cat "$out")" "tool updates: herdr update not in effect" "the wake did not carry the PATH skew report" + pass "the armed check reaches the watcher as an ordinary check wake" +} + +test_path_skew_is_reported_from_every_copy +test_newest_copy_first_on_path_is_silent +test_identical_versions_are_silent +test_one_copy_reached_twice_is_probed_once +test_unreadable_version_is_a_failure_not_a_pass +test_missing_command_is_reported +test_announced_update_is_reported_from_the_tool_itself +test_announcement_is_read_from_a_second_command +test_unusable_announce_pattern_is_reported_not_read_as_silence +test_one_broken_pattern_does_not_blind_the_rest_of_the_sweep +test_an_unchecked_announcement_source_is_not_read_as_current +test_an_announcement_probe_that_does_not_answer_is_reported +test_quiet_tool_with_announce_pattern_is_silent +test_commits_behind_origin_are_reported +test_default_branch_is_detected_when_branch_is_omitted +test_default_branch_is_asked_of_the_remote_when_the_clone_has_no_record +test_current_and_ahead_repositories_are_silent +test_unusable_git_source_is_reported +test_unreadable_remote_is_not_reported_as_a_missing_branch +test_missing_branch_on_a_readable_remote_is_still_reported +test_git_probes_stop_when_the_sweep_budget_is_gone +test_a_git_probe_that_does_not_answer_is_not_an_update +test_a_stalled_repository_probe_is_not_reported_as_not_a_repository +test_absent_registry_is_silent +test_malformed_registry_is_reported_not_ignored +test_findings_are_reported_once_until_they_change +test_an_overlong_report_says_it_was_cut +test_a_finding_past_the_cut_is_still_reported +test_probes_are_skipped_between_intervals +test_an_oversized_budget_is_cut_to_fit_and_reported +test_invalid_environment_and_action_refuse +test_arm_registers_the_check_and_disarm_removes_it +test_arm_refuses_a_symlink_at_the_shim_path +test_a_failed_registration_leaves_no_unregistered_shim +test_a_failed_rearm_leaves_no_shim_the_trust_binding_lost +test_arm_resolves_a_relative_home_into_the_shim +test_armed_check_wakes_the_watcher_with_the_skew_report diff --git a/tests/fm-turnend-guard.test.sh b/tests/fm-turnend-guard.test.sh index ac02c7c37ce..5c21f4d4306 100755 --- a/tests/fm-turnend-guard.test.sh +++ b/tests/fm-turnend-guard.test.sh @@ -1299,6 +1299,83 @@ test_hook_claude_mode_allows_on_fresh_rewake_epoch() { pass "fm-turnend-guard --claude: fresh rewake epoch prevents a duplicate continuation for the same event" } +# The 2026-08-14 lapse: a cycle armed, delivered one rewake, exited, and left its +# owner lock behind holding a live pid. Both Stop participants read that lock as +# "recovery is already under way", so with work in flight and a beacon 40 minutes +# cold every turn ended blind and nothing re-armed. A stale ledger outcome for +# the lock's own pid is the proof that no decision is in flight any more. +test_hook_claude_mode_blocks_on_abandoned_autoarm_claim() { + local dir out status pid + dir=$(make_primary_dir "$TMP_ROOT/hook-claude-abandoned-claim") + : > "$dir/state/task1.meta" + : > "$dir/state/task2.meta" + sleep 60 & + pid=$! + record_autoarm_owner "$dir" "$pid" + printf 'epoch=464 owner_pid=%s outcome=rewake updated_at=1\n' "$pid" > "$dir/state/.claude-autoarm-epoch" + touch -t 202001010000 "$dir/state/.claude-autoarm-epoch" + out=$(FM_CLAUDE_AUTOARM_SYNC_WAIT_MS=200 run_hook_claude "$dir" true); status=$? + kill "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + expect_code 2 "$status" "an owner lock left behind by a finished claim must not pass for recovery under way" + assert_contains "$out" "TURN WOULD END BLIND" "abandoned-claim block must carry the blind-turn banner" + assert_contains "$out" "2 task(s) in flight" "abandoned-claim block must name the unsupervised work" + pass "fm-turnend-guard --claude: an abandoned auto-arm claim no longer allows a blind stop (incident regression)" +} + +# The ledger-blind variant of the same lapse: a session teardown killed the claim's +# process group before it could record any outcome, so its entry still reads +# "arming" - in flight however old, by contract - while the recorded pid now belongs +# to an unrelated live process. The identity the claim wrote into its own lock is +# the only thing that separates that from a real arm still running. +test_hook_claude_mode_blocks_on_pid_reused_arming_claim() { + local dir out status pid identity + dir=$(make_primary_dir "$TMP_ROOT/hook-claude-reused-pid-claim") + : > "$dir/state/task1.meta" + : > "$dir/state/task2.meta" + sleep 60 & + pid=$! + record_autoarm_owner "$dir" "$pid" + # The claim recorded ITS OWN identity; this test shell now stands in for the + # unrelated process that inherited the number. + identity=$(fm_test_pid_identity "$$") || fail "could not compute a claim pid-identity" + printf '%s\n' "$identity" > "$dir/state/.claude-autoarm.lock/pid-identity" + printf 'epoch=464 owner_pid=%s outcome=arming updated_at=1\n' "$pid" > "$dir/state/.claude-autoarm-epoch" + touch -t 202001010000 "$dir/state/.claude-autoarm-epoch" + out=$(FM_CLAUDE_AUTOARM_SYNC_WAIT_MS=200 run_hook_claude "$dir" true); status=$? + kill "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + expect_code 2 "$status" "a claim whose recorded identity no longer matches its live pid must not pass for recovery under way" + assert_contains "$out" "TURN WOULD END BLIND" "reused-pid claim block must carry the blind-turn banner" + assert_contains "$out" "2 task(s) in flight" "reused-pid claim block must name the unsupervised work" + pass "fm-turnend-guard --claude: a claim whose pid was reused stops counting as recovery even while its entry reads arming" +} + +# The same abandoned claim on the terminal path: stepping aside for it allowed the +# stop silently AND spent no attended alarm, so a genuinely broken automatic +# mechanism stayed invisible. The guard must clear the claim and finish instead. +test_hook_claude_mode_terminal_fail_open_clears_abandoned_claim() { + local dir out status pid + dir=$(make_primary_dir "$TMP_ROOT/hook-claude-abandoned-terminal") + : > "$dir/state/task1.meta" + : > "$dir/state/.claude-autoarm-failure-notified" + seed_claude_budget "$dir" 4 3 + sleep 60 & + pid=$! + record_autoarm_owner "$dir" "$pid" + printf 'epoch=3 owner_pid=%s outcome=failed-suppressed updated_at=1\n' "$pid" > "$dir/state/.claude-autoarm-epoch" + touch -t 202001010000 "$dir/state/.claude-autoarm-epoch" + out=$(FM_CLAUDE_AUTOARM_SYNC_WAIT_MS=200 run_hook_claude "$dir" true); status=$? + kill "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + expect_code 0 "$status" "the verified attended fail-open still ends the turn once it is spent" + assert_contains "$out" 'FIRSTMATE SUPERVISION IS GENUINELY DOWN' "an abandoned claim suppressed the episode's attended alarm" + assert_present "$dir/state/.claude-autoarm-failure-alarmed" "abandoned-claim terminal path did not consume the one-time alarm" + assert_absent "$dir/state/.claude-autoarm.lock" "abandoned-claim terminal path left the stale claim in place" + assert_absent "$dir/state/.claude-autoarm.lock.steal" "abandoned-claim reclaim left its serialization mutex behind" + pass "fm-turnend-guard --claude: the terminal path clears an abandoned claim instead of stepping aside silently" +} + test_hook_claude_mode_preserves_fresh_failed_progression() { local dir out status count dir=$(make_primary_dir "$TMP_ROOT/hook-claude-failed-epoch") @@ -1653,6 +1730,9 @@ test_hook_claude_mode_allows_when_autoarm_owner_alive test_hook_claude_mode_repeated_failed_to_arming_interleavings_reach_fail_open test_hook_claude_mode_terminal_boundary_excludes_starting_owner test_hook_claude_mode_allows_on_fresh_rewake_epoch +test_hook_claude_mode_blocks_on_abandoned_autoarm_claim +test_hook_claude_mode_blocks_on_pid_reused_arming_claim +test_hook_claude_mode_terminal_fail_open_clears_abandoned_claim test_hook_claude_mode_preserves_fresh_failed_progression test_hook_claude_mode_integrated_monotonic_fail_open test_hook_claude_mode_recovery_contention_is_not_ordinary_allow diff --git a/tests/fm-voice-relay.test.sh b/tests/fm-voice-relay.test.sh new file mode 100755 index 00000000000..f9c57543aed --- /dev/null +++ b/tests/fm-voice-relay.test.sh @@ -0,0 +1,4741 @@ +#!/usr/bin/env bash +# tests/fm-voice-relay.test.sh - the spoken interface's wire format, read scope and handover. +# +# Every case here runs offline. The three things worth protecting in this feature +# are all offline properties: the frame format the laptop and the desktop agree +# on, WHAT a status answer is allowed to contain, and the fact that real work is +# handed to firstmate rather than done by the voice agent. The latency work that +# motivated the build is a measurement, not an assertion, so it is not here; the +# numbers and the method live in docs/voice-relay.md. +# +# THE CASE THAT MATTERS MOST is the confidentiality boundary. The captain granted +# the voice agent full read access to their records, so the reader defaults to the +# wider scope. What keeps that safe is structural: finished work and free-form +# note bodies are never assembled at all, and those are exactly where commercial +# detail accumulates. This suite plants a marker in both places and fails if it +# ever reaches an answer, so widening the reader later breaks a test instead of +# quietly widening what is sent to a model in another region. +# +# The markers below are invented for this fixture. Real customer names are not +# committed to a test file. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +command -v python3 >/dev/null 2>&1 || { echo "skip: python3 not found"; exit 0; } + +TMP_ROOT=$(fm_test_tmproot fm-voice-relay) +HOME_FIXTURE="$TMP_ROOT/home" + +# NEVER_TOKEN sits in finished work and in a note body: both are excluded by +# construction, so it must never appear at any scope. +NEVER_TOKEN=NEVERLEAVESTHISHOST +# DENY_TOKEN sits in the title of open in-flight work, which the wide scope does +# report. It proves the deny list suppresses something that genuinely would have +# been sent, rather than passing vacuously against text no answer contains. +DENY_TOKEN=DENYMEPLEASE + +seed_home() { + mkdir -p "$HOME_FIXTURE/data" "$HOME_FIXTURE/state" "$HOME_FIXTURE/config" + cat > "$HOME_FIXTURE/data/backlog.md" <<EOF +# Backlog + +## In flight +- [ ] alpha-one - Fix the sign-in redirect (repo: alpha) (kind: ship) (priority: 0) (since 2026-08-01) + Long note body written for someone with the whole file open, mentioning + $NEVER_TOKEN and the rate we agreed. +- [ ] beta-two - Decide the storage shape (repo: beta) (kind: captain) (priority: 1) +- [ ] gamma-three - Migrate the $DENY_TOKEN account onto the new plan (repo: gamma) (kind: ship) + +## Queued +- [ ] delta-four - Add the retry (repo: delta) (kind: ship) (hold-kind: captain) (hold: waiting on the captain) +- [ ] epsilon-five - Tidy the logs (repo: epsilon) (kind: ship) + +## Done +- [x] old-six - Shipped the $NEVER_TOKEN integration (repo: alpha) (done 2026-07-01) +# An unticked line under Done, held for the captain. Two separate mechanisms keep +# finished work out of an answer: the section is never parsed, and a ticked box is +# dropped. A ticked line is blocked by both, so it cannot tell which one broke. +# This line is blocked by the section rule alone, and the list of what waits on +# the captain is assembled with no section filter at all, so it is the one place +# where losing that rule would put finished work into a spoken answer. +- [ ] old-seven - Decide the $NEVER_TOKEN renewal (repo: alpha) (kind: captain) +EOF + + fm_write_meta "$HOME_FIXTURE/state/alpha-one.meta" \ + kind=ship mode=no-mistakes window=firstmate:fm-alpha-one \ + pr=https://github.com/example/alpha/pull/7 + fm_write_meta "$HOME_FIXTURE/state/gamma-three.meta" kind=ship mode=direct-PR + printf 'working: reading the failing test\n' > "$HOME_FIXTURE/state/alpha-one.status" + # The bracketed shape, which is what bin/fm-secondmate-report.sh writes and + # what a keyed decision line looks like. Status metadata sits between the verb + # and the colon, so a reader that only cuts at the colon reads no verb here. + printf 'blocked [key=api-shape]: needs a credential (via-helper)\n' \ + > "$HOME_FIXTURE/state/gamma-three.status" +} + +records_status() { + python3 "$ROOT/bin/fm_voice_records.py" status --home "$HOME_FIXTURE" "$@" +} + +seed_home + +# --- the wire format -------------------------------------------------------- +# +# A desynchronised stream must be a loud error rather than audio interpreted as +# a frame header. The laptop copy of this module is the only other place these +# rules exist, so they are pinned here. + +python3 - "$ROOT/bin" <<'PY' || fail "frame round trip" +import os, io, sys +sys.path.insert(0, sys.argv[1]) +import fm_voice_frame as frame + +def check(cond, label): + if not cond: + sys.exit("frame: " + label) + +# Round trip of every kind, including an empty payload and a large one. +buf = io.BytesIO() +w = frame.Writer(buf) +w.send(frame.TALK_START) +w.send(frame.AUDIO, b"\x01\x02" * 1600) +w.send_json(frame.TEXT, {"role": "USER", "text": "how is the fleet"}) +w.send(frame.TALK_END) +buf.seek(0) +r = frame.Reader(buf) +got = [] +while True: + item = r.read() + if item is None: + break + got.append(item) +check([k for k, _ in got] == [frame.TALK_START, frame.AUDIO, frame.TEXT, + frame.TALK_END], "kinds did not round trip") +check(got[1][1] == b"\x01\x02" * 1600, "audio payload did not round trip") +check(frame.decode_json(got[2][1])["text"] == "how is the fleet", + "json payload did not round trip") + +# A clean close between frames is end of input, not an error. +check(frame.Reader(io.BytesIO(b"")).read() is None, "clean EOF should be None") + +# A stream cut inside a payload is a dropped connection and must say so. +try: + frame.Reader(io.BytesIO(frame.encode(frame.AUDIO, b"12345")[:-2])).read() + sys.exit("frame: truncated payload was accepted") +except frame.FrameError: + pass + +# A payload that never starts at all is the same fault, one byte earlier. +try: + frame.Reader(io.BytesIO(frame.HEADER.pack(frame.AUDIO, 5))).read() + sys.exit("frame: a header with no payload behind it was accepted") +except frame.FrameError: + pass + +# A stream cut inside the HEADER is a dropped connection too, and must NOT come +# back as the clean close checked above. A lost SSH connection does not politely +# end on a frame boundary, and a partial header read as end of input records the +# turn as unanswered with no error, which puts a transport failure into a results +# file as an ordinary turn the model did not answer. +for cut in range(1, frame.HEADER.size): + try: + frame.Reader(io.BytesIO(frame.encode(frame.BYE)[:cut])).read() + sys.exit("frame: %d header bytes then EOF was read as a clean close" % cut) + except frame.FrameError as exc: + check("header" in str(exc), + "a truncated header should name itself: %s" % exc) + +# Audio bytes that happen to look like a header must not be trusted. +for bad in (b"\xffZZZZ", frame.HEADER.pack(frame.AUDIO, frame.MAX_PAYLOAD + 1)): + try: + frame.Reader(io.BytesIO(bad)).read() + sys.exit("frame: accepted a bad header: %r" % bad) + except frame.FrameError: + pass + +try: + frame.encode(b"?") + sys.exit("frame: encoded an unknown kind") +except frame.FrameError: + pass + +try: + frame.encode(frame.AUDIO, b"x" * (frame.MAX_PAYLOAD + 1)) + sys.exit("frame: encoded an oversized payload") +except frame.FrameError: + pass + +# The relay's uplink decodes headers itself, on an asynchronous stream Reader +# cannot drive, and calls this to decide whether to read the payload at all. A +# bogus length has to be refused BEFORE the read, or the relay waits for up to +# four gigabytes while the captain waits for an answer. +for kind, length in ((b"\xff", 0), (frame.AUDIO, frame.MAX_PAYLOAD + 1)): + try: + frame.check_header(kind, length) + sys.exit("frame: check_header accepted %r/%d" % (kind, length)) + except frame.FrameError: + pass +frame.check_header(frame.AUDIO, frame.MAX_PAYLOAD) +PY +pass "wire format round trips and rejects a desynchronised stream" + +# --- the relay's uplink ------------------------------------------------------ +# +# The relay decodes the captain's frames on an asyncio stream, which frame.Reader +# cannot drive, so the rule above has to be exercised on that path as well. The +# failure mode it prevents is not a wrong answer, it is no answer: a relay that +# reads the payload before it checks the length waits inside readexactly for up +# to four gigabytes that will never arrive, while the captain sits in front of a +# client that never replies. The timeout below is what tells those two apart. + +python3 - "$ROOT/bin" <<'PY' || fail "relay uplink" +import asyncio, sys +sys.path.insert(0, sys.argv[1]) +import importlib.util, pathlib +spec = importlib.util.spec_from_file_location( + "relay", str(pathlib.Path(sys.argv[1]) / "fm-voice-relay.py")) +relay = importlib.util.module_from_spec(spec) +spec.loader.exec_module(relay) +import fm_voice_frame as frame + +def check(cond, label): + if not cond: + sys.exit("uplink: " + label) + +async def read(raw): + reader = asyncio.StreamReader() + reader.feed_data(raw) + return await asyncio.wait_for(relay.read_uplink_frame(reader), timeout=5) + +audio = b"\x01\x02" * 8 +check(asyncio.run(read(frame.encode(frame.AUDIO, audio))) == (frame.AUDIO, audio), + "a frame carrying audio did not survive the uplink") +check(asyncio.run(read(frame.encode(frame.TALK_END))) == (frame.TALK_END, b""), + "an empty control frame did not survive the uplink") + +# A header with nothing behind it. Refused on the header, this raises at once; +# read first and checked later, it hangs, so a timeout here is the regression. +for bad in (frame.HEADER.pack(frame.AUDIO, frame.MAX_PAYLOAD + 1), + b"\xff\x00\x00\x10\x00"): + try: + asyncio.run(read(bad)) + sys.exit("uplink: accepted a bad header: %r" % bad) + except frame.FrameError: + pass + except (asyncio.TimeoutError, TimeoutError): + sys.exit("uplink: waited for the payload of a bad header instead of " + "refusing it: %r" % bad) +PY +pass "the relay refuses a desynchronised uplink header instead of waiting for its payload" + +# --- whose account, whose model --------------------------------------------- +# +# A region, a model id and an AWS profile name somebody's account and somebody's +# choices, so nothing here ships one. A home that has configured none of them +# cannot start the relay at all, and it is told which file to write rather than +# quietly reaching an API in somebody else's account. That configuration IS the +# opt-in, so this case is what keeps the feature off by default. + +CONFIG_HOME="$TMP_ROOT/unconfigured" +mkdir -p "$CONFIG_HOME/config" + +python3 - "$ROOT/bin" "$CONFIG_HOME" <<'PY' || fail "relay configuration" +import sys +sys.path.insert(0, sys.argv[1]) +import importlib.util, os, pathlib +spec = importlib.util.spec_from_file_location( + "relay", str(pathlib.Path(sys.argv[1]) / "fm-voice-relay.py")) +relay = importlib.util.module_from_spec(spec) +spec.loader.exec_module(relay) +import fm_voice_records as records + +home = sys.argv[2] +for name in ("FM_VOICE_REGION", "FM_VOICE_MODEL", "FM_VOICE_PROFILE", "FM_VOICE_ID", + "FM_CONFIG_OVERRIDE"): + os.environ.pop(name, None) + +def check(cond, label): + if not cond: + sys.exit("configuration: " + label) + +# An unconfigured home refuses, and the refusal is the path to write. +try: + relay.resolve_settings(relay.parse_args(["--serve", "--home", home])) + sys.exit("configuration: an unconfigured home started the relay") +except records.RecordError as exc: + check("voice-region" in str(exc), + "the refusal should name the file to write: %s" % exc) + check(home in str(exc), "and it should be this home's path: %s" % exc) + +# One file at a time: the region alone is not enough to reach a model. +with open(os.path.join(home, "config", "voice-region"), "w") as handle: + handle.write("# the region this home talks to\neu-somewhere-1\n") +try: + relay.resolve_settings(relay.parse_args(["--serve", "--home", home])) + sys.exit("configuration: a home with no model id started the relay") +except records.RecordError as exc: + check("voice-model" in str(exc), + "the refusal should name the missing model file: %s" % exc) + +with open(os.path.join(home, "config", "voice-model"), "w") as handle: + handle.write("some.model-v1:0\n") +options = relay.resolve_settings(relay.parse_args(["--serve", "--home", home])) +check(options.region == "eu-somewhere-1", + "the configured region should be used, comment and all: %r" % options.region) +check(options.model == "some.model-v1:0", + "the configured model should be used: %r" % options.model) +# No profile configured means ambient credentials only, which is a real choice +# rather than a missing one, so it is not a refusal. +check(options.profile == "", "an absent profile should stay empty: %r" % options.profile) +check(options.voice == relay.VOICE, + "an absent voice should fall back to the shipped one: %r" % options.voice) + +# The environment overrides a file for a single run. +os.environ["FM_VOICE_REGION"] = "eu-elsewhere-2" +os.environ["FM_VOICE_ID"] = "amy" +options = relay.resolve_settings(relay.parse_args(["--serve", "--home", home])) +check(options.region == "eu-elsewhere-2", + "the environment should override the file: %r" % options.region) +check(options.voice == "amy", "the voice should be overridable: %r" % options.voice) + +# And an explicit flag overrides both. +options = relay.resolve_settings( + relay.parse_args(["--serve", "--home", home, "--region", "eu-flag-3"])) +check(options.region == "eu-flag-3", "a flag should win: %r" % options.region) + +os.environ.pop("FM_VOICE_REGION", None) +os.environ.pop("FM_VOICE_ID", None) + +# THE PROFILE IS READ BY PRESENCE, NOT BY TRUTHINESS. An empty FM_VOICE_PROFILE is +# the captain saying "use the credentials I already have", so it must not fall +# through to a configured profile and spend a second exporting from an account +# they just opted out of. fm-inbox.sh reads its own equivalent that way and +# docs/configuration.md promises it for both. +with open(os.path.join(home, "config", "voice-profile"), "w") as handle: + handle.write("a-configured-profile\n") +options = relay.resolve_settings(relay.parse_args(["--serve", "--home", home])) +check(options.profile == "a-configured-profile", + "a configured profile should be used: %r" % options.profile) + +os.environ["FM_VOICE_PROFILE"] = "" +options = relay.resolve_settings(relay.parse_args(["--serve", "--home", home])) +check(options.profile == "", + "an empty FM_VOICE_PROFILE must force ambient credentials: %r" % options.profile) + +os.environ["FM_VOICE_PROFILE"] = "another-profile" +options = relay.resolve_settings(relay.parse_args(["--serve", "--home", home])) +check(options.profile == "another-profile", + "a set FM_VOICE_PROFILE should override the file: %r" % options.profile) +os.environ.pop("FM_VOICE_PROFILE", None) + +# An empty region, by contrast, is not a choice about anything, so it still falls +# through to the file rather than refusing. +os.environ["FM_VOICE_REGION"] = "" +options = relay.resolve_settings(relay.parse_args(["--serve", "--home", home])) +check(options.region == "eu-somewhere-1", + "an empty region variable should fall through to the file: %r" % options.region) +os.environ.pop("FM_VOICE_REGION", None) + +# --help must work in a home that has configured nothing, or the captain cannot +# read how to configure it. +PY +pass "the relay reads whose account to use from this home and refuses to guess" + +set +e +help_out=$(python3 "$ROOT/bin/fm-voice-relay.py" --help 2>&1) +help_code=$? +set -e +expect_code 0 "$help_code" "--help must work with no configuration: $help_out" +assert_contains "$help_out" 'voice-region' \ + "--help should name the files a home has to write" +pass "an unconfigured home can still read how to configure the relay" + +# The captain inbox is the same rule with a different consequence: note, status, +# list and drain make no model call, so they must keep working unconfigured. The +# voice handover depends on note, so that is not a nicety. +# +# EVERY FM_INBOX_ VARIABLE IS NEUTRALIZED HERE, at the harness rather than in each +# case, and the list is read out of the environment rather than written down, so a +# knob added later cannot quietly survive into a refusal case. A shell that +# exports a region and a model id would otherwise walk these cases straight past +# the refusal they assert and into a real model call: an offline suite that can +# spend the operator's credentials is worse than a failing one. +inbox_env=() +while IFS= read -r inbox_knob; do + [ -n "$inbox_knob" ] || continue + inbox_env+=(-u "$inbox_knob") +done < <(env | sed -n 's/^\(FM_INBOX_[A-Za-z0-9_]*\)=.*/\1/p' | sort -u) +inbox_env+=(FM_HOME="$CONFIG_HOME" FM_STATE_OVERRIDE="$CONFIG_HOME/state" + FM_CONFIG_OVERRIDE="$CONFIG_HOME/config") + +# And a stub that records any attempt, so "no model call" is a checked fact rather +# than a claim about control flow. The real aws would need credentials; this one +# leaves evidence and exits non-zero. +INBOX_FAKEBIN=$(fm_fakebin "$TMP_ROOT/inbox-fake") +AWS_CALLED="$TMP_ROOT/aws-was-called" +cat > "$INBOX_FAKEBIN/aws" <<SH +#!/usr/bin/env bash +printf 'aws %s\n' "\$*" >> "$AWS_CALLED" +exit 9 +SH +chmod +x "$INBOX_FAKEBIN/aws" +inbox_env+=(PATH="$INBOX_FAKEBIN:$PATH") + +set +e +ask_out=$(env "${inbox_env[@]}" "$ROOT/bin/fm-inbox.sh" ask "how is the fleet" 2>&1) +ask_code=$? +set -e +[ "$ask_code" -ne 0 ] || fail "ask ran with nothing configured" +assert_contains "$ask_out" 'inbox-region' \ + "the first refusal should name the region file: $ask_out" + +# One file at a time, so each refusal names one thing to do. +printf 'eu-somewhere-1\n' > "$CONFIG_HOME/config/inbox-region" +set +e +ask_out=$(env "${inbox_env[@]}" "$ROOT/bin/fm-inbox.sh" ask "how is the fleet" 2>&1) +ask_code=$? +set -e +[ "$ask_code" -ne 0 ] || fail "ask ran without a configured model" +assert_contains "$ask_out" 'inbox-ask-model' \ + "the refusal should name the model file to write: $ask_out" + +set +e +say_out=$(printf '' | env "${inbox_env[@]}" "$ROOT/bin/fm-inbox.sh" say 2>&1) +say_code=$? +set -e +[ "$say_code" -ne 0 ] || fail "say ran without a configured model" +assert_contains "$say_out" 'inbox-stt-model' \ + "the refusal should name the model file to write: $say_out" + +rm -f "$CONFIG_HOME/config/inbox-region" +unconfigured_note=$(env "${inbox_env[@]}" \ + "$ROOT/bin/fm-inbox.sh" note "the handover must work with no configuration") \ + || fail "note should not need any configuration" +assert_contains "$unconfigured_note" 'queued ' "note should still queue a record" +assert_absent "$AWS_CALLED" \ + "no case above may reach a model: the aws stub recorded an attempt" +pass "the model-backed subcommands refuse by name while note keeps working" + +# --help prints the whole header block, and finds where that block ends rather +# than counting lines to it, so growing the header cannot silently truncate the +# help again. The PRIVACY paragraph is the part that matters: it is the only place +# a new operator is told which subcommands send audio or text off this host, and a +# fixed line range had already dropped it. +inbox_help=$("$ROOT/bin/fm-inbox.sh" --help) || fail "fm-inbox.sh --help failed" +assert_contains "$inbox_help" 'PRIVACY:' \ + "the help must say which subcommands send anything to a model" +assert_contains "$inbox_help" 'make no network call at all' \ + "the help must name the subcommands that stay on this host" +assert_contains "$inbox_help" 'FM_HOME' \ + "the help must keep its environment section" +assert_contains "$inbox_help" 'inbox-ask-model' \ + "the help must name the files a home has to write" +assert_contains "$inbox_help" 'fm-inbox.sh note' \ + "the help must still open with the usage it always had" +pass "fm-inbox.sh --help prints its whole header, privacy paragraph included" + +# --- the tool surface the two sides share ----------------------------------- +# +# The relay declares the tools and fm_voice_records implements them. Renaming one +# side only would leave the agent unable to answer or unable to hand over, and +# the failure would look like a confused model rather than a typo. + +python3 - "$ROOT/bin" <<'PY' || fail "tool surface" +import sys +sys.path.insert(0, sys.argv[1]) +import importlib.util, pathlib +spec = importlib.util.spec_from_file_location( + "relay", str(pathlib.Path(sys.argv[1]) / "fm-voice-relay.py")) +relay = importlib.util.module_from_spec(spec) +spec.loader.exec_module(relay) + +names = sorted(t["toolSpec"]["name"] for t in relay.TOOLS["tools"]) +if names != ["get_fleet_status", "hand_over_to_firstmate"]: + sys.exit("relay declares unexpected tools: %s" % names) + +# The handover tool has to take the request text, or the agent can announce a +# handover it never performed. +handover = [t["toolSpec"] for t in relay.TOOLS["tools"] + if t["toolSpec"]["name"] == "hand_over_to_firstmate"][0] +import json +schema = json.loads(handover["inputSchema"]["json"]) +if schema.get("required") != ["request"]: + sys.exit("hand_over_to_firstmate must require the request text") + +# Push to talk is the default for this build and is meant to be one setting. +options = relay.parse_args(["--self-test", "x.pcm"]) +if options.tail_ms <= 0: + sys.exit("the trailing silence default must be positive; 0 is never answered") +PY +pass "the relay and the reader agree on the tool names and the handover argument" + +# --- credentials ------------------------------------------------------------- +# +# The relay rebuilds the model session on every turn, on purpose. Resolving AWS +# credentials belongs to the relay's start rather than to that rebuild: the +# sandbox profile's credential_process costs about a second, and a second spent +# there is a second added to the delay this whole build exists to keep honest. +# Nothing here talks to AWS; the resolver is replaced with a counter. + +python3 - "$ROOT/bin" <<'PY' || fail "credential reuse" +import asyncio, datetime, os, sys, time +sys.path.insert(0, sys.argv[1]) +import importlib.util, pathlib +spec = importlib.util.spec_from_file_location( + "relay", str(pathlib.Path(sys.argv[1]) / "fm-voice-relay.py")) +relay = importlib.util.module_from_spec(spec) +spec.loader.exec_module(relay) + +def check(cond, label): + if not cond: + sys.exit("credentials: " + label) + +AWS_VARS = ("AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN", + "AWS_CREDENTIAL_EXPIRATION") + +def iso(at): + return datetime.datetime.fromtimestamp(at, datetime.timezone.utc).isoformat() + +# Credentials taken from the environment cannot be refreshed in place, because +# os.environ never gets fresher values while this process runs. So they are only +# preferred while they are usable, and what decides that is the deadline the +# exporter states beside the keys. Treating one as eternal strands a long-lived +# relay: every session after the real deadline is rejected for an expired token. +for name in AWS_VARS: + os.environ.pop(name, None) + +check(relay.ambient_credentials() is None, + "an environment with no keys must send the relay to the profile") + +os.environ["AWS_ACCESS_KEY_ID"] = "AKIAEXAMPLE" +check(relay.ambient_credentials() is None, + "a key id with no secret beside it must be refused, not indexed blindly") + +os.environ["AWS_SECRET_ACCESS_KEY"] = "s3cret" +ambient = relay.ambient_credentials() +check(ambient[0]["aws_access_key_id"] == "AKIAEXAMPLE", + "a complete environment should be used: %r" % (ambient,)) +check(ambient[1] is None, + "long-term keys, with no session token and no stated deadline, do not expire") + +os.environ["AWS_SESSION_TOKEN"] = "t0ken" +check(relay.ambient_credentials()[1] is relay.EXPIRY_UNKNOWN, + "a session token has a deadline whether or not the shell stated it") + +os.environ["AWS_CREDENTIAL_EXPIRATION"] = iso(time.time() + 3600) +check(isinstance(relay.ambient_credentials()[1], float), + "a stated deadline should be carried through as the expiry") +# The environment wins while it is usable, so this never shells out to a profile. +picked = relay.resolve_credentials("no-such-profile") +check(picked[0]["aws_session_token"] == "t0ken", + "the environment should be preferred over the profile while it is usable") +check(picked[2] == relay.FROM_ENVIRONMENT, + "the resolver must say where the credentials came from: %r" % (picked[2],)) + +os.environ["AWS_CREDENTIAL_EXPIRATION"] = iso(time.time() - 1) +check(relay.ambient_credentials() is None, + "expired ambient credentials must send the relay to the profile instead") + +os.environ["AWS_CREDENTIAL_EXPIRATION"] = iso(time.time() + 60) +check(relay.ambient_credentials(margin=300) is None, + "ambient credentials inside the refresh margin must not start a session") +check(relay.ambient_credentials(margin=0) is not None, + "the same credentials are still usable when no margin is asked for") + +# A profile export that fails must be an ordinary exception. SystemExit would walk +# straight through the per-turn boundary in handle_uplink_frame and end the relay, +# and since credentials are resolved lazily this refusal can land mid-conversation. +class Failed: + returncode = 1 + stdout = "" + stderr = "The config profile (nobody) could not be found" + +real_run = relay.subprocess.run +relay.subprocess.run = lambda *a, **k: Failed() +try: + relay.profile_credentials("nobody") + sys.exit("credentials: a failed profile export was accepted") +except relay.CredentialError as exc: + check(isinstance(exc, Exception), + "the refusal must be an ordinary exception, not a SystemExit") + check("nobody" in str(exc), "the refusal should name the profile: %s" % exc) +except SystemExit: + sys.exit("credentials: a failed profile export raised SystemExit") +finally: + relay.subprocess.run = real_run + +# No profile and no environment is also a named refusal rather than a traceback +# from inside the AWS CLI argument list. +try: + relay.profile_credentials("") + sys.exit("credentials: an empty profile was accepted") +except relay.CredentialError as exc: + check("voice-profile" in str(exc), + "the refusal should name the file to write: %s" % exc) + +for name in AWS_VARS: + os.environ.pop(name, None) + +calls = [] +stamp = [""] +delay = [0.0] + +def fake(profile, verbose=False, margin=0, allow_ambient=True): + # Whatever the profile says about expiry reaches the cache through the real + # parser, so the fixture hands it a stamp rather than a decided answer. + calls.append(profile) + time.sleep(delay[0]) + return ({"aws_access_key_id": "AK%d" % len(calls)}, + relay._expires_at(stamp[0]), relay.FROM_PROFILE) + +real_resolve = relay.resolve_credentials +relay.resolve_credentials = fake + +async def take(cache, count): + return [await cache.get() for _ in range(count)] + +# Three sessions, one resolution: the second and third turns pay nothing. +cache = relay.Credentials("a-profile") +got = asyncio.run(take(cache, 3)) +check(len(calls) == 1, "three sessions resolved credentials %d times" % len(calls)) +check([c["aws_access_key_id"] for c in got] == ["AK1"] * 3, + "every session should get the same credentials: %s" % got) + +# A session that edits what it was handed must not edit what the next one gets. +got[0]["aws_access_key_id"] = "tampered" +check(asyncio.run(take(cache, 1))[0]["aws_access_key_id"] == "AK1", + "one session must not be able to corrupt the shared credentials") + +# Credentials with an expiry are refreshed ahead of it, because a relay left +# running outlives them and a dead credential is a dead session. +del calls[:] +stamp[0] = iso(time.time() + relay.Credentials.REFRESH_MARGIN - 1) +cache = relay.Credentials("a-profile") +asyncio.run(take(cache, 2)) +check(len(calls) == 2, + "credentials near expiry should be refreshed, resolved %d times" % len(calls)) + +# An expiry this interpreter cannot read is NOT an expiry that never comes. The +# credential works, its deadline does not, so it is held for the same margin and +# no longer. Read as "never expires" it would be cached past the real deadline +# and every session from then on would fail to start with no way back. +class Bounded(relay.Credentials): + REFRESH_MARGIN = 0.05 + +del calls[:] +stamp[0] = "expires some time on Tuesday" +cache = Bounded("a-profile") +asyncio.run(take(cache, 2)) +check(len(calls) == 1, + "an unreadable expiry should still be reused within the margin: %d" % len(calls)) +time.sleep(0.1) +asyncio.run(take(cache, 1)) +check(len(calls) == 2, + "an unreadable expiry must not be cached for the life of the relay") + +# An absent expiry keeps meaning what it says: this credential does not expire. +del calls[:] +stamp[0] = "" +cache = Bounded("a-profile") +asyncio.run(take(cache, 1)) +time.sleep(0.1) +asyncio.run(take(cache, 1)) +check(len(calls) == 1, + "a credential with no expiry should not be resolved again: %d" % len(calls)) + +# Whenever a resolution does happen it must stay off the event loop, or the +# relay stops reading the captain's audio for as long as it takes. +del calls[:] +stamp[0] = "" +delay[0] = 0.3 + +async def resolve_while_the_loop_runs(): + ticks = [] + + async def tick(): + for _ in range(20): + await asyncio.sleep(0.01) + ticks.append(1) + + task = asyncio.create_task(tick()) + await relay.Credentials("slow-profile").get() + during = len(ticks) + task.cancel() + return during + +during = asyncio.run(resolve_while_the_loop_runs()) +check(during >= 2, + "the event loop ran %d times during a 0.3s credential resolution" % during) + +# GIVING UP ON AMBIENT CREDENTIALS HAS TO STICK. A bound that re-reads the same +# environment is not a bound: os.environ never gets fresher values while this +# process runs, so the same stale keys would come back every time and every +# session past the real deadline would fail while a working profile went untried. +# This drives the real resolver, with only the profile export replaced. +relay.resolve_credentials = real_resolve +exports = [] + +def fake_profile(profile, verbose=False): + exports.append(profile) + return {"aws_access_key_id": "FROM-PROFILE", + "aws_secret_access_key": "s", "aws_session_token": None}, None + +real_profile = relay.profile_credentials +relay.profile_credentials = fake_profile +os.environ["AWS_ACCESS_KEY_ID"] = "AKIAENVIRONMENT" +os.environ["AWS_SECRET_ACCESS_KEY"] = "s3cret" +os.environ["AWS_SESSION_TOKEN"] = "stale-token" +os.environ.pop("AWS_CREDENTIAL_EXPIRATION", None) + +cache = Bounded("a-profile") +first = asyncio.run(cache.get()) +check(first["aws_session_token"] == "stale-token", + "usable ambient credentials should be preferred: %r" % first) +check(exports == [], "the profile should not be consulted while ambient ones hold") +time.sleep(0.1) +later = [asyncio.run(cache.get()) for _ in range(3)] +check([c["aws_access_key_id"] for c in later] == ["FROM-PROFILE"] * 3, + "past the margin the relay must ask the profile, not re-read the " + "environment it already gave up on: %r" % later) +check(len(exports) == 1, + "and the profile answer is then cached like any other: %d exports" % len(exports)) + +# WITH NO PROFILE THERE IS NOTHING TO ESCALATE TO, and a relay configured that +# way is a documented shape. Giving up on the environment there would end every +# turn from the margin onwards, with a message saying there are no credentials in +# the environment while the process is still holding them. The bound becomes a +# re-read instead: the keys may be stale, which is between AWS and whoever +# exported them, but the conversation survives. +del exports[:] +cache = Bounded("") +kept = [asyncio.run(cache.get())] +for _ in range(3): + time.sleep(0.1) + kept.append(asyncio.run(cache.get())) +check([c["aws_session_token"] for c in kept] == ["stale-token"] * 4, + "a profile-free relay must keep answering from the environment: %r" % kept) +check(exports == [], "and must not try to export from a profile it does not have") + +# Even a credential whose stated deadline has already passed, for the same +# reason: there is no fresher source, so refusing is a dead relay rather than a +# safer one. +os.environ["AWS_CREDENTIAL_EXPIRATION"] = iso(time.time() - 60) +cache = Bounded("") +past = asyncio.run(cache.get()) +check(past["aws_session_token"] == "stale-token", + "an expired ambient credential is still the only answer available: %r" % past) +# With a profile, that same credential is abandoned for it, as before. +cache = Bounded("a-profile") +check(asyncio.run(cache.get())["aws_access_key_id"] == "FROM-PROFILE", + "an expired ambient credential should be abandoned when a profile exists") +os.environ.pop("AWS_CREDENTIAL_EXPIRATION", None) + +# A PROFILE THAT CANNOT ANSWER must not cost the environment. Abandoning ambient +# credentials is justified by the profile answering, so it is only decided once the +# profile has: otherwise one failed export strands a relay that is still holding +# keys, and every later turn names a profile while the answer sits in os.environ. +del exports[:] + +def refusing_profile(profile, verbose=False): + exports.append(profile) + raise relay.CredentialError( + "could not get credentials for profile {}".format(profile)) + +relay.profile_credentials = refusing_profile +os.environ["AWS_ACCESS_KEY_ID"] = "AKIAENVIRONMENT" +os.environ["AWS_SECRET_ACCESS_KEY"] = "s3cret" +os.environ["AWS_SESSION_TOKEN"] = "still-held-token" +os.environ.pop("AWS_CREDENTIAL_EXPIRATION", None) + +cache = Bounded("a-profile") +check(asyncio.run(cache.get())["aws_session_token"] == "still-held-token", + "usable ambient credentials should be preferred while they hold") +kept = [] +for _ in range(3): + time.sleep(0.1) + try: + kept.append(asyncio.run(cache.get())["aws_session_token"]) + except relay.CredentialError: + kept.append("CredentialError") +check(kept == ["still-held-token"] * 3, + "a refusing profile must cost one attempt, not the conversation: %r" % kept) +check(exports, "and the profile should have been tried at least once: %r" % exports) + +# An environment with nothing in it and no profile is still a named refusal, so +# this restores the real resolver rather than asking the stub to pretend. +for name in AWS_VARS: + os.environ.pop(name, None) +relay.profile_credentials = real_profile +refused = None +try: + asyncio.run(Bounded("").get()) +except relay.CredentialError as exc: + refused = str(exc) +check(refused is not None, "no credentials anywhere should refuse") +check("voice-profile" in refused, + "and the refusal should name the file to write: %s" % refused) +PY +pass "credentials are resolved once per relay, refreshed before expiry, off the event loop" + +# --- a turn the model refuses ----------------------------------------------- +# +# The relay rebuilds the model session on every turn by design, so every turn +# reaches the model and every turn can fail on its own: a throttle, a dropped +# stream, a token that went stale between turns. That has to cost the captain one +# turn rather than the whole session, because the alternative is a traceback on +# the stderr the client inherits and a relay restarted by hand. + +python3 - "$ROOT/bin" <<'PY' || fail "failed turn" +import asyncio, sys +sys.path.insert(0, sys.argv[1]) +import importlib.util, pathlib +spec = importlib.util.spec_from_file_location( + "relay", str(pathlib.Path(sys.argv[1]) / "fm-voice-relay.py")) +relay = importlib.util.module_from_spec(spec) +spec.loader.exec_module(relay) +import fm_voice_frame as frame + +def check(cond, label): + if not cond: + sys.exit("failed turn: " + label) + +class Down: + def __init__(self): + self.notices = [] + + def send(self, kind, payload=b""): + pass + + def send_json(self, kind, obj): + if kind == frame.NOTICE: + self.notices.append(obj) + +class Stub: + """A session that records what it was asked, or raises where the model would.""" + + def __init__(self, raises=None): + self.raises = raises + self.replies = 0 + self.failed = False + self.ended = asyncio.Event() + self.turn = {} + self.calls = [] + + async def _step(self, name): + self.calls.append(name) + if self.raises is not None: + raise self.raises + + async def talk_start(self): + await self._step("talk_start") + + async def audio(self, pcm): + await self._step("audio:%d" % len(pcm)) + + async def talk_end(self): + await self._step("talk_end") + +options = relay.parse_args(["--serve"]) + +async def drive(session, items): + down = Down() + serving = True + for kind, payload in items: + session, serving = await relay.handle_uplink_frame( + kind, payload, session, options, down) + if not serving: + break + return session, serving, down + +# The ordinary path is unchanged: the frames reach the session in order. +good = Stub() +session, serving, down = asyncio.run(drive(good, [ + (frame.TALK_START, b""), (frame.AUDIO, b"1234"), (frame.TALK_END, b"")])) +check(good.calls == ["talk_start", "audio:4", "talk_end"], + "a good turn should reach the session: %s" % good.calls) +check(serving and not good.failed, + "a good turn must not mark the session spent") +check(down.notices == [], "a good turn should not announce a failure") + +# A model failure mid-turn: the captain is told what happened, the relay stays +# up, and the session is marked spent so nothing reuses a dead stream. +broken = Stub(raises=RuntimeError("ThrottlingException")) +session, serving, down = asyncio.run(drive(broken, [(frame.AUDIO, b"1234")])) +check(serving, "a failed turn must not stop the relay") +check(session is broken and broken.failed, + "a failed session must be marked spent") +check([n["event"] for n in down.notices] == ["turn-failed"], + "a failed turn must be announced to the client: %s" % down.notices) +check("ThrottlingException" in down.notices[0].get("error", ""), + "the notice should name the failure: %s" % down.notices[0]) + +# ONCE PER TURN, not once per frame. The captain is still holding the talk key +# when the failure lands, so the rest of that press is another thirty audio +# frames, one per hundred milliseconds. The client says every notice out loud on +# stderr, so reporting each one would put ten identical lines a second in front of +# the captain while they are still speaking, and would keep calling into a session +# that is already gone. +held = Stub(raises=RuntimeError("ValidationException")) +frames = [(frame.TALK_START, b"")] + [(frame.AUDIO, b"x" * 3200)] * 30 +frames.append((frame.TALK_END, b"")) +session, serving, down = asyncio.run(drive(held, frames)) +check(serving, "a failed turn must not stop the relay") +check(len(down.notices) == 1, + "a failed turn must be announced once, not once per frame: %d notices" + % len(down.notices)) +check(held.calls == ["talk_start"], + "nothing after the failure should reach the dead session: %s" % held.calls) + +# And the next talk key rebuilds instead of reusing it, which is what marking it +# spent is for. +renewed = [] +fresh = Stub() +real_renew = relay.renew + +async def fake_renew(session, options, down): + renewed.append(session) + return fresh + +relay.renew = fake_renew +session, serving, down = asyncio.run(drive(broken, [(frame.TALK_START, b"")])) +check(renewed == [broken], "a spent session must be replaced on the next turn") +check(session is fresh and fresh.calls == ["talk_start"], + "the replacement session must take the turn: %s" % fresh.calls) + +# A reconnect that fails is itself just a failed turn: the captain presses the +# key again rather than restarting the relay. +async def failing_renew(session, options, down): + raise RuntimeError("EndpointConnectionError") + +relay.renew = failing_renew +spent = Stub() +spent.replies = 1 +session, serving, down = asyncio.run(drive(spent, [(frame.TALK_START, b"")])) +check(serving, "a failed reconnect must not stop the relay") +check(spent.failed, "a failed reconnect must leave the session spent") +check([n["event"] for n in down.notices] == ["turn-failed"], + "a failed reconnect must be announced: %s" % down.notices) +check(spent.calls == [], "a session whose reconnect failed must not be spoken to") + +# Quit still ends the loop, so the relay exits when the client says so. +session, serving, down = asyncio.run(drive(Stub(), [(frame.QUIT, b"")])) +check(not serving, "quit must end the loop") + +# A reconnect that fails part way must not strand the session it was building. +# start() opens the model stream and a reader task before it sends anything, and +# the relay now survives the failure and retries, so a session left open here +# would accumulate one live stream and one live task per retry, all of them still +# writing into the shared downlink. +class Partial: + """A session whose start fails after it would have opened the stream.""" + + def __init__(self, *args): + self.closed = 0 + self.credentials = None + self.connect_seconds = None + + async def start(self): + raise RuntimeError("ServiceUnavailableException") + + async def close(self): + self.closed += 1 + +built = [] + +def make_partial(options, down, credentials): + session = Partial() + built.append(session) + return session + +relay.Session = make_partial +outgoing = Partial() +try: + asyncio.run(real_renew(outgoing, options, Down())) + sys.exit("failed turn: a failed reconnect was reported as success") +except RuntimeError: + pass +check(len(built) == 1, "renew should have built one replacement: %d" % len(built)) +check(built[0].closed == 1, + "a session whose start failed must be closed, not stranded: %d closes" + % built[0].closed) +PY +pass "a turn the model refuses is announced and costs one turn, not the relay" + +# --- audio that arrives with no turn open ------------------------------------ +# +# Both listen modes send a talk start before any audio, so audio outside a turn +# means the capture callback raced the key release and a stray chunk landed behind +# the talk end. Opening a block for it would append the captain's stray tenth of a +# second to a session that is already answering, which is the unconditional +# barge-in the per-turn reconnect exists to avoid, and would leave that block open +# so the next turn skipped its own reset and its first-audio mark. + +python3 - "$ROOT/bin" "$TMP_ROOT/stray-home" <<'PY' || fail "stray audio" +import asyncio, os, sys +sys.path.insert(0, sys.argv[1]) +import importlib.util, pathlib +spec = importlib.util.spec_from_file_location( + "relay", str(pathlib.Path(sys.argv[1]) / "fm-voice-relay.py")) +relay = importlib.util.module_from_spec(spec) +spec.loader.exec_module(relay) + +home = sys.argv[2] +os.makedirs(home, exist_ok=True) + +def check(cond, label): + if not cond: + sys.exit("stray audio: " + label) + +class Down: + def send(self, kind, payload=b""): + pass + + def send_json(self, kind, obj): + pass + + def arm_turn(self): + pass + + def first_audio(self): + return None + +sent = [] + +async def record(event): + sent.append(event) + +session = relay.Session(relay.parse_args(["--serve", "--home", home]), Down(), None) +session._send = record + +# No turn open: the stray chunk goes nowhere, and no block is left behind for the +# next turn to trip over. This session has no model stream either, so anything +# that did try to open a block would raise rather than pass quietly. +asyncio.run(session.audio(b"\x01" * 3200)) +check(sent == [], "audio with no turn open must not be forwarded: %r" % sent) +check(session.audio_content is None, + "and must not leave an audio block open: %r" % session.audio_content) + +# Inside a turn it flows, so the guard is about the boundary and not about audio. +asyncio.run(session.talk_start()) +del sent[:] +asyncio.run(session.audio(b"\x01" * 3200)) +check([next(iter(event)) for event in sent] == ["audioInput"], + "audio inside a turn must still be forwarded: %r" % sent) + +# And talk end still pads with its trailing silence before closing the block, +# which is the whole reason a push-to-talk clip gets answered at all. +del sent[:] +asyncio.run(session.talk_end()) +kinds = [next(iter(event)) for event in sent] +check(kinds.count("audioInput") > 0 and kinds[-1] == "contentEnd", + "talk end must pad with silence and then close the block: %r" % kinds) +check(session.audio_content is None, "talk end must close the block") +PY +pass "audio that arrives with no turn open is dropped, not turned into a turn" + +# --- a model stream that dies while answering -------------------------------- +# +# The reader task is the other place a turn can fail, and it fails in the middle +# of work: handling an event reaches back into the model to answer a tool call. A +# failure there must still tell a waiting turn the session is over, and close() +# must absorb it, because close() is the first thing renew does. Neither held +# once, and the cost was not one lost turn but every later one: the reader task +# kept its exception, close() re-raised it on every await, renew never reached +# the line that builds a replacement, and the captain heard the same failure +# forever with no way back short of restarting the relay. +# +# Releasing the waiting turn is only half of it. The client waits for a reply end +# or a notice, so a reader failure that says nothing costs the captain their whole +# timeout and leaves a record that says the turn was not answered without saying +# why. It has to be named, once, and only when it really is a failure: a stream +# that simply ends, and a stream that went away because close() asked it to, are +# both ordinary and neither may look like one. + +python3 - "$ROOT/bin" "$TMP_ROOT/reader-home" <<'PY' || fail "reader failure" +import asyncio, json, os, sys +sys.path.insert(0, sys.argv[1]) +import fm_voice_frame as frame +import importlib.util, pathlib +spec = importlib.util.spec_from_file_location( + "relay", str(pathlib.Path(sys.argv[1]) / "fm-voice-relay.py")) +relay = importlib.util.module_from_spec(spec) +spec.loader.exec_module(relay) + +home = sys.argv[2] +os.makedirs(home, exist_ok=True) +options = relay.parse_args(["--serve", "--home", home]) + +def check(cond, label): + if not cond: + sys.exit("reader failure: " + label) + +class Down: + def __init__(self): + self.notices = [] + + def send(self, kind, payload=b""): + pass + + def send_json(self, kind, obj): + self.notices.append(obj) + + def arm_turn(self): + pass + + def first_audio(self): + return None + +class Input: + async def send(self, chunk): + pass + + async def close(self): + pass + +class Payload: + def __init__(self, raw): + self.bytes_ = raw + +class Result: + def __init__(self, raw): + self.value = Payload(raw) + +class Receiver: + def __init__(self, raw): + self._raw = raw + + async def receive(self): + return None if self._raw is None else Result(self._raw) + +class Stream: + """The scripted events, and then either a clean end or a stream that is gone.""" + + def __init__(self, raws, clean=False): + self._raws = list(raws) + self._clean = clean + self.input_stream = Input() + + async def await_output(self): + if not self._raws: + if self._clean: + return (None, Receiver(None)) + raise RuntimeError("the model stream is gone") + return (None, Receiver(self._raws.pop(0))) + +TOOL_EVENT = json.dumps({"event": {"toolUse": { + "toolName": "no_such_tool", "toolUseId": "t-1", "content": "{}"}}}).encode() + +def failures(down): + return [n for n in down.notices if n.get("event") == "turn-failed"] + +async def poison(how): + """Break a session one of the ways the model side can break it.""" + down = Down() + session = relay.Session(options, down, None) + session.stream = Stream([TOOL_EVENT]) + # A turn is open and waiting for a reply, which is when this costs the most. + session.turn["talk_end"] = 0.0 + if how == "handler": + async def boom(event): + raise RuntimeError("handling blew up") + session._handle = boom + elif how == "tool-result": + # The real _handle and _run_tool, with the tool-result send failing: the + # shape a dropped stream takes while the relay answers a tool call. + async def refuse(obj): + raise RuntimeError("the model stream is gone") + session._send = refuse + elif how == "drop": + # Nothing to read and no clean end: the stream simply goes away, which is + # what a network blip looks like from here. + session.stream = Stream([]) + session.reader_task = asyncio.create_task(session._read_model()) + await asyncio.wait_for(session.ended.wait(), timeout=5) + return down, session + +async def one_case(how): + """Break a session, then take the next turn over the same relay.""" + down, session = await poison(how) + check(session.turn_done.is_set(), + "%s: a waiting turn must be released" % how) + check(session.failed, "%s: the session must be marked spent" % how) + named = failures(down) + check(len(named) == 1, + "%s: the captain must be told once, not never and not twice: %r" + % (how, down.notices)) + check(named[0].get("error"), + "%s: the notice must carry the cause: %r" % (how, named[0])) + check(session.turn.get("failed") == named[0]["error"], + "%s: the run record must carry the same cause: %r" % (how, session.turn)) + + # close() absorbs the stored failure however many times it is asked, which is + # what lets the next turn get as far as building a replacement. + for _ in range(3): + await session.close() + + built = [] + + class Fresh: + def __init__(self, *args): + self.connect_seconds = 0.02 + self.failed = False + self.replies = 0 + self.ended = asyncio.Event() + self.turns = 0 + built.append(self) + + async def start(self): + pass + + async def talk_start(self): + self.turns += 1 + + real, relay.Session = relay.Session, Fresh + try: + used, serving = await relay.handle_uplink_frame( + frame.TALK_START, b"", session, options, down) + finally: + relay.Session = real + check(len(built) == 1, + "%s: the next talk key must build a session: %r" % (how, built)) + check(used is built[0] and serving, + "%s: the relay must go on serving with it: %r %r" % (how, used, serving)) + check(used.turns == 1, "%s: and give it the new turn: %r" % (how, used.turns)) + check(len(failures(down)) == 1, + "%s: recovering must not name the turn again: %r" % (how, down.notices)) + +for how in ("handler", "tool-result", "drop"): + asyncio.run(one_case(how)) + +# A stream that simply ends is the end of a session, not a failed turn. It is +# still said out loud, once, and it says what it is: the client is waiting on a +# turn that is not coming, and only a notice releases it, but calling it a failure +# would tell the captain something broke when the model merely finished. +async def clean_end(): + down = Down() + session = relay.Session(options, down, None) + session.stream = Stream([], clean=True) + session.reader_task = asyncio.create_task(session._read_model()) + await asyncio.wait_for(session.ended.wait(), timeout=5) + return down, session + +down, ended = asyncio.run(clean_end()) +check(ended.turn_done.is_set(), "a clean end must release a waiting turn too") +check(not ended.failed, "a clean end of stream is not a turn failure") +check(not failures(down), + "and no failure should be named to the captain: %r" % (down.notices,)) +check([n.get("event") for n in down.notices] == ["session-ended"], + "a clean end must be announced once, as the end it is: %r" % (down.notices,)) + +# Nor is a stream that went away because close() asked it to. renew closes the +# old session on every single turn, so announcing that would put a failure notice +# in front of the captain on every ordinary turn. +async def torn_down(): + down = Down() + session = relay.Session(options, down, None) + gone = asyncio.Event() + + class Closer: + async def send(self, chunk): + pass + + async def close(self): + # The stream goes away exactly when close() closes the input half, + # which is the ordering every renewed turn goes through. + gone.set() + + class Blocking: + def __init__(self): + self.input_stream = Closer() + + async def await_output(self): + await gone.wait() + raise RuntimeError("the model stream is gone") + + async def quiet(obj): + pass + + session.stream = Blocking() + # The real one builds an SDK event, and the SDK is deliberately not installed + # here; what this case needs is close() getting as far as the input half. + session._send = quiet + session.reader_task = asyncio.create_task(session._read_model()) + await asyncio.sleep(0) + await session.close() + await asyncio.wait_for(session.ended.wait(), timeout=5) + return down, session + +down, closed = asyncio.run(torn_down()) +check(closed.ended.is_set(), "the reader must still report the session over") +check(not closed.failed, "a deliberate close is not a turn failure") +check(not down.notices, + "and an ordinary renew must say nothing at all: %r" % (down.notices,)) +PY +pass "a failure inside the model reader costs one turn, not every later one" +pass "a reader failure is named to the captain, a clean end and a close are not" + +# --- the laptop end --------------------------------------------------------- +# +# The microphone and speaker DEVICES cannot be opened on a host with neither, so +# the first live run is their test and nothing below touches audio hardware. What +# IS testable is everything around them, and these are the pieces whose failure is +# hardest to read from the symptom. A missing -T corrupts audio rather than +# erroring, and a banner-printing login shell desynchronises the stream in a way +# that looks like a protocol bug and is not. The speaker's byte accounting is +# testable as well, being arithmetic rather than device work, and is covered +# further down this block against a stub stream with the callback driven by hand. + +mkdir -p "$TMP_ROOT/client-files" +printf '\0\0\0\0' > "$TMP_ROOT/client-files/clip.pcm" + +python3 - "$ROOT/bin" "$TMP_ROOT/client-files" <<'PY' || fail "laptop client" +import io, os, sys, types +sys.path.insert(0, sys.argv[1]) +import importlib.util, pathlib +spec = importlib.util.spec_from_file_location( + "client", str(pathlib.Path(sys.argv[1]) / "fm-voice-client.py")) +client = importlib.util.module_from_spec(spec) +spec.loader.exec_module(client) +import fm_voice_frame as frame + +TMP = sys.argv[2] + +def check(cond, label): + if not cond: + sys.exit("client: " + label) + +# Where the relay lives on the desktop is one operator's directory layout, so the +# client carries no default for it and says so rather than trying a path that +# belongs to somebody else. The refusal is checked before the variable below is +# set, because after that every other case supplies it. +os.environ.pop("FM_VOICE_RELAY", None) +try: + client.parse_args(["--host", "desk"]) + sys.exit("client: started with no relay path at all") +except SystemExit as exc: + check(exc.code != 0, "a missing relay path must be a refusal, not a default") + +os.environ["FM_VOICE_RELAY"] = "/desktop/firstmate/bin/fm-voice-relay.py" +check(client.parse_args(["--host", "desk"]).relay + == "/desktop/firstmate/bin/fm-voice-relay.py", + "FM_VOICE_RELAY should supply the relay path for a whole shell") +check(client.parse_args(["--host", "desk", "--relay", "/other/relay.py"]).relay + == "/other/relay.py", "an explicit --relay must win over the variable") + +# Push to talk is the default for this build, and the only mode that runs. +check(client.parse_args(["--host", "h"]).listen == client.PUSH_TO_TALK, + "push to talk must be the default") + +# An open microphone needs to know when the captain stopped speaking, and this +# client cannot: it would open a turn and stream forever without ever marking a +# boundary. So the setting is accepted as a value and refuses at parse time, +# before any ssh connection is opened or any model session is paid for. The value +# stays in the accepted set so switching it on later is a small change. +check(client.OPEN_MIC in client.LISTEN_MODES, + "open mic must stay a value the flag accepts") +refusal = None +try: + client.parse_args(["--host", "h", "--listen", "open-mic"]) +except SystemExit as exc: + refusal = exc.code +check(refusal not in (None, 0), + "open mic must refuse rather than start: %r" % (refusal,)) + +# The audio devices themselves cannot be reached from here, but their SELECTOR +# can be, and it is typed: sounddevice reads an int as an index into its device +# list and a str as a name to match, so an index left as text is looked up as a +# device literally called "3" and raises on the captain's first live run. +picked = client.parse_args(["--host", "h", "--input-device", "3", + "--output-device", "External Headphones"]) +check(picked.input_device == 3 and not isinstance(picked.input_device, str), + "a numeric device must arrive as an index: %r" % picked.input_device) +check(picked.output_device == "External Headphones", + "a named device must stay a name: %r" % picked.output_device) +check(client.parse_args( + ["--host", "h", "--input-device", "2 - Built-in Microphone"] + ).input_device == "2 - Built-in Microphone", + "a device name that begins with a digit must stay a name") +check(client.parse_args(["--host", "h"]).input_device is None, + "no device flag must stay unset, so sounddevice picks the default") + +# Over SSH: no pty, or the audio stream is silently rewritten. +argv = client.relay_command(client.parse_args(["--host", "desk"])) +check(argv[:3] == ["ssh", "-T", "desk"], "ssh must be invoked with -T: %s" % argv) +check("--serve" in argv, "the relay must be started in serve mode") + +# Locally: no ssh at all, so the same client can be measured on this host. +argv = client.relay_command(client.parse_args(["--local"])) +check(argv[0] != "ssh", "--local must not invoke ssh: %s" % argv) + +# The interpreter is a setting because the relay needs a virtual environment the +# system interpreter does not have. +argv = client.relay_command(client.parse_args( + ["--host", "desk", "--relay-python", "/opt/venv/bin/python", + "--relay-arg=--scope", "--relay-arg=counts"])) +check("/opt/venv/bin/python" in argv, "the relay interpreter must be passed: %s" % argv) +check(argv[-2:] == ["--scope", "counts"], + "relay arguments must reach the relay: %s" % argv) + +# A relay that dies after the handshake must be reported at once rather than at +# the end of the timeout. Its own one-line error is already on the captain's +# terminal, because the relay's stderr is inherited rather than piped, so the only +# thing a full timeout adds is thirty seconds of watching nothing. This is the +# likely first-run shape: the Bedrock SDK is imported inside the model session, so +# a forgotten --relay-python exits the relay after the handshake. +import time as clock +waiting = client.Client(client.parse_args(["--host", "desk", "--timeout", "5"])) +waiting.closed.set() +began = clock.monotonic() +refused = None +try: + waiting._wait_ready() +except SystemExit as exc: + refused = str(exc) +check(refused is not None, "a closed relay was treated as ready") +check("closed the connection" in refused, + "the refusal should name the closed connection: %s" % refused) +check("by hand" in refused, "and should give the next step: %s" % refused) +took = clock.monotonic() - began +check(took < 2, "a closed relay should be reported at once, waited %.1fs" % took) + +# Ready still wins, and a relay that says nothing at all still times out with the +# message that fits that case instead. +ready = client.Client(client.parse_args(["--host", "desk", "--timeout", "5"])) +ready.ready.set() +ready._wait_ready() + +silent = client.Client(client.parse_args(["--host", "desk", "--timeout", "0.3"])) +timed_out = None +try: + silent._wait_ready() +except SystemExit as exc: + timed_out = str(exc) +check(timed_out is not None, "a silent relay was treated as ready") +check("never reported ready" in timed_out, + "a silent relay should time out with its own message: %s" % timed_out) + +# The uplink can die mid-session - the SSH connection drops, or the relay exits - +# and the next talk start or talk end is then a write to a dead pipe. Every frame +# a turn is made of goes through the one sender thread, so that write has to end +# the thread the same quiet way a dead audio write does. Raising instead killed +# the thread with a traceback and left the queue unserved, so each remaining run +# sat out the full timeout with nothing sending its frames and was reported as an +# unanswered turn rather than as the lost connection the downlink had already seen. +class DeadPipe: + def __init__(self): + self.sent = [] + + def send(self, kind, payload=b""): + self.sent.append(kind) + raise BrokenPipeError(32, "Broken pipe") + +for label, opening in (("talk start", client.START), ("talk end", client.END), + ("audio", b"\x00\x00")): + sending = client.Client(client.parse_args(["--host", "desk"])) + sending.uplink = DeadPipe() + sending.up_q.put(opening) + sending.up_q.put(b"\x01\x01") + sending.up_q.put(None) + try: + sending._sender() + except BaseException as exc: # noqa: BLE001 + sys.exit("client: a broken pipe on %s killed the sender thread: %s: %s" + % (label, type(exc).__name__, exc)) + check(sending.uplink.sent and len(sending.uplink.sent) == 1, + "the sender should stop at the broken pipe on %s rather than keep " + "writing into it: %r" % (label, sending.uplink.sent)) + if opening is client.END: + # Talk end stamps the moment it reached the wire before the write is + # attempted, so uplink_drain_s survives a turn the connection cut short. + check("wire_end" in sending.turn, + "talk end must still record when it reached the wire: %r" + % sending.turn) + +# A connection that drops mid-turn does not wait for a frame boundary, so the +# downlink meets a header cut in half. That is a transport failure and the turn +# record has to say so: a run that only reports answered: false reads in +# runs.jsonl exactly like a turn the model declined, and the latency spread +# docs/voice-relay.md publishes is computed from that file. +import threading as thread_lib + + +class CutStream: + """A downlink that drops mid-header once the turn is under way. + + Held closed until the client has actually opened the turn, so the cut lands + inside the turn being measured rather than before it, which is the sequence + a dropped SSH connection produces and the only one whose record matters. + """ + + def __init__(self, gate): + self._gate = gate + self._half = frame.encode(frame.BYE)[:2] + self._at = 0 + + def read(self, count): + check(self._gate.wait(10), "the turn never opened, so nothing was cut") + chunk = self._half[self._at:self._at + count] + self._at += len(chunk) + return chunk + + +opened = thread_lib.Event() + + +class GateOpeningUplink: + """Discards the uplink and reports when the turn's first frame went out.""" + + def send(self, kind, payload=b""): + if kind == frame.TALK_START: + opened.set() + + +cut = client.Client(client.parse_args( + ["--host", "desk", "--in-file", os.path.join(TMP, "clip.pcm"), + "--out-file", os.path.join(TMP, "reply-cut.pcm"), "--timeout", "5"])) +cut.reader = frame.Reader(CutStream(opened)) +cut.uplink = GateOpeningUplink() +cut.playback = client.FilePlayback(os.path.join(TMP, "reply-cut.pcm")) +cut.capture = client.FileCapture(os.path.join(TMP, "clip.pcm")) +cut.capture.start(cut.up_q, cut.talking) +thread_lib.Thread(target=cut._sender, daemon=True).start() +thread_lib.Thread(target=cut._downlink, daemon=True).start() +cut_record = cut.take_turn(1) +cut.up_q.put(None) +cut.playback.close() + +check(cut.closed.wait(10), "a cut header must end the downlink, not hang it") +check(not cut_record["answered"], "a cut connection cannot have answered: %r" + % cut_record) +check(cut_record["relay_error"], + "a dropped connection must be reported in the turn record rather than " + "leaving it indistinguishable from a turn nobody answered: %r" + % cut_record) +check("the connection was lost before this turn was answered" + in cut_record["relay_error"], + "and it should say the connection went before this turn had its answer, " + "which is the half of that fault this case is: %r" + % cut_record["relay_error"]) + +# The other moment a connection can go is BETWEEN two turns, during the seconds +# the client spends letting the previous answer finish. That wait is most of a +# multi-run session, so it is where a relay that dies between questions dies. +# Opening the next turn anyway cleared the failure the downlink had recorded, +# left nothing on the far end to answer it, and produced a run that came back +# after the whole reply timeout saying answered: false with relay_error: null. +# In runs.jsonl that is indistinguishable from a turn the model declined, and +# runs.jsonl is the file docs/voice-relay.md computes its latency spread from, so +# the invented turn would be averaged into a published number. +import contextlib +import json as json_lib + + +class ClosingStream: + """Serves one whole turn, then closes during the wait after it. + + Both moments are released by the client reaching them rather than by a + timer: the frames wait for the turn to open, and the close waits for the + client to enter the inter-turn wait. So the sequence under test is the same + on a loaded host as on an idle one. + """ + + def __init__(self, gate, waiting): + self._gate = gate + self._waiting = waiting + self._reply = ( + frame.encode(frame.AUDIO, b"\x00\x00" * 1200) + + frame.encode_json(frame.MARK, {"mark": "reply_end", + "since_talk_end": 0.4, + "tool_calls": 1})) + self._at = 0 + + def read(self, count): + check(self._gate.wait(10), "the turn never opened, so nothing was served") + if self._at >= len(self._reply): + check(self._waiting.wait(10), + "fixture: the client never reached the wait between turns") + return b"" + chunk = self._reply[self._at:self._at + count] + self._at += len(chunk) + return chunk + + +class StartGate: + """Discards the uplink and reports when a turn's first frame went out.""" + + def __init__(self, gate): + self._gate = gate + + def send(self, kind, payload=b""): + if kind == frame.TALK_START: + self._gate.set() + + +served, waiting_between = thread_lib.Event(), thread_lib.Event() +closing = client.Client(client.parse_args( + ["--host", "desk", "--in-file", os.path.join(TMP, "clip.pcm"), + "--out-file", os.path.join(TMP, "reply-closing.pcm"), "--runs", "2", + "--timeout", "2", "--audio-idle", "0.05", "--gap-seconds", "0.05"])) +closing.reader = frame.Reader(ClosingStream(served, waiting_between)) +closing.uplink = StartGate(served) +closing.playback = client.FilePlayback(os.path.join(TMP, "reply-closing.pcm")) +closing.capture = client.FileCapture(os.path.join(TMP, "clip.pcm")) +closing.capture.start(closing.up_q, closing.talking) +thread_lib.Thread(target=closing._sender, daemon=True).start() +thread_lib.Thread(target=closing._downlink, daemon=True).start() + +# The wait between turns is the seam: the connection goes while the client is +# inside it, after the first run was reported and before the second could open. +# Waiting for the downlink to see it keeps that ordering exact rather than +# leaving it to whichever thread the scheduler runs next. +finish_wait = closing._let_reply_finish + + +def lose_connection_while_waiting(record): + waiting_between.set() + check(closing.closed.wait(10), + "fixture: the connection never closed during the wait between turns") + return finish_wait(record) + + +closing._let_reply_finish = lose_connection_while_waiting +emitted, spoken = io.StringIO(), io.StringIO() +with contextlib.redirect_stdout(emitted), contextlib.redirect_stderr(spoken): + closing_code = closing.run() +closing.up_q.put(None) +closing.playback.close() + +closing_runs = [json_lib.loads(line) for line in emitted.getvalue().splitlines() + if line.strip()] +check(len(closing_runs) == 1, + "a connection lost between turns must end the session rather than invent a " + "turn nobody took: %d run(s) reported, %r" % (len(closing_runs), closing_runs)) +check(closing_runs[0]["answered"] and closing_runs[0]["relay_error"] is None, + "fixture is wrong: the turn before the connection went should be a good " + "one, so the case cannot pass on a run that failed anyway: %r" + % closing_runs[0]) +# Two runs were asked for and one was taken, so the exit code has to be the +# unhappy one; a session that stops early while reporting success is a +# measurement someone reads as complete. +check(closing_code != 0, + "a session that took 1 of 2 runs must not exit 0, got %r" % closing_code) +check("the connection ended before run 2 of 2" in spoken.getvalue(), + "and the captain should be told why it stopped, in the words the path that " + "stopped it recorded: %r" % spoken.getvalue()) + +# The case above is the connection going while the run loop is watching for it. +# The loop cannot only be watching, though: the downlink names the failure and +# marks the connection closed, and a loop that decides by reading the mark alone +# is blind for as long as those are two separate writes - the connection is gone, +# the failure is on the record, and the check has already passed. So the decision +# belongs where the turn is opened, under the lock both writes are made under. +# With the connection already gone, opening the turn anyway erased the failure the +# downlink had recorded, pushed talk-start into a dead pipe, and printed a run +# that waited out the whole reply timeout as answered: false with relay_error: +# null - the invented turn this whole seam exists to keep out of runs.jsonl. +class LostStream: + """A downlink that is already gone the first time it is read.""" + + def read(self, count): + raise OSError(104, "Connection reset by peer") + + +class CountingUplink: + """Accepts frames and remembers which kinds were pushed at it.""" + + def __init__(self): + self.sent = [] + + def send(self, kind, payload=b""): + self.sent.append(kind) + + +gone = client.Client(client.parse_args( + ["--host", "desk", "--in-file", os.path.join(TMP, "clip.pcm"), + "--out-file", os.path.join(TMP, "reply-gone.pcm"), "--runs", "2", + "--timeout", "2", "--audio-idle", "0.05", "--gap-seconds", "0.05"])) +gone.reader = frame.Reader(LostStream()) +gone.uplink = CountingUplink() +gone.playback = client.FilePlayback(os.path.join(TMP, "reply-gone.pcm")) +gone.capture = client.FileCapture(os.path.join(TMP, "clip.pcm")) +gone.capture.start(gone.up_q, gone.talking) +thread_lib.Thread(target=gone._sender, daemon=True).start() +# Run to completion in this thread rather than in a started one: the connection +# is gone before the first read returns, so the whole downlink is over by the +# time run() begins and there is no ordering left for the scheduler to decide. +gone._downlink() +check(gone.closed.is_set(), + "fixture: a connection lost on the first read must leave the client closed") + +gone_out, gone_said = io.StringIO(), io.StringIO() +with contextlib.redirect_stdout(gone_out), contextlib.redirect_stderr(gone_said): + gone_code = gone.run() +gone.up_q.put(None) +gone.playback.close() + +gone_runs = [json_lib.loads(line) for line in gone_out.getvalue().splitlines() + if line.strip()] +check(gone_runs == [], + "a turn must not be opened on a connection already known gone, and no run " + "reported for one that never opened: %r" % gone_runs) +check(gone.uplink.sent == [], + "and nothing should be pushed into the dead pipe: %r" % gone.uplink.sent) +check(gone_code != 0, + "a session that took none of its 2 runs must not exit 0, got %r" % gone_code) +check("the connection was lost before run 1 of 2" in gone_said.getvalue(), + "and the captain should be told why nothing was taken, naming the run it " + "stopped at and the cause the read that failed recorded: %r" + % gone_said.getvalue()) + +# The third moment is a connection that goes DURING a turn that answered anyway, +# with runs still to take. The answer is real and its record stands, so nothing +# here is a turn failure; what must not happen is the session ending quietly on a +# happy exit code, because two of the three runs asked for are missing and a +# runs.jsonl short of its runs, reported as success, is read later as the whole +# measurement. Refusing the next turn is the one place a closed connection stops +# a session, so it reports the same way wherever the connection went. +class EndingStream: + """Serves one whole turn, then reports end of file without waiting.""" + + def __init__(self, gate): + self._gate = gate + self._reply = ( + frame.encode(frame.AUDIO, b"\x00\x00" * 1200) + + frame.encode_json(frame.MARK, {"mark": "reply_end", + "since_talk_end": 0.4, + "tool_calls": 0})) + self._at = 0 + + def read(self, count): + check(self._gate.wait(10), "the turn never opened, so nothing was served") + chunk = self._reply[self._at:self._at + count] + self._at += len(chunk) + return chunk + + +ending_served = thread_lib.Event() +ending = client.Client(client.parse_args( + ["--host", "desk", "--in-file", os.path.join(TMP, "clip.pcm"), + "--out-file", os.path.join(TMP, "reply-ending.pcm"), "--runs", "3", + "--timeout", "2", "--audio-idle", "0.05", "--gap-seconds", "0.05"])) +ending.reader = frame.Reader(EndingStream(ending_served)) +ending.uplink = StartGate(ending_served) +ending.playback = client.FilePlayback(os.path.join(TMP, "reply-ending.pcm")) +ending.capture = client.FileCapture(os.path.join(TMP, "clip.pcm")) +ending.capture.start(ending.up_q, ending.talking) +thread_lib.Thread(target=ending._sender, daemon=True).start() +thread_lib.Thread(target=ending._downlink, daemon=True).start() + +# The end of file lands inside the first turn, so it is already seen by the time +# the wait after that turn begins. Confirming it here rather than trusting the +# timing keeps the sequence exact on a loaded host as well as an idle one. +finish_after_end = ending._let_reply_finish + + +def confirm_ended_then_wait(record): + check(ending.closed.wait(10), + "fixture: the end of file never reached the client during the turn") + return finish_after_end(record) + + +ending._let_reply_finish = confirm_ended_then_wait +ending_out, ending_said = io.StringIO(), io.StringIO() +with contextlib.redirect_stdout(ending_out), contextlib.redirect_stderr(ending_said): + ending_code = ending.run() +ending.up_q.put(None) +ending.playback.close() + +ending_runs = [json_lib.loads(line) for line in ending_out.getvalue().splitlines() + if line.strip()] +check(len(ending_runs) == 1, + "one turn was served, so exactly one run belongs in the file: %d, %r" + % (len(ending_runs), ending_runs)) +check(ending_runs[0]["answered"] and ending_runs[0]["relay_error"] is None, + "an answered turn whose connection then ended cleanly is not a turn " + "failure, and its record stands: %r" % ending_runs[0]) +check(ending_code != 0, + "but a session that took 1 of 3 runs must not exit 0, got %r" % ending_code) +check("run 2 of 3" in ending_said.getvalue(), + "and it should name the run it stopped at: %r" % ending_said.getvalue()) + +# The case above is the connection ending AFTER a turn was answered, which is the +# negative case: nothing broke inside a turn, so nothing is named. The three +# below are the same three endings landing INSIDE a turn, while it is still +# waiting for its reply, and there each of them has to name itself. +# +# None of them raises. frame.Writer sends whole frames and the relay spends +# almost all of a turn awaiting the model, so a relay killed mid-turn - SIGKILL, +# the host going, the SSH connection dropping between frames - ends its stdout on +# a frame boundary. Reader then reads nothing at all where a header should start +# and reports end of input by design. A goodbye is the same shape by another +# route: the relay sends one from its own teardown after a fault. Released with +# no reason recorded, both come back as answered: false with relay_error: null, +# which in runs.jsonl is the shape of a turn the model declined, and runs.jsonl +# is the file docs/voice-relay.md computes its published latency spread from. An +# infrastructure failure averaged into that number is the whole thing being kept +# out of it. +class ScriptedStream: + """Serves prepared downlink bytes, held until the turn under test has opened. + + One class for every case below, because each of them differs only in the bytes + it serves: the gate and the slicing are the same everywhere, and an empty + script is what an end of stream at a frame boundary looks like. The label is + per case so a fixture that never fires still names which case it belonged to. + + What happens once the script is spent is the one thing a case may choose, and + the default is the clean end of stream every case but one wants. at_end takes + a callable for the case that needs the other shape a dropped link has, a reset + rather than a close, which reaches the client as a raised error instead of an + empty read. + """ + + def __init__(self, gate, payload, label, at_end=None): + self._gate = gate + self._bytes = payload + self._label = label + self._at_end = at_end + self._at = 0 + + def read(self, count): + check(self._gate.wait(10), + "the turn never opened, so %s was never served" % self._label) + if self._at >= len(self._bytes) and self._at_end is not None: + self._at_end() + chunk = self._bytes[self._at:self._at + count] + self._at += len(chunk) + return chunk + + +RELAY_REASON = "FrameError: unknown frame kind: b'\\xff'" +REPLY_END = frame.encode_json(frame.MARK, {"mark": "reply_end", + "since_talk_end": 0.4, + "tool_calls": 0}) + + +def wire_client(args, reader, uplink, playback=None, start=True): + """Build a client wired to stand-ins for both its ends, threads running. + + The same seven lines were written out at every case below. playback defaults to + a file at whatever --out-file the arguments name, which is what most of them + want; a case needing a playback of its own passes one. start=False is for the + one case that has to arrange something before the threads may read. + """ + wired = client.Client(client.parse_args(args)) + wired.reader = frame.Reader(reader) + wired.uplink = uplink + wired.playback = (client.FilePlayback(wired.options.out_file) + if playback is None else playback) + wired.capture = client.FileCapture(os.path.join(TMP, "clip.pcm")) + wired.capture.start(wired.up_q, wired.talking) + if start: + thread_lib.Thread(target=wired._sender, daemon=True).start() + thread_lib.Thread(target=wired._downlink, daemon=True).start() + return wired + + +def turn_lost(label, make_stream, out_name, runs="1"): + """Take one turn against a downlink that ends during it, and return run and stderr.""" + gate = thread_lib.Event() + lost = wire_client( + ["--host", "desk", "--in-file", os.path.join(TMP, "clip.pcm"), + "--out-file", os.path.join(TMP, out_name), "--runs", runs, + "--timeout", "2", "--audio-idle", "0.05"], + make_stream(gate), StartGate(gate)) + out, said = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(said): + code = lost.run() + lost.up_q.put(None) + lost.playback.close() + runs = [json_lib.loads(line) for line in out.getvalue().splitlines() + if line.strip()] + # The record is still emitted: the turn was taken and it really did go + # unanswered, so dropping it would hide the failure instead of naming it. + check(len(runs) == 1, + "%s: the turn was taken, so its run belongs in the file: %d, %r" + % (label, len(runs), runs)) + check(not runs[0]["answered"], + "%s: fixture is wrong, the turn was answered after all: %r" + % (label, runs[0])) + check(code != 0, "%s: an unanswered turn must not exit 0, got %r" + % (label, code)) + check(runs[0]["relay_error"], + "%s: a turn lost to the connection must say so rather than reading in " + "runs.jsonl exactly like a turn the model declined: %r" + % (label, runs[0])) + # With one run asked for, the run loop never reaches its "and the rest were + # not taken" line. A record on stdout and silence on stderr is a captain who + # spoke, heard nothing back, and was told nothing either; the module docstring + # puts everything human on stderr. + check("client:" in said.getvalue(), + "%s: the captain must be told the connection went, not only the file: " + "%r" % (label, said.getvalue())) + return runs[0], said.getvalue() + + +eof_run, eof_said = turn_lost( + "end of stream", + lambda gate: ScriptedStream(gate, b"", "the end of stream"), + "reply-eof.pcm") +check("ended" in eof_run["relay_error"], + "an end of stream should say the connection ended: %r" + % eof_run["relay_error"]) +check("the connection ended" in eof_said, + "and should say so on stderr as well: %r" % eof_said) + +# A goodbye nobody asked for. This end sent no quit, so the relay stopping is the +# relay's own decision, and mid-turn it costs the captain their question. +bye_run, bye_said = turn_lost( + "goodbye", + lambda gate: ScriptedStream(gate, frame.encode(frame.BYE), "the goodbye"), + "reply-bye.pcm") +check("the relay stopped" in bye_run["relay_error"], + "a goodbye nobody asked for mid-turn should say the relay stopped, because " + "that is the relay deciding rather than the connection dying: %r" + % bye_run["relay_error"]) +# Distinct from the line above, because they are distinct faults: a stream that +# stopped, versus a relay that chose to stop. One line for both would send the +# captain looking for the wrong thing. +check("stopped without being asked" in bye_said + and "the connection ended" not in bye_said, + "a goodbye should name itself on stderr rather than borrowing the wording " + "of an ended stream: %r" % bye_said) + +# And when the relay does name the fault, its words are what the record carries. +# This end can only infer that a goodbye arrived; the relay knows what happened, +# so a reason it sent is never replaced by one inferred here. +named_run, _ = turn_lost( + "named fault", + lambda gate: ScriptedStream( + gate, + frame.encode_json(frame.NOTICE, + {"event": "turn-failed", "error": RELAY_REASON}) + + frame.encode(frame.BYE), + "the named fault"), + "reply-named.pcm") +check(named_run["relay_error"] == RELAY_REASON, + "the relay's own reason must reach the record unchanged rather than being " + "overwritten by the goodbye behind it: %r" % named_run["relay_error"]) + +# THE MIDDLE OF THE SCENARIO, which the cases above and below both miss. A relay +# killed mid-turn ends its stdout on a frame boundary, and it spends nearly the +# whole turn streaming a reply, so dying AFTER some of that reply has played is +# the likelier half of the very fault this work exists to name. The cases above +# cover dying before any of it, and the answered-turn case below covers a reply +# that finished, so nothing pinned the middle. +# +# In the middle both halves of the record are true at once and must say so: sound +# reached the captain, so the turn was answered and first_audio_s measures when, +# AND the answer stopped partway, so the reason has to say the reply did not +# finish rather than that the turn was never answered. A record asserting both +# "answered" and "before this turn was answered" sends whoever reads it to the +# wrong end. At the default of one run this also used to clear every path to a +# non-zero exit code and report the session a success. +SOME_REPLY = frame.encode(frame.AUDIO, b"\x00\x00" * 1200) +SESSION_ENDED = frame.encode_json(frame.NOTICE, {"event": "session-ended"}) + + +def turn_cut(label, payload, out_name, at_end=None): + """Take one turn, at the default run count, against a scripted downlink.""" + gate = thread_lib.Event() + cut = wire_client( + ["--host", "desk", "--in-file", os.path.join(TMP, "clip.pcm"), + "--out-file", os.path.join(TMP, out_name), + "--timeout", "2", "--audio-idle", "0.05"], + ScriptedStream(gate, payload, label, at_end), StartGate(gate)) + out, said = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(said): + code = cut.run() + cut.up_q.put(None) + cut.playback.close() + records = [json_lib.loads(line) for line in out.getvalue().splitlines() + if line.strip()] + check(len(records) == 1, + "%s: one turn was taken, so one record belongs in the file: %r" + % (label, records)) + check(cut.options.runs == 1, + "%s: fixture is wrong, this case is about the default run count: %r" + % (label, cut.options.runs)) + return records[0], said.getvalue(), code + + +for label, payload, subject in ( + ("end of stream", b"", "the connection ended"), + ("relay stopped", frame.encode(frame.BYE), "the relay stopped"), + ("session ended", SESSION_ENDED, "the relay ended the session")): + slug = label.replace(" ", "-") + # No audio at all. This half keeps the wording it already had, so the two can + # never be collapsed back into one sentence that fits neither. + silent, _, silent_code = turn_cut( + "%s, nothing played" % label, payload, "reply-silent-%s.pcm" % slug) + check(not silent["answered"] and silent["reply_audio_seconds"] == 0, + "%s: fixture is wrong, this half is the one where nothing played: %r" + % (label, silent)) + check(silent["relay_error"] == "%s before this turn was answered" % subject, + "%s: a turn that got no audio went unanswered and the reason should say " + "so: %r" % (label, silent["relay_error"])) + check(silent_code != 0, + "%s: an unanswered turn must not exit 0, got %r" % (label, silent_code)) + + # Some of the reply played, then the same ending. Same fault, different turn, + # and the record has to describe the turn it actually got. + partial, _, partial_code = turn_cut( + "%s, part of a reply played" % label, SOME_REPLY + payload, + "reply-partial-%s.pcm" % slug) + check(partial["answered"] and partial["reply_audio_seconds"] > 0, + "%s: audio reached the captain, so the turn was answered: %r" + % (label, partial)) + check(partial["first_audio_s"] is not None, + "%s: and when it reached them is a real measurement, not a null: %r" + % (label, partial)) + check(partial["relay_error"] == "%s before the reply finished" % subject, + "%s: a reply cut short must say the reply did not finish: %r" + % (label, partial["relay_error"])) + check("before this turn was answered" not in partial["relay_error"], + "%s: and must not claim nothing arrived, which its own answered field " + "contradicts: %r" % (label, partial["relay_error"])) + # The record and the exit code have to agree with each other as well. + check(partial_code != 0, + "%s: a record naming a fault must not exit 0, even at the default of " + "one run and even though the turn was answered, got %r" + % (label, partial_code)) + +# THE FOURTH SUBJECT, and the one a real captain is likeliest to meet. The three +# above all reach the client as an orderly end: a close, a goodbye frame, or a +# notice. A dropped SSH link is a reset instead, which arrives as a raised error +# from the read rather than as anything the far end chose to send, and that is a +# separate path in the downlink from all three. It has to tell the same two halves +# apart, because the fault is no different from the captain's side. +# +# Its never-played half is the cut-header case far above, which pins the other +# wording, so this is the half that was missing. + + +def a_reset(): + raise OSError(104, "Connection reset by peer") + + +reset, _, reset_code = turn_cut( + "reset, part of a reply played", SOME_REPLY, "reply-partial-reset.pcm", + at_end=a_reset) +check(reset["answered"] and reset["reply_audio_seconds"] > 0, + "a reset after some of the reply played still reached the captain, so the " + "turn was answered: %r" % reset) +check(reset["first_audio_s"] is not None, + "and when it reached them is a real measurement, not a null: %r" % reset) +check(reset["relay_error"].startswith( + "the connection was lost before the reply finished"), + "a reply cut short by a reset must say the reply did not finish, the same " + "as one cut short by a close: %r" % reset["relay_error"]) +check("before this turn was answered" not in reset["relay_error"], + "and must not claim nothing arrived, which its own answered field " + "contradicts: %r" % reset["relay_error"]) +# The clause is what the reader acts on and the detail is what tells a reset from +# a header cut in half, so neither may be lost to the other. +check("Connection reset by peer" in reset["relay_error"], + "and must still carry what the kernel said, which is the only thing that " + "tells a reset from a truncated frame: %r" % reset["relay_error"]) +check(reset_code != 0, + "a record naming a fault must not exit 0, even at the default of one run " + "and even though the turn was answered, got %r" % reset_code) + +# THE OTHER SIDE OF THE SAME LINE, and the complement of the answered case further +# up this block. Those cases are faults that landed while the turn was still owed +# an answer. These two land AFTER the answer was complete, in the gap between the +# reply_end mark arriving and the record being copied, which the client spends +# waiting for the reply audio to go quiet. A turn answered in full must carry no +# reason and must not fail the session, whichever path the late fault takes: an +# end of stream and a reset differ only in what the kernel delivered, and one relay +# death must not produce two different exit codes depending on which it was. +# +# Gated at both ends rather than timed. The fault cannot be delivered until the +# client has passed the reply_end wait, and the record cannot be copied until the +# downlink has marked the connection finished with, which it does only after the +# fault has been applied. Nothing here sleeps and nothing races. +class RaisingAfterReply: + """Serves one whole answer, then raises once the record window is open.""" + + def __init__(self, gate, window): + self._gate = gate + self._window = window + self._bytes = frame.encode(frame.AUDIO, b"\x00\x00" * 1200) + REPLY_END + self._at = 0 + + def read(self, count): + check(self._gate.wait(10), "the turn never opened, so nothing was served") + if self._at < len(self._bytes): + chunk = self._bytes[self._at:self._at + count] + self._at += len(chunk) + return chunk + check(self._window.wait(10), + "fixture: the record window never opened, so nothing raced it") + raise OSError(104, "Connection reset by peer") + + +class FailingAfterReply: + """Serves one whole answer, then a named turn failure in that same window.""" + + def __init__(self, gate, window): + self._gate = gate + self._window = window + self._reply = frame.encode(frame.AUDIO, b"\x00\x00" * 1200) + REPLY_END + self._late = frame.encode_json( + frame.NOTICE, {"event": "turn-failed", + "error": "the model stream dropped"}) + self._at = 0 + self._late_at = 0 + + def read(self, count): + check(self._gate.wait(10), "the turn never opened, so nothing was served") + if self._at < len(self._reply): + chunk = self._reply[self._at:self._at + count] + self._at += len(chunk) + return chunk + check(self._window.wait(10), + "fixture: the record window never opened, so nothing raced it") + if self._late_at < len(self._late): + chunk = self._late[self._late_at:self._late_at + count] + self._late_at += len(chunk) + return chunk + # The notice is handled before this read is reached again, and the end of + # stream behind it is what marks the connection finished with, which is the + # gate the record waits on below. + return b"" + + +def fault_after_answer(label, make_stream, out_name): + """Answer one turn in full, then land a fault before the record is copied.""" + gate, window = thread_lib.Event(), thread_lib.Event() + after = wire_client( + ["--host", "desk", "--in-file", os.path.join(TMP, "clip.pcm"), + "--out-file", os.path.join(TMP, out_name), + "--timeout", "2", "--audio-idle", "0.05"], + make_stream(gate, window), StartGate(gate)) + quiet_wait = after._wait_audio_quiet + + def open_the_window(deadline): + window.set() + quiet_wait(deadline) + # take_turn copies the record on the line after this returns, so the fault + # has to be in before it. closed is the downlink's own mark that it has + # finished with the connection and it is set only after the fault has been + # applied, which makes it the gate rather than any elapsed time. + check(after.closed.wait(10), + "%s: fixture: the late fault never landed before the record" + % label) + + after._wait_audio_quiet = open_the_window + out, said = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(said): + code = after.run() + after.up_q.put(None) + after.playback.close() + records = [json_lib.loads(line) for line in out.getvalue().splitlines() + if line.strip()] + check(len(records) == 1, + "%s: one turn was answered, so one record belongs in the file: %r" + % (label, records)) + check(after.options.runs == 1, + "%s: fixture is wrong, this case is about the default run count: %r" + % (label, after.options.runs)) + check(records[0]["answered"] and records[0]["reply_audio_seconds"] > 0, + "%s: fixture is wrong, the whole reply should have arrived: %r" + % (label, records[0])) + check(records[0]["relay_error"] is None, + "%s: a turn answered in full must carry no reason, whatever arrived " + "afterwards: %r" % (label, records[0]["relay_error"])) + check(code == 0, + "%s: and a session that delivered its answer must exit 0, got %r" + % (label, code)) + return said.getvalue() + + +# The read-failure path: a reset rather than a clean end of stream. +rst_said = fault_after_answer( + "a reset after the answer", RaisingAfterReply, "reply-rst.pcm") +# The guard is on the record and nothing else, so the captain is still told. +check("connection lost" in rst_said, + "the connection going is still said on stderr, only not recorded against a " + "turn it did not cost: %r" % rst_said) + +# The relay fail_turn path: its own model stream broke, after this turn's answer. +failed_said = fault_after_answer( + "a named relay failure after the answer", FailingAfterReply, + "reply-late-fail.pcm") +check("could not finish that turn" in failed_said, + "the relay's own words are still said on stderr for the same reason: %r" + % failed_said) + +# With runs still to take, the loop also has to say why they were not taken, and +# that line has to restate the cause that was recorded rather than asserting a +# default. Told "the relay stopped" and then "the connection closed", the captain +# has been given two causes for one event and has to guess which end to look at. +for label, payload, expected in ( + ("end of stream", b"", "the connection ended"), + ("goodbye", frame.encode(frame.BYE), + "the relay stopped without being asked to")): + _, stopped_said = turn_lost( + "%s, runs remaining" % label, + lambda gate, payload=payload, label=label: ScriptedStream( + gate, payload, "the %s" % label), + "reply-stop-%s.pcm" % label.replace(" ", "-"), runs="2") + check("%s before run 2 of 2" % expected in stopped_said, + "the run loop should restate the recorded cause for %s: %r" + % (label, stopped_said)) + check("the connection closed" not in stopped_said, + "and must not fall back to the default wording for %s: %r" + % (label, stopped_said)) + +# A fault on THIS end, handling a reply that did arrive. The try in the downlink +# used to cover only the read, so the output file refusing the audio, or a payload +# that is not the JSON the wire format promises, killed the reader thread outright +# with the connection neither marked closed nor released. run() then opened every +# remaining turn, each waiting out the whole timeout and printing answered false +# with no reason, so ONE fault cost the session instead of one turn. +# +# The reason it records must also be the right reason: the connection here is +# perfectly healthy, and a record or a message blaming it sends whoever reads it +# to the wrong end of a working link. +class FullDiskPlayback(client.FilePlayback): + """Refuses reply audio the way a filesystem with nothing left does.""" + + def write(self, pcm, turn): + raise OSError(28, "No space left on device") + + +handling_served = thread_lib.Event() +handling = wire_client( + ["--host", "desk", "--in-file", os.path.join(TMP, "clip.pcm"), + "--out-file", os.path.join(TMP, "reply-handling.pcm"), "--runs", "2", + "--timeout", "2", "--audio-idle", "0.05", "--gap-seconds", "0.05"], + ScriptedStream(handling_served, frame.encode(frame.AUDIO, b"\x00\x00" * 600), + "the one reply frame"), + StartGate(handling_served), + FullDiskPlayback(os.path.join(TMP, "reply-handling.pcm"))) +handling_out, handling_said = io.StringIO(), io.StringIO() +with contextlib.redirect_stdout(handling_out), \ + contextlib.redirect_stderr(handling_said): + handling_code = handling.run() +handling.up_q.put(None) +handling.playback.close() +handling_said = handling_said.getvalue() + +handling_runs = [json_lib.loads(line) + for line in handling_out.getvalue().splitlines() if line.strip()] +# One fault, one lost turn. The reader surviving is the whole point: without it +# the second run opens as well and burns another whole timeout for nothing. +check(len(handling_runs) == 1, + "a fault handling a reply must stop the session rather than opening every " + "remaining turn: %d run(s), %r" % (len(handling_runs), handling_runs)) +check(not handling_runs[0]["answered"], + "the audio never reached the file, so the turn was not answered: %r" + % handling_runs[0]) +check(handling_runs[0]["relay_error"], + "a fault this end could name must not be recorded as a turn that merely " + "went unanswered: %r" % handling_runs[0]) +check("No space left on device" in handling_runs[0]["relay_error"], + "the record should carry what actually raised: %r" + % handling_runs[0]["relay_error"]) +# The overriding rule: a wrong cause is worse than a missing one. +check("connection" not in handling_runs[0]["relay_error"], + "the connection was never lost here, so the record must not say it was: %r" + % handling_runs[0]["relay_error"]) +handling_lines = [line for line in handling_said.splitlines() + if line.startswith("client:")] +check(not any("connection" in line for line in handling_lines), + "and the captain must not be sent to a connection that is working: %r" + % handling_lines) +check("No space left on device" in handling_said + and "run 2 of 2" in handling_said, + "the run loop should name the real reason and the run it stopped at: %r" + % handling_said) +# The one line is what the captain and the record get. The traceback is what +# whoever has to find the bug behind it gets, and before the guard existed a dying +# thread printed one, so losing it would make a programming error in the frame +# handling strictly harder to locate than it used to be. +check("Traceback (most recent call last)" in handling_said, + "a fault this end raised should still leave its traceback on stderr: %r" + % handling_said) +check("Traceback" not in (handling_runs[0]["relay_error"] or ""), + "but the record is machine read, so the one-line reason belongs there: %r" + % handling_runs[0]["relay_error"]) +check(handling_code != 0, + "a session that took none of its runs must not exit 0, got %r" + % handling_code) + +# The other end of the same handler: reply audio that arrives after THIS end has +# released the output. close() joins the downlink at five seconds while the relay +# teardown it waits on can take up to ten, so the join can expire with audio still +# in flight, and the chunk behind it then met a closed file. That raised into the +# guard above and printed its fault line and a full traceback on a session that +# answered its turn and exited 0, which is an alarm firing on success: the reader +# learns to skip the line, and the real one is then invisible too. The connection +# was never the problem either, so it was a wrong cause as well as a false one. +# +# Driven through the events the client itself reaches, not a sleep: the turn is +# answered and recorded, the output is released exactly as close() releases it, and +# only then is the late chunk let through. +class LateAudioStream: + """Serves one whole answer, then one more chunk once the output is released.""" + + def __init__(self, opened, released): + self._opened = opened + self._released = released + self._reply = frame.encode(frame.AUDIO, b"\x00\x00" * 1200) + REPLY_END + self._late = frame.encode(frame.AUDIO, b"\x00\x00" * 1200) + self._at = 0 + self._late_at = 0 + self._never = thread_lib.Event() + + def read(self, count): + check(self._opened.wait(10), "the turn never opened, so nothing was served") + if self._at < len(self._reply): + chunk = self._reply[self._at:self._at + count] + self._at += len(chunk) + return chunk + check(self._released.wait(10), + "fixture: the output was never released, so nothing arrived late") + if self._late_at < len(self._late): + chunk = self._late[self._late_at:self._late_at + count] + self._late_at += len(chunk) + return chunk + # Parked rather than ending the stream, so the only lines on stderr are the + # ones this case is about. + self._never.wait(30) + return b"" + + +class CountingPlayback(client.FilePlayback): + """Reports each chunk once it has been handed over, so the test can wait.""" + + def __init__(self, path, handled): + client.FilePlayback.__init__(self, path) + self._handled = handled + self.calls = 0 + + def write(self, pcm, turn): + self.calls += 1 + try: + client.FilePlayback.write(self, pcm, turn) + finally: + # In a finally, so the late chunk raising is as observable as the late + # chunk being discarded and neither outcome hangs the case. + if self.calls >= 2: + self._handled.set() + + +def audio_after_close(label, extra): + """Answer one turn, release the output, then let a late chunk arrive.""" + opened, released, handled = (thread_lib.Event(), thread_lib.Event(), + thread_lib.Event()) + out_file = os.path.join(TMP, "reply-late-%s.pcm" % label) + late = wire_client( + ["--host", "desk", "--in-file", os.path.join(TMP, "clip.pcm"), + "--out-file", out_file, "--timeout", "2", "--audio-idle", "0.05"] + + extra, + LateAudioStream(opened, released), StartGate(opened), + CountingPlayback(out_file, handled)) + out, said = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(said): + code = late.run() + # Exactly what close() does with the output, at the point close() does it. + late.playback.close() + released.set() + check(handled.wait(10), "fixture: the late chunk never reached the output") + # Through close(), not the reporting method directly, so the wiring is + # covered as well as the report: nothing else drives the only production + # caller, and hand-calling it would pass with that call deleted. Returns at + # once here, because the uplink stand-in swallows the quit, no downlink + # thread or relay process was ever assigned, and the file drain is a no-op. + late.close() + late.up_q.put(None) + records = [json_lib.loads(line) for line in out.getvalue().splitlines() + if line.strip()] + check(len(records) == 1, + "%s: one turn was answered, so one record belongs in the file: %r" + % (label, records)) + return late, records[0], said.getvalue(), code, out_file + + +quiet, quiet_run, quiet_said, quiet_code, quiet_file = audio_after_close( + "quiet", []) +# THE POINT. The session worked, so nothing may read as a fault. +check("could not handle the relay's reply" not in quiet_said, + "audio arriving after the output was released is this end's own teardown, " + "not a fault to alarm on: %r" % quiet_said) +check("Traceback" not in quiet_said, + "and it must not leave a traceback behind either: %r" % quiet_said) +check(quiet_code == 0, + "a session whose only turn was answered must still exit 0, got %r" + % quiet_code) +check(quiet_run["answered"] and quiet_run["relay_error"] is None, + "and its record stands, reason-free: %r" % quiet_run) +# Condition 2: the discard is diagnostic only. The late chunk is the same size as +# the answer, so anything crediting it would double both figures. +check(quiet.playback.discarded == 1, + "the late chunk must be counted as discarded: %r" + % quiet.playback.discarded) +check(quiet.playback.turn_bytes == 2400, + "but must not be credited to the turn: %r" % quiet.playback.turn_bytes) +check(quiet_run["reply_audio_seconds"] == 0.05, + "so the reply's own duration stands: %r" % quiet_run["reply_audio_seconds"]) +check(quiet_run["first_audio_s"] is not None, + "and the answer's real first-audio figure stands: %r" + % quiet_run["first_audio_s"]) +check(os.path.getsize(quiet_file) == 2400, + "and the discarded chunk reached no file: %d bytes" + % os.path.getsize(quiet_file)) + +# Condition 1: discarded, not silently swallowed. A write after close outside +# teardown is a real logic bug, so the count has somewhere to be read. +loud, _, loud_said, loud_code, _ = audio_after_close("loud", ["--verbose"]) +check("discarded 1 reply audio chunk" in loud_said, + "--verbose must say how many chunks were dropped: %r" % loud_said) +check("could not handle the relay's reply" not in loud_said + and "Traceback" not in loud_said, + "and saying so is not the same as calling it a fault: %r" % loud_said) +check(loud_code == 0 and loud.playback.turn_bytes == 2400, + "and reporting it changes neither the exit code nor the turn's bytes: %r %r" + % (loud_code, loud.playback.turn_bytes)) + +# Which of the playback's two locks covers what, driven rather than asserted about. +# take_turn calls turn_reset while holding the client's own turn lock, so anything +# turn_reset can wait behind stalls the whole client: neither the downlink nor the +# sender can stamp a thing without that lock. The file write is the one slow step +# here, so it must sit outside the lock turn_reset takes. The handle below parks +# instead of writing, which makes a slow filesystem exact rather than simulated. +parked, release_write, reset_done = (thread_lib.Event(), thread_lib.Event(), + thread_lib.Event()) + + +class ParkingHandle: + """A file whose write blocks until this case lets it through.""" + + def __init__(self): + self.written = 0 + + def write(self, pcm): + self.written += len(pcm) + parked.set() + check(release_write.wait(10), "fixture: the parked write was never freed") + + def close(self): + pass + + +slow = client.FilePlayback(os.path.join(TMP, "reply-slow.pcm")) +slow._handle.close() +slow._handle = ParkingHandle() +slow.turn_reset(1) +thread_lib.Thread(target=slow.write, args=(b"\x00\x00" * 600, 1), + daemon=True).start() +check(parked.wait(10), "fixture: the write never reached the handle") + + +def advance_the_turn(): + slow.turn_reset(2) + reset_done.set() + + +thread_lib.Thread(target=advance_the_turn, daemon=True).start() +# Bounded, and it discriminates in both directions: with the write outside that +# lock the reset completes at once, and with the write inside it the reset cannot +# complete until the line below runs, whatever the machine is doing. +check(reset_done.wait(5), + "a turn advance must not wait behind a file write, because take_turn makes " + "it while holding the lock the downlink and the sender both need") +release_write.set() +# The accounting still had to happen, and under the lock: the chunk was turn one's. +check(slow.turn_bytes == 0 and slow.first_played is None, + "and turn two starts owed nothing and unstamped: %r %r" + % (slow.turn_bytes, slow.first_played)) + +# The record's own two figures for one turn's audio. reply_audio_seconds and +# answered are the same measurement asked twice, and the record is built outside +# the playback's lock, so reading that measurement twice lets a chunk land between +# the two reads and produce a record saying both that no audio arrived and that the +# turn was answered. The reader of runs.jsonl then has to guess which half to +# believe, which is the whole failure this work exists to remove. +# +# The playback below is that interleaving made exact rather than raced for: the +# count grows between one read of it and the next, so a record built from two reads +# cannot agree with itself and a record built from one always does. +class SlippingCount(client.FilePlayback): + """A playback whose byte count grows between one read of it and the next.""" + + def __init__(self, path): + self.reads = 0 + client.FilePlayback.__init__(self, path) + + @property + def turn_bytes(self): + self.reads += 1 + return self._counted if self.reads == 1 else self._counted + 2400 + + @turn_bytes.setter + def turn_bytes(self, count): + self._counted = count + + +slip_open, slip_parked = thread_lib.Event(), thread_lib.Event() + + +def park_the_downlink(): + # Parked rather than ended, because an end of stream would make the downlink + # name a reason, and naming one reads the byte count itself. The turn has to + # reach its record with the count still unread by anything else. + slip_parked.wait(30) + + +slipping = SlippingCount(os.path.join(TMP, "reply-slip.pcm")) +# The timeout boundary is the one moment the first chunk of a reply can land while +# the record is being built, because a turn that timed out with nothing played is +# the only one whose count is still zero when the record reads it. +slipped = wire_client( + ["--host", "desk", "--in-file", os.path.join(TMP, "clip.pcm"), + "--out-file", os.path.join(TMP, "reply-slip.pcm"), + "--timeout", "0.3", "--audio-idle", "0.05"], + ScriptedStream(slip_open, b"", "nothing at all", park_the_downlink), + StartGate(slip_open), slipping) +slip_out, slip_said = io.StringIO(), io.StringIO() +with contextlib.redirect_stdout(slip_out), contextlib.redirect_stderr(slip_said): + slip_code = slipped.run() +slipped.up_q.put(None) +slip_records = [json_lib.loads(line) for line in slip_out.getvalue().splitlines() + if line.strip()] +check(len(slip_records) == 1, + "the turn was taken, so its record belongs in the file: %r" % slip_records) +slip = slip_records[0] +check(slipping.reads >= 1, + "fixture: the record never read the byte count at all, so this case proves " + "nothing: %r" % slipping.reads) +# THE POINT. One record may not answer the same question two ways. +check(slip["answered"] == (slip["reply_audio_seconds"] > 0), + "a record that counts no reply audio must not also call the turn answered: " + "%r" % slip) +check(slip["reply_audio_seconds"] == 0.0 and slip["answered"] is False, + "and the figures are the ones true when the record was built, not one from " + "before the chunk and one from after: %r" % slip) +check(slip_code != 0, + "an unanswered turn must not exit 0, got %r" % slip_code) +# Released under a redirect so the end of stream this case parked cannot print into +# the suite's own output, and waited for so it cannot land after the case has ended. +with contextlib.redirect_stderr(io.StringIO()): + slip_parked.set() + slip_drained = slipped.closed.wait(10) +check(slip_drained, "fixture: the parked stream never ended") + +# An end of stream arriving during THIS end's own teardown. close() bounds the +# relay's exit and the relay's own teardown can outlast that bound, so the child is +# killed with no goodbye written and the still-live downlink reads the empty stream +# it left behind. Every turn was answered and the session exits 0, so the mid-turn +# fault line must not be the last thing the captain reads: an alarm that also fires +# on success is one they learn to skip past, and then the real one is invisible too. +# +# The discriminator is the same one the goodbye branch uses, whether this end asked +# to stop, and the mid-session case further up this file is the other direction of +# it: there the stream ends long before close() is reached, so the line still fires. +class QuitThenEnd: + """Serves one whole answer, then ends the stream once this end says goodbye.""" + + def __init__(self, opened, quit_seen): + self._opened = opened + self._quit = quit_seen + self._bytes = frame.encode(frame.AUDIO, b"\x00\x00" * 1200) + REPLY_END + self._at = 0 + + def read(self, count): + check(self._opened.wait(10), "the turn never opened, so nothing was served") + if self._at < len(self._bytes): + chunk = self._bytes[self._at:self._at + count] + self._at += len(chunk) + return chunk + check(self._quit.wait(10), + "fixture: this end never asked to stop, so the stream never ended") + return b"" + + +class GoodbyeWatchingGate: + """Opens on the turn's first frame and reports this end asking to stop.""" + + def __init__(self, opened, quit_seen): + self._opened = opened + self._quit = quit_seen + + def send(self, kind, payload=b""): + if kind == frame.TALK_START: + self._opened.set() + elif kind == frame.QUIT: + self._quit.set() + + +tidy_open, tidy_quit = thread_lib.Event(), thread_lib.Event() +tidy = wire_client( + ["--host", "desk", "--in-file", os.path.join(TMP, "clip.pcm"), + "--out-file", os.path.join(TMP, "reply-tidy.pcm"), + "--timeout", "2", "--audio-idle", "0.05"], + QuitThenEnd(tidy_open, tidy_quit), GoodbyeWatchingGate(tidy_open, tidy_quit)) +tidy_out, tidy_said = io.StringIO(), io.StringIO() +with contextlib.redirect_stdout(tidy_out), contextlib.redirect_stderr(tidy_said): + tidy_code = tidy.run() + # Gated on the downlink reaching the end of stream rather than on any elapsed + # time: close() does not wait for it here, so without this the case could pass + # by asserting on output the downlink had not written yet. + tidy.close() + tidy_drained = tidy.closed.wait(10) +tidy.up_q.put(None) +check(tidy_drained, "fixture: the downlink never saw the stream end") +tidy_records = [json_lib.loads(line) for line in tidy_out.getvalue().splitlines() + if line.strip()] +check(len(tidy_records) == 1, + "one turn was answered, so one record belongs in the file: %r" % tidy_records) +# Proof the end of stream really was taken, so the silence below is a decision +# rather than a branch this case never reached: the cause is recorded outside the +# test that decides whether to speak. +check(tidy.closed_because == "the connection ended", + "fixture: the downlink did not take the end of stream at all: %r" + % tidy.closed_because) +# THE POINT. The session answered everything asked of it, so nothing may read as a +# fault, least of all as the last line the captain sees. +check("the connection ended" not in tidy_said.getvalue(), + "an end of stream during this end's own teardown is the relay doing as it " + "was told, not a fault to alarm on: %r" % tidy_said.getvalue()) +check(tidy_code == 0, + "and a session whose only turn was answered must still exit 0, got %r" + % tidy_code) +check(tidy_records[0]["answered"] and tidy_records[0]["relay_error"] is None, + "and its record stands, reason-free: %r" % tidy_records[0]) + +# A reply that is handled AFTER the turn it belongs to has already been recorded. +# The downlink reads a frame on one thread and applies it on another, so a turn +# that times out while a notice is in flight used to have that notice applied to +# whatever turn came next: it named the new turn with the old turn's fault, and +# released it before its own answer had arrived. A turn that would have been +# answered normally was reported unanswered, carrying a reason belonging to a turn +# the captain had already been told about. +# +# Driven through the client's own say(), not a sleep: the notice is parked exactly +# where the real window is, after the frame has arrived and before its handler +# takes the lock, and it is released only once the NEXT turn has really opened, +# which the uplink reports when that turn's talk start goes out. +class TurnCountingGate: + """Reports the first turn opening, and the second one separately.""" + + def __init__(self, first, second): + self._first = first + self._second = second + self.starts = 0 + + def send(self, kind, payload=b""): + if kind == frame.TALK_START: + self.starts += 1 + (self._first if self.starts == 1 else self._second).set() + + +stale_open, stale_next = thread_lib.Event(), thread_lib.Event() +# Wired without starting its threads, because the notice has to be parked before +# the downlink is allowed to read it. +stale = wire_client( + ["--host", "desk", "--in-file", os.path.join(TMP, "clip.pcm"), + "--out-file", os.path.join(TMP, "reply-stale.pcm"), "--runs", "2", + "--timeout", "2", "--audio-idle", "0.05", "--gap-seconds", "0.05"], + ScriptedStream( + stale_open, + frame.encode_json(frame.NOTICE, {"event": "turn-failed", + "error": "the model dropped turn one"}) + + frame.encode(frame.AUDIO, b"\x00\x00" * 1200) + REPLY_END, + "turn one's failure and turn two's answer"), + TurnCountingGate(stale_open, stale_next), start=False) + +# The notice is held between arriving and being applied, which is the window the +# turn identity exists to close. Turn one then times out and is recorded, turn two +# opens, and only then is the notice allowed to finish being handled. +said_plainly = client.say + + +def park_the_notice(message): + said_plainly(message) + if "could not finish that turn" in message: + check(stale_next.wait(10), + "fixture: the second turn never opened, so nothing went stale") + + +client.say = park_the_notice +try: + thread_lib.Thread(target=stale._sender, daemon=True).start() + thread_lib.Thread(target=stale._downlink, daemon=True).start() + stale_out, stale_said = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(stale_out), contextlib.redirect_stderr(stale_said): + stale_code = stale.run() +finally: + client.say = said_plainly +stale.up_q.put(None) +stale.playback.close() + +stale_runs = [json_lib.loads(line) for line in stale_out.getvalue().splitlines() + if line.strip()] +check(len(stale_runs) == 2, + "both turns were taken, so both belong in the file: %d, %r" + % (len(stale_runs), stale_runs)) +check(not stale_runs[0]["answered"], + "fixture is wrong: the first turn should time out unanswered: %r" + % stale_runs[0]) +# THE POINT. The stale notice belonged to turn one and must touch nothing else. +check(stale_runs[1]["relay_error"] is None, + "a fault from a turn that has already ended must not be recorded against " + "the next one: %r" % stale_runs[1]) +check(stale_runs[1]["answered"] and stale_runs[1]["reply_audio_seconds"] > 0, + "and the next turn must be left to run normally rather than released " + "before its answer arrived: %r" % stale_runs[1]) +check(stale_code != 0, + "one turn of two was still lost, so the exit code must say so, got %r" + % stale_code) + +# The same window, with AUDIO in it instead of a notice, which is the worse half. +# The frame's timing marks are taken under the turn lock, but the audio itself goes +# to the playback, and the playback is where first_played is stamped, where the +# reply's duration is counted, and where answered comes from. Credited to whatever +# turn is open, one stale chunk gives a turn that was never answered a headline +# latency figure measured from somebody else's reply, reports it answered, and with +# that the whole session exits 0 having lost a turn. +# +# Parked at the playback rather than at say(), because that IS the accounting +# point: the write is released only once the next turn has really opened, which the +# uplink reports when that turn's talk start goes out. No sleeps. +class LatePlayback(client.FilePlayback): + """Holds the first chunk until the turn after the one it belongs to has opened.""" + + def __init__(self, path, opened): + client.FilePlayback.__init__(self, path) + self._opened = opened + self._held = False + + def write(self, pcm, turn): + if not self._held: + self._held = True + check(self._opened.wait(10), + "fixture: the next turn never opened, so nothing went stale") + client.FilePlayback.write(self, pcm, turn) + + +late_open, late_next = thread_lib.Event(), thread_lib.Event() +late_out_file = os.path.join(TMP, "reply-late.pcm") +late = wire_client( + ["--host", "desk", "--in-file", os.path.join(TMP, "clip.pcm"), + "--out-file", late_out_file, "--runs", "2", + "--timeout", "2", "--audio-idle", "0.05", "--gap-seconds", "0.05"], + ScriptedStream(late_open, frame.encode(frame.AUDIO, b"\x00\x00" * 1200), + "turn one's audio"), + TurnCountingGate(late_open, late_next), + LatePlayback(late_out_file, late_next)) +late_out, late_said = io.StringIO(), io.StringIO() +with contextlib.redirect_stdout(late_out), contextlib.redirect_stderr(late_said): + late_code = late.run() +late.up_q.put(None) +late.playback.close() + +late_runs = [json_lib.loads(line) for line in late_out.getvalue().splitlines() + if line.strip()] +check(len(late_runs) == 2, + "both turns were taken, so both belong in the file: %d, %r" + % (len(late_runs), late_runs)) +# THE POINT. Turn two was served nothing at all, and the stale chunk must not make +# it look otherwise on any of the four figures the playback feeds. +second = late_runs[1] +check(not second["answered"], + "a turn served no audio of its own must not be reported answered because " + "an earlier turn's chunk arrived during it: %r" % second) +check(second["reply_audio_seconds"] == 0, + "and none of that chunk's duration belongs to it: %r" % second) +check(second["first_played_s"] is None, + "and it must not be stamped with when that chunk reached the output: %r" + % second) +check(second["first_audio_s"] is None, + "and the headline figure, which prefers the playback stamp, must be absent " + "too rather than measured from another turn's reply: %r" % second) +check(second["first_frame_s"] is None, + "and the frame stamp stays absent, as the turn identity already ensured: %r" + % second) +check(late_code != 0, + "a session that lost a turn must not exit 0, got %r" % late_code) +# And the audio was still written. Attributing it to nobody must not mean dropping +# it: it is the tail of an answer the captain is still listening to. +check(os.path.getsize(late_out_file) == 2400, + "the late chunk must still reach the output, not be discarded to make the " + "figures tidy: %d bytes" % os.path.getsize(late_out_file)) + +# The wait between turns is the last thing a lost session should do. It exists +# only to avoid talking over the model's own speech, and a relay that is already +# gone is not speaking, so waiting it out leaves the captain sitting through the +# previous reply's whole spoken duration before anyone tells them the session +# stopped. The gap here is deliberately long and the reply deliberately short, so +# the elapsed time distinguishes stopping from waiting; nothing in the case sleeps +# on purpose, the code under test is the sleep. +prompt_served = thread_lib.Event() +prompt = wire_client( + ["--host", "desk", "--in-file", os.path.join(TMP, "clip.pcm"), + "--out-file", os.path.join(TMP, "reply-prompt.pcm"), "--runs", "3", + "--timeout", "2", "--audio-idle", "0.05", "--gap-seconds", "5"], + ScriptedStream(prompt_served, + frame.encode(frame.AUDIO, b"\x00\x00" * 1200) + REPLY_END, + "one whole answer and then the end of stream"), + StartGate(prompt_served)) +prompt_out, prompt_said = io.StringIO(), io.StringIO() +prompt_began = clock.monotonic() +with contextlib.redirect_stdout(prompt_out), contextlib.redirect_stderr(prompt_said): + prompt_code = prompt.run() +prompt_took = clock.monotonic() - prompt_began +prompt.up_q.put(None) +prompt.playback.close() + +prompt_runs = [json_lib.loads(line) for line in prompt_out.getvalue().splitlines() + if line.strip()] +check(len(prompt_runs) == 1 and prompt_runs[0]["answered"], + "fixture is wrong: one answered turn should be taken and no more: %r" + % prompt_runs) +check(prompt_code != 0, + "a session that took 1 of 3 runs must not exit 0, got %r" % prompt_code) +check(prompt_took < 3, + "a session that already knows the connection is gone must stop rather than " + "wait out the 5s gap first: took %.2fs" % prompt_took) +check("the connection ended before run 2 of 3" in prompt_said.getvalue(), + "and it should name the cause and the first run it lost: %r" + % prompt_said.getvalue()) + +# The speaker's byte ACCOUNTING, which is not the speaker. No audio device is +# reached here and none can be on this host: the stream is a stub and the device +# callback is called by hand, so nothing below says anything about how a real +# output device behaves. What it does cover is the arithmetic deciding which turn +# a chunk is credited to and whose first-audio clock it stamps, which until now +# was the one piece of that logic with no coverage at all while the file path it +# mirrors had plenty. Its correctness rests on the earlier turns' bytes being a +# prefix of the buffer, and a sign or bound slip in the prefix count would +# silently mismeasure the headline latency on the path the captain will use. +class StubStream: + """Stands in for the output device: accepts the settings, plays nothing.""" + + def __init__(self, **settings): + self.settings = settings + self.latency = 0.011 + self.started = False + + def start(self): + self.started = True + + def stop(self): + pass + + def close(self): + pass + + +stub_audio = types.ModuleType("sounddevice") +stub_audio.RawOutputStream = StubStream +sys.modules["sounddevice"] = stub_audio + + +def fresh_speaker(): + return client.SpeakerPlayback() + + +def pull(speaker, frames): + """Ask the stub device for one block, and return exactly what it was handed.""" + block = bytearray(frames * 2) + speaker._callback(block, frames, None, None) + return bytes(block) + + +opened = fresh_speaker() +check(opened.device_latency == 0.011, + "the reported device latency should come from the stream: %r" + % opened.device_latency) + +# No earlier bytes at all: the first chunk of this turn's own reply is this turn's +# first audio. +speaker = fresh_speaker() +speaker.turn_reset(1) +speaker.write(b"\x01\x01" * 300, 1) +check(speaker.turn_bytes == 600, + "this turn's own audio is credited to it: %r" % speaker.turn_bytes) +check(pull(speaker, 300) == b"\x01\x01" * 300, "and is played unchanged") +check(speaker.first_played is not None, + "and stamps this turn's first audio when it reaches the device") + +# An empty buffer. Silence handed to the device is not the reply arriving, so the +# count taken and the count skipped are equal at zero and nothing is stamped. +speaker = fresh_speaker() +speaker.turn_reset(1) +check(pull(speaker, 100) == b"\x00" * 200, + "an empty buffer is padded with silence") +check(speaker.first_played is None, + "and stamps nothing: no reply audio has reached the device yet") + +# Earlier bytes exactly equal to the block taken. The whole block belongs to a turn +# already recorded, so it plays and stamps nothing. +speaker = fresh_speaker() +speaker.turn_reset(2) +speaker.write(b"\x02\x02" * 300, 1) +check(speaker.turn_bytes == 0, + "a chunk from an earlier turn is credited to nobody: %r" % speaker.turn_bytes) +check(pull(speaker, 300) == b"\x02\x02" * 300, + "but is still played, because it is the tail of an answer being listened to") +check(speaker.first_played is None, + "and must not stamp the later turn's first audio") + +# Earlier bytes larger than the block taken, drained across two blocks. The count +# has to come down by what was taken and no more, or the turn's real first audio +# is stamped by somebody else's tail. +speaker = fresh_speaker() +speaker.turn_reset(2) +speaker.write(b"\x03\x03" * 300, 1) +speaker.write(b"\x04\x04" * 300, 1) +check(pull(speaker, 150) == b"\x03\x03" * 150, "the earlier tail plays in order") +check(speaker.first_played is None, "and stamps nothing on the first block") +check(pull(speaker, 150) == b"\x03\x03" * 150, "nor on the second") +check(speaker.first_played is None, "still nothing stamped") +check(pull(speaker, 300) == b"\x04\x04" * 300, "nor while the rest of it drains") +check(speaker.first_played is None, + "1200 earlier bytes must take 1200 bytes to drain, not one block") +speaker.write(b"\x05\x05" * 300, 2) +check(pull(speaker, 300) == b"\x05\x05" * 300, "then this turn's own reply plays") +check(speaker.first_played is not None, "and that is what stamps the clock") + +# Earlier bytes smaller than the block taken, so one block spans the boundary. The +# first byte past the earlier tail is this turn's first audio, and it is inside a +# block that started with somebody else's. +speaker = fresh_speaker() +speaker.turn_reset(2) +speaker.write(b"\x06\x06" * 150, 1) +speaker.write(b"\x07\x07" * 150, 2) +check(pull(speaker, 300) == b"\x06\x06" * 150 + b"\x07\x07" * 150, + "a block spanning the boundary is played whole") +check(speaker.first_played is not None, + "and the first byte past the earlier tail stamps this turn's first audio") +check(speaker.turn_bytes == 300, + "with only this turn's half credited to it: %r" % speaker.turn_bytes) + +# A turn advancing while the previous answer is still queued, which is the case +# turn_reset itself has to handle: everything already buffered belongs to the turn +# that is being closed, however it got there. +speaker = fresh_speaker() +speaker.turn_reset(1) +speaker.write(b"\x08\x08" * 300, 1) +check(speaker.turn_bytes == 600, "turn one is credited its own reply") +speaker.turn_reset(2) +check(speaker.turn_bytes == 0 and speaker.first_played is None, + "turn two starts owed nothing and unstamped: %r %r" + % (speaker.turn_bytes, speaker.first_played)) +check(pull(speaker, 300) == b"\x08\x08" * 300, + "turn one's undrained tail still reaches the device") +check(speaker.first_played is None, + "but it must not stamp turn two, which is what it would have done before " + "turn_reset counted what was already queued") +speaker.write(b"\x09\x09" * 300, 2) +check(pull(speaker, 300) == b"\x09\x09" * 300 and speaker.first_played is not None, + "and turn two's own reply is what stamps turn two") +check(speaker.turn_bytes == 600, "credited to turn two: %r" % speaker.turn_bytes) + +# A short buffer of earlier bytes: padded with silence, and the padding is not +# mistaken for this turn's reply either. +speaker = fresh_speaker() +speaker.turn_reset(2) +speaker.write(b"\x0a\x0a" * 100, 1) +check(pull(speaker, 300) == b"\x0a\x0a" * 100 + b"\x00" * 400, + "a short earlier tail is padded rather than repeated") +check(speaker.first_played is None, "and the padding stamps nothing") + +# A chunk arriving after close, which is the same teardown race the file path +# already discards and counts. Nothing here goes near a device: the stream is the +# same stub, and stopping it is what makes the chunk unplayable, so queuing it +# would be recording a measurement the captain never heard. +speaker = fresh_speaker() +speaker.turn_reset(1) +speaker.write(b"\x0b\x0b" * 300, 1) +speaker.turn_reset(2) +speaker.write(b"\x0c\x0c" * 150, 2) +was = (speaker.turn_bytes, speaker._earlier, speaker.first_played, + bytes(speaker._buffer)) +check(was == (300, 600, None, b"\x0b\x0b" * 300 + b"\x0c\x0c" * 150), + "fixture: turn two should be owed its own bytes behind turn one's tail: %r" + % (was,)) +speaker.close() +speaker.write(b"\x0d\x0d" * 300, 2) +check(speaker.discarded == 1, + "a chunk arriving after close must be counted, not silently swallowed: %r" + % speaker.discarded) +check(bytes(speaker._buffer) == was[3], + "and must not be queued for a stream that has already stopped") +check((speaker.turn_bytes, speaker._earlier, speaker.first_played) == was[:3], + "and must leave every per-turn figure exactly as it was: %r" + % ((speaker.turn_bytes, speaker._earlier, speaker.first_played),)) + +# And the count reaches the captain on this path through the same close() that +# reports it on the file path, so the counter is not merely present here. +heard = fresh_speaker() +heard.turn_reset(1) +heard.write(b"\x0e\x0e" * 300, 1) +check(pull(heard, 300) == b"\x0e\x0e" * 300, + "fixture: the turn's own reply should drain before the output is released") +heard.close() +heard.write(b"\x0f\x0f" * 300, 1) +reported = client.Client(client.parse_args(["--host", "desk", "--verbose"])) +reported.playback = heard +speaker_said = io.StringIO() +with contextlib.redirect_stderr(speaker_said): + reported.close() +check("discarded 1 reply audio chunk" in speaker_said.getvalue(), + "--verbose must report the count on this path as it does on the file one: %r" + % speaker_said.getvalue()) + +del sys.modules["sounddevice"] + + + +# A startup that refuses part way through releases what it already started, and +# close() therefore has to survive a half-built client. The real devices cannot be +# opened on this host, so these stand in for them; what is tested here is the +# release path and the refusal, not the devices themselves. +class Recorder: + def __init__(self, closed): + self._closed = closed + self.first_played = None + self.device_latency = None + self.discarded = 0 + + def drain(self, timeout=5): + pass + + def close(self): + self._closed.append("closed") + + def start(self, out_q, talking): + pass + +# Nothing built yet: close() must not trip over the fields that are still None. +client.Client(client.parse_args(["--host", "desk"])).close() + +# Built part way, then refused: whatever was started is released once. +half = client.Client(client.parse_args(["--host", "desk"])) +speaker, microphone = [], [] +half.playback = Recorder(speaker) +half.capture = Recorder(microphone) +half.close() +check(speaker == ["closed"] and microphone == ["closed"], + "a half-built client must release both devices: %r %r" % (speaker, microphone)) + +# open() releases them itself when a later step refuses, so no caller has to. +refusing = client.Client(client.parse_args(["--host", "desk"])) +speaker, microphone = [], [] + +def half_start(): + refusing.playback = Recorder(speaker) + refusing.capture = Recorder(microphone) + raise SystemExit("fm-voice-client: the relay closed the connection") + +refusing._start = half_start +raised = None +try: + refusing.open() +except SystemExit as exc: + raised = str(exc) +check(raised is not None, "open must not swallow the refusal") +check(speaker == ["closed"] and microphone == ["closed"], + "open must release the devices it started: %r %r" % (speaker, microphone)) + +# A device that cannot be opened is one named line with a next step, not a +# traceback, because this is the path the guide warns will fail first. The relay +# side is stubbed out here so nothing is launched: the subject is the refusal. +class Boom: + def __init__(self, *args, **kwargs): + raise RuntimeError("PortAudio said no") + +class FakeProc: + def __init__(self, *args, **kwargs): + self.stdin = io.BytesIO() + self.stdout = io.BytesIO(frame.MAGIC) + + def wait(self, timeout=None): + return 0 + + def kill(self): + pass + +real_speaker = client.SpeakerPlayback +real_popen = client.subprocess.Popen +real_sync = client.sync_magic +client.SpeakerPlayback = Boom +client.subprocess.Popen = FakeProc +client.sync_magic = lambda stream, verbose=False: None + +def refusal_for(argv): + """Return how open() refuses, as (exception type name, message).""" + try: + client.Client(client.parse_args(["--host", "desk"] + argv)).open() + except BaseException as exc: # noqa: BLE001 + return type(exc).__name__, str(exc) + return None, "open() did not refuse" + +kind, named = refusal_for([]) +try: + check(kind == "DeviceError", + "a device failure should be a named refusal, got %s: %s" % (kind, named)) + check("could not open the audio device" in named, + "and should say what it could not open: %s" % named) + check("--output-device" in named, + "and should name the flag for that end: %s" % named) + check("--in-file" in named, + "and should name a way to run without a device: %s" % named) + + # The file ends are the ones this host runs and the ones every measured + # figure was taken with, so a path that cannot be opened must say so and name + # the flag that chose it. Calling it a device failure sends the reader to + # --input-device when the thing to fix is the path. + missing = os.path.join(TMP, "no-such-clip.pcm") + kind, named = refusal_for(["--in-file", missing, "--out-file", + os.path.join(TMP, "reply.pcm")]) + check(kind in ("OSError", "FileNotFoundError"), + "a missing clip is not a device failure, got %s: %s" % (kind, named)) + check(missing in named, "the refusal must name the path: %s" % named) + check("--in-file" in named and "--output-device" not in named, + "and the flag that named it, and no device advice: %s" % named) + + nowhere = os.path.join(TMP, "no-such-dir", "reply.pcm") + kind, named = refusal_for(["--in-file", os.path.join(TMP, "clip.pcm"), + "--out-file", nowhere]) + check(kind in ("OSError", "FileNotFoundError"), + "an unwritable reply file is not a device failure either: %s" % named) + check(nowhere in named and "--out-file" in named, + "and must name the path and its flag: %s" % named) +finally: + client.SpeakerPlayback = real_speaker + client.subprocess.Popen = real_popen + client.sync_magic = real_sync + +# A login shell banner is discarded with a warning naming it, not an error. +noise = b"You have mail.\n" +stream = io.BytesIO(noise + frame.MAGIC + frame.encode(frame.BYE)) +client.sync_magic(stream) +check(frame.Reader(stream).read()[0] == frame.BYE, + "the first frame after the handshake must still be readable") + +# Junk with no handshake at all must be a named refusal rather than a hang. +try: + client.sync_magic(io.BytesIO(b"x" * (client.MAX_PREAMBLE + 64))) + sys.exit("client: accepted a stream with no handshake") +except frame.FrameError as exc: + check("not fm-voice-relay.py" in str(exc), + "the refusal should say what is on the far end: %s" % exc) + +# A far end that dies before saying hello must say that, because the useful next +# step is running the relay command by hand. +try: + client.sync_magic(io.BytesIO(b"")) + sys.exit("client: accepted a closed stream") +except frame.FrameError as exc: + check("before it said hello" in str(exc), + "the refusal should name the early close: %s" % exc) + +# The two ends must agree on the sample rates, or the reply plays at the wrong +# pitch and nothing reports an error. +relay_spec = importlib.util.spec_from_file_location( + "relay", str(pathlib.Path(sys.argv[1]) / "fm-voice-relay.py")) +relay = importlib.util.module_from_spec(relay_spec) +relay_spec.loader.exec_module(relay) +check((client.IN_RATE, client.OUT_RATE) == (relay.IN_RATE, relay.OUT_RATE), + "the two ends disagree on the sample rates") +PY +pass "the laptop client builds the right remote command and survives a chatty login shell" + +# docs/voice-relay.md tells the captain to copy exactly two files to the laptop, +# so the real property is that the client runs from a directory holding exactly +# those two and nothing else from bin/. A third local import would leave that +# instruction wrong and the laptop dying at import time, a long way from the +# change that caused it. Run there with no PYTHONPATH, so bin/ cannot supply the +# missing piece the way it does on this host. +LAPTOP="$TMP_ROOT/laptop" +mkdir -p "$LAPTOP" +cp "$ROOT/bin/fm-voice-client.py" "$ROOT/bin/fm_voice_frame.py" "$LAPTOP/" + +set +e +copied_out=$(cd "$LAPTOP" && env -u PYTHONPATH python3 ./fm-voice-client.py --help 2>&1) +copied_code=$? +set -e +expect_code 0 "$copied_code" \ + "the client must start with only the two copied files: $copied_out" +assert_contains "$copied_out" 'fm-voice-client.py' \ + "the copied client should print its own usage" + +# The negative half, so the case above is not passing because bin/ was reachable +# after all: without its one companion the client must fail at import and name it. +SHORT="$TMP_ROOT/laptop-missing-companion" +mkdir -p "$SHORT" +cp "$ROOT/bin/fm-voice-client.py" "$SHORT/" +set +e +short_out=$(cd "$SHORT" && env -u PYTHONPATH python3 ./fm-voice-client.py --help 2>&1) +short_code=$? +set -e +[ "$short_code" -ne 0 ] || fail "the client started without fm_voice_frame.py beside it" +assert_contains "$short_out" 'fm_voice_frame' \ + "the import failure should name the file the laptop is missing" +pass "the client runs from a laptop holding only the two files the guide names" + +# --listen open-mic refuses, out loud and early. The mode has no way to tell when +# the captain stopped speaking, so it would stream a turn that never ends; the +# captain should be told that rather than watching it half work. Early matters as +# much as loud: an ssh stub here records any attempt to reach the desktop, and the +# refusal must come before it, so nothing is opened and nothing is spent. +OPENMIC_FAKEBIN=$(fm_fakebin "$TMP_ROOT/openmic-fake") +SSH_CALLED="$TMP_ROOT/ssh-was-called" +cat > "$OPENMIC_FAKEBIN/ssh" <<SH +#!/usr/bin/env bash +printf 'ssh %s\n' "\$*" >> "$SSH_CALLED" +exit 9 +SH +chmod +x "$OPENMIC_FAKEBIN/ssh" + +set +e +openmic_out=$(PATH="$OPENMIC_FAKEBIN:$PATH" python3 "$ROOT/bin/fm-voice-client.py" \ + --host a-desktop --relay /desktop/bin/fm-voice-relay.py --listen open-mic 2>&1) +openmic_code=$? +set -e +[ "$openmic_code" -ne 0 ] || fail "--listen open-mic started instead of refusing" +assert_contains "$openmic_out" 'end-of-speech' \ + "the refusal should name the missing piece: $openmic_out" +assert_contains "$openmic_out" 'push-to-talk' \ + "the refusal should name the mode that does work: $openmic_out" +assert_absent "$SSH_CALLED" \ + "the refusal must come before anything reaches the desktop" +# The default still starts far enough to try the connection, so the case above is +# a property of the setting rather than of the fixture refusing everything. +set +e +PATH="$OPENMIC_FAKEBIN:$PATH" python3 "$ROOT/bin/fm-voice-client.py" \ + --host a-desktop --relay /desktop/bin/fm-voice-relay.py --talk-seconds 0 \ + >/dev/null 2>&1 +set -e +assert_present "$SSH_CALLED" \ + "fixture is wrong: push to talk should have reached the ssh stub" +pass "--listen open-mic refuses at startup, before it opens anything" + +# --- read scope ------------------------------------------------------------- + +narrow=$(records_status --scope counts) || fail "counts scope failed" +assert_contains "$narrow" '"scope": "counts"' "counts scope should say so" +assert_contains "$narrow" '"in_flight": 3' "counts scope should still count in-flight work" +assert_contains "$narrow" '"queued": 2' "counts scope should still count queued work" +assert_contains "$narrow" '"awaiting_captain": 2' "counts scope should count what waits on the captain" +assert_contains "$narrow" '"open_pull_requests": 1' "counts scope should count open pull requests" +# No record free text is assembled at all at this scope, so there is nothing to +# filter and nothing to get wrong. +assert_not_contains "$narrow" 'alpha-one' "counts scope must not name work" +assert_not_contains "$narrow" 'sign-in redirect' "counts scope must not carry titles" +assert_not_contains "$narrow" 'github.com' "counts scope must not carry pull request links" +pass "the narrow scope answers how much is waiting without saying what it is" + +wide=$(records_status --scope full) || fail "full scope failed" +assert_contains "$wide" '"scope": "full"' "full scope should say so" +assert_contains "$wide" 'alpha-one' "full scope should name in-flight work" +assert_contains "$wide" 'sign-in redirect' "full scope should carry titles" +assert_contains "$wide" 'https://github.com/example/alpha/pull/7' \ + "full scope should carry the pull request link" +assert_contains "$wide" 'beta-two' "full scope should name what waits on the captain" +# The state verb only. The agent speaks to the captain and must not read an +# internal event line aloud. +assert_contains "$wide" '"state": "working"' "full scope should carry the state verb" +assert_not_contains "$wide" 'reading the failing test' \ + "full scope must not carry the raw event line" +# The same rule against the bracketed shape: the verb is still the verb, and the +# metadata and the note stay unspoken. +assert_contains "$wide" '"state": "blocked"' \ + "a status line with a metadata token before the colon should still report its verb" +assert_not_contains "$wide" 'key=api-shape' \ + "full scope must not carry status metadata" +assert_not_contains "$wide" 'needs a credential' \ + "full scope must not carry the raw event line of a bracketed status" +pass "the wide scope names open work and reports state without quoting event lines" + +# THE DEFAULT IS THE NARROW SCOPE. A home that has configured nothing has granted +# nothing, and sending task identifiers, titles and pull request links to a model +# in another region is not something to inherit from somebody else's settings +# file. Widening is one line the captain of those records writes themselves. +default=$(records_status) || fail "default scope failed" +assert_contains "$default" '"scope": "counts"' \ + "an unconfigured home should get the narrow scope" +assert_not_contains "$default" 'alpha-one' \ + "an unconfigured home must not name work" +assert_not_contains "$default" 'sign-in redirect' \ + "an unconfigured home must not carry titles" +assert_not_contains "$default" 'github.com' \ + "an unconfigured home must not carry pull request links" +assert_contains "$default" '"in_flight": 3' \ + "an unconfigured home should still say how much is waiting" +pass "an absent read-scope setting means the narrowest answer, not the widest" + +# Widening is what the file is for, and it takes effect without a flag. +printf 'full\n' > "$HOME_FIXTURE/config/voice-read-scope" +widened=$(records_status) || fail "configured wide scope failed" +assert_contains "$widened" '"scope": "full"' \ + "writing full into config/voice-read-scope should widen the answer" +assert_contains "$widened" 'alpha-one' "the wide scope should then name work" +rm -f "$HOME_FIXTURE/config/voice-read-scope" +pass "a home widens its own read scope by writing the setting" + +# --- the confidentiality boundary ------------------------------------------- +# +# This is the case that lets a home widen to the full scope at all. + +for scope in full counts; do + answer=$(records_status --scope "$scope") || fail "scope $scope failed" + assert_not_contains "$answer" "$NEVER_TOKEN" \ + "finished work and note bodies must never reach a $scope answer" + assert_not_contains "$answer" 'old-six' \ + "finished work must not be named in a $scope answer" + assert_not_contains "$answer" 'old-seven' \ + "an unticked line under finished work must not be named in a $scope answer" + assert_not_contains "$answer" 'the rate we agreed' \ + "a note body must not reach a $scope answer" + # The count is the assertion that bites if the section rule is lost: old-seven + # is held for the captain and that list has no section filter of its own. + assert_contains "$answer" '"awaiting_captain": 2' \ + "finished work must not be counted as waiting on the captain at $scope scope" +done +pass "finished work and note bodies never reach a spoken answer at any scope" + +# The exclusion has to be structural rather than a filter on the way out, so the +# count of in-flight work stays honest while the body stays unread. +assert_contains "$wide" '"in_flight": 3' \ + "excluding note bodies must not change the count of in-flight work" +pass "excluding a note body does not distort the counts" + +# --- the deny list ---------------------------------------------------------- +# +# Reachable first, suppressed second. Without the first assertion the second +# proves nothing. + +assert_contains "$wide" "$DENY_TOKEN" \ + "fixture is wrong: the deny marker should be reachable before it is denied" + +printf '# one plain substring per line\n%s\n' "$DENY_TOKEN" \ + > "$HOME_FIXTURE/config/voice-read-deny" +denied=$(records_status --scope full) || fail "full scope with a deny list failed" +assert_not_contains "$denied" "$DENY_TOKEN" "the deny list must suppress a match" +assert_not_contains "$denied" 'gamma-three' \ + "a denied item must not be named at all" +assert_contains "$denied" '"withheld_as_confidential": 1' \ + "a denied item must still be counted so the captain knows it exists" +assert_contains "$denied" '"in_flight": 3' \ + "denying an item must not change the count of in-flight work" +# The other in-flight work is unaffected: this is a substring list, not a switch. +assert_contains "$denied" 'alpha-one' "the deny list must not suppress everything" +pass "a denied item becomes a withheld count without hiding that work exists" + +# Case-insensitive, because a confidentiality list that depends on the captain +# matching the file's capitalisation is a confidentiality list that fails quietly. +printf '%s\n' "$(printf '%s' "$DENY_TOKEN" | tr '[:upper:]' '[:lower:]')" \ + > "$HOME_FIXTURE/config/voice-read-deny" +lower=$(records_status --scope full) || fail "lowercase deny list failed" +assert_not_contains "$lower" "$DENY_TOKEN" "the deny list must match regardless of case" +pass "the deny list matches regardless of case" + +# The withheld figure counts denied items, not refusals, and the lists overlap by +# design: alpha-one is in flight AND carries a pull request, beta-two is in flight +# AND waiting on the captain. Counting each refusal would tell the captain four +# things are being withheld when two are, which is a wrong number spoken +# confidently about exactly the subject the captain is most careful with. +printf '%s\n%s\n' alpha-one beta-two > "$HOME_FIXTURE/config/voice-read-deny" +overlap=$(records_status --scope full) || fail "overlapping deny list failed" +assert_contains "$overlap" '"withheld_as_confidential": 2' \ + "two denied items appearing in two lists each must be withheld twice, not four times" +assert_not_contains "$overlap" 'alpha-one' "a denied item must not be named" +assert_not_contains "$overlap" 'beta-two' "a denied item must not be named" +assert_not_contains "$overlap" 'github.com' \ + "denying an item must suppress its pull request link too" +assert_contains "$overlap" '"in_flight": 3' \ + "denying items must not change the count of in-flight work" +assert_contains "$overlap" '"open_pull_requests": 1' \ + "denying items must not change the count of open pull requests" +pass "an item denied in more than one list is counted as withheld once" + +rm -f "$HOME_FIXTURE/config/voice-read-deny" + +# THE CASE THE DENY LIST EXISTS FOR, and the one a per-list decision gets wrong. +# The docstring says the list is for a future open task carrying a customer name, +# and a name like that lives in the TITLE or in the HOLD text of an item that is +# also in flight, also waiting on the captain, and also carrying a pull request. +# A decision taken separately in each list, from whichever fields that list +# happens to use, withholds such an item from one list and names it in another. +# That is not a narrower answer, it is a leak with a reassuring count beside it. +# Both items below sit in all three lists, and each is matched on a field only +# one of those lists reads. +# +# The third item is the one an in-flight-only fixture cannot catch: a QUEUED item +# that nothing holds for the captain, so no list iterates it, while its pull +# request link still reaches the answer through the worker records. Assembling its +# fields only where some list walks past it misses a match on its own title. +LEAK_HOME="$TMP_ROOT/deny-every-list" +TITLE_TOKEN=LEAKSBYTITLE +HOLD_TOKEN=LEAKSBYHOLD +QUEUED_TOKEN=LEAKSFROMQUEUED +mkdir -p "$LEAK_HOME/data" "$LEAK_HOME/state" "$LEAK_HOME/config" +cat > "$LEAK_HOME/data/backlog.md" <<EOF +# Backlog + +## In flight +- [ ] omega-nine - Renew the $TITLE_TOKEN contract (repo: omega) (kind: captain) +- [ ] sigma-ten - Move the account onto the new tier (repo: sigma) (kind: ship) (hold-kind: captain) (hold: waiting on the $HOLD_TOKEN owner) + +## Queued +- [ ] zeta-eight - Migrate the $QUEUED_TOKEN estate (repo: zeta) (kind: ship) +EOF +fm_write_meta "$LEAK_HOME/state/omega-nine.meta" \ + kind=captain pr=https://github.com/example/omega/pull/11 +fm_write_meta "$LEAK_HOME/state/sigma-ten.meta" \ + kind=ship pr=https://github.com/example/sigma/pull/12 +fm_write_meta "$LEAK_HOME/state/zeta-eight.meta" \ + kind=ship pr=https://github.com/example/zeta/pull/99 + +leak_status() { + python3 "$ROOT/bin/fm_voice_records.py" status --home "$LEAK_HOME" --scope full +} + +# Reachable in all three lists first, or the suppression below proves nothing. +reachable=$(leak_status) || fail "the deny-every-list fixture failed" +assert_contains "$reachable" "$TITLE_TOKEN" "fixture: the title marker should be reachable" +assert_contains "$reachable" 'omega-nine' "fixture: the item should be named" +assert_contains "$reachable" 'pull/11' "fixture: its pull request should be reachable" +assert_contains "$reachable" 'sigma-ten' "fixture: the held item should be named" +assert_contains "$reachable" 'pull/12' "fixture: its pull request should be reachable" +assert_contains "$reachable" '"awaiting_captain": 2' \ + "fixture: both in-flight items should be waiting on the captain" +assert_contains "$reachable" 'pull/99' \ + "fixture: the queued item should reach the answer through its pull request" +assert_contains "$reachable" '"queued": 1' "fixture: the queued item should be counted" + +# Matched on its title, which only the in-flight list reads. +printf '%s\n' "$TITLE_TOKEN" > "$LEAK_HOME/config/voice-read-deny" +by_title=$(leak_status) || fail "deny by title failed" +assert_not_contains "$by_title" "$TITLE_TOKEN" "a title match must be suppressed" +assert_not_contains "$by_title" 'omega-nine' \ + "a denied item must not be named in any list" +assert_not_contains "$by_title" 'pull/11' \ + "a denied item must not surface through its pull request link" +assert_contains "$by_title" '"withheld_as_confidential": 1' \ + "the denied item should be counted once" +# The other items are untouched, so this is a substring list and not a switch. +assert_contains "$by_title" 'sigma-ten' "the deny list must not suppress everything" +assert_contains "$by_title" 'pull/12' "the other pull requests should still be named" +assert_contains "$by_title" 'pull/99' "the other pull requests should still be named" +assert_contains "$by_title" '"open_pull_requests": 3' \ + "denying an item must not change the count of open pull requests" + +# Matched on its hold text, which only the captain list reads. The mirror of the +# case above: get one list right and this one still leaks. +printf '%s\n' "$HOLD_TOKEN" > "$LEAK_HOME/config/voice-read-deny" +by_hold=$(leak_status) || fail "deny by hold text failed" +assert_not_contains "$by_hold" 'sigma-ten' \ + "an item matched on its hold text must not be named in the in-flight list" +assert_not_contains "$by_hold" 'pull/12' \ + "an item matched on its hold text must not surface through its pull request" +assert_contains "$by_hold" '"withheld_as_confidential": 1' \ + "the denied item should be counted once" +assert_contains "$by_hold" 'omega-nine' "the deny list must not suppress everything" +assert_contains "$by_hold" 'pull/11' "the other pull request should still be named" +assert_contains "$by_hold" '"in_flight": 2' \ + "denying an item must not change the count of in-flight work" + +# Matched on the title of a QUEUED item that no list iterates. Its only way into +# the answer is its pull request link, and the pull request list knows nothing +# about titles, so a field set assembled per list never sees the match at all. +printf '%s\n' "$QUEUED_TOKEN" > "$LEAK_HOME/config/voice-read-deny" +by_queued=$(leak_status) || fail "deny by queued title failed" +assert_not_contains "$by_queued" "$QUEUED_TOKEN" \ + "a queued item's title match must be suppressed" +assert_not_contains "$by_queued" 'zeta-eight' \ + "a denied queued item must not be named" +assert_not_contains "$by_queued" 'pull/99' \ + "a denied queued item must not surface through its pull request link" +assert_contains "$by_queued" '"withheld_as_confidential": 1' \ + "a denied queued item must be counted, so nothing is hidden silently" +assert_contains "$by_queued" '"queued": 1' \ + "denying it must not change the count of queued work" +assert_contains "$by_queued" 'pull/11' "the other pull requests should still be named" +assert_contains "$by_queued" 'pull/12' "the other pull requests should still be named" +pass "one deny decision per item covers every list that item could appear in" + +# --- what a status line may say --------------------------------------------- +# +# A status line is free text a crewmate appended, and the verb taken off the +# front of it is the ONE record-derived string a counts-scope answer says out +# loud. At that scope there is no title and no link, so there is nothing for the +# deny list to filter and no scope setting that makes it safe. The vocabulary is +# therefore closed to the states bin/fm-brief.sh gives every crewmate plus the two +# bin/fm-classify-lib.sh adds when a decision closes, and anything else is a note. +VERB_HOME="$TMP_ROOT/status-verbs" +# Lowercase on purpose. The reader lowercases a verb before it could ever be +# emitted, and assert_not_contains compares case-sensitively, so an uppercase +# marker here would make the assertion below unable to fail on leaking code. +CUSTOMER_TOKEN=acmecorpmigration +mkdir -p "$VERB_HOME/data" "$VERB_HOME/state" +cat > "$VERB_HOME/data/backlog.md" <<'EOF' +# Backlog + +## In flight +- [ ] one - First thing (repo: a) (kind: ship) +- [ ] two - Second thing (repo: b) (kind: ship) +- [ ] three - Third thing (repo: c) (kind: ship) +EOF +fm_write_meta "$VERB_HOME/state/one.meta" kind=ship +fm_write_meta "$VERB_HOME/state/two.meta" kind=ship +fm_write_meta "$VERB_HOME/state/three.meta" kind=ship +printf 'needs-decision [key=shape]: which shape\n' > "$VERB_HOME/state/one.status" +printf '%s: waiting on their security review\n' "$CUSTOMER_TOKEN" \ + > "$VERB_HOME/state/two.status" +# A log past the tail window, so the read is proven to end at the last line +# rather than at the start of whatever window it happened to open. +{ + verb_line=0 + while [ "$verb_line" -lt 400 ]; do + printf 'working: step %s of a long task with a wordy status line\n' "$verb_line" + verb_line=$((verb_line + 1)) + done + printf 'done: shipped it\n' +} > "$VERB_HOME/state/three.status" +[ "$(wc -c < "$VERB_HOME/state/three.status")" -gt 8192 ] \ + || fail "fixture: the long status log should exceed the tail window" + +verb_status() { + python3 "$ROOT/bin/fm_voice_records.py" status --home "$VERB_HOME" "$@" +} + +verbs=$(verb_status --scope counts) || fail "counts scope with odd verbs failed" +assert_not_contains "$verbs" "$CUSTOMER_TOKEN" \ + "a word outside the vocabulary must not be spoken, at the default scope least of all" +assert_contains "$verbs" '"note": 1' \ + "an unrecognised verb should be counted as a note instead" +assert_contains "$verbs" '"needs-decision": 1' \ + "a canonical verb, brackets and all, should survive the fold" +assert_contains "$verbs" '"done": 1' \ + "the last line of a long log is the line that counts" +assert_not_contains "$verbs" '"working"' \ + "an earlier line in the same log must not be reported as the state" +pass "the state verb is a closed vocabulary, so free text cannot ride out on it" + +# The two halves of one answer must come from one home. Every script that sets +# FM_DATA_OVERRIDE sets FM_STATE_OVERRIDE beside it, so a reader that resolved one +# and not the other would count workers and notes from one home while counting +# in-flight work from another, which reads exactly like an ordinary answer. +alt_data="$TMP_ROOT/data-elsewhere" +mkdir -p "$alt_data" +cat > "$alt_data/backlog.md" <<'EOF' +# Backlog + +## In flight +- [ ] moved-one - Work recorded in the overridden data directory (repo: m) (kind: ship) +EOF +moved=$(FM_DATA_OVERRIDE="$alt_data" verb_status --scope full) \ + || fail "status with an overridden data directory failed" +assert_contains "$moved" 'moved-one' \ + "the reader must take the backlog from the overridden data directory" +assert_contains "$moved" '"in_flight": 1' "and count only what that backlog holds" +inbox_moved=$(FM_HOME="$VERB_HOME" FM_STATE_OVERRIDE="$VERB_HOME/state" \ + FM_DATA_OVERRIDE="$alt_data" "$ROOT/bin/fm-inbox.sh" status) \ + || fail "fm-inbox status with an overridden data directory failed" +assert_contains "$inbox_moved" 'moved-one' \ + "the human rendering of the same records must read the same backlog" +pass "the backlog and the state directory always come from the same home" + +# --- pull requests on finished work ----------------------------------------- +# +# A task keeps its state/<id>.meta after its backlog item is marked done, because +# removing the record and moving the item are separate steps. So a reader that took +# every worker carrying a pull request would count and name finished work, which +# this module promises never to read. Worse, the deny list could not reach those +# items: with no open item there is no title in the field set, so a captain +# substring matching the title silently failed for exactly them while working +# everywhere else. Losing the count of a pull request on a finished task is the +# accepted cost of that control applying everywhere it appears to. +DONE_HOME="$TMP_ROOT/finished-pull-requests" +FINISHED_TOKEN=SHIPPEDLASTWEEK +mkdir -p "$DONE_HOME/data" "$DONE_HOME/state" "$DONE_HOME/config" +cat > "$DONE_HOME/data/backlog.md" <<EOF +# Backlog + +## In flight +- [ ] still-open - Fix the retry (repo: a) (kind: ship) +- [x] ticked-two - Renew the $FINISHED_TOKEN contract (repo: b) (kind: ship) + +## Done +- [x] older-three - Migrate the $FINISHED_TOKEN estate (repo: c) (kind: ship) +EOF +fm_write_meta "$DONE_HOME/state/still-open.meta" \ + kind=ship pr=https://github.com/example/a/pull/1 +fm_write_meta "$DONE_HOME/state/ticked-two.meta" \ + kind=ship pr=https://github.com/example/b/pull/2 +fm_write_meta "$DONE_HOME/state/older-three.meta" \ + kind=ship pr=https://github.com/example/c/pull/3 + +done_status() { + python3 "$ROOT/bin/fm_voice_records.py" status --home "$DONE_HOME" --scope full +} + +open_only=$(done_status) || fail "the finished-pull-request fixture failed" +assert_contains "$open_only" '"open_pull_requests": 1' \ + "only open work has an open pull request" +assert_contains "$open_only" 'pull/1' "the open task's pull request should be named" +assert_not_contains "$open_only" 'ticked-two' \ + "a ticked item must not be named through its pull request" +assert_not_contains "$open_only" 'pull/2' \ + "a ticked item's pull request must not be named" +assert_not_contains "$open_only" 'older-three' \ + "an item under Done must not be named through its pull request" +assert_not_contains "$open_only" 'pull/3' \ + "an item under Done must not have its pull request named" +assert_not_contains "$open_only" "$FINISHED_TOKEN" \ + "no finished title may reach the answer at any scope" +# Excluded by construction, not withheld and counted. A later change that put +# finished work back in and leaned on the deny list to hide it would fail here. +assert_contains "$open_only" '"withheld_as_confidential": 0' \ + "finished work is left out rather than counted as withheld" +# The worker count is deliberately NOT open-only: a task keeps its runtime record +# until teardown removes it, and that record is what "on deck" counts. Asserted in +# the same case as the pull request count so the two cannot quietly converge. +assert_contains "$open_only" '"workers_on_deck": 3' \ + "every live runtime record is still on deck, finished or not" +pass "a finished task's pull request is neither counted nor named" + +# The deny list, observed doing its job on the one list that still carries links. +# A substring matching an OPEN task's title takes that task out of the pull +# request detail and says one thing is being withheld, while the count stays +# honest: that split is the contract this module states and the earlier cases +# pin, so the captain learns how much is waiting without learning what it is. +printf '%s\n' 'Fix the retry' > "$DONE_HOME/config/voice-read-deny" +denied_open=$(done_status) || fail "deny by an open title failed" +assert_not_contains "$denied_open" 'still-open' \ + "a denied open task must not be named in the pull request detail" +assert_not_contains "$denied_open" 'pull/1' \ + "a denied open task's pull request link must go with it" +assert_contains "$denied_open" '"withheld_as_confidential": 1' \ + "and the captain must be told one thing is being withheld" +assert_contains "$denied_open" '"open_pull_requests": 1' \ + "while the count of open pull requests stays honest" +rm -f "$DONE_HOME/config/voice-read-deny" +pass "a deny substring on an open title removes its pull request and says so" + +# --- refusals --------------------------------------------------------------- +# +# A misconfigured read scope must stop rather than fall back to the wider one, +# because falling back would widen what is sent on the strength of a typo. + +printf 'everything\n' > "$HOME_FIXTURE/config/voice-read-scope" +set +e +out=$(records_status 2>&1) +code=$? +set -e +expect_code 2 "$code" "an unknown read scope should refuse" +assert_contains "$out" 'voice-read-scope' "the refusal should name the setting" +pass "an unknown read scope refuses instead of widening" + +printf 'counts\n' > "$HOME_FIXTURE/config/voice-read-scope" +configured=$(records_status) || fail "configured scope failed" +assert_contains "$configured" '"scope": "counts"' "the configured scope should be used" +rm -f "$HOME_FIXTURE/config/voice-read-scope" +pass "the configured read scope is honoured" + +# --- handover --------------------------------------------------------------- +# +# The point of the boundary: real work is queued for firstmate, not done by the +# voice agent. It reuses bin/fm-inbox.sh rather than carrying a second queue. + +before=$(find "$HOME_FIXTURE/state" -maxdepth 2 -name '*.note' | wc -l) +[ "$before" = 0 ] || fail "fixture should start with an empty inbox" + +handed=$(FM_HOME="$HOME_FIXTURE" python3 "$ROOT/bin/fm_voice_records.py" queue \ + "Refactor the login module and open a pull request for it" \ + --home "$HOME_FIXTURE") || fail "handover failed" +assert_contains "$handed" '"queued": true' "handover should report the request queued" +assert_contains "$handed" 'did not do the work yourself' \ + "handover should tell the model it handed over rather than acted" + +notes=$(find "$HOME_FIXTURE/state/inbox" -maxdepth 1 -name '*.note' | wc -l) +[ "$notes" = 1 ] || fail "handover should leave exactly one note, found $notes" +note_file=$(find "$HOME_FIXTURE/state/inbox" -maxdepth 1 -name '*.note' | head -1) +assert_grep 'Refactor the login module' "$note_file" \ + "the note should carry the captain's words" + +# Exactly one wake, so a spoken request is presented once at firstmate's next +# check rather than queued twice or lost. +assert_present "$HOME_FIXTURE/state/.wake-queue" \ + "handover should wake firstmate" +wakes=$(grep -c 'inbox:' "$HOME_FIXTURE/state/.wake-queue") +[ "$wakes" = 1 ] || fail "handover should append exactly one wake, found $wakes" + +# The reading half must see what the queueing half just wrote, or the agent says +# the request is queued and then, asked what is waiting, says nothing is. +paired=$(records_status --scope counts) || fail "status after a handover failed" +assert_contains "$paired" '"captain_notes_waiting": 1' \ + "the reader should count the note the handover just queued" +pass "handover queues the request for firstmate and wakes it exactly once" + +# The same pairing when the state directory is moved. bin/fm-inbox.sh resolves +# ${FM_STATE_OVERRIDE:-$FM_HOME/state} and the handover queues through it with +# the ambient environment, so a reader that ignored the override would count +# notes in a directory nothing writes to. +alt_state="$TMP_ROOT/state-elsewhere" +alt_home="$TMP_ROOT/override-home" +mkdir -p "$alt_state" "$alt_home/data" "$alt_home/state" +FM_STATE_OVERRIDE="$alt_state" python3 "$ROOT/bin/fm_voice_records.py" queue \ + "Chase the flaky retry test" --home "$alt_home" >/dev/null \ + || fail "handover with an overridden state directory failed" + +moved=$(find "$alt_state/inbox" -maxdepth 1 -name '*.note' | wc -l) +[ "$moved" = 1 ] || \ + fail "the queue should write into the overridden state directory, found $moved" +[ ! -e "$alt_home/state/inbox" ] || \ + fail "the queue should not have written under the home when the state is moved" + +overridden=$(FM_STATE_OVERRIDE="$alt_state" python3 \ + "$ROOT/bin/fm_voice_records.py" status --home "$alt_home") \ + || fail "status with an overridden state directory failed" +assert_contains "$overridden" '"captain_notes_waiting": 1' \ + "the reader must count notes where the queue actually wrote them" +pass "the reader and the queue resolve the state directory the same way" + +set +e +empty_out=$(python3 "$ROOT/bin/fm_voice_records.py" queue " " \ + --home "$HOME_FIXTURE" 2>&1) +empty_code=$? +set -e +expect_code 2 "$empty_code" "queueing empty text should refuse" +assert_contains "$empty_out" 'empty' "the refusal should say the request was empty" +pass "an empty request is refused rather than queued as a blank note" + +# --- absent records --------------------------------------------------------- +# +# A home with no records at all must answer "nothing" rather than fail, because +# the agent is spoken to and an exception is not an answer. + +bare="$TMP_ROOT/bare" +mkdir -p "$bare" +bare_out=$(python3 "$ROOT/bin/fm_voice_records.py" status --home "$bare") \ + || fail "an empty home should still answer" +assert_contains "$bare_out" '"in_flight": 0' "an empty home should report no work" +assert_contains "$bare_out" '"workers_on_deck": 0' "an empty home should report no workers" +pass "a home with no records answers nothing rather than failing" + +# --- the whole round trip ---------------------------------------------------- +# +# Every case above holds one piece of the spoken interface still. This one runs +# the piece the captain experiences: the laptop client opens the transport, the +# relay answers a spoken question from the records and hands a spoken request for +# real work to firstmate, and the reply audio and the timing come back down the +# same stream. It is the only case that would notice the round trip stopping +# working while all of the pieces still passed. +# +# ONE thing is stood in for: the model. It is a paid service in another region +# and no test has a credential for it. The stand-in below speaks the same event +# protocol Nova Sonic does and composes what it says out of the tool results the +# relay actually hands it, so the words asserted here are the records rather than +# a script, and it records what the session was opened with so this case can +# check the account and the model the relay chose. Everything else is real: the +# client, the frame format, the relay, the reader and bin/fm-inbox.sh. +# +# What only this case can hold: +# the round trip completes at all, in both of its shapes, a status answer and a +# handover, and a second turn is not treated as an interruption of the first; +# the headline figure is measured from the captain's talk end rather than from +# the start of their speech, which on this clip is the difference between half +# a second and two and a half; +# the talk-end silence padding really is sent, which is trap 2 and the +# difference between an answer and no answer; +# the laptop needs no AWS credential: the client runs with an environment that +# has none, and the session is opened with the key only the desktop side holds. + +E2E="$TMP_ROOT/e2e" +E2E_KEY=AKIADESKTOPONLYEXAMPLE +E2E_REGION=eu-north-1 +E2E_MODEL=amazon.nova-2-sonic-v1:0 +E2E_REQUEST="take the flaky sign-in test on alpha and open a pull request for it" +mkdir -p "$E2E/bin" "$E2E/laptop" "$E2E/desktop-home" "$E2E/laptop-home" \ + "$E2E/fakesdk/aws_sdk_bedrock_runtime" \ + "$E2E/home/data" "$E2E/home/state" "$E2E/home/config" + +# The laptop holds the two files the guide says to copy, and nothing else. +cp "$ROOT/bin/fm-voice-client.py" "$ROOT/bin/fm_voice_frame.py" "$E2E/laptop/" + +cat > "$E2E/home/data/backlog.md" <<EOF +# Backlog + +## In flight +- [ ] alpha-one - Fix the sign-in redirect (repo: alpha) (kind: ship) (priority: 0) + A note body, which is never assembled: $NEVER_TOKEN and the rate we agreed. +- [ ] beta-two - Decide the storage shape (repo: beta) (kind: captain) + +## Queued +- [ ] delta-four - Add the retry (repo: delta) (kind: ship) (hold-kind: captain) + +## Done +- [x] old-six - Shipped the $NEVER_TOKEN integration (repo: alpha) (done 2026-07-01) +EOF +fm_write_meta "$E2E/home/state/alpha-one.meta" kind=ship mode=no-mistakes \ + pr=https://github.com/example/alpha/pull/7 +printf 'working: reading the failing test\n' > "$E2E/home/state/alpha-one.status" +printf '%s\n' "$E2E_REGION" > "$E2E/home/config/voice-region" +printf '%s\n' "$E2E_MODEL" > "$E2E/home/config/voice-model" +printf 'full\n' > "$E2E/home/config/voice-read-scope" + +# The model stand-in, at exactly the import boundary bin/fm-voice-relay.py uses. +cat > "$E2E/fakesdk/aws_sdk_bedrock_runtime/__init__.py" <<'PY' +"""A scripted stand-in for Nova Sonic's bidirectional stream. + +It answers with what the relay's own tool results contain, so a spoken answer +here is derived from firstmate's records rather than from a fixture string, and +it appends one JSON line per session describing what that session was opened +with and what it was asked. tests/fm-voice-relay.test.sh reads that record. + + FM_FAKE_SCRIPT comma-separated turn kinds: status | handover | clean-end + FM_FAKE_THINK seconds before the reply begins, standing in for the model + FM_FAKE_STATE file holding the turn counter across the relay's reconnects + FM_FAKE_LOG where to append the per-session record + FM_FAKE_REQUEST the words the captain uses when asking for real work + FM_FAKE_EARLY 1 to answer from the first audio in, not from the talk end + +A clean-end turn is a session the model finishes with while the captain is still +speaking: the output stream simply ends, with no error and no answer. That is an +ordinary end of a Bedrock session rather than a fault, and the relay has to +survive it, so it is a turn kind here rather than a failure injection. + +FM_FAKE_EARLY stands in for the model's own end-of-speech detector firing inside +a clip that already ends in silence: the answer begins before this end of the +stream has said the turn is over. Nova Sonic really does that, and the relay's +own timing figures are negative when it happens, which is the one case where a +fast-looking number is meaningless. +""" + +import asyncio +import base64 +import json +import math +import os +import struct +import sys +import types + +OUT_RATE = 24000 +CHUNK_MS = 100 + +THINK = float(os.environ.get("FM_FAKE_THINK", "0.4")) +REPLY_SECONDS = float(os.environ.get("FM_FAKE_REPLY_SECONDS", "0.4")) +SCRIPT = [s.strip() for s in os.environ.get("FM_FAKE_SCRIPT", "status").split(",") + if s.strip()] +STATE = os.environ.get("FM_FAKE_STATE", "") +LOG = os.environ.get("FM_FAKE_LOG", "") +REQUEST = os.environ.get("FM_FAKE_REQUEST", "open a pull request for the retry") +EARLY = os.environ.get("FM_FAKE_EARLY", "") == "1" + +HEARD = {"status": "how is the fleet doing right now", "handover": REQUEST} + +# How much of the captain's speech a clean-end session takes before its output +# stream ends. Three chunks is 300 ms, so on a two second clip the end lands well +# inside the key press and the rest of that press arrives at a session that is +# already over. +CLEAN_END_AFTER_BYTES = 3200 * 3 + + +def _turn_kind(): + """Return this session's turn kind, advancing a counter that lives on disk. + + The relay reconnects per turn on purpose, so the count cannot live in this + process: each turn is a new stream in a new session. + """ + index = 0 + if STATE: + try: + with open(STATE, encoding="utf-8") as handle: + index = int(handle.read().strip() or "0") + except (OSError, ValueError): + index = 0 + try: + with open(STATE, "w", encoding="utf-8") as handle: + handle.write(str(index + 1)) + except OSError: + pass + if not SCRIPT: + return "status", index + return SCRIPT[index % len(SCRIPT)], index + + +def _speech(seconds): + """Return reply audio: a quiet tone, so a byte count is a duration.""" + out = bytearray() + for n in range(int(OUT_RATE * seconds)): + out += struct.pack("<h", int(6000 * math.sin(2 * math.pi * 220 * n / OUT_RATE))) + return bytes(out) + + +def _status_sentence(result): + """Compose the spoken answer out of what the records reader returned.""" + if result.get("error"): + return "I could not read the records: {}".format(result["error"]) + said = "Right now, {} in flight, {} waiting on you, {} open pull requests.".format( + result.get("in_flight"), result.get("awaiting_captain"), + result.get("open_pull_requests")) + names = [row.get("id") for row in result.get("in_flight_detail", [])][:2] + if names: + said += " The ones moving are {}.".format(" and ".join(names)) + notes = result.get("captain_notes_waiting") or 0 + if notes: + said += " {} note is queued for the first mate.".format(notes) + if result.get("scope") == "counts": + said += " Identifiers are not available by voice at this read scope." + return said + + +def _queued_sentence(result): + if result.get("error"): + return "I could not queue that: {}".format(result["error"]) + return ("That is queued with the first mate as {}. I have not done any of it " + "myself.".format(result.get("note_id") or "a note")) + + +class _Result: + def __init__(self, payload): + self.value = types.SimpleNamespace(bytes_=payload) + + +class _OutputReader: + def __init__(self, queue): + self._queue = queue + + async def receive(self): + item = await self._queue.get() + return None if item is None else _Result(item) + + +class _InputStream: + def __init__(self, stream): + self._stream = stream + + async def send(self, chunk): + await self._stream.on_input(chunk.value.bytes_) + + async def close(self): + await self._stream.finish() + + +class _Stream: + """One bidirectional session, which is one turn the way the relay uses it.""" + + def __init__(self, model_id, config): + self.kind, self.index = _turn_kind() + self.out = asyncio.Queue() + self.input_stream = _InputStream(self) + self._reader = _OutputReader(self.out) + self._audio_content = None + self._pending_use = None + self._tools = {} + self._next_tool = 0 + self._replied = False + self._reply_task = None + self._logged = False + self._ended_early = False + self.record = { + "turn": self.index + 1, + "turn_kind": self.kind, + "model_id": model_id, + "endpoint": config.endpoint_uri, + "region": config.region, + "credential_key_id": config.credentials.get("aws_access_key_id"), + "tool_names_offered": [], + "audio_bytes_in": 0, + "tool_calls": [], + "heard": "", + "said": [], + "reply_audio_bytes": 0, + } + + async def await_output(self): + return (None, self._reader) + + def _emit(self, event): + self.out.put_nowait(json.dumps({"event": event}).encode()) + + async def on_input(self, raw): + event = json.loads(raw.decode()).get("event", {}) + for name, body in event.items(): + if name == "promptStart": + self.record["tool_names_offered"] = [ + t.get("toolSpec", {}).get("name") + for t in body.get("toolConfiguration", {}).get("tools", [])] + elif name == "contentStart" and body.get("type") == "AUDIO": + self._audio_content = body.get("contentName") + elif name == "contentStart" and body.get("type") == "TOOL": + self._pending_use = body.get( + "toolResultInputConfiguration", {}).get("toolUseId") + elif name == "audioInput": + self.record["audio_bytes_in"] += len( + base64.b64decode(body.get("content", ""))) + self._maybe_end_cleanly() + if EARLY and not self._replied and not self._ended_early: + # Answering while the captain's clip is still arriving, which + # is what the model's own endpoint detector does to a clip + # that ends in silence. + self._replied = True + self._reply_task = asyncio.create_task(self._reply()) + elif name == "toolResult": + self._tool_result(body) + elif name == "contentEnd": + if (body.get("contentName") == self._audio_content + and not self._replied and not self._ended_early): + # The captain's talk end. Everything measured is measured + # from here, so the reply starts no earlier than this. + self._replied = True + self._audio_content = None + self._reply_task = asyncio.create_task(self._reply()) + + def _maybe_end_cleanly(self): + """End a clean-end session's output stream, mid-key-press and unannounced. + + None on the output queue is what the SDK gives the reader for a stream + that is simply over: no exception, no stop reason, nothing to report. The + input half stays open, exactly as it does when the model is the side that + finished, so the rest of the captain's key press still arrives here and + goes nowhere. + """ + if (self.kind != "clean-end" or self._ended_early + or self.record["audio_bytes_in"] < CLEAN_END_AFTER_BYTES): + return + self._ended_early = True + self.record["ended_early"] = True + self._write_log() + self.out.put_nowait(None) + + def _tool_result(self, body): + try: + result = json.loads(body.get("content") or "{}") + except ValueError: + result = {} + if self.record["tool_calls"]: + self.record["tool_calls"][-1]["result"] = result + future = self._tools.pop(self._pending_use, None) + if future is not None and not future.done(): + future.set_result(result) + + async def _call_tool(self, name, arguments): + self._next_tool += 1 + use_id = "use-{}-{}".format(self.index + 1, self._next_tool) + future = asyncio.get_running_loop().create_future() + self._tools[use_id] = future + self.record["tool_calls"].append({"name": name, "arguments": arguments}) + self._emit({"toolUse": {"toolName": name, "toolUseId": use_id, + "content": json.dumps(arguments)}}) + try: + return await asyncio.wait_for(future, timeout=15) + except asyncio.TimeoutError: + return {"error": "the relay never answered the tool call"} + + def _say(self, text): + self.record["said"].append(text) + self._emit({"textOutput": {"role": "ASSISTANT", "content": text}}) + + async def _reply(self): + await asyncio.sleep(THINK) + heard = HEARD.get(self.kind, "how is the fleet doing") + self.record["heard"] = heard + self._emit({"textOutput": {"role": "USER", "content": heard}}) + if self.kind == "handover": + self._say("I am not the first mate, so I am handing that to it.") + result = await self._call_tool("hand_over_to_firstmate", + {"request": REQUEST}) + self._say(_queued_sentence(result)) + else: + result = await self._call_tool("get_fleet_status", {}) + self._say(_status_sentence(result)) + pcm = _speech(REPLY_SECONDS) + step = OUT_RATE * 2 * CHUNK_MS // 1000 + for at in range(0, len(pcm), step): + block = pcm[at:at + step] + self._emit({"audioOutput": { + "content": base64.b64encode(block).decode()}}) + self.record["reply_audio_bytes"] += len(block) + await asyncio.sleep(0.01) + # Trap 1: this, not completionEnd, is what says the reply ended. + self._emit({"contentEnd": {"stopReason": "END_TURN"}}) + self._write_log() + + def _write_log(self): + if self._logged or not LOG: + return + self._logged = True + with open(LOG, "a", encoding="utf-8") as handle: + handle.write(json.dumps(self.record) + "\n") + + async def finish(self): + self._write_log() + self.out.put_nowait(None) + + +class AsyncBedrockRuntimeConfig: + def __init__(self, endpoint_uri, region, credentials): + self.endpoint_uri = endpoint_uri + self.region = region + self.credentials = credentials + + @classmethod + async def resolve(cls, endpoint_uri=None, region=None, **credentials): + return cls(endpoint_uri, region, credentials) + + +class AsyncBedrockRuntimeClient: + def __init__(self, config=None): + self.config = config + + async def invoke_model_with_bidirectional_stream(self, operation): + return _Stream(operation.model_id, self.config) + + +class InvokeModelWithBidirectionalStreamOperationInput: + def __init__(self, model_id=None): + self.model_id = model_id + + +class BidirectionalInputPayloadPart: + def __init__(self, bytes_=b""): + self.bytes_ = bytes_ + + +class InvokeModelWithBidirectionalStreamInputChunk: + def __init__(self, value=None): + self.value = value + + +def _submodule(name, **members): + module = types.ModuleType(__name__ + "." + name) + for key, value in members.items(): + setattr(module, key, value) + sys.modules[module.__name__] = module + return module + + +client = _submodule( + "client", + AsyncBedrockRuntimeClient=AsyncBedrockRuntimeClient, + InvokeModelWithBidirectionalStreamOperationInput=( + InvokeModelWithBidirectionalStreamOperationInput)) +config = _submodule("config", AsyncBedrockRuntimeConfig=AsyncBedrockRuntimeConfig) +models = _submodule( + "models", + BidirectionalInputPayloadPart=BidirectionalInputPayloadPart, + InvokeModelWithBidirectionalStreamInputChunk=( + InvokeModelWithBidirectionalStreamInputChunk)) +PY + +# What the desktop side of the connection has, and the laptop side does not. The +# AWS credential is here and nowhere else, which is the whole point of the shape. +cat > "$E2E/bin/desktop.env" <<EOF +PATH=$PATH +HOME=$E2E/desktop-home +PYTHONPATH=$E2E/fakesdk +PYTHONDONTWRITEBYTECODE=1 +FM_HOME=$E2E/home +FM_FAKE_STATE=$E2E/turn-counter +FM_FAKE_LOG=$E2E/model-sessions.jsonl +FM_FAKE_THINK=0.4 +FM_FAKE_REPLY_SECONDS=0.4 +FM_FAKE_SCRIPT=status,handover +FM_FAKE_REQUEST=$E2E_REQUEST +AWS_ACCESS_KEY_ID=$E2E_KEY +AWS_SECRET_ACCESS_KEY=desktop-secret-not-a-real-key +EOF + +# Stands in for ssh, so the client takes its real `ssh -T <host> <relay>` path +# and the desktop's environment is a boundary rather than an assertion: the relay +# starts from env -i and desktop.env, so nothing the laptop holds can reach it. +cat > "$E2E/bin/ssh" <<'SH' +#!/usr/bin/env bash +set -eu +DIR="$(cd "$(dirname "$0")" && pwd)" +if [ "${1:-}" = "-T" ]; then shift; fi +shift # the host, which is this machine +desktop_env=() +while IFS= read -r line; do desktop_env+=("$line"); done < "$DIR/desktop.env" +exec env -i "${desktop_env[@]}" "$@" +SH +chmod +x "$E2E/bin/ssh" + +# Two seconds of speech-shaped audio ending on speech, not silence: the relay's +# own 400 ms of padding is what makes a push-to-talk release answerable, and a +# clip this long makes a clock started at the wrong end unmistakable. +python3 - "$E2E/clip.pcm" <<'PY' || fail "could not write the e2e clip" +import math, struct, sys +out = bytearray() +for n in range(16000 * 2): + swell = 0.5 + 0.5 * math.sin(2 * math.pi * 2.5 * n / 16000) + out += struct.pack("<h", int(9000 * swell * math.sin(2 * math.pi * 190 * n / 16000))) +open(sys.argv[1], "wb").write(bytes(out)) +PY + +printf '0\n' > "$E2E/turn-counter" +: > "$E2E/model-sessions.jsonl" + +# env -i: the laptop has PATH and HOME and nothing else. No AWS variable, no +# interpreter that can reach Bedrock, no firstmate home. +laptop_aws=$(env -i PATH="$E2E/bin:$PATH" HOME="$E2E/laptop-home" env \ + | grep -c '^AWS_' || true) +[ "$laptop_aws" = 0 ] || fail "the laptop end should hold no AWS variables" + +set +e +env -i PATH="$E2E/bin:$PATH" HOME="$E2E/laptop-home" PYTHONDONTWRITEBYTECODE=1 \ + python3 "$E2E/laptop/fm-voice-client.py" \ + --host desktop.example \ + --relay "$ROOT/bin/fm-voice-relay.py" \ + --relay-python python3 \ + --in-file "$E2E/clip.pcm" --out-file "$E2E/reply.pcm" \ + --runs 2 > "$E2E/runs.jsonl" 2> "$E2E/session.log" +e2e_code=$? +set -e +[ "$e2e_code" = 0 ] || { + cat "$E2E/session.log" >&2 + fail "the spoken round trip exited $e2e_code" +} + +# The reader's own answer, taken independently, so the spoken answer is checked +# against the records rather than against itself. +independent=$(python3 "$ROOT/bin/fm_voice_records.py" status \ + --home "$E2E/home" --scope full) || fail "independent status read failed" +printf '%s' "$independent" > "$E2E/independent.json" + +python3 - "$E2E" "$E2E_KEY" "$E2E_REGION" "$E2E_MODEL" "$E2E_REQUEST" \ + "$NEVER_TOKEN" <<'PY' || fail "the spoken round trip did not hold" +import json, os, sys + +root, key, region, model, request, never = sys.argv[1:7] + + +def check(cond, label): + if not cond: + sys.exit("round trip: " + label) + + +def read(name): + with open(os.path.join(root, name), encoding="utf-8") as handle: + return [json.loads(line) for line in handle if line.strip()] + + +runs = read("runs.jsonl") +sessions = read("model-sessions.jsonl") +records = json.load(open(os.path.join(root, "independent.json"), encoding="utf-8")) +transcript = open(os.path.join(root, "session.log"), encoding="utf-8").read() + +# Two turns asked, two turns answered with audio, neither of them lost. +check(len(runs) == 2, "expected two turn records, got %d" % len(runs)) +check(len(sessions) == 2, "expected two model sessions, got %d" % len(sessions)) +for run in runs: + check(run["answered"], "turn %s was not answered" % run["run"]) + check(run["relay_error"] is None, + "turn %s failed: %s" % (run["run"], run["relay_error"])) + check(run["reply_audio_seconds"] > 0, + "turn %s produced no reply audio" % run["run"]) + # Push to talk is the default and the only mode that runs, and the transport + # is the ssh path rather than a local child. + check(run["listen"] == "push-to-talk", "listen mode was %r" % run["listen"]) + check(run["transport"] == "ssh", "transport was %r" % run["transport"]) + # The per-turn reconnect exists so a second question is not barge-in. + check(not run["interrupted"], + "turn %s was treated as an interruption" % run["run"]) + +# Whose account and which model. The relay carries no default for either, so +# this is the home's configuration reaching Bedrock, and the credential is the +# one only the desktop side of the connection holds. +for session in sessions: + check(session["model_id"] == model, "model was %r" % session["model_id"]) + check(session["region"] == region, "region was %r" % session["region"]) + check(session["endpoint"] == + "https://bedrock-runtime.{}.amazonaws.com".format(region), + "endpoint was %r" % session["endpoint"]) + check(session["credential_key_id"] == key, + "session opened with %r" % session["credential_key_id"]) + check(session["tool_names_offered"] == + ["get_fleet_status", "hand_over_to_firstmate"], + "tools offered were %r" % session["tool_names_offered"]) + # Trap 2: a push-to-talk release supplies no trailing silence, so the relay + # appends its own. Without it the model truncates the turn and never answers. + check(session["audio_bytes_in"] == 16000 * 2 * 2 + 400 * 32, + "the uplink carried %d bytes, so the talk-end padding is not being " + "sent" % session["audio_bytes_in"]) + +# The status answer is the records. Every number the agent said aloud came from +# the reader, checked against a separate read of the same home. +status = sessions[0] +check([c["name"] for c in status["tool_calls"]] == ["get_fleet_status"], + "the status turn called %r" % [c["name"] for c in status["tool_calls"]]) +served = status["tool_calls"][0]["result"] +for field in ("in_flight", "awaiting_captain", "open_pull_requests", "queued"): + check(served[field] == records[field], + "the reader served %s=%r but the records say %r" + % (field, served[field], records[field])) +said = " ".join(status["said"]) +check("{} in flight".format(records["in_flight"]) in said, + "the spoken answer did not carry the count: %r" % said) +check(records["in_flight_detail"][0]["id"] in said, + "the spoken answer named no open work: %r" % said) +check(never not in said and never not in json.dumps(served), + "a note body or finished title reached a spoken answer") +check(said in transcript, "the captain never saw the answer: %r" % transcript) + +# The handover turn queues real work and says so. The note is firstmate's own +# queue, written by bin/fm-inbox.sh, and the agent's confirmation carries the id +# that queue gave it, so it cannot be claiming to have queued something it did +# not. +handover = sessions[1] +check([c["name"] for c in handover["tool_calls"]] == ["hand_over_to_firstmate"], + "the handover turn called %r" % [c["name"] for c in handover["tool_calls"]]) +check(handover["tool_calls"][0]["arguments"]["request"] == request, + "the captain's words were rewritten: %r" + % handover["tool_calls"][0]["arguments"]) +queued = handover["tool_calls"][0]["result"] +check(queued.get("queued") is True, "the request was not queued: %r" % queued) +note_id = queued.get("note_id") +check(bool(note_id), "the queue returned no note id: %r" % queued) +check(runs[1]["queued_note"] == note_id, + "the client was told %r, the queue wrote %r" + % (runs[1]["queued_note"], note_id)) +check(note_id in " ".join(handover["said"]), + "the agent did not confirm the queued note: %r" % handover["said"]) +check("handed to the first mate" in transcript, + "the captain was never told it was handed over: %r" % transcript) +note = os.path.join(root, "home", "state", "inbox", note_id + ".note") +check(os.path.exists(note), "no note on disk at %s" % note) +check(request in open(note, encoding="utf-8").read(), + "the note does not carry the captain's words") + +# THE NUMBER THIS BUILD EXISTS TO PRODUCE, and the instant it is measured from. +# The clip is two seconds long and the stand-in waits 0.4 s before speaking, so a +# figure measured from the captain's talk end lands near half a second and one +# measured from the start of their speech lands near two and a half. The bound is +# loose enough for a loaded machine and nowhere near the wrong clock. +for run in runs: + first = run["first_audio_s"] + check(first is not None, "turn %s reported no first audio" % run["run"]) + check(0.2 < first < 1.6, + "turn %s reported first audio at %.3fs, which is not measured from the " + "captain's talk end" % (run["run"], first)) + marks = run["relay_marks_since_talk_end"] + for mark in ("tool_use", "tool_answered", "first_audio", "reply_end"): + check(mark in marks, "turn %s is missing the %s mark" % (run["run"], mark)) + check(marks["tool_use"] <= marks["first_audio"] <= marks["reply_end"], + "turn %s reports its marks out of order: %r" % (run["run"], marks)) + check(run["first_frame_s"] is not None and run["first_played_s"] is not None, + "turn %s reported no wire or playback figure" % run["run"]) + +# The reply audio survived the framing byte for byte. +sent = sum(s["reply_audio_bytes"] for s in sessions) +got = os.path.getsize(os.path.join(root, "reply.pcm")) +check(sent > 0 and sent == got, + "the model sent %d bytes of reply audio and the client wrote %d" % (sent, got)) + +# AND NOTHING ALARMING WAS SAID, which is the other half of every fault line this +# build added. The relay's goodbye arrives on this path too, at the end of every +# ordinary session, so a line that reads as a fault would fire here on a session +# where all of it worked. An alarm that also goes off on success is worse than no +# alarm: the captain learns to skip it, and then the one that means their question +# was lost is invisible as well. The session end still speaks, in its own words. +for alarming in ("stopped without being asked", "could not handle the relay's", + "connection lost", "the connection ended", + "were not taken", "no reply within"): + check(alarming not in transcript, + "a session where every turn was answered said %r: %r" + % (alarming, transcript)) +# And no record named a reason, which is what now decides the exit code as well. +# A fault that fires on a session where everything worked would fail every clean +# run from here on, so the absence is worth pinning next to the exit code itself. +for run in runs: + check(run["relay_error"] is None, + "turn %s recorded a reason on a session that worked: %r" + % (run["run"], run["relay_error"])) +check("the relay signed off" in transcript, + "but the end of the session should still be said, in wording that cannot be " + "read as a fault: %r" % transcript) +PY +pass "a spoken turn goes out and comes back: the records answer, firstmate gets the work" + +# --- a model session that ends while the captain is still talking ------------ +# +# A Bedrock session ending is not a fault. The stream simply stops: no exception, +# no stop reason, nothing to report. It can happen mid-conversation, and when it +# does the captain is usually still holding the talk key, because that is when +# the relay is talking to the model at all. +# +# The relay used to treat that as its own reason to stop, which is the worst +# available failure shape: the relay dies without saying anything the captain can +# act on, and they find out by speaking a whole question into nothing and getting +# no answer. Per-turn reconnect already covers this - the next talk key builds a +# new session, at a measured cost of 0.02 s - and a reconnect that cannot be made +# is spoken to the captain through the turn-failed path. So the session ending +# costs them the remainder of one key press, and nothing else. +# +# This case is the round trip above with one difference: the model finishes with +# the first session 300 ms into a two second key press. What it holds is that the +# relay is still serving afterwards and that the NEXT talk key gets a working +# session rather than a closed pipe - a real answer, out of the real records, over +# the same connection. The relay may exit for three reasons and this is not one of +# them. + +SURVIVE="$E2E/survive" +mkdir -p "$SURVIVE/bin" + +# The same desktop, with its own turn script and its own record of what the model +# was asked, so neither run can read the other's sessions. +grep -v '^FM_FAKE_' "$E2E/bin/desktop.env" > "$SURVIVE/bin/desktop.env" +cat >> "$SURVIVE/bin/desktop.env" <<EOF +FM_FAKE_STATE=$SURVIVE/turn-counter +FM_FAKE_LOG=$SURVIVE/model-sessions.jsonl +FM_FAKE_THINK=0.4 +FM_FAKE_REPLY_SECONDS=0.4 +FM_FAKE_SCRIPT=clean-end,status +EOF +cp "$E2E/bin/ssh" "$SURVIVE/bin/ssh" +printf '0\n' > "$SURVIVE/turn-counter" +: > "$SURVIVE/model-sessions.jsonl" + +# Exit 1 is the honest outcome and what is asserted: one of the two turns really +# was lost, because the model stopped listening part way through it. +set +e +env -i PATH="$SURVIVE/bin:$PATH" HOME="$E2E/laptop-home" PYTHONDONTWRITEBYTECODE=1 \ + python3 "$E2E/laptop/fm-voice-client.py" \ + --host desktop.example \ + --relay "$ROOT/bin/fm-voice-relay.py" \ + --relay-python python3 \ + --in-file "$E2E/clip.pcm" --out-file "$SURVIVE/reply.pcm" \ + --timeout 12 --runs 2 > "$SURVIVE/runs.jsonl" 2> "$SURVIVE/session.log" +survive_code=$? +set -e +[ "$survive_code" = 1 ] || { + cat "$SURVIVE/session.log" >&2 + fail "a lost turn and a good one should exit 1, not $survive_code" +} + +python3 - "$SURVIVE" "$E2E/independent.json" <<'PY' \ + || fail "a model session ending did not leave the relay serving" +import json, os, sys + +root, records_path = sys.argv[1:3] + +CLIP_BYTES = 16000 * 2 * 2 + + +def check(cond, label): + if not cond: + sys.exit("session ended: " + label) + + +def read(name): + with open(os.path.join(root, name), encoding="utf-8") as handle: + return [json.loads(line) for line in handle if line.strip()] + + +runs = read("runs.jsonl") +sessions = read("model-sessions.jsonl") +records = json.load(open(records_path, encoding="utf-8")) +transcript = open(os.path.join(root, "session.log"), encoding="utf-8").read() + +# The first session really did end part way through the captain's key press, +# rather than after answering: it took some of the clip and not all of it. +check(len(sessions) >= 1, "the model was never asked anything") +first = sessions[0] +check(first["turn_kind"] == "clean-end" and first.get("ended_early"), + "the first session did not end early: %r" % first) +check(0 < first["audio_bytes_in"] < CLIP_BYTES, + "the session ended after %d of %d bytes, so it did not end mid-press" + % (first["audio_bytes_in"], CLIP_BYTES)) +check(not first["said"] and not first["reply_audio_bytes"], + "the lost turn was answered after all: %r" % first) + +# THE POINT. The relay was still there for the next talk key, so two turns were +# taken over the one connection and the second one was a whole session of its own. +check(len(runs) == 2, + "the relay stopped serving when the model ended its session: %d turn(s) " + "taken, %r" % (len(runs), transcript)) +check(len(sessions) == 2, + "the next talk key did not get a session: %d opened" % len(sessions)) +check(not runs[0]["answered"], "the lost turn should be the first one: %r" % runs[0]) +# An ordinary session end is not a turn FAILURE - the relay names none, marks the +# session spent for nobody, and the next talk key gets a working one - but the +# turn it landed in still has no answer, and the record has to say why. Left null +# there, the only turn in this whole file that went unanswered for a knowable +# reason reads in runs.jsonl exactly like a turn the model declined, and that file +# is where the published latency spread comes from. +check(runs[0]["relay_error"], + "a turn released by the session ending must name why it has no answer: %r" + % runs[0]) +check("session" in runs[0]["relay_error"], + "and it should name the session ending rather than some other fault: %r" + % runs[0]["relay_error"]) +check("could not finish that turn" not in transcript, + "the relay must not have called this a failed turn: %r" % transcript) + +# And it was a working session rather than a closed pipe: a real answer, composed +# from a real read of the records, spoken to the captain over the same connection. +good = runs[1] +check(good["answered"] and good["reply_audio_seconds"] > 0, + "the next talk key got no answer: %r" % good) +check(good["relay_error"] is None, + "the replacement session failed: %r" % good["relay_error"]) +check([c["name"] for c in sessions[1]["tool_calls"]] == ["get_fleet_status"], + "the replacement turn called %r" + % [c["name"] for c in sessions[1]["tool_calls"]]) +# The whole question, not an answer to nothing: every byte of the clip and the +# talk-end padding reached the replacement session. +check(sessions[1]["audio_bytes_in"] == CLIP_BYTES + 400 * 32, + "the replacement session heard %d bytes of a %d byte question" + % (sessions[1]["audio_bytes_in"], CLIP_BYTES + 400 * 32)) +served = sessions[1]["tool_calls"][0]["result"] +for field in ("in_flight", "open_pull_requests"): + check(served[field] == records[field], + "the replacement session served %s=%r but the records say %r" + % (field, served[field], records[field])) +said = " ".join(sessions[1]["said"]) +check("{} in flight".format(records["in_flight"]) in said, + "the answer did not carry the count: %r" % said) +check(said in transcript, "the captain never heard the answer: %r" % transcript) +check(good["first_audio_s"] is not None and 0.2 < good["first_audio_s"] < 1.6, + "the recovered turn reported first audio at %r, which is not measured from " + "the captain's talk end" % good["first_audio_s"]) +check(os.path.getsize(os.path.join(root, "reply.pcm")) + == sessions[1]["reply_audio_bytes"], + "the reply audio the client wrote is not what the good session sent") + +# Said once. The rest of that key press is another seventeen audio frames, and +# the flag saying the session is over stays set for every one of them, so a +# notice sent from the frame loop instead of from the end itself would put this +# line in front of the captain ten times a second while they were still speaking. +check(transcript.count("the relay ended the session") == 1, + "the session ending was announced %d times: %r" + % (transcript.count("the relay ended the session"), transcript)) +check("connection lost" not in transcript, + "the connection should have outlived the session: %r" % transcript) +PY +pass "a model session that ends mid-conversation costs one turn, not the relay" + +# --- an uplink that stops being a frame stream -------------------------------- +# +# One of the three things that ends the relay, and the only one it diagnoses: the +# captain's uplink desynchronises, so the relay can no longer tell a header from +# audio. It writes what it saw on its own stderr and exits 2, which is honest, +# and it used to tell the client only goodbye. Everything the relay knew stayed +# on a stderr no run file quotes, while the turn the captain was in the middle of +# came back as answered: false with nothing in relay_error - the reason existed +# and was thrown away. +# +# Worse, the teardown closes the model session first, and that sets the flag which +# silences the session reader's own notice, so this path really did send the client +# nothing at all but the goodbye. +# +# The relay is driven directly here rather than through the client, because no +# client sends a bad frame: the desync is the transport corrupting one, and the +# only way to put one on the wire is to be the other end. The model is the same +# stand-in the round trip above uses, and it is never asked to answer. + +set +e +env -i PATH="$PATH" HOME="$E2E/desktop-home" PYTHONPATH="$E2E/fakesdk" \ + PYTHONDONTWRITEBYTECODE=1 FM_HOME="$E2E/home" \ + AWS_ACCESS_KEY_ID="$E2E_KEY" \ + AWS_SECRET_ACCESS_KEY=desktop-secret-not-a-real-key \ + python3 - "$ROOT/bin" <<'PY' +import os, subprocess, sys +sys.path.insert(0, sys.argv[1]) +import fm_voice_frame as frame + +relay = os.path.join(sys.argv[1], "fm-voice-relay.py") + + +def check(cond, label): + if not cond: + sys.exit("desync: " + label) + + +proc = subprocess.Popen( + [sys.executable, relay, "--serve"], + stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) +try: + magic = proc.stdout.read(len(frame.MAGIC)) + check(magic == frame.MAGIC, "the relay did not open the stream: %r" % magic) + reader = frame.Reader(proc.stdout) + kind, payload = reader.read() + check(kind == frame.NOTICE + and frame.decode_json(payload).get("event") == "ready", + "the relay was never ready: %r %r" % (kind, payload)) + + # A turn is open when the uplink goes, which is when it goes in practice: the + # captain is holding the talk key, so that is when there are frames on the + # wire to be corrupted at all. + proc.stdin.write(frame.encode(frame.TALK_START)) + proc.stdin.write(b"\xff\x00\x00\x00\x00") + proc.stdin.flush() + + frames = [] + while True: + got = reader.read() + if got is None: + break + frames.append(got) + if got[0] == frame.BYE: + break +finally: + proc.stdin.close() + said = proc.stderr.read().decode("utf-8", "replace") + code = proc.wait(timeout=30) + +kinds = [k for k, _ in frames] +check(frame.BYE in kinds, "the relay never said goodbye: %r" % kinds) +named = [n for n in (frame.decode_json(p) for k, p in frames if k == frame.NOTICE) + if n.get("event") == "turn-failed"] +check(named, + "the relay diagnosed the fault and told the client only goodbye: %r" % kinds) +check("FrameError" in named[0].get("error", ""), + "the notice should carry what the relay saw: %r" % named[0]) +# Before the goodbye, and before the model session is closed: closing it awaits +# the model stream, which can be slow or raise, and the client must learn the +# reason either way. +failed_at = next(i for i, (k, p) in enumerate(frames) + if k == frame.NOTICE + and frame.decode_json(p).get("event") == "turn-failed") +check(failed_at < kinds.index(frame.BYE), + "the reason must reach the client ahead of the goodbye: %r" % kinds) + +# Still said on the relay's own stderr, which the captain reads over SSH, and +# still the unhappy exit code: naming the fault down the connection does not make +# it a turn the relay recovered from. +check("not a frame stream any more" in said, + "the relay should still name the desync on its stderr: %r" % said) +check(code == 2, "a desynchronised uplink must exit 2, got %r" % code) +PY +desync_code=$? +set -e +[ "$desync_code" = 0 ] || fail "a desynchronised uplink was not named to the client" +pass "a desynchronised uplink is named to the client before the goodbye, not only to stderr" + +# --- the desktop's own check, and the clock it refuses to lie about ---------- +# +# `fm-voice-relay.py --self-test <clip.pcm>` is what docs/voice-relay.md tells the +# captain to run before they touch the laptop, and it is also the instrument the +# direct column of the latency table was measured with. Everything above drives +# the relay through the client; this drives the desktop check itself, because a +# broken --self-test is a captain who cannot tell a configured desktop from an +# unconfigured one, and a number in a table that nobody can reproduce. +# +# The second case is the one worth having. A clip that already ends in silence +# makes the model answer before this end of the stream has said the turn is over, +# so every figure is measured from the wrong instant and comes out negative. The +# reply really is fast and the number really is meaningless, which is the worst +# combination to leave in a results file for someone who was not here. The guard +# has to name the marks and say why, not print the figure. + +SELFTEST="$E2E/self-test" +mkdir -p "$SELFTEST" +printf '0\n' > "$SELFTEST/turn-counter" + +# The same two seconds of speech, with a second of silence glued on the end: the +# shape the docs warn against, and the only way to reach the guard. +cat "$E2E/clip.pcm" > "$SELFTEST/ends-in-silence.pcm" +python3 - "$SELFTEST/ends-in-silence.pcm" <<'PY' \ + || fail "could not write the silence-tailed clip" +import sys +with open(sys.argv[1], "ab") as handle: + handle.write(b"\x00\x00" * 16000) +PY + +# The desktop, and only the desktop: the AWS credential and the SDK live here. +relay_self_test() { + local clip=$1 + shift + env -i PATH="$PATH" HOME="$E2E/desktop-home" PYTHONPATH="$E2E/fakesdk" \ + PYTHONDONTWRITEBYTECODE=1 FM_HOME="$E2E/home" \ + FM_FAKE_STATE="$SELFTEST/turn-counter" FM_FAKE_LOG="$SELFTEST/sessions.jsonl" \ + FM_FAKE_THINK=0.4 FM_FAKE_REPLY_SECONDS=0.4 FM_FAKE_SCRIPT=status \ + AWS_ACCESS_KEY_ID="$E2E_KEY" \ + AWS_SECRET_ACCESS_KEY=desktop-secret-not-a-real-key \ + "$@" python3 "$ROOT/bin/fm-voice-relay.py" --self-test "$clip" +} + +set +e +ok_out=$(relay_self_test "$E2E/clip.pcm" 2> "$SELFTEST/ok.err") +ok_code=$? +set -e +expect_code 0 "$ok_code" "the documented desktop check should answer" +printf '%s\n' "$ok_out" > "$SELFTEST/ok.json" + +python3 - "$SELFTEST/ok.json" "$E2E/independent.json" "$E2E_REGION" "$E2E_MODEL" \ + <<'PY' || fail "the desktop check did not report a usable measurement" +import json, sys + +report = json.load(open(sys.argv[1], encoding="utf-8")) +records = json.load(open(sys.argv[2], encoding="utf-8")) +region, model = sys.argv[3:5] + + +def check(cond, label): + if not cond: + sys.exit("self-test: %s -- %r" % (label, report)) + + +check(report["mode"] == "self-test", "not a self-test report") +check(report["answered"] and report["reply_audio_seconds"] > 0, "it did not answer") +check(report["relay_error"] is None, "it reported an error") +check(report["region"] == region and report["model"] == model, + "it used the wrong account's model") +check(report["tool_names"] == ["get_fleet_status"], "it called the wrong tool") +# The words are the records, the same as over the relay. +check("{} in flight".format(records["in_flight"]) in report["said"], + "the spoken answer did not carry the count") +check(report["heard"], "it reported nothing heard") +# The figure the direct column of the latency table is made of, measured from the +# talk end: the clip is two seconds and the stand-in thinks for 0.4 s, so a clock +# started at the wrong end lands near 2.4. +check(report["clock_unusable"] == [], "it flagged a clip that ends on speech") +for mark in ("tool_use_s", "first_audio_s", "reply_end_s"): + check(report[mark] is not None, "no %s figure" % mark) +check(0.2 < report["first_audio_s"] < 1.6, + "first audio at %r is not measured from the talk end" % report["first_audio_s"]) +check(report["tool_use_s"] <= report["first_audio_s"] <= report["reply_end_s"], + "the figures are out of order") +PY +pass "the desktop's own check answers from the records and times it from the talk end" + +set +e +early_out=$(relay_self_test "$SELFTEST/ends-in-silence.pcm" FM_FAKE_EARLY=1 \ + 2> "$SELFTEST/early.err") +early_code=$? +set -e +expect_code 0 "$early_code" "an answered turn is still an answered turn" +printf '%s\n' "$early_out" > "$SELFTEST/early.json" + +early_err=$(cat "$SELFTEST/early.err") +assert_contains "$early_err" "measured from the wrong instant" \ + "the guard must say why the timings cannot be used" +assert_contains "$early_err" "ends on speech" \ + "the guard must say what clip to pass instead" + +python3 - "$SELFTEST/early.json" <<'PY' \ + || fail "a reply that beat the end of the clip was recorded as a good measurement" +import json, sys + +report = json.load(open(sys.argv[1], encoding="utf-8")) + + +def check(cond, label): + if not cond: + sys.exit("wrong clock: %s -- %r" % (label, report)) + + +# It answered. That is exactly why the figure is dangerous rather than obviously +# broken: a reader sees answered: true and a fast number. +check(report["answered"] and report["reply_audio_seconds"] > 0, + "the turn was not answered at all, so this is not the case under test") +check(report["first_audio_s"] < 0, + "the reply did not beat the end of the clip, so the guard was not reached") +for mark in ("tool_use", "first_audio", "reply_end"): + check(mark in report["clock_unusable"], "%s is not named as unusable" % mark) +PY +pass "a reply that arrives before the end of the clip is named as an unusable clock" + +printf 'all voice relay cases passed\n' diff --git a/tests/fm-watch-arm.test.sh b/tests/fm-watch-arm.test.sh index 0115330671a..2a3a5173c40 100755 --- a/tests/fm-watch-arm.test.sh +++ b/tests/fm-watch-arm.test.sh @@ -121,7 +121,7 @@ ack_wakes() { # <state> rm -f "$err" if [ -z "$sequence" ] || [ -z "$generation" ]; then [ ! -s "$state/.wake-queue" ] || return 1 - case "$(cat "$state/.watcher-down" 2>/dev/null || true)" in pending:*) return 1 ;; esac + case "$(cat "$state/.watcher-down" 2>/dev/null || true)" in pending:*|announced:*) return 1 ;; esac return 0 fi FM_STATE_OVERRIDE="$state" "$DRAIN" --ack-through "$sequence" \ @@ -439,7 +439,7 @@ test_delivery_gap_wake_is_recovered_once() { } test_interrupted_handling_is_redrained_on_rearm() { - local dir home state fakebin first_arm recovery_arm generation_before sequence generation handling_watcher_pid + local dir home state fakebin first_arm recovery_arm sequence generation handling_watcher_pid handling_generation generation_replay dir=$(make_case interrupted-handling-redrain) home="$dir/home" state="$dir/state" @@ -462,32 +462,39 @@ test_interrupted_handling_is_redrained_on_rearm() { grep "$(printf '\tsignal\tinterrupted.status\t')" "$state/.wake-queue" >/dev/null \ || fail "pre-successor crash recovery removed the unacknowledged durable wake" case "$(cat "$state/.watcher-down" 2>/dev/null || true)" in - pending:downtime:*) ;; + pending:downtime:*|announced:downtime:*) ;; *) fail "reason emission marked recovery handled before a successor was established" ;; esac - generation_before=$(sed -n 's/^pending:downtime:\(.*\)$/\1/p' "$state/.watcher-down") + generation_before=$(recovery_marker_generation "$state/.watcher-down") + [ -n "$generation_before" ] || fail "crash-gap recovery left no recovery generation" start_rearm_arm "$home" "$state" "$fakebin" "$dir/reason-emit-crash-replay.out" wait_for_exit "$ARM_PID" 80 || fail "a crash after reason emission stranded the durable wake" recovery_arm=$ARM_PID grep -F 'check: rearm-resurface' "$dir/reason-emit-crash-replay.out" >/dev/null \ || fail "a crash after reason emission did not re-drain recovery" - [ "$(cat "$state/.watcher-down" 2>/dev/null || true)" = "pending:downtime:$generation_before" ] \ - || fail "reason-emission replay replaced or prematurely handled its generation" + generation_replay=$(recovery_marker_generation "$state/.watcher-down") + [ -n "$generation_replay" ] \ + || fail "reason-emission replay left no recovery generation" grep "$(printf '\tsignal\tinterrupted.status\t')" "$state/.wake-queue" >/dev/null \ || fail "reason-emission replay removed the unacknowledged durable wake" start_rearm_arm "$home" "$state" "$fakebin" "$dir/handling-successor-arm.out" "$recovery_arm" is_live_non_zombie "$ARM_PID" \ || fail "expected handling successor looped on the pending durable wake" - [ "$(cat "$state/.watcher-down" 2>/dev/null || true)" = "pending:downtime:$generation_before" ] \ - || fail "successor launch marked recovery handled before prompt delivery" + case "$(cat "$state/.watcher-down" 2>/dev/null || true)" in + announced:downtime:*|pending:downtime:*) ;; + *) fail "successor launch marked recovery handled before prompt delivery" ;; + esac + handling_generation=$(recovery_marker_generation "$state/.watcher-down") handling_watcher_pid=$(sed -n 's/^watcher: started pid=\([0-9][0-9]*\).* recovery-generation=.*$/\1/p' "$dir/handling-successor-arm.out") - FM_HOME="$home" FM_STATE_OVERRIDE="$state" "$WATCH_ARM" --handling-delivered "$generation_before" \ + FM_HOME="$home" FM_STATE_OVERRIDE="$state" "$WATCH_ARM" --handling-delivered "$handling_generation" \ --watcher-pid "$handling_watcher_pid" \ || fail "confirmed prompt delivery did not begin handling" - [ "$(cat "$state/.watcher-down" 2>/dev/null || true)" = "pending:handling:$generation_before" ] \ - || fail "confirmed prompt delivery did not transition its recovery generation" + case "$(cat "$state/.watcher-down" 2>/dev/null || true)" in + pending:handling:"$handling_generation"|announced:handling:"$handling_generation") ;; + *) fail "confirmed prompt delivery did not transition its recovery generation" ;; + esac ! grep -F 'check: rearm-resurface' "$dir/handling-successor-arm.out" >/dev/null \ || fail "expected handling successor emitted a recursive recovery wake" FM_HOME="$home" FM_STATE_OVERRIDE="$state" "$DRAIN" > "$dir/interrupted-drain.out" \ @@ -501,7 +508,7 @@ test_interrupted_handling_is_redrained_on_rearm() { kill -TERM "$ARM_PID" 2>/dev/null || fail "could not interrupt the handling successor" wait "$ARM_PID" 2>/dev/null || true case "$(cat "$state/.watcher-down" 2>/dev/null || true)" in - pending:downtime:*) ;; + pending:downtime:*|announced:downtime:*) ;; *) fail "interrupted pre-handling successor did not persist downtime recovery" ;; esac @@ -620,7 +627,7 @@ test_markerless_legacy_queue_is_recovered_on_arm() { grep -F 'check: rearm-resurface' "$dir/arm.out" >/dev/null \ || fail "markerless legacy queue did not trigger recovery" case "$(cat "$state/.watcher-down" 2>/dev/null || true)" in - pending:downtime:*) ;; + pending:downtime:*|announced:downtime:*) ;; *) fail "markerless legacy queue was not adopted into downtime recovery" ;; esac FM_HOME="$home" FM_STATE_OVERRIDE="$state" "$DRAIN" > "$dir/drain.out" \ diff --git a/tests/fm-watch-recovery-loop.test.sh b/tests/fm-watch-recovery-loop.test.sh new file mode 100755 index 00000000000..9dae5ed3d5a --- /dev/null +++ b/tests/fm-watch-recovery-loop.test.sh @@ -0,0 +1,222 @@ +#!/usr/bin/env bash +# Pin the Pi/OpenCode recovery-loop fix: one announcement per generation, and a +# handling successor that keeps supervising instead of going blind. +set -u + +# shellcheck source=tests/wake-helpers.sh +. "$(dirname "${BASH_SOURCE[0]}")/wake-helpers.sh" + +WATCH="$ROOT/bin/fm-watch.sh" +TMP_ROOT=$(fm_test_tmproot fm-watch-recovery-loop) +export NODE_NO_WARNINGS=1 + +install_pi_watch_extension_fixture() { + local repo=$1 + mkdir -p \ + "$repo/.pi/extensions/lib" \ + "$repo/node_modules/@earendil-works/pi-coding-agent" \ + "$repo/node_modules/@earendil-works/pi-tui" \ + "$repo/node_modules/typebox" \ + "$repo/bin" + cp "$ROOT/.pi/extensions/fm-primary-pi-watch.ts" "$repo/.pi/extensions/fm-primary-pi-watch.ts" + cp "$ROOT/.pi/extensions/lib/fm-calm-visibility.ts" "$repo/.pi/extensions/lib/fm-calm-visibility.ts" + cp "$ROOT/.pi/extensions/lib/fm-operational-input.ts" "$repo/.pi/extensions/lib/fm-operational-input.ts" + cp "$ROOT/bin/fm-operational-input.sh" "$repo/bin/fm-operational-input.sh" + chmod +x "$repo/bin/fm-operational-input.sh" + cat > "$repo/node_modules/@earendil-works/pi-coding-agent/package.json" <<'JSON' +{"name":"@earendil-works/pi-coding-agent","type":"module","exports":"./index.js"} +JSON + cat > "$repo/node_modules/@earendil-works/pi-coding-agent/index.js" <<'JS' +export function getMarkdownTheme() { return {}; } +export class UserMessageComponent { + render() { return []; } + invalidate() {} +} +JS + cat > "$repo/node_modules/@earendil-works/pi-tui/package.json" <<'JSON' +{"name":"@earendil-works/pi-tui","type":"module","exports":"./index.js"} +JSON + cat > "$repo/node_modules/@earendil-works/pi-tui/index.js" <<'JS' +export class Box { + addChild() {} + clear() {} + setBgFn() {} +} +export class Container {} +export class Text {} +JS + cat > "$repo/node_modules/typebox/package.json" <<'JSON' +{"name":"typebox","type":"module","exports":"./index.js"} +JSON + cat > "$repo/node_modules/typebox/index.js" <<'JS' +export const Type = { + Object(properties) { + return { type: "object", properties, additionalProperties: false }; + }, +}; +JS +} + +# T1: a lost --handling-delivered handshake must not re-announce forever. +# The real Pi extension drives the real arm/watcher, with only the handshake +# RPC forced to fail. After the first recovery follow-up, wait past the old +# ~52s loop period so a regression would emit a second follow-up. +test_unacknowledged_recovery_is_announced_once_per_generation() { + local repo home plugin fakebin out status lock_pid messages + repo="$TMP_ROOT/t1-root" + home="$TMP_ROOT/t1-home" + fakebin="$TMP_ROOT/t1-fakebin" + mkdir -p "$repo/bin" "$home/state" "$home/config" "$fakebin" + install_pi_watch_extension_fixture "$repo" + plugin="$repo/.pi/extensions/fm-primary-pi-watch.ts" + cat > "$fakebin/tmux" <<'SH' +#!/usr/bin/env bash +exit 0 +SH + chmod +x "$fakebin/tmux" + cat > "$repo/bin/fm-watch-arm.sh" <<SH +#!/usr/bin/env bash +if [ "\${1:-}" = --handling-delivered ]; then + exit 1 +fi +export FM_ROOT_OVERRIDE="$ROOT" +export PATH="$fakebin:\$PATH" +exec "$ROOT/bin/fm-watch-arm.sh" "\$@" +SH + chmod +x "$repo/bin/fm-watch-arm.sh" + : > "$home/state/seed.meta" + printf 'pending:downtime:seed.1.aaa\n' > "$home/state/.watcher-down" + chmod 600 "$home/state/.watcher-down" + printf '%s\t1\tcheck\tseed\tcheck: seed recovery\n' "$(date +%s)" > "$home/state/.wake-queue" + out=$( + PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" \ + FM_STATE_OVERRIDE="$home/state" PATH="$fakebin:$PATH" \ + FM_POLL=1 FM_SIGNAL_GRACE=0 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 \ + node --input-type=module 2>&1 <<'EOF' +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +let tool = null; +const prompts = []; +const pi = { + on() {}, + registerCommand() {}, + registerTool(candidate) { + if (candidate.name === "fm_watch_arm_pi") tool = candidate; + }, + sendUserMessage: async (message) => { + prompts.push(String(message)); + }, +}; +writeFileSync(`${process.env.FM_HOME}/state/.lock`, `${process.pid}\n`); +const mod = await import(pathToFileURL(process.env.PLUGIN).href); +mod.default(pi); +if (!tool) throw new Error("Pi watch tool was not registered"); +await tool.execute("tool-call-t1", {}, undefined, undefined, {}); +const deadline = Date.now() + 75000; +let firstAt = 0; +while (Date.now() < deadline) { + const rearm = prompts.filter((message) => message.includes("check: rearm-resurface")); + if (rearm.length > 1) { + throw new Error(`unbounded recovery loop: ${rearm.length} rearm-resurface follow-ups`); + } + if (rearm.length === 1 && firstAt === 0) firstAt = Date.now(); + if (firstAt && Date.now() - firstAt >= 55000) break; + await new Promise((resolve) => setTimeout(resolve, 200)); +} +const rearm = prompts.filter((message) => message.includes("check: rearm-resurface")); +if (rearm.length !== 1) { + throw new Error(`expected exactly one recovery follow-up, got ${rearm.length}: ${prompts.join(" || ")}`); +} +const lockPid = existsSync(`${process.env.FM_HOME}/state/.watch.lock/pid`) + ? readFileSync(`${process.env.FM_HOME}/state/.watch.lock/pid`, "utf8").trim() + : ""; +if (!/^[0-9]+$/.test(lockPid)) throw new Error("successor watcher lock pid missing"); +try { + process.kill(Number(lockPid), 0); +} catch { + throw new Error(`successor watcher ${lockPid} is not alive`); +} +const marker = readFileSync(`${process.env.FM_HOME}/state/.watcher-down`, "utf8").trim(); +if (!marker.startsWith("announced:") && !marker.startsWith("pending:")) { + throw new Error(`successor did not keep a live recovery episode: ${marker}`); +} +console.log(`T1_MESSAGES=${rearm.length}`); +console.log(`T1_LOCK_PID=${lockPid}`); +console.log(`T1_MARKER=${marker}`); +process.exit(0); +EOF + ) + status=$? + if [ "${FM_TEST_EVIDENCE:-0}" = 1 ]; then + printf '%s\n' "$out" + fi + lock_pid=$(sed -n 's/^T1_LOCK_PID=//p' <<<"$out" | tail -1) + messages=$(sed -n 's/^T1_MESSAGES=//p' <<<"$out" | tail -1) + if [ -n "$lock_pid" ]; then + kill -TERM "$lock_pid" 2>/dev/null || true + fi + expect_code 0 "$status" "an unacknowledged recovery must be announced at most once per generation: $out" + [ "$messages" = 1 ] || fail "T1 did not report a single recovery follow-up: $out" + pass "unacknowledged recovery is announced at most once per generation and the successor stays alive" +} + +# T2: a handling successor must enter its poll loop immediately and surface a +# real crew event instead of sitting in a pre-loop wait that refreshes the +# liveness beacon and then exits with a synthetic rearm-resurface. +test_handling_successor_does_not_go_blind() { + local dir home state fakebin child event_start now out + dir=$(make_case recovery-gap-successor) + home="$dir/home" + state="$dir/state" + fakebin="$dir/fakebin" + mkdir -p "$home/data" + : > "$state/crew.meta" + printf 'pending:downtime:gap.1.aaa\n' > "$state/.watcher-down" + chmod 600 "$state/.watcher-down" + out="$dir/watch.out" + PATH="$fakebin:$PATH" FM_HOME="$home" FM_STATE_OVERRIDE="$state" \ + FM_POLL=1 FM_SIGNAL_GRACE=0 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=600 \ + FM_WATCH_HANDLING_SUCCESSOR=1 "$WATCH" > "$out" 2>&1 & + child=$! + now=0 + while [ "$now" -lt 40 ]; do + [ "$(cat "$state/.watch.lock/pid" 2>/dev/null || true)" = "$child" ] && break + sleep 0.1 + now=$((now + 1)) + done + [ "$(cat "$state/.watch.lock/pid" 2>/dev/null || true)" = "$child" ] \ + || { kill -TERM "$child" 2>/dev/null || true; fail "handling successor did not take the watcher lock"; } + sleep 0.4 + printf 'done: crew finished its task\n' >> "$state/crew.status" + event_start=$(date +%s) + now=0 + while [ "$now" -lt 5 ]; do + if grep -q '^signal:' "$out" 2>/dev/null; then + break + fi + sleep 0.5 + now=$((now + 1)) + done + if ! grep -q '^signal:' "$out" 2>/dev/null; then + kill -TERM "$child" 2>/dev/null || true + wait "$child" 2>/dev/null || true + fail "handling successor did not surface the crew event within a poll interval or two (waited $(( $(date +%s) - event_start ))s): $(cat "$out")" + fi + grep -F 'crew.status' "$out" >/dev/null \ + || { kill -TERM "$child" 2>/dev/null || true; fail "handling successor did not name the crew status file: $(cat "$out")"; } + grep "$(printf '\tsignal\tcrew.status\t')" "$state/.wake-queue" >/dev/null \ + || { kill -TERM "$child" 2>/dev/null || true; fail "handling successor did not enqueue a durable row for the crew event"; } + ! grep -F 'check: rearm-resurface' "$out" >/dev/null \ + || { kill -TERM "$child" 2>/dev/null || true; fail "handling successor emitted synthetic recovery instead of supervising: $(cat "$out")"; } + if [ "${FM_TEST_EVIDENCE:-0}" = 1 ]; then + printf 'T2_WATCH_OUTPUT=%s\n' "$(tr '\n' ' ' < "$out")" + printf 'T2_QUEUE_ROW=%s\n' "$(grep "$(printf '\tsignal\tcrew.status\t')" "$state/.wake-queue" | tail -1)" + fi + kill -TERM "$child" 2>/dev/null || true + wait "$child" 2>/dev/null || true + pass "a resurfacing handling successor stays alive and supervises instead of going blind" +} + +test_handling_successor_does_not_go_blind +test_unacknowledged_recovery_is_announced_once_per_generation diff --git a/tests/fm-watch-triage.test.sh b/tests/fm-watch-triage.test.sh index 87979184a89..1bee5cab9bb 100755 --- a/tests/fm-watch-triage.test.sh +++ b/tests/fm-watch-triage.test.sh @@ -337,7 +337,17 @@ test_status_is_paused_classifier() { || fail "captain-held transfer not recognized by the bounded-idle classifier" status_is_paused_or_captain_held 'resolved [key=route]: captain answered' \ && fail "resolved decision remained classed as captain-held" - pass "status_is_paused: only the leading paused verb matches, and paused is not captain-relevant" + # The two declarations share one cadence but block on different humans, so the + # combined predicate cannot be the only discriminator: a recheck has to know which + # verb it is naming. + status_is_captain_held 'captain-held [key=route]: tracked by task-decision-route' \ + || fail "captain-held verb not recognized" + status_is_captain_held 'paused: holding for the upstream release' \ + && fail "a declared pause matched the captain-held verb" + status_is_captain_held 'working: the captain-held backlog item is next' \ + && fail "a working line mentioning captain-held false-matched" + status_is_captain_held '' && fail "empty line classified as captain-held" + pass "status_is_paused: only the leading paused verb matches, paused is not captain-relevant, and the two declared-wait verbs stay separable" } # crew_absorb_class: the single fm-crew-state.sh read that returns BOTH absorb @@ -367,6 +377,175 @@ test_crew_absorb_class_classifier() { pass "crew_absorb_class: working/paused/none from one read; crew_is_paused and crew_is_provably_working agree" } +# The wedge detector's third liveness input: writes inside the crew's own recorded +# worktree. Every negative outcome must report "no evidence" so the caller keeps +# its existing escalation schedule, and a supervisor-side git read (which touches +# .git, never tracked files) must not be able to fake a positive. +test_crew_worktree_written_since_classifier() { + local dir state anchor wt home statedir_wt + dir=$(make_case classify-worktree-writes); state="$dir/state" + anchor="$state/anchor"; wt="$dir/wt"; home="$dir/mate-home"; statedir_wt="$dir/wt-with-state" + mkdir -p "$wt/src" "$wt/.git/objects" + printf 'old\n' > "$wt/src/existing.c" + set_mtime "$(( $(date +%s) - 300 ))" "$wt/src/existing.c" + : > "$anchor" + set_mtime "$(( $(date +%s) - 120 ))" "$anchor" + + # No recorded worktree at all: absence of evidence, never a positive. + printf 'window=test:fm-a\nkind=ship\n' > "$state/a.meta" + ! crew_worktree_written_since a "$state" "$anchor" \ + || fail "a task with no recorded worktree reported write evidence" + # Recorded but gone (torn down): still no evidence. + printf 'window=test:fm-b\nkind=ship\nworktree=%s\n' "$dir/missing" > "$state/b.meta" + ! crew_worktree_written_since b "$state" "$anchor" \ + || fail "a torn-down worktree reported write evidence" + # Present, but nothing written since the anchor. + printf 'window=test:fm-c\nkind=ship\nworktree=%s\n' "$wt" > "$state/c.meta" + ! crew_worktree_written_since c "$state" "$anchor" \ + || fail "a quiet worktree reported write evidence" + # A missing anchor cannot be compared against: no evidence. + ! crew_worktree_written_since c "$state" "$state/absent-anchor" \ + || fail "a missing anchor reported write evidence" + # Only .git churn (what firstmate's own read-only git commands touch): pruned. + printf 'pack\n' > "$wt/.git/objects/fresh" + printf 'ref\n' > "$wt/.git/index" + ! crew_worktree_written_since c "$state" "$anchor" \ + || fail ".git churn alone reported write evidence (a supervisor read could fake liveness)" + # A real file written after the anchor: positive evidence. + printf 'new\n' > "$wt/src/new.c" + crew_worktree_written_since c "$state" "$anchor" \ + || fail "a file written after the anchor was not reported as write evidence" + # An empty id is never evidence. + ! crew_worktree_written_since "" "$state" "$anchor" || fail "an empty id reported write evidence" + + # A secondmate records a provisioned firstmate home, not a code tree, and such a + # home supervises itself: its own watcher beacon, pane hashes, and heartbeats keep + # its state/ churning whether or not the mate produced anything. + mkdir -p "$home/state" + printf 'sm-classify-1\n' > "$home/.fm-secondmate-home" + printf 'beat\n' > "$home/state/.last-watcher-beat" + printf 'window=remote:sm\nkind=secondmate\nworktree=%s\n' "$home" > "$state/sm.meta" + ! crew_worktree_written_since sm "$state" "$anchor" \ + || fail "a secondmate's own home supervision churn reported crew write evidence" + # The home marker alone is enough, even when the record does not say secondmate. + printf 'window=test:fm-sm2\nkind=ship\nworktree=%s\n' "$home" > "$state/sm2.meta" + ! crew_worktree_written_since sm2 "$state" "$anchor" \ + || fail "a marked firstmate home reported crew write evidence" + # But an ordinary worktree that merely holds a directory named state is real + # work: only the home is excluded, never a source directory of that name. + mkdir -p "$statedir_wt/state" + printf 'machine\n' > "$statedir_wt/state/machine.go" + printf 'window=test:fm-d\nkind=ship\nworktree=%s\n' "$statedir_wt" > "$state/d.meta" + crew_worktree_written_since d "$state" "$anchor" \ + || fail "a source directory named state was hidden from the write probe" + pass "crew_worktree_written_since: real writes are evidence; no worktree, no anchor, quiet trees, .git churn and a mate's own home are not" +} + +# FM_WORKTREE_WRITE_PRUNE is a skip list, so clearing it skips nothing and is the +# obvious way to widen the probe to the whole depth-bounded tree. An empty list must +# therefore widen the walk rather than report no evidence at all, which would +# quietly cost the wedge detector its third liveness input on a home that cleared +# the knob to get more coverage, not less. +test_empty_write_prune_widens_the_probe() { + local dir state anchor wt saved + dir=$(make_case classify-empty-write-prune); state="$dir/state" + anchor="$state/anchor"; wt="$dir/wt" + mkdir -p "$wt/src" "$wt/.git" + : > "$anchor" + set_mtime "$(( $(date +%s) - 120 ))" "$anchor" + printf 'window=test:fm-e\nkind=ship\nworktree=%s\n' "$wt" > "$state/e.meta" + saved=$FM_WORKTREE_WRITE_PRUNE + FM_WORKTREE_WRITE_PRUNE='' + # A quiet tree is still no evidence, so the caller's schedule is untouched. + ! crew_worktree_written_since e "$state" "$anchor" \ + || fail "an empty prune list reported write evidence for a quiet worktree" + printf 'new\n' > "$wt/src/new.c" + crew_worktree_written_since e "$state" "$anchor" \ + || fail "an empty prune list disabled the probe instead of widening it" + # Widened means nothing is skipped, including what the default list prunes. + set_mtime "$(( $(date +%s) - 900 ))" "$wt/src/new.c" + printf 'pack\n' > "$wt/.git/index" + crew_worktree_written_since e "$state" "$anchor" \ + || fail "an empty prune list still skipped a directory the default list prunes" + # Restoring the default prunes .git again, so a supervisor's own read-only git + # command still cannot fake liveness. + FM_WORKTREE_WRITE_PRUNE=$saved + ! crew_worktree_written_since e "$state" "$anchor" \ + || fail "the default prune list stopped keeping .git out of the probe" + pass "an empty FM_WORKTREE_WRITE_PRUNE widens the probe to the whole depth-bounded tree instead of disabling it" +} + +# The same widening, reached the way a home actually configures it: through the +# process ENVIRONMENT, not an in-process assignment made after the library was +# sourced. An empty exported value must survive as empty, because defaulting it with +# the colon form reads "explicitly cleared" as "never set" and hands the default skip +# list straight back to the one home that asked for a wider walk. +# shellcheck disable=SC2016 # single quotes are deliberate: the library path, state dir, and anchor expand inside the bash -c child, not here +test_empty_write_prune_from_the_environment_widens_the_probe() { + local dir state anchor wt + dir=$(make_case classify-empty-write-prune-env); state="$dir/state" + anchor="$state/anchor"; wt="$dir/wt" + mkdir -p "$wt/.git/objects" + : > "$anchor" + set_mtime "$(( $(date +%s) - 120 ))" "$anchor" + printf 'window=test:fm-wenv\nkind=ship\nworktree=%s\n' "$wt" > "$state/wenv.meta" + # The one thing written since the anchor sits exactly where the DEFAULT list prunes. + printf 'pack\n' > "$wt/.git/objects/fresh" + env -u FM_WORKTREE_WRITE_PRUNE \ + bash -c '. "$1"; crew_worktree_written_since wenv "$2" "$3"' _ \ + "$ROOT/bin/fm-classify-lib.sh" "$state" "$anchor" \ + && fail "the default skip list let .git churn count as write evidence" + FM_WORKTREE_WRITE_PRUNE='' \ + bash -c '. "$1"; crew_worktree_written_since wenv "$2" "$3"' _ \ + "$ROOT/bin/fm-classify-lib.sh" "$state" "$anchor" \ + || fail "an empty FM_WORKTREE_WRITE_PRUNE in the environment fell back to the default skip list instead of widening the probe" + pass "an empty FM_WORKTREE_WRITE_PRUNE exported into the environment prunes nothing, widening the probe" +} + +# The probe's walk runs synchronously inside the poll that was about to escalate, so +# it must be wall-clock bounded: -xdev keeps it out of a nested mount, but a worktree +# root that is ITSELF on a hung mount would otherwise stall the very supervisor that +# exists to notice a wedge. A fake find that never returns in time stands in for that +# mount. Hitting the bound must read as NO evidence, exactly like every other +# negative outcome, so the caller's escalation schedule is untouched. +test_worktree_write_probe_is_wall_clock_bounded() { + local dir state anchor wt slowbin fastbin started elapsed + dir=$(make_case classify-write-probe-bound); state="$dir/state" + anchor="$state/anchor"; wt="$dir/wt"; slowbin="$dir/slowbin"; fastbin="$dir/fastbin" + mkdir -p "$wt/src" "$slowbin" "$fastbin" + : > "$anchor" + set_mtime "$(( $(date +%s) - 120 ))" "$anchor" + printf 'window=test:fm-slow\nkind=ship\nworktree=%s\n' "$wt" > "$state/slow.meta" + # Both stand-ins report the same hit; only one of them takes longer than the bound + # to do it, so the prompt one shows what a positive outcome looks like and the + # bounded assertion below cannot pass merely because the fake failed. + cat > "$fastbin/find" <<'SH' +#!/usr/bin/env bash +set -u +printf '%s\n' "$1/hit" +SH + cat > "$slowbin/find" <<'SH' +#!/usr/bin/env bash +set -u +sleep 30 +printf '%s\n' "$1/hit" +SH + chmod +x "$fastbin/find" "$slowbin/find" + PATH="$fastbin:$PATH" \ + bash -c '. "$1"; crew_worktree_written_since slow "$2" "$3"' _ \ + "$ROOT/bin/fm-classify-lib.sh" "$state" "$anchor" \ + || fail "a walk that reported a hit inside its bound was not read as write evidence" + started=$(date +%s) + PATH="$slowbin:$PATH" FM_WORKTREE_WRITE_TIMEOUT=1 \ + bash -c '. "$1"; crew_worktree_written_since slow "$2" "$3"' _ \ + "$ROOT/bin/fm-classify-lib.sh" "$state" "$anchor" \ + && fail "a walk that outlived its bound was reported as write evidence" + elapsed=$(( $(date +%s) - started )) + [ "$elapsed" -lt 10 ] \ + || fail "the worktree write probe was not wall-clock bounded: one walk held the caller for ${elapsed}s" + pass "the worktree write probe is wall-clock bounded, and hitting the bound reads as no write evidence" +} + # signal_crew_provably_working: a no-verb "signal:" wake is benign ONLY when EVERY # task it references is provably working; if any crew has stopped, or no task can be # resolved, it surfaces. Files map to ids by stripping .status / .turn-ended. @@ -882,8 +1061,10 @@ test_exited_declared_pause_is_bounded_but_live_gate_surfaces() { FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & pid=$! wait_for_exit "$pid" 100 || fail "captain-held dead-agent pane did not re-surface on the bounded cadence" + grep -F "awaiting the captain" "$state/.wake-queue" >/dev/null \ + || fail "captain-held dead-agent pane surfaced as a stopped crew instead of a captain-owned recheck: $(cat "$state/.wake-queue")" grep -F "awaiting external" "$state/.wake-queue" >/dev/null \ - || fail "captain-held dead-agent pane surfaced as a stopped crew" + && fail "captain-held dead-agent pane borrowed the pause verb's external-wait wording" dir=$(make_case alive-decision-gate); state="$dir/state"; fakebin="$dir/fakebin" out="$dir/watch.out"; capture_file="$dir/pane.txt"; statusf="$state/gate.status" @@ -955,11 +1136,46 @@ test_secondmate_paused_resurfaces_in_normal_mode() { wait_for_exit "$pid" 100 || fail "watcher did not re-surface a paused secondmate" grep -F "stale: $window" "$out" >/dev/null || fail "paused secondmate did not emit a stale recheck" grep -F "awaiting external" "$out" >/dev/null || fail "paused secondmate recheck omitted its external-wait reason" + grep -F "awaiting the captain" "$out" >/dev/null && fail "paused secondmate recheck named the captain instead of its external dependency" grep -F "possible wedge" "$out" >/dev/null && fail "paused secondmate was mislabeled a wedge" unset FM_FAKE_CREW_STATE pass "a declared paused secondmate re-surfaces on the bounded normal-mode cadence" } +# A captain hold is the other declared wait, but unlike paused: it has no +# current-state mapping, so a held mate reports `unknown` rather than `paused`. +# The bounded re-surface must still reach it, or a mate's hold rots invisibly: +# nothing else re-reads a quiet mate's endpoint. +test_secondmate_captain_held_resurfaces_in_normal_mode() { + local dir state fakebin out capture_file statusf window key pane_hash sig pid back + dir=$(make_case secondmate-held-resurface); state="$dir/state"; fakebin="$dir/fakebin" + out="$dir/watch.out"; capture_file="$dir/pane.txt"; statusf="$state/secondmate-hold.status" + window="test:fm-secondmate-hold" + printf 'idle awaiting the captain\n' > "$capture_file" + printf 'window=%s\nkind=secondmate\n' "$window" > "$state/secondmate-hold.meta" + printf 'captain-held [key=route]: tracked by task-decision-route\n' > "$statusf" + back=$(( $(date +%s) - 500 )) + if [ "$(uname)" = Darwin ]; then touch -mt "$(date -r "$back" '+%Y%m%d%H%M.%S')" "$statusf" + else touch -m -d "@$back" "$statusf"; fi + sig=$(seen_sig "$statusf"); printf '%s' "$sig" > "$state/.seen-secondmate-hold_status" + key=$(printf '%s' "$window" | tr '.:/' '___') + pane_hash=$(hash_text "idle awaiting the captain") + printf '%s' "$pane_hash" > "$state/.hash-$key" + printf '1\n' > "$state/.count-$key" + export FM_FAKE_CREW_STATE='state: unknown · source: none · no current-state source available' + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" FM_PAUSE_RESURFACE_SECS=240 FM_POLL=1 FM_SIGNAL_GRACE=1 \ + FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & + pid=$! + wait_for_exit "$pid" 100 || fail "watcher did not re-surface a captain-held secondmate" + grep -F "stale: $window" "$out" >/dev/null || fail "captain-held secondmate did not emit a stale recheck" + grep -F "awaiting the captain" "$out" >/dev/null || fail "captain-held secondmate recheck did not name the captain as the blocker: $(cat "$out")" + grep -F "awaiting external" "$out" >/dev/null && fail "captain-held secondmate recheck claimed an external wait" + grep -F "possible wedge" "$out" >/dev/null && fail "captain-held secondmate was mislabeled a wedge" + unset FM_FAKE_CREW_STATE + pass "a captain-held secondmate re-surfaces on the bounded normal-mode cadence" +} + test_secondmate_nonpaused_stale_remains_suppressed() { local dir state fakebin out capture_file statusf window key pane_hash sig pid dir=$(make_case secondmate-stale-suppressed); state="$dir/state"; fakebin="$dir/fakebin" @@ -1636,6 +1852,306 @@ test_nonterminal_stale_repairs_missing_or_corrupt_timer() { pass "matching non-terminal stale suppressors repair missing or corrupt stale-since timers" } +# --- quiet pane, worktree still being written: deferred, never wedge-escalated - +# The live 2026-08-14 case: one crew produced eight consecutive possible-wedge +# escalations in an afternoon, three of them demanding deep inspection, while it +# was demonstrably writing source, then tests, then documentation. The detector's +# two inputs (pane quietness, run step) cannot see that, so the pane looks frozen. +# Both halves of the contract are asserted on the SAME fixture, because the whole +# point is that only the worktree evidence differs: writing defers, silent +# escalates on the unchanged schedule. +# Every wait below is the file's standard one (wait_poll_cycle for an absorbing +# watcher, a 100-tick wait_for_exit for an escalating one), because the poll these +# tests assert on is the ONE poll that spawns the bounded worktree walk: on a +# loaded runner it outlives a fixed liveness budget, and a round reaped before it +# finished reports a lost deferral instead of the deferral under test. +test_wedge_escalation_deferred_while_worktree_is_written() { + local dir state fakebin out drain_out capture_file window key pane_hash sig pid wt back + dir=$(make_case wedge-worktree-writes); state="$dir/state"; fakebin="$dir/fakebin" + out="$dir/watch.out"; drain_out="$dir/drain.out"; capture_file="$dir/pane.txt" + window="test:fm-writing"; wt="$dir/wt" + mkdir -p "$wt/src" + printf 'idle building output' > "$capture_file" + printf 'window=%s\nkind=ship\nworktree=%s\n' "$window" "$wt" > "$state/writing.meta" + printf 'working: implementing\n' > "$state/writing.status" + sig=$(seen_sig "$state/writing.status"); printf '%s' "$sig" > "$state/.seen-writing_status" + key=$(printf '%s' "$window" | tr ':/.' '___') + pane_hash=$(hash_text "idle building output") + printf '%s' "$pane_hash" > "$state/.hash-$key" + printf '1\n' > "$state/.count-$key" + # Already-classified hash with an idle window that opened 500s ago, so the very + # first stale poll lands straight on the at-threshold wedge branch (this repeat + # path never re-reads crew state, so the worktree evidence is the only input + # that can change the outcome). + printf '%s' "$pane_hash" > "$state/.stale-$key" + back=$(( $(date +%s) - 500 )) + echo "$back" > "$state/.stale-since-$key" + set_mtime "$back" "$state/.stale-since-$key" + + # Phase A: the crew wrote a file after the idle window opened. Deferred. + printf 'int main(void) { return 0; }\n' > "$wt/src/main.c" + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" FM_STALE_ESCALATE_SECS=240 \ + FM_PAUSE_RESURFACE_SECS=999 FM_POLL=1 FM_SIGNAL_GRACE=1 \ + FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & + pid=$! + if ! wait_poll_cycle "$state" "$pid"; then + reap "$pid"; fail "watcher wedge-escalated a quiet pane whose worktree was being written: $(cat "$out")" + fi + [ ! -s "$out" ] || { reap "$pid"; fail "a written-worktree deferral printed a wake reason: $(cat "$out")"; } + [ ! -s "$state/.wake-queue" ] || { reap "$pid"; fail "a written-worktree deferral enqueued a wake"; } + [ -e "$state/.writing-since-$key" ] || { reap "$pid"; fail "the write-deferral chain marker was not recorded"; } + [ ! -e "$state/.wedge-escalations-$key" ] || { reap "$pid"; fail "a deferral advanced the wedge escalation counter"; } + [ "$(cat "$state/.stale-since-$key" 2>/dev/null || echo 0)" -gt "$back" ] \ + || { reap "$pid"; fail "a deferral did not restart the idle timer, so the next window cannot re-probe"; } + reap "$pid" + ack_stopped_cycle "$state" || fail "could not acknowledge the intentional phase-A watcher stop" + + # Phase B: same fixture, same quiet pane, but nothing written during this idle + # window (the crew really is stalled). The unchanged schedule must still fire. + set_mtime "$(( $(date +%s) - 900 ))" "$wt/src/main.c" + echo "$back" > "$state/.stale-since-$key" + set_mtime "$back" "$state/.stale-since-$key" + : > "$out" + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" FM_STALE_ESCALATE_SECS=240 \ + FM_PAUSE_RESURFACE_SECS=999 FM_POLL=1 FM_SIGNAL_GRACE=1 \ + FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & + pid=$! + wait_for_exit "$pid" 100 || fail "a stalled crew that wrote nothing did not wedge-escalate on the existing schedule" + grep -F "stale: $window" "$out" >/dev/null || fail "the stalled-crew escalation did not print a stale wake" + grep -F "possible wedge" "$out" >/dev/null || fail "the stalled-crew escalation did not flag a possible wedge" + [ "$(cat "$state/.wedge-escalations-$key" 2>/dev/null || true)" = 1 ] || fail "the stalled-crew escalation was not counted" + [ ! -e "$state/.stale-since-$key" ] || fail "the idle timer was not cleared after a real escalation" + [ ! -e "$state/.writing-since-$key" ] || fail "the write-deferral chain outlived a real escalation" + FM_STATE_OVERRIDE="$state" "$DRAIN" > "$drain_out" 2>/dev/null || fail "drain after the stalled-crew escalation failed" + grep "$(printf '\tstale\t')" "$drain_out" | grep -F "$window" >/dev/null || fail "the stalled-crew escalation was not queued" + pass "a quiet pane writing its own worktree is deferred, while one writing nothing still wedge-escalates on the unchanged schedule" +} + +# A deferral is not silence. A worktree can churn without real progress (a +# rewritten log, a build touching the same file), so the whole deferral chain ages +# and re-surfaces once per PAUSE_RESURFACE_SECS - the same bounded cadence a +# declared pause uses - labeled as a recheck rather than a wedge. +test_write_deferral_resurfaces_on_the_bounded_cadence() { + local dir state fakebin out drain_out capture_file window key pane_hash sig pid wt back + dir=$(make_case wedge-worktree-resurface); state="$dir/state"; fakebin="$dir/fakebin" + out="$dir/watch.out"; drain_out="$dir/drain.out"; capture_file="$dir/pane.txt" + window="test:fm-churn"; wt="$dir/wt" + mkdir -p "$wt/src" + printf 'idle building output' > "$capture_file" + printf 'window=%s\nkind=ship\nworktree=%s\n' "$window" "$wt" > "$state/churn.meta" + printf 'working: implementing\n' > "$state/churn.status" + sig=$(seen_sig "$state/churn.status"); printf '%s' "$sig" > "$state/.seen-churn_status" + key=$(printf '%s' "$window" | tr ':/.' '___') + pane_hash=$(hash_text "idle building output") + printf '%s' "$pane_hash" > "$state/.hash-$key" + printf '1\n' > "$state/.count-$key" + printf '%s' "$pane_hash" > "$state/.stale-$key" + back=$(( $(date +%s) - 500 )) + echo "$back" > "$state/.stale-since-$key" + set_mtime "$back" "$state/.stale-since-$key" + # This pane has been deferring on write evidence for 500s already. + : > "$state/.writing-since-$key" + set_mtime "$back" "$state/.writing-since-$key" + printf 'churn\n' > "$wt/src/main.c" + + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" FM_STALE_ESCALATE_SECS=240 \ + FM_PAUSE_RESURFACE_SECS=240 FM_POLL=1 FM_SIGNAL_GRACE=1 \ + FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & + pid=$! + wait_for_exit "$pid" 100 || fail "a long-running write deferral never re-surfaced on the bounded cadence" + grep -F "stale: $window" "$out" >/dev/null || fail "the write-deferral recheck did not print a stale wake" + grep -F "writing its worktree" "$out" >/dev/null || fail "the write-deferral recheck was not labeled as such" + grep -F "possible wedge" "$out" >/dev/null && fail "a write-deferral recheck was mislabeled a possible wedge" + [ -e "$state/.writing-resurfaced-$key" ] || fail "the write-deferral re-surface throttle marker was not recorded" + [ ! -e "$state/.wedge-escalations-$key" ] || fail "a write-deferral recheck advanced the wedge escalation counter" + FM_STATE_OVERRIDE="$state" "$DRAIN" > "$drain_out" 2>/dev/null || fail "drain after the write-deferral recheck failed" + grep "$(printf '\tstale\t')" "$drain_out" | grep -F "$window" >/dev/null || fail "the write-deferral recheck was not queued" + pass "a write deferral re-surfaces once on the bounded pause cadence, so a churning worktree cannot stay invisible" +} + +# The worktree recorded for a secondmate is a provisioned firstmate home, and that +# home runs its OWN supervision inside itself: its watcher beacon, pane hashes and +# heartbeats keep state/ churning whether or not the mate produced anything. Reading +# that as crew progress would quietly relax the kind-agnostic busy-turn backstop from +# the escalation cadence to the hourly recheck for work that produced nothing, so the +# probe must report no evidence and the unchanged schedule must still fire. +test_secondmate_home_supervision_churn_is_not_write_evidence() { + local dir state fakebin out drain_out capture_file window key sig pid home back + dir=$(make_case secondmate-home-churn); state="$dir/state"; fakebin="$dir/fakebin" + out="$dir/watch.out"; drain_out="$dir/drain.out"; capture_file="$dir/pane.txt" + window="test:fm-mate"; home="$dir/mate-home" + mkdir -p "$home/state" + printf 'sm-mate\n' > "$home/.fm-secondmate-home" + printf 'Working... (12.3s)' > "$capture_file" + printf 'window=%s\nkind=ship\nharness=pi\nworktree=%s\n' "$window" "$home" > "$state/mate.meta" + record_pi_busy "$state" mate + # An ordinary crew recording a provisioned mate home is the route that actually + # reaches the probe: a kind=secondmate window of its own is triaged only under a + # declared pause, and a declared pause takes the bounded recheck cadence instead of + # the wedge timer. The home marker alone is what excludes the walk, so the exclusion + # is what this asserts. A busy pane is bounded by its completed-turn age; no turn + # ever completed here, so the spawn record itself is aged past the bound that routes + # it into the wedge timer. + printf 'working: implementing\n' > "$state/mate.status" + sig=$(seen_sig "$state/mate.status"); printf '%s' "$sig" > "$state/.seen-mate_status" + key=$(printf '%s' "$window" | tr ':/.' '___') + set_mtime "$(( $(date +%s) - 4000 ))" "$state/mate.meta" + back=$(( $(date +%s) - 500 )) + echo "$back" > "$state/.stale-since-$key" + set_mtime "$back" "$state/.stale-since-$key" + # The only thing written since the idle window opened is the mate home's own + # supervision bookkeeping. + printf 'beat\n' > "$home/state/.last-watcher-beat" + + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" \ + FM_STALE_ESCALATE_SECS=240 FM_BUSY_TURN_MAX_SECS=1 FM_PAUSE_RESURFACE_SECS=999 \ + FM_POLL=1 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & + pid=$! + wait_for_exit "$pid" 100 || fail "a mate home's own supervision churn deferred an escalation it must not defer" + grep -F "stale: $window" "$out" >/dev/null || fail "the mate-home escalation did not print a stale wake" + grep -F "possible wedge" "$out" >/dev/null || fail "the mate-home escalation did not flag a possible wedge" + [ ! -e "$state/.writing-since-$key" ] || fail "a mate's provisioned home was probed as if it were a code tree" + [ "$(cat "$state/.wedge-escalations-$key" 2>/dev/null || true)" = 1 ] || fail "the mate escalation was not counted" + FM_STATE_OVERRIDE="$state" "$DRAIN" > "$drain_out" 2>/dev/null || fail "drain after the mate escalation failed" + grep "$(printf '\tstale\t')" "$drain_out" | grep -F "$window" >/dev/null || fail "the mate escalation was not queued" + pass "a secondmate's own home supervision churn is not crew write evidence, so a pane recording that home keeps the unchanged escalation schedule" +} + +# A write deferral is a bounded chain, not a permanent one: its .writing-since +# marker ages the whole chain so a churning worktree still re-surfaces once per +# PAUSE_RESURFACE_SECS. That only holds while the chain belongs to the CURRENT quiet +# stretch, so every path that restarts the idle-window timer must drop it too. The +# reachable case is a pane that deferred on write evidence and later has its timer +# repaired: a long-finished chain would make the first deferral of the new window +# re-surface immediately instead of after a fresh window. +test_timer_repair_drops_a_finished_write_deferral_chain() { + local dir state fakebin out capture_file window key pane_hash sig pid wt back + dir=$(make_case wedge-write-chain-timer-repair); state="$dir/state"; fakebin="$dir/fakebin" + out="$dir/watch.out"; capture_file="$dir/pane.txt" + window="test:fm-chain-repair"; wt="$dir/wt" + mkdir -p "$wt/src" + printf 'idle building output' > "$capture_file" + printf 'window=%s\nkind=ship\nworktree=%s\n' "$window" "$wt" > "$state/chain-repair.meta" + printf 'working: implementing\n' > "$state/chain-repair.status" + sig=$(seen_sig "$state/chain-repair.status"); printf '%s' "$sig" > "$state/.seen-chain-repair_status" + key=$(printf '%s' "$window" | tr ':/.' '___') + pane_hash=$(hash_text "idle building output") + printf '%s' "$pane_hash" > "$state/.hash-$key" + printf '1\n' > "$state/.count-$key" + printf '%s' "$pane_hash" > "$state/.stale-$key" + # A deferral chain left over from an earlier quiet stretch, already well past the + # bounded re-surface window. + back=$(( $(date +%s) - 5000 )) + : > "$state/.writing-since-$key" + set_mtime "$back" "$state/.writing-since-$key" + # The idle-window timer is corrupt, so this poll repairs it and opens a NEW quiet + # window without probing the worktree at all. + printf 'corrupt\n' > "$state/.stale-since-$key" + + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" \ + FM_STALE_ESCALATE_SECS=240 FM_PAUSE_RESURFACE_SECS=240 FM_POLL=1 FM_SIGNAL_GRACE=1 \ + FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & + pid=$! + wait_numeric_file "$state/.stale-since-$key" 30 \ + || { reap "$pid"; fail "the corrupt idle-window timer was not repaired"; } + [ ! -e "$state/.writing-since-$key" ] \ + || { reap "$pid"; fail "an idle-window timer repair kept a finished write-deferral chain"; } + [ ! -s "$state/.wake-queue" ] || { reap "$pid"; fail "the idle-window timer repair enqueued a wake"; } + reap "$pid" + ack_stopped_cycle "$state" || fail "could not acknowledge the intentional timer-repair watcher stop" + + # The new quiet window now crosses the escalation threshold while the crew writes + # its worktree. That deferral must get a FRESH re-surface window rather than + # inheriting the finished chain's age. + back=$(( $(date +%s) - 500 )) + echo "$back" > "$state/.stale-since-$key" + set_mtime "$back" "$state/.stale-since-$key" + printf 'int main(void) { return 0; }\n' > "$wt/src/main.c" + : > "$out" + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" \ + FM_STALE_ESCALATE_SECS=240 FM_PAUSE_RESURFACE_SECS=240 FM_POLL=1 FM_SIGNAL_GRACE=1 \ + FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & + pid=$! + if ! wait_poll_cycle "$state" "$pid"; then + reap "$pid" + fail "the first deferral of a new quiet window re-surfaced at once, so it inherited a finished chain: $(cat "$out")" + fi + [ ! -s "$out" ] || { reap "$pid"; fail "a fresh write deferral printed a wake reason: $(cat "$out")"; } + [ ! -s "$state/.wake-queue" ] || { reap "$pid"; fail "a fresh write deferral enqueued a wake"; } + [ -e "$state/.writing-since-$key" ] || { reap "$pid"; fail "the new deferral recorded no chain marker"; } + [ ! -e "$state/.writing-resurfaced-$key" ] \ + || { reap "$pid"; fail "a fresh write deferral spent its bounded re-surface on the first poll"; } + reap "$pid" + pass "an idle-window timer repair drops a finished write-deferral chain, so the next deferral gets a fresh re-surface window" +} + +# The same chain must not outlive either first-sight path through a captain-relevant +# status line, because both also open a new idle window: the provably-working absorb +# and the plain surface. +test_terminal_first_sight_drops_a_finished_write_deferral_chain() { + local dir state fakebin out capture_file window key pane_hash sig pid wt back + dir=$(make_case wedge-write-chain-first-sight); state="$dir/state"; fakebin="$dir/fakebin" + out="$dir/watch.out"; capture_file="$dir/pane.txt" + window="test:fm-chain-firstsight"; wt="$dir/wt" + mkdir -p "$wt/src" + printf 'no-mistakes axi run: validating...' > "$capture_file" + printf 'window=%s\nkind=ship\nworktree=%s\n' "$window" "$wt" > "$state/chain-first.meta" + printf 'done: implementation complete, ready to validate\n' > "$state/chain-first.status" + sig=$(seen_sig "$state/chain-first.status"); printf '%s' "$sig" > "$state/.seen-chain-first_status" + key=$(printf '%s' "$window" | tr ':/.' '___') + pane_hash=$(hash_text "no-mistakes axi run: validating...") + printf '%s' "$pane_hash" > "$state/.hash-$key" + printf '1\n' > "$state/.count-$key" + back=$(( $(date +%s) - 5000 )) + : > "$state/.writing-since-$key" + set_mtime "$back" "$state/.writing-since-$key" + export FM_FAKE_CREW_STATE='state: working · source: run-step · validating (running)' + + # First sight of this hash, absorbed because the active run outranks the stale + # captain-relevant line. The absorb opens a new idle window, so the finished chain + # must go with it. + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" \ + FM_STALE_ESCALATE_SECS=999 FM_PAUSE_RESURFACE_SECS=240 FM_POLL=1 FM_SIGNAL_GRACE=1 \ + FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & + pid=$! + if ! wait_poll_cycle "$state" "$pid"; then + reap "$pid"; fail "the overridden terminal status was not absorbed on first sight: $(cat "$out")" + fi + [ "$(cat "$state/.stale-$key" 2>/dev/null || true)" = "$pane_hash" ] \ + || { reap "$pid"; fail "the first-sight absorb did not advance the stale suppressor"; } + [ ! -e "$state/.writing-since-$key" ] \ + || { reap "$pid"; fail "the provably-working first-sight absorb kept a finished write-deferral chain"; } + reap "$pid" + ack_stopped_cycle "$state" || fail "could not acknowledge the intentional first-sight absorb stop" + + # Same pane, first sight again, but nothing overrides the status line now, so it + # surfaces. That path drops the idle-window timer, so it must drop the chain too. + rm -f "$state/.stale-$key" "$state/.stale-since-$key" + printf '1\n' > "$state/.count-$key" + : > "$state/.writing-since-$key" + set_mtime "$back" "$state/.writing-since-$key" + FM_FAKE_CREW_STATE='state: unknown · source: none · no run, no busy pane' + : > "$out" + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" \ + FM_STALE_ESCALATE_SECS=999 FM_PAUSE_RESURFACE_SECS=240 FM_POLL=1 FM_SIGNAL_GRACE=1 \ + FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & + pid=$! + wait_for_exit "$pid" 100 || fail "a first-sight captain-relevant status was not surfaced" + grep -F "stale: $window" "$out" >/dev/null || fail "the first-sight surface did not print a stale wake" + [ ! -e "$state/.writing-since-$key" ] \ + || fail "the first-sight surface kept a finished write-deferral chain" + unset FM_FAKE_CREW_STATE + pass "both first-sight paths through a captain-relevant status drop a finished write-deferral chain with the idle window" +} + # --- triage debug log stays size capped ------------------------------------- test_triage_log_size_cap_accepts_spaced_wc_counts() { @@ -2097,6 +2613,10 @@ test_classifier_primitives test_crew_is_provably_working_classifier test_status_is_paused_classifier test_crew_absorb_class_classifier +test_crew_worktree_written_since_classifier +test_empty_write_prune_widens_the_probe +test_empty_write_prune_from_the_environment_widens_the_probe +test_worktree_write_probe_is_wall_clock_bounded test_signal_crew_provably_working_classifier test_secondmate_status_signal_never_absorbed_classifier test_provably_working_signal_absorbed @@ -2122,12 +2642,18 @@ test_nonterminal_stale_not_working_surfaced test_nonterminal_stale_paused_absorbed_then_resurfaced test_exited_declared_pause_is_bounded_but_live_gate_surfaces test_secondmate_paused_resurfaces_in_normal_mode +test_secondmate_captain_held_resurfaces_in_normal_mode test_secondmate_nonpaused_stale_remains_suppressed test_secondmate_unpause_clears_pause_tracking test_nonterminal_stale_pause_transitions_reclassify_unchanged_hash test_nonterminal_paused_rechecks_authoritative_state test_paused_authoritative_working_preserves_wedge_timer test_nonterminal_stale_repairs_missing_or_corrupt_timer +test_wedge_escalation_deferred_while_worktree_is_written +test_write_deferral_resurfaces_on_the_bounded_cadence +test_secondmate_home_supervision_churn_is_not_write_evidence +test_timer_repair_drops_a_finished_write_deferral_chain +test_terminal_first_sight_drops_a_finished_write_deferral_chain test_triage_log_size_cap_accepts_spaced_wc_counts test_procevent_captured_result_surfaces_proactively test_procevent_unacknowledged_result_redrains_until_handled diff --git a/tests/fm-watcher-lock.test.sh b/tests/fm-watcher-lock.test.sh index a3628b1694f..482e425a9f5 100755 --- a/tests/fm-watcher-lock.test.sh +++ b/tests/fm-watcher-lock.test.sh @@ -13,6 +13,13 @@ WATCH_ARM="$ROOT/bin/fm-watch-arm.sh" DRAIN="$ROOT/bin/fm-wake-drain.sh" LIB="$ROOT/bin/fm-wake-lib.sh" +# An arm only reports its typed failure after wait_for_healthy_successor has +# spent the whole confirmation budget, so cases that wait for that failure must +# outlast the largest production default (30s on MSYS, 10s elsewhere - see +# ARM_CONFIRM_DEFAULT in bin/fm-watch-arm.sh). This is a ceiling spent only when +# an arm genuinely fails to exit; a passing case returns as soon as it does. +ARM_FAIL_EXIT_POLLS=400 + TMP_ROOT=$(fm_test_tmproot fm-watcher-lock-tests) mark_pr_check_migration_complete() { @@ -536,7 +543,14 @@ test_arm_self_eviction_is_loud_without_successor() { fakebin="$dir/fakebin" armout="$dir/arm.out" mark_pr_check_migration_complete "$state" - PATH="$fakebin:$PATH" FM_STATE_OVERRIDE="$state" FM_POLL=0.2 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 FM_ARM_CONFIRM_TIMEOUT=1 "$WATCH_ARM" > "$armout" & + # The arm's confirmation budget bounds a REAL child startup (fork, exec, lock + # acquisition, beacon publication), so this case holds the arm to production's + # own budget rather than a shrunken fixture one: a one-second budget turned + # ordinary CPU contention into an honest "FAILED - no live watcher with a fresh + # beacon" and broke this case's premise under full-suite load (issue #2844). + # It stays at the production default rather than something roomier because the + # same budget bounds the successor wait this case deliberately spends below. + PATH="$fakebin:$PATH" FM_STATE_OVERRIDE="$state" FM_POLL=0.2 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH_ARM" > "$armout" & armpid=$! i=0 while [ "$i" -lt 80 ]; do @@ -551,7 +565,7 @@ test_arm_self_eviction_is_loud_without_successor() { # self-evict normally. With no verified successor, the arm must turn that # otherwise clean empty close into the typed nonzero failure. printf '%s\n' "$$" > "$state/.watch.lock/pid" - wait_for_exit "$armpid" 80 + wait_for_exit "$armpid" "$ARM_FAIL_EXIT_POLLS" status=$? [ "$status" -ne 0 ] && [ "$status" -ne 124 ] || fail "self-evicted arm did not fail nonzero (status $status)" grep -qF 'watcher: FAILED - cycle ended without an actionable reason' "$armout" || fail "self-evicted arm omitted the typed cycle-end failure" @@ -742,7 +756,13 @@ SH FM_STATE_OVERRIDE="$state" "$ROOT/bin/fm-check-register.sh" task >/dev/null \ || fail "could not register immediate-wake custom check" rc=0 - PATH="$fakebin:$PATH" FM_STATE_OVERRIDE="$state" FM_GUARD_GRACE=0 FM_POLL=5 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=0 FM_HEARTBEAT=999999 "$WATCH_ARM" > "$armout" || rc=$? + # This case asserts wake propagation, not the confirmation deadline, and its + # child must also run the registered check before exiting: measured at 1.9-2.3s + # idle but 9.1-13.1s at 3x CPU oversubscription, against an 11s production + # budget. An explicit budget takes the deadline out of the assertion and costs + # nothing on a passing run, because the arm returns as soon as the child + # settles (issue #2844). + PATH="$fakebin:$PATH" FM_STATE_OVERRIDE="$state" FM_GUARD_GRACE=0 FM_POLL=5 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=0 FM_HEARTBEAT=999999 FM_ARM_CONFIRM_TIMEOUT=60 "$WATCH_ARM" > "$armout" || rc=$? [ "$rc" -eq 0 ] || fail "arm returned non-zero for an immediate wake (status $rc): $(cat "$armout")" grep -F "check: $check_file: merged: https://example.test/pr/7" "$armout" >/dev/null || fail "arm did not propagate the immediate check wake" ! grep -qF 'watcher: FAILED' "$armout" || fail "arm printed FAILED after a valid immediate wake" @@ -766,12 +786,15 @@ test_arm_waits_for_peer_beacon_after_child_stands_down() { printf '%s\n' "$dir" > "$state/.watch.lock/fm-home" printf '%s\n' "$WATCH" > "$state/.watch.lock/watcher-path" printf '%s\n' "$identity" > "$state/.watch.lock/pid-identity" - PATH="$fakebin:$PATH" FM_HOME="$dir" FM_POLL=5 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 FM_ARM_CONFIRM_TIMEOUT=1 FM_ARM_ATTACH_POLL=0.1 "$WATCH_ARM" > "$armout" & + # Same budget contract as the self-eviction case: the owned child's real + # startup and stand-down happen inside the arm's confirmation window, so the + # window stays production-sized (issue #2844). + PATH="$fakebin:$PATH" FM_HOME="$dir" FM_POLL=5 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 FM_ARM_ATTACH_POLL=0.1 "$WATCH_ARM" > "$armout" & armpid=$! # Synchronize on the owned child declining the live peer lock before making - # the peer healthy. Sleeping for the same one-second budget as the arm made - # this regression fixture race the confirmation deadline under full-suite - # load, rather than testing the intended successor-handshake boundary. + # the peer healthy. Sleeping for the same budget the arm spends made this + # regression fixture race the confirmation deadline under full-suite load, + # rather than testing the intended successor-handshake boundary. i=0 while [ "$i" -lt 80 ]; do grep -qF "watcher: already running pid $peer" "$state"/.watch-arm-output.* 2>/dev/null && break @@ -793,7 +816,7 @@ test_arm_waits_for_peer_beacon_after_child_stands_down() { # After the peer dies without a successor, the attached arm must fail loudly. kill "$peer" 2>/dev/null || true wait "$peer" 2>/dev/null || true - wait_for_exit "$armpid" 80 + wait_for_exit "$armpid" "$ARM_FAIL_EXIT_POLLS" status=$? [ "$status" -ne 0 ] && [ "$status" -ne 124 ] || fail "attached arm did not fail after peer died (status $status): $(cat "$armout")" grep -qF 'watcher: FAILED - cycle ended without an actionable reason' "$armout" || fail "peer-attached arm did not emit the typed cycle-end failure" diff --git a/tests/wake-helpers.sh b/tests/wake-helpers.sh index 99481201cb2..8e6281a5763 100644 --- a/tests/wake-helpers.sh +++ b/tests/wake-helpers.sh @@ -123,6 +123,11 @@ prime_status_seen() { # <state> <file> ' _ "$ROOT/bin/fm-wake-lib.sh" "$1" "$2" } +# Print the generation from a recovery marker token of any status/kind. +recovery_marker_generation() { # <marker-file> + sed -n 's/^[^:]*:[^:]*:\(.*\)$/\1/p' "$1" +} + # Acknowledge a drain from its captured stderr (the WAKE_ACK_REQUIRED line). ack_drain_err() { # <state> <stderr-file> local state=$1 err=$2 sequence generation