diff --git a/.agents/skills/afk/SKILL.md b/.agents/skills/afk/SKILL.md index 89cd8de65ce..4642075eb31 100644 --- a/.agents/skills/afk/SKILL.md +++ b/.agents/skills/afk/SKILL.md @@ -125,9 +125,7 @@ For tmux that confirmation is normally a proven cleared composer from the shared Without that baseline, busy state never converts an `unknown` composer into confirmation. For herdr, idle-baseline submits first seek native agent-state showing a real turn started, then use the shared classifier when native state remains idle: a cleared composer confirms delivery, while pending text retries Enter and reaches the shared busy-queue verdict only after the retry budget. A bordered-empty or ghost-only composer is recognized as empty where that backend uses composer confirmation, rather than mistaken for a swallowed Enter. -`fm-send.sh` uses the same primitive and exits non-zero -when a steer's Enter is positively swallowed, so firstmate learns an instruction -did not land instead of leaving it unsubmitted. +`fm-send.sh` uses the same primitive only on its typed plane and exits non-zero when that plane's Enter is positively swallowed; ordinary local text steers use the durable inbox and do not treat doorbell submission as delivery proof. **Busy-queued Enter exception (opencode 1.18.4).** OpenCode keeps queued text visible while it is mid-turn, so tmux and herdr delegate the final delivery decision to `fm_composer_queued_enter_verdict` in `bin/fm-composer-lib.sh` rather than treating visible text alone as a swallowed Enter. The daemon still clears its buffer only on the backend's `empty` success verdict; [`docs/tmux-backend.md`](../../../docs/tmux-backend.md) and [`docs/herdr-backend.md`](../../../docs/herdr-backend.md) own the backend-specific confirmation signals. diff --git a/.agents/skills/bootstrap-diagnostics/SKILL.md b/.agents/skills/bootstrap-diagnostics/SKILL.md index a195ce64422..f6c7b18ad14 100644 --- a/.agents/skills/bootstrap-diagnostics/SKILL.md +++ b/.agents/skills/bootstrap-diagnostics/SKILL.md @@ -53,8 +53,8 @@ When any diagnostic needs captain attention, report the plain consequence and re - `SECONDMATE_SYNC: secondmate : skipped: ` - secondmate convergence left a live home on its existing checkout because the home was dirty, diverged, unsafe, on the wrong branch, missing its placement-specific target commit, unreachable, or otherwise not fast-forwardable, or because inherited local-material propagation failed; bootstrap continued, but inspect the reason because the secondmate's tracked instructions, inherited settings, or shared captain preferences may be stale after a primary update. - `SECONDMATE_LIVENESS: secondmate : skipped: |respawn failed after : ` - the session-start liveness sweep could not guarantee that the registered secondmate is running a real agent process. Investigate the reason because that secondmate is not guaranteed live. -- `SECONDMATE_HANDOFF: secondmate : pending delivery: item(s)` - queued work has already left the main dispatchable backlog and remains safe in the named remote route's backlog-format outbox. - Preserve that outbox and rerun `bin/fm-backlog-handoff.sh --resume-pending` after same-host connectivity returns; never re-add or dispatch the items from the main backlog. +- `SECONDMATE_HANDOFF: secondmate : pending delivery: item(s)` - queued work has already left the main dispatchable backlog and remains safe in the named remote route's backlog-format outbox, pending backlog receipt or receiver-wake confirmation. + Preserve that outbox and rerun `bin/fm-backlog-handoff.sh --resume-pending` after the route or endpoint problem is resolved; never re-add or dispatch the items from the main backlog. An unsafe-outbox variant requires path and file-type inspection before any retry. - `NUDGE_SECONDMATES: secondmate : send failed: ` - secondmate convergence changed a running home's loaded instructions or inherited config, but the deterministic `fm-send.sh fm-` re-read nudge failed. Inspect the reason, keep the pending marker under `state/.secondmate-nudge-pending/` intact, and rerun session start after the endpoint or metadata issue is fixed so bootstrap can retry the exact same marked send on the same local or remote route. diff --git a/.agents/skills/firstmate-orca/SKILL.md b/.agents/skills/firstmate-orca/SKILL.md index c6c23b07121..939f6698b9b 100644 --- a/.agents/skills/firstmate-orca/SKILL.md +++ b/.agents/skills/firstmate-orca/SKILL.md @@ -52,15 +52,15 @@ Do not manually patch metadata to make an externally-created Orca terminal look ## Supervision Use `bin/fm-peek.sh`, `bin/fm-send.sh`, `bin/fm-crew-state.sh`, and `bin/fm-teardown.sh` for routine operation. -For steer messages, send short lines through `bin/fm-send.sh '...'`; the stable `fm-` alias also works. -Put long instructions in the task brief or a temporary file and point the crewmate at that file. +For steer messages, use `bin/fm-send.sh '...'`; the stable `fm-` alias also works, and ordinary local text steers may contain newlines because they ride the durable inbox. +Keep initial scope in the task brief; a temporary file remains useful when the instruction includes supporting material the worker should inspect separately. When supervising, treat `state/.meta` as the routing record and Orca's own ids as backend implementation details. The stable firstmate alias is `fm-`. The recorded `terminal=` and `orca_worktree_id=` fields are what backend helpers use under the hood. -If `fm-send` fails to submit, do not immediately repeat the same long instruction. -Peek first, then decide whether the target is busy, waiting on a prompt, stuck behind a popup, or genuinely wedged. +If an ordinary steer fails to enqueue, or a typed-plane `fm-send` fails to submit, do not immediately repeat the instruction. +Read the reported failure and peek first, then decide whether the record exists or the target is busy, waiting on a prompt, stuck behind a popup, or genuinely wedged. For harness-specific interrupts or exits, load `harness-adapters`. ## Recovery diff --git a/.agents/skills/harness-adapters/SKILL.md b/.agents/skills/harness-adapters/SKILL.md index 1b3c36ecc49..d20a7dfaa62 100644 --- a/.agents/skills/harness-adapters/SKILL.md +++ b/.agents/skills/harness-adapters/SKILL.md @@ -258,9 +258,7 @@ If a pane shows the exit banner, relaunch with `--continue` to resume the sessio While opencode is mid-turn, the composer accepts Enter as a "send when the turn ends" keystroke but does not clear the typed text from the composer until the turn actually finishes. -Without a conversion, every `fm-send` to a busy opencode pane exits non-zero on a -false "Enter swallowed", and every daemon escalation that lands while the -primary is mid-turn is treated as wedged. +Without a conversion, every typed-plane `fm-send` to a busy opencode pane exits non-zero on a false "Enter swallowed", and every daemon escalation that lands while the primary is mid-turn is treated as wedged. Both tmux and herdr delegate this exception to the one policy in `fm_composer_queued_enter_verdict` (`bin/fm-composer-lib.sh`), with backend-specific signals documented in `docs/tmux-backend.md` and `docs/herdr-backend.md`. Regression coverage is `tests/fm-tmux-submit-busy.test.sh`, `tests/fm-composer-lib.test.sh`, and `tests/fm-backend-herdr.test.sh`; the live Herdr Claude guard is `FM_HERDR_SUBMIT_CONFIRM_LIVE=1 tests/fm-herdr-submit-confirm-live-e2e.test.sh`. @@ -407,8 +405,8 @@ Match that TOKEN and never the spinner verb: the same version rendered `Working` **Delivery confirmation is verified on tmux and Herdr only.** Herdr reports a Cursor pane `blocked` in EVERY state - idle, mid-turn, and after - so its native idle-baseline submit path is unreachable for Cursor and the composer branch runs instead; that branch reads a mid-turn row carrying the placeholder beside `ctrl+c to stop`, which is `pending`. `bin/backends/herdr.sh` therefore confirms a Cursor submit from a rendered-footer idle-to-busy transition, taking the baseline before the first Enter so an already-busy pane never confirms. -Zellij, cmux, and Orca share a submit core that never consults that footer, so a Cursor steer there LANDS but `bin/fm-send.sh` reports delivery unconfirmed and exits non-zero. -Treat that as a known limitation of those three backends rather than a lost message: the steer is in the pane and the worker's own recorded state still comes from its transcript fold. +Zellij, cmux, and Orca share a submit core that never consults that footer, so a typed-plane Cursor send there (a harness-native invocation or an explicit backend target; ordinary text steers ride the durable inbox and exit 0 at enqueue) LANDS but `bin/fm-send.sh` reports delivery unconfirmed and exits non-zero. +Treat that as a known limitation of those three backends rather than a lost message: the text is in the pane and the worker's own recorded state still comes from its transcript fold. Teaching the shared core the same transition is deliberately separate work, because it changes the submit path for every harness on those three backends and needs its own live validation on each. The composer's reverse-video placeholder remnant is taught to the ONE fleet-wide screen classifier in `bin/fm-composer-lib.sh`, not to any adapter. diff --git a/.agents/skills/secondmate-provisioning/SKILL.md b/.agents/skills/secondmate-provisioning/SKILL.md index cf26c791c03..6f6981dc2b7 100644 --- a/.agents/skills/secondmate-provisioning/SKILL.md +++ b/.agents/skills/secondmate-provisioning/SKILL.md @@ -190,7 +190,9 @@ After seeding, run this handoff for the new secondmate's in-scope queued items. For an existing or inherited domain, complete record intake first so no already-shipped plan row is handed off as open work. For a local route, the helper resolves and validates the secondmate home from `data/secondmates.md`, then delegates the item move to `tasks-axi mv` (the single owner of the backlog format), which moves each named item - and a whole connected set, blocker plus dependents, atomically - from the main `data/backlog.md` into the secondmate home's `data/backlog.md`. For a remote route, the same helper first moves the dependency-closed set atomically from the main backlog into `data/handoff/.outbox.md`, then transfers that backlog-format outbox through `fm-on.sh` and lets the remote home's `fm-backlog-receive.sh` move every not-already-present key under the destination lock. -The outbox is the whole recovery record: its presence means delivery is unfinished, `--resume-pending` safely re-delivers it, and confirmed receipt removes it. +After a new local placement or a remote outbox receipt becomes durable, the helper sends one marked routed-work instruction through the receiving secondmate's recorded endpoint; missing or failed delivery makes the command fail loudly with the moved work intact, and the same handoff command retries known-undelivered wake intent without moving an already-present item again. +An unresolved delivery attempt is never blindly resent. +For a remote route, the outbox remains until both backlog receipt and receiver wake are confirmed; `--resume-pending` retries unfinished outboxes, while the script header owns its stable wake-correlation recovery state. There is no two-phase handoff journal and no tasks-axi release beyond the already-required atomic `mv` capability. Bootstrap retries pending outboxes when mutation is authorized and emits `SECONDMATE_HANDOFF:` for any that remain. This delegated route remains required when `config/backlog-backend=manual`, which controls only routine firstmate backlog edits. diff --git a/.agents/skills/stow/SKILL.md b/.agents/skills/stow/SKILL.md index c7d96ce30db..348a9975471 100644 --- a/.agents/skills/stow/SKILL.md +++ b/.agents/skills/stow/SKILL.md @@ -20,6 +20,8 @@ Markers are compact trailing HTML comments, deliberately cheap because marker by - `` - an `aging` entry; the embedded date is its last-reinforced date. - `` - a `perishable` entry; the embedded date is its last-reinforced date. +- `` - only in a home that has opted in to the pass horizon below: either dated marker may carry `/N`, the number of passes that evaluated the entry without reinforcing it. + An absent `/N` means zero, so an entry the fleet keeps exercising costs no counter bytes at all, and a home that has not opted in never writes one. - `` - an explicitly `pinned` entry in a file whose default tier is not `pinned`. - `` - migration-only: an unconfirmed legacy entry that has consumed its one grace cycle, carrying no date because grace is not reinforcement. @@ -27,6 +29,7 @@ Markers are compact trailing HTML comments, deliberately cheap because marker by - Treehouse pool slots share one repo, so workers must create their task branch before editing. - While state/.afk exists, the away-daemon owns triage (until the afk-wake fix lands; tracked: afk-pi-wake-bypass-r1). - Never restart the shared no-mistakes daemon while runs are active. +- Codex writes its trust prompt to stderr, not stdout. ``` The tier names say what the pass does with an entry: @@ -43,13 +46,33 @@ Marking rules: - An entry matching its file's `pinned` default carries no marker at all; every `aging` and `perishable` entry always carries its dated marker, whose letter names the tier, so a clock-carrying entry is never ambiguous with unmarked legacy material. - Marker and header-pointer bytes count toward the startup-memory budget: the pass's own bookkeeping is costed content, never free, which is why the spellings above are as short as they are. - Each memory file's header carries at most a one-line pointer naming this skill as the scheme owner, such as ``. - This skill text is the single owner of tier semantics, marker spellings, and clocks - deliberately policy, not configuration - and no memory file header may restate them. + This skill text is the single owner of tier semantics, marker spellings, and clocks, and no memory file header may restate them. + The one exception is the `config/stow-pass-horizon` presence flag below, which turns a single extra horizon on for this home and changes nothing else on this page. - Inspect each editable file's header pointer on every pass and add or correct it; for a read-only `data/captain-shared.md`, leave the file byte-identical and route a missing or outdated pointer to the primary owner. The required receipt action for that file is `routed`, not `unchanged`; name the ownership exception and do not declare the session reset-safe. - A pre-existing missing or hand-dropped marker is never grounds for destructive treatment: it means the file's default tier; an unmarked entry in a default-pinned file is simply pinned, while an unmarked entry in a file whose default tier carries a clock follows the migration rule below. Decay advances only when a pass runs, so a home stowed less often than a clock experiences that clock at its stow interval. +### Optional pass horizon (config/stow-pass-horizon) + +The wall-clock horizons above are this skill's default contract, and a home gets exactly them unless it asks for more. +A home may opt in to a second, per-pass horizon by creating the local, gitignored `config/stow-pass-horizon` presence flag. +While that file is absent nothing else in this section applies: no counter is written, no counter already in a file is read, and every entry decays on its date alone. + +Opt in where admission and decay are not commensurable. +A pass admits the findings that pass produced, so growth is a per-pass quantity, while a wall-clock horizon alone is a per-day one. +In a home that stows daily those two rates diverge by the stow cadence, an entry the fleet keeps exercising never sits unreinforced for 30 wall-clock days, and the date horizon is evaluated vacuously every pass while the file only grows. +A home stowed monthly already exceeds its date horizon on a single pass and gains nothing from the flag. + +While the flag is present: + +- An `aging` entry is stale at whichever horizon it reaches first: 10 passes that evaluated it without reinforcing it, or 30 days since its last-reinforced date. +- A `perishable` entry is stale at whichever it reaches first: 3 unreinforced passes, or 7 days. +- Reinforcement refreshes the date and clears the counter, and nothing else clears it, so the evidence hard rule in step 4 stays the only way an entry renews its lease. +- An existing dated marker with no `/N` reads as counter zero, so a home that opts in migrates nothing. +- Removing the flag returns the home to the default contract on its next pass: any `/N` already written is then neither read nor advanced, and is left in place rather than rewritten. + ## Required startup-memory pass Every `/stow` invocation performs this complete pass, even when the session contains no new finding: @@ -72,10 +95,12 @@ Every `/stow` invocation performs this complete pass, even when the session cont Retain lower-utility material only while budget remains. 4. Reinforce and stamp. Refresh an entry's last-reinforced date to today only when this session actually exercised, confirmed, or re-derived it. + Where the optional pass horizon is enabled, refreshing that date also clears the entry's unreinforced-pass counter, and nothing else clears it. **Hard rule: reinforcement requires independent evidence from this session that you can name in the receipt; plausibility, importance, prior knowledge, and the entry's own text are not evidence, and any explicit statement that no confirming session evidence exists requires the no-evidence path.** For an unmarked `data/learnings.md` entry with no such evidence, the no-evidence path is always to append `` and retain it for this entire pass; never stamp or archive it during that same invocation. Stamp each newly written entry with today's date and its tier per the marking rules, and admit a new `perishable` entry only with its named checkable expiry condition in the prose. 5. Evaluate every dated entry in each editable memory file against its tier clock. + Where the optional pass horizon is enabled, first increment the unreinforced-pass counter of every dated entry step 4 did not reinforce - that increment is the pass tick - then judge each dated entry against both of its horizons and treat it as stale at whichever it reaches first. Re-validate a stale `aging` entry from current evidence and refresh its date, or archive it. Re-confirm a stale `perishable` entry against its named condition: still open means refresh the date, while resolved, expired, or no longer checkable means archive it in this pass. Promote `perishable` to `aging` when its condition keeps proving durable past its expected life, and retier in place when a supersession changes an entry's lifetime. @@ -108,6 +133,7 @@ Never describe the session as reset-safe while the memory total is over budget o Stale never means deleted: pruning an entry from an editable memory file always means moving it to `data/memory-archive.md`, this home's append-only, never-injected cold tier, gitignored with the rest of `data/` and never counted by the budget report. Each archived entry keeps its provenance under a dated pass heading: source file, tier, last-reinforced date, and the reason it left. +Include the unreinforced-pass counter only when the optional pass horizon itself made the entry stale, using the exact reason `unreinforced p`; omit the counter when the wall-clock horizon or any other reason caused archival, even if the active marker carried one. Archive provenance stays verbose rather than compact because the cold tier is never budget-counted. ```markdown @@ -115,7 +141,7 @@ Archive provenance stays verbose rather than compact because the cold tier is ne - (from learnings.md, tier: perishable, reinforced: 2026-06-30) While state/.afk exists, the away-daemon owns triage... [archived: unreinforced 39d] ``` -Reasons include `unreinforced d`, `budget oldest-first`, and `legacy-unvalidated`. +Reasons include `unreinforced d`, `unreinforced p`, `budget oldest-first`, and `legacy-unvalidated`. Archiving is a move, not a removal, and recovery is `grep` plus copy back with no tooling. Each home keeps its own archive, the archive never cascades, and truncating a grown archive is a captain decision, not a mechanism. diff --git a/.agents/skills/stuck-crewmate-recovery/SKILL.md b/.agents/skills/stuck-crewmate-recovery/SKILL.md index b9b94b27d43..64d809c798d 100644 --- a/.agents/skills/stuck-crewmate-recovery/SKILL.md +++ b/.agents/skills/stuck-crewmate-recovery/SKILL.md @@ -43,7 +43,7 @@ If the worktree or ownership cannot be reconciled safely, leave all state intact Escalate in order: -1. Peek the pane. +1. Peek the pane, and check the task's steering inbox (`state/.inbox/`) for unhandled `*.msg` records - a stale wake naming an unread firstmate instruction means the worker never acknowledged a durable steer, and the record itself shows exactly what was intended. 2. If the crewmate is waiting on a question its brief already answers, answer in one line via `FM_HOME= bin/fm-send.sh` from an active firstmate session unless `FM_HOME` is already set to the active firstmate home. 3. If the crewmate is confused or looping, interrupt with `FM_HOME= bin/fm-control.sh interrupt`, then redirect with one corrective line through `fm-send`. 4. If the crewmate is genuinely wedged after redirection, relaunch it with `FM_HOME= bin/fm-control.sh relaunch --note ''`, which stops the agent, carries the brief plus that note into a replacement in the same local copy, and restores the prior record if the replacement cannot start. diff --git a/AGENTS.md b/AGENTS.md index c3327293c94..977dba22c18 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -72,6 +72,7 @@ config/backlog-backend backlog backend override; LOCAL, gitignored; absent or " config/backend runtime session-provider backend override for new tasks; LOCAL, gitignored; absent = falls through to runtime auto-detection (the runtime firstmate itself is executing inside), then tmux; tmux is the verified reference backend (docs/tmux-backend.md), while herdr, zellij, orca, and cmux are experimental spawn backends (docs/herdr-backend.md, docs/zellij-backend.md, docs/orca-backend.md, docs/cmux-backend.md) - herdr and cmux can also be selected by runtime auto-detection, zellij and orca never are (always explicit), and codex-app is not accepted; see docs/codex-app-backend.md; inherited by secondmate homes under the primary-authoritative contract in secondmate-provisioning config/calm Pi Calm presentation preference; LOCAL, gitignored, and not inherited; see docs/configuration.md "Pi Calm preference" config/startup-memory-budget primary-authoritative per-home startup-memory budget; LOCAL, gitignored, materialized as 7,500 estimated tokens by locked primary bootstrap and inherited into secondmate homes; see docs/configuration.md "Startup memory budget" +config/stow-pass-horizon optional presence flag opting this home in to /stow's default-off pass-count decay horizon; LOCAL, gitignored, and not inherited; see docs/configuration.md "Stow pass horizon" config/herdr-presentation-spaces optional "off" opt-out from, or "on" opt-in to, Herdr's default-on disposable single-task visual projection, which is unconfigured-default-on only at or above a Herdr version floor; LOCAL, gitignored; inherited by secondmate homes; see docs/herdr-backend.md "Presentation spaces" config/trace-context optional presence flag enabling default-off native W3C trace-context propagation to spawned agents; LOCAL, gitignored; inherited by secondmate homes; see docs/configuration.md "Trace context propagation" and docs/trace-context.md config/cmux-socket-password optional cmux control-socket password; LOCAL, gitignored; read fresh on every cmux CLI call and passed through without ever overriding an operator's own ambient CMUX_SOCKET_PASSWORD when absent (docs/cmux-backend.md "Setup") @@ -95,6 +96,7 @@ state/ runtime records and signals; gitignored .kimi-turnend-token firstmate-owned Kimi hook registry token for the task; removed by teardown .muse-session muse busy-source binding (sessions root plus task worktree) written by fm-spawn; removed by teardown .cursor-session cursor busy-source binding (projects root, task worktree, prior conversations) written by fm-spawn; removed by teardown + .inbox/ durable steering inbox: sequenced firstmate instruction records the worker acknowledges by moving them into its handled/ subdirectory; written by fm-send, re-rung and escalated by the watcher, removed by teardown (bin/fm-task-inbox-lib.sh) .meta task metadata; each producer script's header owns its exact fields and mutation contract, with docs/configuration.md routing operator-facing backend and trace-context details .herdr-presentation quarantinable attempt and restart-binding journal for Herdr's optional visual projection; never task or endpoint authority; see docs/herdr-backend.md "Presentation spaces" .check.sh authenticated slow poll; the watcher dispatches validated PR data and the byte-identified Relay shim through trusted repository scripts, runs registered custom checks from hash-validated private snapshots, and rejects every other state check without execution @@ -301,7 +303,8 @@ The spawn must resolve a genuine isolated task worktree distinct from the primar After spawning, confirm the worker is processing the brief, handle any trust dialog through `harness-adapters`, and record ship or scout work as under way. A persistent secondmate is recorded in the secondmate registry and runtime state, never as a backlog work item. -Steer a worker with short single-line messages through fail-closed `fm-send`; put long instructions in a file. +Steer a worker with ordinary text through fail-closed `fm-send`: the message becomes a durable record in the task's steering inbox (multi-line text is legal, local and remote alike) and the worker's terminal receives only a constant doorbell line, with the watcher re-ringing an unacknowledged local message and escalating a stuck one (`bin/fm-task-inbox-lib.sh`; `bin/fm-send.sh` owns the typed-plane carve-outs). +A remote secondmate steer rides the same durable-inbox model through the remote transport; after an unconfirmed delivery, only the exact `FM_PENDING_REPLY_EXISTING_CORR=` resend command printed by `fm-send` is safe because it preserves the request body for remote enqueue deduplication (`bin/fm-send.sh` header). When a steer answers an open keyed decision or blocker, pass `fm-send`'s `--resolve-key` so the answer itself closes that decision record at answer time, identically for local and remote workers (contract: `bin/fm-send.sh` header). `fm-send` is the data plane for text the worker should read; never use its key or text paths for interrupt, exit, or other lifecycle control, because routing-marked lifecycle text becomes chat the worker reasons about instead of executing. Drive a worker's lifecycle through `bin/fm-control.sh interrupt|exit|relaunch`, which owns the per-runtime mechanics, verifies each action, and never tears down or discards anything ([`docs/agent-control.md`](docs/agent-control.md)). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 98cc88a5f68..19aa158b093 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -45,7 +45,8 @@ See the [no-mistakes quick start](https://kunchenguid.github.io/no-mistakes/star - Helper scripts in `bin/` are plain bash. Each starts with a usage header comment; keep it accurate when you change behavior. Test scripts and helpers in `tests/` are plain bash too. - `bin/fm-lint.sh` must pass: it is the single owner of the lint definition (the shellcheck file set, config, pinned shellcheck version, and pinned actionlint workflow lint), and both CI and the no-mistakes pre-push gate run it, so local and CI can never diverge. + `bin/fm-lint.sh` must pass: it is the single owner of the lint definition (the shellcheck file set, config, pinned shellcheck version, and pinned actionlint workflow lint), and both CI and the no-mistakes pre-push gate run its no-argument full-analysis path. + Its header and `--help` output own the exact local lint modes and flags. A malformed `.github/workflows/*.yml`, including a self-broken `ci.yml`, fails that local lint path before merge because a broken workflow cannot report its own breakage. It pins one exact shellcheck version and one exact actionlint version and refuses to run under any other. Print the shellcheck pin with `bin/fm-lint.sh --required-version` and the actionlint pin with `bin/fm-lint-workflows.sh --required-version`. @@ -103,6 +104,8 @@ Family selection is the ordinary local path; `--all` is deliberate full regressi CI owns broad regression across required portable parallel shards, the portable serial lane's separate-runner shards, the Herdr lane, lint, invariants, the coverage guard, and stock macOS Bash compatibility in [`.github/workflows/ci.yml`](.github/workflows/ci.yml). Use `bin/fm-test-run.sh --list-lanes` for exact lane names and `--help` for `--jobs` rules and required gate-skip flags when reproducing a lane locally. Discover tests by listing `tests/*.test.sh`: each is a self-contained bash script named `.test.sh`, and its header comment describes what it covers, so pass one to `bin/fm-test-run.sh` to focus on a subject with canonical timing output. +A fixture may shorten a production timeout to keep a failure path prompt, but never below what the real work inside that window costs on a loaded machine: a fork, an exec, a lock acquisition, a beacon publication, or a first-poll check. +Where a case's assertion is not about the timeout itself, give that window headroom over the measured loaded cost, and bound the test's own waiting with iteration-counted poll loops, which stretch under load where a wall-clock budget does not. Tests that need a real optional backend or an explicit opt-in (real herdr/zellij/cmux smoke tests, the live Pi regression) skip themselves and print the tool or environment gate needed to enable them, so the portable suite remains safe on machines without those tools. The [Herdr backend guide](docs/herdr-backend.md#destructive-lab-safety) owns the lane's isolation boundary, while [runtime backend verification](docs/verification/runtime-backends.md#herdr) owns active empirical evidence; live harness credential tests remain opt-in. diff --git a/bin/fm-backlog-handoff.sh b/bin/fm-backlog-handoff.sh index 97bda75c331..fa729c9d1b6 100755 --- a/bin/fm-backlog-handoff.sh +++ b/bin/fm-backlog-handoff.sh @@ -50,7 +50,16 @@ # Remote routes use an outbox handoff: one atomic local tasks-axi mv removes the # selected set from the dispatchable backlog into data/handoff/.outbox.md, # then an idempotent confined transfer and fm-backlog-receive.sh deliver it. -# A present outbox is the whole recovery record. No two-phase journal exists. +# A present outbox remains the remote retry trigger until backlog receipt and +# receiver wake are both confirmed; a companion pending-reply correlation makes +# crash recovery reconcile an attempted or confirmed wake instead of blindly +# resending it. A prepared local wake is bound to the exact sorted +# requested-key batch; an unrelated handoff to that mate refuses until the +# original batch is retried, so it cannot discard wake intent for work that +# already moved. No two-phase journal exists. +# Every newly durable backlog delivery also sends one marked wake to the +# receiving endpoint. A missing endpoint or a live endpoint that rejects the +# wake makes the handoff fail with the delivered backlog intact. # Usage: fm-backlog-handoff.sh ... # fm-backlog-handoff.sh --resume-pending set -eu @@ -70,6 +79,10 @@ MAIN_BACKLOG="$DATA/backlog.md" . "$SCRIPT_DIR/fm-wake-lib.sh" # shellcheck source=bin/fm-public-followup-lib.sh . "$SCRIPT_DIR/fm-public-followup-lib.sh" +# shellcheck source=bin/fm-pending-reply-lib.sh +. "$SCRIPT_DIR/fm-pending-reply-lib.sh" + +RECEIVER_WAKE_MESSAGE='New routed work is in your backlog. Run bin/fm-session-start.sh now, then act on the routed task.' ACTIVE_HANDOFF_LOCK= ACTIVE_REGISTRY_LOCK= @@ -99,6 +112,7 @@ if [ "${1:-}" = --resume-pending ]; then else [ "$#" -ge 2 ] || { echo "usage: fm-backlog-handoff.sh ..." >&2; exit 1; } ID=$1 + case "$ID" in ''|*[!A-Za-z0-9._-]*) echo "error: unsafe secondmate id: $ID" >&2; exit 1 ;; esac shift fi @@ -300,12 +314,212 @@ warn_stale_public_commitments() { # ... return 0 } +# Wake a live receiver after its backlog has become durable. The marked message +# uses the normal endpoint route, so local and remote secondmates share the same +# verified submit and failure semantics. A seeded but not-yet-spawned home is a +# valid handoff destination, but its missing endpoint is reported rather than +# pretending the task was started. +receiver_wake_batch_id() { # ... + local digest + if command -v shasum >/dev/null 2>&1; then + digest=$(printf '%s\n' "$@" | LC_ALL=C sort | shasum -a 256 2>/dev/null | awk '{print $1}') + else + digest=$(printf '%s\n' "$@" | LC_ALL=C sort | sha256sum 2>/dev/null | awk '{print $1}') + fi + printf '%s' "$digest" | grep -Eq '^[a-f0-9]{64}$' || return 1 + printf '%s' "${digest:0:16}" +} + +receiver_wake_state_write() { # + local id=$1 value=$2 marker="$STATE/.backlog-handoff-$1.wake-pending" tmp + case "$id" in ''|*[!A-Za-z0-9._-]*) return 1 ;; esac + case "$value" in + pending|confirmed) ;; + prepared:*) printf '%s' "$value" | grep -Eq '^prepared:[a-f0-9]{16}:[a-f0-9]{16}$' || return 1 ;; + pending:*) printf '%s' "$value" | grep -Eq '^pending:[a-f0-9]{16}$' || return 1 ;; + confirmed:*) printf '%s' "$value" | grep -Eq '^confirmed:[a-f0-9]{16}$' || return 1 ;; + *) return 1 ;; + esac + tmp=$(umask 077; mktemp "$STATE/.backlog-handoff-wake.XXXXXX") || return 1 + if ! printf '%s\n' "$value" > "$tmp" || ! chmod 600 "$tmp" || ! mv -f -- "$tmp" "$marker"; then + rm -f -- "$tmp" + return 1 + fi +} + +receiver_wake_mark() { # [batch-id] + local id=$1 wake_phase=$2 batch=${3:-} marker="$STATE/.backlog-handoff-$1.wake-pending" value corr rec + local wake_state + case "$wake_phase" in prepared|pending) ;; *) return 1 ;; esac + if [ -e "$marker" ] || [ -L "$marker" ]; then + [ -f "$marker" ] && [ ! -L "$marker" ] || return 1 + value=$(cat "$marker" 2>/dev/null || true) + case "$value" in + prepared:*|pending:*) + corr=${value#*:} + corr=${corr%%:*} + rec=$(fm_pending_reply_path "$STATE" "$corr") + [ -f "$rec" ] && [ ! -L "$rec" ] \ + && [ "$(fm_pending_reply_get "$rec" task_id)" = "$id" ] + return $? + ;; + pending) ;; + *) return 1 ;; + esac + fi + corr=$(fm_pending_reply_create "$FM_HOME" "$STATE" "$id" "$RECEIVER_WAKE_MESSAGE") || return 1 + wake_state="$wake_phase:$corr" + if [ "$wake_phase" = prepared ]; then + printf '%s' "$batch" | grep -Eq '^[a-f0-9]{16}$' || return 1 + wake_state="$wake_state:$batch" + fi + if ! receiver_wake_state_write "$id" "$wake_state"; then + fm_pending_reply_discard_undelivered "$STATE" "$corr" || true + return 1 + fi +} + +receiver_wake_mark_pending() { # + receiver_wake_mark "$1" pending +} + +receiver_wake_mark_prepared() { # + receiver_wake_mark "$1" prepared "$2" +} + +receiver_wake_discard_prepared() { # + local id=$1 marker="$STATE/.backlog-handoff-$1.wake-pending" value corr + [ -f "$marker" ] && [ ! -L "$marker" ] || return 1 + value=$(cat "$marker" 2>/dev/null || true) + case "$value" in + prepared:*) + corr=${value#prepared:} + corr=${corr%%:*} + ;; + *) return 1 ;; + esac + fm_pending_reply_discard_undelivered "$STATE" "$corr" || return 1 + rm -f -- "$marker" +} + +receiver_wake_promote_prepared() { # + local id=$1 batch=$2 marker="$STATE/.backlog-handoff-$1.wake-pending" value corr + [ -f "$marker" ] && [ ! -L "$marker" ] || return 1 + value=$(cat "$marker" 2>/dev/null || true) + case "$value" in + prepared:*:"$batch") + corr=${value#prepared:} + corr=${corr%%:*} + ;; + pending:*) return 0 ;; + *) return 1 ;; + esac + receiver_wake_state_write "$id" "pending:$corr" +} + +receiver_wake_discard_pending() { # + local id=$1 marker="$STATE/.backlog-handoff-$1.wake-pending" value corr + [ -f "$marker" ] && [ ! -L "$marker" ] || return 1 + value=$(cat "$marker" 2>/dev/null || true) + case "$value" in + pending:*) + corr=${value#pending:} + fm_pending_reply_discard_undelivered "$STATE" "$corr" || return 1 + ;; + pending) ;; + *) return 1 ;; + esac + rm -f -- "$marker" +} + +receiver_wake_clear_confirmed() { # + local id=$1 marker="$STATE/.backlog-handoff-$1.wake-pending" value + [ -e "$marker" ] || [ -L "$marker" ] || return 0 + [ -f "$marker" ] && [ ! -L "$marker" ] || return 1 + value=$(cat "$marker" 2>/dev/null || true) + case "$value" in + pending|pending:*) return 0 ;; + confirmed|confirmed:*) rm -f -- "$marker" ;; + *) return 1 ;; + esac +} + +wake_secondmate_receiver() { # + local id=$1 corr=$2 meta="$STATE/$1.meta" out rc=0 + if [ ! -f "$meta" ] || [ -L "$meta" ]; then + printf 'error: handed off work to secondmate %s, but no live receiver endpoint is recorded; the destination backlog is durable and the receiver was not woken\n' "$id" >&2 + return 1 + fi + [ "$(grep '^kind=' "$meta" | cut -d= -f2-)" = secondmate ] || { + printf 'error: secondmate %s has non-secondmate endpoint metadata; backlog is durable but the receiver was not woken\n' "$id" >&2 + return 1 + } + out=$(FM_HOME="$FM_HOME" FM_STATE_OVERRIDE="$STATE" FM_ROOT_OVERRIDE="$FM_ROOT" \ + FM_PENDING_REPLY_EXISTING_CORR="$corr" \ + "$SCRIPT_DIR/fm-send.sh" "$id" "$RECEIVER_WAKE_MESSAGE" 2>&1) || rc=$? + if [ "$rc" -ne 0 ]; then + [ -z "$out" ] || printf '%s\n' "$out" >&2 + printf 'error: backlog delivery to secondmate %s succeeded, but its receiver wake failed; rerun this handoff to retry the wake\n' "$id" >&2 + return 1 + fi + [ -z "$out" ] || printf '%s\n' "$out" +} + +wake_pending_secondmate_receiver() { # [retain-confirmed] + local id=$1 retain=${2:-0} marker="$STATE/.backlog-handoff-$1.wake-pending" value corr rec delivered + [ -e "$marker" ] || [ -L "$marker" ] || return 0 + if [ ! -f "$marker" ] || [ -L "$marker" ]; then + printf 'error: receiver wake state for secondmate %s is unsafe or invalid\n' "$id" >&2 + return 1 + fi + value=$(cat "$marker" 2>/dev/null || true) + case "$value" in + confirmed|confirmed:*) return 0 ;; + prepared|prepared:*) + printf 'error: receiver wake for secondmate %s was prepared before its backlog became durable\n' "$id" >&2 + return 1 + ;; + pending) + receiver_wake_mark_pending "$id" || return 1 + value=$(cat "$marker" 2>/dev/null || true) + ;; + esac + case "$value" in pending:*) corr=${value#pending:} ;; *) + printf 'error: receiver wake state for secondmate %s is unsafe or invalid\n' "$id" >&2 + return 1 + ;; + esac + rec=$(fm_pending_reply_path "$STATE" "$corr") + [ -f "$rec" ] && [ ! -L "$rec" ] \ + && [ "$(fm_pending_reply_get "$rec" task_id)" = "$id" ] || return 1 + fm_pending_reply_reconcile_delivery "$STATE" "$corr" >/dev/null 2>&1 || true + delivered=$(fm_pending_reply_get "$rec" delivered_epoch) + if [ -z "$delivered" ]; then + fm_pending_reply_corr_reusable "$STATE" "$corr" "$id" || { + printf 'error: receiver wake delivery for secondmate %s is unresolved; refusing to resend correlation %s\n' "$id" "$corr" >&2 + return 1 + } + wake_secondmate_receiver "$id" "$corr" || return 1 + fi + if [ "$retain" = 1 ]; then + receiver_wake_state_write "$id" "confirmed:$corr" || { + printf 'error: receiver wake for secondmate %s was confirmed, but confirmed state could not be recorded\n' "$id" >&2 + return 1 + } + else + rm -f -- "$marker" || { + printf 'error: receiver wake for secondmate %s was confirmed, but pending state could not be cleared\n' "$id" >&2 + return 1 + } + fi +} + outbox_item_count() { # awk '/^- \[[ x]\] / { count++ } END { print count + 0 }' "$1" } remote_deliver_outbox() { # - local id=$1 outbox=$2 remote_rel receive_out snapshot bytes hash generation counter counter_tmp current + local id=$1 outbox=$2 remote_rel receive_out snapshot bytes hash generation counter counter_tmp current marker [ -f "$outbox" ] && [ ! -L "$outbox" ] || { echo "error: pending outbox is unavailable or unsafe: $outbox" >&2 return 1 @@ -348,8 +562,24 @@ remote_deliver_outbox() { # echo "error: handoff receipt by $id was unavailable or completion is unknown; outbox preserved at $outbox" >&2 return 1 fi + marker="$STATE/.backlog-handoff-$id.wake-pending" + case "$(cat "$marker" 2>/dev/null || true)" in + pending:*|confirmed|confirmed:*) ;; + *) receiver_wake_mark_pending "$id" || { + echo "error: remote backlog is durable at $id, but receiver wake state could not be recorded; outbox preserved at $outbox" >&2 + return 1 + } ;; + esac + if ! wake_pending_secondmate_receiver "$id" 1; then + echo "error: remote backlog is durable at $id; outbox preserved at $outbox for wake retry" >&2 + return 1 + fi rm -f -- "$outbox" || { - echo "error: remote receipt was confirmed but local outbox cleanup failed: $outbox" >&2 + echo "error: receiver wake was confirmed but local outbox cleanup failed: $outbox" >&2 + return 1 + } + rm -f -- "$marker" || { + echo "error: remote outbox cleanup succeeded but confirmed receiver wake state could not be cleared: $marker" >&2 return 1 } printf '%s\n' "$receive_out" @@ -388,6 +618,12 @@ remote_handoff() { # outbox="$DATA/handoff/$id.outbox.md" validate_backlog_file "main backlog" "$MAIN_BACKLOG" || return 1 validate_backlog_file "remote handoff outbox" "$outbox" || return 1 + if [ ! -e "$outbox" ] && [ ! -L "$outbox" ]; then + receiver_wake_clear_confirmed "$id" || { + echo "error: stale receiver wake state for secondmate $id could not be cleared" >&2 + return 1 + } + fi fm_tasks_axi_compatible || { echo "error: a compatible tasks-axi with atomic multi-ID mv support is required to stage remote handoffs; run bin/fm-bootstrap.sh for the required version" >&2 return 1 @@ -429,6 +665,18 @@ remote_handoff() { # return 1 done < <(backlog_key_noncanonical_body_lines "$MAIN_BACKLOG" "$key") done + # Do not append a fresh handoff to an older recovery batch. In particular, a + # confirmed wake can survive when outbox cleanup fails; if new work were + # staged into that outbox, the old confirmation would suppress the wake for + # the new work. Finish receipt, wake reconciliation, and cleanup for the old + # batch first. A failure leaves the fresh items dispatchable in main. + if [ "${#to_move[@]}" -gt 0 ] && [ -f "$outbox" ] \ + && [ "$(outbox_item_count "$outbox")" -gt 0 ]; then + remote_deliver_outbox "$id" "$outbox" || { + echo "error: previous remote handoff for secondmate $id could not be completed; nothing new was staged" >&2 + return 1 + } + fi seed_backlog_scaffold "$outbox" if [ "${#to_move[@]}" -gt 0 ]; then if ! mv_out=$(tasks-axi mv "${to_move[@]}" --file "$MAIN_BACKLOG" --to "$outbox" 2>&1); then @@ -502,7 +750,10 @@ if [ "$REMOTE" = 1 ]; then release_remote_locks exit "$rc" fi -release_remote_locks +ACTIVE_HANDOFF_LOCK="$STATE/.backlog-handoff-$ID.lock" +fm_lock_acquire_wait "$ACTIVE_HANDOFF_LOCK" +fm_lock_release "$ACTIVE_REGISTRY_LOCK" +ACTIVE_REGISTRY_LOCK= RAW_HOME=$(secondmate_home "$ID") || exit 1 [ -n "$RAW_HOME" ] || { echo "error: secondmate $ID has no home in $REG" >&2; exit 1; } @@ -556,8 +807,22 @@ if [ "$FAILED" -ne 0 ]; then exit 1 fi +REQUESTED_BATCH=$(receiver_wake_batch_id "$@") || { + echo "error: receiver wake batch identity could not be recorded; nothing was moved" >&2 + exit 1 +} + if [ "${#TO_MOVE[@]}" -eq 0 ]; then + WAKE_PENDING_MARKER="$STATE/.backlog-handoff-$ID.wake-pending" + case "$(cat "$WAKE_PENDING_MARKER" 2>/dev/null || true)" in + prepared:*:"$REQUESTED_BATCH") receiver_wake_promote_prepared "$ID" "$REQUESTED_BATCH" || exit 1 ;; + prepared:*) + echo "error: a prepared receiver wake for secondmate $ID belongs to a different routed batch; retry that original handoff before handling ${ALREADY[*]}" >&2 + exit 1 + ;; + esac echo "nothing to move: ${ALREADY[*]:-no keys} already present in $SUB_BACKLOG" + wake_pending_secondmate_receiver "$ID" || exit 1 exit 0 fi @@ -579,6 +844,27 @@ if ! fm_tasks_axi_compatible; then exit 1 fi +WAKE_PENDING_MARKER="$STATE/.backlog-handoff-$ID.wake-pending" +if [ -e "$WAKE_PENDING_MARKER" ] || [ -L "$WAKE_PENDING_MARKER" ]; then + case "$(cat "$WAKE_PENDING_MARKER" 2>/dev/null || true)" in + prepared:*:"$REQUESTED_BATCH") receiver_wake_discard_prepared "$ID" || exit 1 ;; + prepared:*) + echo "error: a prepared receiver wake for secondmate $ID belongs to a different routed batch; retry that original handoff before moving ${TO_MOVE[*]}" >&2 + exit 1 + ;; + *) + wake_pending_secondmate_receiver "$ID" || { + echo "error: previous receiver wake for secondmate $ID is unresolved; nothing new was moved" >&2 + exit 1 + } + ;; + esac +fi +receiver_wake_mark_prepared "$ID" "$REQUESTED_BATCH" || { + echo "error: receiver wake state for secondmate $ID could not be recorded; nothing was moved" >&2 + exit 1 +} + # Seed the destination with firstmate's standard three-section scaffold when it # does not exist yet, so the moved item lands under the right section. (Left to # create the file itself, tasks-axi mv writes its own `# Backlog` title format, @@ -599,6 +885,10 @@ if ! MV_OUT=$(tasks-axi mv "${TO_MOVE[@]}" --file "$MAIN_BACKLOG" --to "$SUB_BAC if [ "$SUB_CREATED" -eq 1 ]; then rm -f "$SUB_BACKLOG" fi + receiver_wake_discard_prepared "$ID" || { + echo "error: tasks-axi mv failed and receiver wake state could not be cleared" >&2 + exit 1 + } if [ -n "$MV_OUT" ]; then printf '%s\n' "$MV_OUT" >&2 fi @@ -608,6 +898,11 @@ fi echo "handed off ${#TO_MOVE[@]} item(s) to $ID: ${TO_MOVE[*]}" echo " into $SUB_BACKLOG" +receiver_wake_promote_prepared "$ID" "$REQUESTED_BATCH" || { + echo "error: handed off work to secondmate $ID, but durable receiver wake state could not be recorded" >&2 + exit 1 +} +wake_pending_secondmate_receiver "$ID" || exit 1 if [ "${#ALREADY[@]}" -gt 0 ]; then echo " already present (skipped): ${ALREADY[*]}" fi diff --git a/bin/fm-brief.sh b/bin/fm-brief.sh index ee2b76e2c9d..4897271c04d 100755 --- a/bin/fm-brief.sh +++ b/bin/fm-brief.sh @@ -51,6 +51,9 @@ # declared-external-wait verb (FM_CLASSIFY_PAUSED_VERB, default "paused") from # "blocked:": pause for a known external wait expected to clear on its own, # blocked when firstmate must act. +# Every scaffold also carries the steering-inbox receive-and-ack section: +# process state/.inbox/*.msg in order and acknowledge each by moving it to +# handled/ (record, doorbell, and ladder owned by bin/fm-task-inbox-lib.sh). # Ship tasks include a project-memory section so durable project-intrinsic # learnings can be committed to AGENTS.md through the project's delivery path; # it carries the AGENTS.md authoring bar (widely useful knowledge only, pointers @@ -179,6 +182,20 @@ shell_quote() { } STATUS_FILE=$(shell_quote "$STATE/$ID.status") +INBOX_DIR=$(shell_quote "$STATE/$ID.inbox") + +# The receive-and-ack half of the steering-inbox contract, included in every +# scaffold kind. The record format, doorbell line, and re-ring ladder are +# owned by bin/fm-task-inbox-lib.sh; the doorbell itself is self-describing, +# so this section is reinforcement for the natural-checkpoint habit, not the +# only carrier of the instruction. +IFS= read -r -d '' INBOX_SECTION <`; Linux (GNU) +# stat uses `-c `. Do NOT collapse this into `stat -f ... || stat -c +# ...`: on GNU `-f` is *filesystem* stat, so it reads the format string as +# a path, reports that on stderr, prints a partial filesystem dump (" File: +# ...") on stdout, and still exits 0 - the fallback never runs and the caller +# gets a non-numeric token. Detect the platform once and pick the right form, +# exactly as bin/fm-watch.sh does. +if [ "$(uname)" = Darwin ]; then + lock_mtime() { stat -f %m "$1" 2>/dev/null; } +else + lock_mtime() { stat -c %Y "$1" 2>/dev/null; } +fi + # Serialize writers. The lock protects seq advancement and the sidecar/record # pair; a holder that died mid-write is broken after FM_BUSY_LOCK_STALE_SECS. lock_acquire() { @@ -103,7 +116,11 @@ lock_acquire() { tries=$((tries + 1)) if [ "$tries" -ge 40 ]; then now=$(date +%s) - mtime=$(stat -f %m "$LOCK" 2>/dev/null || stat -c %Y "$LOCK" 2>/dev/null || echo "$now") + mtime=$(lock_mtime "$LOCK" || true) + # Anything unreadable or non-numeric reads as "just created", so an + # unforeseen stat surprise degrades to a lock-timeout refusal instead of + # aborting the writer - and its caller, fm-teardown.sh - under `set -u`. + case "$mtime" in ''|*[!0-9]*) mtime=$now ;; esac age=$((now - mtime)) if [ "$age" -ge "${FM_BUSY_LOCK_STALE_SECS:-5}" ]; then rmdir "$LOCK" 2>/dev/null || rm -rf "$LOCK" 2>/dev/null || true diff --git a/bin/fm-control.sh b/bin/fm-control.sh index b714df73da5..b73fde56b67 100755 --- a/bin/fm-control.sh +++ b/bin/fm-control.sh @@ -760,6 +760,10 @@ record_note() { echo "This task was relaunched. Continue from here; the local copy and every" echo "uncommitted change are exactly as the previous worker left them." echo + echo "First, check your instruction inbox: list $STATE/$ID.inbox/*.msg, act on" + echo "each message in numeric order, then mv each handled file into" + echo "$STATE/$ID.inbox/handled/. A steer sent before the relaunch survives there." + echo printf '%s\n' "$NOTE" } >> "$RELAUNCH_BRIEF" \ || die "could not append the progress note to task $ID's instructions" diff --git a/bin/fm-fleet-sync.sh b/bin/fm-fleet-sync.sh index d5c951e1a74..dd00be86baa 100755 --- a/bin/fm-fleet-sync.sh +++ b/bin/fm-fleet-sync.sh @@ -13,6 +13,11 @@ # stashed, or discarded. # Still skips (benignly) local-only/no-origin projects, missing remotes/branches, # and fetch failures. +# A candidate under projects/ must be the root of its own work tree: git discovery +# walks up, so a plain nested directory would otherwise resolve to the enclosing +# repository (the firstmate checkout) and be synced under that directory's label. +# Anything else is reported as "skipped: not a clone root" naming the repository +# that would have been touched. # Pruning never deletes the checked-out branch or a branch that still has a # worktree, so it cannot discard unlanded work; set FM_FLEET_PRUNE=0 to disable it. # When the fetch fails on an orphaned .git/packed-refs.lock (left by a ref rewrite @@ -300,10 +305,25 @@ sync_project() { echo "$label: skipped: not a directory" return 0 fi - if ! git -C "$PROJ" rev-parse --is-inside-work-tree >/dev/null 2>&1; then + # Git repository discovery walks UP from $PROJ, so a plain directory merely + # nested inside a repository - a worktree container left under projects/, say - + # resolves to the ENCLOSING repository, which in a firstmate home is the + # firstmate checkout itself. Every later `git -C "$PROJ"` would then read, prune + # and fast-forward that repository under this project's label, turning a routine + # refresh into an unrequested self-update reported as a project sync. Require + # $PROJ to be the root of its own work tree before any other git command runs. + proj_top=$(git -C "$PROJ" rev-parse --show-toplevel 2>/dev/null) || proj_top="" + if [ -z "$proj_top" ]; then echo "$label: skipped: not a git repo" return 0 fi + # Both sides are physical paths (git resolves --show-toplevel through symlinks), + # so a symlinked clone dir still compares equal to its own root. + proj_abs=$(cd "$PROJ" && pwd -P) || proj_abs="" + if [ "$proj_top" != "$proj_abs" ]; then + echo "$label: skipped: not a clone root (git would act on $proj_top)" + return 0 + fi mode_line=$("$FM_ROOT/bin/fm-project-mode.sh" "$label" 2>/dev/null || echo "no-mistakes off") mode=${mode_line%% *} if [ "$mode" = "local-only" ]; then diff --git a/bin/fm-inbox.sh b/bin/fm-inbox.sh index 3f967fd80f2..f314a12f7a1 100755 --- a/bin/fm-inbox.sh +++ b/bin/fm-inbox.sh @@ -168,10 +168,12 @@ queue_note() { [ -n "${body//[[:space:]]/}" ] || die "refusing to queue an empty note" mkdir -p "$INBOX" - local tmp id summary + local tmp id summary staging_name tmp=$(mktemp "$INBOX/.staging-XXXXXX") + staging_name=$(basename "$tmp") + id="$(date +%s)-${staging_name#.staging-}" { - printf 'id=PENDING\n' + printf 'id=%s\n' "$id" printf 'at=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" printf 'source=%s\n' "$source" [ -z "$extra" ] || printf '%s\n' "$extra" @@ -179,9 +181,7 @@ queue_note() { printf '%s\n' "$body" } >"$tmp" - id="$(date +%s)-$(basename "$tmp" | sed 's/^\.staging-//')" - # Rewrite the id line now that we know it, then publish atomically. - sed -i "s/^id=PENDING$/id=$id/" "$tmp" + # Publish the completed note atomically. mv "$tmp" "$INBOX/$id.note" # One-line summary for the wake payload; the full body stays in the file. diff --git a/bin/fm-lint.sh b/bin/fm-lint.sh index 53a3f0cff89..3eb5b53609d 100755 --- a/bin/fm-lint.sh +++ b/bin/fm-lint.sh @@ -5,6 +5,9 @@ # ambient configuration disabled, and one exact ShellCheck version. CI and # no-mistakes both invoke this script with no arguments, so the rule set, # version, bounded execution, and diagnostics ordering cannot drift. +# The explicit --fast mode is local-only and disables ShellCheck's extended +# dataflow analysis while preserving ordinary shell lint checks. CI and +# no-mistakes keep the full-analysis no-argument default. # Tests stop source analysis at imported production modules because every # production shell is already a canonical, source-aware root of this same run. # The default (no explicit-path) path also runs bin/fm-lint-workflows.sh so a @@ -34,6 +37,7 @@ # # Usage: # fm-lint.sh lint the context-selected file set (see above) +# fm-lint.sh --fast [path]... local lint with extended analysis disabled # fm-lint.sh ... lint explicit roots with the same config # fm-lint.sh --jobs <1|2> [path]... override bounded worker count # fm-lint.sh --telemetry ... write a quiet metrics snapshot @@ -59,7 +63,7 @@ fm_lint_worker_stop() { fm_lint_worker() { # local manifest=$1 output_dir=$2 shard_index=$3 tab index path output rc=0 - local -a roots + local -a roots shellcheck_args roots=() tab=$(printf '\t') while IFS="$tab" read -r index path || [ -n "${index:-}${path:-}" ]; do @@ -71,7 +75,11 @@ fm_lint_worker() { # trap 'fm_lint_worker_stop; exit 129' HUP trap 'fm_lint_worker_stop; exit 130' INT trap 'fm_lint_worker_stop; exit 143' TERM - "$FM_LINT_SHELLCHECK" --norc --external-sources -- "${roots[@]}" > "$output.out" 2>&1 & + shellcheck_args=(--norc --external-sources) + if [ "${FM_LINT_INTERNAL_FAST:-0}" -eq 1 ]; then + shellcheck_args+=(--extended-analysis=false) + fi + "$FM_LINT_SHELLCHECK" "${shellcheck_args[@]}" -- "${roots[@]}" > "$output.out" 2>&1 & FM_LINT_WORKER_SHELLCHECK_PID=$! wait "$FM_LINT_WORKER_SHELLCHECK_PID" || rc=$? FM_LINT_WORKER_SHELLCHECK_PID= @@ -100,7 +108,11 @@ if [ "${1:-}" = "--required-version" ]; then fi fm_lint_usage() { - sed -n '2,42{s/^# \{0,1\}//;p;}' "$SELF" + awk ' + NR == 1 { next } + /^#/ { sub(/^# ?/, ""); print; next } + { exit } + ' "$SELF" } # Default no-args lint also validates GitHub workflows. Explicit paths stay a @@ -112,6 +124,8 @@ fm_lint_run_workflows() { JOBS=${FM_LINT_JOBS:-2} TELEMETRY=${FM_LINT_TELEMETRY:-} +FAST=0 +ANALYSIS_MODE=full LIST_FILES=0 while [ "$#" -gt 0 ]; do case "$1" in @@ -133,6 +147,11 @@ while [ "$#" -gt 0 ]; do TELEMETRY=${1#*=} shift ;; + --fast) + FAST=1 + ANALYSIS_MODE=fast + shift + ;; --list-files) LIST_FILES=1 shift @@ -154,6 +173,11 @@ case "$JOBS" in *) printf 'fm-lint.sh: jobs must be 1 or 2, got %s.\n' "$JOBS" >&2; exit 2 ;; esac +if [ "$FAST" -eq 1 ] && { [ "${GITHUB_ACTIONS:-}" = true ] || [ "${CI:-}" = true ]; }; then + printf 'fm-lint.sh: --fast is local-only; CI uses full ShellCheck analysis.\n' >&2 + exit 2 +fi + # fm_lint_changed_base_ref prints the ref to diff the working branch against: # the local origin/main tracking ref when present, else local main. Returns # nonzero when neither is resolvable, which the caller treats as "no @@ -247,6 +271,11 @@ if [ "$resolved" != "$REQUIRED_SHELLCHECK" ]; then "$REQUIRED_SHELLCHECK" "$resolved" "$REQUIRED_SHELLCHECK" >&2 exit 1 fi +if [ "$FAST" -eq 1 ]; then + printf 'fm-lint.sh: fast local mode; ShellCheck extended analysis disabled\n' >&2 +else + printf 'fm-lint.sh: full ShellCheck extended analysis enabled\n' >&2 +fi if [ "$CHANGED_MODE" -eq 1 ] && [ "$ROOT_COUNT" -eq 0 ]; then printf 'fm-lint.sh: no changed lint targets\n' @@ -379,18 +408,18 @@ fm_lint_run_worker() { # if [ "$(uname)" = Darwin ]; then exec "$PERL_BIN" -e 'setpgrp(0, 0) or die "setpgrp: $!"; exec @ARGV or die "exec: $!"' \ /usr/bin/time -lp -o "$timing" \ - env FM_LINT_INTERNAL=1 FM_LINT_SHELLCHECK="$SHELLCHECK_BIN" \ + env FM_LINT_INTERNAL=1 FM_LINT_INTERNAL_FAST="$FAST" FM_LINT_SHELLCHECK="$SHELLCHECK_BIN" \ "${BASH:-bash}" "$SELF" --internal-worker "$manifest" "$OUTPUT_DIR" "$worker_index" else exec "$PERL_BIN" -e 'setpgrp(0, 0) or die "setpgrp: $!"; exec @ARGV or die "exec: $!"' \ /usr/bin/time -f 'wall_seconds=%e\nuser_seconds=%U\nsystem_seconds=%S\nmax_rss_kib=%M' -o "$timing" \ - env FM_LINT_INTERNAL=1 FM_LINT_SHELLCHECK="$SHELLCHECK_BIN" \ + env FM_LINT_INTERNAL=1 FM_LINT_INTERNAL_FAST="$FAST" FM_LINT_SHELLCHECK="$SHELLCHECK_BIN" \ "${BASH:-bash}" "$SELF" --internal-worker "$manifest" "$OUTPUT_DIR" "$worker_index" fi else [ -z "$TELEMETRY" ] || printf 'timing_unavailable=1\n' > "$timing" exec "$PERL_BIN" -e 'setpgrp(0, 0) or die "setpgrp: $!"; exec @ARGV or die "exec: $!"' \ - env FM_LINT_INTERNAL=1 FM_LINT_SHELLCHECK="$SHELLCHECK_BIN" \ + env FM_LINT_INTERNAL=1 FM_LINT_INTERNAL_FAST="$FAST" FM_LINT_SHELLCHECK="$SHELLCHECK_BIN" \ "${BASH:-bash}" "$SELF" --internal-worker "$manifest" "$OUTPUT_DIR" "$worker_index" fi } @@ -521,6 +550,7 @@ EOF printf 'git_head\t%s\n' "$git_head" printf 'content_cksum\t%s\n' "$content_cksum" printf 'shellcheck_version\t%s\n' "$resolved" + printf 'analysis_mode\t%s\n' "$ANALYSIS_MODE" printf 'jobs\t%s\n' "$JOBS" printf 'root_count\t%s\n' "$ROOT_COUNT" printf 'direct_lines\t%s\n' "$direct_lines" diff --git a/bin/fm-pending-reply-lib.sh b/bin/fm-pending-reply-lib.sh index 5453585d0e2..8db770288a7 100755 --- a/bin/fm-pending-reply-lib.sh +++ b/bin/fm-pending-reply-lib.sh @@ -178,7 +178,7 @@ fm_pending_reply_get() { # } fm_pending_reply_corr_reusable() { # - local state=$1 corr=$2 task_id=$3 rec phase + local state=$1 corr=$2 task_id=$3 rec phase delivered printf '%s' "$corr" | grep -Eq '^[A-Fa-f0-9]{16}$' || return 1 rec=$(fm_pending_reply_path "$state" "$corr") [ -f "$rec" ] || return 1 @@ -186,6 +186,11 @@ fm_pending_reply_corr_reusable() { # phase=$(fm_pending_reply_get "$rec" phase) case "$phase" in awaiting_report|recovery_sending|recovery_sent) return 0 ;; + delivery_unknown) + delivered=$(fm_pending_reply_get "$rec" delivered_epoch) + [ -z "$delivered" ] + return $? + ;; esac return 1 } @@ -344,6 +349,18 @@ fm_pending_reply_prepare_delivery() { # } fm_pending_reply_confirm_delivery() { # + local state=$1 corr=$2 lock rc=0 + local STATE FM_WAKE_QUEUE FM_WAKE_QUEUE_LOCK + STATE=$state + lock="$state/.pending-reply-$corr.lock" + . "$_FM_PENDING_REPLY_LIB_DIR/fm-wake-lib.sh" + fm_lock_acquire_wait "$lock" || return 1 + _fm_pending_reply_confirm_delivery_locked "$@" || rc=$? + fm_lock_release "$lock" + return "$rc" +} + +_fm_pending_reply_confirm_delivery_locked() { # local state=$1 corr=$2 now marker marker=$(fm_pending_reply_delivery_confirmation_path "$state" "$corr") if ! fm_pending_reply_prepare_delivery "$state" "$corr"; then @@ -372,7 +389,7 @@ fm_pending_reply_mark_delivery_unknown() { # fm_pending_reply_set "$rec" phase delivery_unknown } -fm_pending_reply_reconcile_delivery() { # +_fm_pending_reply_reconcile_delivery_locked() { # local state=$1 corr=$2 rec delivered marker entry delivery_state value epoch local grace now age phase rec=$(fm_pending_reply_path "$state" "$corr") @@ -412,6 +429,68 @@ fm_pending_reply_reconcile_delivery() { # return 1 } +fm_pending_reply_reconcile_delivery() { # + local state=$1 corr=$2 lock rc=0 + local STATE FM_WAKE_QUEUE FM_WAKE_QUEUE_LOCK + STATE=$state + lock="$state/.pending-reply-$corr.lock" + . "$_FM_PENDING_REPLY_LIB_DIR/fm-wake-lib.sh" + fm_lock_acquire_wait "$lock" || return 1 + _fm_pending_reply_reconcile_delivery_locked "$@" || rc=$? + fm_lock_release "$lock" + return "$rc" +} + +fm_pending_reply_delivery_attempt_unresolved() { # + local state=$1 corr=$2 rec delivered marker entry + rec=$(fm_pending_reply_path "$state" "$corr") + [ -f "$rec" ] && [ ! -L "$rec" ] || return 1 + delivered=$(fm_pending_reply_get "$rec" delivered_epoch) + [ -z "$delivered" ] || return 1 + marker=$(fm_pending_reply_delivery_confirmation_path "$state" "$corr") + [ -f "$marker" ] && [ ! -L "$marker" ] || return 1 + entry=$(cat "$marker" 2>/dev/null || true) + case "$entry" in attempted=*) return 0 ;; esac + return 1 +} + +# A definitive backend rejection makes the existing correlation retryable again. +# Reconciliation may have aged the same attempted sidecar to delivery_unknown +# while the backend call was in flight, so both undelivered phases converge here +# under the per-correlation lock; a confirmed delivery can never be reset. +fm_pending_reply_reset_known_undelivered() { # + local state=$1 corr=$2 lock rc=0 + local STATE FM_WAKE_QUEUE FM_WAKE_QUEUE_LOCK + STATE=$state + lock="$state/.pending-reply-$corr.lock" + . "$_FM_PENDING_REPLY_LIB_DIR/fm-wake-lib.sh" + fm_lock_acquire_wait "$lock" || return 1 + _fm_pending_reply_reset_known_undelivered_locked "$@" || rc=$? + fm_lock_release "$lock" + return "$rc" +} + +_fm_pending_reply_reset_known_undelivered_locked() { # + local state=$1 corr=$2 rec delivered phase marker entry + rec=$(fm_pending_reply_path "$state" "$corr") + [ -f "$rec" ] && [ ! -L "$rec" ] || return 1 + delivered=$(fm_pending_reply_get "$rec" delivered_epoch) + [ -z "$delivered" ] || return 1 + phase=$(fm_pending_reply_get "$rec" phase) + case "$phase" in awaiting_report|delivery_unknown) ;; *) return 1 ;; esac + marker=$(fm_pending_reply_delivery_confirmation_path "$state" "$corr") + [ -e "$marker" ] || [ -L "$marker" ] || { + [ "$phase" = awaiting_report ] + return $? + } + [ -f "$marker" ] && [ ! -L "$marker" ] || return 1 + entry=$(cat "$marker" 2>/dev/null || true) + case "$entry" in attempted=*) ;; *) return 1 ;; esac + [ "$phase" = awaiting_report ] \ + || fm_pending_reply_set "$rec" phase awaiting_report || return 1 + rm -f -- "$marker" +} + # Drop an undelivered expectation after a failed send so transport failure does # not masquerade as a missed report later. fm_pending_reply_discard_undelivered() { # @@ -1049,7 +1128,7 @@ _fm_pending_reply_maybe_escalate_locked() { # [ -f "$rec" ] || return 1 phase=$(fm_pending_reply_get "$rec" phase) if [ "$phase" = delivery_unknown ]; then - fm_pending_reply_reconcile_delivery "$state" "$corr" || true + _fm_pending_reply_reconcile_delivery_locked "$state" "$corr" || true phase=$(fm_pending_reply_get "$rec" phase) [ "$phase" = delivery_unknown ] || return 0 fi diff --git a/bin/fm-procevent-atelier.sh b/bin/fm-procevent-atelier.sh index 034577023f7..66b7813c9b5 100755 --- a/bin/fm-procevent-atelier.sh +++ b/bin/fm-procevent-atelier.sh @@ -8,9 +8,14 @@ # fm-procevent-atelier.sh answers # fm-procevent-atelier.sh source-id # fm-procevent-atelier.sh retire +# fm-procevent-atelier.sh poll # # classify Print the lifecycle state a handler should act on: feedback, ended, # waiting, missing, or unknown. +# poll The registered listener command `arm` publishes, not a command to +# run in a conversational turn. It runs the published blocking poll +# and prints its response verbatim, absorbing only the one exact +# transient interruption described below. # terminal Exit 0 when the captured result means this Atelier source will never # produce another result, so the runner may retire it; any other exit # keeps it armed. This is the generic adapter contract bin/fm-procevent.sh @@ -41,6 +46,23 @@ # server-side events. It adds no periodic discovery, no timer fallback, and no # dependency on any unreleased capability. # +# BOUNDED QUIET RETRY, owned here and nowhere else. A live listener can be cut +# short by the server with exactly this two-line response while the session's +# marks remain available: +# +# error: Atelier Editor poll response was interrupted +# code: SERVER_ERROR +# +# That is an internal retry, not news, so registering the raw poll made the +# generic runner capture it and wake the whole fleet. `poll` therefore re-runs +# the published poll up to POLL_RETRY_LIMIT times for that exact response, with +# POLL_RETRY_DELAY_DEFAULT seconds between attempts. The match is exact and +# deliberately narrow: real feedback, ended and missing sessions, any other +# SERVER_ERROR, and the same interruption still standing after the bound is +# spent are all printed straight through and captured normally. The retry is an +# Atelier fact, so the generic runner in bin/fm-procevent.sh stays +# adapter-agnostic and learns nothing about it. +# # LOSS LIMITATION, stated plainly. The published poll destructively clears # feedback before returning it. A result lost after that clearing and before the # runner reads the process output is unrecoverable, and no Firstmate wrapper can @@ -61,7 +83,7 @@ FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" . "$SCRIPT_DIR/fm-procevent-lib.sh" die() { printf 'error: %s\n' "$1" >&2; exit 1; } -usage() { sed -n '2,49p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 2; } +usage() { sed -n '2,71p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 2; } # Canonical identity is physical, not the path string: Atelier itself keys a # session on the realpath of the artifact, so two names for one file are one @@ -85,11 +107,16 @@ cmd_arm() { [ -n "$artifact" ] || usage [ "$#" -eq 1 ] || usage command -v atelier-axi >/dev/null 2>&1 || die "atelier-axi is not installed" + poll_retry_delay >/dev/null id=$(cmd_source_id "$artifact") || exit 1 real=$(perl -MCwd=realpath -e '$p = realpath($ARGV[0]); defined($p) or exit 1; print "$p\n"' "$artifact" 2>/dev/null) \ || die "cannot resolve the artifact path: $artifact" - # The plain blocking form: no --timeout-ms, so completion is a server event. - "$SCRIPT_DIR/fm-procevent.sh" register atelier "$id" -- atelier-axi poll "$real" || exit 1 + # This adapter's own listener command, which runs the plain blocking form with + # no --timeout-ms so completion is a server event, and absorbs only the exact + # transient interruption. Registering raw poll output is what let that + # interruption reach the runner as a captured result. + "$SCRIPT_DIR/fm-procevent.sh" register atelier "$id" \ + -- "$SCRIPT_DIR/fm-procevent-atelier.sh" poll "$real" || exit 1 printf 'armed: %s\n' "$id" printf 'artifact: %s\n' "$real" } @@ -101,6 +128,124 @@ cmd_retire() { "$SCRIPT_DIR/fm-procevent.sh" retire "$id" } +# The bounded quiet retry described in the header. The bound is a constant +# because it is a property of the transient response, not an operator choice; +# only the delay takes an override, so a test can exercise the real bound +# without waiting it out. +POLL_RETRY_LIMIT=12 +POLL_RETRY_DELAY_DEFAULT=5 +POLL_RETRY_DELAY_MAX=60 + +# Exit 0 only for the exact two-line interruption, and nothing else. The whole +# response must be those two lines with those exact bytes: whitespace variants, +# a longer response that merely opens with them, and any other SERVER_ERROR are +# genuine errors this adapter must never swallow. +poll_response_filter() { # + perl -e ' + use strict; + use warnings; + my ($stage) = @ARGV; + my $expected = "error: Atelier Editor poll response was interrupted\ncode: SERVER_ERROR\n"; + open my $staged, ">", $stage or exit 2; + binmode STDIN; + binmode STDOUT; + binmode $staged; + my ($candidate, $streaming) = ("", 0); + sub write_all { + my ($handle, $bytes) = @_; + my $offset = 0; + while ($offset < length $bytes) { + my $written = syswrite $handle, $bytes, length($bytes) - $offset, $offset; + exit 2 unless defined $written; + $offset += $written; + } + } + while (1) { + my $count = sysread STDIN, my $chunk, 65536; + exit 2 unless defined $count; + last if $count == 0; + if ($streaming) { + write_all(*STDOUT, $chunk); + next; + } + my $room = length($expected) + 1 - length($candidate); + my $take = length($chunk) < $room ? length($chunk) : $room; + my $prefix = substr($chunk, 0, $take); + $candidate .= $prefix; + write_all($staged, $prefix); + my $matches_prefix = length($candidate) <= length($expected) + && substr($expected, 0, length($candidate)) eq $candidate; + if (!$matches_prefix) { + write_all(*STDOUT, $candidate); + write_all(*STDOUT, substr($chunk, $take)); + $streaming = 1; + } + } + exit 10 if !$streaming && $candidate eq $expected; + write_all(*STDOUT, $candidate) unless $streaming; + ' "$1" +} + +# Seconds between retries. FM_ATELIER_POLL_RETRY_DELAY is a bounded test +# override; a malformed or out-of-range value is refused rather than quietly +# rounded, because silently changing a retry cadence is how a bound stops +# meaning anything. +poll_retry_delay() { + local delay=${FM_ATELIER_POLL_RETRY_DELAY-} + if [ -z "$delay" ]; then + printf '%s\n' "$POLL_RETRY_DELAY_DEFAULT" + return 0 + fi + case "$delay" in + *[!0-9]*) die "FM_ATELIER_POLL_RETRY_DELAY must be whole seconds from 0 to $POLL_RETRY_DELAY_MAX: $delay" ;; + esac + [ "$delay" -le "$POLL_RETRY_DELAY_MAX" ] \ + || die "FM_ATELIER_POLL_RETRY_DELAY must be whole seconds from 0 to $POLL_RETRY_DELAY_MAX: $delay" + printf '%s\n' "$delay" +} + +cmd_poll() { + local artifact=${1-} delay attempt=0 response cleanup_command rc filter_rc + local pipeline_status + [ -n "$artifact" ] || usage + [ "$#" -eq 1 ] || usage + command -v atelier-axi >/dev/null 2>&1 || die "atelier-axi is not installed" + delay=$(poll_retry_delay) || exit 1 + response=$(mktemp "${TMPDIR:-/tmp}/fm-atelier-poll.XXXXXX") || die "cannot stage the poll response" + printf -v cleanup_command 'rm -f -- %q' "$response" + # shellcheck disable=SC2064 # $cleanup_command must expand now, while the staged path is still set. + trap "$cleanup_command" EXIT + # Retirement stops this listener by signalling its process group, and bash runs + # no EXIT trap for an uncaught signal, so each one cleans up the staged + # response and then re-raises itself with the default disposition, leaving the + # process dying exactly as the runner expects. + local signal + for signal in INT TERM HUP; do + # shellcheck disable=SC2064 # Same reason: expand now, while both are set. + trap "$cleanup_command; trap - $signal; kill -$signal $$" "$signal" + done + while :; do + atelier-axi poll "$artifact" | poll_response_filter "$response" + pipeline_status=("${PIPESTATUS[@]}") + rc=${pipeline_status[0]} + filter_rc=${pipeline_status[1]} + case "$filter_rc" in + 0) break ;; + 10) + if [ "$attempt" -lt "$POLL_RETRY_LIMIT" ]; then + attempt=$((attempt + 1)) + sleep "$delay" + else + cat -- "$response" + break + fi + ;; + *) die "cannot classify the poll response" ;; + esac + done + return "$rc" +} + # Read one field of the response's leading `session:` block. Those fields are # INDENTED, so each is read as the first indented match inside that block rather # than an anchored whole-line match; anchoring on "^status:" silently never @@ -244,6 +389,7 @@ cmd_answers() { case "${1-}" in arm) shift; cmd_arm "$@" ;; retire) shift; cmd_retire "$@" ;; + poll) shift; cmd_poll "$@" ;; source-id) shift; cmd_source_id "$@" ;; classify) shift; cmd_classify "$@" ;; terminal) shift; cmd_terminal "$@" ;; diff --git a/bin/fm-remote-job-lib.sh b/bin/fm-remote-job-lib.sh index 73bffa54c70..25d7bb73b40 100755 --- a/bin/fm-remote-job-lib.sh +++ b/bin/fm-remote-job-lib.sh @@ -30,7 +30,10 @@ # PATH, HOME, FM_HOME, FM_ROOT_OVERRIDE, and FM_REMOTE_JOB_ACTIVE=1. The PATH # is intentionally filesystem-discovered rather than login-shell-derived: # ~/.local/bin; nvm, asdf, and mise shims/install bins; Nix; Homebrew; and the -# system tail. No shell startup files are evaluated. +# system tail. No shell startup files are evaluated. Each discovered set is +# appended in the shell's own sorted pathname-expansion order, so which install +# of a multi-version tool wins is fixed by this composition rather than by the +# order the filesystem happens to return. # # On macOS the worker is Firstmate's Aqua LaunchAgent # dev.firstmate.remote-job at ~/Library/LaunchAgents/dev.firstmate.remote-job.plist @@ -130,11 +133,18 @@ fm_remote_job_path_append_resolved_dir() { # fm_remote_job_path_append "$physical" } -fm_remote_job_append_glob_dirs() { # - local pattern=$1 directory - while IFS= read -r directory; do +# Callers pass an already-expanded glob rather than the pattern, because only +# the shell's own pathname expansion sorts its matches: bash sorts +# glob_filename's result in pathexp.c, while `compgen -G` reaches the same +# glob_filename through pcomplete.c, which does not sort. On bash 3.2 (macOS +# /bin/bash) that handed back raw readdir order, so which install of a +# multi-version tool a remote job resolved depended on the filesystem instead +# of on this composition. +fm_remote_job_append_dirs() { # + local directory + for directory in "$@"; do fm_remote_job_path_append_if_dir "$directory" - done < <(compgen -G "$pattern" || true) + done } fm_remote_job_nvm_default_selector() { # @@ -217,11 +227,11 @@ fm_remote_job_compose_operator_path() { # nvm_bin=$(fm_remote_job_nvm_selected_bin "$account_home" 2>/dev/null || true) [ -z "$nvm_bin" ] || fm_remote_job_path_append "$nvm_bin" fm_remote_job_path_append_if_dir "$account_home/.asdf/shims" - fm_remote_job_append_glob_dirs "$account_home/.asdf/installs/*/*/bin" + fm_remote_job_append_dirs "$account_home"/.asdf/installs/*/*/bin fm_remote_job_path_append_if_dir "$account_home/.local/share/mise/shims" fm_remote_job_path_append_if_dir "$account_home/.mise/shims" - fm_remote_job_append_glob_dirs "$account_home/.local/share/mise/installs/*/*/bin" - fm_remote_job_append_glob_dirs "$account_home/.mise/installs/*/*/bin" + fm_remote_job_append_dirs "$account_home"/.local/share/mise/installs/*/*/bin + fm_remote_job_append_dirs "$account_home"/.mise/installs/*/*/bin fm_remote_job_path_append_resolved_dir "$account_home/.nix-profile/bin" account_user=$(id -un 2>/dev/null || true) if [ -n "$account_user" ]; then diff --git a/bin/fm-remote-secondmate-control.sh b/bin/fm-remote-secondmate-control.sh index aa17c952860..fc8cc5ec72e 100755 --- a/bin/fm-remote-secondmate-control.sh +++ b/bin/fm-remote-secondmate-control.sh @@ -46,6 +46,8 @@ REMOTE_HERDR_SESSION=fm-remote . "$SCRIPT_DIR/fm-backend.sh" # shellcheck source=bin/fm-pending-reply-lib.sh . "$SCRIPT_DIR/fm-pending-reply-lib.sh" +# shellcheck source=bin/fm-task-inbox-lib.sh +. "$SCRIPT_DIR/fm-task-inbox-lib.sh" die() { printf 'error: %s\n' "$1" >&2; exit 1; } usage() { sed -n '2,23p' "$0" | sed 's/^# \{0,1\}//'; exit 2; } @@ -184,18 +186,44 @@ cmd_launch() { } cmd_send() { - local id=$1 message=$2 + local id=$1 message=$2 rec ring_rc=0 meta meta_lock validate_id "$id" validate_home "$id" - remote_endpoint_require "$id" - # fm-send's exit status is the delivery verdict the parent home acts on - # (0 = confirmed, 3 = delivered with the submit read-back unconfirmed, other - # nonzero = failed; see bin/fm-send.sh's header). The job worker, entrypoint, - # and ssh all preserve it, so no mapping may happen here: flattening exit 3 - # into a generic failure is exactly the false-negative the parent's remote - # send path exists to avoid. - FM_HOME="$TARGET_HOME" FM_ROOT_OVERRIDE="$FM_ROOT" FM_STATE_OVERRIDE="$TARGET_HOME/state" \ - "$SCRIPT_DIR/fm-send.sh" "$REMOTE_ENDPOINT_TARGET" "$message" + meta=$(meta_path "$id") + meta_lock=$(fm_meta_lock_path "$meta") || die "remote secondmate metadata lock path is invalid" + fm_task_inbox_lock_acquire "$meta_lock" \ + || die "remote secondmate endpoint metadata could not be locked for final delivery validation" + if ! remote_endpoint_load "$id"; then + fm_lock_release "$meta_lock" + die "$REMOTE_ENDPOINT_ERROR" + fi + # A remote steer is delivered by durable record, never by typing its payload + # into the pane: write it into this secondmate's host-local steering inbox, + # then ring the constant self-describing doorbell into the recorded pane, + # best-effort (bin/fm-task-inbox-lib.sh owns the record and doorbell). The + # write is idempotent - re-running the same request after an ambiguous + # transport failure lands on the existing record instead of a duplicate - so + # the parent may safely repeat this leg. Exit 0 once the record durably + # exists; no ring outcome changes it, because the parent's pending-reply + # reconciliation owns loss detection for a remote request from here. + if ! rec=$(fm_task_inbox_write_idempotent "$CONTROL_STATE" "$id" "$message"); then + fm_lock_release "$meta_lock" + die "steering-inbox record could not be written under $CONTROL_STATE/$id.inbox" + fi + fm_lock_release "$meta_lock" + case "$rec" in + */handled/*) + # The dedup landed on a record the worker already acknowledged: the + # steer was delivered and acted on, so there is nothing to announce. + printf 'notice: this steer was already delivered and acknowledged at %s; nothing re-rung\n' "$rec" >&2 + return 0 + ;; + esac + fm_task_inbox_ring "$REMOTE_ENDPOINT_BACKEND" "$REMOTE_ENDPOINT_TARGET" "$rec" "fm-$id" || ring_rc=$? + case "$ring_rc" in + 1) printf 'notice: doorbell skipped (composer visibly holds pending text); the steer is durably recorded at %s\n' "$rec" >&2 ;; + 2) printf 'notice: doorbell did not reach %s; the steer is durably recorded at %s\n' "$REMOTE_ENDPOINT_TARGET" "$rec" >&2 ;; + esac } cmd_key() { diff --git a/bin/fm-send.sh b/bin/fm-send.sh index 512df6245c7..99feadfb50c 100755 --- a/bin/fm-send.sh +++ b/bin/fm-send.sh @@ -1,5 +1,6 @@ #!/usr/bin/env bash -# Send one line of literal text to a crewmate endpoint, then Enter. +# Steer a task by durable record: write the message into the task's steering +# inbox and ring a constant doorbell line into its terminal, best-effort. # Usage: fm-send.sh [--resolve-key ]... # may be an exact task id, a legacy fm- task label resolved # through this home's state/.meta, or an explicit well-formed backend @@ -10,68 +11,138 @@ # Key support is backend-specific: tmux/herdr support Escape, Enter, and C-c; # Orca currently supports Enter and C-c only, and rejects Escape. # -# Text submission is verified: the line is typed ONCE, then Enter is sent and -# retried (Enter only, never retyped) until the target backend confirms a -# submit or reports an inconclusive send. If a swallowed Enter is positively -# confirmed, fm-send exits NON-ZERO so the caller knows the steer did not land -# instead of silently leaving an unsubmitted instruction. -# Exit status contract: 0 = submit confirmed (or, for a remote secondmate -# target, delivered with confirmation pending - see the remote paragraph); -# 3 = the text was typed into the live endpoint and Enter was sent, but the -# submit read-back stayed unconfirmed (verify the pane before any resend, and -# never re-type blindly; a marked request's pending-reply expectation stays -# armed because this outcome is not a proven failure); any other nonzero = the -# send failed and nothing may be assumed delivered. -# Submission dispatches through the target's recorded backend; the tmux adapter -# shares its composer/submit core with the away-mode daemon via bin/fm-tmux-lib.sh. -# Tune with FM_SEND_RETRIES (default 3) / FM_SEND_SLEEP (0.4). -# Slash commands, and codex `$...` skill invocations resolved through harness -# meta, get a longer pre-Enter settle so completion popups do not swallow Enter. +# Two data planes: +# +# INBOX - the default for text to a task recorded in this home, local and +# remote alike. The message is appended as a durable sequenced record under +# the task's steering inbox (newlines are legal) - state/.inbox/ for a +# local task, or the remote home's host-local inbox reached through fm-on.sh +# for a remote secondmate - and the terminal receives only one short constant +# self-describing doorbell line plus Enter, best-effort. The durable record IS +# the delivery, so the record's fate alone governs the exit: 0 = the steer is +# durably sent (recorded); nonzero = nothing was confirmed delivered and a +# resend is appropriate (unresolvable target, an endpoint that cannot be +# locked and revalidated or that retired or changed, an unwritable record, a +# failed or lost remote transport) or a decision-close append failed after +# delivery (the error then carries the exact manual close). The remote enqueue +# is idempotent: the remote leg deduplicates an exact re-run of the same +# request onto the existing record (bin/fm-task-inbox-lib.sh), so after a lost +# transport (ssh exit 255, completion unknown) fm-send retries the same leg +# once itself. A later re-run is idempotent only through the printed +# FM_PENDING_REPLY_EXISTING_CORR= command: it preserves the same +# correlation, body, and record, while a plain re-run mints a new correlation +# and delivers a separate record. A still-unconfirmed marked request keeps its +# reply expectation preserved for the record that may have landed. +# Pending-reply bookkeeping trouble after a durable enqueue NEVER exits +# nonzero: with the recovery marker stored the watcher reconciles it silently, +# and with both the commit and the marker lost the send prints a distinct +# "reply-tracking-degraded (steer delivered, do not resend)" warning instead, +# because a resend-inviting status there would duplicate a delivered +# instruction. There is no delivered-unconfirmed +# outcome on this plane: "did the doorbell land" is no longer the question - +# "was the message acted on" is, and that is answered asynchronously by the +# worker's acknowledgement move into handled/, with the watcher re-ringing an +# unacknowledged message and escalating a stuck one. bin/fm-task-inbox-lib.sh +# owns the record format, the doorbell line, and the re-ring ladder. The +# composer pre-check before the ring is ADVISORY only: when the composer +# visibly holds pending text the ring is skipped with a notice and the watcher +# re-rings later; no composer verdict is delivery proof on this plane, and a +# failed ring never fails the send. +# +# TYPED - the LOCAL text that must reach the terminal itself: a harness-native +# invocation (a leading "/", or a leading "$" to a codex target) must reach +# the harness's own parser, and an explicit backend target names an endpoint, +# not a task, so it stays typed even when local metadata happens to match it +# (the same boundary that keeps it unmarked and outside --resolve-key). These +# type the literal +# text through the target backend's verified submit core: typed ONCE, then +# Enter retried (never retyped) until the backend confirms a submit or reports +# an inconclusive send. Typed-plane exit contract: 0 = submit confirmed; +# 3 = the text was typed into the live endpoint and +# Enter was sent, but the submit read-back stayed unconfirmed (verify the pane +# before any resend, and never re-type blindly; a marked request's +# pending-reply expectation stays armed because this outcome is not a proven +# failure); any other nonzero = the send failed and nothing may be assumed +# delivered. Submission dispatches through the target's recorded backend; the +# tmux adapter shares its composer/submit core with the away-mode daemon via +# bin/fm-tmux-lib.sh. Tune with FM_SEND_RETRIES (default 3) / FM_SEND_SLEEP +# (0.4). Slash commands, and codex `$...` skill invocations resolved through +# harness meta, get a longer pre-Enter settle so completion popups do not +# swallow Enter. A remote secondmate target has no typed text plane at all: +# every remote text steer rides the inbox (a marked secondmate request already +# reaches the harness as marker-prefixed chat rather than a parser command, so +# routing a remote "/..." or "$..." through the record changes nothing the +# parser would have seen); only --key still crosses to the remote pane as a +# keystroke. +# +# Stage-1 compatibility boundary: classification uses the original pre-marker +# text, but secondmate marking still precedes every typed submission. Therefore +# a marked parser-native secondmate invocation intentionally reaches the harness +# as marker-prefixed chat rather than executing as a parser command. This is a +# pre-existing interaction retained for byte compatibility in this local-inbox +# stage; do not move the marker behind the invocation or omit it here. Follow-up +# fm-send-secondmate-harness-invocation-r1 owns that behavior. # # From-firstmate marker: when the resolved target is a task selector whose meta -# records kind=secondmate, the text uses the live-charter-compatible +# records kind=secondmate, the message uses the live-charter-compatible # from-firstmate carrier owned by bin/fm-operational-input.sh so the secondmate # routes its reply via its status file or a status-pointed doc instead of -# stranding it in chat the main firstmate never reads. A crewmate/scout target, +# stranding it in chat the main firstmate never reads. On the inbox plane the +# marker travels verbatim inside the recorded body. A crewmate/scout target, # an explicit backend-target escape-hatch target, and the --key path are never # marked - their behavior is unchanged. # # Parent-owned pending-reply expectation: every newly marked secondmate request # also receives a privacy-safe correlation id and a durable parent record under # state/pending-replies/ before delivery (bin/fm-pending-reply-lib.sh). Delivery -# success and reply success are separate facts: a successful submit never -# resolves the expectation, and an unconfirmed submit (exit 3) keeps it armed -# rather than dropping it; only a proven send failure discards it. Set -# FM_PENDING_REPLY_EXISTING_CORR= when re-sending a recovery request for an -# already-open expectation so a second record is not created. Direct unmarked -# captain input never creates one. +# success and reply success are separate facts: delivery never resolves the +# expectation. On the inbox plane the durable enqueue IS delivery to the task's +# record, so the expectation is marked delivered at enqueue time; when that +# bookkeeping commit fails after its durable recovery marker is stored, the +# send remains successful and watcher reconciliation owns the repair, and when +# the commit and marker are BOTH lost the send still remains successful with a +# reply-tracking-degraded warning naming the expectation an operator must +# inspect (it can no longer reconcile or escalate on its own). Only a +# failed enqueue discards the expectation. On the typed plane an unconfirmed submit (exit 3) keeps +# it armed rather than dropping it, and only a proven send failure discards it. +# Set FM_PENDING_REPLY_EXISTING_CORR= when re-sending a recovery request +# for an already-open expectation so a second record is not created. Direct +# unmarked captain input never creates one. # # Remote secondmate delivery: the send crosses fm-on.sh to a host-local leg -# (bin/fm-remote-secondmate-control.sh cmd_send) that runs this same verified -# submit against the recorded remote Herdr pane and relays its exit status -# unchanged. A leg that delivered the text into the live verified pane but -# could not synchronously confirm the submit (exit 3 - typically a busy mate -# whose harness queues the steer and keeps rendering it) is reported here as -# DELIVERED with confirmation pending: fm-send prints a non-error notice, -# exits 0, marks the pending-reply expectation delivered, and closes any -# --resolve-key decisions. Empirically that pattern is a delivered steer, a -# resend duplicates the instruction, and the parent's pending-reply -# recovery/escalation still surfaces the rare genuinely lost request. Transport -# loss (ssh exit 255, completion unknown) and every real remote failure keep -# failing loudly with the remote leg's own stderr attached. +# (bin/fm-remote-secondmate-control.sh cmd_send) that writes the message as a +# durable record into the remote home's steering inbox and rings the remote +# doorbell, best-effort. The remote record is the delivery, exactly as it is +# locally: leg exit 0 means durably recorded (fm-send then exits 0, marks the +# pending-reply expectation delivered, and closes any --resolve-key +# decisions), and any real remote failure fails loudly with the remote leg's +# own stderr attached. Transport loss (ssh exit 255) means completion unknown, +# so fm-send retries the identical leg once - safe because the remote write +# deduplicates the same request onto the same record - and a still-lost +# transport exits nonzero while preserving a marked request's reply +# expectation, since the record may have landed. Its error prints the exact +# FM_PENDING_REPLY_EXISTING_CORR= resend command that preserves the body +# and makes a later remote enqueue deduplicate onto that same record. The +# remote host runs no re-ring ladder of +# its own: a swallowed remote doorbell surfaces through the parent's +# pending-reply recovery and escalation, whose recovery request re-rings the +# remote doorbell when it is enqueued. # # Decision closure (answerer-closes): pass --resolve-key (repeatable, # before the message) when this send answers an open keyed needs-decision: or -# blocked: record in the target task's state/.status. After the submit is -# confirmed, fm-send itself appends the closing -# "resolved [key=]: answered: " line to that status file, -# so the captain-facing OPEN DECISIONS record closes at answer time and never -# depends on the busy worker writing a matching resolved line. The close is a -# LOCAL append for every target kind - crewmate, scout, local secondmate, and -# remote secondmate alike - because the open-decision ledger fm-wake-drain -# folds lives in this home's own state dir (a remote mate's escalations reach -# it through the parent-replies ingest); only the answer message crosses the -# backend or remote transport. +# blocked: record in the target task's state/.status. fm-send itself +# appends the closing "resolved [key=]: answered: " line +# to that status file, so the captain-facing OPEN DECISIONS record closes at +# answer time and never depends on the busy worker writing a matching resolved +# line. On the inbox plane the close happens at ENQUEUE time, because enqueue +# is durable delivery to the task's record; the worker reading the answer late +# is covered by the acknowledgement re-ring ladder. On the typed plane it +# still waits for the confirmed submit. The close is a LOCAL append for every +# target kind - crewmate, scout, local secondmate, and remote secondmate alike +# - because the open-decision ledger fm-wake-drain folds lives in this home's +# own state dir (a remote mate's escalations reach it through the +# parent-replies ingest); only the answer message crosses the backend or +# remote transport. # # Chat is also a channel that carries keyed captain answers, so the same flag # feeds bin/fm-captain-hold.sh's one keyed-answer intake for any key that names @@ -87,9 +158,7 @@ # in this home's status log per status_open_decisions (bin/fm-classify-lib.sh), or # a still-open captain-held task resolved as above. A key in neither is refused # before sending, so a mistyped key cannot deliver an answer while silently -# orphaning the decision. A failed or unconfirmed send never closes a key (a remote -# delivered-with-pending-confirmation outcome counts as delivered - see the -# remote paragraph above); a +# orphaning the decision. A failed or unconfirmed send never closes a key; a # delivered answer whose closing append fails exits nonzero with the exact # manual close command, leaving the decision open to re-surface (the safe # direction). A send without the flag never closes anything: a routine steer, @@ -97,14 +166,16 @@ # refused with --key, with an explicit backend target (no task ledger in this # home), and with an empty message. # -# After a successful text submit fm-send pauses FM_SEND_SETTLE seconds (default 1, -# 0 disables) before returning: submit confirmation only proves the text was -# accepted, but the harness needs a beat to spin up the turn before its busy -# footer appears, so an immediate peek would otherwise see the stale idle pane. -# The pause is fm-send-only; the shared submit core (used by the away-mode daemon, -# which only needs "submitted") does not pay it, and the --key path is unaffected. +# After a successful TYPED-plane submit fm-send pauses FM_SEND_SETTLE seconds +# (default 1, 0 disables) before returning: submit confirmation only proves the +# text was accepted, but the harness needs a beat to spin up the turn before its +# busy footer appears, so an immediate peek would otherwise see the stale idle +# pane. The pause is typed-plane-only; the inbox plane, the shared submit core +# (used by the away-mode daemon, which only needs "submitted"), and the --key +# path do not pay it. set -eu +FM_SEND_ORIGINAL_ARGS=("$@") SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" @@ -143,6 +214,8 @@ fi . "$SCRIPT_DIR/fm-line-cap-lib.sh" # shellcheck source=bin/fm-wake-lib.sh . "$SCRIPT_DIR/fm-wake-lib.sh" +# shellcheck source=bin/fm-task-inbox-lib.sh +. "$SCRIPT_DIR/fm-task-inbox-lib.sh" FM_GUARD_CONTINUE_LINE='This is a supervision warning only; the requested message WILL still be sent.' "$SCRIPT_DIR/fm-guard.sh" || true @@ -231,6 +304,7 @@ fm_send_resolve_target() { # TARGET_META="" TARGET_SELECTOR="" TARGET_REMOTE_ID="" + TARGET_REMOTE_HOST="" RESOLUTION_TRIED="" meta=$(fm_backend_meta_for_selector "$raw" "$STATE" 2>/dev/null || true) @@ -244,6 +318,7 @@ fm_send_resolve_target() { # EXPECTED_LABEL="fm-$id" TARGET_SELECTOR=1 TARGET_REMOTE_ID=$id + TARGET_REMOTE_HOST=$(fm_meta_get "$meta" remote_host) RESOLUTION_TRIED="meta=$meta; placement=remote" return 0 fi @@ -376,6 +451,14 @@ MARK_FROM_FIRSTMATE=0 PENDING_REPLY_CORR= PENDING_REPLY_CREATED=0 TARGET_TASK_ID= +fm_send_known_undelivered_cleanup() { + [ -n "$PENDING_REPLY_CORR" ] || return 0 + if [ "$PENDING_REPLY_CREATED" = 1 ]; then + fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" + else + fm_pending_reply_reset_known_undelivered "$STATE" "$PENDING_REPLY_CORR" + fi +} if [ -n "$TARGET_SELECTOR" ] && [ -n "$TARGET_META" ] && [ "$(fm_meta_get "$TARGET_META" kind)" = secondmate ]; then MARK_FROM_FIRSTMATE=1 TARGET_TASK_ID=$(fm_send_id_from_meta "$TARGET_META") @@ -453,8 +536,9 @@ if [ -n "$RESOLVE_KEYS" ]; then done fi -# Close each answered decision in this home's ledger, only after delivery is -# fully confirmed. An append failure exits nonzero with the manual close +# Close each answered decision in this home's ledger, only after the answer is +# durably sent: enqueued on the inbox plane, submit-confirmed on the typed +# plane. An append failure exits nonzero with the manual close # command; the decision then stays open and re-surfaces, never silently lost. # The close is this home's own bookkeeping, written by the very turn that # answered the decision, so it goes through the guarded self-announced append @@ -534,11 +618,21 @@ else # Reuse an existing correlation id for recovery resends; otherwise create a # durable parent expectation before delivery. Transport success never # resolves that expectation (see fm-pending-reply-lib.sh). - existing_corr=${FM_PENDING_REPLY_EXISTING_CORR:-$(fm_pending_reply_extract_corr "$MESSAGE")} + existing_corr_explicit=0 + if [ "${FM_PENDING_REPLY_EXISTING_CORR+x}" = x ]; then + existing_corr_explicit=1 + existing_corr=$FM_PENDING_REPLY_EXISTING_CORR + else + existing_corr=$(fm_pending_reply_extract_corr "$MESSAGE") + fi if [ -n "$existing_corr" ] \ && fm_pending_reply_corr_reusable "$STATE" "$existing_corr" "$TARGET_TASK_ID"; then PENDING_REPLY_CORR=$existing_corr else + if [ "$existing_corr_explicit" = 1 ]; then + echo "error: explicitly requested pending-reply correlation '${existing_corr:-empty}' is not reusable for $TARGET_TASK_ID; refusing to mint a replacement correlation" >&2 + exit 1 + fi if [ -z "$TARGET_TASK_ID" ]; then echo "error: cannot create pending-reply expectation without a resolvable secondmate task id" >&2 exit 1 @@ -548,13 +642,213 @@ else PENDING_REPLY_CREATED=1 fi fm_pending_reply_embed_corr "$MESSAGE" "$PENDING_REPLY_CORR" MESSAGE - if [ "$PENDING_REPLY_CREATED" = 1 ] \ - && ! fm_pending_reply_prepare_delivery "$STATE" "$PENDING_REPLY_CORR"; then - fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" || true + if [ "$PENDING_REPLY_CREATED" != 1 ] \ + && fm_pending_reply_delivery_attempt_unresolved "$STATE" "$PENDING_REPLY_CORR"; then + if [ "$TARGET_BACKEND" = remote ]; then + if ! fm_pending_reply_reset_known_undelivered "$STATE" "$PENDING_REPLY_CORR"; then + echo "error: pending-reply delivery for $TARGET_TASK_ID could not be reset for an idempotent remote resend of correlation $PENDING_REPLY_CORR" >&2 + exit 1 + fi + else + echo "error: pending-reply delivery for $TARGET_TASK_ID is unresolved; refusing to resend correlation $PENDING_REPLY_CORR" >&2 + exit 1 + fi + fi + if ! fm_pending_reply_prepare_delivery "$STATE" "$PENDING_REPLY_CORR"; then + [ "$PENDING_REPLY_CREATED" != 1 ] \ + || fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" || true echo "error: failed to durably prepare pending-reply delivery for $TARGET_TASK_ID" >&2 exit 1 fi fi + # Data-plane selection (see the header): text addressed to a task selector + # resolved through this home's metadata rides the inbox plane, unless it is + # a LOCAL harness-native invocation that must reach the harness's own parser + # - a leading "/" (slash command), or a leading "$" to a codex target (skill + # invocation). A remote secondmate selector always rides the inbox: its + # requests are marked, and a marked request reaches the harness as + # marker-prefixed chat rather than a parser command anyway, so no remote + # text has a typed plane to lose. An explicit backend target stays typed + # even when it happens to match local metadata: it names an endpoint, not a + # task, the same boundary that keeps it unmarked and outside --resolve-key. + # Classification reads the pre-marker text so a marked secondmate request + # and a plain crewmate steer classify identically. It deliberately does NOT + # promise that a marked parser-native secondmate request executes as a parser + # command: the pre-existing marker-first wire bytes are retained in stage 1. + INBOX_PLANE=0 + if [ -n "$TARGET_SELECTOR" ]; then + if [ "$TARGET_BACKEND" = remote ]; then + INBOX_PLANE=1 + else + case "$RESOLVE_ANSWER_TEXT" in + /*) ;; + \$*) [ "$TARGET_HARNESS" = codex ] || INBOX_PLANE=1 ;; + *) INBOX_PLANE=1 ;; + esac + fi + fi + if [ "$INBOX_PLANE" = 1 ] && [ "$TARGET_BACKEND" = remote ]; then + # Remote inbox leg: the message becomes a durable record in the remote + # home's steering inbox, written idempotently by the host-local leg, then + # the remote doorbell rings, best-effort. One identical retry after ssh + # 255 is safe by that idempotence; a still-lost transport preserves a + # marked request's reply expectation because the record may have landed. + REMOTE_META_LOCK=$(fm_meta_lock_path "$TARGET_META") || exit 1 + if ! fm_task_inbox_lock_acquire "$REMOTE_META_LOCK"; then + if [ "$PENDING_REPLY_CREATED" = 1 ] && [ -n "$PENDING_REPLY_CORR" ]; then + fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" || true + fi + echo "error: steer not sent to remote secondmate $TARGET_REMOTE_ID: its parent task metadata could not be locked for final delivery validation" >&2 + exit 1 + fi + CURRENT_REMOTE_ID= + CURRENT_REMOTE_HOST= + if [ -f "$TARGET_META" ]; then + CURRENT_REMOTE_ID=$(fm_send_id_from_meta "$TARGET_META") + CURRENT_REMOTE_HOST=$(fm_meta_get "$TARGET_META" remote_host) + fi + if [ "$CURRENT_REMOTE_ID" != "$TARGET_REMOTE_ID" ] \ + || [ -z "$CURRENT_REMOTE_HOST" ] \ + || [ "$CURRENT_REMOTE_HOST" != "$TARGET_REMOTE_HOST" ]; then + fm_lock_release "$REMOTE_META_LOCK" + if [ "$PENDING_REPLY_CREATED" = 1 ] && [ -n "$PENDING_REPLY_CORR" ]; then + fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" || true + fi + echo "error: steer not sent to remote secondmate $TARGET_REMOTE_ID: its parent task retired or changed route during target resolution" >&2 + exit 1 + fi + remote_rc=0 + remote_completion_unknown=0 + "$SCRIPT_DIR/fm-on.sh" "$TARGET_REMOTE_ID" fm-remote-secondmate-control.sh send \ + "$TARGET_REMOTE_ID" "$MESSAGE" < /dev/null || remote_rc=$? + if [ "$remote_rc" -eq 255 ]; then + remote_completion_unknown=1 + remote_rc=0 + "$SCRIPT_DIR/fm-on.sh" "$TARGET_REMOTE_ID" fm-remote-secondmate-control.sh send \ + "$TARGET_REMOTE_ID" "$MESSAGE" < /dev/null || remote_rc=$? + fi + fm_lock_release "$REMOTE_META_LOCK" + if [ "$remote_rc" -ne 0 ] && [ "$remote_completion_unknown" -eq 1 ]; then + if [ -n "$PENDING_REPLY_CORR" ]; then + fm_pending_reply_mark_delivery_unknown "$STATE" "$PENDING_REPLY_CORR" || true + fi + if [ "$remote_rc" -eq 255 ]; then + echo "error: steer to remote secondmate $TARGET_REMOTE_ID is unconfirmed (transport lost twice; remote completion unknown). Only the correlation-reusing resend below is idempotent and lands on the same remote inbox record:" >&2 + else + echo "error: steer to remote secondmate $TARGET_REMOTE_ID is unconfirmed (the first transport attempt had unknown completion and the retry failed). Only the correlation-reusing resend below is idempotent and lands on the same remote inbox record:" >&2 + fi + resend_home=$(cd "$FM_HOME" 2>/dev/null && pwd) || resend_home=$FM_HOME + printf 'FM_HOME=%q ' "$resend_home" >&2 + if [ "${FM_STATE_OVERRIDE+x}" = x ]; then + resend_state=$(cd "$STATE" 2>/dev/null && pwd) || resend_state=$STATE + printf 'FM_STATE_OVERRIDE=%q ' "$resend_state" >&2 + fi + printf 'FM_PENDING_REPLY_EXISTING_CORR=%q %q' "$PENDING_REPLY_CORR" "$SCRIPT_DIR/fm-send.sh" >&2 + for resend_arg in "${FM_SEND_ORIGINAL_ARGS[@]}"; do + printf ' %q' "$resend_arg" >&2 + done + printf '\n' >&2 + exit 1 + fi + if [ "$remote_rc" -ne 0 ]; then + fm_send_known_undelivered_cleanup || \ + echo "error: known-undelivered pending-reply state could not be reset for $TARGET_TASK_ID" >&2 + echo "error: steer not sent to remote secondmate $TARGET_REMOTE_ID (the remote steering-inbox record could not be written; the remote leg's stderr above has the reason)" >&2 + exit 1 + fi + # The remote record is durable delivery, exactly as a local enqueue is. + if [ -n "$PENDING_REPLY_CORR" ]; then + if fm_pending_reply_confirm_delivery "$STATE" "$PENDING_REPLY_CORR"; then + : + else + delivery_commit_status=$? + if [ "$delivery_commit_status" = 2 ]; then + echo "notice: the steer was durably recorded in the remote inbox, but its pending-reply delivery commit failed; a durable recovery marker was stored and the watcher will reconcile it. Do not resend." >&2 + else + echo "warning: reply-tracking-degraded (steer delivered, do not resend): the steer was durably recorded in the remote inbox, but its pending-reply delivery commit and recovery marker both failed, so the reply expectation for this request may not reconcile on its own. Inspect $STATE." >&2 + fi + fi + fi + if [ -n "$RESOLVE_KEYS" ]; then + fm_send_close_resolved_keys "$RESOLVE_ANSWER_TEXT" || exit 1 + fm_send_feed_resolved_holds "$RESOLVE_ANSWER_TEXT" || exit 1 + fi + exit 0 + fi + if [ "$INBOX_PLANE" = 1 ]; then + INBOX_TASK_ID=$(fm_send_id_from_meta "$TARGET_META") + INBOX_META_LOCK=$(fm_meta_lock_path "$TARGET_META") || exit 1 + if ! fm_task_inbox_lock_acquire "$INBOX_META_LOCK"; then + if [ "$PENDING_REPLY_CREATED" = 1 ] && [ -n "$PENDING_REPLY_CORR" ]; then + fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" || true + fi + echo "error: steer not sent to $INBOX_TASK_ID: its task metadata could not be locked for final delivery validation" >&2 + exit 1 + fi + CURRENT_INBOX_TARGET= + CURRENT_INBOX_BACKEND= + if [ -f "$TARGET_META" ]; then + CURRENT_INBOX_TARGET=$(fm_backend_target_of_meta "$TARGET_META") + CURRENT_INBOX_BACKEND=$(fm_backend_of_meta "$TARGET_META") + fi + if [ "$CURRENT_INBOX_TARGET" != "$T" ] \ + || [ "$CURRENT_INBOX_BACKEND" != "$TARGET_BACKEND" ] \ + || [ -n "$(fm_meta_get "$TARGET_META" remote_host)" ]; then + fm_lock_release "$INBOX_META_LOCK" + if [ "$PENDING_REPLY_CREATED" = 1 ] && [ -n "$PENDING_REPLY_CORR" ]; then + fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" || true + fi + echo "error: steer not sent to $INBOX_TASK_ID: the task retired or changed endpoint during target resolution" >&2 + exit 1 + fi + if ! INBOX_RECORD=$(fm_task_inbox_write "$STATE" "$INBOX_TASK_ID" "$MESSAGE"); then + fm_lock_release "$INBOX_META_LOCK" + if [ "$PENDING_REPLY_CREATED" = 1 ] && [ -n "$PENDING_REPLY_CORR" ]; then + fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" || true + fi + echo "error: steer not sent to $INBOX_TASK_ID: its inbox record could not be written under $STATE/$INBOX_TASK_ID.inbox" >&2 + exit 1 + fi + fm_lock_release "$INBOX_META_LOCK" + # Enqueue IS durable delivery to the task's record: mark the pending + # expectation delivered now, without resolving it - only a correlated + # parent report acknowledges the request. + if [ -n "$PENDING_REPLY_CORR" ]; then + if fm_pending_reply_confirm_delivery "$STATE" "$PENDING_REPLY_CORR"; then + : + else + delivery_commit_status=$? + if [ "$delivery_commit_status" = 2 ]; then + echo "notice: the steer was recorded at $INBOX_RECORD, but its pending-reply delivery commit failed; a durable recovery marker was stored and the watcher will reconcile it. Do not resend." >&2 + else + # Both the commit and its recovery marker failed. The durable inbox + # record is what delivers the steer, so the send still SUCCEEDED: + # a nonzero here would read as undelivered to every automated caller + # and invite a duplicate enqueue - the exact defect this plane + # removes. Surface the degradation as its own distinct, + # non-resend-inviting condition instead: reply tracking for this + # request may not resolve or escalate on its own until an operator + # inspects it. + echo "warning: reply-tracking-degraded (steer delivered, do not resend): the steer was durably recorded at $INBOX_RECORD, but its pending-reply delivery commit and recovery marker both failed, so the reply expectation for this request may not reconcile on its own. Inspect $STATE." >&2 + fi + fi + fi + # The answer is durably sent: close each answered decision at enqueue time + # (answerer-closes; see the header contract). + if [ -n "$RESOLVE_KEYS" ]; then + fm_send_close_resolved_keys "$RESOLVE_ANSWER_TEXT" || exit 1 + fm_send_feed_resolved_holds "$RESOLVE_ANSWER_TEXT" || exit 1 + fi + # Ring the doorbell, best-effort: no ring outcome changes the exit status, + # because the watcher's re-ring ladder owns loss detection from here. + ring_rc=0 + fm_task_inbox_ring "$TARGET_BACKEND" "$T" "$INBOX_RECORD" "$EXPECTED_LABEL" || ring_rc=$? + case "$ring_rc" in + 1) echo "fm-send: doorbell skipped (composer visibly holds pending text); the steer is durably recorded at $INBOX_RECORD and the watcher will re-ring" >&2 ;; + 2) echo "fm-send: doorbell did not reach $T; the steer is durably recorded at $INBOX_RECORD and the watcher will re-ring" >&2 ;; + esac + exit 0 + fi # Slash commands open a completion popup in some TUIs (verified on codex); # submitting too fast selects nothing, so give the popup time to settle before # the (retried) Enter. Codex opens the same kind of popup for a `$` @@ -573,44 +867,18 @@ else retries=${FM_SEND_RETRIES:-3} sleep_s=${FM_SEND_SLEEP:-0.4} # Type once, submit, verify. Only exact empty confirms delivery; every other - # verdict preserves the loud refusal boundary. + # verdict preserves the loud refusal boundary. Only LOCAL targets reach this + # block: remote text rides the inbox leg above, and remote --key exits + # earlier. send_rc=0 - REMOTE_DELIVERY_NOTICE=0 - if [ "$TARGET_BACKEND" = remote ]; then - # The remote leg is this same script running host-locally against the - # recorded Herdr pane (cmd_send in fm-remote-secondmate-control.sh), so its - # submit verification IS the local one, and fm-on/the remote worker relay - # its exit status unchanged. Exit 3 is the delivered-unconfirmed contract - # (see this script's header) crossing the ssh boundary: the text reached - # the live verified pane and Enter was sent; only the synchronous read-back - # stayed unconfirmed. The remote stderr is held back and replayed only for - # a real failure, so a delivered outcome does not surface the inner leg's - # diagnostics as alarm. - remote_err=$("$SCRIPT_DIR/fm-on.sh" "$TARGET_REMOTE_ID" fm-remote-secondmate-control.sh send "$TARGET_REMOTE_ID" "$MESSAGE" < /dev/null 2>&1 >/dev/null) || send_rc=$? - if [ "$send_rc" -eq 0 ]; then - verdict=empty - elif [ "$send_rc" -eq 3 ]; then - verdict=empty - send_rc=0 - REMOTE_DELIVERY_NOTICE=1 - else - verdict=send-failed - [ -z "$remote_err" ] || printf '%s\n' "$remote_err" >&2 - fi - elif verdict=$(fm_backend_send_text_submit "$TARGET_BACKEND" "$T" "$MESSAGE" "$retries" "$sleep_s" "$settle" "$EXPECTED_LABEL"); then + if verdict=$(fm_backend_send_text_submit "$TARGET_BACKEND" "$T" "$MESSAGE" "$retries" "$sleep_s" "$settle" "$EXPECTED_LABEL"); then : else send_rc=$? fi if [ "$send_rc" -ne 0 ]; then - if [ "$TARGET_BACKEND" = remote ] && [ "$send_rc" -eq 255 ] && [ -n "$PENDING_REPLY_CORR" ]; then - fm_pending_reply_mark_delivery_unknown "$STATE" "$PENDING_REPLY_CORR" || true - echo "error: text delivery to remote secondmate $TARGET_REMOTE_ID is unknown; do not resend - same-host reconciliation is required" >&2 - exit 1 - fi - if [ "$PENDING_REPLY_CREATED" = 1 ] && [ -n "$PENDING_REPLY_CORR" ]; then - fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" || true - fi + fm_send_known_undelivered_cleanup || \ + echo "error: known-undelivered pending-reply state could not be reset for $TARGET_TASK_ID" >&2 echo "error: text not sent to $T ($TARGET_BACKEND send failed; tried $RESOLUTION_TRIED)" >&2 exit 1 fi @@ -618,9 +886,8 @@ else empty) ;; send-failed) - if [ "$PENDING_REPLY_CREATED" = 1 ] && [ -n "$PENDING_REPLY_CORR" ]; then - fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" || true - fi + fm_send_known_undelivered_cleanup || \ + echo "error: known-undelivered pending-reply state could not be reset for $TARGET_TASK_ID" >&2 echo "error: text not sent to $T ($TARGET_BACKEND send failed; tried $RESOLUTION_TRIED)" >&2 exit 1 ;; @@ -629,11 +896,9 @@ else # submit read-back stayed unconfirmed (e.g. a busy harness queues the # steer and keeps rendering it). That is not a proven failure, so never # re-type the message: verify the pane instead. Exit 3 is the documented - # delivered-unconfirmed status, and the remote send leg above depends on - # it crossing the ssh boundary intact. - # The pending-reply expectation is deliberately NOT discarded here: this - # is the same not-a-failure outcome the remote leg reports as delivered, - # so dropping it would silently stop tracking a marked request that very + # delivered-unconfirmed status. + # The pending-reply expectation is deliberately NOT discarded here: + # dropping it would silently stop tracking a marked request that very # likely landed. It stays armed on its unconfirmed-delivery marker, so a # correlated report still resolves it and an unanswered one still # surfaces through the library's own reconciliation @@ -670,12 +935,6 @@ else fm_send_close_resolved_keys "$RESOLVE_ANSWER_TEXT" || exit 1 fm_send_feed_resolved_holds "$RESOLVE_ANSWER_TEXT" || exit 1 fi - # Remote delivered-with-pending-confirmation: the outcome above is treated as - # delivered (expectation marked, keys closed), and this one non-error notice - # carries the remaining nuance so nobody re-sends the steer. - if [ "$REMOTE_DELIVERY_NOTICE" = 1 ]; then - echo "fm-send: delivered to remote secondmate $TARGET_REMOTE_ID; the remote pane accepted the text and Enter, and only the synchronous submit confirmation is still pending. This is not a failure - do not resend; the pending-reply expectation stays armed." >&2 - fi # Submit landed with exact empty. Confirmation only proves the text was # accepted; the harness still needs a beat to spin up the # turn before its busy footer shows. Pause so an immediate peek catches the diff --git a/bin/fm-task-inbox-lib.sh b/bin/fm-task-inbox-lib.sh new file mode 100644 index 00000000000..6ad482eafd3 --- /dev/null +++ b/bin/fm-task-inbox-lib.sh @@ -0,0 +1,381 @@ +#!/usr/bin/env bash +# fm-task-inbox-lib.sh - the per-task steering inbox: durable records plus a +# constant doorbell. +# +# ONE owner of the steering-inbox contract: the record format, sequence +# allocation, the idempotent re-enqueue dedup, the handled/ acknowledgement, +# the self-describing doorbell line, and the watcher's re-ring ladder policy. +# bin/fm-send.sh writes and rings locally, the host-local remote steer leg +# (bin/fm-remote-secondmate-control.sh cmd_send) writes idempotently and rings +# on the remote host, bin/fm-watch.sh polls and re-rings, and the brief +# scaffold (bin/fm-brief.sh) tells the worker how to read and acknowledge; +# none of them restates the format. +# +# Design (captain-adopted, data/fm-send-reliability-reframe-s1/report.md): the +# payload moves to the filesystem, which is reliable; the terminal carries only +# a short constant doorbell line, which does not need to be reliable because +# ringing it again is free. A duplicated doorbell is a no-op by construction +# (the worker finds the inbox empty or already handled), a swallowed doorbell +# is detected by the absence of the worker's acknowledgement and re-rung on a +# bounded schedule, and a worker that never acknowledges surfaces through the +# ordinary stale wake into stuck-crewmate-recovery. +# +# Layout under : +# .inbox/NNN.msg one durable steer, numeric sequence, atomic rename +# .inbox/handled/ the worker's `mv` here IS the acknowledgement +# .inbox/.seq.lock serializes sequence allocation across writers +# (the session and the away daemon) +# .inbox/.ring-state watcher re-ring ladder: "\t\t" +# .inbox/.escalated oldest-message name already surfaced as stale, +# so later polls suppress another escalation +# +# Record format (fm_task_inbox_write / fm_task_inbox_body): +# schema=fm-task-inbox.v1 +# at= +# -- +# +# +# Sequence numbers are never reused within a task: allocation scans both the +# inbox root and handled/, so a message is processed at most once per worker +# lifetime even if every doorbell is duplicated. Concurrent writers serialize +# on .seq.lock; the worst racing outcome is ordering, never loss. +# +# Re-ring ladder (fm_task_inbox_due_action): an unhandled message older than +# FM_TASK_INBOX_GRACE_SECS is due one delivery attempt per grace period; an +# attempt may ring or be skipped to protect proven pending composer text. After +# FM_TASK_INBOX_RING_MAX attempts without an acknowledgement it escalates. The +# caller owns the busy check (a busy pane just waits - the record is durable and +# the worker reaches a turn boundary) and the wake emission; this library owns +# only the schedule. If attempt bookkeeping cannot be persisted while the record +# remains unhandled, the caller surfaces that failure instead of retrying +# silently; a concurrently removed inbox is a quiet no-op. Escalation +# deliberately queues the wake before writing the +# deduplication marker: normal polls surface a message once, while a crash or +# marker failure may produce a rare duplicate rather than silently lose a wake. +# +# fm_task_inbox_ring requires bin/fm-backend.sh's dispatch (sourced below); the +# other helpers are dependency-light. Sourced by bin/fm-send.sh, bin/fm-watch.sh, +# and tests. No side effects on source beyond its sourced libraries. +# +# Tunables (env): +# FM_TASK_INBOX_GRACE_SECS default 90; delivery-attempt grace and spacing +# FM_TASK_INBOX_RING_MAX default 3; delivery attempts before escalation + +_FM_TASK_INBOX_LIB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# Both dependencies are canonical lint roots in their own right. Keep them as +# analysis boundaries here so ShellCheck's external-source traversal does not +# recursively duplicate the full backend graph for every inbox consumer. +# shellcheck source=/dev/null +. "$_FM_TASK_INBOX_LIB_DIR/fm-wake-lib.sh" +# shellcheck source=/dev/null +. "$_FM_TASK_INBOX_LIB_DIR/fm-backend.sh" + +FM_TASK_INBOX_SCHEMA='fm-task-inbox.v1' +FM_TASK_INBOX_GRACE_DEFAULT=90 +FM_TASK_INBOX_RING_MAX_DEFAULT=3 +FM_TASK_INBOX_LOCK_WAIT_DEFAULT=5 + +fm_task_inbox_grace_secs() { + local g=${FM_TASK_INBOX_GRACE_SECS:-$FM_TASK_INBOX_GRACE_DEFAULT} + case "$g" in ''|*[!0-9]*) g=$FM_TASK_INBOX_GRACE_DEFAULT ;; esac + printf '%s' "$g" +} + +fm_task_inbox_ring_max() { + local m=${FM_TASK_INBOX_RING_MAX:-$FM_TASK_INBOX_RING_MAX_DEFAULT} + case "$m" in ''|*[!0-9]*) m=$FM_TASK_INBOX_RING_MAX_DEFAULT ;; esac + printf '%s' "$m" +} + +fm_task_inbox_dir() { # + printf '%s/%s.inbox' "$1" "$2" +} + +fm_task_inbox_handled_dir() { # + printf '%s/%s.inbox/handled' "$1" "$2" +} + +# Numeric sequence of one record basename, or fail for a non-record name. +fm_task_inbox_seq_of() { # + local n=${1%.msg} + [ "$n" != "$1" ] || return 1 + case "$n" in ''|*[!0-9]*) return 1 ;; esac + printf '%s' "$((10#$n))" +} + +# Next unused sequence, scanning the inbox root AND handled/ so an +# acknowledged sequence is never reissued. Caller must hold .seq.lock. +fm_task_inbox_next_seq() { # + local dir=$1 max=0 d f n + for d in "$dir" "$dir/handled"; do + for f in "$d"/*.msg; do + [ -e "$f" ] || continue + n=$(fm_task_inbox_seq_of "${f##*/}") || continue + [ "$n" -le "$max" ] || max=$n + done + done + printf '%03d' "$((max + 1))" +} + +fm_task_inbox_lock_acquire() { # + local lock=$1 wait=${FM_TASK_INBOX_LOCK_WAIT_SECS:-$FM_TASK_INBOX_LOCK_WAIT_DEFAULT} + local deadline probe + case "$wait" in ''|*[!0-9]*) wait=$FM_TASK_INBOX_LOCK_WAIT_DEFAULT ;; esac + probe=$(mktemp "${lock%/*}/.lock-probe.XXXXXX") || return 1 + rm -f "$probe" || return 1 + if [ ! -e "$lock" ] && [ ! -L "$lock" ]; then + fm_lock_try_create "$lock" && return 0 + [ -e "$lock" ] || [ -L "$lock" ] || return 1 + fi + deadline=$(( $(date +%s) + wait )) + while ! fm_lock_try_acquire "$lock"; do + [ "$(date +%s)" -lt "$deadline" ] || return 1 + sleep 0.1 + done +} + +# Write one record into the next sequence slot: temp-write, then atomic +# rename. Prints the record path. Caller must hold .seq.lock. +_fm_task_inbox_write_record_locked() { # + local dir=$1 text=$2 seq tmp rec status=0 + seq=$(fm_task_inbox_next_seq "$dir") + rec="$dir/$seq.msg" + tmp=$(mktemp "$dir/.staging.XXXXXX") || return 1 + { + printf 'schema=%s\n' "$FM_TASK_INBOX_SCHEMA" + printf 'at=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" + printf -- '--\n' + printf '%s' "$text" + } > "$tmp" && mv "$tmp" "$rec" || status=1 + [ "$status" -eq 0 ] || { rm -f "$tmp"; return 1; } + printf '%s' "$rec" +} + +# Durably enqueue one steer: temp-write, then atomic rename into the next +# sequence slot. Prints the record path. Fails without a partial record. +fm_task_inbox_write() { # + local state=$1 task=$2 text=$3 dir lock rec status=0 + dir=$(fm_task_inbox_dir "$state" "$task") + mkdir -p "$dir/handled" || return 1 + lock="$dir/.seq.lock" + fm_task_inbox_lock_acquire "$lock" || return 1 + rec=$(_fm_task_inbox_write_record_locked "$dir" "$text") || status=1 + fm_lock_release "$lock" + [ "$status" -eq 0 ] || return 1 + printf '%s' "$rec" +} + +# Durably enqueue one steer at most once: when a record with the exact same +# body already exists - unhandled or already acknowledged in handled/ - no new +# record is written and the existing record's path is printed instead. +# This is the enqueue primitive for a transport that can fail with completion +# unknown (the remote steer leg over ssh): the caller's safe recovery is to run +# the same enqueue again, and this dedup is what makes the re-run land on the +# same record instead of a duplicate the worker would act on twice. Two +# distinct logical requests never collapse in practice because a marked +# secondmate request embeds a per-request correlation token in its body. The +# local plane keeps plain fm_task_inbox_write: its outcome is synchronous, so +# a repeated identical local steer is a deliberate new instruction. +fm_task_inbox_write_idempotent() { # + local state=$1 task=$2 text=$3 dir lock want have f rec='' status=0 + dir=$(fm_task_inbox_dir "$state" "$task") + mkdir -p "$dir/handled" || return 1 + lock="$dir/.seq.lock" + fm_task_inbox_lock_acquire "$lock" || return 1 + if want=$(mktemp "$dir/.dedup.XXXXXX") && have=$(mktemp "$dir/.dedup.XXXXXX"); then + if printf '%s' "$text" > "$want"; then + for f in "$dir"/*.msg "$dir/handled"/*.msg; do + if [ ! -e "$f" ]; then + case "$f" in + "$dir"/*.msg) + f="$dir/handled/${f##*/}" + [ -e "$f" ] || continue + ;; + *) continue ;; + esac + fi + if ! fm_task_inbox_body "$f" > "$have" 2>/dev/null; then + case "$f" in + "$dir"/*.msg) + f="$dir/handled/${f##*/}" + fm_task_inbox_body "$f" > "$have" 2>/dev/null || continue + ;; + *) continue ;; + esac + fi + cmp -s "$want" "$have" || continue + [ ! -e "$dir/handled/${f##*/}" ] || f="$dir/handled/${f##*/}" + rec=$f + break + done + else + status=1 + fi + rm -f "$want" "$have" + else + rm -f "${want:-}" 2>/dev/null || true + status=1 + fi + if [ "$status" -eq 0 ] && [ -z "$rec" ]; then + rec=$(_fm_task_inbox_write_record_locked "$dir" "$text") || status=1 + fi + fm_lock_release "$lock" + [ "$status" -eq 0 ] || return 1 + printf '%s' "$rec" +} + +# The exact enqueued text back out of a record. +fm_task_inbox_body() { # + local line + [ -f "$1" ] || return 1 + while IFS= read -r line; do + if [ "$line" = -- ]; then + cat + return 0 + fi + done < "$1" + return 1 +} + +# The constant self-describing doorbell line for the inbox containing a record. +# Self-describing on purpose: a worker whose brief predates the inbox contract +# still receives the complete instruction in the line itself. +fm_task_inbox_doorbell_line() { # + local dir=${1%/*} abs + abs=$(cd "$dir" 2>/dev/null && pwd) || abs=$dir + printf 'Firstmate instruction waiting: list %s/*.msg and, in numeric order, read and act on each, then mv each handled file to %s/handled/.' \ + "$abs" "$abs" +} + +# Ring the doorbell, best-effort: one advisory composer pre-check, then the +# backend's submit machinery with a minimal retry budget, verdict discarded. +# Returns 0 rang, 1 skipped because the composer PROVENLY holds pending text +# (the watcher re-rings later), 2 the backend send failed. No return value is +# delivery proof; the acknowledgement move is the only delivery signal. +# The skip is deliberately narrow: only an exact `pending` verdict defers, +# because there our Enter could submit someone's real half-typed content. +# `pending-unproven` and `unknown` still ring - the worst outcome is a garbled +# CONSTANT line the worker recovers semantically, while skipping on ambiguous +# verdicts would starve a harness whose idle screen the classifier cannot +# positively identify (that classifier is advisory here by design). +fm_task_inbox_ring() { # [expected-label] + local backend=$1 target=$2 rec=$3 label=${4:-} line cstate verdict + line=$(fm_task_inbox_doorbell_line "$rec") + cstate=$(fm_backend_composer_state "$backend" "$target" "$label" 2>/dev/null) || cstate=unknown + case "$cstate" in + pending) return 1 ;; + esac + if ! verdict=$(fm_backend_send_text_submit "$backend" "$target" "$line" 1 0.4 0.3 "$label" 2>/dev/null); then + return 2 + fi + # The verdict is read only to report a failed keystroke; every other value + # (empty, pending, unknown, ...) is deliberately ignored, never proof. + [ "$verdict" != send-failed ] || return 2 + return 0 +} + +# Oldest unhandled record by sequence, or fail when the inbox is empty. +fm_task_inbox_oldest_unhandled() { # + local dir best='' best_n=0 f n + dir=$(fm_task_inbox_dir "$1" "$2") + for f in "$dir"/*.msg; do + [ -e "$f" ] || continue + n=$(fm_task_inbox_seq_of "${f##*/}") || continue + if [ -z "$best" ] || [ "$n" -lt "$best_n" ]; then + best=$f + best_n=$n + fi + done + [ -n "$best" ] || return 1 + printf '%s' "$best" +} + +# The re-ring ladder decision for one task. Prints exactly one of: +# quiet nothing due (healthy, within grace or spacing, +# or already escalated for the current oldest) +# ring one doorbell re-ring is due +# escalate attempt budget spent; surface as stale +# An empty inbox also resets the ladder bookkeeping so the next message starts +# a fresh ladder. +fm_task_inbox_due_action() { # + local dir oldest base now grace max ladder rec_base count last + dir=$(fm_task_inbox_dir "$1" "$2") + if ! oldest=$(fm_task_inbox_oldest_unhandled "$1" "$2"); then + rm -f "$dir/.ring-state" "$dir/.escalated" 2>/dev/null || true + printf 'quiet' + return 0 + fi + base=${oldest##*/} + grace=$(fm_task_inbox_grace_secs) + if [ "$(fm_path_age "$oldest")" -lt "$grace" ]; then + printf 'quiet' + return 0 + fi + count=0 + last=0 + ladder=$(cat "$dir/.ring-state" 2>/dev/null || true) + IFS=$(printf '\t') read -r rec_base count last </dev/null || true + fi + case "$count" in ''|*[!0-9]*) count=0 ;; esac + case "$last" in ''|*[!0-9]*) last=0 ;; esac + if [ "$(cat "$dir/.escalated" 2>/dev/null || true)" = "$base" ]; then + printf 'quiet' + return 0 + fi + max=$(fm_task_inbox_ring_max) + if [ "$count" -ge "$max" ]; then + printf 'escalate %s %s' "$oldest" "$count" + return 0 + fi + now=$(date +%s) + if [ "$((now - last))" -lt "$grace" ]; then + printf 'quiet' + return 0 + fi + printf 'ring %s' "$oldest" +} + +# Advance the ladder after a delivery attempt. A failed ring or a composer- +# protected skip still consumes budget so neither a dead pane nor permanently +# blocked composer can retry silently forever. A concurrently removed inbox is +# a successful no-op; otherwise failure means the caller must surface the +# unwritable ladder while the record remains unhandled. +fm_task_inbox_record_ring() { # + local dir base ladder rec_base count last + dir=$(fm_task_inbox_dir "$1" "$2") + base=${3##*/} + count=0 + ladder=$(cat "$dir/.ring-state" 2>/dev/null || true) + IFS=$(printf '\t') read -r rec_base count last < "$dir/.ring-state"; } 2>/dev/null; then + [ -d "$dir" ] || return 0 + return 1 + fi +} + +# Mark the current oldest as escalated after its stale wake is durably queued, +# suppressing another wake on later polls. Wake-before-marker ordering favors +# at-least-once recovery: a crash or marker failure can cause a rare duplicate; +# stuck-crewmate-recovery owns the message from here. +fm_task_inbox_record_escalated() { # + local dir + dir=$(fm_task_inbox_dir "$1" "$2") + [ -d "$dir" ] || return 0 + if ! { printf '%s\n' "${3##*/}" > "$dir/.escalated"; } 2>/dev/null; then + [ -d "$dir" ] || return 0 + return 1 + fi +} diff --git a/bin/fm-teardown.sh b/bin/fm-teardown.sh index c84669d3632..595c73e6a13 100755 --- a/bin/fm-teardown.sh +++ b/bin/fm-teardown.sh @@ -50,8 +50,12 @@ # is the approved discard path that prevalidates child removal targets, locks each # descendant home's task set before enumeration, and holds those locks through # child cleanup. Contention refuses the complete forced teardown before child -# mutation. It then discards child work, kills child runtime endpoints, and removes -# the retired home. Removing a leased home releases its durable treehouse lease so the pool slot is freed, +# mutation. Local and remote retirement serialize their destructive phase with +# that mate's backlog-handoff lock under the registry lock. Pending handoff wake +# state is retired with the home, and local removal failure restores that state +# before preserving the route for retry. Teardown then discards child work, kills +# child runtime endpoints, and removes the retired home. Removing a leased home +# releases its durable treehouse lease so the pool slot is freed, # never left leased forever. If the treehouse return fails, teardown leaves the # leased home and state in place instead of hiding a still-held lease. # Usage: fm-teardown.sh [--force] @@ -166,6 +170,8 @@ SUB_HOME_PARENT_MARKER=".fm-secondmate-parent" . "$SCRIPT_DIR/fm-secondmate-parent-lib.sh" # shellcheck source=bin/fm-wake-lib.sh . "$SCRIPT_DIR/fm-wake-lib.sh" +# shellcheck source=bin/fm-pending-reply-lib.sh +. "$SCRIPT_DIR/fm-pending-reply-lib.sh" # shellcheck source=bin/fm-nm-run-lib.sh . "$SCRIPT_DIR/fm-nm-run-lib.sh" if [ "$#" -lt 1 ] || ! fm_task_id_path_safe "$1"; then @@ -194,6 +200,18 @@ teardown_release_locks() { fm_lock_release "${DESCENDANT_LOCK_PATHS[$i]}" || true done DESCENDANT_LOCK_PATHS=() + if [ -n "${HANDOFF_WAKE_RETIRE_LOCK:-}" ]; then + fm_lock_release "$HANDOFF_WAKE_RETIRE_LOCK" || true + HANDOFF_WAKE_RETIRE_LOCK= + fi + if [ -n "${LOCAL_HANDOFF_LOCK:-}" ]; then + fm_lock_release "$LOCAL_HANDOFF_LOCK" || true + LOCAL_HANDOFF_LOCK= + fi + if [ -n "${LOCAL_REGISTRY_LOCK:-}" ]; then + fm_lock_release "$LOCAL_REGISTRY_LOCK" || true + LOCAL_REGISTRY_LOCK= + fi if [ "$META_LOCK_HELD" = 1 ]; then fm_lock_release "$META_LOCK" || true META_LOCK_HELD=0 @@ -230,6 +248,208 @@ REMOTE_PENDING_DIR_REAL= REMOTE_HANDOFF_LOCK= REMOTE_REGISTRY_LOCK= REMOTE_REPLY_LIFECYCLE_LOCK= +LOCAL_HANDOFF_LOCK= +LOCAL_REGISTRY_LOCK= +HANDOFF_WAKE_RETIRE_MARKER= +HANDOFF_WAKE_RETIRE_VALUE= +HANDOFF_WAKE_RETIRE_CORR= +HANDOFF_WAKE_RETIRE_LOCK= +HANDOFF_WAKE_RETIRE_STAGE= + +handoff_wake_retire_validate() { + local marker="$STATE/.backlog-handoff-$ID.wake-pending" value corr rec confirmation + HANDOFF_WAKE_RETIRE_MARKER= + HANDOFF_WAKE_RETIRE_VALUE= + HANDOFF_WAKE_RETIRE_CORR= + [ -e "$marker" ] || [ -L "$marker" ] || return 0 + [ -f "$marker" ] && [ ! -L "$marker" ] || { + echo "REFUSED: receiver wake state for secondmate $ID is unsafe" >&2 + return 1 + } + value=$(cat "$marker" 2>/dev/null || true) + case "$value" in + pending|confirmed) ;; + prepared:*) + corr=${value#prepared:} + corr=${corr%%:*} + printf '%s' "$value" | grep -Eq '^prepared:[a-f0-9]{16}:[a-f0-9]{16}$' || { + echo "REFUSED: receiver wake state for secondmate $ID is invalid" >&2 + return 1 + } + ;; + pending:*|confirmed:*) + corr=${value#*:} + printf '%s' "$corr" | grep -Eq '^[a-f0-9]{16}$' || { + echo "REFUSED: receiver wake state for secondmate $ID is invalid" >&2 + return 1 + } + ;; + *) + echo "REFUSED: receiver wake state for secondmate $ID is invalid" >&2 + return 1 + ;; + esac + if [ -n "$corr" ]; then + rec=$(fm_pending_reply_path "$STATE" "$corr") + if [ -e "$rec" ] || [ -L "$rec" ]; then + [ -f "$rec" ] && [ ! -L "$rec" ] \ + && [ "$(fm_pending_reply_get "$rec" task_id)" = "$ID" ] || { + echo "REFUSED: receiver wake correlation for secondmate $ID is unsafe or belongs to another task" >&2 + return 1 + } + fi + confirmation=$(fm_pending_reply_delivery_confirmation_path "$STATE" "$corr") + if [ -e "$confirmation" ] || [ -L "$confirmation" ]; then + [ -f "$confirmation" ] && [ ! -L "$confirmation" ] || { + echo "REFUSED: receiver wake delivery state for secondmate $ID is unsafe" >&2 + return 1 + } + fi + HANDOFF_WAKE_RETIRE_CORR=$corr + fi + HANDOFF_WAKE_RETIRE_MARKER=$marker + HANDOFF_WAKE_RETIRE_VALUE=$value +} + +handoff_wake_retire() { + local marker=$HANDOFF_WAKE_RETIRE_MARKER corr=$HANDOFF_WAKE_RETIRE_CORR lock rec confirmation rc=0 + [ -n "$marker" ] || return 0 + [ -f "$marker" ] && [ ! -L "$marker" ] \ + && [ "$(cat "$marker" 2>/dev/null || true)" = "$HANDOFF_WAKE_RETIRE_VALUE" ] || return 1 + if [ -n "$corr" ]; then + lock="$STATE/.pending-reply-$corr.lock" + fm_lock_acquire_wait "$lock" || return 1 + rec=$(fm_pending_reply_path "$STATE" "$corr") + confirmation=$(fm_pending_reply_delivery_confirmation_path "$STATE" "$corr") + if { [ ! -e "$rec" ] && [ ! -L "$rec" ]; } \ + || { [ -f "$rec" ] && [ ! -L "$rec" ] \ + && [ "$(fm_pending_reply_get "$rec" task_id)" = "$ID" ]; }; then + rm -f -- "$confirmation" "$rec" "$marker" || rc=$? + else + rc=1 + fi + fm_lock_release "$lock" + return "$rc" + fi + rm -f -- "$marker" +} + +handoff_wake_retire_stage_restore() { + local stage=$HANDOFF_WAKE_RETIRE_STAGE marker rec confirmation name destination + [ -n "$stage" ] || return 0 + marker="$STATE/.backlog-handoff-$ID.wake-pending" + rec= + confirmation= + if [ -n "$HANDOFF_WAKE_RETIRE_CORR" ]; then + rec=$(fm_pending_reply_path "$STATE" "$HANDOFF_WAKE_RETIRE_CORR") + confirmation=$(fm_pending_reply_delivery_confirmation_path "$STATE" "$HANDOFF_WAKE_RETIRE_CORR") + fi + for name in record confirmation marker; do + [ -e "$stage/$name" ] || continue + case "$name" in + record) destination=$rec ;; + confirmation) destination=$confirmation ;; + marker) destination=$marker ;; + esac + [ -n "$destination" ] && [ ! -e "$destination" ] && [ ! -L "$destination" ] \ + && mv -- "$stage/$name" "$destination" || return 1 + done + rm -f -- "$stage/corr" || return 1 + rmdir -- "$stage" || return 1 + if [ -n "$HANDOFF_WAKE_RETIRE_LOCK" ]; then + fm_lock_release "$HANDOFF_WAKE_RETIRE_LOCK" || return 1 + HANDOFF_WAKE_RETIRE_LOCK= + fi + HANDOFF_WAKE_RETIRE_STAGE= +} + +handoff_wake_retire_stage_commit() { + local stage=$HANDOFF_WAKE_RETIRE_STAGE retired + [ -n "$stage" ] || return 0 + retired="$stage.retired.$$" + [ ! -e "$retired" ] && [ ! -L "$retired" ] || return 1 + mv -- "$stage" "$retired" || return 1 + HANDOFF_WAKE_RETIRE_STAGE= + if [ -n "$HANDOFF_WAKE_RETIRE_LOCK" ]; then + fm_lock_release "$HANDOFF_WAKE_RETIRE_LOCK" || return 1 + HANDOFF_WAKE_RETIRE_LOCK= + fi + rm -rf -- "$retired" || echo "warning: retired receiver wake state remains at $retired" >&2 +} + +handoff_wake_retire_stage_recover() { + local home=$1 stage="$STATE/.backlog-handoff-$ID.wake-retiring" corr + [ -e "$stage" ] || [ -L "$stage" ] || return 0 + [ -d "$stage" ] && [ ! -L "$stage" ] || { + echo "REFUSED: receiver wake retirement state for secondmate $ID is unsafe" >&2 + return 1 + } + if [ ! -e "$stage/corr" ] && [ ! -L "$stage/corr" ]; then + rmdir -- "$stage" 2>/dev/null && return 0 + echo "REFUSED: receiver wake retirement state for secondmate $ID is incomplete" >&2 + return 1 + fi + [ -f "$stage/corr" ] && [ ! -L "$stage/corr" ] || { + echo "REFUSED: receiver wake retirement state for secondmate $ID is unsafe" >&2 + return 1 + } + corr=$(cat "$stage/corr" 2>/dev/null || true) + [ -z "$corr" ] || printf '%s' "$corr" | grep -Eq '^[a-f0-9]{16}$' || { + echo "REFUSED: receiver wake retirement correlation for secondmate $ID is invalid" >&2 + return 1 + } + local staged + for staged in "$stage/marker" "$stage/record" "$stage/confirmation"; do + [ ! -e "$staged" ] && [ ! -L "$staged" ] && continue + [ -f "$staged" ] && [ ! -L "$staged" ] || { + echo "REFUSED: receiver wake retirement state for secondmate $ID is unsafe" >&2 + return 1 + } + done + HANDOFF_WAKE_RETIRE_CORR=$corr + HANDOFF_WAKE_RETIRE_STAGE=$stage + if [ -n "$corr" ]; then + HANDOFF_WAKE_RETIRE_LOCK="$STATE/.pending-reply-$corr.lock" + fm_lock_acquire_wait "$HANDOFF_WAKE_RETIRE_LOCK" || return 1 + fi + if [ -e "$home" ] || [ -L "$home" ]; then + handoff_wake_retire_stage_restore + else + handoff_wake_retire_stage_commit + fi +} + +handoff_wake_retire_stage() { + local stage="$STATE/.backlog-handoff-$ID.wake-retiring" marker=$HANDOFF_WAKE_RETIRE_MARKER + local corr=$HANDOFF_WAKE_RETIRE_CORR rec confirmation + [ -n "$marker" ] || return 0 + [ ! -e "$stage" ] && [ ! -L "$stage" ] || return 1 + (umask 077; mkdir -- "$stage") || return 1 + HANDOFF_WAKE_RETIRE_STAGE=$stage + printf '%s\n' "$corr" > "$stage/corr" || { handoff_wake_retire_stage_restore || true; return 1; } + if [ -n "$corr" ]; then + HANDOFF_WAKE_RETIRE_LOCK="$STATE/.pending-reply-$corr.lock" + fm_lock_acquire_wait "$HANDOFF_WAKE_RETIRE_LOCK" || { + HANDOFF_WAKE_RETIRE_LOCK= + handoff_wake_retire_stage_restore || true + return 1 + } + rec=$(fm_pending_reply_path "$STATE" "$corr") + confirmation=$(fm_pending_reply_delivery_confirmation_path "$STATE" "$corr") + if [ -e "$rec" ] && ! mv -- "$rec" "$stage/record"; then + handoff_wake_retire_stage_restore || true + return 1 + fi + if [ -e "$confirmation" ] && ! mv -- "$confirmation" "$stage/confirmation"; then + handoff_wake_retire_stage_restore || true + return 1 + fi + fi + if ! mv -- "$marker" "$stage/marker"; then + handoff_wake_retire_stage_restore || true + return 1 + fi +} remote_teardown_locks_release() { if [ -n "$REMOTE_REPLY_LIFECYCLE_LOCK" ]; then @@ -342,6 +562,7 @@ remote_secondmate_teardown() { [ "$route_host" = "$remote_host" ] && [ "$route_root" = "$remote_root" ] && [ "$route_home" = "$remote_home" ] \ || { echo "REFUSED: remote secondmate metadata does not match its registry route" >&2; return 1; } [ -z "$FORCE" ] || [ "$FORCE" = --force ] || { echo "error: invalid teardown option: $FORCE" >&2; return 2; } + handoff_wake_retire_validate || return 1 remote_recovery_paths_validate initial || return 1 if [ "$FORCE" != --force ] && [ "$REMOTE_OUTBOX_PRESENT" -eq 1 ]; then echo "REFUSED: remote secondmate $ID still has a pending backlog outbox; deliver it or explicitly discard with --force" >&2 @@ -391,6 +612,8 @@ remote_secondmate_teardown() { fi remote_pending_replies_cleanup \ || { echo "error: remote pending-reply cleanup failed; preserving the local route for retry" >&2; return 1; } + handoff_wake_retire \ + || { echo "error: remote receiver wake cleanup failed; preserving the local route for retry" >&2; return 1; } tmp="$SECONDMATE_REG.tmp.$$" grep -vE "^- $ID( |$)" "$SECONDMATE_REG" > "$tmp" || true mv -f -- "$tmp" "$SECONDMATE_REG" @@ -2267,21 +2490,30 @@ cleanup_firstmate_home_children() { } remove_secondmate_registry_entry() { - local id=$1 tmp lock rc=0 + local id=$1 tmp lock rc=0 acquired=0 [ -f "$SECONDMATE_REG" ] || return 0 lock=$(secondmate_registry_lock_path "$STATE") - fm_lock_acquire_wait "$lock" || return 1 + if [ "$LOCAL_REGISTRY_LOCK" != "$lock" ]; then + fm_lock_acquire_wait "$lock" || return 1 + acquired=1 + fi tmp="$SECONDMATE_REG.tmp.$$" grep -vE "^- $id( |$)" "$SECONDMATE_REG" > "$tmp" || true mv "$tmp" "$SECONDMATE_REG" || rc=$? - fm_lock_release "$lock" + [ "$acquired" -eq 0 ] || fm_lock_release "$lock" return "$rc" } validate_pr_poll_cleanup "$STATE" "$ID" || exit 1 if [ "$KIND" = secondmate ]; then + LOCAL_REGISTRY_LOCK=$(secondmate_registry_lock_path "$STATE") + fm_lock_acquire_wait "$LOCAL_REGISTRY_LOCK" || exit 1 + LOCAL_HANDOFF_LOCK="$STATE/.backlog-handoff-$ID.lock" + fm_lock_acquire_wait "$LOCAL_HANDOFF_LOCK" || exit 1 [ -n "$HOME_PATH" ] || HOME_PATH=$WT + handoff_wake_retire_stage_recover "$HOME_PATH" || exit 1 + handoff_wake_retire_validate || exit 1 validate_firstmate_home_for_removal "$HOME_PATH" "secondmate home" "$ID" >/dev/null || exit 1 if [ "$FORCE" = "--force" ]; then validate_firstmate_home_children_removal "$HOME_PATH" || exit 1 @@ -2542,7 +2774,18 @@ if [ "$BACKEND" = herdr ]; then fi if [ "$KIND" = secondmate ]; then [ -n "$HOME_PATH" ] || HOME_PATH=$WT - remove_firstmate_home "$HOME_PATH" "secondmate home" "$ID" || exit $? + handoff_wake_retire_stage \ + || { echo "error: receiver wake cleanup could not be staged; preserving the secondmate home and route" >&2; exit 1; } + if remove_firstmate_home "$HOME_PATH" "secondmate home" "$ID"; then + : + else + rc=$? + handoff_wake_retire_stage_restore \ + || echo "error: receiver wake restoration failed; recovery state remains at $HANDOFF_WAKE_RETIRE_STAGE" >&2 + exit "$rc" + fi + handoff_wake_retire_stage_commit \ + || { echo "error: receiver wake cleanup failed; preserving the secondmate route for retry" >&2; exit 1; } remove_secondmate_registry_entry "$ID" fi remove_grok_turnend_auth "$STATE" "$ID" || exit 1 @@ -2560,6 +2803,10 @@ rm -f "$STATE/$ID.turn-ended" "$STATE/$ID.meta" \ "$STATE/$ID.muse-session-current" "$STATE/$ID.cursor-session" \ "$STATE/$ID.control-relaunch" "$STATE/$ID.control-relaunch.meta-prior" \ "$STATE/$ID.control-relaunch.brief-prior" "$STATE/$ID.control-relaunch.note" +# The steering inbox (bin/fm-task-inbox-lib.sh) is runtime state for the +# retired endpoint; teardown only runs after landing is confirmed, so any +# leftover unhandled steer here is moot rather than unlanded work. +rm -rf "$STATE/$ID.inbox" fm_lock_release "$META_LOCK" META_LOCK_HELD=0 if [ "$KIND" != scout ] && [ "$KIND" != secondmate ] && [ "$MODE" != local-only ]; then diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh index 3a9e1d7ae9f..d8739f206e9 100755 --- a/bin/fm-test-run.sh +++ b/bin/fm-test-run.sh @@ -157,7 +157,7 @@ family_for_basename() { fm-wake-drain-unread-status.test.sh|\ fm-tool-update-check.test.sh|\ fm-wake-queue.test.sh|fm-watch-arm.test.sh|fm-watch-checkpoint.test.sh|fm-watch-recovery-loop.test.sh|\ - fm-watch-triage.test.sh|\ + fm-watch-triage.test.sh|fm-task-inbox.test.sh|\ fm-watcher-lock.test.sh|fm-inactive-reconcile.test.sh) printf '%s\n' watcher-wake-lock ;; @@ -196,13 +196,15 @@ family_for_basename() { fm-opencode-primary-live-e2e.test.sh|fm-pi-primary-live-e2e.test.sh|\ fm-sessionstart-hook-live-e2e.test.sh|fm-sessionstart-instruction-refresh-live-e2e.test.sh|\ fm-quota-array-dispatch-live-e2e.test.sh|fm-send-secondmate-marker-herdr-e2e.test.sh|\ + fm-send-inbox-doorbell-live-e2e.test.sh|\ fm-herdr-submit-confirm-live-e2e.test.sh) printf '%s\n' live-harness-optin ;; fm-backend-herdr.test.sh|fm-backend-tmux-smoke.test.sh|fm-backend.test.sh|\ fm-tmux-agent-liveness.test.sh|\ fm-control.test.sh|fm-control-relaunch.test.sh|\ - fm-herdr-session-cleanup.test.sh|fm-send-resolve-key.test.sh|fm-send-strict.test.sh|fm-spawn-batch.test.sh|\ + fm-herdr-session-cleanup.test.sh|fm-send-resolve-key.test.sh|fm-send-strict.test.sh|\ + fm-send-inbox.test.sh|fm-spawn-batch.test.sh|\ fm-spawn-dispatch-profile.test.sh|\ fm-trace-context-spawn.test.sh|fm-spawn-worktree-settle.test.sh|\ fm-teardown-endpoint-safety.test.sh) @@ -1003,6 +1005,14 @@ families_for_changed_path() { printf '%s\n' backend-dispatch printf '%s\n' pure-contract-unit ;; + bin/fm-task-inbox-lib.sh) + # The steering-inbox record/doorbell/ladder owner: fm-send's data plane + # (backend-dispatch), the watcher's re-ring check (watcher-wake-lock), + # and the live doorbell guard against real harnesses. + printf '%s\n' backend-dispatch + printf '%s\n' watcher-wake-lock + printf '%s\n' live-harness-optin + ;; bin/fm-bearings-snapshot.sh|bin/fm-fleet-snapshot.sh|bin/fm-fleet-view.sh) printf '%s\n' snapshot-bearings ;; diff --git a/bin/fm-wake-drain.sh b/bin/fm-wake-drain.sh index 203765be80f..14599aaf8da 100755 --- a/bin/fm-wake-drain.sh +++ b/bin/fm-wake-drain.sh @@ -298,6 +298,10 @@ if [ -n "$ACK_THROUGH" ]; then awk -F '\t' -v cutoff="$ACK_THROUGH" ' NF < 5 || $2 !~ /^[0-9]+$/ || $2 > cutoff { print } ' "$FM_WAKE_QUEUE" > "$DRAIN_TMP" || exit 1 + fm_wake_commit_secondmate_stall_receipts_through "$ACK_THROUGH" || { + echo "wake drain: secondmate stall receipt could not be recorded safely" >&2 + exit 1 + } if [ ! -s "$DRAIN_TMP" ]; then fm_recovery_marker_ack "$RECOVERY_MARKER" "$ACK_GENERATION" RECOVERY_ACK_STATUS=$? diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index 28249b661f3..8ce2195ac8d 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -1176,6 +1176,69 @@ fm_wake_queued_keys_locked() { "$FM_WAKE_QUEUE" 2>/dev/null || true } +fm_wake_secondmate_stall_marker_write() { # + local task=$1 row_key=$2 marker tmp + case "$task" in ''|*[!A-Za-z0-9._-]*) return 1 ;; esac + case "$row_key" in ''|*[!0-9-]*) return 1 ;; esac + marker="$STATE/.secondmate-wake-stall-$task" + if [ -e "$marker" ] || [ -L "$marker" ]; then + [ -f "$marker" ] && [ ! -L "$marker" ] || return 1 + fi + tmp=$(mktemp "$STATE/.secondmate-wake-stall.XXXXXX") || return 1 + if ! printf '%s\n' "$row_key" > "$tmp" || ! chmod 0600 "$tmp" \ + || ! _fm_atomic_replace "$tmp" "$marker"; then + rm -f -- "$tmp" + return 1 + fi +} + +fm_wake_secondmate_stall_receipt_write() { # + local task=$1 row_key=$2 root task_dir receipt tmp + case "$task" in ''|*[!A-Za-z0-9._-]*) return 1 ;; esac + case "$row_key" in ''|*[!0-9-]*) return 1 ;; esac + root="$STATE/.secondmate-wake-stall-receipts" + task_dir="$root/$task" + if [ -e "$root" ] || [ -L "$root" ]; then + [ -d "$root" ] && [ ! -L "$root" ] || return 1 + else + mkdir "$root" || return 1 + chmod 0700 "$root" || return 1 + fi + if [ -e "$task_dir" ] || [ -L "$task_dir" ]; then + [ -d "$task_dir" ] && [ ! -L "$task_dir" ] || return 1 + else + mkdir "$task_dir" || return 1 + chmod 0700 "$task_dir" || return 1 + fi + receipt="$task_dir/$row_key" + [ "$(cat "$receipt" 2>/dev/null || true)" != "$row_key" ] || return 0 + tmp=$(mktemp "$task_dir/.receipt.XXXXXX") || return 1 + if ! printf '%s\n' "$row_key" > "$tmp" || ! chmod 0600 "$tmp" \ + || ! _fm_atomic_replace "$tmp" "$receipt"; then + rm -f -- "$tmp" + return 1 + fi +} + +fm_wake_commit_secondmate_stall_receipts_through() { # + local cutoff=$1 key seq rest epoch task row_key + while IFS= read -r key; do + seq=${key##*-} + rest=${key%-*} + epoch=${rest##*-} + task=${rest#secondmate-wake-loop-} + task=${task%-"$epoch"} + case "$seq" in ''|*[!0-9]*) return 1 ;; esac + case "$epoch" in ''|*[!0-9]*) return 1 ;; esac + case "$task" in ''|*[!A-Za-z0-9._-]*) return 1 ;; esac + row_key="$epoch-$seq" + fm_wake_secondmate_stall_receipt_write "$task" "$row_key" || return 1 + done < <(awk -F '\t' -v cutoff="$cutoff" ' + NF >= 5 && $2 ~ /^[0-9]+$/ && $2 <= cutoff && $3 == "check" \ + && $4 ~ /^secondmate-wake-loop-[A-Za-z0-9._-]+-[0-9]+-[0-9]+$/ { print $4 } + ' "$FM_WAKE_QUEUE" 2>/dev/null) +} + fm_wake_restore_queue() { local drained=$1 restore restore="$STATE/.wake-queue.restore.$(fm_current_pid)" diff --git a/bin/fm-watch.sh b/bin/fm-watch.sh index 1e56b8631c5..e4e20fe8013 100755 --- a/bin/fm-watch.sh +++ b/bin/fm-watch.sh @@ -58,6 +58,13 @@ # exempt from the working deferral above: the busy signal # a re-verification would read is the very one # BUSY_TURN_MAX_SECS stopped trusting. +# stale: (unread firstmate instruction: ...) +# the steering-inbox ladder spent its delivery-attempt +# budget on an idle pane without an acknowledgement +# stale: (steering-inbox ladder bookkeeping unwritable: ...) +# an unhandled record's ladder cannot advance; quiet +# successful attempts never wake firstmate +# (bin/fm-task-inbox-lib.sh owns the ladder policy) # check: