feat(bin): deliver remote steers through durable inboxes

[kunchenguid]

Intent

STAGE 2 (remote leg) of the inbox+doorbell steer channel, following merged stage 1 (PR #2856), captain-authorized. Two required parts. Part 1: put REMOTE secondmate steers on the durable-inbox + doorbell model - a remote steer writes the message as a durable sequenced inbox record into the REMOTE home's task inbox over the existing fm-on.sh transport (the host-local leg fm-remote-secondmate-control.sh cmd_send now writes to the remote parent-route steering inbox via a new idempotent enqueue in fm-task-inbox-lib.sh), then rings the REMOTE doorbell (the one short constant self-describing line), idempotently re-runnable: re-running the identical request lands on the same record (exact-body dedup against both unhandled and handled records, safe because a marked request's body carries a unique correlation token) and re-rings, both safe. Exit 0 when the remote record is durably written; nonzero only for a real transport/write failure. The from-firstmate marker and correlation id travel verbatim inside the record body (reusing the existing marker and pending-reply owners); the pending-reply REPLY-direction machinery stays as-is. Deliberate design decisions: fm-send retries the identical remote leg exactly once on ssh exit 255 (safe by idempotence); after a second 255 it exits nonzero with resend-safe guidance and PRESERVES a marked request's reply expectation as delivery_unknown because the record may have landed (deleting the do-not-resend TRAP, not the expectation preservation the reply-direction owner documents); every remote text including a leading '/' or '$' rides the inbox, because remote sends are marked and a marked request already reaches the harness as marker-prefixed chat rather than a parser command, so no remote typed text plane remains and only --key still crosses as a keystroke; the remote host runs no doorbell re-ring ladder - loss detection is owned by the parent's pending-reply recovery/escalation, whose recovery request re-rings when enqueued; --resolve-key closes at enqueue for remote too, identically to local; the remote leg skips the ring with a notice when dedup lands on an already-acknowledged handled record. Part 2 (MANDATORY cleanup, per the captain's accounting in data/fm-inbox-doorbell-design-board-s1/report.md section 4): DELETE the dead remote typed-payload transport that #2856 deliberately left behind - the fm-send.sh parent remote send via fm-on cmd_send with the typed message payload plus the exit-3-to-delivered remap (REMOTE_DELIVERY_NOTICE), the remote ssh-255 fm_pending_reply_mark_delivery_unknown 'do not resend' typed-hop handling, the REMOTE_DELIVERY_NOTICE stderr notice, the header paragraphs defining remote delivery as verified typed submit / exit 3 / 255-do-not-resend, and fm-remote-secondmate-control.sh cmd_send's inner payload-typing fm-send at an explicit pane target; rewrite tests/fm-send-remote-delivery.test.sh around remote inbox+doorbell asserting the deleted behaviors (exit-3-as-delivered, 255 do-not-resend for a task steer) are GONE without weakening assertions; update leftover remote typed-submit wording (found in AGENTS.md section 7 and docs/remote-secondmates.md; docs/configuration.md, docs/scripts.md, and docs/verification/runtime-backends.md were audited and carry no remote typed leftovers). MUST STAY (explicitly out of scope to delete): fm_task_inbox_ring, fm_backend_send_text_submit, fm_backend_composer_state, fm_tmux_submit_core, fm_tmux_submit_enter_core, fm_composer_classify_screen (doorbell + spawn + daemon + control consumers); fm-send.sh local slash / Codex '$' / --key / explicit-backend typed branches and their local exit-3 contract; cmd_key; inbox close-at-enqueue for --resolve-key. Regression tests required: a remote steer writes the record into the remote home's inbox and rings the remote doorbell with no payload typed across ssh; re-running is idempotent (same record, no duplicate); --resolve-key closes at enqueue for remote; the deleted typed-payload behaviors are gone. The e2e lifecycle test's failed-reread-nudge scenario now induces failure via an unwritable remote inbox because a swallowed doorbell alone no longer fails a durably recorded steer. This is firstmate shared tracked transport core: repo style is one sentence per line in Markdown, plain dash, shellcheck-clean bin scripts, colocated tests asserting executable behavior. The PR should note this removes the last typed-payload steer transport and completes the inbox+doorbell migration. One PR; do not merge (yolo off, captain merges).

What Changed

• Route all remote secondmate text through durable task inbox records with best-effort doorbells and enqueue-time decision closure.
• Add exact-body enqueue deduplication, handled-record detection, and one safe retry after SSH exit 255 while preserving correlation-based reply expectations.
• Remove the remote typed-payload and exit-3 delivery paths, and update documentation and regression coverage for the completed inbox migration.

Risk Assessment

✅ Low: The remote steer path is consistently migrated to durable idempotent inbox delivery, preserves ambiguous expectations, revalidates lifecycle ownership under locks, and includes behavior-focused regressions without a substantiated remaining defect.

Testing

Remote steers were exercised end-to-end through the simulated SSH boundary, including durable inbox persistence, doorbell behavior, exact-body deduplication, correlation-preserving resend, route-retirement locking, decision closure, failure semantics, and the changed lifecycle failure scenario; targeted tests passed, with the broader lifecycle script timing out only after the relevant scenario had passed.

Evidence: End-to-end ambiguous-send and correlation-reusing resend evidence, showing one persisted record with the same correlation and delivered pending-reply state

Source: End-to-end ambiguous-send and correlation-reusing resend evidence, showing one persisted record with the same correlation and delivered pending-reply state

END-USER SCENARIO: ambiguous remote send followed by the printed correlation-reusing resend

--- CLI guidance after two completion-unknown SSH results ---
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
●  WATCHER DOWN - SUPERVISION IS OFF
●  1 task(s) in flight, but no watcher has a fresh beacon (last beat: never, grace 300s).
●  Trust the emitted supervision protocol for this harness; do not use shell & for watcher repair.
●  This is a supervision warning only; the requested message WILL still be sent.
●  repair a missing or failed watcher cycle with the Pi tool fm_watch_arm_pi, or restart Pi with -e /Users/kunchen/.no-mistakes/worktrees/016d88035d58/01M0R4V0YRSKM0GCEBJ8MXCPQ1/.pi/extensions/fm-primary-turnend-guard.ts -e /Users/kunchen/.no-mistakes/worktrees/016d88035d58/01M0R4V0YRSKM0GCEBJ8MXCPQ1/.pi/extensions/fm-primary-pi-watch.ts if the extensions are not loaded.
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
notice: doorbell did not reach fm-remote:p1; the steer is durably recorded at /var/folders/0k/bf8mwt2n5qddzk24r20gfk0c0000gn/T/fm-send-remote-delivery.T2HDfJ/remote-idem-rhome/state/parent-route/rsm.inbox/001.msg
notice: doorbell did not reach fm-remote:p1; the steer is durably recorded at /var/folders/0k/bf8mwt2n5qddzk24r20gfk0c0000gn/T/fm-send-remote-delivery.T2HDfJ/remote-idem-rhome/state/parent-route/rsm.inbox/001.msg
error: steer to remote secondmate rsm is unconfirmed (transport lost twice; remote completion unknown). Only the correlation-reusing resend below is idempotent and lands on the same remote inbox record:
FM_HOME=/var/folders/0k/bf8mwt2n5qddzk24r20gfk0c0000gn/T/fm-send-remote-delivery.T2HDfJ/remote-idem-11711 FM_PENDING_REPLY_EXISTING_CORR=c47bb8db5a19aa4f /Users/kunchen/.no-mistakes/worktrees/016d88035d58/01M0R4V0YRSKM0GCEBJ8MXCPQ1/bin/fm-send.sh rsm please\ rename\ the\ metric

--- Durable remote inbox state after executing the printed resend ---
record_count=1
record_name=001.msg
schema=fm-task-inbox.v1
at=2026-08-23T21:05:12Z
--
[fm-from-firstmate]⁣corr=c47bb8db5a19aa4f please rename the metric
--- Parent pending-reply state after successful resend ---
corr_id=c47bb8db5a19aa4f
task_id=rsm
phase=awaiting_report
delivered_epoch=1787519113

Evidence: Targeted remote delivery test transcript

Source: Targeted remote delivery test transcript

ok - fm-send remote: a text steer lands as a durable remote inbox record and exits 0 at enqueue
ok - fm-send remote: the printed correlation-reusing resend deduplicates onto the same record
ok - fm-send remote: a failed retry cannot erase an earlier ambiguous delivery
ok - remote control: send revalidates endpoint ownership under the metadata lock
ok - fm-send remote: enqueue revalidates the parent route under its metadata lock
ok - fm-send remote: a --resolve-key answer closes its decision at enqueue
ok - fm-send remote: every remote text, harness-native included, rides the inbox
ok - fm-send remote: a real remote failure still fails loudly with the remote diagnostics
ok - fm-send remote: exit 3 from the remote leg is a failure, never a delivery claim
ok - fm-send remote: ssh 255 fails with resend-safe guidance and preserves the expectation
ok - fm-send local: an unconfirmed submit exits 3 with an honest non-error report
ok - fm-send local: an unconfirmed submit still never closes a --resolve-key decision
ok - fm-send local: an unconfirmed secondmate send keeps its reply expectation armed
all fm-send-remote-delivery tests passed

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 5 issues found → auto-fixed (5) ✅

• 🚨 bin/fm-send.sh:684 - The required promise that “re-running the identical request lands on the same record” is not satisfied by the emitted “Re-running this send is safe” guidance. After two SSH-255 results, rerunning the same CLI command creates a fresh pending-reply correlation token, changing the record body; exact-body dedup therefore creates a second record and can execute the instruction twice. Preserve/reuse the original correlation through a supported rerun path before claiming resend safety.
• 🚨 bin/fm-send.sh:687 - If the first attempt exits 255 after writing remotely but the retry returns another nonzero status, this branch treats delivery as known-undelivered and discards/resets the expectation. The first attempt remains completion-unknown, so the record may exist. Track whether any attempt returned 255 and preserve delivery_unknown unless a later attempt confirms success.
• 🚨 bin/fm-task-inbox-lib.sh:188 - Exact-body dedup can miss a record while the worker moves it from the inbox root to handled/: glob paths are expanded before iteration, and the acknowledgement does not take .seq.lock. If the root path vanishes after expansion while handled was expanded before the move, the retry writes a duplicate. Recheck the corresponding handled path or perform a stable rescan before allocating a new record.
• 🚨 bin/fm-remote-secondmate-control.sh:192 - cmd_send validates endpoint metadata without taking its metadata lock, then writes later. Concurrent teardown can remove the endpoint metadata and inbox after validation; the send then recreates an orphan inbox, ignores the failed ring, and exits 0 as delivered although no worker owns it. Acquire the endpoint metadata lock, revalidate under it, and hold it through enqueue, matching the local inbox boundary.
• ⚠️ docs/architecture.md:115 - This documentation still lists “the remote leg” as part of fm-send's typed plane, contradicting the new inbox-only remote behavior. docs/tmux-backend.md:88 has the same stale claim. Update both so operators are not instructed using the deleted transport model.

🔧 Fix: Harden remote inbox delivery against lifecycle races
1 error still open:

• 🚨 bin/fm-send.sh:687 - The required invariant “re-running the identical request lands on the same record” remains violated. Re-running the same CLI command creates a new pending-reply correlation token, changing the record body, while explicitly reusing the old correlation is refused because its delivery attempt remains unresolved. Exact-body dedup therefore writes a second record despite the emitted “Re-running this send is safe” guidance. Provide a supported rerun path that preserves the original correlation before claiming resend safety.

🔧 Fix: Enable correlation-preserving remote steer resends
3 issues (2 errors, 1 warning) still open:

• 🚨 bin/fm-send.sh:617 - An explicitly supplied FM_PENDING_REPLY_EXISTING_CORR silently falls back to creating a new correlation when the old expectation is no longer reusable. For example, after double-255 leaves delivery_unknown, the watcher can advance it to escalated before the operator runs the printed command; fm-send then embeds a new corr, exact-body dedup misses, and the instruction executes twice. When the environment variable is explicitly set, fail closed rather than minting a replacement correlation.
• 🚨 tests/fm-remote-secondmate-lifecycle-e2e.test.sh:881 - The changed lifecycle test still requires the removed text “Re-running this send is safe,” while fm-send now emits “Only the correlation-reusing resend below is idempotent.” This executable regression will fail before reaching its remaining lifecycle assertions; update it to assert the correlation-reusing command contract.
• ⚠️ AGENTS.md:306 - This still tells operators that an unconfirmed remote delivery is safe to re-run, but only a resend preserving FM_PENDING_REPLY_EXISTING_CORR is idempotent; a plain rerun creates a new body and record. docs/remote-secondmates.md:173 repeats the same unsafe guidance. Document the exact correlation-reusing resend boundary in both places.

🔧 Fix: Fail closed on stale correlation resends
2 errors still open:

• 🚨 bin/fm-send.sh:711 - The promised exact resend command omits the mandatory FM_HOME environment. For the documented invocation FM_HOME=<primary-home> bin/fm-send.sh ..., that assignment is command-scoped, so copying the printed command later fails before resending. Include the resolved FM_HOME (and any active state override needed to reach the same expectation) in the shell-quoted command.
• 🚨 bin/fm-send.sh:31 - The header still says “re-running the send later remains safe,” contradicting the required contract that only the printed FM_PENDING_REPLY_EXISTING_CORR=<id> resend is idempotent and that a plain rerun mints a new correlation and record. Make this paragraph explicitly require correlation reuse.

🔧 Fix: Include home context in remote resend commands
3 issues (2 errors, 1 warning) still open:

• 🚨 bin/fm-send.sh:686 - The remote inbox path does not acquire or revalidate the parent task's metadata lock. A send can resolve the route, then parent teardown can acquire the lock and begin retirement before the send creates its expectation; if the send's host-local enqueue wins the remote lock, retirement can immediately delete that inbox and the send still exits 0. Acquire the parent metadata lock, revalidate the remote route under it, and hold it through the remote enqueue, matching the local inbox boundary.
• 🚨 bin/fm-send.sh:31 - The header says “re-running the send later remains safe,” but the supported invariant now requires the printed FM_PENDING_REPLY_EXISTING_CORR=<id> command; a plain rerun mints a different correlation and body. This contradicts the required correlation-preserving resend contract. State explicitly that only the correlation-reusing resend is idempotent.
• ⚠️ bin/fm-send.sh:711 - The printed resend command copies FM_HOME and FM_STATE_OVERRIDE verbatim rather than resolving relative paths. For a valid invocation such as FM_HOME=primary ..., running the printed command later from another directory fails before resending. Print canonical paths so the promised standalone resend remains executable.

🔧 Fix: Lock and revalidate remote parent routes
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

• ./tests/fm-send-remote-delivery.test.sh
• bash tests/fm-task-inbox.test.sh (rerun via Bash after the non-executable test file returned permission denied)
• ./tests/fm-remote-secondmate-lifecycle-e2e.test.sh (the changed failed-reread-nudge scenario passed; the broader lifecycle command later exceeded the timeout)
• Evidence replay of the ambiguous double-255 send followed by executing the printed correlation-reusing resend command

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

_{Reviews (2): Last reviewed commit: "no-mistakes: apply CI fixes" | Re-trigger Greptile}