Skip to content

feat(bin): make capability green require first-try validation - #96

Merged
Amplify-Logic merged 4 commits into
mainfrom
fm/fm-capability-truegreen-signal-t3
Aug 23, 2026
Merged

feat(bin): make capability green require first-try validation#96
Amplify-Logic merged 4 commits into
mainfrom
fm/fm-capability-truegreen-signal-t3

Conversation

@Amplify-Logic

Copy link
Copy Markdown
Owner

Intent

Improve firstmate's capability-outcome logging so green means what it claims. Accepted definition: green means the task's CI passed on the FIRST try; today teardown approximates success by any non-forced teardown, a proxy that must be replaced by deriving the outcome from recorded validation/CI results at the teardown-time capability logging path (bin/fm-teardown.sh plus the bin/fm-capability-lib.sh helpers it uses), with docs/configuration.md's Capability outcome log section kept accurate for field additions.

Acceptance criteria:

  • The outcome log's success signal is derived from the recorded validation/CI result (first-try pass), never from teardown mode.
  • Record the number of validation-pipeline fix rounds and supervisor steers when derivable from local task records; absent otherwise, never guessed. These are capturable extras, not gates.
  • Backward compatible: existing six-field log lines keep parsing, and the reader (bin/fm-dispatch-select.sh capability path) handles old and new lines.
  • Colocated executable tests cover first-try green, green-after-fix-rounds (not first-try, so it must NOT count as green), failed, and old-format parsing; any new tests/*.test.sh needs chmod +x or CI fails with exit 126.

Deliberate implementation decisions accepted by the captain (not mistakes):

  • Outcome values are green, fixed, failed, unknown, discarded: exactly one completed pipeline attempt for the branch is green with fix-rounds 0; completion after earlier attempts is fixed with fix-rounds counting prior attempts; newest attempt not completed is failed; anything underivable (scout reports, direct-PR/local-only delivery, unavailable run records) is unknown rather than guessed; --force still records discarded without derived counts.
  • Evidence source: a bounded read-only no-mistakes runs --limit 200 probe run from the worktree before destructive cleanup, parsing newest-first completed/failed/cancelled rows for the exact branch name.
  • Steer recording: bin/fm-send.sh appends one line to state/.steers on every confirmed text submit to a meta-resolved task target; teardown reads then deletes that volatile file; explicit backend targets and the --key path never write it.
  • Trailing counts are omitted entirely when absent; the reader itself needed no code change because its positional awk parsing already tolerates extra fields, proven by mixed-format tests.

Repo style constraints: one sentence per line in tracked Markdown, plain dashes only, shellcheck-clean bin scripts, colocated tests named .test.sh, and never an agent name as commit co-author.

What Changed

  • Derive teardown capability outcomes from bounded no-mistakes run history, reserving green for first-try passes and recording fixed, failed, or unknown when appropriate.
  • Extend the backward-compatible outcome format with optional fix-round and confirmed supervisor-steer counts, while excluding explicit backend targets and key sends from steer tracking.
  • Update capability documentation and executable coverage for outcome derivation, mixed-format parsing, steer counting, and teardown cleanup.

Risk Assessment

✅ Low: Captain, the change is well-bounded, all prior findings are resolved with matching regression coverage, and no remaining material risks were found in the full diff.

Testing

At target commit 03e2640, focused capability, steer-count, teardown, and temporary-directory regression tests passed. The captured CLI transcript demonstrates first-try green|0, post-fix fixed rather than green, failed and unknown outcomes, derived steer counts, mixed old/new log parsing, and executable test permissions; the worktree remained clean.

Evidence: End-to-end capability outcome transcript
End-to-end capability outcome validation
Commit: 03e26402dc3da7daec5d895d8bd4727175112efa

[tests/fm-capability.test.sh]
ok - log append writes wire lines and the reader applies the 7-day window
ok - summarize reports green density per harness/model/effort
ok - teardown recorder writes derived outcomes with optional counts and skips secondmate
ok - run-table derivation yields first-try green, fixed, failed, or unknown
ok - reader handles old and new wire lines side by side
ok - capability-recent ranks within cost rules and scout tax stays advisory
ok - absent select keeps first-profile selection with evidence only advisory
ok - 0%-green sampled keeps configured order over earlier untried
ok - scout tax rate above 100 clamps to 100
ok - log append refuses pipe-bearing fields
# all fm-capability tests passed

[tests/fm-send-steer-count.test.sh]
ok - fm-send appends one steer line per confirmed task-target submit
ok - explicit backend targets leave no steer counter
ok - explicit recorded backend targets leave no steer counter
ok - the --key path leaves no steer counter
ok - unconfirmed submits leave no steer counter
# all fm-send-steer-count tests passed

[tests/fm-teardown.test.sh]
ok - local-only worktree with HEAD on a fork remote is torn down (fix holds)
ok - treehouse return clears the durable path claim before later cleanup
ok - provider return and concurrent allocation are serialized by the lease owner
ok - teardown prompts tasks-axi backlog refresh when compatible
ok - teardown honors config/backlog-backend=manual even when tasks-axi is compatible
ok - local-only worktree with truly unpushed work is refused (safety preserved)
ok - local-only worktree with work merged into local main is torn down (no regression)
ok - no-mistakes worktree with HEAD on origin is torn down (no regression)
ok - teardown derives first-try green from recorded runs and consumes the steer counter
ok - teardown records fixed after fix rounds and keeps green first-try only
ok - teardown records failed when no pipeline attempt completed
ok - teardown records unknown when validation evidence is absent
ok - teardown records unknown when the branch name is unavailable
ok - no-mistakes worktree with genuinely unlanded work is refused (safety preserved)
ok - local-only worktree with unpushed work is torn down under --force (escape hatch)
ok - herdr teardown removes pane-owned escalation dedupe state
ok - teardown collects both captain-held one-shot namespaces so a same-id relaunch still surfaces
ok - squash-merged + deleted-branch worktree (PR merged) is torn down (the fix)
ok - squash-merged PR accepts a local HEAD that is an ancestor of the final PR head
ok - teardown discovers a merged PR by branch name and tears down when no pr= was ever recorded
ok - squash-merged PR accepts replayed unpushed local patches contained in the PR head
ok - merged PR does not allow teardown after a later local commit
ok - fm-pr-check does not refresh PR head after HEAD moves
ok - fm-pr-check records the remote PR head when the local worktree lags
ok - worktree whose content already landed in the default branch is torn down (content fallback)
ok - content fallback refreshes origin default before comparing trees
ok - dirty worktree is refused even when its committed work has landed (dirty always wins)
ok - gh lookup error with content not in default refuses (fail-safe)
ok - provably-stale worktree index.lock (old, no live holder) is cleared and teardown succeeds
ok - live-held worktree index.lock is never removed and teardown refuses
ok - lsof errors leave worktree index.lock in place and refuse teardown
ok - stale lock cleanup rechecks and refuses dirty worktree before return
ok - normal repo index.lock is resolved from the worktree and cleared when stale
ok - lock mtime read failures leave worktree index.lock in place and refuse teardown
ok - transient index.lock cleared after first failed return is retried successfully without force-remove
ok - persistent index.lock exhausts retries and refuses without force-removing the lock
ok - empty retry wait overrides use the default without aborting teardown
ok - fractional legacy retry wait remains supported without arithmetic

[executable modes]
-rwxr-xr-x tests/fm-capability.test.sh
-rwxr-xr-x tests/fm-teardown.test.sh
-rwxr-xr-x tests/fm-send-steer-count.test.sh

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 2 issues found → auto-fixed (2) ✅
  • 🚨 bin/fm-capability-lib.sh:118 - The required trailing counts are misidentified when fix rounds are absent but steers are known. This code omits the empty fix-rounds slot and writes steers as field 7, so records such as failed|1 or unknown|1 document 1 as fix-rounds rather than steers. This contradicts “Record the number of validation-pipeline fix rounds and supervisor steers when derivable ... never guessed.” Preserve positional meaning, such as by retaining an empty placeholder or using explicitly keyed fields.
  • 🚨 bin/fm-teardown.sh:186 - When the worktree branch cannot be read or is detached, CAP_OUTCOME remains empty and the recorder silently skips the capability line. The accepted behavior requires underivable results to be recorded as unknown, including unavailable run records, rather than omitted. Initialize non-forced ship/scout outcomes to unknown before attempting the probe, then replace that value only when evidence is derived.

🔧 Fix: Preserve capability count slots and record unknown outcomes
1 error still open:

  • 🚨 bin/fm-send.sh:304 - The required rule says “explicit backend targets ... never write” the steer file, but an explicit recorded window/terminal is resolved through fm_backend_meta_for_window, populating TARGET_META, and this condition then records a steer. The test only covers an unmatched external target, so fm-send.sh sess:fm-build ... incorrectly increments build.steers. Gate recording on the task-selector signal (TARGET_SELECTOR) and cover an explicit target that matches task metadata.

🔧 Fix: Exclude explicit targets from steer counts
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

  • tests/fm-capability.test.sh
  • tests/fm-send-steer-count.test.sh
  • tests/fm-teardown.test.sh
  • tests/fm-gotmp.test.sh
  • stat -f '%Sp %N' tests/fm-capability.test.sh tests/fm-teardown.test.sh tests/fm-send-steer-count.test.sh
  • git status --short
  • git diff --check 3418df5656d8dc5a613319525b68a1a7e3cbf3c1..03e26402dc3da7daec5d895d8bd4727175112efa
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

Teardown derives the capability outcome from the repo-scoped no-mistakes run records for the task branch: exactly one completed attempt is green, a pass after earlier attempts is fixed with its fix-round count, a never-completed history is failed, and anything underivable is unknown rather than guessed. fm-send records confirmed supervisor sends per task so teardown can log the steer count. Trailing counts are written only when derivable, and older six-field lines keep parsing side by side with the new format.
@Amplify-Logic
Amplify-Logic merged commit 361861c into main Aug 23, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant