feat(bin): make capability green require first-try validation - #96
Merged
Conversation
Teardown derives the capability outcome from the repo-scoped no-mistakes run records for the task branch: exactly one completed attempt is green, a pass after earlier attempts is fixed with its fix-round count, a never-completed history is failed, and anything underivable is unknown rather than guessed. fm-send records confirmed supervisor sends per task so teardown can log the steer count. Trailing counts are written only when derivable, and older six-field lines keep parsing side by side with the new format.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Improve firstmate's capability-outcome logging so green means what it claims. Accepted definition: green means the task's CI passed on the FIRST try; today teardown approximates success by any non-forced teardown, a proxy that must be replaced by deriving the outcome from recorded validation/CI results at the teardown-time capability logging path (bin/fm-teardown.sh plus the bin/fm-capability-lib.sh helpers it uses), with docs/configuration.md's Capability outcome log section kept accurate for field additions.
Acceptance criteria:
Deliberate implementation decisions accepted by the captain (not mistakes):
Repo style constraints: one sentence per line in tracked Markdown, plain dashes only, shellcheck-clean bin scripts, colocated tests named .test.sh, and never an agent name as commit co-author.
What Changed
no-mistakesrun history, reservinggreenfor first-try passes and recordingfixed,failed, orunknownwhen appropriate.Risk Assessment
✅ Low: Captain, the change is well-bounded, all prior findings are resolved with matching regression coverage, and no remaining material risks were found in the full diff.
Testing
At target commit 03e2640, focused capability, steer-count, teardown, and temporary-directory regression tests passed. The captured CLI transcript demonstrates first-try
green|0, post-fixfixedrather than green, failed and unknown outcomes, derived steer counts, mixed old/new log parsing, and executable test permissions; the worktree remained clean.Evidence: End-to-end capability outcome transcript
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 2 issues found → auto-fixed (2) ✅
bin/fm-capability-lib.sh:118- The required trailing counts are misidentified when fix rounds are absent but steers are known. This code omits the empty fix-rounds slot and writes steers as field 7, so records such asfailed|1orunknown|1document1as fix-rounds rather than steers. This contradicts “Record the number of validation-pipeline fix rounds and supervisor steers when derivable ... never guessed.” Preserve positional meaning, such as by retaining an empty placeholder or using explicitly keyed fields.bin/fm-teardown.sh:186- When the worktree branch cannot be read or is detached,CAP_OUTCOMEremains empty and the recorder silently skips the capability line. The accepted behavior requires underivable results to be recorded asunknown, including unavailable run records, rather than omitted. Initialize non-forced ship/scout outcomes tounknownbefore attempting the probe, then replace that value only when evidence is derived.🔧 Fix: Preserve capability count slots and record unknown outcomes
1 error still open:
bin/fm-send.sh:304- The required rule says “explicit backend targets ... never write” the steer file, but an explicit recorded window/terminal is resolved throughfm_backend_meta_for_window, populatingTARGET_META, and this condition then records a steer. The test only covers an unmatched external target, sofm-send.sh sess:fm-build ...incorrectly incrementsbuild.steers. Gate recording on the task-selector signal (TARGET_SELECTOR) and cover an explicit target that matches task metadata.🔧 Fix: Exclude explicit targets from steer counts
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
tests/fm-capability.test.shtests/fm-send-steer-count.test.shtests/fm-teardown.test.shtests/fm-gotmp.test.shstat -f '%Sp %N' tests/fm-capability.test.sh tests/fm-teardown.test.sh tests/fm-send-steer-count.test.shgit status --shortgit diff --check 3418df5656d8dc5a613319525b68a1a7e3cbf3c1..03e26402dc3da7daec5d895d8bd4727175112efa✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.