feat(bin): promote a fleet learning to a binding check, add recurrence measurement - #2865
Open
pramendra wants to merge 5 commits into
Open
feat(bin): promote a fleet learning to a binding check, add recurrence measurement#2865pramendra wants to merge 5 commits into
pramendra wants to merge 5 commits into
Conversation
added 2 commits
August 23, 2026 19:59
…surement Workstream 5 of the fleet engineering plan depends on workstream 4's executable-contract mechanism, which has not landed (fleet-ws4-executable- contracts is still in flight, zero commits ahead of main at time of writing). Scoped down per the brief: pick one learnings.md entry checkable without that machinery, rather than a design-only pass. Promoted learning (data/learnings.md, recorded 2026-08-22): spawned Claude workers inherit firstmate's CLAUDE_CONFIG_DIR, which has no user-level skills directory, so an instruction to invoke a skill by name gets "Unknown skill" and stalls. bin/fm-brief.sh's no-mistakes and direct-PR delivery blocks still told every worker to "run /no-mistakes" - live evidence of recurrence: this exact defect reached the crewmate dispatching this change, in its own brief, filed but undispatched since 2026-08-22 as brief-tells-workers-to-invoke-unreachable-skill. - Fix bin/fm-brief.sh to route validation through the CLI (`no-mistakes axi run`) instead of a literal "/no-mistakes" invocation, in every mode. - Add the binding check: tests/fm-brief.test.sh now fails if any generated ship brief ever contains "/no-mistakes" again, in any mode. Mutation- verified: reverted the fix (including a differently-worded variant) and confirmed the check goes red; restored and confirmed green. - Add bin/fm-learning-recurrence.sh + -lib.sh: measures recurrence from git history itself (commits that add the banned pattern to a guarded path), split into pre-rule introductions and recurrences on/after the recorded date, rather than a hand-kept tally. Measured against this repo: 8 historical introductions, 0 lines added, 0 recurrences since the rule was recorded. tests/fm-learning-recurrence.test.sh proves the counter against a fixture repo, including a real reintroduction it must detect. - docs/architecture.md documents the promotion pattern for future learnings. Not built: promotion for any other learnings.md entry, and no workstream-4-style registry/drift-guard machinery (out of scope until workstream 4 lands). The recurrence tool measures tracked-source reintroduction only, not every live brief a worker may have hit before this fix landed - that instance-level count isn't reconstructable from git history and isn't claimed here.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains. Reviews (3): Last reviewed commit: "no-mistakes(document): Fix stale harness..." | Re-trigger Greptile |
added 3 commits
August 23, 2026 22:33
…mmit The no-mistakes pipeline's document fix step swept .serena/project.yml and .serena/.gitignore into its commit alongside the intended AGENTS.md fix - environment leakage from a globally-registered Serena MCP server writing into the isolated worktree, not intentional content. Untrack them and ignore .serena/ repo-wide so this cannot happen again.
…c contradicting new no-mistakes CLI rule
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Workstream 5 of the fleet engineering plan (data/fleet-engineering-plan.html), sequenced last because it depends on workstream 4 landing first. Build the mechanism that promotes a recorded learning (in data/learnings.md) into an actual binding check, refusal, or brief clause - never leaving it as prose alone - and measures recurrence: how many times the SAME defect class has shown up since the rule was written. A rule with recurring hits is a wish, per this fleet's own stated doctrine (data/learnings.md self-critically notes it reached 16 entries and 99% of its startup memory budget before being rewritten once - evidence that write-prose-and-hope-it-is-remembered does not scale). Done when: at least one existing learnings.md entry is demonstrably promoted to a real, testable check (not just referenced in a brief), and there is a mechanism - even a simple one - for tracking how many times that defect class has recurred since. Scope note: this depends on workstream 4's executable-contract mechanism; if workstream 4 has not landed when starting, say so plainly and either scope down to a design-only pass or pick a learning that can be checked without workstream 4's machinery. Acceptance criteria: the mechanism must be built and demonstrably work (show it, not just assert it); any check added must be proven able to fail by mutating what it guards and showing it go red; claim only what was measured, stating plainly what was NOT built and why scope was narrowed if it was. At task start, workstream 4 (fleet-ws4-executable-contracts) had not landed - it was live on a sibling worktree with zero commits ahead of main - so this task was scoped down accordingly: promote one learning checkable without workstream 4's machinery, rather than build a design-only pass or invent registry/drift-guard generality beyond what is needed for that one rule. Follow-up commit: the prior pipeline run's document-fix step accidentally committed .serena/project.yml and .serena/.gitignore (Serena MCP environment leakage into the worktree); this run additionally untracks those files and adds .serena/ to .gitignore so the PR does not ship that artifact.
What Changed
bin/fm-learning-recurrence-lib.sh, a shared helper (introductions_for_path) that walks a guarded file's git history and counts commits whose diff adds a line matching a banned pattern, excluding comment lines and excluding rewrites of a pattern that was already present (only counts genuine (re)introductions).bin/fm-learning-recurrence.sh, a CLI wrapper with a--list/<rule-id>interface that reportsintroductions_totalandrecurrences_since_recordedfor a promoted learning, splitting counts before/after the rule's recorded date; registers the first promoted rule,no-mistakes-cli-not-skill, guardingbin/fm-brief.shagainst the literal/no-mistakespattern.tests/fm-brief.test.sh:test_ship_briefs_never_instruct_skill_invocation) that fails if any ship brief (no-mistakes, direct-PR, or local-only mode) instructs a worker to invoke/no-mistakesas a skill instead of routing through theno-mistakesCLI, and rewordsbin/fm-brief.sh's generated brief text,AGENTS.md's Validate section, and.agents/skills/harness-adapters/SKILL.mdaccordingly.tests/fm-learning-recurrence.test.sh, which exercisesintroductions_for_pathand the CLI wrapper against a fixture git repo (isolated from this repo's own history) to prove pre-rule introductions, fixes, post-rule recurrences, comment exclusion, and rewrite-of-still-present-pattern exclusion are all counted correctly.docs/architecture.md, and untrack.serena/project.yml/.serena/.gitignore(accidental Serena MCP artifacts committed by a prior pipeline run) by adding.serena/to.gitignore.Risk Assessment
✅ Low: The change promotes exactly one learning (no-mistakes CLI vs. skill invocation) to a binding, mutation-provable test, adds a git-history-based recurrence counter with its own fixture-driven behavioral tests (including a CI-round fix for a real double-counting bug on reformatted lines), updates docs consistently, and separately cleans up an accidental Serena artifact leak; scope matches the stated workstream-4-unavailable narrowing and no risky or irreversible operations are introduced.
Testing
Ran the new fm-learning-recurrence test suite and the fm-brief suite (both fully pass), then manually mutated bin/fm-brief.sh to reintroduce the banned no-mistakes skill-invocation phrase and confirmed the promoted binding check goes red before reverting, demonstrating the learning is a real provably-failing check rather than prose, with the recurrence counter separately verified against a fixture repo and end-to-end against the real production rule and pattern.
Evidence: tests/fm-learning-recurrence.test.sh full pass
Evidence: Mutation proof: reintroducing the no-mistakes skill-invocation instruction turns the binding check red
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
✅ **Test** - passed
✅ No issues found.
bash tests/fm-learning-recurrence.test.sh (3 of 3 pass: additions-only counting, recorded-date split against the real production rule, and rewrite-of-still-present-pattern exclusion)bash tests/fm-brief.test.sh (full suite including new test_ship_briefs_never_instruct_skill_invocation, all pass)Manual mutation: reintroduced the literal /no-mistakes skill-invocation phrase into bin/fm-brief.sh, re-ran tests/fm-brief.test.sh, confirmed it goes red (not ok), then reverted the mutation and confirmed git status --short is cleanConfirmed .serena/project.yml and .serena/.gitignore are untracked from git history while .serena/ is now in .gitignore, matching the intended cleanup commit✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.