Skip to content

feat(bin): promote a fleet learning to a binding check, add recurrence measurement - #2865

Open
pramendra wants to merge 5 commits into
kunchenguid:mainfrom
pramendra:fm/fleet-ws5-learning-loop
Open

feat(bin): promote a fleet learning to a binding check, add recurrence measurement#2865
pramendra wants to merge 5 commits into
kunchenguid:mainfrom
pramendra:fm/fleet-ws5-learning-loop

Conversation

@pramendra

@pramendra pramendra commented Aug 23, 2026

Copy link
Copy Markdown

Intent

Workstream 5 of the fleet engineering plan (data/fleet-engineering-plan.html), sequenced last because it depends on workstream 4 landing first. Build the mechanism that promotes a recorded learning (in data/learnings.md) into an actual binding check, refusal, or brief clause - never leaving it as prose alone - and measures recurrence: how many times the SAME defect class has shown up since the rule was written. A rule with recurring hits is a wish, per this fleet's own stated doctrine (data/learnings.md self-critically notes it reached 16 entries and 99% of its startup memory budget before being rewritten once - evidence that write-prose-and-hope-it-is-remembered does not scale). Done when: at least one existing learnings.md entry is demonstrably promoted to a real, testable check (not just referenced in a brief), and there is a mechanism - even a simple one - for tracking how many times that defect class has recurred since. Scope note: this depends on workstream 4's executable-contract mechanism; if workstream 4 has not landed when starting, say so plainly and either scope down to a design-only pass or pick a learning that can be checked without workstream 4's machinery. Acceptance criteria: the mechanism must be built and demonstrably work (show it, not just assert it); any check added must be proven able to fail by mutating what it guards and showing it go red; claim only what was measured, stating plainly what was NOT built and why scope was narrowed if it was. At task start, workstream 4 (fleet-ws4-executable-contracts) had not landed - it was live on a sibling worktree with zero commits ahead of main - so this task was scoped down accordingly: promote one learning checkable without workstream 4's machinery, rather than build a design-only pass or invent registry/drift-guard generality beyond what is needed for that one rule. Follow-up commit: the prior pipeline run's document-fix step accidentally committed .serena/project.yml and .serena/.gitignore (Serena MCP environment leakage into the worktree); this run additionally untracks those files and adds .serena/ to .gitignore so the PR does not ship that artifact.

What Changed

  • Add bin/fm-learning-recurrence-lib.sh, a shared helper (introductions_for_path) that walks a guarded file's git history and counts commits whose diff adds a line matching a banned pattern, excluding comment lines and excluding rewrites of a pattern that was already present (only counts genuine (re)introductions).
  • Add bin/fm-learning-recurrence.sh, a CLI wrapper with a --list/<rule-id> interface that reports introductions_total and recurrences_since_recorded for a promoted learning, splitting counts before/after the rule's recorded date; registers the first promoted rule, no-mistakes-cli-not-skill, guarding bin/fm-brief.sh against the literal /no-mistakes pattern.
  • Promote that learning into a binding, mutation-proven regression test (tests/fm-brief.test.sh:test_ship_briefs_never_instruct_skill_invocation) that fails if any ship brief (no-mistakes, direct-PR, or local-only mode) instructs a worker to invoke /no-mistakes as a skill instead of routing through the no-mistakes CLI, and rewords bin/fm-brief.sh's generated brief text, AGENTS.md's Validate section, and .agents/skills/harness-adapters/SKILL.md accordingly.
  • Add tests/fm-learning-recurrence.test.sh, which exercises introductions_for_path and the CLI wrapper against a fixture git repo (isolated from this repo's own history) to prove pre-rule introductions, fixes, post-rule recurrences, comment exclusion, and rewrite-of-still-present-pattern exclusion are all counted correctly.
  • Document the promotion mechanism in docs/architecture.md, and untrack .serena/project.yml/.serena/.gitignore (accidental Serena MCP artifacts committed by a prior pipeline run) by adding .serena/ to .gitignore.

Risk Assessment

✅ Low: The change promotes exactly one learning (no-mistakes CLI vs. skill invocation) to a binding, mutation-provable test, adds a git-history-based recurrence counter with its own fixture-driven behavioral tests (including a CI-round fix for a real double-counting bug on reformatted lines), updates docs consistently, and separately cleans up an accidental Serena artifact leak; scope matches the stated workstream-4-unavailable narrowing and no risky or irreversible operations are introduced.

Testing

Ran the new fm-learning-recurrence test suite and the fm-brief suite (both fully pass), then manually mutated bin/fm-brief.sh to reintroduce the banned no-mistakes skill-invocation phrase and confirmed the promoted binding check goes red before reverting, demonstrating the learning is a real provably-failing check rather than prose, with the recurrence counter separately verified against a fixture repo and end-to-end against the real production rule and pattern.

Evidence: tests/fm-learning-recurrence.test.sh full pass
ok - fm-learning-recurrence-lib.sh: introductions_for_path counts additions only, excludes comments
ok - fm-learning-recurrence.sh: CLI wrapper measures the real rule and splits on its recorded date
ok - fm-learning-recurrence-lib.sh: rewriting an already-present pattern line does not inflate the recurrence count
Evidence: Mutation proof: reintroducing the no-mistakes skill-invocation instruction turns the binding check red
sed replaced the CLI-routing sentence in bin/fm-brief.sh with a literal /no-mistakes instruction; bash tests/fm-brief.test.sh then reported not ok on the affected test; mutation was reverted and git status --short confirmed clean

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • bash tests/fm-learning-recurrence.test.sh (3 of 3 pass: additions-only counting, recorded-date split against the real production rule, and rewrite-of-still-present-pattern exclusion)
  • bash tests/fm-brief.test.sh (full suite including new test_ship_briefs_never_instruct_skill_invocation, all pass)
  • Manual mutation: reintroduced the literal /no-mistakes skill-invocation phrase into bin/fm-brief.sh, re-ran tests/fm-brief.test.sh, confirmed it goes red (not ok), then reverted the mutation and confirmed git status --short is clean
  • Confirmed .serena/project.yml and .serena/.gitignore are untracked from git history while .serena/ is now in .gitignore, matching the intended cleanup commit
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

Your Name added 2 commits August 23, 2026 19:59
…surement

Workstream 5 of the fleet engineering plan depends on workstream 4's
executable-contract mechanism, which has not landed (fleet-ws4-executable-
contracts is still in flight, zero commits ahead of main at time of writing).
Scoped down per the brief: pick one learnings.md entry checkable without
that machinery, rather than a design-only pass.

Promoted learning (data/learnings.md, recorded 2026-08-22): spawned Claude
workers inherit firstmate's CLAUDE_CONFIG_DIR, which has no user-level
skills directory, so an instruction to invoke a skill by name gets "Unknown
skill" and stalls. bin/fm-brief.sh's no-mistakes and direct-PR delivery
blocks still told every worker to "run /no-mistakes" - live evidence of
recurrence: this exact defect reached the crewmate dispatching this change,
in its own brief, filed but undispatched since 2026-08-22 as
brief-tells-workers-to-invoke-unreachable-skill.

- Fix bin/fm-brief.sh to route validation through the CLI (`no-mistakes axi
  run`) instead of a literal "/no-mistakes" invocation, in every mode.
- Add the binding check: tests/fm-brief.test.sh now fails if any generated
  ship brief ever contains "/no-mistakes" again, in any mode. Mutation-
  verified: reverted the fix (including a differently-worded variant) and
  confirmed the check goes red; restored and confirmed green.
- Add bin/fm-learning-recurrence.sh + -lib.sh: measures recurrence from git
  history itself (commits that add the banned pattern to a guarded path),
  split into pre-rule introductions and recurrences on/after the recorded
  date, rather than a hand-kept tally. Measured against this repo: 8
  historical introductions, 0 lines added, 0 recurrences since the rule was
  recorded. tests/fm-learning-recurrence.test.sh proves the counter against
  a fixture repo, including a real reintroduction it must detect.
- docs/architecture.md documents the promotion pattern for future learnings.

Not built: promotion for any other learnings.md entry, and no
workstream-4-style registry/drift-guard machinery (out of scope until
workstream 4 lands). The recurrence tool measures tracked-source
reintroduction only, not every live brief a worker may have hit before this
fix landed - that instance-level count isn't reconstructable from git
history and isn't claimed here.
@greptile-apps

greptile-apps Bot commented Aug 23, 2026

Copy link
Copy Markdown

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Reviews (3): Last reviewed commit: "no-mistakes(document): Fix stale harness..." | Re-trigger Greptile

Comment thread bin/fm-learning-recurrence-lib.sh
Your Name added 3 commits August 23, 2026 22:33
…mmit

The no-mistakes pipeline's document fix step swept .serena/project.yml and
.serena/.gitignore into its commit alongside the intended AGENTS.md fix -
environment leakage from a globally-registered Serena MCP server writing
into the isolated worktree, not intentional content. Untrack them and
ignore .serena/ repo-wide so this cannot happen again.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant