Repository navigation
feat: route lessons to /app-control:map refresh or /corrections:codify #10072
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
6571309
feat(app-control): route lessons to map refresh or /corrections:codify
kyle-sexton 3f166d1
fix(app-control): let a handed-in lesson or feature id run past a ref…
kyle-sexton 1b4d5b4
docs(app-control): wrap refresh's skip and live-pass lines to the fil…
kyle-sexton 43c25ef
fix(app-control): keep map's argument-hint inside the 100-character b…
kyle-sexton 5599e58
fix(app-control): resolve refresh's feature id as drive does and narr…
kyle-sexton bb5c364
test(app-control): tie the lesson regexes to the Gotchas section and …
kyle-sexton 69b5bdf
Merge branch 'main' into feat/9584-lesson-routing
kyle-sexton 8a6cc04
fix(app-control,verification): answer the ready reviews of lesson rou…
kyle-sexton d3d6072
Merge main into feat/9584-lesson-routing
melodic-automation-lanes[bot] 4d76ed7
test(app-control): label the lesson cases' judge graders for calibration
kyle-sexton 7b523c5
Merge branch 'feat/9584-lesson-routing' of ssh://github.com/melodic-s…
kyle-sexton cfcd6b7
docs(app-control): keep refresh's feature id to the repository's cont…
kyle-sexton de19fce
docs(app-control): block a refresh feature id no repository control s…
kyle-sexton cb7f4a7
fix(app-control,verification): carry the lesson text on confirm's map…
kyle-sexton File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| --- | ||
| bump: minor | ||
| --- | ||
|
|
||
| ### Added | ||
|
|
||
| - `/app-control:map` create and refresh place a lesson learned while driving by its scope: a lesson about one feature goes in that feature's `## Gotchas`, one that holds for every feature goes in a new optional `## Gotchas` section of the control skill's hub, and any other lesson is written nowhere in the control skill and reported for `/corrections:codify`. In refresh a placed lesson is doc drift and ships in the upkeep pull request. | ||
| - `/app-control:map refresh <feature>` narrows the source wave and live pass to one feature of the control skill that holds it; a clean narrowed pass records no state. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| bump: minor | ||
| --- | ||
|
|
||
| ### Added | ||
|
|
||
| - `/verification:confirm` lists the lessons a run learned in its report, each with its scope and route, and its `## Next` names `/app-control:map refresh <feature>` for a lesson about a feature or drive step and `/corrections:codify` for any other. It never edits a control skill itself. |
19 changes: 19 additions & 0 deletions
19
plugins/app-control/evals/map-lesson-every-feature/case.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| schema_version: "1.1" | ||
| name: map-lesson-every-feature | ||
| description: Refresh places a lesson that holds for every feature in the hub's Gotchas once, not in each feature file | ||
| tags: [map, refresh, lessons, invented] | ||
| expected_outcome: "The reply puts the cookie-banner lesson once in a `## Gotchas` section of the hub, .claude/skills/control-clockwork/SKILL.md, rather than copying it into each of the four feature files, treats it as doc drift so the outcome is changed, ships it on control-skill-upkeep/control-clockwork in one pull request, does not hand it to /corrections:codify, and writes nothing." | ||
| execution: | ||
| prompt: "/app-control:map refresh .claude/skills/control-clockwork. Do not read the repository, drive the app or write any file. Treat the source wave and the live pass as already run. A /verification:confirm report handed in one lesson: on the first page load of a new browser session the app shows a cookie banner across the bottom of the page, and a click on anything under it lands on the banner, so a drive has to dismiss it first. The live pass saw it at the start of all four features (weekly-timesheet-export, start-a-timer, invoice-email, team-report) and each drive worked once the banner was dismissed. Ready passed and nothing else differs. Name the file and section the lesson goes to, the outcome, and how it ships, then stop." | ||
| allowed_tools: [Read, Glob, Grep] | ||
| max_turns: 6 | ||
| timeout_seconds: 300 | ||
| graders: | ||
| - name: names-hub-gotchas | ||
| type: regex | ||
| pattern: 'SKILL\.md[\s\S]{0,120}Gotchas|Gotchas[\s\S]{0,120}SKILL\.md|\bhub''?s?\s+`?(## )?Gotchas' | ||
| flags: i | ||
| arm: both | ||
| - name: hub-gotchas-once | ||
| type: llm | ||
| criteria: "PASS only if the reply (1) puts the lesson in a `## Gotchas` section of the hub, .claude/skills/control-clockwork/SKILL.md, (2) does not copy it into the Gotchas of each feature file, (3) says it ships as a commit on control-skill-upkeep/control-clockwork through one pull request, with the outcome changed, and (4) does not route it to /corrections:codify or any instruction file outside the control skill. FAIL if it writes the lesson into each of the four feature files, writes it outside .claude/skills/control-clockwork/, calls the outcome clean, or hands it off instead of placing it." | ||
14 changes: 14 additions & 0 deletions
14
plugins/app-control/evals/map-lesson-every-feature/samples/hub-gotchas-once.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": "It holds for every feature, so it goes once in the `## Gotchas` section of the hub, .claude/skills/control-clockwork/SKILL.md, and in no feature file. It is doc drift: the outcome is changed and it ships as a commit on control-skill-upkeep/control-clockwork through one pull request.", | ||
| "why": "hub Gotchas once, changed outcome, upkeep branch and one pull request" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": "Add the banner note to the hub's `## Gotchas` in SKILL.md and to the `## Gotchas` of each of the four feature READMEs, so every drive sees it. Outcome changed; it ships on control-skill-upkeep/control-clockwork in one pull request.", | ||
| "why": "copies it into each feature file as well" | ||
| } | ||
| ] | ||
| } |
18 changes: 18 additions & 0 deletions
18
plugins/app-control/evals/map-lesson-every-feature/samples/names-hub-gotchas.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": "It holds for every feature, so it goes once in the hub's ## Gotchas in .claude/skills/control-clockwork/SKILL.md. Doc drift; the outcome is changed and it ships on control-skill-upkeep/control-clockwork.", | ||
| "why": "names the hub's Gotchas" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": "I would add the dismiss-the-banner step to the Drive of each of the four feature READMEs.", | ||
| "why": "copies it into every feature file" | ||
| }, | ||
| { | ||
| "answer": "Add the dismiss-the-banner note to each of the four feature READMEs' Gotchas and open a PR on GitHub.", | ||
| "why": "copies it into every feature; GitHub must not read as the hub" | ||
| } | ||
| ] | ||
| } |
18 changes: 18 additions & 0 deletions
18
plugins/app-control/evals/map-lesson-not-a-feature/case.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| schema_version: "1.1" | ||
| name: map-lesson-not-a-feature | ||
| description: Refresh writes nothing into the control skill for a lesson that is not about driving the app and leaves it for /corrections:codify | ||
| tags: [map, refresh, lessons, invented] | ||
| expected_outcome: "The reply says the dist/ staging lesson is not about a feature or a drive step, writes it into no file of .claude/skills/control-clockwork/ (neither a feature's Gotchas nor the hub's), reports it for /corrections:codify to place, says the outcome is clean because nothing else differs, names no instruction-placement or write-for-agents skill, and writes nothing." | ||
| execution: | ||
| prompt: "/app-control:map refresh .claude/skills/control-clockwork. Do not read the repository, drive the app or write any file. Treat the source wave and the live pass as already run; every feature drove as written and Ready passed. A /verification:confirm report handed in one lesson: agents in this repository keep staging the generated dist/ folder in their commits, and it should never be committed. Say where the lesson goes, what you would write and where, and the outcome, then stop." | ||
| allowed_tools: [Read, Glob, Grep] | ||
| max_turns: 6 | ||
| timeout_seconds: 300 | ||
| graders: | ||
| - name: names-codify | ||
| type: regex | ||
| pattern: '/corrections:codify' | ||
| arm: both | ||
| - name: hands-off-writes-nothing | ||
| type: llm | ||
| criteria: "PASS only if the reply (1) says the lesson is not about a feature or a drive step of the app, (2) writes or plans to write it into no file under .claude/skills/control-clockwork/, including the hub's and every feature's Gotchas, (3) leaves it for /corrections:codify to place, and (4) calls the outcome clean, with no upkeep commit or pull request for this lesson. FAIL if it puts the lesson in any Gotchas section or other control skill file, writes it into an instruction file itself, names any destination other than /corrections:codify, or opens a pull request for it." |
14 changes: 14 additions & 0 deletions
14
plugins/app-control/evals/map-lesson-not-a-feature/samples/hands-off-writes-nothing.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": "This lesson is about how agents work in this repository, not about a feature or a drive step, so I write nothing into .claude/skills/control-clockwork/. I report it for /corrections:codify, which places it. Every feature drove as written, so the outcome is clean: no upkeep commit and no pull request.", | ||
| "why": "not a feature lesson, nothing written, left for codify, clean" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": "Not a feature lesson, so I report it for /corrections:codify, and I also add 'never stage dist/' to the hub's `## Gotchas` so drives see it. That edit ships in the upkeep pull request on control-skill-upkeep/control-clockwork.", | ||
| "why": "names codify but still writes the lesson into the control skill and opens a pull request" | ||
| } | ||
| ] | ||
| } |
14 changes: 14 additions & 0 deletions
14
plugins/app-control/evals/map-lesson-not-a-feature/samples/names-codify.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": "Not a feature or drive-step lesson, so nothing goes into the control skill. I report it for /corrections:codify, which places it. Nothing else differs: the outcome is clean.", | ||
| "why": "hands the lesson to codify" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": "I would add 'never stage dist/' to the hub's Gotchas so drives see it, and ship it in the upkeep pull request.", | ||
| "why": "writes a non-feature lesson into the control skill" | ||
| } | ||
| ] | ||
| } |
18 changes: 18 additions & 0 deletions
18
plugins/app-control/evals/map-lesson-one-feature/case.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| schema_version: "1.1" | ||
| name: map-lesson-one-feature | ||
| description: Refresh places a lesson that holds for one feature in that feature's Gotchas and ships it in the upkeep pull request | ||
| tags: [map, refresh, lessons, invented] | ||
| expected_outcome: "The reply puts the wait-for-Export lesson in the `## Gotchas` section of .claude/skills/control-clockwork/features/weekly-timesheet-export/README.md, not in the hub or another feature, treats it as doc drift so the outcome is changed, ships it on control-skill-upkeep/control-clockwork in one pull request, does not hand it to /corrections:codify, and writes nothing." | ||
| execution: | ||
| prompt: "/app-control:map refresh weekly-timesheet-export. Do not read the repository, drive the app or write any file. Treat the source wave and the live pass as already run on .claude/skills/control-clockwork, the only control skill. A /verification:confirm report handed in one lesson: on the weekly timesheet page the Export button stays disabled for about two seconds after the week loads, and a click before it enables does nothing, so the drive has to wait for the button to enable. The live pass confirmed it: the export worked once the step waited. No other feature has that button. Ready passed and nothing else differs. Name the file and section the lesson goes to, the outcome, and how it ships, then stop." | ||
| allowed_tools: [Read, Glob, Grep] | ||
| max_turns: 6 | ||
| timeout_seconds: 300 | ||
| graders: | ||
| - name: names-feature-file | ||
| type: regex | ||
| pattern: 'features/weekly-timesheet-export/README\.md[\s\S]{0,120}Gotchas|Gotchas[\s\S]{0,120}features/weekly-timesheet-export/README\.md' | ||
| arm: both | ||
| - name: feature-gotchas-in-pull-request | ||
| type: llm | ||
| criteria: "PASS only if the reply (1) puts the lesson in the `## Gotchas` section of features/weekly-timesheet-export/README.md, (2) does not also put it in the hub (SKILL.md) or any other feature file, (3) says it ships as a commit on control-skill-upkeep/control-clockwork through one pull request, with the outcome changed, and (4) does not route it to /corrections:codify or any instruction file. FAIL if it places the lesson in the hub's Gotchas, writes it into a file outside .claude/skills/control-clockwork/, calls the outcome clean, or hands it off instead of placing it." |
14 changes: 14 additions & 0 deletions
14
...ins/app-control/evals/map-lesson-one-feature/samples/feature-gotchas-in-pull-request.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": "The lesson goes in the `## Gotchas` section of .claude/skills/control-clockwork/features/weekly-timesheet-export/README.md: wait for the Export button to enable before clicking it. Nothing goes in the hub or another feature. It is doc drift, so the outcome is changed, and it ships as a commit on control-skill-upkeep/control-clockwork through one pull request.", | ||
| "why": "one feature's Gotchas, changed outcome, upkeep branch and one pull request" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": "The lesson goes in the `## Gotchas` of .claude/skills/control-clockwork/features/weekly-timesheet-export/README.md and also in the hub's `## Gotchas` in SKILL.md so every drive sees it. Outcome changed; it ships on control-skill-upkeep/control-clockwork in one pull request.", | ||
| "why": "also copies a one-feature lesson into the hub" | ||
| } | ||
| ] | ||
| } |
18 changes: 18 additions & 0 deletions
18
plugins/app-control/evals/map-lesson-one-feature/samples/names-feature-file.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": "The lesson goes in the Gotchas of .claude/skills/control-clockwork/features/weekly-timesheet-export/README.md: wait for Export to enable before clicking. It is doc drift, so the outcome is changed and it ships on control-skill-upkeep/control-clockwork.", | ||
| "why": "names the one feature's file" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": "The lesson goes in the hub's Gotchas in .claude/skills/control-clockwork/SKILL.md so every drive sees it.", | ||
| "why": "places a one-feature lesson in the hub" | ||
| }, | ||
| { | ||
| "answer": "Open features/weekly-timesheet-export/README.md and add a wait-for-Export step to its Drive.", | ||
| "why": "names the right file but no Gotchas section" | ||
| } | ||
| ] | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.