Repository navigation
Commit 5230ec3
authored
fix(agent-context): tune prompts and codegen skill triggers with eval set (#438)
* feat(agent-context): trigger evaluation for the prompts and codegen skill descriptions
Add should-trigger and should-not-trigger cases (12 each) for metaobjects-prompts and
metaobjects-codegen, a runner that wraps the skills in a throwaway plugin for
claude plugin eval, and a CI test that the cases parse and name existing skills.
First run: codegen 36/36, prompts 32/36 (one run of an entities-only near-miss
fired the skill). Narrow the prompts description to projects that declare
template.prompt; codegen needed no change. Rerun: 36/36 and 36/36.
* no-mistakes(review): Scope prompts skill gate; add adoption eval case
* no-mistakes(test): Tune codegen skill description to trigger on dialect config
* chore(agent-context): drop the parse-only trigger-case test; the evaluation is manual-only
A test that only parses the case files proves nothing about trigger behaviour, so the cases
and the documented run command are the evidence. README no longer claims a CI check.1 parent 66056ae commit 5230ec3
64 files changed
Lines changed: 435 additions & 12 deletions
File tree
- agent-context
- evals
- skill-triggers
- metaobjects-codegen
- 01-pos-write-openapi-generator
- graders
- 02-pos-eject-reference-generator
- graders
- 03-pos-configure-generators
- graders
- 04-pos-regen-hand-edits
- graders
- 05-pos-json-schema-output
- graders
- 06-pos-dialect-config
- graders
- 07-neg-add-entity-field
- graders
- 08-neg-verify-failure
- graders
- 09-neg-openapi-no-metaobjects
- graders
- 10-neg-prisma-codegen
- graders
- 11-neg-prompt-template
- graders
- 12-neg-runtime-query
- graders
- metaobjects-prompts
- 01-pos-fstring-prompt
- graders
- 02-pos-declare-template-prompt
- graders
- 03-pos-regex-reply-parse
- graders
- 04-pos-test-harness-prompt
- graders
- 05-pos-prompt-drift-verify
- graders
- 06-pos-review-json-parse
- graders
- 07-neg-fstring-no-metaobjects
- graders
- 08-neg-langchain-system-prompt
- graders
- 09-neg-json-reply-no-metaobjects
- graders
- 10-neg-entities-only-project-prompt-string
- graders
- 11-neg-add-entity-field
- graders
- 12-neg-non-llm-string-building
- graders
- 13-pos-adopt-first-prompt
- graders
- skills
- metaobjects-codegen
- metaobjects-prompts
- fixtures/agent-context-conformance
- java-kotlin-react-tanstack/expected/.claude/skills
- metaobjects-codegen
- metaobjects-prompts
- java-react/expected/.claude/skills
- metaobjects-codegen
- metaobjects-prompts
- python/expected/.claude/skills
- metaobjects-codegen
- metaobjects-prompts
- ts-react-tanstack/expected/.claude/skills
- metaobjects-codegen
- metaobjects-prompts
- ts-requirements/expected/.claude/skills
- metaobjects-codegen
- metaobjects-prompts
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
0 commit comments