Skip to content

Commit 5230ec3

Browse files
authored
fix(agent-context): tune prompts and codegen skill triggers with eval set (#438)
* feat(agent-context): trigger evaluation for the prompts and codegen skill descriptions Add should-trigger and should-not-trigger cases (12 each) for metaobjects-prompts and metaobjects-codegen, a runner that wraps the skills in a throwaway plugin for claude plugin eval, and a CI test that the cases parse and name existing skills. First run: codegen 36/36, prompts 32/36 (one run of an entities-only near-miss fired the skill). Narrow the prompts description to projects that declare template.prompt; codegen needed no change. Rerun: 36/36 and 36/36. * no-mistakes(review): Scope prompts skill gate; add adoption eval case * no-mistakes(test): Tune codegen skill description to trigger on dialect config * chore(agent-context): drop the parse-only trigger-case test; the evaluation is manual-only A test that only parses the case files proves nothing about trigger behaviour, so the cases and the documented run command are the evidence. README no longer claims a CI check.
1 parent 66056ae commit 5230ec3

64 files changed

Lines changed: 435 additions & 12 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎agent-context/evals/README.md‎

Lines changed: 26 additions & 0 deletions
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
#!/usr/bin/env bash
2+
# Run the skill trigger evaluation (agent-context/evals/skill-triggers) with `claude plugin eval`.
3+
# Manual, not CI: it drives real model sessions on your own Claude login (subscription quota).
4+
#
5+
# usage: agent-context/evals/run-skill-triggers.sh [OUT_DIR] [extra `claude plugin eval` flags]
6+
# OUT_DIR defaults to a fresh temp dir; the scored result is written to OUT_DIR/result.json.
7+
# e.g. agent-context/evals/run-skill-triggers.sh /tmp/triggers --case 'metaobjects-codegen/*'
8+
set -euo pipefail
9+
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
10+
skills="$here/../skills"
11+
out="${1:-$(mktemp -d)}"; [ $# -gt 0 ] && shift
12+
mkdir -p "$out"
13+
plugin="$out/plugin"
14+
rm -rf "$plugin"
15+
mkdir -p "$plugin/.claude-plugin"
16+
# `claude plugin eval` refuses an evals/ link that points outside the plugin, so copy.
17+
echo '{"name":"mo-skills","version":"0.0.0","description":"MetaObjects skills, wrapped for trigger evaluation"}' >"$plugin/.claude-plugin/plugin.json"
18+
cp -R "$skills" "$plugin/skills"
19+
cp -R "$here/skill-triggers" "$plugin/evals"
20+
cd "$plugin"
21+
# --ablation none: one arm, so the `tool_used: Skill` graders count toward the score.
22+
exec claude plugin eval . --ablation none --no-publish --trust-plugin -j 4 --threshold 0 \
23+
--output-dir "$out/runs" --json "$out/result.json" "$@"
Lines changed: 6 additions & 0 deletions
Lines changed: 8 additions & 0 deletions
Lines changed: 6 additions & 0 deletions
Lines changed: 8 additions & 0 deletions
Lines changed: 6 additions & 0 deletions
Lines changed: 8 additions & 0 deletions
Lines changed: 6 additions & 0 deletions
Lines changed: 8 additions & 0 deletions

0 commit comments

Comments
 (0)