Skip to content

feat(skill): keep non-ACS fixes out of acs-governed run labels. - #325

Open
alex ngo (ango10) wants to merge 1 commit into
mainfrom
ango10/skill-non-acs-fix-guidance
Open

feat(skill): keep non-ACS fixes out of acs-governed run labels.#325
alex ngo (ango10) wants to merge 1 commit into
mainfrom
ango10/skill-non-acs-fix-guidance

Conversation

@ango10

Copy link
Copy Markdown
Collaborator

Summary

Adds one piece of guidance to the run-assert-eval skill: a non-ACS fix (model upgrade, system-prompt rewrite, agent change) must not be recorded under an acs-governed run id. Doc-only, applied identically to all three front-doors.

Motivation / linked issue

No issue. This came out of running the skill end-to-end on a real internal eval.

Step 8 asks the agent to govern a failure with ACS and re-measure to prove the rate dropped. While troubleshooting, a coding agent can reach for a non-ACS fix instead: such as swapping the target model. Those are frequently the right fix, but they are agent changes, not governance. ACS can only restrict or gate an output, it can never add a capability the model lacks.

Nothing in the skill drew that line. So a governed re-measure could bank a model upgrade's improvement under an acs-governed run id and hand ACS credit for a delta it didn't produce, which corrupts exactly the comparison govern-and-remeasure.md exists to make.

Changes

  • .claude/skills/run-assert-eval/SKILL.md: new paragraph after the "Suggested next step" guidance.
  • .cursor/rules/assert.mdc: same guidance as a bullet in the matching list.
  • .github/prompts/run-assert-eval.prompt.md: same guidance in the matching section.

All three say the same thing: non-ACS fixes are legitimate and are the user's call, but they're outside this skill's loop, and the run should be named for what actually produced the numbers (e.g. model-upgrade) rather than labeled acs-governed.

Testing

  • pytest .claude/skills/run-assert-eval/tests/ -x -q: 68 passed (the command CI runs in regression.yml).
  • Confirmed these are the only three front-doors: Suggested next step appears in exactly these files repo-wide, so no fourth surface is left inconsistent.
  • Verified the guidance is self-contained: it references no example, config, or fixture, so it reads correctly on its own.

Checklist

  • Tests pass locally (pytest and/or viewer checks as applicable).
  • Docs updated if behavior or public API changed.
  • No secrets, credentials, or customer data committed.
  • No breaking change, or a CHANGELOG.md entry is included.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant