feat(skill): keep non-ACS fixes out of acs-governed run labels. - #325
Open
alex ngo (ango10) wants to merge 1 commit into
Open
feat(skill): keep non-ACS fixes out of acs-governed run labels.#325alex ngo (ango10) wants to merge 1 commit into
alex ngo (ango10) wants to merge 1 commit into
Conversation
alex ngo (ango10)
requested review from
Aaron Aspinwall (AaronAspinwall123),
Chang Liu (changliu2),
Jake Present (jakepresent) and
tangym
as code owners
August 19, 2026 23:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds one piece of guidance to the
run-assert-evalskill: a non-ACS fix (model upgrade, system-prompt rewrite, agent change) must not be recorded under anacs-governedrun id. Doc-only, applied identically to all three front-doors.Motivation / linked issue
No issue. This came out of running the skill end-to-end on a real internal eval.
Step 8 asks the agent to govern a failure with ACS and re-measure to prove the rate dropped. While troubleshooting, a coding agent can reach for a non-ACS fix instead: such as swapping the target model. Those are frequently the right fix, but they are agent changes, not governance. ACS can only restrict or gate an output, it can never add a capability the model lacks.
Nothing in the skill drew that line. So a governed re-measure could bank a model upgrade's improvement under an
acs-governedrun id and hand ACS credit for a delta it didn't produce, which corrupts exactly the comparisongovern-and-remeasure.mdexists to make.Changes
.claude/skills/run-assert-eval/SKILL.md: new paragraph after the "Suggested next step" guidance..cursor/rules/assert.mdc: same guidance as a bullet in the matching list..github/prompts/run-assert-eval.prompt.md: same guidance in the matching section.All three say the same thing: non-ACS fixes are legitimate and are the user's call, but they're outside this skill's loop, and the run should be named for what actually produced the numbers (e.g.
model-upgrade) rather than labeledacs-governed.Testing
pytest .claude/skills/run-assert-eval/tests/ -x -q: 68 passed (the command CI runs inregression.yml).Suggested next stepappears in exactly these files repo-wide, so no fourth surface is left inconsistent.Checklist
pytestand/or viewer checks as applicable).CHANGELOG.mdentry is included.