fix(gate): replay a script gate's evals, and name its script by file not address - #156
Merged
Merged
Conversation
…not address Two things a policy shipping a `kind: script` gate hit the moment it was adopted from a catalog. `chock check --only evals` replays a staged-files case in a throwaway repository holding only the case's own files. That is whole for a declarative gate, whose JSON is the entire check, and empty-handed for a script gate, whose program lives under the policy's implementations/ and is resolved from the repository root: every case observed "not installed" and refused, so a suite could only pass by expecting block. The runner now copies the policy's implementations/ to where the compiled gate names the script, as `chock sync` would have, before running the case. Copied, never staged: the material a case puts before the gate stays the files it lists. The ambient line rendered the compiled `script` param, the file's path from the repository root, which differs between a catalog tree (base/<id>/...) and an adopter (.agents/policies/<id>/...). The packaged SKILL.md carries that line, so `chock plugin build --check` could not be clean in both places at once. The bare file name is rendered now, which is what the manifest declared. The stability-script golden moves with it: an emitter change, so the next release is a minor one under the stability rule. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two things a policy shipping a
kind: scriptgate (0.9.3) hits the moment it is adopted from a catalog. Found by movingjava-securityin open-coder-ai/chock-catalog onto the script gate against the v0.9.3 pin.chock check --only evalsreplays a staged-files case against the compiled gate in a throwaway repository holding only the case's own files. That is whole for a declarative gate, whose JSON is the entire check, and empty-handed for a script gate, whose program lives under the policy'simplementations/and is resolved from the repository root: every case observedscript gate: '...' is not installed -- refusingand blocked, so a suite could only pass by expectingblock(25 cases: 13 false passes, 12 fails).execute.pynow copies the policy'simplementations/to where the compiled gate names the script, aschock syncwould have, before running the case. Copied, never staged: the material a case puts before the gate stays the files it lists, and a test pins that a script which would refuse its own text does not refuse every case.SKILL.mddiffered by address. The ambient line rendered the compiledscriptparam, the file's path from the repository root, which isbase/<id>/implementations/x.pyin a catalog tree and.agents/policies/<id>/implementations/x.pyonce adopted.chock plugin build --checktherefore reportedplugin_driftin the adopter for a package that was clean in the catalog.advisory.pyrenders the bare file name, which is what the manifest declared, so one packaged skill is right in both places.Docs:
spec/gate-dsl.md(both behaviours),CHANGELOG.mdunder Unreleased.Definition of done
chock check→ 0 errors, 0 warnings (the 5 infos are pre-existing onmain)chock check --only matrixpasses (32 invariants); no behavior change to the matrixchock sync --repo . --checkcleanchock check --only verifycleanpytest -qgreen: 1428 passed, 2 skipped (26 script-kind tests, 6 new: replay allow/block throughrun_case, the script is never in the writes, and the ambient line is identical for the policy at.agents/policies/and under a catalog tree)pytest acceptance/ -c acceptance/pytest.ini --rootdir=acceptancegreen: 21 passedscript; nothing to migratestability-scriptgolden'sambient-rule/ambient.mdis regenerated withCHOCK_REGEN_GOLDENS=1(one line: the script param); no other golden changedruff check .andruff format --check .cleanEmitter output changes for the script kind's ambient line, so under CONTRIBUTING's stability rule the release carrying this is a minor one, even though the kind shipped yesterday and nothing published uses it yet. A maintainer may reasonably call it a patch on that ground; the golden regeneration is in this PR either way.
Claims
INSTALLED_SURFACES, the coverage table anddocs/enforcement-surfaces.mdare untouched: neither change alters what is emitted or installed, only what an eval replays against and what one rendered line says.🤖 Generated with Claude Code
Generated by Claude Code