Skip to content

fix(gate): replay a script gate's evals, and name its script by file not address - #156

Merged
jothimani-rajendran merged 1 commit into
mainfrom
claude/script-gate-eval-replay
Sep 22, 2026
Merged

jothimani-rajendran merged 1 commit into
mainfrom
claude/script-gate-eval-replay

Conversation

@jothimani-rajendran

Copy link
Copy Markdown
Collaborator

What

Two things a policy shipping a kind: script gate (0.9.3) hits the moment it is adopted from a catalog. Found by moving java-security in open-coder-ai/chock-catalog onto the script gate against the v0.9.3 pin.

  1. A script gate's evals could not pass. chock check --only evals replays a staged-files case against the compiled gate in a throwaway repository holding only the case's own files. That is whole for a declarative gate, whose JSON is the entire check, and empty-handed for a script gate, whose program lives under the policy's implementations/ and is resolved from the repository root: every case observed script gate: '...' is not installed -- refusing and blocked, so a suite could only pass by expecting block (25 cases: 13 false passes, 12 fails). execute.py now copies the policy's implementations/ to where the compiled gate names the script, as chock sync would have, before running the case. Copied, never staged: the material a case puts before the gate stays the files it lists, and a test pins that a script which would refuse its own text does not refuse every case.
  2. A script gate's packaged SKILL.md differed by address. The ambient line rendered the compiled script param, the file's path from the repository root, which is base/<id>/implementations/x.py in a catalog tree and .agents/policies/<id>/implementations/x.py once adopted. chock plugin build --check therefore reported plugin_drift in the adopter for a package that was clean in the catalog. advisory.py renders the bare file name, which is what the manifest declared, so one packaged skill is right in both places.

Docs: spec/gate-dsl.md (both behaviours), CHANGELOG.md under Unreleased.

Definition of done

  • chock check → 0 errors, 0 warnings (the 5 infos are pre-existing on main)
  • chock check --only matrix passes (32 invariants); no behavior change to the matrix
  • chock sync --repo . --check clean
  • chock check --only verify clean
  • Registry rescanned; no stale entries
  • pytest -q green: 1428 passed, 2 skipped (26 script-kind tests, 6 new: replay allow/block through run_case, the script is never in the writes, and the ambient line is identical for the policy at .agents/policies/ and under a catalog tree)
  • pytest acceptance/ -c acceptance/pytest.ini --rootdir=acceptance green: 21 passed
  • Existing artifacts: no policy in this repo uses script; nothing to migrate
  • Touched manifests: none. The stability-script golden's ambient-rule/ambient.md is regenerated with CHOCK_REGEN_GOLDENS=1 (one line: the script param); no other golden changed
  • ruff check . and ruff format --check . clean

Emitter output changes for the script kind's ambient line, so under CONTRIBUTING's stability rule the release carrying this is a minor one, even though the kind shipped yesterday and nothing published uses it yet. A maintainer may reasonably call it a patch on that ground; the golden regeneration is in this PR either way.

Claims

  • No surface is described as enforcing more than it installs. INSTALLED_SURFACES, the coverage table and docs/enforcement-surfaces.md are untouched: neither change alters what is emitted or installed, only what an eval replays against and what one rendered line says.

🤖 Generated with Claude Code


Generated by Claude Code

…not address

Two things a policy shipping a `kind: script` gate hit the moment it was
adopted from a catalog.

`chock check --only evals` replays a staged-files case in a throwaway
repository holding only the case's own files. That is whole for a
declarative gate, whose JSON is the entire check, and empty-handed for a
script gate, whose program lives under the policy's implementations/ and is
resolved from the repository root: every case observed "not installed" and
refused, so a suite could only pass by expecting block. The runner now
copies the policy's implementations/ to where the compiled gate names the
script, as `chock sync` would have, before running the case. Copied, never
staged: the material a case puts before the gate stays the files it lists.

The ambient line rendered the compiled `script` param, the file's path from
the repository root, which differs between a catalog tree (base/<id>/...)
and an adopter (.agents/policies/<id>/...). The packaged SKILL.md carries
that line, so `chock plugin build --check` could not be clean in both
places at once. The bare file name is rendered now, which is what the
manifest declared. The stability-script golden moves with it: an emitter
change, so the next release is a minor one under the stability rule.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Claude <noreply@anthropic.com>
@jothimani-rajendran
jothimani-rajendran marked this pull request as ready for review September 22, 2026 14:38
@jothimani-rajendran
jothimani-rajendran merged commit 246a4c9 into main Sep 22, 2026
19 checks passed
@jothimani-rajendran jothimani-rajendran mentioned this pull request Sep 22, 2026
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants