Skip to content

docs: correct what chock writes, grades and enforces - #161

Merged
jothimani-rajendran merged 3 commits into
mainfrom
claude/docs-refresh-0.11.0
Sep 23, 2026
Merged

docs: correct what chock writes, grades and enforces#161
jothimani-rajendran merged 3 commits into
mainfrom
claude/docs-refresh-0.11.0

Conversation

@jothimani-rajendran

@jothimani-rajendran jothimani-rajendran commented Sep 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

What

A full audit of README.md and every page under docs/ against the source, run in scratch repos rather than read off the previous prose. No behaviour changes: the only source edits are two comments and one help string.

The honesty claim was the least honest passage in the README

It said every policy × agent is graded enforced, enforced-at-commit or advisory, that a level is claimed only once there is an install witness, and that a native control "that fails closed" is enforced. All three are wrong:

  • COVERAGE_LEVELS has eight members, not three.
  • A git hook earns enforced-at-commit from emission alone; witnessed is false on every commit-time and advisory cell.
  • No agent reaches enforced. This repo's own .chock/coverage.json: 168 advisory, 72 enforced-at-commit, 41 best-effort, 5 enforceable, 0 enforced.
  • No client fails closed. fail_mode is open for all ten in-agent vendors except Cursor, which is configurable — which is exactly why Cursor is the only one reaching enforceable.

For a project whose pitch is not overclaiming enforcement, this was the wrong paragraph to be wrong in.

The example manifest would have failed the check printed beneath it

The README shows a manifest as the thing you write, directly above chock check # validate every artifact against the spec. It was missing version, provenance, lifecycle and the security baseline. Verified by writing it into a scratch repo and running the real check; it now passes.

The adapter pages named files chock never writes

Checked against chock init . --agent-agnostic in a clean repo. Seven of the documented agents read AGENTS.md natively and get no wrapper file at all — Cursor, Codex, Copilot, VS Code, Gemini, Windsurf, Kimi Code. Claude Code's wrapper is the repo-root CLAUDE.md, not .claude/CLAUDE.md. The files that are written are named agentseam.*, not chock.*. Junie was missing from both lists despite being written. Thirteen pages also carried a literal broken-markdown line with nested backticks.

getting-started promised enforcement the policy does not have

It said protect-main-branch stops Claude Code before it runs the command. That gate declares on: [commit, push], so it compiles no pre-tool surface: .chock/compiled/protect-main-branch/ holds ambient-rule, ci-gate, git-hook and managed-setting only. The same page misdescribed what init and new policy leave on disk and put chock.lock inside .chock/.

Everything else, each verified against source

Page Was Is
README, docs six stop vendors, eight surfaces, four plugin formats, two write vendors seven, nine, five, three
README agent table .github/hooks/agentseam.json chock.json — agentseam.json is agentseam's path, not the file chock owns
authoring-policies, security SEC-7 byte-matches rule.text repointed to chock check --only index, per spec/enforcement-matrix.md, which records the byte-match was never performed
rules/README rules compile into AGENTS.md into .agents/policies/INDEX.md; AGENTS.md keeps a pointer block
coverage-levels cells are bare strings objects with level, basis, witnessed — the page already said so 15 lines later
validation Ubuntu and Windows, Python 3.11/3.12 Linux-only by default across 3.11/3.12/3.13; Windows on a v* tag or manual dispatch
reviewer-evidence chock@v0.1.0 v0.11.0, matching action.yml
cli-reference, tools --only list missing conflicts and baseline; eval export shown as a check flag both documented; eval export is its own command
concepts, authoring five gate kinds, three trust tiers, subagents/ six kinds, four tiers, .agents/skills/<id>/
baseline-policies four declarative policies seven; four ship shell implementations
registry-and-lockfile source: local always add records catalog provenance; the next sync resets it — the two pages contradicted each other

Definition of done

  • chock check → 0 errors, 0 warnings, 0 infos
  • chock check --only matrix passes; no behaviour changed
  • chock sync --repo . --check clean
  • chock check --only verify clean
  • Registry rescanned; no stale entries
  • pytest -q green (1460 passed, 2 skipped)
  • pytest acceptance/ — not applicable, packaging/init/add/hook installation unchanged
  • Existing artifacts migrated — not applicable, no check added or extended
  • Touched manifests — none
  • ruff check . and ruff format --check . clean

Claims

  • No surface is described as enforcing more than it installs. Every change here narrows a claim toward what the code does; the grading paragraph and the getting-started promise were both overclaiming.

Worth a separate look: nothing in CI compares any of these numbers or paths against the source. The figure check stayed green the whole time the prose beside it drifted, and the adapter pages have been wrong for as long as those agents have read AGENTS.md natively.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CzNYfzP8ymU3r4JB9Sz8Ha

@claude
docs: bring the surface and vendor counts up to 0.11.0
4a36f98 
The 0.11.0 release added the stop surface and Cursor's write vocabulary, and
the figures regenerated, but the hand-written prose around them did not.

The README's alt text still described eight surfaces and omitted stop, so the
accessible text disagreed with the picture beside it; it now mirrors the
figure's own generated description. The stop count read six vendors where
stop_vendors returns seven. architecture.md and concepts.md both enumerated
eight surfaces without stop, against nine in the Surface enum.
enforcement-surfaces.md's callout had both halves wrong: stop wires seven
vendors, and the write path wires three now that Cursor records Write.

agent-plugins.md listed four plugin formats against five in HOOK_FORMATS,
conflating the format count with the four distribution repos. cli-reference.md
described a package as a guard and an adapter, which has been incomplete since
the gate, its runner and a script gate's implementations began riding along,
and it described Cursor as beforeShellExecution alone, with no mention of the
write gate or the turn's end.

The README also pointed Copilot and VS Code at .github/hooks/agentseam.json.
chock writes chock.json there; agentseam.json is agentseam's recorded vendor
path, not the file chock owns.

Two source comments drifted the same way: surfaces.py still listed cursor
among the agents that cannot refuse a finished turn, and both it and
in_agent.py miscounted the vendors with no write vocabulary. The cursor line
in plugin build's own output named only beforeShellExecution.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Claude <noreply@anthropic.com>
@claude
docs: the catalog holds 42 policies, not 39
066f780 
The ecosystem table's count was last true three policies ago; the catalog's
own README badge reads 42.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Claude <noreply@anthropic.com>
@claude
docs: correct what chock writes, grades and enforces
5ab5033 
A full audit of README.md and docs/ against the source, run in scratch repos
rather than read off the previous prose.

The honesty claim was the least honest passage in the README. It named three
grades where COVERAGE_LEVELS has eight, said a level is claimed only with an
install witness when a git hook earns its grade from emission alone, and
described 'enforced' as what a control that fails closed reaches. Nothing
reaches enforced: this repo's own coverage.json records 168 advisory, 72
enforced-at-commit, 41 best-effort, 5 enforceable and zero enforced, and every
in-agent hook fails open except Cursor's, which is configurable.

The example manifest a reader is told to write would have failed the check
shown directly beneath it: version, provenance, lifecycle and the security
baseline were all missing. Verified by running chock check against it.

The adapter pages named instruction files chock never writes. Seven of those
agents read AGENTS.md natively and get no wrapper at all; Claude Code's is the
repo-root CLAUDE.md, not .claude/CLAUDE.md; and the files that are written are
named agentseam.*, not chock.*. Checked against chock init --agent-agnostic.

getting-started promised that protect-main-branch stops an agent before it runs
a command. Its gate declares commit and push, so it compiles no pre-tool
surface at all. The same page misdescribed what init and new policy leave on
disk, and put chock.lock inside .chock/.

Also: SEC-7 repointed to what it actually checks, per spec/enforcement-matrix;
the rules page moved from AGENTS.md to INDEX.md; the coverage.json example
rewritten in the shape the compiler writes; the CI matrix corrected to
Linux-only across three Python versions; the action pin moved off v0.1.0; the
conflicts and baseline check targets documented; six gate kinds and four trust
tiers, not five and three.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Claude <noreply@anthropic.com>
@jothimani-rajendran jothimani-rajendran changed the title docs: bring the surface and vendor counts up to 0.11.0 docs: correct what chock writes, grades and enforces Sep 23, 2026
@jothimani-rajendran
jothimani-rajendran marked this pull request as ready for review September 23, 2026 21:21
@jothimani-rajendran
jothimani-rajendran merged commit 1c577c4 into main Sep 23, 2026
20 checks passed
@jothimani-rajendran jothimani-rajendran mentioned this pull request Sep 23, 2026
Merged
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jothimani-rajendran @claude