Skip to content

fork-kit/review: one general review skill; tiers, failure table, nit-only check and a drift test - #845

Merged
scuttletoby merged 6 commits into
mainfrom
fork/kit-review-one
Nov 22, 2026
Merged

scuttletoby merged 6 commits into
mainfrom
fork/kit-review-one

Conversation

@scuttletoby

Copy link
Copy Markdown
Owner

Audit #834, item G6: one review skill in the fork kit; a project keeps only its lens config.

What and why

Two review skills drifted apart: this kit's fork-kit/review and the one a downstream project's agents actually use
(that project's own skill, its order and lens-comment pages, and its eval folder). This PR makes the kit's
SKILL.md the one general skill. Every general rule of the project copy is merged in; the
project copy can then shrink to a thin adapter (a follow-up PR in that project, not here).

No code change to the gate or the kit tools: only the skill text, COMMENTS.md (two new checked examples), REVIEW.md,
the eval lens text and one new test class.

Drift table (section by section)

G = general (now in the kit), P = project fact (stays in the project), C = conflict (pick below).

Section Project skill Kit before Kind Now
Tiers tiers 0-3 by what the change touches, tier per module in a registry file, "unsure: higher tier" weights critical/normal/light from the review block C + P the gate's weight is the tier; table with defaults; a project maps its own tier names onto the three weights in its rules (P)
Tier 1 = CI only, no review owner decision light = CI only G light row
No-CI projects ("light" in other projects' prose: one Opus review) - fallback: normal, one lens C named: no queue config = normal, one lens, by hand
Author: why, root cause, where else, red-to-green yes yes G kept
Failure table (one row per write step: crash, rerun, old/empty start; skips and fallbacks) yes one sentence G added to §2, plus "a lens checks the table; a missing row is a finding"
Size (~400 changed lines, else a stack) yes - G added
Draft until lenses are done yes - P (merge path) "a project may open drafts"
CI side by side, lenses do not wait yes yes G kept
Outside reviewers extra, never required; which ones, when yes - G / P rule in kit; the list stays in the project
Lenses get the problem as reported, no summaries yes yes G kept, plus "no other lens's report"
Batch per area (<= ~1500 lines, one comment per PR) yes - G added
Verify, one push per round yes yes G kept
At most 2 rounds, then the orchestrator / override, then one audit yes yes (lens_rounds) G kept
Nit-only quick check (nit-only: yes/no) yes - G added; COMMENTS.md example checked
Minors into one follow-up issue yes, with the project's labels yes (minors) G / P kept; labels are the project's
Ready twice, auto-merge, push nothing after the project's ready command - P "the queue config's ready_command; push nothing after"
State question "...existing/old/empty state (rerun, half-applied, legacy data, empty history)", answered in checks: default question, answered on state: C state: line; question stays the kit setting state_question
Lens list correctness, design, tests + item/data integrity, exploits, concurrency, performance, game client, domain expert, visual correctness, design, tests, data, security, concurrency G / P added performance, domain expert, visual, forged messages under security; item/game-client lenses stay the project's (domain lenses)
Attacks leave failing tests yes yes G kept
Twin lens names (DI-1, DI-2), focus: runs twice and first yes - G added
covered: rule (not listed = not reviewed; every changed file in some lens's final covered) yes - G added
Model table critical lenses 2x Opus xhigh; others Opus high; Fable audit; Fable never a lens runner Opus high; audit Fable high; low/max forbidden C + P default stays Opus high (the owner's tier rule); xhigh for twin lenses is a project's lens_runners; "audit model never a lens" and "a cheaper model never the only reviewer" added
Comment format finding: <sev> ..., findings: none, no state: line state:, blocker:, major:, minor: C state: + blocker:/major:/minor: (both shapes parse; see picks)
After-post check of the live comment project command review_kit.py check before posting G / P both: kit check before posting, a project's own check right after
Flagged comment handling (not a lens:) yes yes G kept
Gate comment rules page project page COMMENTS.md, tested C COMMENTS.md; the project's lens_format_doc should point there (REVIEW.md)
Risk labels on follow-up issues project labels - P stays in the project
Eval method, prompt, judge prompt /tmp paths, project names general, {workdir}, harness review_eval.py G kit's; merged the concurrency part and the covered: rule into the eval lens
Eval truth set and case briefs project data example config P stay in the project, read by the kit harness through its eval config
Eval process reaper scripts host-specific - P stays with the host, not in the kit

Picks for the conflicts (what the gate parses decides)

  • Tier = the gate's weight. The review block's rules (weight per path glob) are what the gate reads, so a
    separate tier key would be a second source. Defaults: min_lenses 2/1/0 from review.recommended.json, runner
    Opus high. The new test pins the skill's table to the gate's WEIGHTS and those counts.
  • State answer on a state: line, not in checks:. The gate parses state_line (not enforced yet), the kit's
    check requires it, and T3's lens prompt (Review rules T3a: lens_start plan, prompt, refusals and request ids (pure logic) #825) asks for "a line starting state:". COMMENTS.md now shows the
    older shape (state only in checks:) and that check refuses it.
  • Findings as blocker: / major: / minor: lines. Both shapes parse the same in the gate
    (finding: <sev> lines and findings: none count as a findings section), but the gate's release note asks lens
    prompts to write blocker: none and major: none, and Review rules T3a: lens_start plan, prompt, refusals and request ids (pure logic) #825 writes the same. finding: lines stay documented as
    read.
  • Default runner Opus high for critical lenses, not xhigh: the owner's tier rule is two independent Opus high
    lenses; a project that wants xhigh for its twin lenses sets lens_runners.
  • The gate's light (CI only, no lens) keeps its meaning; "light" for projects without CI is named "no queue
    config: normal, one lens".

Config keys (unchanged; now documented as the tier source)

  • Queue config review block (read by the gate, keys in the pr-queue README "Review rules"): rules[].weight is
    the tier per path pattern; min_lenses the per-tier lens count; lenses, rounds, findings,
    override_authors, state_line. Also lens_format_doc (only quoted): point it at the kit's COMMENTS.md.
  • .github/review-kit.json (kit only): runner, lens_runners, audit_runner, forbidden_efforts,
    lens_rounds, minors, state_question, queue_config. REVIEW.md now has a "what stays in the project" table.

Test

fork-kit/review/tests/comments.test.py, new class SkillFormat:

  • every report template the skill (§6), the eval lens and COMMENTS.md ("The shape") tell a lens to write is filled
    in as a clean read and fed through the gate's own parser (lens.py, review_judge.py via review_kit.verdict):
    it must read read, clean, state, and check must exit 0;
  • the same template without its state: or blocker: line reads no state / no findings section;
  • the state question is word for word in SKILL.md, eval/lens.md and REVIEW.md;
  • the skill's tier table rows are the gate's WEIGHTS, with the lens counts of review.recommended.json.

Red checks done by hand: removing the state: line from the skill template, changing the normal tier's count,
dropping the lightest choice from the eval lens's finding: placeholder, and rewording the state question each
turned the class red. vp fmt --check fork-kit/review is clean.

python3 -I -B fork-kit/review/tests/comments.test.py   # 31 tests OK
python3 -I -B fork-kit/review/tests/plan.test.py       # OK
python3 -I -B fork-kit/review/tests/eval.test.py       # OK

State question for this change: the docs and the test hold no state; the test reads files only and gives the same
answer on every run.

General vs project

General: all of it. No project names, paths or machines; examples use made-up lens names and paths. Project facts
(its tier per module mapped to weights, its lens list, its domain lenses such as item duplication attacks, its
commands, labels and outside reviewers) stay in that project's review block, lens prompts, kit settings and a thin
adapter skill, done in a follow-up PR in that project after it moves its fork-ci pin to a release that has the
review rules.

Left

  • The downstream project's adapter PR and its fork-ci pin move (separate PRs there).
  • state_line is parsed but not enforced by the gate; enforcing it is gate work, not part of this PR.

🤖 Generated with Claude Code

…only check and a drift test

The kit's SKILL.md becomes the one general review skill, so a project keeps only its facts (review block, lens
prompts, kit settings) and a thin adapter. Merged in from the project skill: the tier table as the gate's weights,
the author's failure table, nit-only quick check, batch per area, twin lens names, focus lenses, covered rule, the
extra general lenses, and a concurrency part in the eval lens. Conflicts go to what the gate parses: the state answer
on a `state:` line and `blocker:`/`major:`/`minor:` lines.

New test (comments.test.py, SkillFormat): every report shape the skill, the eval lens and COMMENTS.md tell a lens to
write reads as a clean read with a state line through the gate's own parser; the skill's tiers match the gate's
weights and recommended min_lenses; the state question is word for word in the skill, the eval lens and REVIEW.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

not a lens: Lens A (#845), review notes for head bb08b8e (a plain note, so the gate does not count it as a lens read)

Verdict: NEEDS FIX (one major, five minors). The skill matches the gate on tiers, lens counts, comment format, overrides, twin-lens names and the reads-nothing rules, but its nit-only quick check runs into a gate rule the skill never mentions.

Major

  1. The nit-only quick check counts as a gate round and makes the approach lens required; the skill never mentions the approach lens. fork-kit/review/SKILL.md:78-80 (and §3 step 7 as a whole).
    The gate counts a round as every lens header that read a different own change (review_judge.review_round, fork-ci/actions/pr-queue/review_judge.py, the review_round function). The quick check is a lens comment on the fix head, so it is a new round. The common case is round 1 on A, round 2 on fix B, then a nit-only fix C with a quick check on C. That is gate round 3, which is the recommended rounds.approach_at (3). In enforce the approach lens then becomes required (fork-ci/actions/head-check/head_check.py:363-364). The orchestrator's override from step 7 does not supply a missing required lens (head_check.py:214,229-232: missing = required key not in reads). It only waives the approach lens when its author is in override_authors (decided), and that list is empty by default. So an agent that follows the skill exactly has its head held for a lens named approach that the skill does not mention.
    Reproduced with the gate's own judge_rounds on review.recommended.json set to enforce, with history code@A, code r2@B, quick-check@C. Result: round 3 of 4, approach lens approach required from round 3.
    Fix: in §3 say that every lens read of a new head is a gate round (the quick check too), and name rounds.approach_at and approach_lens, the lens the gate adds from that round. Allow the quick check only while the next round stays below approach_at; past that, the orchestrator checks the fix itself and posts the override, as the skill already says for "the rest". Or tell the project to keep approach_at above lens_rounds + 1. The plan already prints asks for the lens 'approach' from round 3, but nothing in the skill says what that means.

Minor

  1. focus: <lens> is undefined in the kit and the gate. SKILL.md:116-118. Nothing in fork-kit/ or fork-ci/ reads or defines focus:, and "the owner asked" is one project's own phrasing. Either define it in general terms (a person asks for a lens to run twice: give it two names in the review block) or move it to the project's adapter skill.
  2. The tier table leaves out the built-in protected paths. SKILL.md:28 (light row: "comments, config values with no code path change"). The gate always counts the queue config, every lens prompt file and .github/workflows/** as critical (review_judge.protected_paths / is_protected). The light row reads as if config changes are light. "Comments" is not a path either: a comment-only change under src/ takes its path's tier. Add one line: "the queue config, lens prompts and workflows are always critical".
  3. The not a lens: way out is incomplete. SKILL.md:171-172 says "no line below it to hold lens:". The gate also refuses a not a lens: comment that has an unquoted hex read at on any line (lens.py OPT_OUT_READ, read_problem). So an edit that keeps reviewer: x · read at: <sha> without the word lens: still reads nothing. COMMENTS.md states the full rule, so add "and no read at sha outside a > quote".
  4. check exit 1 for a real finding. SKILL.md:131 and SKILL.md:168. A lens with a real blocker or major always gets exit 1 from check. Nothing says that is expected, and an agent told to "check before posting" may soften the finding to get exit 0. Add: "a report with a blocker or major exits 1 by design: post it when the words read read, blocker|major, state".
  5. Test coverage of the tier table. fork-kit/review/tests/comments.test.py (test_the_skill_tiers_are_the_gates_weights_and_recommended_lens_counts) pins the tier names and lens counts. It does not pin the "Opus 5.5 high" runner text in the same table to review_kit.DEFAULTS['runner'], so a change to the kit default would leave the table stale. Low impact.

Checked and fine

  • Parser: SkillFormat uses review_kit.verdict and check, which import the gate's lens.py, review_judge.py and config.py (asserted by Doc.test_the_parser_is_the_gates), never a copy. Drift is caught both ways. A doc template that stops reading read, clean, state fails, and so does a gate parser change that stops accepting blocker:/major:, finding: or findings: none. Every doc must still show a template (assertTrue(found)). A new or renamed gate weight fails the tier-row test (list(tiers) == list(WEIGHTS)).
  • The skill against the gate: heaviest weight decides, first matching rule takes a path, min_lenses 2/1/0 as config-time floors, overrides cover reads and clear findings only for override_authors, a lens with a blocker or major is held until it reads a fixed head, twin names are distinct lens_keys, an edited or hidden lens comment reads nothing until a new post, and a lens-looking comment that reads nothing keeps blocking until it is deleted or opted out. All match lens.py, review_judge.py and head_check.py.
  • The state question:
    • A config with no review block: the plan falls back to normal with one review lens and says so, and the gate runs no rounds or findings (its required-lenses input decides).
    • Old finding: lines: the gate reads them as a findings section, check refuses them without a state: line, and COMMENTS.md has a checked example of this.
    • A wrong tier name in weight: the gate refuses the config and plan exits 2 (fails closed).
    • A project tier mapped to a lighter weight: silent, but that is the project's config, which the gate decides by.
    • The new test holds no state and gives the same result on every run.
  • Public and general: no names, IPs, VM ids, home paths or game lenses in the diff. Example lens names and paths are made up. The only project leftovers are focus: and "owner" (finding 2).
  • Length: SKILL.md is 172 lines. There is no overlap with docs/fork (only the reviewRules setting is documented there). The report shape now appears in both SKILL §6 and COMMENTS.md "The shape", but the test checks both, so they cannot drift apart.
  • Tests: at the head, run through heavy: comments.test.py 31 OK, plan.test.py 27 OK, eval.test.py 28 OK.

state: docs and a read-only test. Rerun, a stop half-way, and old, empty or missing review state are covered above; the one problem found is round counting (finding 1).

scuttletoby and others added 2 commits November 21, 2026 23:46
…d check exits

- A quick check is a lens read of a new head, so the gate counts it as a
  round; the skill names rounds.approach_at / approach_lens / max, how to run
  the approach lens, and that only an override by a listed override_authors
  login waives it.
- The tier table names the gate's always-critical paths (queue config, lens
  prompts, .github/workflows/**); the light row no longer says config is light.
- focus: removed (nothing reads it); twin lenses described in general terms.
- check exits 1 for a real blocker or major by design; never soften a finding.
- not a lens: also needs no read-at sha outside a quote (COMMENTS.md rule).
- Drift test reads the round keys and recommended approach_at through the
  gate's parser and judge_rounds, the protected paths from review_judge, and
  the runner defaults from review_kit.DEFAULTS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: A r2 · reviewer: lens agent (Opus 5.5 high) · read at: 6f0d833
covered: git diff bb08b8e2 6f0d8330 -- fork-kit/review in full (SKILL.md, COMMENTS.md, tests/comments.test.py); against the gate: review_judge.py (protected_paths, is_protected, judge, round_marker, review_round, judge_rounds), review_config.py (ROUND_KEYS, parse_review), lens.py (lens_key, read_problem, lens_reads, reviewed_head), head-check/head_check.py (verdict, the review-rounds block), lib/review_kit.py (DEFAULTS, check, run)
checks: the kit's three test files through heavy: comments 34 OK, plan 27 OK, eval 28 OK. Six mutations of the skill or the gate against comments.test.py (results below). A repro script that feeds the skill's quick-check flow to the gate's own lens.Review.reviewed_head and review_judge.judge_rounds (recommended rules, set to enforce). The not a lens: rule: five comment variants through the kit's check.
state: docs and a read-only test, so no stored state. Runs again: the tests hold no state and give the same result every run. Stops half-way: nothing is written. Old or empty state: a config with no review block still falls back to one review lens; the new tests read review.recommended.json through the gate's parser, so a broken file fails them instead of passing. The one problem is the quick-check flow below, which leaves a head held.
blocker: none
major: SKILL.md:87-92 now sets the quick check against the override ("or skip the quick check and use the override below, which adds no round"), but a quick check alone never lets the head through, and the skill does not say the override is needed after it. Details below.
minor: tests/comments.test.py:352 checks for a new protected rule only through upper-case names in is_protected's source. Details below.
minor: SKILL.md:83 says "a r3 marker in a lens name counts too". The gate takes the higher of the change count and the marker (review_round), not the sum. COMMENTS.md says it right ("whichever is higher"), so use the same words here.

Result: NOT SAFE to merge as is (one major, two minors). Every round-1 finding is fixed at its root.

Major: a quick check alone holds the head

The gate passes a head only when every lens's newest read names that one head, or a newer override covers it (lens.py reviewed_head, lines 510-517; head_check.verdict). The quick check posts under its own lens name (quick-check in COMMENTS.md:107), so the lenses of the earlier round still name the old head. Reproduced with the gate's own code, history code@A, code r2@B, fix head C:

comments on the pull request reviewed_head rounds
quick-check@C the lenses did not all read one head: code B, quick-check C round 3, approach required
quick-check@C + approach@C (what the skill now says to do) the lenses did not all read one head round 3, approach required
quick-check@C + approach@C + override for C every lens read it round 3, approach required
override for C only every lens read it round 2

So an agent that follows step 7 (quick check, nit-only: yes, and the approach lens at round approach_at) still has its head held. The override is needed on both paths. Skipping the quick check saves a round and the approach lens; it does not replace the override.

The old text was already unclear ("The rest the orchestrator checks itself"). The fix's new "or skip the quick check and use the override" makes the wrong reading explicit. COMMENTS.md:103 ("The nit-only: line is for the orchestrator") does not say what the orchestrator does on yes either.

Fix: in step 7 say that on nit-only: yes the orchestrator still posts the override for the fix head, because that is what covers the lenses that read the old head. The quick check only adds an independent read, at the cost of a round. Then reword the "or skip the quick check" clause so it does not suggest the override is the alternative. Add a test in test_the_skill_counts_rounds_as_the_gate that feeds this history to lens.Review(...).reviewed_head and pins that the quick check alone gives no head and the override gives head C. The current test checks only judge_rounds, which is why it missed this.

Minor: one gate change the protected-paths test does not catch

tests/comments.test.py:352 looks for a new kind of protected path only through upper-case constant names in inspect.getsource(review_judge.is_protected). I changed is_protected to add or path.startswith('.github/actions/'), and the test file still passed (34 OK). A rule written inline, with no new constant, gets past the test while the skill's "Always critical" list goes stale. One fix: compare is_protected's answer with the skill's list on a few probe paths outside that list (such as .github/actions/x.yml and .github/CODEOWNERS), so a new rule fails the test. Low impact.

Round-1 findings: each fixed at its root

  1. Quick check and the approach lens: SKILL.md:82-96 names rounds.approach_at, rounds.max and rounds.approach_lens, says the quick check is a gate round, and says an orchestrator's override never replaces the approach lens. Each statement matches review_round, judge_rounds and head_check.py:363-364. My round-1 repro (code@A, code r2@B, quick-check@C) still gives "round 3 of 4, approach lens approach required from round 3", which the skill now says. The new major above is in the same step but is a different gate rule.
  2. focus: and "owner" are gone from SKILL.md and COMMENTS.md (grep finds nothing). The twin-lens text is now general.
  3. SKILL.md:28-32: the light row no longer says config, and the "Always critical" list matches protected_paths and WORKFLOWS. These paths count for the first critical rule (judge), so calling them critical is right. "A comment-only change takes its path's tier" is right.
  4. SKILL.md:185-188 matches lens.py:398-400 (OPT_OUT_READ, unquoted_lines). Checked with the kit's check on five not a lens: variants: a quoted read-at sha and a quoted lens header give not a lens. An unquoted reviewer: x · read at: <sha>, an unquoted mid-line read at <sha> and one inside a code block give reads nothing.
  5. Exit 1 for a real blocker or major is stated at SKILL.md:144-145 and :181-183, with "never softened". run exits 2 only for a usage error, which prints no words, so "otherwise exit 1" is right for a read comment.
  6. test_the_skill_runner_text_is_the_kits_default reads review_kit.DEFAULTS and checks both the §5 text and every tier row that has lenses.

The new drift tests read the gate's real code and fail when the skill drifts

review_config is the gate's module (fork-kit/review/lib holds no copy). review_judge is review_kit.review_judge. The recommended rules go through review_config.parse_review. Results of the mutations:

mutation result
skill says approach_at is 4 FAIL (rounds test)
skill drops the workflows glob FAIL (protected test)
normal row says Fable 5.2 FAIL (runner test)
skill drops "the gate counts it as a round" FAIL (rounds test)
gate stops counting a quick check as a round FAIL (rounds test)
gate adds an inline protected rule passes (minor above)

All mutations were reverted. My temp checkout was clean afterwards.

scuttletoby and others added 2 commits November 22, 2026 03:03
…ck never does

The gate passes a head only when every lens's newest read names it or a newer
override covers them (lens.Review.reviewed_head). A quick check is one lens on
the fix head while the earlier lenses still name the old head, so it alone
holds the head. SKILL.md step 7 now says which head passes and that on
`nit-only: yes` the orchestrator still posts the override; COMMENTS.md says the
same. The round count says "whichever is higher", as review_round does.

Tests: a history (code at A, code r2 at B, fix C with a quick check) through the
gate's real reviewed_head and judge_rounds, pinned to the skill's words; the
protected-paths test now compares is_protected's answer with the skill's list
on every tracked file and on probe paths, so an inline rule fails it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: A r3 · reviewer: lens agent (Opus 5.5 high) · read at: a8c263c
covered: git diff 6f0d8330 a8c263c7 -- fork-kit/review in full (SKILL.md, COMMENTS.md, tests/comments.test.py); the rest of the head is a merge of main. Against the gate: lens.py (Review.override_sha, lens_reads, reviewed_head), review_judge.py (is_protected, protected_paths, WORKFLOWS, review_round), config.py (override), lib/review_kit.py (reader, the plan's rounds line), REVIEW.md
checks: the kit's three test files through heavy: comments 36 OK, plan 27 OK, eval 28 OK. Seven mutations of the gate or the docs against comments.test.py, each reverted (results below). The checkout was clean afterwards.
state: docs and read-only tests, so no stored state. Runs again: the new tests hold no state; repo_paths reads git ls-files fresh each run and gives the same answer. Stops half-way: nothing is written. Old or empty state: with no git (a tarball), repo_paths returns nothing and the 21 probe paths still run, so the test still fails on a new gate rule. A broken review.recommended.json fails the tests through the gate's parser, it does not pass them.
blocker: none
major: none
minor: SKILL.md:98 and :106 against REVIEW.md:63 and the plan's line (review_kit.py:223). The fix removed "The rest the orchestrator checks itself ... then posts the override". Now the skill does not say what happens to a fix that is not nit-only once lens_rounds is used up. Details below.
minor: SKILL.md:89 says the override "counts only from the account with" override.author_association. That setting is an author association (a kind of account, such as OWNER or COLLABORATOR), not one account. The gate also needs the comment to be never edited and not hidden (lens.py override_sha). COMMENTS.md:63 says it correctly ("the configured author association"). Use the same words, and add "never edited, not hidden".

Result: SAFE TO MERGE (no blocker, no major, two wording minors for the follow-up issue). Every round-2 finding is fixed at its root.

Minor: a fix that is not nit-only, after the last lens round

Before this round, step 7 ended with "The rest the orchestrator checks itself from the fix diff and the lenses' proof tests, then posts the override". REVIEW.md:63 (lens_rounds: "lens rounds before the orchestrator checks fix diffs itself") and the plan ("after that the orchestrator checks each fix diff itself") still say that. The new text gives the orchestrator's own check only to a nit-only fix. On nit-only: no it says "the fix goes to the next round" (SKILL.md:98), and :106 says "No round past the limit". With the default lens_rounds 2, a quick check after round 2 is already gate round 3, so "the next round" goes past lens_rounds. One case shows the gap: minors: same-round, round 2 is clean except for a minor, and the fix touches code. The skill does not say who checks that fix. Fix: add one sentence after the nit-only bullet: "Once lens_rounds is used up, the orchestrator checks any later fix that has no blocker or major left in the same way, and posts the override. A blocker or major left is stuck (below)."

Round-2 findings: each fixed at its root

  1. The quick check does not replace the override. SKILL.md:87-101 has a new "Which head passes" bullet, which states the gate's rule from lens.Review.reviewed_head and lens_reads (every lens's newest read names the head, or a newer override covers it). The "Nit-only fix" bullet now says the override is always needed, and the quick check only adds a read and a round. "skip the quick check" is gone. COMMENTS.md:103-105 says the same about yes. The new test test_the_skill_says_what_passes_a_fix_head_after_a_quick_check (comments.test.py:427) runs my round-2 history through the gate's real lens.Review (review_kit.reader builds lens.Review; no copy). Quick check alone: no head. Quick check with the approach lens: no head. Either one with the override: head C. Override alone: head C at round 2. It also runs judge_rounds for the round count, and it pins the skill's and COMMENTS.md's words.
  2. Protected paths. comments.test.py:379-387 now compares is_protected's answer, not its source, with the skill's list (the exact paths plus WORKFLOWS). It does this on every tracked file, 21 probe paths and the exact paths. My round-2 mutation (or path.startswith('.github/actions/')) now fails.
  3. "Whichever is higher". SKILL.md:82-84 matches review_round (max, not a sum). test_the_skill_says_whichever_is_higher checks the gate's function on one change plus an r3 marker, which gives 3 (a count alone would give 1, a sum 4). It pins the words too.

The new tests read the gate's real code and fail when the docs drift

mutation result
gate: is_protected also takes .github/actions/ (inline prefix) FAIL (7)
gate: is_protected also takes *CODEOWNERS (inline suffix) FAIL (1)
gate: reviewed_head passes the newest-looking sha when the lenses disagree FAIL (2: quick check alone, quick check and approach lens)
gate: review_round adds the marker to the count FAIL (3)
SKILL.md drops "it does not replace the override" FAIL (1)
SKILL.md says "Or skip the quick check and use the override" FAIL (2)
COMMENTS.md drops "on yes it still posts the override" FAIL (1)

No new error: the new bullets match override_sha (one association, never edited, not hidden, apart from the wording minor), lens_reads (an override covers only the reads older than it) and head_check (an orchestrator's override never waives a required approach lens). Every COMMENTS.md example still reads as the doc says (the comments test feeds each one to the gate).

@scuttletoby
scuttletoby merged commit 500289b into main Nov 22, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant