Repository navigation
fork-kit/review: one general review skill; tiers, failure table, nit-only check and a drift test - #845
Conversation
…only check and a drift test The kit's SKILL.md becomes the one general review skill, so a project keeps only its facts (review block, lens prompts, kit settings) and a thin adapter. Merged in from the project skill: the tier table as the gate's weights, the author's failure table, nit-only quick check, batch per area, twin lens names, focus lenses, covered rule, the extra general lenses, and a concurrency part in the eval lens. Conflicts go to what the gate parses: the state answer on a `state:` line and `blocker:`/`major:`/`minor:` lines. New test (comments.test.py, SkillFormat): every report shape the skill, the eval lens and COMMENTS.md tell a lens to write reads as a clean read with a state line through the gate's own parser; the skill's tiers match the gate's weights and recommended min_lenses; the state question is word for word in the skill, the eval lens and REVIEW.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
not a lens: Lens A (#845), review notes for head bb08b8e (a plain note, so the gate does not count it as a lens read) Verdict: NEEDS FIX (one major, five minors). The skill matches the gate on tiers, lens counts, comment format, overrides, twin-lens names and the reads-nothing rules, but its nit-only quick check runs into a gate rule the skill never mentions. Major
Minor
Checked and fine
state: docs and a read-only test. Rerun, a stop half-way, and old, empty or missing review state are covered above; the one problem found is round counting (finding 1). |
…d check exits - A quick check is a lens read of a new head, so the gate counts it as a round; the skill names rounds.approach_at / approach_lens / max, how to run the approach lens, and that only an override by a listed override_authors login waives it. - The tier table names the gate's always-critical paths (queue config, lens prompts, .github/workflows/**); the light row no longer says config is light. - focus: removed (nothing reads it); twin lenses described in general terms. - check exits 1 for a real blocker or major by design; never soften a finding. - not a lens: also needs no read-at sha outside a quote (COMMENTS.md rule). - Drift test reads the round keys and recommended approach_at through the gate's parser and judge_rounds, the protected paths from review_judge, and the runner defaults from review_kit.DEFAULTS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
lens: A r2 · reviewer: lens agent (Opus 5.5 high) · read at: 6f0d833 Result: NOT SAFE to merge as is (one major, two minors). Every round-1 finding is fixed at its root. Major: a quick check alone holds the headThe gate passes a head only when every lens's newest read names that one head, or a newer override covers it (
So an agent that follows step 7 (quick check, The old text was already unclear ("The rest the orchestrator checks itself"). The fix's new "or skip the quick check and use the override" makes the wrong reading explicit. COMMENTS.md:103 ("The Fix: in step 7 say that on Minor: one gate change the protected-paths test does not catch
Round-1 findings: each fixed at its root
The new drift tests read the gate's real code and fail when the skill drifts
All mutations were reverted. My temp checkout was clean afterwards. |
…ck never does The gate passes a head only when every lens's newest read names it or a newer override covers them (lens.Review.reviewed_head). A quick check is one lens on the fix head while the earlier lenses still name the old head, so it alone holds the head. SKILL.md step 7 now says which head passes and that on `nit-only: yes` the orchestrator still posts the override; COMMENTS.md says the same. The round count says "whichever is higher", as review_round does. Tests: a history (code at A, code r2 at B, fix C with a quick check) through the gate's real reviewed_head and judge_rounds, pinned to the skill's words; the protected-paths test now compares is_protected's answer with the skill's list on every tracked file and on probe paths, so an inline rule fails it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
lens: A r3 · reviewer: lens agent (Opus 5.5 high) · read at: a8c263c Result: SAFE TO MERGE (no blocker, no major, two wording minors for the follow-up issue). Every round-2 finding is fixed at its root. Minor: a fix that is not nit-only, after the last lens roundBefore this round, step 7 ended with "The rest the orchestrator checks itself from the fix diff and the lenses' proof tests, then posts the override". REVIEW.md:63 ( Round-2 findings: each fixed at its root
The new tests read the gate's real code and fail when the docs drift
No new error: the new bullets match |
Audit #834, item G6: one review skill in the fork kit; a project keeps only its lens config.
What and why
Two review skills drifted apart: this kit's
fork-kit/reviewand the one a downstream project's agents actually use(that project's own skill, its order and lens-comment pages, and its eval folder). This PR makes the kit's
SKILL.md the one general skill. Every general rule of the project copy is merged in; the
project copy can then shrink to a thin adapter (a follow-up PR in that project, not here).
No code change to the gate or the kit tools: only the skill text, COMMENTS.md (two new checked examples), REVIEW.md,
the eval lens text and one new test class.
Drift table (section by section)
G = general (now in the kit), P = project fact (stays in the project), C = conflict (pick below).
weightis the tier; table with defaults; a project maps its own tier names onto the three weights in its rules (P)lens_rounds)nit-only: yes/no)minors)ready_command; push nothing after"checks:state:state:line; question stays the kit settingstate_questionDI-1,DI-2),focus:runs twice and firstcovered:rule (not listed = not reviewed; every changed file in some lens's final covered)lens_runners; "audit model never a lens" and "a cheaper model never the only reviewer" addedfinding: <sev> ...,findings: none, nostate:linestate:,blocker:,major:,minor:state:+blocker:/major:/minor:(both shapes parse; see picks)review_kit.py checkbefore postingnot a lens:)lens_format_docshould point there (REVIEW.md)/tmppaths, project names{workdir}, harnessreview_eval.pycovered:rule into the eval lensPicks for the conflicts (what the gate parses decides)
weight. The review block's rules (weightper path glob) are what the gate reads, so aseparate
tierkey would be a second source. Defaults:min_lenses2/1/0 fromreview.recommended.json, runnerOpus high. The new test pins the skill's table to the gate's
WEIGHTSand those counts.state:line, not inchecks:. The gate parsesstate_line(not enforced yet), the kit'scheckrequires it, and T3's lens prompt (Review rules T3a: lens_start plan, prompt, refusals and request ids (pure logic) #825) asks for "a line startingstate:". COMMENTS.md now shows theolder shape (state only in
checks:) and thatcheckrefuses it.blocker:/major:/minor:lines. Both shapes parse the same in the gate(
finding: <sev>lines andfindings: nonecount as a findings section), but the gate's release note asks lensprompts to write
blocker: noneandmajor: none, and Review rules T3a: lens_start plan, prompt, refusals and request ids (pure logic) #825 writes the same.finding:lines stay documented asread.
lenses; a project that wants xhigh for its twin lenses sets
lens_runners.light(CI only, no lens) keeps its meaning; "light" for projects without CI is named "no queueconfig: normal, one lens".
Config keys (unchanged; now documented as the tier source)
reviewblock (read by the gate, keys in the pr-queue README "Review rules"):rules[].weightisthe tier per path pattern;
min_lensesthe per-tier lens count;lenses,rounds,findings,override_authors,state_line. Alsolens_format_doc(only quoted): point it at the kit's COMMENTS.md..github/review-kit.json(kit only):runner,lens_runners,audit_runner,forbidden_efforts,lens_rounds,minors,state_question,queue_config. REVIEW.md now has a "what stays in the project" table.Test
fork-kit/review/tests/comments.test.py, new classSkillFormat:in as a clean read and fed through the gate's own parser (
lens.py,review_judge.pyviareview_kit.verdict):it must read
read, clean, state, andcheckmust exit 0;state:orblocker:line readsno state/no findings section;WEIGHTS, with the lens counts ofreview.recommended.json.Red checks done by hand: removing the
state:line from the skill template, changing the normal tier's count,dropping the lightest choice from the eval lens's
finding:placeholder, and rewording the state question eachturned the class red.
vp fmt --check fork-kit/reviewis clean.State question for this change: the docs and the test hold no state; the test reads files only and gives the same
answer on every run.
General vs project
General: all of it. No project names, paths or machines; examples use made-up lens names and paths. Project facts
(its tier per module mapped to weights, its lens list, its domain lenses such as item duplication attacks, its
commands, labels and outside reviewers) stay in that project's review block, lens prompts, kit settings and a thin
adapter skill, done in a follow-up PR in that project after it moves its fork-ci pin to a release that has the
review rules.
Left
state_lineis parsed but not enforced by the gate; enforcing it is gate work, not part of this PR.🤖 Generated with Claude Code