From bb08b8e2fba7d84f6b8a2c888fb7b205847214c5 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sat, 21 Nov 2026 23:37:32 +0000 Subject: [PATCH 1/3] fork-kit/review: one general review skill; tiers, failure table, nit-only check and a drift test The kit's SKILL.md becomes the one general review skill, so a project keeps only its facts (review block, lens prompts, kit settings) and a thin adapter. Merged in from the project skill: the tier table as the gate's weights, the author's failure table, nit-only quick check, batch per area, twin lens names, focus lenses, covered rule, the extra general lenses, and a concurrency part in the eval lens. Conflicts go to what the gate parses: the state answer on a `state:` line and `blocker:`/`major:`/`minor:` lines. New test (comments.test.py, SkillFormat): every report shape the skill, the eval lens and COMMENTS.md tell a lens to write reads as a clean read with a state line through the gate's own parser; the skill's tiers match the gate's weights and recommended min_lenses; the state question is word for word in the skill, the eval lens and REVIEW.md. Co-Authored-By: Claude Opus 5.5 --- fork-kit/review/COMMENTS.md | 26 +++++ fork-kit/review/README.md | 16 +-- fork-kit/review/REVIEW.md | 27 +++-- fork-kit/review/SKILL.md | 143 +++++++++++++++++-------- fork-kit/review/eval/lens.md | 11 +- fork-kit/review/tests/comments.test.py | 84 +++++++++++++++ 6 files changed, 246 insertions(+), 61 deletions(-) diff --git a/fork-kit/review/COMMENTS.md b/fork-kit/review/COMMENTS.md index 6c13bb8851bcd..25105884b2637 100644 --- a/fork-kit/review/COMMENTS.md +++ b/fork-kit/review/COMMENTS.md @@ -100,6 +100,32 @@ state: an old config without the new key is read with the default findings: none ``` +A quick check of a nit-only fix (SKILL.md, "Order", step 7). The `nit-only:` line is for the orchestrator; the gate +reads the report as any other. + +The gate reads: `lens quick-check, read, clean, state` + +```text +lens: quick-check · reviewer: lens agent · read at: 0123456789abcdef0123456789abcdef01234567 +covered: the fix diff since round 1 (tests and message text only) +nit-only: yes +state: no write step changed +blocker: none +major: none +``` + +The state answer only in `checks:` (an older shape): the gate takes the read, but the kit's `check` refuses it (no state +line). Write the answer on a `state:` line. + +The gate reads: `lens correctness, read, clean, no state` + +```text +lens: correctness · reviewer: lens agent · read at: 0123456789abcdef0123456789abcdef01234567 +covered: src/queue.ts +checks: a second run finds the entry and does nothing; an empty queue file is read as empty +finding: minor src/queue.ts:12 the error text names the wrong flag +``` + ## Reports that block A major finding. The head is held until this lens posts a new read on a fixed head (or a person decides). diff --git a/fork-kit/review/README.md b/fork-kit/review/README.md index faa500b2b0222..c322bb0031135 100644 --- a/fork-kit/review/README.md +++ b/fork-kit/review/README.md @@ -6,14 +6,14 @@ question every time, one or two independent lenses, at most two lens rounds, roo failing tests. Unlike the other kits it installs nothing on a host: an agent reads the skill and runs the tools from a checkout of the fork. -| File | What | -| -------------------------------------------------- | --------------------------------------------------------------------------------------- | -| [SKILL.md](SKILL.md) | the review skill: plan, state question, order, rounds, minors, lenses, runners, posting | -| [REVIEW.md](REVIEW.md) | how a project adopts it: the review block, lens prompt files, the gate, kit settings | -| [COMMENTS.md](COMMENTS.md) | the lens comment format, with examples checked against the gate's own parser | -| [review-kit.example.json](review-kit.example.json) | the kit settings (`.github/review-kit.json` in the project; optional) | -| [lib/review_kit.py](lib/review_kit.py) | `plan` (which lenses a change needs), `check` (how the gate reads a comment), `doc` | -| [eval/](eval/README.md) | the model eval: which model and effort finds the confirmed bugs, from one config file | +| File | What | +| -------------------------------------------------- | ---------------------------------------------------------------------------------------- | +| [SKILL.md](SKILL.md) | the review skill: tiers, state question, order, rounds, minors, lenses, runners, posting | +| [REVIEW.md](REVIEW.md) | how a project adopts it: the review block, lens prompt files, the gate, kit settings | +| [COMMENTS.md](COMMENTS.md) | the lens comment format, with examples checked against the gate's own parser | +| [review-kit.example.json](review-kit.example.json) | the kit settings (`.github/review-kit.json` in the project; optional) | +| [lib/review_kit.py](lib/review_kit.py) | `plan` (which lenses a change needs), `check` (how the gate reads a comment), `doc` | +| [eval/](eval/README.md) | the model eval: which model and effort finds the confirmed bugs, from one config file | The kit holds no project facts: which paths are critical and which lenses they need are the project's `review` block (read by the gate), and who runs a lens is the project's kit settings. The kit imports the gate's own parser diff --git a/fork-kit/review/REVIEW.md b/fork-kit/review/REVIEW.md index 5dd43754c1059..cfe59792f97ee 100644 --- a/fork-kit/review/REVIEW.md +++ b/fork-kit/review/REVIEW.md @@ -11,15 +11,21 @@ Add a `review` block to the queue config (`.github/pr-queue.json` by default), i then only reports what it would ask for. Start from the gate's [review.recommended.json](../../fork-ci/actions/pr-queue/review.recommended.json): -- `rules`: the project's paths, heaviest first; the first rule that matches a path takes it. Critical paths (stored - data, migrations, deploy, security, the merge path itself) name **two** lenses; the last rule is the catch-all - `**`, normal, one lens. +- `rules`: the project's paths, heaviest first; the first rule that matches a path takes it. A rule's `weight` is + the tier of its paths (`critical`, `normal` or `light`; SKILL.md, "Tier"). Critical paths (stored data, + migrations, deploy, security, the merge path itself) name **two** lenses; the last rule is the catch-all `**`, + normal, one lens. A project with its own tier names (a tier per module or plugin) writes one rule per module + path with the weight its tier maps to; the tier list itself stays in the project's own files. - `lenses`: each lens's name and its prompt file. - `min_lenses`: at least 2 for critical, 1 for normal, 0 for light. - `rounds`: the gate's own cap. Keep `max` at or above the kit's `lens_rounds` (the plan warns otherwise). - `findings: true`: the gate holds a head while a lens's newest read has an open blocker or major. - `state_line: true`: the gate parses it but does not enforce it yet; the kit's `check` does. +Also set the queue config's `lens_format_doc` (a text the gate only quotes in its comments) to where agents read +the format: the kit's [COMMENTS.md](COMMENTS.md) at the fork tag the project pins, or the project's adapter skill +if it links there. + Check it: `python3 -I fork-kit/review/lib/review_kit.py plan --root . --changes ` with a `git diff --name-status -z -M` output, or `--base origin/main` on a branch. A config the gate refuses gives a plain message and exit 2. @@ -68,6 +74,15 @@ rule order would let a lighter rule take a heavier rule's paths. ## What stays in the project -The review block, the lens prompt files with the project's own hints, the kit settings, and anything about outside -reviewers. The general rules (tiers by path, the state question, one or two independent lenses, rounds, minors, -attacks with tests, the comment format) are the kit's, so a project does not keep its own copy. +| Project fact | Where | +| ---------------------------------------------------------------- | -------------------------------------------------------------- | +| which paths are critical, normal or light; the lenses each needs | the `review` block of the queue config (read by the gate) | +| its lens prompts, with its domain's typical bugs and attacks | the lens prompt files the block names | +| who runs each lens, rounds, minors, the state question | `.github/review-kit.json` | +| the override prefix, ready and push commands, lens authors | the queue config (`override`, `ready_command`, `push_command`) | +| outside reviewers, follow-up issue labels, its own checks | the project's adapter skill or docs | + +The general rules (tiers by path, the state question and the failure table, one or two independent lenses, twin lens +names, rounds and the nit-only quick check, minors, attacks with tests, the comment format) are the kit's, so a +project does not keep its own copy. A project's adapter skill is a few lines: it points at [SKILL.md](SKILL.md) and +adds only its domain lenses and commands. diff --git a/fork-kit/review/SKILL.md b/fork-kit/review/SKILL.md index ee2a60b877bdb..821b2152293cf 100644 --- a/fork-kit/review/SKILL.md +++ b/fork-kit/review/SKILL.md @@ -8,26 +8,36 @@ description: Review a pull request with lenses picked by the changed paths (the Goal: find bugs early and fix them at the root, so they cannot come back in a slightly different case. CI and lenses check side by side; the merge gate lets a head merge only when both pass. A lens is one separate read-only agent. -The project's facts are in two files, never in this text: the `review` block of its queue config (which paths are -critical, normal or light, and which lenses each needs; read by the gate) and its kit settings -`.github/review-kit.json` (who runs a lens, how many lens rounds, what happens to minors). Adopting the kit: -[REVIEW.md](REVIEW.md). The comment format the gate reads: [COMMENTS.md](COMMENTS.md). +This is the one general review skill. A project keeps only its facts, in two files, never in this text: the `review` +block of its queue config (which paths are critical, normal or light, and which lenses each needs; read by the gate) +and its kit settings `.github/review-kit.json` (who runs a lens, how many lens rounds, what happens to minors, the +state question). A project's own skill, if it has one, is a thin adapter: it points here and adds only its domain +lenses and commands. Adopting the kit: [REVIEW.md](REVIEW.md). The comment format the gate reads: +[COMMENTS.md](COMMENTS.md). -## 1. Plan: which lenses, from the changed paths +## 1. Tier: from the changed paths + +The tier of a change is the gate's `weight`: each rule of the review block gives its paths a weight, and the heaviest +weight any changed path hits decides. The tiers, with their defaults (the gate's `min_lenses` in +[review.recommended.json](../../fork-ci/actions/pr-queue/review.recommended.json) and the kit's `runner`): + +| Tier | Typical paths | Lenses (default) | +| -------- | ----------------------------------------------------------------------------------- | ----------------------------------------------------- | +| critical | stored data, migrations, deploy, security, the merge path, anything that loses data | at least **2 independent** lenses, each Opus 5.5 high | +| normal | other code | at least **1** lens, Opus 5.5 high | +| light | docs, comments, config values with no code path change | **0**: CI only | + +A project with its own tier names maps them onto these three in its rules (for example tiers 0-1 light, 2 normal, 3 +critical). A project with **no CI** (no queue config) has no gate: every change is normal with one lens, by hand. ``` python3 -I fork-kit/review/lib/review_kit.py plan --root --base origin/main ``` -It judges the change with the gate's own rules and prints the weight, the lenses (each with its prompt file and its -runner), why, the state question, the round limit and the minors rule. Do what it says; never pick fewer lenses by -hand. Unsure, or a path the rules do not cover: take the heavier weight. - -- **critical:** every lens it names, **two independent lenses** at least (separate agents, no shared context, none - sees another's report). Two independent reads miss different bugs; a second identical read of normal code adds - almost nothing. -- **normal:** one lens. -- **light:** no lens; CI only. +The plan judges the change with the gate's own rules and prints the tier, the lenses (each with its prompt file and +its runner), why, the state question, the round limit and the minors rule. Do what it says; never pick fewer lenses +by hand. Unsure, or a path the rules do not cover: take the heavier tier. Two independent reads miss different bugs; +a second identical read of normal code adds almost nothing. No queue config, or one without a `review` block: the plan says so and treats every change as normal with one lens `review`. That is a plain fallback, never a guess at which paths are critical. @@ -36,51 +46,80 @@ No queue config, or one without a `review` block: the plan says so and treats ev Every lens prompt contains, word for word, the project's state question (default: _What happens when this runs again, stops half-way, or meets existing, old or empty state?_), and every lens answers it on a `state:` line. In the -eval that shaped this kit, no model at any effort found the rerun and empty-history bugs without it. The author asks -it first: a change that saves, migrates, deploys or retries lists each write step with what a crash right after it, -a second run and an old, empty or half-written start state do, each with a test. +eval that shaped this kit, no model at any effort found the rerun and empty-history bugs without it. + +The author asks it first. A change that saves, migrates, deploys or changes state carries a **failure table** in its +description: one row per write step, with what a crash right after it, a second run, and an old, empty or +half-written start state do, each with its test; and every skip or fallback the code has, with how it reports +itself (no quiet fallbacks). Every lens checks that table: a write step without a row, or a row without a test that +fails without the fix, is a finding (major on a critical or normal change). ## 3. Order -1. **Author** (not a lens): why this approach, the root cause and where else the pattern occurs (with the search - used), every new check shown red on the old code and green on the fix. -2. **CI** starts with the push. Lenses do not wait for it. -3. **Lenses** get the diff, the files it calls into and the problem _as reported_, never the author's summary or - another reviewer's list (they would anchor on it). -4. **Orchestrator** merges the lens lists. -5. **Verify:** another agent reproduces every blocker and major (a test or an exact trace). False positive: a - one-line reason. Confirmed: a root-cause fix with a red-to-green test. All fixes of a round go up in one push. -6. **Rounds:** at most `lens_rounds` (default **2**): round 1 lenses, one fix round, round 2 lenses on the fix (only +Each step is done by another agent than the author. + +1. **Author** (not a lens): why this approach and the rejected ones, the root cause and where else the pattern occurs + (with the search used), every new check shown red on the old code and green on the fix, the failure table (§2). + One behaviour or a batch of small related fixes per pull request, about 400 changed code lines at most; bigger is + a stack. A project may open critical and normal changes as drafts until the lenses are done. +2. **CI** starts with the push. Lenses do not wait for it. Red: fix the cause. +3. **Outside reviewers** (bots, other tools) are extra opinions, never required; the project lists its own. +4. **Lenses** get the diff, the files it calls into and the problem _as reported_, never the author's summary, an + outside reviewer's list or another lens's report (they would anchor on it). **Batch per area:** a stack or a group + of small pull requests in one area may get one lens pass of at most about 1500 changed lines (never mixing + areas); the lens posts one comment per pull request, each read at that pull request's head. +5. **Orchestrator** merges the lens lists (and the outside reviewers'; an outside finding no lens confirmed gets one + verifier). +6. **Verify:** another agent reproduces every blocker and major (a test or an exact trace). False positive: a + one-line reason. Confirmed: a root-cause fix with a red-to-green test. All fixes of a round go up in **one push**. +7. **Rounds:** at most `lens_rounds` (default **2**): round 1 lenses, one fix round, round 2 lenses on the fix (only those whose area the fix touches, and every lens that reported a blocker or major: the gate holds the head until that lens reads a fixed head). Stop early when a round finds no blocker and no major. + - **Nit-only fix:** a fix diff that touches only tests, docs, comments, message text or names (judged by the diff, + not by the findings' severity) gets one quick check instead of a round: one lens on the fix diff whose comment + has a line `nit-only: yes` or `nit-only: no`. On `no` the fix goes to the next round. - The rest the orchestrator checks itself from the fix diff and the lenses' proof tests, then posts the override comment (` : `, COMMENTS.md). An override covers lens reads; it clears a blocker or major only when its author is in the review block's `override_authors`. - A note between rounds that is no lens report starts with `not a lens:` (COMMENTS.md). - **No round past the limit.** A pull request still stuck (a blocker or major left) gets one audit by the `audit_runner` ("good enough to merge, or what next"), posted as a `not a lens:` comment; then a person decides. -7. **Minors** (`minors`, default `issue`): minors not fixed when the pull request stops go into **one** follow-up - issue, linked on the pull request. With `same-round`, minors are fixed in the round that found them and any left - at the last round are listed in the pull request description with a reason each. +8. **Minors** (`minors`, default `issue`): minors and anything left when the pull request stops go into **one** + follow-up issue, with the project's labels, linked on the pull request. With `same-round`, minors are fixed in the + round that found them and any left at the last round are listed in the pull request description with a reason + each. +9. **Ready:** the project's ready command (the queue config's `ready_command`) arms the head every lens read. Push + nothing after it; a new head needs the lenses again. ## 4. Lenses Each lens is a prompt file named in the review block. Every prompt holds the state question and the comment format. -Common lenses, picked by the rules: +Common lenses, picked by the rules; a project adds its own domain lenses (its typical bugs and attacks) in its +prompt files: | Lens | Looks for | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------- | -| correctness | null, empty and bounds, error paths, the unhappy branch of every `if`, units; is this the root cause? | +| correctness | null, empty and bounds, error paths, the unhappy branch of every `if`, units; is this the root cause, where else? | | design / root cause | forms its own view from the problem as reported; the simpler or more general fix | | tests | every requirement and fix has a test that fails without it; assertions that cannot fail; skipped checks | | data and state | **attacks**: double submit, two processes, disconnect mid-action, kill after each write, retry; each with a **failing test** | -| security | input from outside, permissions, secrets, injection | +| security | input from outside, permissions, forged or out-of-order messages, secrets, injection | | concurrency | races, ordering, shutdown | +| performance | hot paths and queries, with numbers at the expected load | +| domain expert | how such systems are built and broken elsewhere; says what is general knowledge and what it checked in this code | +| visual | only when people see the change: real screenshots | **Attacks leave failing tests.** A data or state lens that finds a way to lose, duplicate or lock data writes the test that shows it (red on the head it read) and names it in the finding; the fix makes it green and keeps it. An opinion without a test or exact trace is not a finding. +**Twin lenses** (the two independent lenses of a critical change, or a lens the owner asked to run twice with +`focus: `) have different names in the review block (`data-1`, `data-2`): the gate counts one name as one lens. +A `focus:` lens runs twice and first. + +**Covered:** a lens lists every file and function it read in full; anything not listed counts as not reviewed. +Every changed file must be in some lens's `covered:` in its final state; a fix round lists the files it changes. + ### Lens prompt A lens with no prompt file of its own (the fallback lens `review`) uses this text, filled in: @@ -89,27 +128,45 @@ A lens with no prompt file of its own (the fallback lens `review`) uses this tex > full and what they call into. Find bugs at their root: the unhappy branch of every `if`, error paths, empty and > bounds, and where else the same pattern occurs. Answer the state question on a `state:` line: __ > Every finding needs file:line and inputs or state leading to the wrong result. Post one comment in the format of -> COMMENTS.md, the header first: `lens: · reviewer: · read at: `, then `covered:`, -> `checks:`, `state:`, `blocker:`, `major:`, `minor:`. Check it with `review_kit.py check --head ` before -> posting; never edit a posted comment (post a new one). +> §6, the header first. Check it with `review_kit.py check --head ` before posting; never edit a posted comment +> (post a new one). ## 5. Who runs what Runners are kit settings, not rules: `runner` (default `claude-opus-5-5`, effort `high`) for every lens, -`lens_runners` for a lens that needs another, `audit_runner` (default `claude-fable-5-2`, effort `high`) for the -stuck-PR audit, and `forbidden_efforts` (default `low` and `max`: the highest effort cost about 3x for no reliable -gain). The plan prints the runner of each lens. A project changes them from eval numbers ([eval/](eval/README.md)): -the cheapest setting whose worst-case recall equals the best. +`lens_runners` for a lens that needs another (a project may give its critical twin lenses `xhigh`), `audit_runner` +(default `claude-fable-5-2`, effort `high`) for the stuck-PR audit and design options, and `forbidden_efforts` +(default `low` and `max`: the highest effort cost about 3x for no reliable gain). The audit model is never a lens, and +a cheaper model is never the only reviewer of important code (it missed a blocker in both its eval runs). The plan +prints the runner of each lens. A project changes them from eval numbers ([eval/](eval/README.md)): the cheapest +setting whose worst-case recall equals the best. ## 6. Posting -One comment per lens, header on the first line, full 40-character sha, `state:`, `blocker:` and `major:` lines. -Check before posting: +One plain pull request comment per lens (not an inline comment on a diff line, not inside a code block or quote), +in this shape: + +```text +lens: · reviewer: · read at: +covered: +checks: +state: +blocker: none +major: none +minor: +``` + +A blocker or major replaces `none` with the finding: file:line, the inputs or state leading to the wrong result, and +the test that shows it. `finding: ` lines and `findings: none` are read too (COMMENTS.md), +but every report keeps the `state:` line. The sha is `git rev-parse HEAD` of the head read; a short sha reads nothing. +Templates in briefs write ``, never a real one. Check before posting: ``` python3 -I fork-kit/review/lib/review_kit.py check --root --head comment.md ``` Exit 0 only for a read of that head with a findings section, no blocker or major, and a state line; otherwise it -prints how the gate reads it. A comment the gate reads as `reads nothing` blocks the head: delete it, or edit its -first line to start with `not a lens:` (COMMENTS.md). +prints how the gate reads it. A project may also have a check of the posted comment (who wrote it, edits): run it +right after posting. A comment is never edited or hidden (an edited or hidden lens comment reads nothing): post a new +one. A comment the gate reads as `reads nothing` keeps blocking the head even after a new post: delete it, or edit +its first line to start with `not a lens:` and no line below it to hold `lens:` (COMMENTS.md). diff --git a/fork-kit/review/eval/lens.md b/fork-kit/review/eval/lens.md index 8cd0fa6766417..f159ffa264fb3 100644 --- a/fork-kit/review/eval/lens.md +++ b/fork-kit/review/eval/lens.md @@ -1,4 +1,4 @@ -## Your lens: correctness + state + data integrity (one read-only reviewer) +## Your lens: correctness + state + data integrity + concurrency (one read-only reviewer) Goal: find the bugs early and fix them at their root, so the same bug cannot come back in a slightly different case. @@ -9,7 +9,10 @@ clauses, and _is this the root cause, and where else does the pattern occur?_ meets existing, old or empty state? **Data integrity** (anything that saves, moves or counts data): attacks such as a double submit, two processes at -once, a disconnect mid-action, a kill after each write, a retry. Each finding comes with a failing test. +once, a disconnect mid-action, a kill after each write, a retry, input at the wrong moment. Each finding comes with a +failing test. + +**Concurrency** (threads, async work, schedulers, messages between processes): races, ordering, shutdown. In this exercise you cannot run builds, tests or servers. Replace "failing test" with an **exact trace** (file:line, the inputs and state, step by step, to the wrong result) plus the name and content of the test you would add. @@ -21,8 +24,8 @@ impact. Style nits go in a separate optional list. ## Output format (write exactly this to your result file) ``` -lens: correctness+state+data · reviewer: RUN-ID · read at: -covered: +lens: correctness+state+data+concurrency · reviewer: RUN-ID · read at: +covered: # anything not listed counts as not reviewed checks: state: finding: wrong result> (confidence) [test: : ] diff --git a/fork-kit/review/tests/comments.test.py b/fork-kit/review/tests/comments.test.py index 4b8dddca65b0c..4feec20ee7d34 100644 --- a/fork-kit/review/tests/comments.test.py +++ b/fork-kit/review/tests/comments.test.py @@ -234,6 +234,90 @@ def test_not_utf8_is_a_plain_message(self): } +TEMPLATE_DOCS = ('SKILL.md', os.path.join('eval', 'lens.md'), 'COMMENTS.md') +FENCE_RE = re.compile(r'^(`{3,})[a-z]*\s*$') +PLACEHOLDER_RE = re.compile(r'<([^<>\n]*)>') + + +def kit_text(name): + with open(os.path.join(KIT, name), encoding='utf-8') as f: + return f.read() + + +def templates(text): + """Every fenced block whose first line is a lens header with a ``: the report shapes the docs tell a + lens to write (the checked examples of COMMENTS.md hold real shas, so they are not templates).""" + lines, out, i = text.splitlines(), [], 0 + while i < len(lines): + m = FENCE_RE.match(lines[i]) + if not m: + i += 1 + continue + body, i = [], i + 1 + while i < len(lines) and lines[i].strip() != m.group(1): + body.append(lines[i]) + i += 1 + i += 1 + if body and re.match(r'lens\s*:', body[0], re.I) and PLACEHOLDER_RE.search(body[0]): + out.append('\n'.join(body)) + return out + + +def fill(template): + """A template as a lens would fill it on a clean read: a sha placeholder takes a full sha, a choice + (``) its last, lightest option, anything else plain text.""" + def one(m): + inner = m.group(1) + if 'sha' in inner: + return SHA + if '|' in inner: + return inner.split('|')[-1].strip() + return 'text' + return PLACEHOLDER_RE.sub(one, template) + + +class SkillFormat(unittest.TestCase): + """The report shape the skill, the eval lens and COMMENTS.md tell a lens to write is a clean read with a state line + to the gate's own parser, and the skill's tiers and state question are the gate's and the kit's: the skill text and + the gate cannot drift apart again.""" + + def test_every_documented_report_shape_reads_clean_with_the_gates_parser(self): + for name in TEMPLATE_DOCS: + found = templates(kit_text(name)) + with self.subTest(doc=name): + self.assertTrue(found, f'{name} no longer shows the report shape') + for t in found: + with self.subTest(doc=name, header=t.splitlines()[0]): + words = review_kit.verdict(fill(t), 'tool ready --head', SHA) + self.assertEqual(words[1:], ['read', 'clean', 'state']) + self.assertEqual(review_kit.check(fill(t), 'tool ready --head', SHA)[0], 0) + + def test_a_shape_without_its_state_or_findings_lines_fails(self): + t = templates(kit_text('SKILL.md'))[0] + no_state = '\n'.join(x for x in t.splitlines() if not x.startswith('state:')) + self.assertIn('no state', review_kit.verdict(fill(no_state), 'tool ready --head', SHA)) + no_blocker = '\n'.join(x for x in t.splitlines() if not x.startswith('blocker:')) + self.assertIn('no findings section', review_kit.verdict(fill(no_blocker), 'tool ready --head', SHA)) + + def test_the_skill_carries_the_state_question_word_for_word(self): + for name in ('SKILL.md', os.path.join('eval', 'lens.md'), 'REVIEW.md'): + with self.subTest(doc=name): + self.assertIn(review_kit.STATE_QUESTION, ' '.join(kit_text(name).split())) + + def test_the_skill_tiers_are_the_gates_weights_and_recommended_lens_counts(self): + import json + import review_config + gate = os.path.join(REPO, 'fork-ci', 'actions', 'pr-queue') + with open(os.path.join(gate, 'review.recommended.json'), encoding='utf-8') as f: + mins = json.load(f)['min_lenses'] + rows = re.findall(r'^\| (\w+) +\|[^|]*\| ([^|]*)\|\s*$', kit_text('SKILL.md'), re.M) + tiers = {w: cell for w, cell in rows if w in review_config.WEIGHTS} + self.assertEqual(list(tiers), list(review_config.WEIGHTS)) + for weight, cell in tiers.items(): + with self.subTest(weight=weight): + self.assertEqual(int(re.search(r'\*\*(\d+)', cell).group(1)), mins[weight]) + + class RedCheck(unittest.TestCase): def test_each_rule_has_a_test_that_fails_without_it(self): with open(LIB, encoding='utf-8') as f: From 6f0d8330097561086e2dca5defa39e21899c57c5 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sat, 21 Nov 2026 23:49:19 +0000 Subject: [PATCH 2/3] fork-kit/review: skill matches the gate on rounds, protected paths and check exits - A quick check is a lens read of a new head, so the gate counts it as a round; the skill names rounds.approach_at / approach_lens / max, how to run the approach lens, and that only an override by a listed override_authors login waives it. - The tier table names the gate's always-critical paths (queue config, lens prompts, .github/workflows/**); the light row no longer says config is light. - focus: removed (nothing reads it); twin lenses described in general terms. - check exits 1 for a real blocker or major by design; never soften a finding. - not a lens: also needs no read-at sha outside a quote (COMMENTS.md rule). - Drift test reads the round keys and recommended approach_at through the gate's parser and judge_rounds, the protected paths from review_judge, and the runner defaults from review_kit.DEFAULTS. Co-Authored-By: Claude Opus 5.5 --- fork-kit/review/COMMENTS.md | 2 +- fork-kit/review/SKILL.md | 44 ++++++++++----- fork-kit/review/tests/comments.test.py | 75 +++++++++++++++++++++++++- 3 files changed, 105 insertions(+), 16 deletions(-) diff --git a/fork-kit/review/COMMENTS.md b/fork-kit/review/COMMENTS.md index 25105884b2637..e5960e07e738c 100644 --- a/fork-kit/review/COMMENTS.md +++ b/fork-kit/review/COMMENTS.md @@ -101,7 +101,7 @@ findings: none ``` A quick check of a nit-only fix (SKILL.md, "Order", step 7). The `nit-only:` line is for the orchestrator; the gate -reads the report as any other. +reads the report as any other, and counts it as a round (a read of a new head). The gate reads: `lens quick-check, read, clean, state` diff --git a/fork-kit/review/SKILL.md b/fork-kit/review/SKILL.md index 821b2152293cf..f7cef10de93e5 100644 --- a/fork-kit/review/SKILL.md +++ b/fork-kit/review/SKILL.md @@ -25,7 +25,11 @@ weight any changed path hits decides. The tiers, with their defaults (the gate's | -------- | ----------------------------------------------------------------------------------- | ----------------------------------------------------- | | critical | stored data, migrations, deploy, security, the merge path, anything that loses data | at least **2 independent** lenses, each Opus 5.5 high | | normal | other code | at least **1** lens, Opus 5.5 high | -| light | docs, comments, config values with no code path change | **0**: CI only | +| light | docs and other paths the rules mark light (never a protected path, below) | **0**: CI only | + +**Always critical** (the gate's built-in protected paths, whatever rule they fall under): the queue config file, every +lens prompt file named in the review block's `lenses`, and `.github/workflows/**`. A comment-only change takes its +path's tier. A project with its own tier names maps them onto these three in its rules (for example tiers 0-1 light, 2 normal, 3 critical). A project with **no CI** (no queue config) has no gate: every change is normal with one lens, by hand. @@ -75,12 +79,22 @@ Each step is done by another agent than the author. 7. **Rounds:** at most `lens_rounds` (default **2**): round 1 lenses, one fix round, round 2 lenses on the fix (only those whose area the fix touches, and every lens that reported a blocker or major: the gate holds the head until that lens reads a fixed head). Stop early when a round finds no blocker and no major. + - **How the gate counts rounds:** every lens comment that reads a different change of the pull request is one more + round (a head that only merges the base in is the same change; a `r3` marker in a lens name counts too). From + round `rounds.approach_at` of the review block (3 in review.recommended.json; `plan` prints the project's) the + gate requires one more lens, `rounds.approach_lens`, on the same head as the others; past `rounds.max` it fails + the head (in `enforce`; `shadow` only reports both). - **Nit-only fix:** a fix diff that touches only tests, docs, comments, message text or names (judged by the diff, - not by the findings' severity) gets one quick check instead of a round: one lens on the fix diff whose comment - has a line `nit-only: yes` or `nit-only: no`. On `no` the fix goes to the next round. + not by the findings' severity) may get one quick check: one lens on the fix head whose comment has a line + `nit-only: yes` or `nit-only: no`. On `no` the fix goes to the next round. The quick check is a lens read of a + new head, so **the gate counts it as a round**. When that round reaches `rounds.approach_at`, run the approach + lens too: one more independent read-only agent with the prompt file the review block names for that lens, on + the same head, posting its own report under that lens name; or skip the quick check and use the override below, + which adds no round. - The rest the orchestrator checks itself from the fix diff and the lenses' proof tests, then posts the override comment (` : `, COMMENTS.md). An override covers lens - reads; it clears a blocker or major only when its author is in the review block's `override_authors`. + reads; it clears a blocker or major, and waives a required approach lens, only when its author is in the review + block's `override_authors` (empty by default). An orchestrator's override never replaces the approach lens. - A note between rounds that is no lens report starts with `not a lens:` (COMMENTS.md). - **No round past the limit.** A pull request still stuck (a blocker or major left) gets one audit by the `audit_runner` ("good enough to merge, or what next"), posted as a `not a lens:` comment; then a person decides. @@ -113,9 +127,8 @@ prompt files: test that shows it (red on the head it read) and names it in the finding; the fix makes it green and keeps it. An opinion without a test or exact trace is not a finding. -**Twin lenses** (the two independent lenses of a critical change, or a lens the owner asked to run twice with -`focus: `) have different names in the review block (`data-1`, `data-2`): the gate counts one name as one lens. -A `focus:` lens runs twice and first. +**Twin lenses** (the two independent lenses of a critical change, or a lens the person who decides asks to run twice) +have different names in the review block (`data-1`, `data-2`): the gate counts one name as one lens. **Covered:** a lens lists every file and function it read in full; anything not listed counts as not reviewed. Every changed file must be in some lens's `covered:` in its final state; a fix round lists the files it changes. @@ -128,8 +141,8 @@ A lens with no prompt file of its own (the fallback lens `review`) uses this tex > full and what they call into. Find bugs at their root: the unhappy branch of every `if`, error paths, empty and > bounds, and where else the same pattern occurs. Answer the state question on a `state:` line: __ > Every finding needs file:line and inputs or state leading to the wrong result. Post one comment in the format of -> §6, the header first. Check it with `review_kit.py check --head ` before posting; never edit a posted comment -> (post a new one). +> §6, the header first. Check it with `review_kit.py check --head ` before posting (a blocker or major exits 1 by +> design: post it as found, never softened); never edit a posted comment (post a new one). ## 5. Who runs what @@ -165,8 +178,11 @@ Templates in briefs write ``, never a real one. Check before posting: python3 -I fork-kit/review/lib/review_kit.py check --root --head comment.md ``` -Exit 0 only for a read of that head with a findings section, no blocker or major, and a state line; otherwise it -prints how the gate reads it. A project may also have a check of the posted comment (who wrote it, edits): run it -right after posting. A comment is never edited or hidden (an edited or hidden lens comment reads nothing): post a new -one. A comment the gate reads as `reads nothing` keeps blocking the head even after a new post: delete it, or edit -its first line to start with `not a lens:` and no line below it to hold `lens:` (COMMENTS.md). +Exit 0 only for a read of that head with a findings section, no blocker or major, and a state line; otherwise exit 1, +and it prints how the gate reads it. A report with a real blocker or major exits 1 **by design**: post it when the +words read `read, blocker, state` or `read, major, state`. Never soften or drop a finding to get exit 0. A project may +also have a check of the posted comment (who wrote it, edits): run it right after posting. A comment is never edited +or hidden (an edited or hidden lens comment reads nothing): post a new one. A comment the gate reads as +`reads nothing` keeps blocking the head even after a new post: delete it, or edit its first line to start with +`not a lens:`, with no line below it holding `lens:` and no `read at` sha outside a `>` quote (COMMENTS.md, the +`not a lens:` example). diff --git a/fork-kit/review/tests/comments.test.py b/fork-kit/review/tests/comments.test.py index 4feec20ee7d34..d2116e69d26df 100644 --- a/fork-kit/review/tests/comments.test.py +++ b/fork-kit/review/tests/comments.test.py @@ -5,7 +5,7 @@ python3 -I -B fork-kit/review/tests/comments.test.py """ -import os, re, subprocess, sys, tempfile, shutil, types, unittest +import inspect, json, os, re, subprocess, sys, tempfile, shutil, types, unittest sys.dont_write_bytecode = True # the CI runner has no -B; keep the kit folder free of __pycache__ @@ -15,6 +15,8 @@ DOC = os.path.join(KIT, 'COMMENTS.md') sys.path.insert(0, os.path.join(KIT, 'lib')) import review_kit # noqa: E402 +import review_config # noqa: E402 (the gate's, on the path review_kit set) +review_judge = review_kit.review_judge SHA = '0123456789abcdef0123456789abcdef01234567' OTHER = '89abcdef0123456789abcdef0123456789abcdef' @@ -244,6 +246,12 @@ def kit_text(name): return f.read() +def recommended(): + """review.recommended.json through the gate's own parser.""" + with open(os.path.join(REPO, 'fork-ci', 'actions', 'pr-queue', 'review.recommended.json'), encoding='utf-8') as f: + return review_config.parse_review(json.load(f)) + + def templates(text): """Every fenced block whose first line is a lens header with a ``: the report shapes the docs tell a lens to write (the checked examples of COMMENTS.md hold real shas, so they are not templates).""" @@ -317,6 +325,71 @@ def test_the_skill_tiers_are_the_gates_weights_and_recommended_lens_counts(self) with self.subTest(weight=weight): self.assertEqual(int(re.search(r'\*\*(\d+)', cell).group(1)), mins[weight]) + def test_the_skill_runner_text_is_the_kits_default(self): + """The tier table's `Opus 5.5 high` and §5's runner defaults are the kit's DEFAULTS, read, not copied.""" + skill = ' '.join(kit_text('SKILL.md').split()) + for key in ('runner', 'audit_runner'): + r = review_kit.DEFAULTS[key] + with self.subTest(setting=key): + self.assertIn(f"`{key}` (default `{r['model']}`, effort `{r['effort']}`)", skill) + r = review_kit.DEFAULTS['runner'] + m = re.fullmatch(r'claude-([a-z]+)-(\d+)-(\d+)', r['model']) + self.assertTrue(m, f"the default runner {r['model']!r} has no display name rule here") + shown = f"{m.group(1).capitalize()} {m.group(2)}.{m.group(3)} {r['effort']}" + rows = re.findall(r'^\| (\w+) +\|[^|]*\| ([^|]*)\|\s*$', kit_text('SKILL.md'), re.M) + with_lenses = [(w, cell) for w, cell in rows if w in review_config.WEIGHTS and 'lens' in cell] + self.assertTrue(with_lenses) + for weight, cell in with_lenses: + with self.subTest(weight=weight): + self.assertIn(shown, cell) + + def test_the_skill_names_the_gates_protected_paths(self): + """The skill's always-critical list is the gate's: the config, every lens prompt and the workflows glob, and + nothing else (a new kind of protected path in the gate fails here until the skill names it).""" + rv = recommended() + cfg = '.github/pr-queue.json' + self.assertEqual(review_judge.protected_paths(rv, cfg), {cfg, *(x.prompt for x in rv.lenses)}) + used = set(re.findall(r'\b[A-Z][A-Z_]{2,}\b', inspect.getsource(review_judge.is_protected))) + self.assertEqual(used, {'WORKFLOWS'}, 'is_protected reads another list: name it in SKILL.md, "Always critical"') + skill = ' '.join(kit_text('SKILL.md').split()) + line = re.search(r'\*\*Always critical\*\*.*?\.(?= |$)', skill) # to the sentence's end + self.assertTrue(line, 'SKILL.md no longer lists the always-critical paths') + for words in ('the queue config file', 'every lens prompt file', f'`{review_judge.WORKFLOWS}`'): + with self.subTest(path=words): + self.assertIn(words, line.group(0)) + # the gate really judges them critical, even under a light rule (`**/*.md` is light in the recommended rules) + for path in (cfg, rv.lenses[0].prompt, '.github/workflows/ci.yml'): + with self.subTest(judged=path): + self.assertEqual(review_judge.judge(rv, [('M', (path,))], cfg).weight, 'critical') + light = re.search(r'^\| light +\|([^|]*)\|', kit_text('SKILL.md'), re.M).group(1) + self.assertNotIn('config', light) + + def test_the_skill_counts_rounds_as_the_gate(self): + """A quick check is a lens read of a new head: one more gate round, and from `rounds.approach_at` the approach + lens is required unless a listed person decided. The skill names the gate's own round keys and the recommended + approach_at as review_config parses it.""" + rv = recommended() + skill = ' '.join(kit_text('SKILL.md').split()) + for key in review_config.ROUND_KEYS: + with self.subTest(key=key): + self.assertIn(f'`rounds.{key}`', skill) + said = re.search(r'`rounds\.approach_at` of the review block \((\d+) in review\.recommended\.json', skill) + self.assertTrue(said, 'SKILL.md no longer says the recommended rounds.approach_at') + self.assertEqual(int(said.group(1)), rv.rounds.approach_at) + for words in ('the gate counts it as a round', '`override_authors`', + "An orchestrator's override never replaces the approach lens"): + with self.subTest(says=words): + self.assertIn(words, skill) + # the gate's own round count: rounds before the quick check, then the quick check on a new head + n = rv.rounds.approach_at + history = [('code' if i == 0 else f'code r{i + 1}', f'{i:040x}') for i in range(n - 1)] + before = review_judge.judge_rounds(rv, history, {}) + self.assertEqual((before.round, before.approach), (n - 1, False)) + history.append(('quick-check', f'{n:040x}')) + after = review_judge.judge_rounds(rv, history, {}) + self.assertEqual((after.round, after.approach), (n, True)) + self.assertFalse(review_judge.judge_rounds(rv, history, {}, decided=True).approach) + class RedCheck(unittest.TestCase): def test_each_rule_has_a_test_that_fails_without_it(self): From a8c263c774e82254025f77a6131b112a894b4ee7 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 22 Nov 2026 03:05:33 +0000 Subject: [PATCH 3/3] fork-kit/review: the override passes a nit-only fix head, a quick check never does The gate passes a head only when every lens's newest read names it or a newer override covers them (lens.Review.reviewed_head). A quick check is one lens on the fix head while the earlier lenses still name the old head, so it alone holds the head. SKILL.md step 7 now says which head passes and that on `nit-only: yes` the orchestrator still posts the override; COMMENTS.md says the same. The round count says "whichever is higher", as review_round does. Tests: a history (code at A, code r2 at B, fix C with a quick check) through the gate's real reviewed_head and judge_rounds, pinned to the skill's words; the protected-paths test now compares is_protected's answer with the skill's list on every tracked file and on probe paths, so an inline rule fails it. Co-Authored-By: Claude Opus 5.5 --- fork-kit/review/COMMENTS.md | 5 +- fork-kit/review/SKILL.md | 37 ++++++----- fork-kit/review/tests/comments.test.py | 86 ++++++++++++++++++++++++++ 3 files changed, 111 insertions(+), 17 deletions(-) diff --git a/fork-kit/review/COMMENTS.md b/fork-kit/review/COMMENTS.md index e5960e07e738c..4b48bb2ea443d 100644 --- a/fork-kit/review/COMMENTS.md +++ b/fork-kit/review/COMMENTS.md @@ -100,8 +100,9 @@ state: an old config without the new key is read with the default findings: none ``` -A quick check of a nit-only fix (SKILL.md, "Order", step 7). The `nit-only:` line is for the orchestrator; the gate -reads the report as any other, and counts it as a round (a read of a new head). +A quick check of a nit-only fix (SKILL.md, "Order", step 7). The `nit-only:` line is for the orchestrator: on `yes` it +still posts the override for the fix head, because the other lenses still name the old head. The gate reads the report +as any other, and counts it as a round (a read of a new head). The gate reads: `lens quick-check, read, clean, state` diff --git a/fork-kit/review/SKILL.md b/fork-kit/review/SKILL.md index f7cef10de93e5..b51925f6006a5 100644 --- a/fork-kit/review/SKILL.md +++ b/fork-kit/review/SKILL.md @@ -79,22 +79,29 @@ Each step is done by another agent than the author. 7. **Rounds:** at most `lens_rounds` (default **2**): round 1 lenses, one fix round, round 2 lenses on the fix (only those whose area the fix touches, and every lens that reported a blocker or major: the gate holds the head until that lens reads a fixed head). Stop early when a round finds no blocker and no major. - - **How the gate counts rounds:** every lens comment that reads a different change of the pull request is one more - round (a head that only merges the base in is the same change; a `r3` marker in a lens name counts too). From - round `rounds.approach_at` of the review block (3 in review.recommended.json; `plan` prints the project's) the - gate requires one more lens, `rounds.approach_lens`, on the same head as the others; past `rounds.max` it fails - the head (in `enforce`; `shadow` only reports both). + - **How the gate counts rounds:** the round is the number of different changes of the pull request the lenses read + (a head that only merges the base in is the same change) or the highest `r2`, `r3` marker in a lens name, + whichever is higher. From round `rounds.approach_at` of the review block (3 in review.recommended.json; `plan` + prints the project's) the gate requires one more lens, `rounds.approach_lens`, on the same head as the others; + past `rounds.max` it fails the head (in `enforce`; `shadow` only reports both). + - **Which head passes:** the gate passes a head only when every lens's newest read names that head, or an override + comment newer than a lens's read covers it (` : `, + COMMENTS.md; it counts only from the account with the queue config's `override.author_association`). So after a + fix the lenses of the last round still name the old head, and the fix head passes only when every lens reads it + again (a new round) or the orchestrator posts the override for it (no new round). - **Nit-only fix:** a fix diff that touches only tests, docs, comments, message text or names (judged by the diff, - not by the findings' severity) may get one quick check: one lens on the fix head whose comment has a line - `nit-only: yes` or `nit-only: no`. On `no` the fix goes to the next round. The quick check is a lens read of a - new head, so **the gate counts it as a round**. When that round reaches `rounds.approach_at`, run the approach - lens too: one more independent read-only agent with the prompt file the review block names for that lens, on - the same head, posting its own report under that lens name; or skip the quick check and use the override below, - which adds no round. - - The rest the orchestrator checks itself from the fix diff and the lenses' proof tests, then posts the override - comment (` : `, COMMENTS.md). An override covers lens - reads; it clears a blocker or major, and waives a required approach lens, only when its author is in the review - block's `override_authors` (empty by default). An orchestrator's override never replaces the approach lens. + not by the findings' severity) needs no new lens round: the orchestrator checks the fix diff and the lenses' proof + tests itself and posts the override for the fix head. Before that it may ask for one quick check: one lens on the + fix head whose comment has a line `nit-only: yes` or `nit-only: no`. The quick check only adds an independent + read; **it does not replace the override**: it is one lens on a new head while the other lenses still name the + old one. It is a lens read of a new head, so **the gate counts it as a round**. On `nit-only: yes` the + orchestrator still posts the override for the fix head; on `no` the fix goes to the next round. When the quick + check's round reaches `rounds.approach_at`, run the approach lens too: one more independent read-only agent with + the prompt file the review block names for that lens, on the same head, posting its own report under that lens + name. The override alone, with no quick check, adds no round. + - An override covers lens reads; it clears a blocker or major, and waives a required approach lens, only when its + author is in the review block's `override_authors` (empty by default). An orchestrator's override never replaces + the approach lens. - A note between rounds that is no lens report starts with `not a lens:` (COMMENTS.md). - **No round past the limit.** A pull request still stuck (a blocker or major left) gets one audit by the `audit_runner` ("good enough to merge, or what next"), posted as a `not a lens:` comment; then a person decides. diff --git a/fork-kit/review/tests/comments.test.py b/fork-kit/review/tests/comments.test.py index d2116e69d26df..d5e660a06a839 100644 --- a/fork-kit/review/tests/comments.test.py +++ b/fork-kit/review/tests/comments.test.py @@ -16,6 +16,7 @@ sys.path.insert(0, os.path.join(KIT, 'lib')) import review_kit # noqa: E402 import review_config # noqa: E402 (the gate's, on the path review_kit set) +import review_glob # noqa: E402 (the gate's too) review_judge = review_kit.review_judge SHA = '0123456789abcdef0123456789abcdef01234567' @@ -252,6 +253,30 @@ def recommended(): return review_config.parse_review(json.load(f)) +PROTECTED_PROBES = ('.github/actions/x/action.yml', '.github/CODEOWNERS', '.github/dependabot.yml', + '.github/pr-queue.example.json', '.github/scripts/x.cjs', '.github/workflows', '.github/workflow/x.yml', + 'github/workflows/x.yml', 'x/.github/workflows/ci.yml', '.gitattributes', '.gitignore', 'package.json', + 'Dockerfile', 'Makefile', 'scripts/release.sh', 'docs/review.md', 'fork-ci/actions/pr-queue/lens.py', + 'fork-kit/review/SKILL.md', 'fork-kit/review/eval/lens.md', 'review/x.md', 'lenses/new.md') + + +def repo_paths(): + """Every file git tracks in this repository (none when git cannot say, and the probes still run).""" + try: + out = subprocess.run(['git', '-C', REPO, 'ls-files', '-z'], capture_output=True, check=True, timeout=60).stdout + except (OSError, subprocess.SubprocessError): + return [] + return [p for p in out.decode('utf-8', 'replace').split('\0') if p] + + +def lens_pr(*posts): + """A pull request whose comments are `posts` ((body), oldest first) as the gate reads them: written by the owner, + never edited, not hidden.""" + return {'comments': [{'body': body, 'createdAt': f'2026-01-01T00:00:{i:02d}Z', 'authorAssociation': 'OWNER', + 'includesCreatedEdit': False, 'author': {'login': 'someone'}} + for i, body in enumerate(posts)]} + + def templates(text): """Every fenced block whose first line is a lens header with a ``: the report shapes the docs tell a lens to write (the checked examples of COMMENTS.md hold real shas, so they are not templates).""" @@ -351,6 +376,15 @@ def test_the_skill_names_the_gates_protected_paths(self): self.assertEqual(review_judge.protected_paths(rv, cfg), {cfg, *(x.prompt for x in rv.lenses)}) used = set(re.findall(r'\b[A-Z][A-Z_]{2,}\b', inspect.getsource(review_judge.is_protected))) self.assertEqual(used, {'WORKFLOWS'}, 'is_protected reads another list: name it in SKILL.md, "Always critical"') + # The gate's answer, not its source: on every file of this repository and on probe paths next to the listed + # ones, is_protected says exactly what the skill's list says. A new rule of any form (a constant, an inline + # prefix, a suffix) that makes one more path protected fails here until the skill names it. + exact = review_judge.protected_paths(rv, cfg) + skill_says = lambda p: p in exact or review_glob.matches(review_judge.WORKFLOWS, p) # noqa: E731 + for path in sorted(set(repo_paths()) | set(PROTECTED_PROBES) | exact): + with self.subTest(probe=path): + self.assertEqual(review_judge.is_protected(path, exact), skill_says(path), + 'is_protected and SKILL.md "Always critical" disagree on this path') skill = ' '.join(kit_text('SKILL.md').split()) line = re.search(r'\*\*Always critical\*\*.*?\.(?= |$)', skill) # to the sentence's end self.assertTrue(line, 'SKILL.md no longer lists the always-critical paths') @@ -390,6 +424,58 @@ def test_the_skill_counts_rounds_as_the_gate(self): self.assertEqual((after.round, after.approach), (n, True)) self.assertFalse(review_judge.judge_rounds(rv, history, {}, decided=True).approach) + def test_the_skill_says_what_passes_a_fix_head_after_a_quick_check(self): + """The gate passes a head only when every lens's newest read names it, or a newer override covers them + (lens.Review.reviewed_head). The history: round 1 `code` at A, round 2 `code r2` at B, then a nit-only fix C + with a quick check: the quick check alone holds C, the override for C passes it, and the override alone adds + no round. The skill must say so: the quick check never replaces the override.""" + rv = recommended() + a, b, c = (f'{n:040x}' for n in (0xa, 0xb, 0xc)) + prefix = 'tool ready --head' + review = review_kit.reader(prefix) + + def report(name, sha): + return f'lens: {name} · reviewer: a · read at: {sha}\nstate: nothing is stored\nblocker: none\nmajor: none\n' + + override = f'{prefix} {c}: read the fix diff and its tests' + approach = rv.rounds.approach_lens + rounds = [report('code', a), report('code r2', b)] + cases = ( + ('quick check alone', rounds + [report('quick-check', c)], None), + ('quick check and the approach lens', rounds + [report('quick-check', c), report(approach, c)], None), + ('quick check, approach lens, override', rounds + [report('quick-check', c), report(approach, c), override], + c), + ('override alone', rounds + [override], c), + ) + for name, posts, passes in cases: + with self.subTest(history=name): + reviewed, sha, why = review.reviewed_head(lens_pr(*posts)) + self.assertTrue(reviewed) + self.assertEqual(sha, passes, why) + with_quick = review_judge.judge_rounds(rv, [('code', a), ('code r2', b), ('quick-check', c)], {}) + self.assertEqual((with_quick.round, with_quick.approach), (3, rv.rounds.approach_at <= 3)) + alone = review_judge.judge_rounds(rv, [('code', a), ('code r2', b)], {}) + self.assertEqual((alone.round, alone.approach), (2, rv.rounds.approach_at <= 2)) + skill = ' '.join(kit_text('SKILL.md').split()) + for words in ('the gate passes a head only when every lens\'s newest read names that head, or an override', + 'the fix head passes only when every lens reads it again (a new round) or the orchestrator posts ' + 'the override for it (no new round)', + '**it does not replace the override**', + 'On `nit-only: yes` the orchestrator still posts the override for the fix head', + 'The override alone, with no quick check, adds no round'): + with self.subTest(says=words): + self.assertIn(words, skill) + self.assertNotIn('skip the quick check', skill) + comments = ' '.join(kit_text('COMMENTS.md').split()) + self.assertIn('on `yes` it still posts the override for the fix head', comments) + + def test_the_skill_says_whichever_is_higher(self): + """review_round takes the higher of the change count and the round marker, never the sum.""" + self.assertEqual(review_judge.review_round([('code', 'a' * 40), ('code r3', 'a' * 40)]), 3) + self.assertEqual(review_judge.review_round([('code', 'a' * 40), ('code', 'b' * 40), ('code r2', 'c' * 40)]), 3) + skill = ' '.join(kit_text('SKILL.md').split()) + self.assertIn('the highest `r2`, `r3` marker in a lens name, whichever is higher', skill) + class RedCheck(unittest.TestCase): def test_each_rule_has_a_test_that_fails_without_it(self):