Skip to content

Require an attached file to be cited, not merely checked - #38

Merged
cabibbz merged 1 commit into
mainfrom
claude/pwa-chatgpt-model-selection-o38216
Jul 29, 2026
Merged

cabibbz merged 1 commit into
mainfrom
claude/pwa-chatgpt-model-selection-o38216

Conversation

@cabibbz

@cabibbz cabibbz commented Jul 29, 2026

Copy link
Copy Markdown
Owner

Eighteen files attached to one packet, and the review referenced none of them. Two causes, both real.

The policies never asked for the path

They required the reviewer to check a claim against an attachment, and to cite a path when the two disagreed. Nothing asked it to name the file otherwise — so a finding about attached code could be written with no path at all, which is exactly what happened.

Added to all four policies:

Attached files are the only code you can actually verify, so use them and show that you did. Cite the path whenever a finding concerns a file that is attached: a statement about attached code that names no path cannot be checked by the operator or acted on by the assistant, and reads as an opinion about code nobody can confirm you read. Where an attached file settles a question, quote the line that settles it. Where the attachments as a whole bear on the decision, say which ones you relied on. You need not discuss a file with no bearing on the decision, but never assert anything about a file whose contents you did not read.

The last clause matters as much as the first: this asks for grounding, not decoration. A file with no bearing on the decision still needn't be mentioned.

The measurement was also wrong

References were matched against the released text. A withheld or unselected candidate releases Bob Regress, so that match could find nothing however thoroughly the reviewer had cited its attachments. It now reads what the reviewer actually wrote.

Ungrounded reviews now look like a problem

None are referenced was a neutral grey note. Files sent and never used are files that did no work, so it now reads in amber and says so:

None of the 18 are referenced anywhere in the review, so the review is not grounded in the code you sent. Either those files do not bear on the decision, or the reviewer did not use them.

FileEvidence carries a level — used, partial, unused, none — and the card is styled from it.

Tests

  • Every policy, in both research modes, requires the path, the settling line, and forbids a claim about an unread file
  • A withheld candidate is matched against the reviewer's text: Bob Regress alone finds 0, plus the reviewer's own words finds 2
  • Levels: all cited → used, some → partial, none → unused, no attachments → none

121 unit tests (120 pass / 1 Windows-only skip), mocked e2e, lint, build clean, browser check at 390×844.

What to expect

The policy change only affects packets sent after you deploy. Reviews already in history keep their existing text — the card will now read them correctly, but it can't add citations that were never written.


Generated by Claude Code

Eighteen files were attached to one packet and the review referenced
none of them. Two causes, and both were real.

The policies asked the reviewer to check a claim against an attachment
and to cite a path when the two disagreed. They never asked it to name
the file otherwise, so a finding about attached code could be written
without a path at all. Every policy now requires the path whenever a
finding concerns an attached file, requires the settling line to be
quoted where an attachment settles a question, and forbids asserting
anything about a file whose contents were not read.

The measurement was also wrong. References were matched against the
released text, and a withheld or unselected candidate releases the
fixed terminal response, so that match could find nothing however
thoroughly the reviewer had cited its attachments. It now reads what
the reviewer actually wrote.

A review that references none of its attachments now reads in amber
and says so plainly. Files sent and never used are files that did no
work, and that belongs on the screen as a problem rather than as a
neutral observation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxxEQsJEXq9hZmjpNH5PxY
@vercel

vercel Bot commented Jul 29, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
sol-review-gate Building Building Preview, Comment Jul 29, 2026 2:32pm

Request Review

@cabibbz
cabibbz merged commit 17d4884 into main Jul 29, 2026
2 of 3 checks passed

This branch was successfully deployed

1 active deployment
Preview — b92616ed Deployed Jul 29, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants