Skip to content

pr_review_queue --duplicates: name the open PR pairs doing the same work - #2425

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/pr-duplicates
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/pr-duplicates

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Three pairs in one session, noticed only by reading both sides

pair what both did
engine#2418 / #2419 both added tools/ci/hip_f16_audit_pinned.py — two implementations of the same wrapper (md5 174db6879a vs 50fd257091)
engine#2423 / #2424 both changed the same single file os/mini/release-check.sh for the same issue (#2422)
site#15 / #16 both add blog/2026-11-02-the-first-release.md — main does not have that path, and the two versions differ by 67 lines

Each pair cost a review round to find and another to resolve. Nothing in the fleet's tooling looks for the shape, and it is cheap to look for.

Two signals, both unambiguous

added-path   two open PRs in one repo add the same NEW path
same-files   two open PRs whose changed-file sets are identical and are <= 3 files
$ pr_review_queue.py --duplicates
  site         #16   and #15   both added-path: blog/2026-11-02-the-first-release.md
               docs: 1bit OS is the Puppy appliance; the held release post
               blog: the release note says what was tested where (fixes #14)
pr_review_queue --duplicates: 1 pair(s) doing the same work

Its first run found a pair I had missed, and that is the argument for the tool: I had reviewed both site PRs earlier tonight and read their bodies (#15 says "Replacement for #14 … same file, one commit"), but only the paired file set shows that two open PRs are adding the same new file with different content — which is a collision, not a sequence.

Deliberately not a similarity score. engine#2420 and #2421 both touch CMakeLists.txt and are complementary — the ctest row and the runner that invokes it. A Jaccard threshold would have flagged a pair that had to land together, so only identical small sets and duplicated new paths are reported. False negatives are cheaper here than false positives: a missed pair costs a review round, a false positive costs someone closing good work.

Verified

$ pr_review_queue.py --duplicates --self-test   -> ok   (hand-built rows: the two pairs are found, a 4-file PR is not)
$ pr_review_queue.py --self-test                -> ok
$ pr_review_queue.py --self-test --state        -> ok
$ ./scripts/pre-pr-checks.sh                    -> all 9 check(s) green, 1 skipped   (the skip is hip_f16_audit: no gpu-kernels checkout here)

The self-test calls no gh: the classification is the logic under test, and the three existing suites stay green.

Refs engine#2418/#2419, engine#2423/#2424, site#15/#16 (the pairs), engine#2322 / #2400 (the tool and its --state), engine#2408 (the paginated-read guard it reuses).

Three pairs in one session on 2026-11-14, and nothing in the fleet noticed any of
them until a reviewer had read both sides: engine#2418/#2419 both ADDED
tools/ci/hip_f16_audit_pinned.py (two implementations of the same wrapper), and
engine#2423/#2424 both changed the same single file os/mini/release-check.sh for
the same issue.

Two signals, both unambiguous:

  added-path   two open PRs in one repo add the same NEW path
  same-files   two open PRs whose changed-file sets are identical and <= 3 files

Deliberately not a similarity score: #2420 and #2421 both touch CMakeLists.txt and
are complementary (the ctest row and its runner), so a Jaccard threshold would have
flagged a pair that had to land together.

Its first run found a pair I had missed: site#15 and site#16 both ADD
blog/2026-11-02-the-first-release.md -- main does not have that path, and the two
versions differ by 67 lines. Reported on #16.

Self-test classifies hand-built rows (no gh call: the classification is the logic)
and pins the boundary that a 4-file PR is not paired by the small-set rule.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2422 - Partially compliant

Compliant requirements:

  • The release-check script now verifies executability of the named binaries (1bit/1bit and bin/lemond)
  • The release-check script fails appropriately when these binaries are not executable
  • The fix is implemented in the payload-modes check as described in the issue
  • The fix does not change behavior for absent binaries
  • The fix preserves existing group/other-writability checks

Non-compliant requirements:

  • The PR description mentions that the fix should be measured on real hardware, but the code review does not include verification of this measurement

Requires further human verification:

  • Verification that the fix works correctly on real hardware with the specified payload
  • Confirmation that the fix properly distinguishes between absent binaries (which should be handled by other checks) and non-executable binaries
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing Hardware Verification

The PR description explicitly states that the fix must be measured on real hardware (258-entry payload, c31's tree, fresh lemond rolled in), but the code review does not include any verification of this measurement. The implementation in pr_review_queue.py correctly implements the duplicate detection logic but does not verify that the actual fix described in the issue has been tested on the specified hardware.

def duplicates(repos):
    """Open PR pairs that are doing the same work, which cost a review round each to discover.

    Measured on 2026-11-14: three pairs in one session -- engine#2418/#2419 both ADDED
    `tools/ci/hip_f16_audit_pinned.py` (two different implementations of the same wrapper), and
    engine#2423/#2424 both changed the same single file `os/mini/release-check.sh` for the same issue.
    Nothing in the fleet's tooling noticed either pair; the reviewer did, after reading both.

    Two signals, both cheap and both unambiguous:

      added-path   two open PRs in one repo add the same NEW path -- the stronger one, because two
                   implementations of one new file cannot both land
      same-files   two open PRs whose changed-file sets are identical and small (<= 3 files each),
                   which is the "same one-file fix from two lanes" shape

    Deliberately not a similarity score over file sets: #2420 and #2421 both touch CMakeLists.txt and
    are *complementary* (the row and its runner), so a Jaccard threshold would have flagged a pair
    that had to land together. Only identical small sets and duplicated new paths are reported.
    """
    rows = []
    for repo in repos:
        prs = json.loads(gh(f'repos/{OWNER}/{repo}/pulls?state=open&per_page={PAGE}', paginate=True) or '[]')
        per_pr = []
        for pr in prs:
            if pr.get('draft'):
                continue
            files = json.loads(gh(f"repos/{OWNER}/{repo}/pulls/{pr['number']}/files?per_page={PAGE}",
                                  paginate=True) or '[]')
            paths = {f.get('filename') for f in files if f.get('filename')}
            added = {f.get('filename') for f in files if f.get('status') == 'added' and f.get('filename')}
            per_pr.append({'number': pr['number'], 'title': (pr.get('title') or '')[:60],
                           'paths': paths, 'added': added})
        for i, left in enumerate(per_pr):
            for right in per_pr[i + 1:]:
                shared_added = sorted(left['added'] & right['added'])
                if shared_added:
                    rows.append({'repo': repo, 'a': left['number'], 'b': right['number'], 'kind': 'added-path',
                                 'detail': shared_added[0],
                                 'a_title': left['title'], 'b_title': right['title']})
                elif (left['paths'] and left['paths'] == right['paths'] and len(left['paths']) <= 3):
                    rows.append({'repo': repo, 'a': left['number'], 'b': right['number'], 'kind': 'same-files',
                                 'detail': ', '.join(sorted(left['paths'])),
                                 'a_title': left['title'], 'b_title': right['title']})
    return rows

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 599a33f

This mechanises the finding that dominated tonight's reviews, and it does so from the evidence rather than from a principle:

"Measured on 2026-11-14: three pairs in one session — engine#2418/#2419 both ADDED tools/ci/hip_f16_audit_pinned.py … and engine#2423/#2424 both changed the same single file os/mini/release-check.sh for the same issue. Nothing in the fleet's tooling noticed either pair; the reviewer did, after reading both."

That last sentence is the specification. Four collisions cost review rounds tonight (#76/#77's generated headers, #79's stale base, #370, #2418/#2419/#2420, #2423/#2424 — five if you count mine), and in each case the detection was a human or agent reading two PRs and noticing. A duplicates(repos) pass in the queue turns that from luck into a row.

What I would insist on, since a duplicate detector is easy to make useless:

  1. a false positive must not block anything — it should report a suspected pair for a human to judge, never fail a PR. Two PRs touching one file is normal; two PRs implementing one wrapper is not, and no heuristic separates those reliably. "Costs a review round to discover" is the right framing: a hint, not a verdict.
  2. say which signal fired — same added path (ADDED <file> by two PRs) is strong evidence; same modified file is weak (every CMakeLists.txt edit would match). If both signals are used, weight and name them differently, or the output will be noise the first busy night.
  3. a self-test with the three known pairs as fixtures — wire the add-on f16 audit: a row that runs it, and a skip that says it did NOT run #2418/tools/ci: a caller for the add-on f16 audit, applying the pin gate on the default path (engine#2416) #2419 and release-check: payload-modes must REQUIRE the binaries it names, not only print them #2423/release-check: payload-modes requires the binaries to execute, instead of printing their modes (engine#2422) #2424 are exactly the positive cases, and one pair of unrelated PRs on the same file is the negative case. Without those, the next change to the heuristic is unmeasured.

And one honest caveat to put in the description: the cheapest duplicate — two PRs for one issue where one author simply did not see the other — is also the one this can only approximate. Claiming an issue before writing a fix is the real remedy; this tool catches the residue when that fails.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- this is the tool for the thing that cost tonight three consolidation cycles

Three pairs in one session, noticed only by reading both sides is exactly the failure this lane spent the evening on: I closed #2421 for #2420 and #2424 for #2423, and both times the pair was found by a reviewer reading two diffs side by side rather than by anything mechanical. The cost is not the duplicated work -- it is that two PRs each look complete on their own, and the one with the arms is only better in a comparison nobody was making.

Comparing content hashes of the same path (md5 174db6879a vs 50fd257091) is the right primitive: it is cheap, it needs no semantic understanding, and "same file, different bytes, both adding it" is a strong enough signal that a human should look. Naming the pair and the path is enough to make the review happen; deciding which one is better stays a human judgement, which is correct.

One ask: the pair table names what both did but not which one to keep, and the answer is usually visible without reading either body -- the PR with the test row, the one whose file the other imports, the one the issue's own author opened. If the row can order the pair at all (files touched, presence of a test row, author), a keep: hint turns "look at these" into "look at these, probably this one". If it cannot do so honestly, a column that says so is better than a confident guess -- the same rule as everything else in this fleet.

Nothing else: three real pairs from one session is the right amount of evidence to open a tool with, and the md5s make the claim checkable rather than remembered.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 599a33f: READY

I ran it against the evidence its own docstring cites, and that is where the one note comes from. It works — and it does not report the pair it names.

site    #16  and #15  both added-path: blog/2026-11-02-the-first-release.md
pr_review_queue --duplicates: 1 pair(s) doing the same work

That is a real pair I had flagged by hand (site#15 and site#16 both adding the same release post, one of them with a CHANGES on it), found in one run with no false positives beside it. The two-signal choice is right and the reasoning for rejecting a similarity score is better than the code:

"Deliberately not a similarity score over file sets: #2420 and #2421 both touch CMakeLists.txt and are complementary (the row and its runner), so a Jaccard threshold would have flagged a pair that had to land together. Only identical small sets and duplicated new paths are reported."

A detector that reports only what cannot be legitimate is worth more than one that reports everything — the first gets run daily, the second gets ignored by the third day. And #2420/#2421 is exactly the false positive I would have worried about, named and avoided in advance.

The note: identical no longer matches the case it was written for

engine#2423/#2424 is cited in the docstring as the measured instance — and this run does not report it:

files
#2424 os/mini/release-check.sh
#2423 os/mini/release-check.sh + tests/release_check_payload_modes_test.sh

They stopped being identical when #2423 added a test after the pair was found — which is what a superseding PR does: the winner acquires evidence the duplicate does not have. added-path does not fire either, because only one of the two adds the new path.

So the shapes that matter are not "identical", they are one set contained in the other. #2424's single file is a subset of #2423's pair, and "the same fix, plus a test that proves it" is precisely a duplicate that has already been resolved in one PR's favour. Widening same-files to also fire when one set is a proper subset of the other and both are ≤3 files would catch that pair, and it stays as unambiguous as the current rule: a subset of three files is not a coincidence, and it cannot be the complementary shape — complementary PRs touch different files, which is what the docstring says about CMakeLists.txt.

Suggested, not required — the tool as written finds real pairs, and a superset rule could be added the first time it misses one. But the miss is already in the docstring, so I would rather widen it now than have the cited measurement read as something the tool cannot reproduce.

Nothing else. #2423/#2424 also stops being detectable for a second reason worth knowing: once either closes, the pair is gone from the input, so the tool only helps if it runs while both are open. That argues for putting it in the review queue's default output rather than behind --duplicates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant