Skip to content

fix(secret-sweep): sweeps of a ref one at a time; a reset run never falls back to the old record (#791) - #948

Merged
scuttletoby merged 4 commits into
mainfrom
fork/secret-sweep-791
Nov 22, 2026
Merged

scuttletoby merged 4 commits into
mainfrom
fork/secret-sweep-791

Conversation

@scuttletoby

@scuttletoby scuttletoby commented Nov 22, 2026 •

Copy link
Copy Markdown
Owner

Fixes #791

Point 3 of #791 (the live check, result "not shown") is now tracked in #950, so this PR still closes #791.

What changed

The five follow-ups from the two reviews of #782. Only fork-owned files: fork-secret-sweep.yml, fork-secret-scan.yml (the read step) and fork-contracts.test.py.

  1. Header wording (fork-secret-sweep.yml). A cancelled reset run is started again "as a new run by hand with reset_record, not a re-run" (a re-run stays the older run, so the next sweep would read the newer sweep's record). "First attempt" now says "newest earlier attempt" (the upload uses overwrite, so from attempt 3 on it is the newest one). The read step's comments say the same.
  2. An undone reset is no longer silent: sweeps of one ref run one at a time. fork-secret-sweep.yml gets a workflow concurrency group fork-secret-sweep-${{ github.ref }}, cancel-in-progress: false.
    • A sweep that starts while a reset run is running now waits. Then it reads the reset run's record, so it cannot undo the reset.
    • Why it cannot cancel or starve a scheduled sweep for ever: GitHub keeps one run waiting per group, and a newer run cancels the one that waits, never the one running. The running run always ends (job time-outs of 30 and 5 minutes), so the waiting one always starts, unless a newer run replaced it. The newer run is the same workflow on the same ref: it sweeps every ref and alerts the same way (the alert job runs for a schedule or for the default branch, and both share the ref). So a scheduled sweep can be replaced, but the sweep and its alert still happen.
    • A reset run that a newer run replaces while it waits shows as cancelled and changed nothing, which is visible in the runs list. It is not read later (the reader takes only success or failure). The header says to start it again by hand.
    • Groups are per ref, and the record is per branch, so a sweep on another branch never waits for main's.
    • The group is only in the sweep workflow. fork-secret-scan.yml still has none: a push's scan must not wait or be cancelled (the existing contract still checks that).
    • I chose this over the run-name marker: it prevents the undo instead of detecting it, and needs no new API reads.
  3. Live check of GITHUB_WORKFLOW_REF in the called workflow: see "Live check" below.
  4. A reset run never falls back to the old record. In the read step, none failed in a reset run on a branch now writes the empty record (read=reset) and logs the reason as a warning. Before, a failed tag check, a tag named like the branch, or a failure to list, compare, download or read its own record gave read=failed. If attempt 1 had kept nothing (for example it was cancelled before the keep step), attempt 2 then kept nothing unless it ran to the end, and the next sweep read the record from before the reset. The empty record trusts nothing, so it is always safe and costs only progress. A reset run on a tag is unchanged (no record; the keep step never keeps one on a tag).
  5. The time-limit check fails instead of skipping. If the ((SECONDS > N)) line is missing or reworded, lookup_problems now reports "no lookups' time limit". Before, if limit and … skipped the check, and only the exact-text contract held the line.

Fix round after the two lens reviews (both "safe to merge", minors only)

  • The alert job's concurrency group must differ from the workflow group. If the alert job used the new workflow group (fork-secret-sweep-${{ github.ref }}), GitHub would see a deadlock and cancel the alert, so a failed scheduled sweep would alert no one. The contract now refuses an alert group equal to the workflow group (spaces inside ${{ }} ignored). Two new mutants: the alert given the sweep's group, and the same group spaced otherwise.
  • Header wording (fork-secret-sweep.yml and the read step in fork-secret-scan.yml): a re-run going on from its own earlier attempt's record is meant to work but is not yet confirmed live (Secret sweep: a re-run does not find its own earlier attempt's record #950). "A sweep cancelled so loses nothing" is replaced by the true statement: a new sweep cancelled so loses only its own run, but a re-run cancelled so makes the whole run cancelled, and its earlier attempt's record is no longer read. Added: never re-run a finished reset run; start a new reset run by hand.
  • The branch is updated from main (a merge, no force push).

Tests

  • python3 .github/scripts/fork-contracts.test.py (Linux, bash 5.2): all passed: 732 ok, 0 skipped (after the fix round; 730 before it).
  • This test file against main's workflows: 16 fail, among them:
    • all six new reset re-run cases (own record cannot be listed, tag check fails, tag named like the branch, own record cannot be downloaded, cannot be read, commit check fails). Each gives read=failed and no record on main;
    • the new read-then-scan case: a reset run's re-run whose own record cannot be listed, then a lookup fails half-way. On main nothing is kept; now the proven tips are kept in a fresh record;
    • the sweep's concurrency contract, and its four mutants (group removed, cancel-in-progress: true, a per-run group, the alert's group);
    • the two new in-suite mutants of the read step (the empty-record fallback removed, and its branch check removed);
    • the exact-text contracts (expected).
  • Point 5 without its fix (only the if not limit hunk of the test file reverted): both new checks fail with "no time-limit problem": the line reworded to >=, and the line gone.
  • actionlint 1.7.12 with shellcheck 0.10.0 on both changed workflows: no findings.
  • vp fmt --check on the touched files: clean (the two workflows; the formatter does not cover .py).
  • Fix round, alert group check without its fix (only the new elif same_group(...) lines removed): both new mutants fail ("2 failed").
  • fork-seam-check.cjs --config-ref origin/main origin/main HEAD: only the pinned seams (no upstream file touched).

Live check (point 3)

Result: not shown. The re-run did not read its own record, so this check does not prove the called workflow sees the caller's GITHUB_WORKFLOW_REF. Follow-up: #950.

  • No sweep had run on main since fix(secret-sweep): a reset run's re-run goes on from its own record; gates held by behaviour (#703) #782 merged (the last one, run 44905498728, was before it and was attempt 1). So I dispatched exactly one normal run (no reset_record) on main: run 45017065212. A first attempt can only show "record from run ". "Record from run " appears only on a re-run, so I also re-ran that run once (attempt 2). Nothing else was dispatched.
  • Attempt 1 warned "no earlier sweep on main kept a record (or it expired)". That is correct: run 44905498728 kept a record, but its commit is no longer in main (compare says diverged, 779 ahead / 379 behind, because main was rewritten since). The read step skips a record whose commit the branch does not have. Attempt 1 kept its own record (2222 bytes).
  • Attempt 2 also warned "no earlier sweep on main kept a record (or it expired)". It did not show "the sweep's record from run 45017065212". Attempt 1's record still existed when attempt 2 read (attempt 2 replaced it about 4 minutes later, overwrite: true; the old artifact id now returns 404). So the own-record branch was skipped, and the log cannot say which of two causes did it:
    1. in the called workflow, GITHUB_WORKFLOW_REF is not …/fork-secret-sweep.yml@…, so self stays empty and the run skips its own id; or
    2. at that point, the run's artifact listing does not include an earlier attempt's artifact.
  • Either way this PR stays safe. A reset run's re-run that cannot use its own record now takes the empty record, never the one from before the reset. A normal run's re-run reads the newest earlier sweep, as a first attempt does. What does not work live is the secret sweep record: follow-ups after #701 #703 "a re-run continues from its own earlier attempt" path. Suggested follow-up: on a re-run, add a notice with the two facts (whether the ref matched, and how many own-run records were listed). One more re-run would then settle it.

The state question

  • Runs again:
    • A re-run of a reset run is meant to read its own newest earlier attempt's record (not yet confirmed live, Secret sweep: a re-run does not find its own earlier attempt's record #950). Never re-run a finished reset run; start a new reset run by hand. If it cannot be read, or anything about the read is in doubt, it uses the empty record and never an older run's (6 new cases, 2 mutants).
    • A second sweep on the same ref waits for the first and then reads its record.
  • Stops half-way:
    • A reset run's re-run that stops in the lookups now keeps a fresh partial record, because read=reset allows it (the new read-then-scan case). Before, it kept none, which left the pre-reset record in use.
    • A reset run cancelled by hand, or replaced while waiting, has the conclusion cancelled, so later sweeps do not read it. That is visible, not silent.
    • Partial-record rules after a normal failed read are unchanged (failed keeps none).
  • Old, empty or missing record:
    • A normal run is unchanged: none when no run kept a record; failed when one may exist but cannot be read; a corrupt record is rejected.
    • A reset run always ends with the empty record or a record written after the reset. It never ends with the old one.
    • The record format is unchanged ("record 2"), so the first sweep after the merge reads main's current record.

General vs project

  • General: the per-ref sweep concurrency, the reset run's empty-record fallback and the time-limit check are part of the fork's generic secret-scan and sweep workflows and their contract tests. They hold for any repository that uses them.
  • Project: nothing new. The workflow names and the 20- and 30-minute values are as before.

🤖 Generated with Claude Code

…alls back to the old record (#791)

- fork-secret-sweep.yml: a per-ref concurrency group (cancel-in-progress false), so a sweep that
  starts during a reset run waits for it and reads its record; header reworded (a new run by hand
  with reset_record, not a re-run; "newest earlier attempt").
- fork-secret-scan.yml: in a reset run on a branch, any failed or doubtful read (tag check, own
  record listed, compared, downloaded or read) gives the empty record (read=reset), never
  read=failed, so the record from before the reset cannot stay in use.
- fork-contracts.test.py: the sweep's concurrency contract and mutants; six reset re-run read cases
  and a read-then-scan case; the time-limit check fails when its line is missing or reworded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: merge-path · reviewer: lens agent (Opus 5.5) · read at: 68479a3
verdict: safe to merge
covered: issue #791 (all five points), the PR body, the full diff of fork-secret-sweep.yml, fork-secret-scan.yml (read step, scan step's partial and keep step) and fork-contracts.test.py; every concurrency group in .github/workflows (no other group can equal fork-secret-sweep-refs/...; nothing else calls fork-secret-sweep.yml; release workflows use their own groups)
checks: ran fork-contracts.test.py on this head: all passed, 730 ok, 0 skipped. Own mutants: (1) the alert job's group set to the new workflow group: still 730 ok (see minor 1); (2) the time limit raised to 99999 s: 6 FAIL; (3) the limit line reworded to >= with the old text left in a comment above it: 6 FAIL (the exact-text contract and the e2e case catch it, so the regex finding a comment does not open a gap). Traced every none call in a reset run by hand.
state: runs again: a second sweep of the same ref waits for the running one and then reads its record; a newer run replaces only the waiting run, never the running one, so a reset run that is running cannot be undone; a re-run of a reset run reads its own newest earlier attempt or, on any failed or doubtful read, the empty record (never the pre-reset one). Stops half-way: a reset run's re-run whose read fails now gets read=reset, so a lookup failure keeps a fresh partial record of tips proven in this run; a cancelled reset run is cancelled, not read, and visible. Old or empty state: the empty record trusts nothing (every deleted tip is looked up again), so the fallback costs only progress and cannot hide a leak; a normal run's failed read still gives read=failed and keeps the old record; record format unchanged.
blocker: none
major: none
minor: 1. .github/scripts/fork-contracts.test.py:1669 (alert job concurrency check) only asks for a non-empty group with cancel-in-progress false. Setting the alert job's group to fork-secret-sweep-${{ github.ref }} (the new workflow group) passes all 730 tests, but GitHub detects that as a deadlock and cancels the alert job, so a failed scheduled sweep would again alert no one. Before this PR there was no workflow group to collide with. Fix: require the alert group to differ from SWEEP_CONCURRENCY["group"] (or pin it exactly), plus a mutant.
minor: 2. .github/workflows/fork-secret-sweep.yml:20-21 and fork-secret-scan.yml:76-77 state as fact that a re-run "goes on from its own newest earlier attempt's record". The PR's own live check (point 3 of #791) did not show that path working: attempt 2 did not read attempt 1's record. The safe fallback holds either way, but the header should say this path is not yet confirmed live (pointing at the follow-up issue), so nobody plans a reset re-run around it.
minor: 3. PR body "Fixes #791": point 3 (the live check) is not done; its result is "not shown". On merge this closes #791 with that point open. Use "Part of #791", or file the follow-up issue first and link it from #791 before merging.
minor: 4. Concurrency, checked and fine: a waiting scheduled sweep can be replaced only by a newer run of the same workflow on the same ref (schedule runs on the default branch, so the replacer also alerts); a waiting reset run replaced so shows as cancelled, as the header says; the running run ends within the 30 + 5 minute job time-outs, so nothing waits for ever; the scan workflow still has no group (contract holds), so pushes, pull requests and fork-tests' release/main calls are never held by it.

@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: state · reviewer: lens agent · read at: 68479a3
verdict: safe to merge
covered: issue #791 (all five points), the PR body, the full diff (fork-secret-sweep.yml, the read and keep steps of fork-secret-scan.yml, fork-contracts.test.py), and the whole read, scan and keep steps the change sits in.
checks: ran fork-contracts.test.py on the head: all passed, 730 ok, 0 skipped. Ran the head's test file against the parent commit's workflows: 16 failed, the same 16 the PR lists (the six reset re-run read cases, the read-then-scan case, the four concurrency mutants plus the contract, the two in-suite read-step mutants, and the exact-text contracts). The new tests fail on the old code. Each fix line has a test that goes red without it: the fallback and its branch check (in-suite mutants), the concurrency block (four mutants), and the time-limit check (two cases).
state: Runs again: a reset run's re-run always ends with its own record (read=found) or the empty record (read=reset). Every failure that goes through none() on a branch now gives read=reset, so the keep step always keeps a post-reset record. The older run's record is never left in use. If the live self path does not work (point 3), self stays empty, so a reset re-run takes the empty record straight away, which is still safe. A second sweep on the same ref now waits, which also closes the old window where a sweep skipped a reset run still in progress (the reader takes only completed runs). Stops half-way: a cancelled first attempt keeps nothing and its run is not read; a re-run after a read failure keeps the empty record or a partial one written after the reset. The only way a reset re-run still keeps nothing is a hard failure in the read step outside none() (the runner dies), which is negligible. Old records: the format is unchanged (record 2). The first sweep after the merge reads the newest completed main run whose commit main has (today the live-check run, attempt 2's record). Records from before main was rewritten are skipped (compare says diverged), and a compare that errors gives read=failed with a full check, which is also safe. No record at all: read=none and a full check, as before. Both fallbacks cost progress only, never coverage. The one way left for a reset to be undone is the re-run case in the first minor below.
blocker: none
major: none
minor: fork-secret-sweep.yml:24-26 says "a sweep cancelled so loses nothing". That holds for a new run, but not for a re-run attempt. If someone re-runs the finished reset run while another sweep is running, the re-run waits. A newer sweep then cancels it, and the whole reset run's conclusion becomes cancelled. The reader (fork-secret-scan.yml:140) takes only success or failure, so the reset run's record (from attempt 1) is no longer read. If the sweep in between also keeps no record (it failed before keep), the next sweep reads the record from before the reset, and nothing warns. This needs three things to happen together and starts with a person's re-run. Suggested fix: one sentence in the header, such as "never re-run a finished reset run; start a new one". Or a test, plus a reader that takes a cancelled run's record (the keep step uploads only whole records, also on cancel).
minor: fork-secret-sweep.yml:20-21 and fork-secret-scan.yml:76-77 state as fact that a re-run "goes on from its own newest earlier attempt's record". The PR's own live check (point 3) did not show that: attempt 2 of run 45017065212 did not read attempt 1's record. Suggested fix: word it as intended behaviour that is not yet confirmed live, and name the follow-up issue (until then a re-run starts from the empty record, or from the newest earlier sweep's record, which is safe).
minor: the PR body says "Fixes #791", but point 3 of #791 (the live check) ended as "not shown". Merging closes the issue with that point still open. Suggested fix: "Part of #791", or file the follow-up issue first and link it in the body.

scuttletoby and others added 3 commits November 22, 2026 12:07
…un path unconfirmed (#791, #950)

- Contract: the sweep's alert job's concurrency group must differ from the
  new workflow-level group (the same group is a deadlock: GitHub cancels
  the alert). Two mutants: the sweep's group, and the same group spaced
  otherwise.
- Headers of fork-secret-sweep.yml and the read step of
  fork-secret-scan.yml: a re-run going on from its own earlier attempt's
  record is intended but not yet confirmed live (#950). A cancelled
  re-run makes the whole run cancelled and its record unread, so never
  re-run a finished reset run; start a new reset run by hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby
scuttletoby merged commit 94c36bb into main Nov 22, 2026
40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Secret sweep: review follow-ups from #782

1 participant