Repository navigation
fix(pr-pipeline): class a kill-switch refusal as a breaker trip in work-unblock - #9946
Conversation
…rk-unblock A run the token broker refused for the lanes kill switch did no work: at broker-check's broker ask (run 40127074215) it never reached a resolve-issue run job, and at a lane token request after the model ran (run 40127093047) no write runner minted. The jobs listing reads the same for a workflow-modified refusal, so the sweep treated the first as a failed run, spending the once-per-version re-fire, and the second as a failed mint, counting toward the failed-mint bound and its "fault in the lanes" backoff. work-unblock now reads one log per such run (broker-check's refused job, else the first failed mint's), capped at 1 MiB, and checks only the failed step's last line before `Process completed` for `lane-check-denied` or `lane-token-denied: the broker answered 403 with reason kill-switch`. A breaker trip is re-fired an hour after the run ended with a record of its own, and past limits.limit-stops-per-issue trips on the stranded backoff. Its records never spend the once-per-version re-fire, count toward the failed-mint bound, or raise another cause's backoff. Any other reason, or a log it cannot read, keeps the class it had before. createGitHub gains a maxBytes option: one fetch through the rate guard's latch, no retry, the body returned as text and refused past the cap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkaXdCkYZqLSTZKK8GkNnM
…ker-class # Conflicts: # .github/scripts/work-unblock/label-unblocked.mjs
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
claude-security-review reviewed No security issues found in the incremental change ( What I checked
PR context: The description contains no embedded instructions aimed at the review lane. Spend: The change adds up to one capped job-log fetch per run, drawing on the existing GitHub API rate budget only. That is not a billed charge. manifest fields: none touched (no critical-findings: 0 |
|
claude-review reviewed Incremental review of changes since 3a37d51, covering I read the incremental diff, the PR context, and the surrounding code in
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3a37d51c6a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…lper The capped log read lived in createGitHub (.github/actions/check-trusted-trigger), which is in the takeover set, so any change there blocks the pull request. It moves to .github/scripts/work-unblock/job-log.mjs: one fetch through the rate guard's latch, no retry, a body over the cap refused by its Content-Length or as it streams. main() builds it from ACTIONS_TOKEN and threads it to the refusal check; with none, a refusal keeps the class it had before. check-trusted-trigger.mjs and its tests are back to main's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkaXdCkYZqLSTZKK8GkNnM
…ckoff count The stranded backoff skipped every App re-add whose newest marker was a breaker record. The backoff comment is dated by its last edit, so an edit moved it past the re-adds it had made, and those then read as a breaker trip's: three ordinary re-fires around one trip counted as two and re-fired after 12 hours instead of 24. Now only the App's first re-add after each breaker record, dated by when it was posted, is skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkaXdCkYZqLSTZKK8GkNnM
No related issue: defect seen live in the 2026-10-24 kill-switch window (runs 40127074215 and 40127093047), fixed directly
Summary
When the token broker refuses a work-resolve-issue run because the lanes kill switch is on, the run does no work. The jobs API cannot tell that refusal apart from a
workflow-modifiedone, so.github/scripts/work-unblock/label-unblocked.mjshandled it like any other failure:Ask the broker whether the lane may run(run 40127074215, job 120970852314), the sweep counted it as a failed run and spent the issue's once-per-version re-fire. A real failure later on the same workflow files then went straight to the stranded backoff.Request the lane tokenstep after the model ran (run 40127093047, job 120972368518), the sweep counted it as a failed mint. That counted toward thelimits.limit-stops-per-issuefailed-mint bound, and past the bound it posted the infrastructure backoff comment, which says the fault is in the lanes.Both job logs end their failed step with the line
lane-check-deniedorlane-token-denied: the broker answered 403 with reason kill-switch: ..., followed directly by##[error]Process completed with exit code 1.(read this session).Fix
didNoWorkadds abreakerclass. It ranks abovecappedandblockedand belowmint-failed. For each run it reads one log, and only when no job minted a token. That log comes from broker-check's refused job, or else from the first job whose mint failed. The read is capped at 1 MiB; the real logs were 45 KB and 94 KB.Process completedline is matched, against an anchored pattern for the step's own mode (checkortoken), and only the reason word is read. A different reason, a lookalike, a line with no timestamp, an earlier forged line, an unreadable log, or a log over the cap all keep the class the run had before. A failed read logs only the job id and the error.limits.reset-clamp-minhours after the run ended, with its own<!-- work-unblock refired <key> breaker -->record. Afterlimits.limit-stops-per-issuetrips, counted since a listed human last addedagent-readyor a run last reported, it waits on the stranded backoff..github/scripts/work-unblock/job-log.mjs, kept outside the takeover set (check-trusted-triggeris unchanged). It makes one fetch through the rate guard's latch (.github/actions/log-api-budget/rate-guard.mjs) with no retry, and refuses a body past the cap, byContent-Lengthor as it streams. With noACTIONS_TOKENthere is no reader, and a refusal keeps its old class.actions: readcomment inwork-unblock.yml,docs/conventions/pr-pipeline.yamland apr-pipelinepatch changelog fragment are updated.Verification
.github/scripts/work-unblock/*.test.mjs178/178 (job-log 4, label-unblocked 174),.github/actions/log-api-budget/*.test.mjs64/64, and.github/actions/check-trusted-trigger/*.test.mjs362/362 with that action unchanged from main.agent-ready. They also cover seven refusals that are not breaker trips.scripts/check-changelog-fragments.sh --checkpasses.kill-switch, no CR bytes.Metric: person hand-offs (
needs-human) caused by breaker refusals.needs-humanwrite, atlabel-unblocked.mjs:1276, comes fromlimited/fundinglimit-stop markers, and a breaker trip leaves neither.Related
.github/actions/request-lane-token/action.yml(fail "$denied: the broker answered $status with reason $reason...").🤖 Generated with Claude Code
https://claude.ai/code/session_01FkaXdCkYZqLSTZKK8GkNnM