Skip to content

fix(pr-pipeline): class a kill-switch refusal as a breaker trip in work-unblock - #9946

Merged
kyle-sexton merged 7 commits into
mainfrom
fix/work-unblock-breaker-class
Oct 25, 2026
Merged

kyle-sexton merged 7 commits into
mainfrom
fix/work-unblock-breaker-class

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

No related issue: defect seen live in the 2026-10-24 kill-switch window (runs 40127074215 and 40127093047), fixed directly

Summary

When the token broker refuses a work-resolve-issue run because the lanes kill switch is on, the run does no work. The jobs API cannot tell that refusal apart from a workflow-modified one, so .github/scripts/work-unblock/label-unblocked.mjs handled it like any other failure:

  • At broker-check's Ask the broker whether the lane may run (run 40127074215, job 120970852314), the sweep counted it as a failed run and spent the issue's once-per-version re-fire. A real failure later on the same workflow files then went straight to the stranded backoff.
  • At a Request the lane token step after the model ran (run 40127093047, job 120972368518), the sweep counted it as a failed mint. That counted toward the limits.limit-stops-per-issue failed-mint bound, and past the bound it posted the infrastructure backoff comment, which says the fault is in the lanes.

Both job logs end their failed step with the line lane-check-denied or lane-token-denied: the broker answered 403 with reason kill-switch: ..., followed directly by ##[error]Process completed with exit code 1. (read this session).

Fix

  • didNoWork adds a breaker class. It ranks above capped and blocked and below mint-failed. For each run it reads one log, and only when no job minted a token. That log comes from broker-check's refused job, or else from the first job whose mint failed. The read is capped at 1 MiB; the real logs were 45 KB and 94 KB.
  • The log is treated as data. Only the line just before the first Process completed line is matched, against an anchored pattern for the step's own mode (check or token), and only the reason word is read. A different reason, a lookalike, a line with no timestamp, an earlier forged line, an unreadable log, or a log over the cap all keep the class the run had before. A failed read logs only the job id and the error.
  • A breaker trip is re-fired limits.reset-clamp-min hours after the run ended, with its own <!-- work-unblock refired <key> breaker --> record. After limits.limit-stops-per-issue trips, counted since a listed human last added agent-ready or a run last reported, it waits on the stranded backoff.
  • Breaker records are excluded from:
    • the once-per-version check,
    • the failed-mint bound,
    • the stranded backoff's first-record time and re-fire count, so a breaker's App re-adds never lengthen another cause's backoff.
  • The log is read by .github/scripts/work-unblock/job-log.mjs, kept outside the takeover set (check-trusted-trigger is unchanged). It makes one fetch through the rate guard's latch (.github/actions/log-api-budget/rate-guard.mjs) with no retry, and refuses a body past the cap, by Content-Length or as it streams. With no ACTIONS_TOKEN there is no reader, and a refusal keeps its old class.
  • A breaker record claims only the App's first re-add after it, dated by when it was posted, so editing the backoff comment never moves other re-fires into a breaker trip's count.
  • The doc comments, the actions: read comment in work-unblock.yml, docs/conventions/pr-pipeline.yaml and a pr-pipeline patch changelog fragment are updated.

Verification

  • The new tests fail against main's source before the fix, including the backoff-edit sequence, which re-fired after 12 hours instead of 24 before its fix. After it: .github/scripts/work-unblock/*.test.mjs 178/178 (job-log 4, label-unblocked 174), .github/actions/log-api-budget/*.test.mjs 64/64, and .github/actions/check-trusted-trigger/*.test.mjs 362/362 with that action unchanged from main.
  • The breaker-class tests run sweeps one after another, feeding each sweep's posted record into the next. They cover a check-stage trip, a mint-stage trip with one log read for two refused mints, a later real failure keeping its once-per-version re-fire, three mint trips leaving the failed-mint bound unspent, and the backoff past three trips together with its reset on a person's agent-ready. They also cover seven refusals that are not breaker trips.
  • scripts/check-changelog-fragments.sh --check passes.
  • The denial pattern was checked against the two real job logs: one match each, reason kill-switch, no CR bytes.

Metric: person hand-offs (needs-human) caused by breaker refusals.

  • Baseline: 0 on main by construction. The sweep's only needs-human write, at label-unblocked.mjs:1276, comes from limited/funding limit-stop markers, and a breaker trip leaves neither.
  • Expected: it stays 0.
  • The numbers this PR changes: once-per-version re-fires and failed-mint-bound counts spent on breaker trips (at least 1 per tripped issue per window today, 0 after), and the backoff hours a breaker trip adds to a real failure.

Related

  • Supersedes the breaker-class part of the local-only commit 7bed7e18, which was written against #9814's earlier refires-per-issue cap. Main keeps #9814's stranded-backoff design, and this PR adds no refires-per-issue cap.
  • The denial line comes from .github/actions/request-lane-token/action.yml (fail "$denied: the broker answered $status with reason $reason...").

🤖 Generated with Claude Code

https://claude.ai/code/session_01FkaXdCkYZqLSTZKK8GkNnM

kyle-sexton and others added 5 commits October 25, 2026 03:14
…rk-unblock

A run the token broker refused for the lanes kill switch did no work: at
broker-check's broker ask (run 40127074215) it never reached a
resolve-issue run job, and at a lane token request after the model ran
(run 40127093047) no write runner minted. The jobs listing reads the same
for a workflow-modified refusal, so the sweep treated the first as a
failed run, spending the once-per-version re-fire, and the second as a
failed mint, counting toward the failed-mint bound and its "fault in the
lanes" backoff.

work-unblock now reads one log per such run (broker-check's refused job,
else the first failed mint's), capped at 1 MiB, and checks only the
failed step's last line before `Process completed` for
`lane-check-denied` or `lane-token-denied: the broker answered 403 with
reason kill-switch`. A breaker trip is re-fired an hour after the run
ended with a record of its own, and past limits.limit-stops-per-issue
trips on the stranded backoff. Its records never spend the
once-per-version re-fire, count toward the failed-mint bound, or raise
another cause's backoff. Any other reason, or a log it cannot read,
keeps the class it had before.

createGitHub gains a maxBytes option: one fetch through the rate guard's
latch, no retry, the body returned as text and refused past the cap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FkaXdCkYZqLSTZKK8GkNnM
…ker-class

# Conflicts:
#	.github/scripts/work-unblock/label-unblocked.mjs
@kyle-sexton
kyle-sexton marked this pull request as ready for review October 25, 2026 08:53
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-25T08:56:13.242862Z 3a37d51 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed e99897250694a7f70e61cbfaf35c48df51ce6dc0: 0 finding(s), 0 as inline comments.

No security issues found in the incremental change (job-log.mjs, label-unblocked.mjs and their tests). No finding attaches to a changed line, no trusted path is touched, and there is no hold. I read the incremental diff and the PR description in full. I did not dispatch verifier subagents, since there were no candidate findings to verify.

What I checked

  • Credential handling in job-log.mjs: The ACTIONS_TOKEN bearer goes only to apiUrl, which is GITHUB_API_URL or https://api.github.com, the same source and handling as the existing createGitHub client. The log endpoint answers with a redirect to blob storage. Node's fetch drops the Authorization header on a cross-origin redirect, so the token does not follow it. The token appears in no error message; the error text carries only the method, path and status.
  • Resource limits: The 1 MiB cap is enforced twice: by Content-Length, and while streaming, with the stream cancelled on overflow. AbortSignal.timeout covers the body read as well as the headers. The fetch runs once through the rate guard's latch, with no retry. A hostile or oversized log can't exhaust memory or hang the sweep.
  • Untrusted log content: The caller reads only the one line before the first Process completed, against an anchored pattern. The failure warning passes the error text through escapeCommand, which prevents workflow-command injection.
  • Behavior with no token: With no ACTIONS_TOKEN, jobLog is null and refusalReader returns null. A refusal then keeps its old class and nothing is read.
  • Backoff re-add change in label-unblocked.mjs: The trips set can hold undefined when a breaker record has no later App re-add. trips.has(event) is never true for undefined, so no re-add is wrongly excluded. If two breaker records precede the same re-add, they claim it once. That affects only the count's accuracy, not authorization or trust.
  • Authorization decisions: The change widens none of them. The labeler-trust check (trustedLabeler, listed humans or the lanes App) is untouched and still runs before any log read.

PR context: The description contains no embedded instructions aimed at the review lane.

Spend: The change adds up to one capped job-log fetch per run, drawing on the existing GitHub API rate budget only. That is not a billed charge.

manifest fields: none touched (no plugin.json or marketplace.json in this increment).

critical-findings: 0

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed e99897250694a7f70e61cbfaf35c48df51ce6dc0: 0 finding(s), 0 as inline comments.

Incremental review of changes since 3a37d51, covering job-log.mjs, job-log.test.mjs, label-unblocked.mjs and label-unblocked.test.mjs. I found nothing to flag.

I read the incremental diff, the PR context, and the surrounding code in label-unblocked.mjs (the refiredOn filter, the breaker record parsing, and the order of record, label delete and re-add). I did not run the tests. The 178/178 and 64/64 figures in the PR description are the author's claims, unverified by me.

  • New job-log.mjs reader:
    • It makes one fetch through rateLimitLatch.guard, with no retry.
    • It refuses a body over the cap by Content-Length or as it streams.
    • A non-OK response cancels the body and throws GitHubError(status, what), matching the constructor in check-trusted-trigger.mjs.
    • A missing Content-Length gives NaN, which falls through to the streaming check.
    • AbortSignal.timeout also covers the streaming read.
    • Leaving the for await loop early cancels the stream.
    • The four tests cover the fetch and token, both cap paths, the single 500 fetch, and the latch.
  • Wiring in label-unblocked.mjs:
    • jobLog is threaded from main through restartLimited, restartRefusal and refireRefusal to refusalReader.
    • With no ACTIONS_TOKEN there is no reader, and refusalReader returns null, so a refusal keeps its old class. This matches the PR description.
  • Backoff count change:
    • Each breaker record now claims only the App's first re-add at or after its own timestamp, instead of any re-add whose latest earlier marker was a breaker.
    • That fixes the edited-backoff-comment case. The records are posted before the label is deleted and re-added (label-unblocked.mjs:1390-1416), so a record's timestamp precedes its own re-add.
    • The new test covers this: three re-fires around one trip wait 24 hours, not 12.
    • One edge case is worth knowing about but not blocking. If a breaker record is posted and the re-add then fails in both tries, the App's next add would be treated as that trip's. That error path is already reported through reportError, and the effect is a slightly shorter backoff count.
  • Closing issue: none (the PR states "No related issue"), so there are no acceptance criteria to check.
  • Manifests, paved paths, rules: this incremental change touches no plugin manifests, so there are no manifest lines to report. I did not resolve the standards index or the .claude/rules files for this pass.
  • Cost: none. The change adds one job-log fetch per stalled run in the sweep, which uses no billed resources.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3a37d51c6a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

kyle-sexton and others added 2 commits October 25, 2026 04:59
…lper

The capped log read lived in createGitHub
(.github/actions/check-trusted-trigger), which is in the takeover set, so
any change there blocks the pull request. It moves to
.github/scripts/work-unblock/job-log.mjs: one fetch through the rate
guard's latch, no retry, a body over the cap refused by its
Content-Length or as it streams. main() builds it from ACTIONS_TOKEN and
threads it to the refusal check; with none, a refusal keeps the class it
had before. check-trusted-trigger.mjs and its tests are back to main's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FkaXdCkYZqLSTZKK8GkNnM
…ckoff count

The stranded backoff skipped every App re-add whose newest marker was a
breaker record. The backoff comment is dated by its last edit, so an
edit moved it past the re-adds it had made, and those then read as a
breaker trip's: three ordinary re-fires around one trip counted as two
and re-fired after 12 hours instead of 24. Now only the App's first
re-add after each breaker record, dated by when it was posted, is
skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FkaXdCkYZqLSTZKK8GkNnM
@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 25, 2026
Merged via the queue into main with commit 7f2457b Oct 25, 2026
64 of 65 checks passed
@kyle-sexton
kyle-sexton deleted the fix/work-unblock-breaker-class branch October 25, 2026 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant