Repository navigation
feat: pause and resume every lane on an upstream outage without a human re-arm #9205
Description
No related issue: a new cross-lane posture the owner decided on 2026-10-22 (Q677).
Summary
During GitHub's API incidents on 2026-10-22 (bad reads 14:18 to about 17:41Z), degraded answers made pr-merge file false demotions that needed the owner to close them (#9145, #9167, #9168). The owner wants every lane to pause on an outage and resume by itself as soon as the service recovers, without hammering it. Owner decision Q677 (2026-10-22, "Yes, go with recommendations."): adopt this posture for every lane.
Fix
The posture, from the research in this session (sources: AWS Builders' Library, timeouts, retries and backoff with jitter, GitHub REST rate limits, Google SRE, handling overload):
- Scripts retry; the model never does. At most 3 attempts per call. Wait for
retry-afterif present; for a primary rate limit wait untilx-ratelimit-reset; for a secondary limit at least 60 s; otherwise capped exponential backoff with full jitter. A wait beyond a small per-step budget means unknown (2), not a sleep. Non-idempotent writes are not retried. - A read that fails, or a 200 whose completeness check fails, makes the verdict unknown: hold, file nothing, merge nothing, exit green with a notice.
- The next existing cron tick is the probe. No stored breaker, no re-arm, no status-page trigger.
- Demote or file a failure only on positive evidence from a complete read. A record a lane filed is closed by the lane when a later complete read contradicts it.
- The GitHub status page never gates, opens or closes anything; it may be quoted in a hold notice as context. On 2026-10-22 it showed only the Actions component, 14:18 to 15:18Z.
- Don't amplify load: serial reads, conditional requests (
etag) for repeated polls, default single-pending concurrency on scheduled lanes, optional work skipped while unknown, other lanes off minute 0.
Slices, in order:
- A shared retry helper that the lanes' scripts call, replacing pr-merge's fixed 15 s/30 s listing loop in
pr-merge.yml. - Element 4's self-close for lane-filed records (for example a
post-merge-failurerecord such as Post-merge dispatch failure after #9163..#9166: lane not demoted #9177 closing itself when main's next full run is green). - Each lane audited against elements 1-6, one slice per lane.
Open gaps from the research, to settle in the first slice: whether a degraded GitHub 200 reports the true total_count; Claude Code's own API retry policy; how idle in-step waits bill against Actions minutes.
Verification
- Each slice names the element it implements and a test for it.
- A degraded-read test (a listing answering 200 with items missing) holds the lane without filing or demoting, and the next complete read clears it.
Activity
kyle-sexton commentedon Oct 22, 2026
Decision recorded. Owner Q677 (2026-10-22, "Yes, go with recommendations."): adopt this posture for every lane, starting with the shared retry helper.
github-actions commentedon Oct 22, 2026
Proposed work class: work-class: structural
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: none
#9205 (comment)
This decision settles the open questions. The issue is now slice 1: a bash retry helper beside
rate-stop.bashthat replaces the fixed listing loop inpr-merge.yml. It holds green with a notice when the read is unknown. It comes with the retry, budget, no-write-retry and degraded-read tests, and the PR answers the three research gaps. Self-close and per-lane audits are separate follow-up issues.Automated triage, applied by the lanes App.
github-actions commentedon Oct 26, 2026
The triage sweep dispatched intake triage for this issue (attempt 1 of 2).
melodic-automation-lanes commentedon Oct 26, 2026
Not admitted to the issue lane: a criterion names .github/workflows/pr-merge.yml, which the issue lane cannot push (patch-protected-path); a builder session takes it. Admission takes the issue once each of those is met, when intake triages it again (
.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Oct 26, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.melodic-automation-lanes commentedon Oct 26, 2026
Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.
Intended outcome: A shared retry helper that lane scripts call replaces pr-merge's fixed 15 s/30 s listing loop, as the first slice of the pause-and-resume posture.
Related work
Open questions
RECOMMENDED: Yes: keep this issue to slice 1, and file slice 2 (self-close of lane-filed records) and one audit issue per lane afterward. Basis: judgment: the triage comment says the work is too big for one change and must be split into the listed slices
RECOMMENDED: A bash helper beside the existing .github/scripts/pr-merge/rate-stop.bash and .github/actions/log-api-budget/api-budget.bash, since the pr-merge listing loop is bash and already sources those. Basis: .base/.github/workflows/pr-merge.yml:739
RECOMMENDED: A 60 s budget per step; above it, exit 0 with a ::notice:: hold. The current exit 1 on exhausted retries and on a rate stop becomes a green hold, per element 2. Basis: .base/.github/workflows/pr-merge.yml:753-758
RECOMMENDED: No: the slice records what it finds on each in a doc or the PR, and the helper ships with a completeness check that is pluggable per caller. Basis: judgment: the triage comment says the gaps can be answered inside the first slice
RECOMMENDED: A bats or pytest test that stubs gh to fail twice then succeed (3 attempts max), honors retry-after, and returns unknown without sleeping past the budget; plus a degraded-listing test where a 200 has missing items holds without filing, and a later complete read clears it. Basis: The issue's Verification section
Sources read
Automated refinement by the lanes App. Answer with a comment whose first line is
**Decision recorded.**.kyle-sexton commentedon Oct 27, 2026
Decision recorded. The owner answered in the Refinement and Issue Triage sessions (2026-10-25), deciding the open items below on the facts:
The posture itself is owner decision Q677, recorded on the issue (#9205 (comment)):
The ledger row is
.work/poteto-review/interview-checklist.md:722("one posture issue, shared retry helper first"). No owner ledger names #9205 otherwise.Questions from intake's refinement (#9205 (comment)):
Decided: Yes. This issue is slice 1: the shared retry helper and the swap in pr-merge's listing loop. Basis: Q677 says "starting with the shared retry helper", and admission holds the issue as Size large pending the split (feat: pause and resume every lane on an upstream outage without a human re-arm #9205 (comment)). The body's other two slices become new issues that Issue Triage files: (a) element 4's self-close of lane-filed records, and (b) one audit issue per lane against elements 1-6. This record files nothing.
Decided: A bash file beside
.github/scripts/pr-merge/rate-stop.bash, which the listing step already sources (.github/workflows/pr-merge.yml:745-746). The existing retry helper,retry-fetch.mjs, is a Node fetch wrapper (.github/scripts/work-resolve-issue/retry-fetch.mjs:1-16) and cannot wrap the bashgh pr listloop, so it does not cover this.The helper adds no header parsing of its own. The rate guard already turns
retry-after,x-ratelimit-resetand the 60 s secondary-limit fallback into a recorded stop (.github/actions/log-api-budget/api-budget.bash:257,:273,:277,:164), readable withapi_budget_stop_until(:185) andrate_stop_from(.github/scripts/pr-merge/rate-stop.bash). The helper waits until that stop when it fits the budget.Decided: Green with a notice, per element 2 of the body ("hold, file nothing, merge nothing, exit green with a notice"). Today the loop exits 1 on a rate stop and after 3 failures (
.github/workflows/pr-merge.yml:758-765). Neither the body nor Q677 gives a budget number ("a small per-step budget"), so this record sets none: the PR states the value it passes and why.Decided: No. The PR records its answer to each. Basis: the body says they are "to settle in the first slice" (feat: pause and resume every lane on an upstream outage without a human re-arm #9205).
Decided: A test per element the slice implements, and the degraded-read test. Basis: the body's Verification section (feat: pause and resume every lane on an upstream outage without a human re-arm #9205).
Build: host session
Builder: Automation Lanes
Acceptance criteria:
.github/scripts/pr-merge/rate-stop.bashruns a call at most 3 times. For a rate limit it waits until the rate guard's recorded stop; otherwise it waits a capped exponential backoff with full jitter. A wait past the caller's budget is not slept: the helper returns unknown. A non-idempotent write is never retried..github/workflows/pr-merge.ymlcalls the helper in place of its fixedsleep $((attempt * 15))loop.::notice::hold, and the job lists, demotes, files and merges nothing.ghto fail twice then succeed and passes in 3 attempts; one asserts a wait past the budget returns unknown without sleeping; one asserts a write is not retried.total_count, Claude Code's own API retry policy, and how idle in-step waits bill against Actions minutes.kyle-sexton commentedon Nov 2, 2026
Delivered: closing as completed.
The 2026-10-27 record narrowed this issue to slice 1, and #11596 (32cf9a9) delivered it:
.github/scripts/pr-merge/retry-call.bashsits besiderate-stop.bash, with at most 3 runs, the rate guard's stop, full-jitter backoff and a budget that returns unknown.Merge lane holdnotice.The two remaining slices are filed so their scope stays open:
post-merge-failure. Demotion records stay person-closed.Author: the issue-triage session.