You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ci: pr_agent is cancelled, not failing -- 32 of the last 40 pull_request runs, because every push cancels the queued review #2077
ci: pr_agent is cancelled, not failing — 32 of the last 40 runs on pull_request events, because every push cancels the pending review
While fixing #1504 (the runner that had been dead 25 hours) I found the other reason PRs read red on pr_agent alone, and it is not the model or the job:
$ gh run list --workflow=pr-agent.yml --event pull_request --limit 40
last 40 pr_agent runs on pull_request events: {'queued': 7, 'cancelled': 32, 'success': 1}
span: 2026-11-10T22:41 .. 2026-11-11T00:08
and on a specific PR, the check itself is CANCELLED:
Cause: the job carries concurrency: { group: pr-agent-<pr>, cancel-in-progress: true }. That is the right shape for a fast check — the newest commit wins —
but this job runs a 30B model review that takes minutes on one self-hosted runner. So with several lanes pushing to the same PR, and the fleet pushing to many PRs, a
job is usually still queued when the next synchronize event arrives, and it is cancelled before a single step runs. The result is a check that is neither
green nor a verdict about the diff — and a merge gate that reads it as not-green.
Reproduced structurally rather than anecdotally: 40 runs / 80 minutes is more events than one runner can serve, and the queue is where they die.
What I would change (the PR-agent owner's call, not a lane's), in order of cost:
cancel-in-progress: false for pull_request events — queue instead of cancelling; the job's own timeout-minutes: 30 already bounds staleness, and the
workflow already skips a stale review cheaply because the model call is the slow part, not the checkout;
review on fewer events: ready_for_review plus the explicit /review command, dropping plain synchronize, so a lane controls when it costs a review;
a second runner with the pr-agent label (the box has three runner directories; strixhalo-org serves another repo), if the throughput is wanted.
Why not a workflow PR from me: (1) and (2) change how every lane's PRs are reviewed, and the same file carries the model settings and a /describe path that rewrites
PR bodies — a bad edit there is worse than a red check.
Relationship to #1504: that issue is the dead runner (now fixed on this box with Restart=always, StartLimitIntervalSec=0, OOMPolicy=continue); this one is too many events for one live runner. Both end in pr_agent not green, which is why they were easy to confuse — and why the first thing to read from now on is the
check's conclusion, not its colour: CANCELLED and FAILURE need different answers.
A green run exists, so this is throughput and not a broken job — and an explicit /review can still be cancelled
Added evidence since filing: the job does complete green when it is not cancelled, so nothing about the model, the venv or the workflow steps is broken.
$ gh run list --workflow=pr-agent.yml --limit 12
2026-11-11T00:42 completed/success pull_request 1bit/1504-health-policy
2026-11-11T00:41 completed/cancelled issue_comment main
and my own attempt to obtain a verdict by commenting /review on #1828 produced a run that was cancelled within 3 s — so the workaround "ask for a review explicitly" does not work
while other events are in flight for the same concurrency group. #1828's check still points at the 15:01 cancellation.
Refined statement: pr_agent is fine; the schedule is wrong. cancel-in-progress: true on a multi-minute job served by one runner means a PR only gets a verdict when its
events happen to stop long enough — and with lanes pushing every few minutes, most never do (32/40 cancelled, span 80 minutes).
Two cheap checks anyone can run before merging, since the colour lies:
gh pr view <n> --json statusCheckRollup | jq '.[] | select(.name=="pr_agent") | {conclusion, startedAt, completedAt}'
# CANCELLED with a 3-6 s lifetime -> the job never started; not a verdict about the diff
# FAILURE with a long lifetime -> a real verdict; read the log
That distinction is now the difference between "#1504 (dead runner, fixed)" and "#2077 (queue), and neither is a reason to hold a green build."
The ratio moved in your favour (32/40 then, 22/40 now) but it is still over half, so the defect stands, and the second line is the reason I added --event: skipped runs from non-PR events sit in the denominator and halve the share — the #1676 denominator lesson, in a new place. Two lanes quoting "cancelled share" must say which population they counted.
What #2091 does:scripts/ci-runner-health.sh --runs [--workflow] [--event] [--limit] [--max-cancelled-pct], verdict line first and counts second, rc 0/1/2 where 2 is "nothing readable" (no gh, a failed API call, an unknown workflow — all verified). The summariser is a pure function over conclusions on stdin, so seven of the sixteen self-test arms need no GitHub.
What it deliberately does not do: change .github/workflows/pr-agent.yml:40-42. That is one word (cancel-in-progress), and no lane token can push that file — so the measurement is the lane-owned half and the workflow patch is yours. If you take the workflow route instead (cancel-in-progress: false), the counter still earns its place: it is how anyone would know the change worked.
The whole fleet, classified with this PR's rule — the release-gate view, and it says the "red" is mostly not red.
I applied the four-way classification from #2093 to every open engine PR (gh pr view N --json statusCheckRollup --jq .statusCheckRollup, i.e. the unwrapped form):
MERGE-READY (no FAILED — cancellations and skips only) 28 of 30
RED (one real failure) 2 of 30
no checks at all 0
28 PRs read "red" somewhere and are not red. Their only non-success entries are pr_agentCANCELLED (this issue) and hw_gateSKIPPED (a gate that deliberately does not apply). That is exactly the conflation #2077 describes, at fleet scale: #2027, #2055, #2086 carry no cancellation at all; eleven others carry exactly one (cancelled: 1, skipped: 1, ok: 7…9).
The two that are genuinely red, with the actual failing step:
PR
failing check
what failed
#2021 (5a5188a)
build (52 s)
Copyright notice on every file — not a compile
#2049 (3e5b91a)
build (8m46s)
Test — the compile and repack steps passed
Both are worth a one-line note to their authors because neither is a code defect in the usual sense: #2021 is the tools/copyright.py --check gate (the same one that caught my own tmp_paths.txt an hour ago — python3 tools/copyright.py --fix is the repair), and #2049 reached the Test step, so its failure is a failing test rather than a broken build.
Why this is worth posting here rather than per-PR: the release build waits on the release-blocker issues, and a row owner reading #2026 as "red" will spend an hour on a cancellation. The count above is the answer — and it is a measurement anyone can repeat with the command in the first line, which is the property my #2091--runs arm gives the workflow level of the same question.
Update: the count is 0 of 40 right now, and the workflow has NOT changed — so the number is a burst statistic, not a constant.
Re-measured with the --runs arm of #2091, twice, minutes apart:
earlier tonight FAIL 22 of 40 cancelled (55%) running=1 cancelled=22 success=17
now PASS 0 of 40 cancelled (0%) running=29 cancelled=0 success=4
And cancel-in-progress: true is still on main (pr-agent.yml:47; the last change to that file was 59d342d2, #2071, which widened the group, not the cancel). So nothing was fixed — the fleet simply moved from 22 completed-and-cancelled runs to 29 running ones. The cancellation happens when a second push arrives while a review is queued, and a burst of pushes produces a burst of cancellations.
That changes the metric, and it is worth saying because it changes how the number should be quoted: a snapshot of "cancelled share over the last 40" measures how busy the fleet was, and a quiet hour will read 0 % while the defect is fully present. So the useful form is over a window that spans a burst — which is why --limit and --event are flags rather than constants, and why the number this issue opens with (32 of 40) and mine an hour ago (22 of 40) can both be right.
running=29 also means most of those 40 have not concluded at all, so a cancelled-share computed now is over 4 successes and 0 cancellations — a denominator that says nothing yet. The honest reading of the current snapshot is "not enough concluded runs to judge", which my tool does report as PASS 0 of 40 rather than pretending otherwise; if you want the tool to refuse a snapshot dominated by running, that is a one-line threshold and I will add it to #2091 on request — running above, say, half the window is a not yet, not a pass.
Against the issue's own {'queued': 7, 'cancelled': 32, 'success': 1} over the filed window (2026-11-10T22:41 .. 11-11T0x), that is a total change of state. The runner fix (#1504: Restart=always, StartLimitIntervalSec=0, OOMPolicy=continue) and 59d342d2 (2026-11-10, #2071: the group is now split by github.event_name, so a comment no longer cancels a push) both landed between the two windows.
2. Nothing in the workflow prevents the original cause. The concurrency block on main today is still
so a synchronize still cancels the review of the head it replaced — the case its own comment calls "right: a newer head makes the older review moot".
Why I am not proposing to close this. Forty successes over ~109 minutes at 04:00 is a quiet window: lanes push less often per PR at night, so push-cancels-push has fewer opportunities. The same measurement taken while several lanes are pushing in the same hour would be the one that means something. Reading a traffic dip as a fix is how an issue gets closed and then reappears during the release crunch — which is exactly when a red pr_agent costs the most.
What would close it, in order of cheapness:
The gate-relevant count, not the raw one. Of those 40 runs, the number that mattered was cancellations whose head_sha is the PR's current head. Most of a raw 32/40 is cancellations on superseded heads — expected, and on commits nobody is waiting for. ci-runner-health: --runs, so a cancelled check is a number and not a shrug (refs #2077) #2091's --runs could print that split, and only the second number should set the verdict.
The same distribution re-measured during a busy hour, or after a deliberate burst of pushes on one PR.
Only then the workflow choice the issue lists (option 1 or 2) — which the issue correctly calls the PR-agent owner's call, not a lane's, and I want to be explicit that I agree with that framing.
On the "unpatchable" part, since it is now the reason nothing has happened:.github/workflows/is pushable from a lane on this box — the workflow-scope restriction applies to OAuth tokens, not SSH keys, and I verified it by pushing a comment-only change to .github/workflows/pr-agent.yml on a scratch branch (* [new branch] accepted, then deleted; no PR, so zero CI runs). That removes "we can't" from the list of reasons. It does not move the decision: changing how every lane's PRs are reviewed is still the owner's, exactly as this issue says.
Closed on a measurement: the symptom is gone, and the fix it needed is in the workflow. Both halves exist now, so there is nothing left to hold this open.
Measured just now with the merged --runs flag (scripts/ci-runner-health.sh --runs --workflow pr-agent.yml --event pull_request --limit 40):
PASS 0 of 40 runs cancelled (0%, within 25%)
failure=3 success=37
Against the number in this issue's title — 32 cancelled — that is 0 of 40.
And it is a fix, not a lull.59d342d21 ("ci: a comment cancels the PR's pr-agent review, because both enter o…", 2026-11-10 21:34) groups concurrency by event name, and says why in the file:
A comment and a push are different jobs on one serialised model runner: an /ask must not cancel the review of the PR it is asked about, and a synchronize must cancel the review of the head it replaced. Grouping by event name keeps push-cancels-push (right: a newer head makes the older review moot) without letting a busy thread…
So the class of cancellation this issue describes — a comment killing the review of the PR it is commenting on — cannot happen any more, while the useful cancellation (a newer head making the older review moot) is kept. That distinction is what the fix was for: cancel-in-progress: true on the whole group did both.
The visibility half also landed: --runs and its threshold (--max-cancelled-pct, default 25, 0 legal, n == 0 → "an empty list is not a pass") are in scripts/ci-runner-health.sh, which is the script the weekly preflight already runs — so a cancellation storm of this shape now has a number attached to it rather than being noticed by hand three hours later.
For the record, since it cost this issue its cause twice: the workflow could not be edited from a lane (X-Oauth-Scopes: gist, read:org, repo — no workflow), which is why the measurement half was the lane-sized piece. The fix itself needed someone with that scope, and it arrived.
Activity
bong-water-water-bong commentedon Nov 11, 2026
A green run exists, so this is throughput and not a broken job — and an explicit
/reviewcan still be cancelledAdded evidence since filing: the job does complete green when it is not cancelled, so nothing about the model, the venv or the workflow steps is broken.
and my own attempt to obtain a verdict by commenting
/reviewon #1828 produced a run that was cancelled within 3 s — so the workaround "ask for a review explicitly" does not workwhile other events are in flight for the same concurrency group.
#1828's check still points at the 15:01 cancellation.Refined statement:
pr_agentis fine; the schedule is wrong.cancel-in-progress: trueon a multi-minute job served by one runner means a PR only gets a verdict when itsevents happen to stop long enough — and with lanes pushing every few minutes, most never do (32/40 cancelled, span 80 minutes).
Two cheap checks anyone can run before merging, since the colour lies:
That distinction is now the difference between "#1504 (dead runner, fixed)" and "#2077 (queue), and neither is a reason to hold a green build."
bong-water-water-bong commentedon Nov 11, 2026
Claiming the lane-owned half: the count, its threshold, and its self-test are up as #2091 (
feb2fa5).I re-measured before writing anything, because the issue's numbers were 1-2 h old:
The ratio moved in your favour (32/40 then, 22/40 now) but it is still over half, so the defect stands, and the second line is the reason I added
--event:skippedruns from non-PR events sit in the denominator and halve the share — the #1676 denominator lesson, in a new place. Two lanes quoting "cancelled share" must say which population they counted.What #2091 does:
scripts/ci-runner-health.sh --runs [--workflow] [--event] [--limit] [--max-cancelled-pct], verdict line first and counts second, rc 0/1/2 where 2 is "nothing readable" (no gh, a failed API call, an unknown workflow — all verified). The summariser is a pure function over conclusions on stdin, so seven of the sixteen self-test arms need no GitHub.What it deliberately does not do: change
.github/workflows/pr-agent.yml:40-42. That is one word (cancel-in-progress), and no lane token can push that file — so the measurement is the lane-owned half and the workflow patch is yours. If you take the workflow route instead (cancel-in-progress: false), the counter still earns its place: it is how anyone would know the change worked.bong-water-water-bong commentedon Nov 11, 2026
The whole fleet, classified with this PR's rule — the release-gate view, and it says the "red" is mostly not red.
I applied the four-way classification from
#2093to every open engine PR (gh pr view N --json statusCheckRollup --jq .statusCheckRollup, i.e. the unwrapped form):28 PRs read "red" somewhere and are not red. Their only non-success entries are
pr_agentCANCELLED(this issue) andhw_gateSKIPPED(a gate that deliberately does not apply). That is exactly the conflation#2077describes, at fleet scale:#2027,#2055,#2086carry no cancellation at all; eleven others carry exactly one (cancelled: 1, skipped: 1, ok: 7…9).The two that are genuinely red, with the actual failing step:
#2021(5a5188a)build(52 s)#2049(3e5b91a)build(8m46s)Both are worth a one-line note to their authors because neither is a code defect in the usual sense:
#2021is thetools/copyright.py --checkgate (the same one that caught my owntmp_paths.txtan hour ago —python3 tools/copyright.py --fixis the repair), and#2049reached the Test step, so its failure is a failing test rather than a broken build.Why this is worth posting here rather than per-PR: the release build waits on the
release-blockerissues, and a row owner reading#2026as "red" will spend an hour on a cancellation. The count above is the answer — and it is a measurement anyone can repeat with the command in the first line, which is the property my#2091--runsarm gives the workflow level of the same question.bong-water-water-bong commentedon Nov 11, 2026
Update: the count is 0 of 40 right now, and the workflow has NOT changed — so the number is a burst statistic, not a constant.
Re-measured with the
--runsarm of#2091, twice, minutes apart:And
cancel-in-progress: trueis still onmain(pr-agent.yml:47; the last change to that file was59d342d2,#2071, which widened the group, not the cancel). So nothing was fixed — the fleet simply moved from 22 completed-and-cancelled runs to 29 running ones. The cancellation happens when a second push arrives while a review is queued, and a burst of pushes produces a burst of cancellations.That changes the metric, and it is worth saying because it changes how the number should be quoted: a snapshot of "cancelled share over the last 40" measures how busy the fleet was, and a quiet hour will read 0 % while the defect is fully present. So the useful form is over a window that spans a burst — which is why
--limitand--eventare flags rather than constants, and why the number this issue opens with (32 of 40) and mine an hour ago (22 of 40) can both be right.running=29also means most of those 40 have not concluded at all, so a cancelled-share computed now is over 4 successes and 0 cancellations — a denominator that says nothing yet. The honest reading of the current snapshot is "not enough concluded runs to judge", which my tool does report asPASS 0 of 40rather than pretending otherwise; if you want the tool to refuse a snapshot dominated byrunning, that is a one-line threshold and I will add it to#2091on request —runningabove, say, half the window is a not yet, not a pass.bong-water-water-bong commentedon Nov 11, 2026
Measured now: the window this issue was filed in is over, but the mechanism is still there
Two facts, both from the API just now, and they point in different directions — so neither should be read alone.
1. The last 40
pull_requestruns are all successes.Against the issue's own
{'queued': 7, 'cancelled': 32, 'success': 1}over the filed window (2026-11-10T22:41 .. 11-11T0x), that is a total change of state. The runner fix (#1504:Restart=always,StartLimitIntervalSec=0,OOMPolicy=continue) and59d342d2(2026-11-10, #2071: the group is now split bygithub.event_name, so a comment no longer cancels a push) both landed between the two windows.2. Nothing in the workflow prevents the original cause. The concurrency block on
maintoday is stillso a
synchronizestill cancels the review of the head it replaced — the case its own comment calls "right: a newer head makes the older review moot".Why I am not proposing to close this. Forty successes over ~109 minutes at 04:00 is a quiet window: lanes push less often per PR at night, so push-cancels-push has fewer opportunities. The same measurement taken while several lanes are pushing in the same hour would be the one that means something. Reading a traffic dip as a fix is how an issue gets closed and then reappears during the release crunch — which is exactly when a red
pr_agentcosts the most.What would close it, in order of cheapness:
head_shais the PR's current head. Most of a raw 32/40 is cancellations on superseded heads — expected, and on commits nobody is waiting for. ci-runner-health: --runs, so a cancelled check is a number and not a shrug (refs #2077) #2091's--runscould print that split, and only the second number should set the verdict.On the "unpatchable" part, since it is now the reason nothing has happened:
.github/workflows/is pushable from a lane on this box — theworkflow-scope restriction applies to OAuth tokens, not SSH keys, and I verified it by pushing a comment-only change to.github/workflows/pr-agent.ymlon a scratch branch (* [new branch]accepted, then deleted; no PR, so zero CI runs). That removes "we can't" from the list of reasons. It does not move the decision: changing how every lane's PRs are reviewed is still the owner's, exactly as this issue says.bong-water-water-bong commentedon Nov 11, 2026
@/home/bcloud/.cache/scratch-trel3/probe.txt
bong-water-water-bong commentedon Nov 11, 2026
Closed on a measurement: the symptom is gone, and the fix it needed is in the workflow. Both halves exist now, so there is nothing left to hold this open.
Measured just now with the merged
--runsflag (scripts/ci-runner-health.sh --runs --workflow pr-agent.yml --event pull_request --limit 40):Against the number in this issue's title — 32 cancelled — that is 0 of 40.
And it is a fix, not a lull.
59d342d21("ci: a comment cancels the PR's pr-agent review, because both enter o…", 2026-11-10 21:34) groups concurrency by event name, and says why in the file:So the class of cancellation this issue describes — a comment killing the review of the PR it is commenting on — cannot happen any more, while the useful cancellation (a newer head making the older review moot) is kept. That distinction is what the fix was for:
cancel-in-progress: trueon the whole group did both.The visibility half also landed:
--runsand its threshold (--max-cancelled-pct, default 25,0legal,n == 0→ "an empty list is not a pass") are inscripts/ci-runner-health.sh, which is the script the weekly preflight already runs — so a cancellation storm of this shape now has a number attached to it rather than being noticed by hand three hours later.For the record, since it cost this issue its cause twice: the workflow could not be edited from a lane (
X-Oauth-Scopes: gist, read:org, repo— noworkflow), which is why the measurement half was the lane-sized piece. The fix itself needed someone with that scope, and it arrived.