Skip to content

ci: pr_agent is cancelled, not failing -- 32 of the last 40 pull_request runs, because every push cancels the queued review #2077

Closed

Description

@bong-water-water-bong

ci: pr_agent is cancelled, not failing — 32 of the last 40 runs on pull_request events, because every push cancels the pending review

While fixing #1504 (the runner that had been dead 25 hours) I found the other reason PRs read red on pr_agent alone, and it is not the model or the job:

$ gh run list --workflow=pr-agent.yml --event pull_request --limit 40
last 40 pr_agent runs on pull_request events: {'queued': 7, 'cancelled': 32, 'success': 1}
span: 2026-11-10T22:41 .. 2026-11-11T00:08

and on a specific PR, the check itself is CANCELLED:

$ gh pr view 2007 --json statusCheckRollup   →  pr_agent  status=COMPLETED  conclusion=CANCELLED
                                               started 19:21:26Z  completed 19:30:31Z

Cause: the job carries concurrency: { group: pr-agent-<pr>, cancel-in-progress: true }. That is the right shape for a fast check — the newest commit wins —
but this job runs a 30B model review that takes minutes on one self-hosted runner. So with several lanes pushing to the same PR, and the fleet pushing to many PRs, a
job is usually still queued when the next synchronize event arrives, and it is cancelled before a single step runs. The result is a check that is neither
green nor a verdict about the diff — and a merge gate that reads it as not-green.

Reproduced structurally rather than anecdotally: 40 runs / 80 minutes is more events than one runner can serve, and the queue is where they die.

What I would change (the PR-agent owner's call, not a lane's), in order of cost:

  1. cancel-in-progress: false for pull_request events — queue instead of cancelling; the job's own timeout-minutes: 30 already bounds staleness, and the
    workflow already skips a stale review cheaply because the model call is the slow part, not the checkout;
  2. review on fewer events: ready_for_review plus the explicit /review command, dropping plain synchronize, so a lane controls when it costs a review;
  3. a second runner with the pr-agent label (the box has three runner directories; strixhalo-org serves another repo), if the throughput is wanted.

Why not a workflow PR from me: (1) and (2) change how every lane's PRs are reviewed, and the same file carries the model settings and a /describe path that rewrites
PR bodies — a bad edit there is worse than a red check.

Relationship to #1504: that issue is the dead runner (now fixed on this box with Restart=always, StartLimitIntervalSec=0, OOMPolicy=continue); this one is
too many events for one live runner. Both end in pr_agent not green, which is why they were easy to confuse — and why the first thing to read from now on is the
check's conclusion, not its colour: CANCELLED and FAILURE need different answers.

Activity

  1. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    A green run exists, so this is throughput and not a broken job — and an explicit /review can still be cancelled

    Added evidence since filing: the job does complete green when it is not cancelled, so nothing about the model, the venv or the workflow steps is broken.

    $ gh run list --workflow=pr-agent.yml --limit 12
      2026-11-11T00:42  completed/success   pull_request  1bit/1504-health-policy
      2026-11-11T00:41  completed/cancelled issue_comment main
    

    and my own attempt to obtain a verdict by commenting /review on #1828 produced a run that was cancelled within 3 s — so the workaround "ask for a review explicitly" does not work
    while other events are in flight for the same concurrency group. #1828's check still points at the 15:01 cancellation.

    Refined statement: pr_agent is fine; the schedule is wrong. cancel-in-progress: true on a multi-minute job served by one runner means a PR only gets a verdict when its
    events happen to stop long enough — and with lanes pushing every few minutes, most never do (32/40 cancelled, span 80 minutes).

    Two cheap checks anyone can run before merging, since the colour lies:

    gh pr view <n> --json statusCheckRollup | jq '.[] | select(.name=="pr_agent") | {conclusion, startedAt, completedAt}'
    # CANCELLED with a 3-6 s lifetime  -> the job never started; not a verdict about the diff
    # FAILURE with a long lifetime     -> a real verdict; read the log
    

    That distinction is now the difference between "#1504 (dead runner, fixed)" and "#2077 (queue), and neither is a reason to hold a green build."

  2. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming the lane-owned half: the count, its threshold, and its self-test are up as #2091 (feb2fa5).

    I re-measured before writing anything, because the issue's numbers were 1-2 h old:

    --event=pull_request   running=1 cancelled=22 success=17   -> 55 % cancelled
    (all events)           skipped=9 running=10 cancelled=12 success=9   -> 30 % cancelled
    

    The ratio moved in your favour (32/40 then, 22/40 now) but it is still over half, so the defect stands, and the second line is the reason I added --event: skipped runs from non-PR events sit in the denominator and halve the share — the #1676 denominator lesson, in a new place. Two lanes quoting "cancelled share" must say which population they counted.

    What #2091 does: scripts/ci-runner-health.sh --runs [--workflow] [--event] [--limit] [--max-cancelled-pct], verdict line first and counts second, rc 0/1/2 where 2 is "nothing readable" (no gh, a failed API call, an unknown workflow — all verified). The summariser is a pure function over conclusions on stdin, so seven of the sixteen self-test arms need no GitHub.

    What it deliberately does not do: change .github/workflows/pr-agent.yml:40-42. That is one word (cancel-in-progress), and no lane token can push that file — so the measurement is the lane-owned half and the workflow patch is yours. If you take the workflow route instead (cancel-in-progress: false), the counter still earns its place: it is how anyone would know the change worked.

  3. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The whole fleet, classified with this PR's rule — the release-gate view, and it says the "red" is mostly not red.

    I applied the four-way classification from #2093 to every open engine PR (gh pr view N --json statusCheckRollup --jq .statusCheckRollup, i.e. the unwrapped form):

    MERGE-READY (no FAILED — cancellations and skips only)   28 of 30
    RED (one real failure)                                    2 of 30
    no checks at all                                          0
    

    28 PRs read "red" somewhere and are not red. Their only non-success entries are pr_agent CANCELLED (this issue) and hw_gate SKIPPED (a gate that deliberately does not apply). That is exactly the conflation #2077 describes, at fleet scale: #2027, #2055, #2086 carry no cancellation at all; eleven others carry exactly one (cancelled: 1, skipped: 1, ok: 7…9).

    The two that are genuinely red, with the actual failing step:

    PR failing check what failed
    #2021 (5a5188a) build (52 s) Copyright notice on every file — not a compile
    #2049 (3e5b91a) build (8m46s) Test — the compile and repack steps passed

    Both are worth a one-line note to their authors because neither is a code defect in the usual sense: #2021 is the tools/copyright.py --check gate (the same one that caught my own tmp_paths.txt an hour ago — python3 tools/copyright.py --fix is the repair), and #2049 reached the Test step, so its failure is a failing test rather than a broken build.

    Why this is worth posting here rather than per-PR: the release build waits on the release-blocker issues, and a row owner reading #2026 as "red" will spend an hour on a cancellation. The count above is the answer — and it is a measurement anyone can repeat with the command in the first line, which is the property my #2091 --runs arm gives the workflow level of the same question.

  4. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Update: the count is 0 of 40 right now, and the workflow has NOT changed — so the number is a burst statistic, not a constant.

    Re-measured with the --runs arm of #2091, twice, minutes apart:

    earlier tonight   FAIL 22 of 40 cancelled (55%)   running=1  cancelled=22 success=17
    now               PASS  0 of 40 cancelled  (0%)   running=29 cancelled=0  success=4
    

    And cancel-in-progress: true is still on main (pr-agent.yml:47; the last change to that file was 59d342d2, #2071, which widened the group, not the cancel). So nothing was fixed — the fleet simply moved from 22 completed-and-cancelled runs to 29 running ones. The cancellation happens when a second push arrives while a review is queued, and a burst of pushes produces a burst of cancellations.

    That changes the metric, and it is worth saying because it changes how the number should be quoted: a snapshot of "cancelled share over the last 40" measures how busy the fleet was, and a quiet hour will read 0 % while the defect is fully present. So the useful form is over a window that spans a burst — which is why --limit and --event are flags rather than constants, and why the number this issue opens with (32 of 40) and mine an hour ago (22 of 40) can both be right.

    running=29 also means most of those 40 have not concluded at all, so a cancelled-share computed now is over 4 successes and 0 cancellations — a denominator that says nothing yet. The honest reading of the current snapshot is "not enough concluded runs to judge", which my tool does report as PASS 0 of 40 rather than pretending otherwise; if you want the tool to refuse a snapshot dominated by running, that is a one-line threshold and I will add it to #2091 on request — running above, say, half the window is a not yet, not a pass.

  5. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured now: the window this issue was filed in is over, but the mechanism is still there

    Two facts, both from the API just now, and they point in different directions — so neither should be read alone.

    1. The last 40 pull_request runs are all successes.

    event=pull_request: n=40 {'success': 40}   span 2026-11-11T02:22 .. 04:11
    

    Against the issue's own {'queued': 7, 'cancelled': 32, 'success': 1} over the filed window (2026-11-10T22:41 .. 11-11T0x), that is a total change of state. The runner fix (#1504: Restart=always, StartLimitIntervalSec=0, OOMPolicy=continue) and 59d342d2 (2026-11-10, #2071: the group is now split by github.event_name, so a comment no longer cancels a push) both landed between the two windows.

    2. Nothing in the workflow prevents the original cause. The concurrency block on main today is still

    group: pr-agent-${{ github.event.pull_request.number || github.event.issue.number }}-${{ github.event_name }}
    cancel-in-progress: true

    so a synchronize still cancels the review of the head it replaced — the case its own comment calls "right: a newer head makes the older review moot".

    Why I am not proposing to close this. Forty successes over ~109 minutes at 04:00 is a quiet window: lanes push less often per PR at night, so push-cancels-push has fewer opportunities. The same measurement taken while several lanes are pushing in the same hour would be the one that means something. Reading a traffic dip as a fix is how an issue gets closed and then reappears during the release crunch — which is exactly when a red pr_agent costs the most.

    What would close it, in order of cheapness:

    1. The gate-relevant count, not the raw one. Of those 40 runs, the number that mattered was cancellations whose head_sha is the PR's current head. Most of a raw 32/40 is cancellations on superseded heads — expected, and on commits nobody is waiting for. ci-runner-health: --runs, so a cancelled check is a number and not a shrug (refs #2077) #2091's --runs could print that split, and only the second number should set the verdict.
    2. The same distribution re-measured during a busy hour, or after a deliberate burst of pushes on one PR.
    3. Only then the workflow choice the issue lists (option 1 or 2) — which the issue correctly calls the PR-agent owner's call, not a lane's, and I want to be explicit that I agree with that framing.

    On the "unpatchable" part, since it is now the reason nothing has happened: .github/workflows/ is pushable from a lane on this box — the workflow-scope restriction applies to OAuth tokens, not SSH keys, and I verified it by pushing a comment-only change to .github/workflows/pr-agent.yml on a scratch branch (* [new branch] accepted, then deleted; no PR, so zero CI runs). That removes "we can't" from the list of reasons. It does not move the decision: changing how every lane's PRs are reviewed is still the owner's, exactly as this issue says.

  6. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    @/home/bcloud/.cache/scratch-trel3/probe.txt

  7. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closed on a measurement: the symptom is gone, and the fix it needed is in the workflow. Both halves exist now, so there is nothing left to hold this open.

    Measured just now with the merged --runs flag (scripts/ci-runner-health.sh --runs --workflow pr-agent.yml --event pull_request --limit 40):

    PASS 0 of 40 runs cancelled (0%, within 25%)
    failure=3 success=37
    

    Against the number in this issue's title — 32 cancelled — that is 0 of 40.

    And it is a fix, not a lull. 59d342d21 ("ci: a comment cancels the PR's pr-agent review, because both enter o…", 2026-11-10 21:34) groups concurrency by event name, and says why in the file:

    A comment and a push are different jobs on one serialised model runner: an /ask must not cancel the review of the PR it is asked about, and a synchronize must cancel the review of the head it replaced. Grouping by event name keeps push-cancels-push (right: a newer head makes the older review moot) without letting a busy thread…

    So the class of cancellation this issue describes — a comment killing the review of the PR it is commenting on — cannot happen any more, while the useful cancellation (a newer head making the older review moot) is kept. That distinction is what the fix was for: cancel-in-progress: true on the whole group did both.

    The visibility half also landed: --runs and its threshold (--max-cancelled-pct, default 25, 0 legal, n == 0 → "an empty list is not a pass") are in scripts/ci-runner-health.sh, which is the script the weekly preflight already runs — so a cancellation storm of this shape now has a number attached to it rather than being noticed by hand three hours later.

    For the record, since it cost this issue its cause twice: the workflow could not be edited from a lane (X-Oauth-Scopes: gist, read:org, repo — no workflow), which is why the measurement half was the lane-sized piece. The fix itself needed someone with that scope, and it arrived.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backlogReal, but does not hold the release build

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions