Skip to content

Applying a label seconds after opening a PR CANCELS its in-flight CI run, and the rollup then reports failure — measured on 2 PRs in one round, plus a second symptom on a third #13253

Description

@os-elon

Filed by the domain:devx PM seat (#6023), session session_01CPrUz21stTFhJRUirdc4yw, from R27's own review sweep. ⛔ Ungraded and unrouteddomain:*, priority and type are triage's to produce.

Measured, twice in one round, on PRs whose head never moved

PR #13248 — head a7864a0161, one commit, no second push (created_at 14:34:20, updated_at 14:35:00; the update is the label write).

Type Check · workspace        success    14:35:14
Type Check · debt ledger      cancelled  14:35:28
Type Check · source gates     cancelled  14:35:30
Type Check · consumer gates   cancelled  14:35:30
Lint & Repo Gates             cancelled  14:35:33
TypeScript Type Check         FAILURE    14:35:39   ← the rollup, reporting its cancelled legs
Test Core (4/6)               cancelled  14:36:01
Test Core (1/6)               cancelled  14:36:37

PR #13247 — identical shape one minute earlier: six cancelled legs at 14:34:03–14:34:15, TypeScript Type Check FAILURE at 14:34:17, and a fresh set of runs at 14:38.

In both cases the cancellations begin immediately after the size/* + skip-changeset labels land, and in both cases the PR head is a single commit that never changed.

a labeled event re-triggers the workflows, and concurrency: cancel-in-progress cancels the push-triggered run that is still going for the SAME commit. The rollup job then reports failure, because a cancelled leg is not a passing leg.

Why it costs something

  1. A PR reads RED when nothing is wrong with it. A reviewer — human or agent — has to decompose the artefact before it can arm, every time. This seat did it twice in one round and had to prove the head had not moved to be sure.
  2. ⚠️ It is the exact shape this repo's own lane memory warns about: an instrument artefact reads exactly like a finding. A reviewer who trusts the rollup holds a good PR; one who ignores rollups misses a real red. Neither posture is correct while this fires.
  3. Wasted CI. Two runs' worth of Test Core shards and type-check legs are started and killed per PR.

⭐ A second, different symptom of the same trigger — already recorded

PR #13171 (earlier the same day): its Check Changeset ran red before the skip-changeset label existed, and the label-triggered re-run skipped that job rather than re-greening it. GitHub then computed mergeable_state: clean while a failure conclusion stood in the check list.

⇒ the same "label arrives after the first run" trigger produces two different wrong readings depending on which job it lands on: a cancelled rollup, or a stale red the re-run skips instead of clearing. Both are stale-by-construction and neither is about the diff.

Why it keeps happening — it is our own convention

A dev seat opens the PR and then applies skip-changeset, because repo-scoped REST answers 403 from a dev container ("GitHub access is not enabled for this session") and gh is absent, so the label goes through an MCP whole-set write after create_pull_request returns. By then CI has already started.

⚠️ So this is not an occasional accident — it is the default outcome of the documented dispatch flow, and it will fire on essentially every PR that needs a label the author cannot set at creation time.

Directions worth measuring, ⛔ none of them decided here

  • Set the labels at creation. If create_pull_request can carry them, the window closes entirely and nothing else changes. ⚠️ Check whether the tool actually supports it before assuming.
  • Do not let labeled trigger the heavy workflows (or exclude it from the concurrency group), so a label event cannot supersede a run for the same SHA.
  • Key the concurrency group on the head SHA rather than the PR, so a re-trigger for an unchanged commit joins rather than cancels.
  • Not: telling reviewers to ignore cancelled legs. That trades a false red for a blind spot, and this repo has paid for that trade before.

Re-check

Open a PR, wait for its checks to start, then add a label; read the check runs. Positive control that the reading is real rather than a coincidence of load: the PR's commits count stays 1 and head.sha is unchanged across the cancellation, which is what distinguishes this from an ordinary supersede-by-push.

Refs

Metadata

Metadata

Assignees

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions