-
Notifications
You must be signed in to change notification settings - Fork 10
808 lines (774 loc) · 47.4 KB
/
Copy pathmerge-queue-triage.yml
File metadata and controls
808 lines (774 loc) · 47.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
name: Merge Queue Triage
# Why this exists (#4859): red queue builds were being blind-requeued. On
# 2026-08-03 one PR failed the queue four times and landed unchanged on the
# fifth attempt (09:19 → 10:22), and every failure evicted and rebuilt every
# entry queued behind it — the queue's perceived slowness that morning was
# mostly this amplification, not build duration.
#
# A queue failure is a different animal from a PR failure: the PR's own CI ran
# affected-only, while the queue runs the FULL suite on the speculative merge
# result. The failing test is therefore often in a package the PR never
# touched — a flake, or a semantic conflict with another queued PR — and
# neither of those is fixed by re-queueing; re-queueing just burns another
# ~10-minute build for every entry behind it.
#
# So: every red merge_group CI run gets a triage comment on its PR — the
# failed jobs/steps, the failing test lines pulled from the logs (best
# effort) WITH the deciding failure-reason line beside each one, how many
# times THIS PR has already failed in the queue in the last 24 h, the
# queue-wide failure count, and which OTHER PRs the same failing test file has
# ejected in the last 24 h. The checklist tells the author (human or agent) to
# diagnose before re-queueing. The comment is the machine-readable signal the
# PM dispatch loop can key on.
#
# ## The reason line, and why it is not decoration (#10128, limb ①)
#
# Until 2026-08-20 the excerpt carried the `FAIL <file> > <suite> > <test>`
# line and NOT the line under it that says WHY. Those two failure modes look
# identical in the excerpt and are diagnosed in opposite directions:
#
# FAIL src/dev-plugin-security-enforcement-warning.test.ts > … > bail #1 …
# Error: Test timed out in 5000ms. <- load/timing defect
#
# FAIL src/dev-plugin-security-enforcement-warning.test.ts > … > bail #1 …
# AssertionError: SecurityPlugin.init() ran: expected false to be true
# <- a real regression
#
# The FAIL lines above are byte-identical; only the second line differs. On
# 2026-08-20 a first responder reproduced the ASSERTION form of that very file
# locally, matched it against an excerpt that was actually reporting the
# TIMEOUT form, and filed a confidently wrong diagnosis (#10112, "missing build
# edge"); a whole dispatch was then spent falsifying it, and the real fix
# (#10120, via #10115) was a module-load cost that no assertion was ever
# involved in. One grep in this file would have separated them, so it is here.
#
# ## The reason line's named exception: what the assertion MEASURES (#13830)
#
# The rule above is right about the common case and is doing real work: the
# same morning the exception below was measured, a seat used the rule to
# REFUSE a flake reading on an `AssertionError` (#13683), correctly, and that
# refusal is why that card exists. But "assertion ⇒ real behaviour change" has
# a counterexample class, and it is the expensive direction. #13691 ejected
# the queue with:
#
# AssertionError: this run took over a second, so second-precision stamps
# could have differed too: expected 1006 to be less than 1000
#
# That assertion measures nothing about the product. It is the test's own
# VALIDITY PRECONDITION — the run has to fit inside one second so that a
# truncated second-precision stamp COULD NOT have distinguished the two
# updates, which is what gives the distinctness assertion below it any
# meaning. The invariant actually under test (monotonicity, asserted a line
# above) HELD. The experiment was invalidated; the product was not falsified —
# confirmed by evidence, not by taste: same head, same test, re-queued once
# with no change → success.
#
# So the discriminator is not `AssertionError` vs timeout. It is what the
# assertion MEASURES. Product behaviour (a value, a shape, a refusal) reads
# the way the rule says. A precondition of the experiment's OWN validity
# (elapsed time, ordering under load, anything that only holds inside a time
# budget) reads WITH the timeouts, where one re-queue is a legitimate
# discriminator. The tell is cheap and mechanical: the assertion's message or
# its compared values name a duration, a timestamp, or a count of elapsed
# units.
#
# Both readings are expensive, in opposite directions, and the rule alone only
# protects one of them: a behaviour assertion read as a flake gets a real
# defect re-queued until it passes (that is the failure the rule prevents),
# and a validity-precondition assertion read as a behaviour change sends an
# author to debug a product that is not broken, on a red they cannot reproduce.
#
# ⛔ This NAMES an exception; it does not soften the rule. ⛔ And it changes
# only how a red is READ — which tests may be re-queued is governed elsewhere
# and is untouched here.
#
# ## Cross-PR aggregation, and its boundary (#10128, limb ②)
#
# The same test file ejected #10105, then #10003, then #10008 — three unrelated
# PRs — before a human joined the dots at 04:08Z. Each victim got its own
# comment and nothing connected them, so the first shared record was hand-filed
# and carried the wrong diagnosis. This job now keys each ejection by the
# FAILING TEST FILE PATH, records that sighting in its own comment as a
# machine-readable marker, and reads 24 h of those markers back on the next
# ejection. When one key has ejected >= 2 DISTINCT PRs inside the window it
# files ONE anchor issue for the key — or refreshes the open one, matched on a
# stable marker in the issue body, the same idempotency idiom as this file's
# own per-run comment marker — and links it from every later victim's comment.
#
# ⛔ The boundary is the point: this job NAMES and AGGREGATES, and DECIDES
# NOTHING. No auto-requeue, no auto-quarantine, no label on anybody's PR, no
# `skip`/`only` written anywhere. Weakening a gate stays a human act, and an
# anchor issue is a place for that conversation, not a verdict in it.
#
# Fires on conclusion == failure ONLY. 'cancelled' is the queue evicting an
# entry because something AHEAD of it failed (or a manual cancel) — it says
# nothing about this PR, so it gets no comment (the same run-lifecycle
# reasoning as dogfood-gate's cancelled handling in ci.yml, from the other
# side).
#
# workflow_run executes in the DEFAULT branch's context: this file must be on
# main before it fires, it never checks out or runs PR code, and it holds the
# minimum permissions (actions: read for logs, pull-requests: write for the
# comment, issues: write for the anchor issue limb ② files and refreshes).
#
# Delivery is retried, never assumed (#9424). A transient GitHub API failure used
# to kill this job outright — github-script hands any throw from the script to
# `main().catch(handleError)` → `core.setFailed` — and it took the diagnosis with
# it, because the comment body exists NOWHERE but the failed request. The step now
# retries the transient class (declared in its `retries:` inputs below) and, when
# delivery is refused anyway, writes the whole triage into the run's job summary
# before failing. That leaves one invariant worth keying on:
#
# this job is green ⇔ the triage comment is on the PR
#
# It is deliberately NOT green-on-undelivered, which is what the sibling fix in
# docs-drift-check.yml (#9373) chose. There the job's conclusion is a check on the
# PR and its comment is a courtesy, so a red costs a reader's attention for nothing.
# Here the job is `workflow_run`: its conclusion is on no check list, gates nothing
# and is in no required context, so a red costs nothing — while a green that
# quietly delivered nothing is the same silence this workflow exists to break, one
# layer further in.
#
# ⚠️ That invariant is about the COMMENT, and limb ② deliberately does not join
# it. The anchor issue is a convenience — a single place for one conversation —
# and every fact it carries is also printed inline in the comment, so an anchor
# that could not be written loses no diagnosis. It is therefore reported as an
# annotation AND as a line in the comment, and does not change this job's
# verdict. Failing the job on it would be claiming an equivalence that is false.
on:
workflow_run:
workflows: [CI]
types: [completed]
permissions: {}
jobs:
triage:
name: Comment queue-failure triage on the PR
if: >-
github.event.workflow_run.event == 'merge_group' &&
github.event.workflow_run.conclusion == 'failure'
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
actions: read
pull-requests: write
issues: write
steps:
- name: Post the triage comment
uses: actions/github-script@v9
with:
# The transient-retry policy, declared rather than hand-written (#9424):
# octokit's retry plugin re-issues any request whose status is NOT exempt
# below, plus network-level failures. Nothing here is swallowed — a spent
# retry still throws, so this widens no verdict, only the number of times
# the request is asked.
#
# 403 is REMOVED from the action's default exempt list
# (400,401,403,404,422) on purpose: GitHub answers a SECONDARY rate limit
# with 403 as well as with 429, and this job is rate-limit-shaped — it
# paginates the run's jobs, pulls up to four job logs, then paginates 24 h
# of merge_group runs. The price is that a genuine permission denial (a
# wrong `permissions:` block) now takes four attempts to fail instead of
# one. It still fails.
#
# 400/401/404/422 stay exempt: a malformed request, a wrong target, or a
# body past GitHub's 65536-character comment limit is this repo's own bug,
# answered on the first try and not improved by asking again.
retries: 3
retry-exempt-status-codes: 400,401,404,422
script: |
const run = context.payload.workflow_run;
const { owner, repo } = context.repo;
// Queue branches are named gh-readonly-queue/<base>/pr-<N>-<sha>, and
// BOTH captures are load-bearing. The trailing sha is not decoration:
// it is the commit the queue built this PR ON TOP OF, so it names the
// PR's position in GitHub's speculative stack. Reading it is what lets
// limb ② tell "another PR hit this too" from "another PR inherited this
// one's tree" — see `stackEdges` below. It was matched but discarded
// until 2026-08-29, which is why the victim counts read as flake
// evidence when they were measuring queue depth.
//
// The base-branch segment is `.+` rather than `[^/]+` because a base
// branch may itself contain slashes (`release/v5`).
const QUEUE_REF = /^gh-readonly-queue\/.+\/pr-(\d+)-([0-9a-f]{40})$/;
const m = QUEUE_REF.exec(run.head_branch ?? '');
if (!m) {
core.info(`head_branch '${run.head_branch}' is not a merge-queue branch — nothing to do.`);
return;
}
const prNumber = Number(m[1]);
const marker = `<!-- merge-queue-triage:${run.id} -->`;
// `HTTP 503: No server is currently available to service your request`,
// `ECONNRESET: ...` — enough for a reader to tell platform weather from
// a 403 that means the `permissions:` block above is wrong.
const describe = (error) => {
const kind = typeof error?.status === 'number'
? `HTTP ${error.status}`
: (error?.code || 'error');
return `${kind}: ${String(error?.message || '').replace(/\s*\.\s*$/, '')}`;
};
// Idempotency: workflow_run deliveries can repeat; one comment per run.
// The marker is scoped to THIS run id, so the only thing this listing can
// prevent is a second copy of this very comment.
//
// When the listing cannot be read at all, this degrades to AT-LEAST-ONCE
// on purpose and posts without knowing. The two mistakes are not
// symmetric here: a duplicate is inert — same run, same text, and the
// shared marker makes the pair self-evident — while a miss is the whole
// defect, since this comment IS the machine-readable signal the PM
// dispatch loop keys on and the cross-PR flake evidence lives in it.
// #9423 chose the opposite for docs-drift-check.yml, and correctly: that
// marker is STABLE across runs, so posting blind there strands a second
// advisory which the dedup then updates forever alongside the first.
//
// This early return also fences limb ② off from repeat deliveries: a
// redelivered workflow_run stops HERE, before any sighting is recorded
// and before any anchor is touched, so one queue build contributes its
// signature exactly once however many times GitHub delivers it.
let alreadyPosted = false;
try {
const existing = await github.rest.issues.listComments({
owner, repo, issue_number: prNumber, per_page: 100,
});
alreadyPosted = existing.data.some((c) => (c.body ?? '').includes(marker));
} catch (error) {
core.warning(
`Could not read #${prNumber}'s comments to check for an existing triage `
+ `comment (${describe(error)}). Posting anyway: a duplicate triage comment `
+ `is inert, a missing one loses the queue-failure signal.`,
{ title: 'Triage comment de-duplication skipped' },
);
}
if (alreadyPosted) {
core.info('triage comment for this run already exists — skipping.');
return;
}
// Failed jobs and their failed steps.
const jobs = await github.paginate(github.rest.actions.listJobsForWorkflowRun, {
owner, repo, run_id: run.id, per_page: 100,
});
const failedJobs = jobs.filter((j) => j.conclusion === 'failure');
// ── Log harvest ────────────────────────────────────────────────
//
// Best-effort: the lines a human would grep for first, plus the line
// that says WHY. Aggregate gate jobs (Test Core / Dogfood Regression
// Gate) fail with no information of their own, so prefer real jobs
// when both are present. A 4xx on the logs endpoint degrades to
// names only.
//
// The ESC byte is built rather than typed: a raw control byte in a
// repo file makes grep treat the whole file as binary and is a gate
// failure here (scripts/check-nul-bytes.mjs).
const ANSI = new RegExp(`${String.fromCharCode(27)}\\[[0-9;]*m`, 'g');
const plain = (l) => l.replace(ANSI, '').replace(/\r/g, '');
// A FAIL line, matched on the ANSI-STRIPPED text. The three cross
// marks must sit at the START of their line, which is where vitest
// prints them (` × test name 14ms`). Anywhere-matching cost real
// budget: on 2026-08-20 four of the twelve excerpted lines on PR
// #10008 were `stdout | …` lines whose only qualification was the
// multiplication sign inside a test TITLE (`(#7986 × #7799/#8022)`),
// crowding out lines that carried the failure.
const FAIL_LINE = /^\s*(?:✗|×|✕)\s|(?:^|[\s|])(?:FAIL\s|AssertionError|STALL)/;
// The deciding line (#10128 limb ①). `Error:` last and unanchored is
// the card's "first `Error:` after the FAIL" fallback; the three
// timeout spellings are vitest's own (`testTimeout`, `hookTimeout`,
// `teardownTimeout`) and are listed explicitly so the label stays
// right when the fallback would also have matched them.
const REASON_LINE = /(?:Test|Hook|Teardown) timed out in \d+\s*ms|(?:^|[\s|>])(?:AssertionError|Error):\s/;
// The aggregation key (#10128 limb ②): the failing test FILE.
const TEST_FILE = /(?:^|[\s|"'(\[])((?:[\w.@-]+\/)*[\w.@-]+\.(?:test|spec)\.[cm]?[jt]sx?)/;
const REASON_WINDOW = 12;
const MAX_FAILS = 12;
const informative = failedJobs.filter((j) => (j.steps ?? []).some(
(s) => s.conclusion === 'failure' && !/^Verify .* results$/.test(s.name)));
const details = [];
let logsRefused = 0;
for (const job of (informative.length ? informative : failedJobs).slice(0, 4)) {
const steps = (job.steps ?? [])
.filter((s) => s.conclusion === 'failure')
.map((s) => s.name);
let lines = [];
let keys = [];
try {
const res = await github.request(
'GET /repos/{owner}/{repo}/actions/jobs/{job_id}/logs',
{ owner, repo, job_id: job.id });
const text = typeof res.data === 'string'
? res.data
: Buffer.from(res.data).toString('utf8');
const all = text.split('\n').map((l) => plain(l.replace(/^[^ ]*Z /, '')));
const hits = [];
// A line already printed as somebody's reason is not printed
// again as an excerpt line of its own. `AssertionError`
// qualifies under both patterns, so without this the assertion
// case reports every failure twice and spends half the budget
// saying the same thing.
const consumed = new Set();
for (let i = 0; i < all.length && hits.length < MAX_FAILS; i++) {
if (consumed.has(i)) continue;
if (!FAIL_LINE.test(all[i])) continue;
// vitest prints the reason UNDER the `FAIL <file> > <test>`
// header of its Failed-Tests block, never under the `× <test>`
// line of the file list. So only a FAIL-shaped line is asked
// for a reason, and only it can report one as missing --
// "no reason found" printed under a line that never carries
// one is noise that reads like a finding.
const wantsReason = /(?:^|[\s|])FAIL\s/.test(all[i]);
let reason = null;
if (wantsReason) {
// Look ahead from the FAIL line, and STOP at the next FAIL
// line so one failure cannot borrow the next one's reason.
const last = Math.min(i + REASON_WINDOW, all.length - 1);
for (let j = i + 1; j <= last; j++) {
// REASON first, deliberately: `AssertionError` qualifies
// under BOTH patterns, and testing the FAIL pattern first
// makes the assertion case break out on the very line it
// was looking for -- a timeout would then be reported and
// an assertion never would, which is the worse half of
// the split this limb exists to end.
if (REASON_LINE.test(all[j])) {
reason = all[j].trim().slice(0, 200);
consumed.add(j);
break;
}
if (FAIL_LINE.test(all[j])) break;
}
}
hits.push({ fail: all[i].trim().slice(0, 200), reason, wantsReason });
const f = TEST_FILE.exec(all[i]);
if (f) keys.push(f[1]);
}
lines = hits.flatMap((h) => {
if (!h.wantsReason) return [h.fail];
return [
h.fail,
h.reason
? ` ↳ 失败原因: ${h.reason}`
: ' ↳ 失败原因: (这条 FAIL 之后 12 行内没有可识别的原因行 —— 点进 job 看)',
];
});
keys = [...new Set(keys)];
} catch (e) {
logsRefused++;
core.info(`logs unavailable for job ${job.id}: ${e.message}`);
}
details.push({ name: job.name, url: job.html_url, steps, lines, keys });
}
// History: this PR's earlier queue failures + queue-wide count, 24 h.
const since = new Date(Date.now() - 24 * 3600 * 1000).toISOString();
const recent = await github.paginate(github.rest.actions.listWorkflowRunsForRepo, {
owner, repo, event: 'merge_group', created: `>=${since}`, per_page: 100,
});
const ciRuns = recent.filter((r) => r.workflow_id === run.workflow_id && r.id !== run.id);
const priorFailuresThisPr = ciRuns.filter((r) =>
r.conclusion === 'failure' && (r.head_branch ?? '').includes(`/pr-${prNumber}-`)).length;
const queueFailures24h = ciRuns.filter((r) => r.conclusion === 'failure').length;
// runId -> { base, head } for every merge_group run in the window,
// plus this one. `recent` is ALREADY in hand for the two counts above,
// and a run object already carries `head_branch` and `head_sha` — so
// the speculative stack costs ZERO additional API calls. That is the
// whole reason this discriminator is affordable in a job whose header
// calls itself rate-limit-shaped.
//
// `recent` is deliberately NOT narrowed to this workflow_id here: the
// sightings below are keyed by run id, and a wider index can only
// resolve more of them.
const stackEdges = new Map();
const noteRun = (r) => {
const q = QUEUE_REF.exec(r?.head_branch ?? '');
// A head_sha that is not a full sha cannot be compared to a base by
// equality, and an equality that silently never holds is the same
// failure as no data at all — so it is left OUT of the index rather
// than stored as something a comparison would quietly reject.
if (!q || !/^[0-9a-f]{40}$/.test(String(r?.head_sha ?? ''))) return;
stackEdges.set(String(r.id), { base: q[2], head: String(r.head_sha) });
};
for (const r of recent) noteRun(r);
noteRun(run);
/**
* Partition a key's victim PRs into speculative STACKS.
*
* Under speculative stacking GitHub builds each queued PR on top of
* the previous entry's queue head, so a build whose BASE commit IS
* another victim's queue HEAD contains that victim's tree by
* construction. One deterministic break therefore ejects everyone
* behind it, and the raw victim count measures QUEUE DEPTH.
*
* ⚠️ The test is full-sha equality in the strict direction: one
* victim's base == another victim's head. ⛔ Never a base PREFIX and
* ⛔ never time ordering — both would fuse genuinely independent hits
* into one stack, trading this limb's false positive for a false
* negative, which is strictly worse. The count exists to be true.
*
* A victim whose runs are not in the index (a run older than the
* window, or a ref that is not a queue ref) is left as its own group:
* "cannot prove downstream" must read as INDEPENDENT, never as
* same-stack. `unresolved` reports how many of those there were, so a
* partial index cannot pass as a complete one.
*/
const stacksFor = (byPr) => {
const parent = new Map([...byPr.keys()].map((p) => [p, p]));
const find = (x) => { while (parent.get(x) !== x) x = parent.get(x); return x; };
const union = (a, b) => { const ra = find(a); const rb = find(b); if (ra !== rb) parent.set(ra, rb); };
// head sha -> the victim PRs that PRODUCED it.
const producers = new Map();
for (const [pr, runs] of byPr) {
for (const id of runs) {
const e = stackEdges.get(String(id));
if (!e) continue;
if (!producers.has(e.head)) producers.set(e.head, new Set());
producers.get(e.head).add(pr);
}
}
const inherited = new Set();
let unresolved = 0;
for (const [pr, runs] of byPr) {
let resolved = false;
for (const id of runs) {
const e = stackEdges.get(String(id));
if (!e) continue;
resolved = true;
for (const upstream of producers.get(e.base) ?? []) {
if (upstream === pr) continue;
inherited.add(pr);
union(pr, upstream);
}
}
if (!resolved) unresolved++;
}
const groups = new Map();
for (const pr of byPr.keys()) {
const root = find(pr);
if (!groups.has(root)) groups.set(root, []);
groups.get(root).push(pr);
}
const ordered = [...groups.values()]
.map((g) => g.sort((x, y) => x - y))
.sort((a, b) => a[0] - b[0]);
const stackOf = new Map();
ordered.forEach((g, i) => { for (const pr of g) stackOf.set(pr, { index: i + 1, size: g.length }); });
return { groups: ordered, stackOf, inherited, unresolved };
};
// ── Limb ②: cross-PR signature aggregation ─────────────────────
//
// The ledger is this workflow's OWN comments. Every triage comment
// carries one machine-readable sighting marker per failing test
// file, and a later ejection reads 24 h of them back. That keeps the
// evidence where the header always said it lived, needs no state
// outside the repo, and cannot disagree with what a human reading
// the comments would conclude.
const SIGHTING = /<!-- queue-signature:([^|]+)\|pr=(\d+)\|run=(\d+) -->/g;
const runKeys = [...new Set(details.flatMap((d) => d.keys))];
/** key -> Map(prNumber -> Set(runId)) */
const sightings = new Map();
const note = (key, pr, runId) => {
if (!sightings.has(key)) sightings.set(key, new Map());
const byPr = sightings.get(key);
if (!byPr.has(pr)) byPr.set(pr, new Set());
byPr.get(pr).add(String(runId));
};
for (const k of runKeys) note(k, prNumber, run.id);
// A bounded read: newest-first inside the window, at most 5 pages.
// Reaching the cap is NOT "no other PR hit this key" — it is "I did
// not look at the whole window", and the two are reported
// differently on purpose. An enumerator that cannot say which of
// those it found is the no-op this limb would otherwise decay into.
const MAX_LEDGER_PAGES = 5;
let ledgerComplete = true;
let ledgerRefusal = null;
try {
let page = 1;
for (; page <= MAX_LEDGER_PAGES; page++) {
const res = await github.rest.issues.listCommentsForRepo({
owner, repo, since, sort: 'created', direction: 'desc', per_page: 100, page,
});
for (const c of res.data) {
for (const s of String(c.body ?? '').matchAll(SIGHTING)) {
note(s[1], Number(s[2]), s[3]);
}
}
if (res.data.length < 100) break;
if (page === MAX_LEDGER_PAGES) ledgerComplete = false;
}
} catch (error) {
ledgerComplete = false;
ledgerRefusal = describe(error);
}
// Only keys THIS ejection actually hit are aggregated: an ejection
// must not re-file anchors for signatures it had nothing to do with.
// ⛔ The trigger stays `a.prs.size >= 2`. Stack inheritance changes
// what this limb REPORTS, never what it notices: raising the bar to
// "2 independent hits" would silence a real deterministic break that
// has already eaten the whole queue behind it, which is precisely the
// case a reader most needs to see. Honest counting, not a quieter one.
const aggregated = runKeys
.map((key) => {
const prs = sightings.get(key) ?? new Map();
return { key, prs, ...stacksFor(prs) };
})
.filter((a) => a.prs.size >= 2)
.sort((a, b) => b.prs.size - a.prs.size);
/**
* The one sentence four call sites need: how many PRs, and how many of
* them are actually independent.
*
* Single-sourced because the four `anchorNotes` branches below (refresh
* / refresh-failed / not-established / create-failed) each render it,
* and a phrase copied four times is a phrase that drifts in three.
*/
const victimPhrase = (a) => {
const n = a.prs.size;
const c = a.groups.length;
const list = [...a.prs.keys()].sort((x, y) => x - y).map((p) => `#${p}`).join('、');
const tail = a.unresolved > 0
? `(其中 ${a.unresolved} 个的队列构建不在 24h 窗口索引内,无法判定栈关系,已按独立计)`
: '';
if (c === n) return `24h 内已弹出 **${n} 个互相独立的 PR**(${list})${tail}`;
if (c === 1) return `24h 内已弹出 **${n} 个 PR**(${list}),但它们同属 **1 条投机栈** ⇒ **1 次独立命中**${tail}`;
return `24h 内已弹出 **${n} 个 PR**(${list}),分属 **${c} 条投机栈** ⇒ **${c} 次独立命中**${tail}`;
};
const ANCHOR_LABEL = 'finding';
const MAX_ANCHOR_PAGES = 3;
const anchorNotes = [];
for (const a of aggregated.slice(0, 3)) {
const anchorMarker = `<!-- queue-signature-anchor:${a.key} -->`;
// Stable across refreshes — the victim count lives in the body, so
// a growing count cannot make the anchor unfindable by title.
const title = `Queue-flake anchor: ${a.key}`;
const prRows = [...a.prs.entries()].sort((x, y) => x[0] - y[0]).map(([pr, runs]) => {
const s = a.stackOf.get(pr);
const cell = !s || s.size < 2
? 'independent'
: `S${s.index} · ${a.inherited.has(pr) ? 'inherited' : 'root'}`;
return `| #${pr} | ${cell} | ${[...runs].map((r) => `[${r}](https://github.com/${owner}/${repo}/actions/runs/${r})`).join(' · ')} |`;
});
const independent = a.groups.length;
const body = [
`\`${a.key}\` has ejected **${a.prs.size} pull request${a.prs.size === 1 ? '' : 's'}** from the merge`,
`queue within a rolling 24 hours — **${independent} independent hit${independent === 1 ? '' : 's'}** once`,
"GitHub's speculative stacking is accounted for. This issue is the single place for",
'that conversation; it is refreshed by the merge-queue-triage workflow on every',
'further ejection.',
'',
'| PR | stack | queue build |',
'|---|---|---|',
...prRows,
'',
...(independent < a.prs.size
? [
"⚠️ **Queue depth is not evidence.** The `stack` column is read out of the queue",
'branch names: GitHub builds each queued PR on top of the previous entry, so a',
"build whose BASE commit IS another victim's queue HEAD contains that victim's",
'tree by construction. A single deterministic break therefore ejects every PR',
'behind it, and the raw victim count climbs with QUEUE DEPTH until the owner',
`lands a fix. Start with the ${independent === 1 ? 'root' : 'roots'} above; an `
+ '`inherited` row is a bystander until shown otherwise.',
'',
]
: []),
...(a.unresolved > 0
? [
`⚠️ ${a.unresolved} of the rows above could not be placed in a stack at all (their`,
'queue build fell outside the 24 h run index). They are counted as INDEPENDENT',
'here, which is the safe direction but may overstate the independent-hit count.',
'',
]
: []),
'**This issue is a NAME, not a diagnosis.** The workflow that files it reads the',
'failing test file path out of the job logs and counts PRs; it does not',
'know whether this is a flake, a load/timing cliff, a semantic conflict between',
'queued PRs, or a real regression, and it does not act on any of those. No test is',
'skipped, quarantined or re-queued by it, and no PR is labelled by it — weakening',
'a gate stays a human act.',
'',
'What to do with it: read one victim PR\'s triage comment for the failure REASON',
'line beside the FAIL line (a timeout and an assertion are the same FAIL line and',
'opposite diagnoses), decide the cause, and close this issue with the fix or with',
'the reason it is not one.',
'',
`Last refreshed by queue build [${run.id}](${run.html_url}) (PR #${prNumber}).`,
'',
'---',
'_Filed by the merge-queue-triage workflow (#4859, aggregation #10128)._',
'',
anchorMarker,
].join('\n');
let existing = null;
let scanComplete = true;
let refusal = null;
try {
for (let p = 1; p <= MAX_ANCHOR_PAGES; p++) {
const res = await github.rest.issues.listForRepo({
owner, repo, state: 'open', labels: ANCHOR_LABEL,
sort: 'created', direction: 'desc', per_page: 100, page: p,
});
// Identity is the body marker; the exact title is a second
// way in, because a body is the one channel GitHub is known
// to rewrite and an anchor that cannot be found is an anchor
// that gets duplicated.
existing = res.data.find((i) => !i.pull_request
&& (String(i.body ?? '').includes(anchorMarker) || i.title === title)) ?? null;
if (existing || res.data.length < 100) break;
if (p === MAX_ANCHOR_PAGES) scanComplete = false;
}
} catch (error) {
scanComplete = false;
refusal = describe(error);
}
if (existing) {
try {
await github.rest.issues.update({
owner, repo, issue_number: existing.number, body,
});
anchorNotes.push(`- \`${a.key}\` — ${victimPhrase(a)}。汇总 issue:#${existing.number}(已刷新)`);
} catch (error) {
anchorNotes.push(`- \`${a.key}\` — ${victimPhrase(a)}。⚠️ 汇总 issue #${existing.number} 刷新失败(${describe(error)}),上面的名单就是全部事实。`);
core.warning(`Could not refresh the anchor issue #${existing.number} for ${a.key} (${describe(error)}).`,
{ title: 'Queue-signature anchor not refreshed' });
}
continue;
}
if (!scanComplete) {
// Not found AND not fully looked for. Creating here would risk a
// second anchor for a key that already has one, which is exactly
// the duplication the marker exists to prevent — so this says so
// instead of guessing.
anchorNotes.push(`- \`${a.key}\` — ${victimPhrase(a)}。⚠️ 没能确认是否已有汇总 issue(${refusal ?? `open issue 列表超过 ${MAX_ANCHOR_PAGES} 页仍未匹配`}),本次不新建,避免开出重复的锚点。`);
core.warning(`Could not establish whether an anchor issue already exists for ${a.key}; not creating one.`,
{ title: 'Queue-signature anchor not created' });
continue;
}
try {
const created = await github.rest.issues.create({
owner, repo, title, body, labels: [ANCHOR_LABEL],
});
anchorNotes.push(`- \`${a.key}\` — ${victimPhrase(a)}。汇总 issue:#${created.data.number}(新建)`);
} catch (error) {
anchorNotes.push(`- \`${a.key}\` — ${victimPhrase(a)}。⚠️ 汇总 issue 新建失败(${describe(error)}),上面的名单就是全部事实。`);
core.warning(`Could not file the anchor issue for ${a.key} (${describe(error)}).`,
{ title: 'Queue-signature anchor not created' });
}
}
// Keys this run saw that are NOT yet shared with another PR still get
// a line: "seen once" and "not looked at" must not read alike.
for (const key of runKeys) {
const prs = sightings.get(key) ?? new Map();
if (prs.size >= 2) continue;
anchorNotes.push(`- \`${key}\` — 24h 窗口内只有本 PR 撞到过,暂不汇总(再有一个不同 PR 撞到就会自动开汇总 issue)。`);
}
// The anti-no-op limb. An enumerator over a corpus it could not read
// prints exactly what an enumerator over a clean corpus prints, and
// both of them are green. These three say which one happened.
if (runKeys.length === 0) {
const why = logsRefused > 0
? `${logsRefused} 个 job 的日志读不到`
: '日志里没有能解析出测试文件名的 FAIL 行';
anchorNotes.push(`- ⚠️ **本次没有可用的聚合签名**(${why})—— 这不是「没有同签名的其他 PR」,是**这一轮没测到**。跨 PR 聚合本次不可用,请手工比对其他 PR 的同类评论。`);
core.warning(
`No signature key could be derived from this run's logs (${why}), so the cross-PR `
+ 'aggregation did nothing this time. That is an absent measurement, not an absent signal.',
{ title: 'Queue-signature aggregation had nothing to key on' },
);
}
if (!ledgerComplete) {
anchorNotes.push(`- ⚠️ 24h 评论账本**没读完**(${ledgerRefusal ?? `超过 ${MAX_LEDGER_PAGES} 页仍未读到窗口尽头`}),所以上面的「不同 PR 数」是**下界**,不是全量。`);
core.warning(
`The 24h sighting ledger was truncated (${ledgerRefusal ?? 'page cap reached'}); `
+ 'distinct-PR counts in this comment are lower bounds.',
{ title: 'Queue-signature ledger incomplete' },
);
}
const jobSections = details.map((d) => {
const head = `- **[${d.name}](${d.url})** — 失败步骤: ${d.steps.join('、') || '(无步骤级结论)'}`;
return d.lines.length
? `${head}\n\n \`\`\`\n ${d.lines.join('\n ')}\n \`\`\``
: `${head}(日志不可读,点进 job 看)`;
}).join('\n');
const flakeHint = priorFailuresThisPr > 0
? `⚠️ **本 PR 过去 24h 已在队列失败 ${priorFailuresThisPr} 次(不含本次)。** 内容未变而反复失败 ⇒ 高度怀疑 flaky 测试或与同组 PR 的语义冲突,重排不解决。`
: '本 PR 过去 24h 无队列失败记录(首次)。';
// The sighting markers this comment contributes to the 24 h ledger.
const sightingMarkers = runKeys.map((k) => `<!-- queue-signature:${k}|pr=${prNumber}|run=${run.id} -->`);
const body = [
'### ⛔ merge queue 构建失败 — 先分诊,再决定要不要重排',
'',
`队列构建 [${run.id}](${run.html_url}) 红了。队列跑的是**全量**套件(PR 侧 CI 只跑 affected 子集),`,
'所以失败的测试可能在本 PR 没碰过的包里 —— 那不是重排能修的。每次盲目重排都会让排在后面的所有 PR 重建一轮。',
'',
'**失败的 job(日志抽取,best effort):**',
'',
jobSections || '- (没拿到 job 级信息,点上面的 run 链接看)',
'',
'> `↳ 失败原因` 是判读的关键:**超时**(`Test timed out in …` / `Hook timed out in …`)多半是负载/时序,不是本 PR 的回归;',
'> **断言**(`AssertionError: …`)才指向真实的行为改变。两者的 `FAIL` 行长得一模一样,只有这一行能区分。',
'>',
'> ⚠️ **断言这一侧有一类例外,判据是断言在测什么,不是它是不是 `AssertionError`。** 断言的对象是**产品行为**(一个值、一个形状、一次拒收)⇒ 照上面读:真实的行为改变,去查,⛔ 不要重排掉;',
'> 断言的对象是**这次实验自身的有效性前提**(跑完的耗时、负载下的先后、任何只在时间预算内才成立的条件)⇒ 它跟**超时**是同一类,同样对负载敏感,重排一次是合法的判别手段。',
'> 识别是机械的:断言的消息或它比较的值本身点名了一段**时长、一个时间戳、一个耗时计数**。实测过的一对 —— `AssertionError: SecurityPlugin.init() ran: expected false to be true` 测的是产品行为(真回归);',
'> `AssertionError: this run took over a second, so second-precision stamps could have differed too: expected 1006 to be less than 1000` 测的是实验前提:它守护的那条不变式当时是绿的,同一个 head 原样重排一次即成功。',
'> 穿着 `AssertionError` 外衣的时间测量,仍然是时间测量。(⛔ 这只改「怎么读一次红」,不改「哪些测试可以重排」——后者由别处管。)',
'',
'**跨 PR 相同签名(24h,按失败测试文件聚合):**',
'',
anchorNotes.join('\n') || '- (本次没有可聚合的失败测试文件)',
'',
'**历史信号:**',
`- ${flakeHint}`,
`- 过去 24h 队列共有 ${queueFailures24h} 个失败构建(不含本次)。`,
'',
'**分诊清单:**',
'1. 失败测试在本 PR 改动的包里 → 真回归,修 PR。',
'2. 失败测试与本 PR 无关 → 看上面的「跨 PR 相同签名」;已有汇总 issue ⇒ flaky/环境问题实锤,去那张 issue 上谈,修好前重排只会再烧一轮全队列。',
'3. 两者都不是 → 可能与同组 PR 语义冲突;等前面的 PR 落地或失败出队后再重排一次即可,不要连续重排。',
'',
marker,
...sightingMarkers,
'---',
'_Generated by [Claude Code](https://claude.ai/code) · merge-queue-triage workflow (#4859)_',
].join('\n');
// Everything above is the DIAGNOSIS; the call below only carries it to
// the PR. The step's `retries:` have already absorbed a blip by the time
// anything lands here, so what is left is a refusal that outlived them —
// and `body` exists nowhere else, so letting it throw (which is what this
// step did until #9424) loses the entire queue-failure triage and leaves
// the run page showing `Unhandled error: HttpError` and nothing more.
//
// So the diagnosis is written where it outlives the request, and THEN the
// job fails. Failing is the point: see the invariant in this file's header
// — green means delivered, and nothing about this workflow_run job's
// conclusion costs anything to anybody. Tolerance is scoped to the retry,
// never to the outcome, and it never reaches detection: a failure above
// this line still fails exactly as it always did.
try {
await github.rest.issues.createComment({ owner, repo, issue_number: prNumber, body });
core.info(`triage comment posted on #${prNumber}.`);
} catch (error) {
const reason = describe(error);
try {
await core.summary.addRaw([
`## ⚠️ 队列失败分诊已生成,但没能发到 PR #${prNumber}`,
'',
`\`issues.createComment\` 最终被拒绝:\`${reason}\`。`,
'',
`- 这是**投递**失败,不是分诊失败。下面就是本次运行算出来的完整分诊内容 —— PR #${prNumber} 上没有它。`,
'- 可以直接复制到 PR 上;也可以在 API 恢复后 re-run 本 job,重复投递会被评论里的 marker 挡掉。',
'- 怎么分辨:5xx / 429 / 网络错误码是平台抖动,re-run 即可;4xx(422 正文超长、403 权限)是本仓自己的 bug,re-run 不会变绿,去修它。',
'- ⚠️ 本次的签名 sighting 也随这条评论一起丢了:跨 PR 聚合账本读的就是这些评论,所以在补发之前,本次弹出对后续聚合是不可见的。',
'- 本 job 判红是有意的:这个 workflow 的结论不出现在任何 check 列表上,红的代价是零,而绿会让「信号丢了」跟「信号送到了」长得一模一样。',
'',
'---',
'',
body,
'',
].join('\n')).write();
} catch (summaryError) {
// The summary is the richer channel, the annotation the reliable one.
// Losing the richer one must not restore the silence this prevents.
core.info(`Could not write the job summary: ${summaryError.message}`);
}
core.setFailed(
`The queue-failure triage for PR #${prNumber} was computed but could NOT be `
+ `posted (${reason}). It is reproduced in full in this run's job summary: copy `
+ `it onto the PR, or re-run this job to retry delivery once the API recovers. `
+ `Queue run ${run.id}: ${run.html_url}`,
);
}