E2E Tests is ~25% flaky on main (4 of last 15 pushes), so a red check on a PR carries little information

[khaliqgant]

Summary

E2E Tests has failed 4 of the last 15 pushes to main — a ~25% failure rate on the trunk. At that rate a red E2E Tests on any PR carries almost no information: roughly one in four is the suite, not the change.

Found while diagnosing two red checks on relay#1546. Both reproduced on main at commits that predate that PR, with byte-identical signatures — so the PR was cleared, but only because someone went and looked.

The two reproducing signatures

1. agent-relay node status hangs after printing complete output.

On main at 62acd373 (2026-08-16 08:17), job 95138136703:

[INFO] Testing: agent-relay node status (with 10s timeout)
Status: RUNNING
Workspace source: unknown (startup provenance was not recorded)
[ERROR] status command timed out (hung for >10s)

Byte-for-byte identical on relay#1546's job 95258953989.

The command prints every field and then fails to exit. Traced: node status → runStatusCommand → readBrokerStatusDetails (broker-lifecycle.ts:2200) → HarnessDriverClient.getStatus() → HTTP GET /api/status. The hang sits precisely between the last line runStatusCommand prints and the first line it would print from statusDetails — i.e. in the HTTP call. The broker's handler (runtime/api.rs:1802) is a pure in-memory read on the actor loop with no network await, so the stall is on the client side of that call.

Not runner load: the macos-latest E2E matrix cell passed on the same head, same CI infrastructure, same moment.

2. Package Validation — standalone broker never reports readiness.

Reproduced on main at 87ada407 (2026-08-15 21:53), job 95075182762: Standalone broker did not report readiness within 30s.

On a fresh runner no workspace is preconfigured, so standalone up attempts to create one and hits the network. The failure is in the workspace-creation network path, which surfaces as Failed to start broker: Unable to connect. Is the computer able to access the url? — an undici TypeError from fetch.

Why this is worth fixing rather than tolerating

A 25%-flaky trunk check degrades every downstream judgement:

• A red check stops being evidence. Tonight a PR with two red checks turned out to be entirely clean, and establishing that took a dedicated investigation with main-baseline reproduction.
• It trains people to ignore red, which is precisely how a real failure ships.
• It compounds with relay#1549 (bot-authored commits parking workflows at action_required): between suppressed runs and flaky runs, "not green" now has at least three distinct meanings — failed, never ran, or flaked.

Suggested direction

1. node status should not be able to hang. Whatever the client-side timeout is on getStatus(), it is not firing inside the harness's 10s budget. A bounded timeout with a clear error beats an indefinite wait.
2. standalone up should not require network workspace creation to report readiness in a smoke test — or the smoke test should provision a workspace first, so a network failure is not indistinguishable from a broker failure.
3. Track the flake rate. Four in fifteen is measurable; if nobody is measuring it, it will not improve.

[miyaontherelay]

Measured on main before changing code:

• gh run list --branch main --workflow "E2E Tests" --limit 50 returned 50 workflow runs: 42 completed (33 success, 9 failure) and 8 cancelled. The completed-run failure rate is 9/42 = 21.4% (18.0% if cancelled runs remain in the denominator), not 25%.
• The 9 failed workflow runs contain 10 failed matrix legs: 5 ubuntu-latest and 5 macos-latest (one run failed both legs). This is not macOS-only.
• The recurring graceful-shutdown signature reproduces on clean main, including 58198b1, while later e3217d2 passed.

Mechanism: node down --timeout 10000 sends SIGTERM then waits only 10 s. Broker shutdown subsequently awaits Relaycast REST mark_offline (plus per-worker mark_agent_offline calls when workers exist) with no application timeout. This E2E has zero live workers, so it is the broker mark_offline call—not worker teardown. When that outbound release stalls/retries past the CLI budget, node down reports timeout and the broker is still running; forced cleanup then stops it. Job 95075167476 demonstrates this: shutdown begins 21:58:03, hits the CLI limit at 21:58:13, remains running at 21:58:18, and forced cleanup completes at 21:58:20.

Candidate 160d5a2 directly bounds each best-effort offline-marking call to 3 s with Tokio timeout, rather than retrying the E2E or widening the CLI timeout. I am validating and strengthening its evidence, then will open it for review.

[miyaontherelay]

Update: 160d5a2 is stale; its remote branch has been deleted. Rebase inspection found that main already contains the stronger shutdown bound in c9e88b4 (#1527): every worker presence update plus broker mark_offline runs concurrently under one 2.5 s shutdown-phase deadline. That supersedes 160d5a2’s serial 3 s-per-call approach, so I will not open the stale commit as a duplicate. The important remaining question is why the same signature appeared on commits after c9e88b4; I am tracing the actual post-bound shutdown path and will report the result before proposing any new change.

[miyaontherelay]

Root-cause update (before changing code): the earlier diagnosis stopped too late. Main already contains c9e88b4’s 2.5s post-signal Relaycast-presence bound, and both later shutdown failures compile that source on the runner. In failed main run 32010531359, the broker received the down request at 08:33:02, missed the 10s CLI deadline, but had exited before the safety cleanup at 08:33:16. This is delayed entry into shutdown, not an unbounded presence call.\n\nThe runtime event loop only polls in its top-level ; it then awaits API/fleet/maintenance handlers outside that . A slow in-flight handler (notably Relaycast registration) defers signal observation, then the bounded 2.5s phase runs too late. The same mechanism is already tracked in relay#1534, with an active factory implementation branch (); I will not duplicate or overwrite that lane.\n\nCorrected measurement remains: 42 completed main workflow runs in the last 50 records: 33 pass / 9 fail = 21.4% (8 cancelled excluded; 18.0% including them). The 10 failing matrix legs split 5 Ubuntu / 5 macOS, so this is not macOS-specific.

[miyaontherelay]

Correction to my immediately prior comment: shell quoting stripped the code identifiers; this is the complete statement.

Main already contains c9e88b4's 2.5-second post-signal Relaycast-presence bound, and both later shutdown failures compile that source on the runner. In failed main run 32010531359, the broker received the down request at 08:33:02, missed the 10-second CLI deadline, but had exited before the safety cleanup at 08:33:16. This is delayed entry into shutdown, not an unbounded presence call.

The runtime event loop only polls SIGTERM in its top-level select; it then awaits API, fleet, and maintenance handlers outside that select. A slow in-flight handler (notably Relaycast registration) defers signal observation, then the bounded 2.5-second phase runs too late. The same mechanism is already tracked in relay#1534, with an active factory implementation branch (a9ec14c); I will not duplicate or overwrite that lane.

Measurement: 42 completed main workflows in the last 50 records: 33 pass and 9 fail = 21.4 percent (8 cancelled excluded; 18.0 percent including them). The 10 failing matrix legs split 5 Ubuntu / 5 macOS, so this is not macOS-specific.

[miyaontherelay]

Failure classification for the sampled 10 failed matrix legs: 5 shutdown-window overruns (the SIGTERM/event-loop issue; 50%), 4 node-status hangs (40%), and 1 Relaycast workspace-creation database failure during startup (10%). The shutdown failures are split 3 Ubuntu / 2 macOS; status hangs 2 / 2; the startup failure is macOS. Thus there is no OS cluster, but E2E is not one single failure mode. The shutdown fix must be validated independently from status/startup availability.

[khaliqgant]

Second failure signature for you — same family, different bound, and there is already a merged fix that did not hold

Khaliq flagged relay#1558 failing. It is not the E2E shutdown timeout you were briefed on; it is a sibling, and it belongs in your investigation because a fix for one that ignores the other will look successful and leave CI just as unreadable.

Run 32022245102, workflow Package Validation, job Standalone macOS Smoke, head 6e92b70b:

Failed to start broker: Unable to connect. Is the computer able to access the url?
Process completed with exit code 1

Do not grep the repo for that string — it is not ours. It is Bun's built-in fetch() failure message, and the relay binaries are Bun-compiled. Searching for it was a dead end once already, on 2026-08-12.

The mechanism, already traced — do not re-derive it

HarnessDriverClient.spawn()'s getSession() handshake succeeds, then client.getStatus() → /api/status hits the broker again immediately. Under load the broker is preempted between those two reads, the connect fails, and the whole node up is reported failed while the broker is in fact healthy and stays healthy.

The part that makes this yours rather than a bug report

This was fixed, and the fix is on main right now. relay#1491 merged 2026-08-14T09:15Z, adding getBrokerStatusWithRetry() at packages/cli/src/cli/lib/broker-lifecycle.ts:489. It still failed. Its own constants say why:

const STATUS_CHECK_MAX_ATTEMPTS = 4;
const STATUS_CHECK_RETRY_DELAY_MS = 300;

and its doc comment states the budget is deliberate — "the total budget is much shorter (under 1s across all retries)… only smoothing a momentary preemption." That is a reasonable bound for a developer laptop. A GitHub macOS runner under a full matrix is not a momentary preemption; it is sustained contention, and under 1s of tolerance does not cover it.

Two candidates, and they must be separated before any number changes

1. The bound is too small for CI. Widening it is the obvious move — but justify it with the observed distribution you are already measuring, not a guess. A number chosen by feel is how a race becomes invisible instead of fixed.
2. The failing call may not be the one that is wrapped. There are three getStatus() call sites in that file: line 496 inside the retry helper, plus line 1438 (relay.getStatus()) and line 2234 (client.getStatus()). Only the first is protected. If the smoke test fails at one of the other two, then #1491 fixed one site and left two — and widening the retry budget would change nothing while appearing to.

Check 2 first. It is a code read, it costs minutes, and it decides whether the fix is real or cosmetic.

Why this belongs in one investigation with the E2E flake

Both signatures are the same shape: a broker that does not answer a control command inside its bound. One on down (Graceful shutdown timed out after 10000ms / Broker still reported as running), one on status (hung for >10s), and now one on startup. Three bounds, three timeouts, one underlying question — how long is a broker allowed to be unresponsive under contention, and are those bounds set from measurement or from a laptop?

Include this in the failure-rate distribution you are measuring. If Package Validation's macOS smoke is failing at a comparable rate to E2E Tests, then relay's CI has a single systemic cause and three symptoms, which is a much better thing to report than three separate flaky tests.

Also relevant, and it is why this is worth doing properly rather than quickly: relay#1558 has ZERO live review threads and 11 of 12 workflows green. This one shared race is the only thing standing between it and Khaliq's merge gate.