Skip to content

CI-only failure: runtime::relaycast_events::tests::spawn_request_returns_the_verified_process_failure (ubuntu-latest, timing-sensitive) #1516

Description

@miyaontherelay

Summary

runtime::relaycast_events::tests::spawn_request_returns_the_verified_process_failure failed on the Rust Tests (ubuntu-latest) job of PR #1431 (head 539293e1c), attempt 1.

Not calling this a flake outright. The fixture margin is thin (see below) and the failure is consistent with CI-runner load, but that has not been proven — only asserted as a hypothesis.

Test

crates/broker/src/runtime/relaycast_events.rs:871 — asserts that a sidecar process exiting during the spawn stability window causes the spawn to fail with "process exited during startup", via .expect_err(...).

Failure evidence

  • Run: https://github.com/AgentWorkforce/relay/actions/runs/31791291936 (attempt 1, Rust Tests (ubuntu-latest), created 2026-08-14T10:12:42Z)
  • Panic: crates/broker/src/runtime/relaycast_events.rs:871:10a sidecar that exits during the stability window must fail the spawn: ()
  • test result: FAILED. 953 passed; 1 failed; 4 ignored
  • 14 other required checks on this head: SUCCESS. Rust Tests (macos-latest) on the same head: SUCCESS. Only ubuntu-latest failed.

Timing margin

The fixture spawns sh -c "sleep 0.05; exit 23" (50ms exit) against WORKER_SPAWN_STABILITY_WINDOW = Duration::from_millis(250) (crates/broker/src/worker.rs:59) — nominally 5x margin, but that is an absolute 200ms slack, thin for a loaded shared CI runner. Note: the fixture line traces (via git log -S) to f3733a180 — one of PR #1431's own commits — not to a pre-existing/inherited commit as initially reported in internal handoff notes; worth correcting that attribution if it surfaces elsewhere. This does not change the failure's shape, only whether it's a pre-existing flaky test or a new test with an as-shipped thin margin.

Pattern — third occurrence today, same job, same crate

  • relay#1508 — delivery_retry_transient_blip_emits_failed_event_for_present_worker (1ms retry window)
  • relay#1513 — init_worker_send_failure_cleans_up_like_a_startup_rejection (subprocess EPIPE race)
  • This issue — spawn_request_returns_the_verified_process_failure (50ms exit vs 250ms window)

Three timing-sensitive broker tests failing on Rust Tests (ubuntu-latest) in one day, same crate, same shape (thin-margin real-subprocess timing assertions passing locally, failing intermittently in CI). Filing a linking meta-issue separately rather than treating each as independent.

Suggested next step (not investigated in depth)

  • Rerun this specific test in a loop on ubuntu-latest to establish flake vs. deterministic before touching the fixture or the production WORKER_SPAWN_STABILITY_WINDOW.
  • If flake, widen the fixture's sleep margin (not the production window) — the production window is a real behavioral constant, not test scaffolding.
  • Rule out (not yet ruled out) whether rebasing fix(broker): confirm fleet spawn success #1431 onto feat(fleet): publish declared worker metadata #1504's merged changes altered spawn-path timing; a local discriminator run (this test x5 on origin/main vs. 539293e1c) would settle that.

Consequence for #1431

This is what puts #1431 into mergeStateStatus: BLOCKED, independent of the review-coverage gap already flagged on this PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions