Skip to content

test(recorder): wait for progress instead of a fixed window in the stall tests - #995

Merged
srperens merged 1 commit into
mainfrom
test/recorder-stall-flake
Oct 7, 2026
Merged

srperens merged 1 commit into
mainfrom
test/recorder-stall-flake

Conversation

@srperens

@srperens srperens commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The recorder's stall tests (backend/tests/recorder_stall_test.rs) count buffers over a fixed 2 s window. They fail now and then on the Windows runner:

A debug x264enc on a busy Windows runner can fall behind real time, so a running track can miss the minimum in a fixed window.

Change

All three tests now wait, up to 30 s, until each counted track has handed the muxer the same minimum (MIN_IN_WINDOW, 18 buffers). They no longer count over a fixed window. The sleeps before counting (multiples of the 1 s stall timeout) are unchanged, so the watchdog still has had time to misfire before the count starts.

A track that was ended, or a muxer frozen on a dead track, delivers nothing however long the test waits, so the extra time only helps a slow runner.

This replaces this PR's earlier commit, which changed the same test in recorder_test.rs before #1000 moved it into its own binary.

Evidence

  • Still a guard: with spawn_track_stall_watchdog not called (local edit, reverted), a_track_that_stops_does_not_freeze_the_recording fails: the recording froze on the track that stopped: 0 video buffers reached the muxer within 30s (0 bytes written).
  • The Windows failure on bfccbdb is unverified as slowness. If the watchdog really ended the live audio track there, this test keeps failing (0 buffers in 30 s) rather than going green, which would then be a real watchdog bug to chase.

Tests

Ran (macOS, GStreamer 1.28.6): cargo test --test recorder_stall_test, 3 passed; once with the watchdog disabled, failing as quoted above.
Not run: Windows. This PR carries ci:windows.

🤖 Generated with Claude Code

@srperens srperens added the ci:windows Run Build (Windows) on this pull request label Oct 5, 2026

@srperens srperens left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: Approve. The test now waits for progress, not a fixed amount of wall-clock time. It asserts the same 45-buffer threshold, so a frozen muxer still fails it, and a slow runner gets up to 30 s instead of 5 s.

Claims

Claim Verdict Evidence
The test still fails when the muxer freezes CONFIRMED The loop exits early only once 45 buffers have arrived. Otherwise it runs to the deadline (backend/tests/recorder_test.rs:1125 — let deadline = Instant::now() + Duration::from_secs(30);), and the unchanged assert still requires backend/tests/recorder_test.rs:1142 — buffers_after - buffers_before >= 45,. The waiting can delay a failure but cannot turn one into a pass.
A frozen splitmuxsink takes 0 video buffers EXTERNAL This is splitmuxsink behaviour. The test's own doc comment says the same ("pins the counted video at zero"), and the body reports 0 in 30 s with the watchdog disabled.
The test runs in CI on Linux and Windows CONFIRMED stalled_track::a_track_that_stops_does_not_freeze_the_recording ... ok appears in both Check (Linux) and Build (Windows) (run 37333970488). The skip path goes through common::plugins_available, which panics under CI's STROM_REQUIRE_GST_PLUGINS (backend/tests/common/mod.rs:29 — strom_types::env::var_opt("STROM_REQUIRE_GST_PLUGINS").is_none(),).

Diagnosis. The failure message (40 buffers, 273 kB written) shows a muxer that was still running, so the cause was a slow encoder, not a regression. The coverage tradeoff: the test no longer catches a recording that recovers but runs well below real time. That was never this test's purpose, and the body says so. One Windows pass does not prove an intermittent flake is gone. It does show the mechanism works on that runner.

Radius. LOCAL. One test, with no backend/src change.

Overlaps.

  • #762 adds two more stalled-track tests with the same fixed std::thread::sleep(Duration::from_secs(5)) window and the same >= 45 assert. #927 adds 5 s windows too. Neither conflicts with this PR (trial merges are clean), but both inherit the Windows flake. Whichever lands after this should adopt the polling loop.
  • The same pattern stays in this file at backend/tests/recorder_test.rs:1173 — std::thread::sleep(Duration::from_secs(5)); (tracks_that_keep_running_are_left_alone) and backend/tests/recorder_test.rs:1254 — std::thread::sleep(Duration::from_secs(5)); (a_muxer_that_stalls_for_every_track_ends_none). The body leaves these alone on purpose, and that is reasonable until they flake.

Tests & CI. At d29a18f: Check (Linux), Build (Windows) (via ci:windows), both Linux builds, WASM, API Contract Check and Static JS Tests pass. macOS was skipped, and nothing here is platform-specific. Nothing was built or run here.

Confidence: HIGH

@srperens srperens mentioned this pull request Oct 6, 2026
…all tests

All three stall tests counted buffers over a fixed 2 s window. A slow
Windows runner missed the minimum in it (0 audio buffers in
tracks_that_keep_running_are_left_alone on main bfccbdb), although an
ended track delivers nothing however long the test waits. Wait up to
30 s for the same minimum instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@srperens
srperens force-pushed the test/recorder-stall-flake branch from d29a18f to ca9a878 Compare October 7, 2026 09:00
@srperens srperens changed the title test(recorder): wait for progress in the stalled-track recovery test test(recorder): wait for progress instead of a fixed window in the stall tests Oct 7, 2026
@srperens
srperens merged commit 5cb7b3f into main Oct 7, 2026
9 checks passed
@srperens
srperens deleted the test/recorder-stall-flake branch October 8, 2026 08:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:windows Run Build (Windows) on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant