Repository navigation
Stinger clip source sometimes never parks its first clip on Windows #1069
Description
What happens
On the Windows CI runner, a stinger clip source sometimes never parks its first clip. The cue thread started at flow start gives up after CUE_TIMEOUT (5 s):
WARN strom::blocks::builtin::mediaplayer::builder: Media Player sting-restart: could not park stinger clip 0: clip 0 decoded no frame within 5s
After that, nothing tries again, so the stinger stays not ready (ready: false) until someone cues by hand. An operator would see a stinger that never arms after a flow start. It hits about 2 to 3 of the 13 tests in stinger_test.rs per run on Windows, and a different set each time. It has not been seen on Linux or on a developer machine.
What is known
Measured on the runner with why_not_parked() (added in #1067) and STINGER_TEST_LOG:
paused true, parked false, running false, cued index 0, current index 0.- The loaded clip matches the file on disk: same stamp (mtime, length), and same software-decode verdict. Stale-file detection is ruled out.
- The bridge builds and links the video chain (
Internal pad added: video_0,Linked internal video chain,clocksync_video paces from running time 0 ...). Then no buffer ever enters the park clocksync: the one-shot park probe on its sink pad never fires. - The probe is armed before the upstream pad is linked (
bridge.rs,arm_park_probebeforesrc_pad.link), so a first frame slipping past an unarmed probe is unlikely. - It often hits the second start of a flow in the same test (
a_take_outlived_by_a_restart_leaves_the_new_run_alone,stopping_mid_take_releases_the_pipeline), but also a first start (cpu_stingers_play_classic,a_failed_analysis_is_kept_and_shown_until_a_reload).
Next steps
- GStreamer debug from the runner for a failing case (
GST_DEBUGondecodebin3,multiqueue,qtdemux,clocksync,GST_PADS, written toGST_DEBUG_FILEand uploaded as an artifact), to see where the first buffer stops. - Decide whether a cue that times out at flow start should retry, whatever the root cause is.
Tests skipped on the Windows runner until this is fixed
#1067 skips these through common::skip_known_ci_failure(&["windows"], ...). Remove "windows" from each when this is fixed:
stinger_test.rs:stopping_mid_take_releases_the_pipeline,a_take_outlived_by_a_restart_leaves_the_new_run_alone,a_take_plays_with_no_park_probe_on_its_path,a_clip_with_early_audio_lands_its_first_video_frame_on_the_take,cpu_stingers_play_classicstinger_api_test.rs:a_take_is_followed_by_its_id
Other tests in stinger_test.rs still run on Windows and can still hit this now and then: a_downgraded_mask_never_shows_its_matte, a_clip_that_will_not_play_still_changes_the_program and a_failed_analysis_is_kept_and_shown_until_a_reload have each failed this way once.
Activity
ci: drop the DIAG step; point the Windows stinger skips at issue #1069
test(ci): skip tests that pass locally but fail on the macOS and Wind…
test(ci): skip two more stinger tests that never park on the Windows …
srperens commentedon Oct 14, 2026
CONFIRMED, for the part this issue asks to decide. A flow-start cue that times out is never retried. Why the first frame goes missing cannot be settled from the code; the
GST_DEBUGrun is how to find it.backend/src/blocks/builtin/mediaplayer/builder.rs:460—match cue_state.cue(index) {. Thestinger-cuethread makes this one call. OnErr, it logs (backend/src/blocks/builtin/mediaplayer/builder.rs:467—Err(e) => tracing::warn!() and exits. The 5 s limit isbackend/src/blocks/builtin/mediaplayer/state.rs:864—if started.elapsed() > CUE_TIMEOUT {.A manual cue after the timeout takes a different path than the first cue did. The clip is loaded and paused, so
cue()rewinds instead of loading. It arms a fresh probe (backend/src/blocks/builtin/mediaplayer/state.rs:838—self.stinger.arm_park_probe(&sink);), then pauses and seeks to 0. That fits a hand cue recovering.Option Retry. On a timeout at flow start, the cue thread calls
cue(index)once more. That second call takes the rewind path, the same one a hand cue takes. CLOSED loop: it stops once parked, and costs nothing on a healthy start. Keep the existing warning on the first timeout so the root cause stays visible in logs. SPECULATIVE (not verified): a flush-seek gets the frame past whatever held it at load.Option Wait. Collect the runner's
GST_DEBUGfirst and change nothing here. If the frame is stuck indecodebin3/multiqueueat preroll, the fix belongs at load time, and a retry would only hide it. The Windows skips from #1067 stay either way.Overlaps. No open item owns cue retry or the park path. #1072 edits
mediaplayer/builder.rs, but only the bridge appsrc latency, not the cue thread.Radius.
LOCAL. Option Retry changes one thread in one block's builder.Guarding test. In
backend/tests/stinger_test.rs, hold the first parked frame back for longer thanCUE_TIMEOUTwith a drop probe ahead of the park clocksync. Then assert that the stinger reportsready: truewith no hand cue. Fails today; costs about 6 s.Recommendation. Option Retry. Operators lose an armed stinger today, and the retry does not block the debug run.
Ask: Option Retry or Option Wait? Reply
/agent-fix Retryor/agent-fix Wait. Other phrasing needs a human.Confidence: MED
srperens commentedon Oct 14, 2026
Findings from tracing on the Windows runner
Four rounds on a temporary
diag/stinger-park-windowsbranch. Each round runsstinger_testrepeatedly onwindows-latestwith no CI skips, keeps the GStreamer logs and dot dumps from failing iterations, and takes thread stacks withcdb. Runs: 38376439472, 38385569894, 38387714329, 38389696473.The park probe is not at fault. The clip never produces a video pad, so the bridge never builds a chain for the probe to sit on.
What a failing load looks like (dot dump and pad dump at the cue timeout):
urisourcebintypefinds the clip,qtdemuxexposesvideo_0, andpngparseand parsebin exposesrc_0. About 266 ms ofimage/pngwaits in urisourcebin's multiqueue.decodebin3'ssink_0. Its inputidentityhas its src pad blocked and unlinked, and decodebin3's own multiqueue has no pads.gst_decodebin_input_unblock_streamsis never logged for that decodebin3; every working one logs it.uridecodebin3, no frame reaches the park clocksync, and the cue times out afterCUE_TIMEOUT(5 s). Nothing retries.It is not a deadlock. At the timeout every GStreamer streaming thread (typefind, qtdemux, multiqueue src) is idle in
gst_task_funcwaiting on its condition variable. Nothing holds a lock another thread waits for, so the unblock looks lost rather than stuck behind a lock. This differs from #963.Where and how often:
a_take_outlived_by_a_restart_leaves_the_new_run_alonein both failures of the last round).stinger_test, which is roughly 1 load in 200. Seen on Windows only, on GStreamer 1.26.10.GST_STATES:5,multiqueue:5and more) it showed one step earlier instead: typefind never completed for 3 of 4 parsebins that started at the same moment (the player's and three clip analyses). That is probably the same family of startup race, but this is not established.Side finding:
a_take_that_fails_to_program_leaves_the_old_source_on_airhas a second, unrelated flake on Windows. A frame 66 ms in is not the old source's colour. It failed 6 of 15 runs under heavy logging and is already skipped on Windows.Next
CUE_TIMEOUT, load the clip again once, with a newuridecodebin3. This also covers operators, whose stinger otherwise stays dead until they reload by hand. After that, rerun the diag loop and drop the Windows skips that then pass.decodebin3:6, to have something to report upstream.fix(media player): hold typefind back when a stinger clip loads paused (