fix(recorder): record a stalled track again once it resumes

[wagenet]

Important

This PR can probably be closed. Note added by Claude (Claude Code), on behalf of @wagenet, 2026-10-08.

Live Recorder (builtin.liverecorder, #1034, merged 2026-10-08) is set to replace this block, and it does not have the problem this PR fixes: a track that stalls does not hold up the others, and comes back in the same file (an_audio_stall_holds_nothing_up_and_audio_returns_in_the_same_file, a_video_that_comes_back_resumes_on_a_keyframe_without_waiting_a_gop). #967 and #762 are stacked on this PR and carry the same note.

This PR stays open, in draft, only as a fallback until Live Recorder has run in a real production. It needs no review in the meantime.

Description

Written by Claude.

A recorder track that stalls and then resumes is now recorded again, in a new file, without restarting the flow.

The stall watchdog ends a track that has run dry while another backs up behind it, so one dead input cannot freeze the recording. Until now that was permanent. On a live rig the program video stalled for about 7 s, the watchdog ended it, and the recording stayed audio-only for three hours.

Why a new file

The current file cannot take the track back. splitmuxsink copies a pad's EOS into every later fragment and keeps it across state changes, mp4mux refuses request pads once a file has started, GAP events do not release splitmuxsink's wait, and repeated-frame filler is per codec. A new file works for every track and container, at one extra file per recovery.

What changed

• Each video track has a parser ahead of the point where it is parked. Parking, the keyframe cut and the stash read keyframes from DELTA_UNIT, which tsdemux output (SRT passthrough) never sets. An h264parse or h265parse that lives for the whole recording marks keyframes and repeats SPS/PPS before each one.
• Ending a track parks its input. An IDLE probe unlinks the input and sends the EOS into its chain, so splitmuxsink's pad ends as before while the input itself never carries EOS. While parked, a probe drops buffers (upstream sees no flow error; the tee also feeds the vision mixer) and keeps everything from the first keyframe of the current run of data.
• A track is back once its run spans 3 s, with no gap longer than the 5 s stall timeout. A longer gap starts the kept buffers over. So a source that sends a frame now and then stays out, instead of starting a new file and freezing the other tracks again each time, while a source slow enough to be recorded before a stall is recorded after one.
• On a resume, start_next_file switches files:
  1. It parks the recording tracks, which closes the current file, and catches that file's EOS before the pipeline sees it. A video track still being recorded is parked at its next keyframe, after a key-unit request, so neither file loses frames. With no keyframe within 15 s it is ended instead: splitmuxsink opens no file without its primary video.
  2. It takes splitmuxsink to NULL and requests fresh pads, since only a fresh pad clears splitmuxsink's per-pad EOS record, then relinks every track carrying data through a new parser and queue and restarts splitmuxsink. start-index keeps the _%05d numbering.
  3. Each track replays what it kept, on its own thread, then goes live. The new file starts on the run's first keyframe, so the 3 s wait loses no video, even from a remote encoder whose GOP is longer than the stall timeout. From the moment the switch takes a track, a gap no longer drops what it kept, and frames arriving during the replay follow it. The stall check keeps running, so a track that dies during the switch is still ended.
• Kept buffers are capped at 32 MiB per track; a full stash drops its oldest GOP. The park probe takes a lock and can allocate, an exception to the buffer-probe rule that applies only while a track is out. The keyframe cut is a buffer probe present only during a switch.
• A track whose input refuses to push its caps and segment, as one that took an EOS while parked does, gets them straight into the new chain. Buffers without a segment leave splitmuxsink unable to place them.
• A flow stop waits out a relink. Blocks can register a pre-stop hook, which the pipeline manager runs before setting the pipeline to NULL, both in stop() and on drop. The recorder's hook closes a gate: a relink under way is waited for (10 s at most, then the stop goes ahead with a warning), and none starts after it. The watchdog and every wait in the switch give up once the gate is closed, so a stop during a wait logs no timeout warnings. The switch still holds the pipeline only while relinking, and the stop now waits for it to let go.
• The hook then sends FLUSH_START into splitmuxsink. Opening a file overwrites the STOPPED state that PAUSED→READY sets, so a queue thread inside splitmuxsink can go back to waiting after the state change and setting NULL never returns. A flush releases that wait. In production stop() gives up after its 30 s deadline rather than hanging; the new stop-during-relink test, which sets NULL directly, hung on it.
• Probes discard events with Handled rather than Drop: before GStreamer 1.24.8 (CI has 1.24.2) a dropped event is freed twice and logs a CRITICAL. That was the park probe's RECONFIGURE and the switch's EOS catch, seven per switch, plus two test stand-ins (an encoder ignoring key-unit requests, a source whose EOS is swallowed). Buffers are still dropped.
• The parked input drops the RECONFIGURE that relinking sends upstream, so an encoder does not answer it with an ALLOCATION query that sits in the new queue and makes it read as drained to the watchdog.
• If the old file is not written out within 10 s, the switch waits until the file sink is idle before closing it, since stopping mp4mux mid-write posts an error.
• A track still quiet, or whose source really sent EOS, stays out of the new file.

Costs and gaps

• Each stall still freezes the other tracks for the 5 s it takes to detect. On main that happens once, since the track never comes back. Here a source that returns for 3 s or more and drops out again costs it once per dropout, and a new file each time.
• A returning track reaches the file 3 s after it is back, or at its first keyframe if that comes later, and the file starts on that keyframe. Above about 70 Mbit/s the 32 MiB cap is reached during that wait, and the new file starts a GOP or two later.
• A switch waits up to 15 s for the recorded video's keyframe. A keyframe inside a buffer list is not split out.
• A video that comes back in another codec still fails the flow, as on main (Recorder: a video track that changes codec fails the flow #931). With max_size_time_secs set, an mp4 recording still fails the flow on a video stall longer than about one split, as on main (Recorder: mp4 with max_size_time fails the flow on a long video stall #932).
• A video left out of a switch for want of a keyframe, which then breaks for over 5 s during that switch, carries the break into the file that takes it back (Recorder: a video left out of a file switch keeps a break it should drop #939).
• The segment fallback for a refused push is not guarded: the refusal without EOS was seen once in about 20 runs of one test, as a hang at stop, and not reproduced since.
• async-handling is untested; live pipelines ignore a child's ASYNC_START. A connected track whose first caps arrive during a restart could reactivate splitmuxsink early.
• Not guarded, each passing every test when reverted: replaying tracks on separate threads (replaying audio first in turn hangs the slow-storage test, but the recorder replays video first); dropping RECONFIGURE (its only test was a track back for one frame, which no longer resumes); parking a track once when the keyframe cut and its fallback both fire; the stall check skipping a track still replaying.
• Not run on the live rig.
• fix(recorder): retry an EOS the stalled track refused #762 also changes end_stalled_track. This should land first; fix(recorder): retry an EOS the stalled track refused #762 then needs a rebase, since its test file moved in test: consolidate the integration tests and share their helpers #912.

Related Issue

None. Found on a live rig recording.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update

Checklist

• I have run cargo fmt --all
• I have run cargo clippy --workspace --all-targets -- -D warnings
• I have run cargo test --workspace
• I have updated documentation as needed
• I have added tests for new functionality

Tests

Ran locally on macOS with GStreamer 1.28.6:

• A track_resume module in backend/tests/recorder_test.rs stalls and resumes real x264enc and avenc_aac sources through RecorderBuilder: video and audio resumes, two stalls in a row, a 10 s GOP that ignores key-unit requests (returning and still recording), a track back for one frame, a keyframe every 6 s, a frame every 4 s, a track that stops again as the new file takes it, a 5.5 s break while the switch closes the old file, a track that dies during the switch, a video that stops before the keyframe the switch waits for, storage blocked for 14 s, video with no keyframe flags (returning, cut, and no loss at a short GOP), a stop during a switch, a stop while the switch relinks, a stop while a track is out, a real EOS, and an EOS during the switch. After merging main: recorder_test 30/30, recorder_stall_test 3/3, pipeline_lifecycle_test 4/4.

• Linux, in an Ubuntu 24.04 container with CI's package list (GStreamer 1.24.2): recorder_test 30/30 and recorder_stall_test 3/3, with no CRITICALs. At 125e7c1 the same run logged 131, matching CI's 130.

• Unit tests: the stop gate (a stop waits for a relink and none starts after it; a stop gives up on a relink that never ends), plan_next_file, the stash's keyframe, overflow, run and replay rules, and the parked input answering OK upstream.

• Revert checks, each failing its test:

  Reverted	Result
  never starting a new file	eleven resume tests fail
  the 3 s run	the one-frame and keyframe-every-6-s tests start new files
  the run measured to now instead of to its last frame	the one-frame test starts a new file
  restarting kept buffers after a gap	the keyframe-every-6-s test starts new files
  a 3 s gap limit instead of the stall timeout	the frame-every-4-s test never resumes
  dropping kept buffers after a gap once the switch has the track	the 5.5 s break test loses the video
  clearing a full stash instead of its oldest GOP; dropping a frame that arrives during the replay; keeping a run's start across a forgiven gap	unit tests fail
  the EOS catch	early EOS
  a real EOS treated as a resume	the recording loops into empty files
  the EOS pushed into a parked chain	the stop-while-out test hangs
  the parked-EOS forwarding	the EOS-during-switch test fails
  discarding the stash instead of replaying it	the two-stalls, stops-again and stops-before-keyframe tests fail
  the idle check before closing a late file	the slow-storage test fails with mp4mux's error
  parking the recorded video at once	the long-GOP recorded-track test fails
  the stall check during replay	the dies-during-switch test fails
  ending a video with no keyframe in time	the stops-before-keyframe test fails, as with no time limit
  no keyframe parser	the three no-flags tests fail
  holding the pipeline through the switch	the stop-during-switch test fails
  the caps sent straight to a chain after an EOS	the EOS-during-switch test fails with a flow error
  the stop gate (closing it returns at once)	the stop-while-relinking test fails the leak check, 4/4
  the FLUSH_START at stop	the stop-while-relinking test hangs, 3/6

• cargo test --workspace: 1335 passed, 0 failed, 8 ignored.

• Adversarial scratch tests that held: resumes in mpegts and mkv, three tracks with one finished, two video tracks, five stalls in a row, a resolution change while out, real tsdemux output, a real EOS during the keyframe cut, a source flapping 4 s on and 6 s off, a stop during a blocked replay, and a track ended during the replay that comes back later. The codec change and splitting cases above fail as on main.

🤖 Generated with Claude Code

[srperens]

Noted as a draft, so I am holding off on a full review until it is marked ready for review. If you want one before then, say so here or request me as a reviewer.

[srperens]

Verdict: Request changes. Recovering in a new file is the right approach; the body rules out each in-file alternative. The healthy-path probes keep to CLAUDE.md, and all 32 recorder tests ran in CI. But one claim in the body fails for a stop that lands in step 3 of the switch, and the new tests multiply a GStreamer CRITICAL by about 130 that nobody has attributed.

Requested changes

1. Close the step-3 window, or narrow the claim and guard it. In start_next_file, backend/src/blocks/builtin/recorder.rs:1106 — let Some(bin) = splitmuxsink takes the pipeline after a single state check, backend/src/blocks/builtin/recorder.rs:1109 — .filter(|b| b.current_state() >= gst::State::Paused). It holds the pipeline until backend/src/blocks/builtin/recorder.rs:1217 — drop(bin);, across splitmuxsink's NULL and the relinking. A stop_flow whose drop lands in that window fails its leak check, backend/src/state/mod.rs:1526 — let pipeline_survived = pipeline_weak.upgrade().is_some();. That logs the ERROR CLAUDE.md treats as P0, and the relink then runs against a pipeline that is going to NULL. Either make stop wait for an in-flight switch before the drop, or state the window and add a test that stops inside step 3.
2. Attribute the CRITICALs. Check (Linux) at 125e7c1 prints gst_mini_object_unref: assertion 'mini_object != NULL' failed 130 times in recorder_test, mostly in bursts of 7–8 while track_resume runs. The same binary on #944, which is based on main, prints it once. Say whether these come from the switch or from the test sources. If they come from the switch, fix them.

Claims

Claim	Verdict	Evidence
A flow stopped mid-switch is not kept alive past the leak check	CONTRADICTED	Holds during the waits. Fails in step 3 (item 1). The guarding test covers only the keyframe wait: backend/tests/recorder_test.rs:2519 — /// reports a leak if anything still holds it. A stop while a switch waits for
The healthy path adds no locking probe	CONFIRMED	The only permanent BUFFER probes are the retired-input probe and the intake probe, and both use atomics only. The park, cut and write-tracking probes exist only while a track is out or a switch runs
The new closures hold no strong element references	CONFIRMED	The caps probes, the keyframe park and WatchedTrack hold WeakRefs. The replay threads are scoped and joined before start_next_file returns
The resume tests ran	CONFIRMED	Check (Linux), run 36870108266: test result: ok. 32 passed, 183 s, under STROM_REQUIRE_GST_PLUGINS: 1

Lifetime: the watchdog thread lives as long as the flow. It holds a strong splitmuxsink for a whole switch, from backend/src/blocks/builtin/recorder.rs:1391 — let Some(sink) = splitmuxsink.upgrade() else {; that can be 15 s or more, but it does not keep the pipeline alive. A stop during a wait makes the next two warnings report timeouts that never happened.

Radius: LOCAL. One block. Every recorder flow with two or more tracks runs the new watchdog paths.

Overlaps: #762 also changes end_stalled_track, and the body asks to land first. Both conflict textually. #944 assumes that only the first file can start before its reference track. A switch here restarts splitmuxsink, which breaks that assumption, so whichever lands second must treat every restart as a first file (see my review there). #967 changes the stop path that item 1 touches.

Tests & CI: all checks green at 125e7c1, which is 25 commits behind main. macOS and Windows were skipped, and nothing here is platform-specific. recorder.rs is now 2927 lines. CLAUDE.md asks for a split above 1500 lines, best done after this and #944 land.

Confidence: MED

[wagenet]

Written by Claude.

Both addressed, and main is merged in.

1. A stop during step 3. Stop now waits for an in-flight switch. Blocks can register a pre-stop hook, which the pipeline manager runs before setting NULL (in stop() and on drop). The recorder's hook closes a gate that the relink holds from taking bin until splitmuxsink is reactivated: a relink under way is waited for (10 s at most), and none starts after it. Every wait in the switch and the watchdog loop return once the gate is closed, so a stop mid-wait no longer logs timeout warnings either.

Waiting for the relink surfaced a splitmuxsink hang: opening a file overwrites the STOPPED state that PAUSED→READY sets, so a queue thread goes back to waiting and setting NULL never returns. The hook sends FLUSH_START into splitmuxsink, which releases it.

New test, a_flow_stopped_while_the_switch_relinks_leaves_nothing_holding_its_pipeline, holds the relink open from pad-removed and stops on another thread. With the gate disabled it fails the leak check 4/4; without the flush it hangs 3/6; with both it passes every run.

2. The CRITICALs. All of them were probes returning Drop for an event, which GStreamer frees twice before 1.24.8. Seven per switch came from the switch itself (the park probe's RECONFIGURE, the EOS catch); the rest from two test stand-ins, including main's one in recorder_stall_test. Those now return Handled. In an Ubuntu 24.04 container with CI's packages: 131 before, 0 after, recorder_test 30/30 and recorder_stall_test 3/3.

Also changed: a replay whose segment push is refused for any reason now sends the sticky events to the new chain directly, not only after EOS. I saw the refusal once without EOS, as a hang at stop, and could not reproduce it, so that path has no guard. The PR body covers it.

[srperens]

Supersedes my review of 2026-10-05 at 125e7c1. That Request changes verdict does not stand: both requested changes are done.

Verdict: Comment. The stop gate closes the step-3 window, and the CRITICALs are gone from recorder_test. But #967 adds a second "block work before NULL" hook at the same line of stop(). Merged in the wrong order, the two cancel each other out, so a maintainer has to choose the shape.

Requested changes (or a maintainer decision instead)

1. Agree one ordered stop hook with #967 before either PR lands. This PR adds self.run_pre_stop_hooks(); where #967 adds self.run_stop_drains();, directly after self.volume_ramps.clear(); in backend/src/gst/pipeline/lifecycle.rs. That is a textual conflict, and the order you resolve it in changes behaviour:

   • Flush first, then drain. The hook's FLUSH_START puts splitmuxsink's sink pads into flushing. #967's EOS, which it sends into the queue in front of each pad, is then discarded. Every recorder stop waits out #967's 5 s timeout and the file is still unfinished, which is the bug #967 fixes.
   • Drain first, then gate. A drain can run during a relink, and #967 aims its EOS at whatever queue peer() finds while this PR is replacing those queues.

   The order that works for both is: close the gate, then drain, then flush. Neither PR can express that on its own. Option: land this first and give #967's drain a slot between the gate and the flush. Or land #967 first and rebase this onto register_stop_drain. Or wait, and merge the two APIs into one.

Claims

Claim	Verdict	Evidence
A stop no longer lands inside the relink	CONFIRMED	bin is taken only after the gate is entered and released before the gate is left (citations below). The gate's SeqCst store-then-load on each side rules out both sides missing each other
The new tests run in CI and guard the stop window	CONFIRMED	Check (Linux) at e5c4ab3: recorder_test reported 30 passed in 173 s, including a_flow_stopped_while_the_switch_relinks_leaves_nothing_holding_its_pipeline. The two RelinkGate unit tests and pipeline_lifecycle_test also ran
The CRITICALs came from the switch, and Handled removes them	CONFIRMED	The same job printed gst_mini_object_unref 0 times in recorder_test, against 130 at 125e7c1. 125 of the 129 in this job are in output_refusal_test, which this diff does not touch. I have not attributed those 125
FLUSH_START releases a splitmuxsink queue that NULL cannot	EXTERNAL	Depends on splitmuxsink internals. The assumption is the author's observed hang without it.

Lines on this head:
backend/src/blocks/builtin/recorder.rs:1220 — let Some(relinking) = gate.enter() else {
backend/src/blocks/builtin/recorder.rs:1334 — drop(bin);
backend/src/blocks/builtin/recorder.rs:1341 — drop(relinking);
backend/src/blocks/builtin/recorder.rs:2318 — pad.send_event(gst::event::FlushStart::new());
backend/src/gst/pipeline/lifecycle.rs:274 — self.run_pre_stop_hooks();

Radius: SHARED. This PR now adds a BlockBuildContext API and a step to PipelineManager::stop() and Drop. Both are no-ops for flows without a recorder, because the hook list is empty. PreStopFn is FnOnce, and a flow restart builds a new PipelineManager (start_flow), so a closed gate never outlives its flow.

Overlaps: #967 (the same code path, and the same design decision, described above). #944: the restart-as-first-file conflict from my previous review still stands. #762: this PR's body still asks to land first.

Tests & CI: Linux, WASM and API checks are green at e5c4ab3. macOS and Windows were skipped, and nothing here is platform-specific.

Confidence: HIGH

Superseded by my re-review at e5c4ab3: both requested changes were addressed.

[wagenet]

Written by Claude.

Agreed on the order: close the gate, then drain, then flush. The plan is to land this PR first and resolve the order in #967's rebase:

• The recorder's pre-stop closure splits in two. Closing the gate stays a pre-stop hook. The FLUSH_START moves to a new hook that runs after the drains.
• stop() runs pre-stop hooks, then run_stop_drains(), then the post-drain hooks. Drop runs the same steps without the drain.
• fix(recorder): finish the current file when the flow stops #967's drain stays its own API rather than going into the recorder's closure. That keeps every block's drain in parallel under one shared 5 s deadline. With one closure per recorder, recorders would drain one after another.

This PR does not change. The rebase also needs a test for a stop that comes after a switch while one track is still out. A retired track may never carry EOS in the current file, so the drain's completion check could wait out its timeout.

[srperens]

Comment by Claude (AI), on behalf of @srperens.

Thank you for this, and for the adversarial tests behind it: #931, #932 and #939 came out of them, and they shaped what we built next. The diagnosis is right. The trouble is splitmuxsink itself: it waits for every pad at each GOP, and it keeps a pad's EOS for good. A recovery built around it has to take splitmuxsink apart to bring a track back.

In #1034 a stalled track comes back in the same file, with a gap where it was missing. We measured splitmuxsink freeze the video intake for an entire 30 s audio stall, where the live muxers kept writing. #932 also goes away, because nothing reserves header room.

Where we're heading. We're aiming to deprecate the Recorder in favour of a new block, Live Recorder (builtin.liverecorder), in #1034. It replaces splitmuxsink with a live aggregator muxer (isofmp4mux, matroskamux or mpegtsmux) and a small sink of our own that does the splitting. A track that stops then holds nothing up, and when it comes back it continues in the same file. There is no watchdog, no ended track, and no new file per stall. Both blocks will exist side by side until the new one has run in real productions. The Recorder stays as it is in the meantime.

We'd really like your help with this:

• Test it on the rigs where you found these bugs (WHIP guests, SRT inputs, per-seat and program recordings). Swap builtin.recorder for builtin.liverecorder; the pads and properties are the same. It builds and runs on GStreamer 1.24 and later, like the rest of Strom.
• Tell us what you think of the approach, and what it still lacks for your use.
• Review feat(live recorder): a recorder where a quiet track holds nothing up #1034. You know the failure cases better than anyone, and its "Out of scope" section lists the gaps we know about.

[wagenet]

Comment by Claude (AI), on behalf of @wagenet.

Closing. Live Recorder (builtin.liverecorder, #1034) does not have the problem this PR fixes: a stalled track leaves a gap and returns in the same file, so nothing needs a second file or a splitmuxsink teardown. The branch wagenet/recorder-track-resume stays as it is, in case Live Recorder fails in a real production and this approach is needed again.

Thanks @srperens for the pointer to #1034 and for the earlier reviews.