Repository navigation
Conversation
The stall watchdog ends a track that has run dry while another backs up behind it. Until now the track never came back, and the recording stayed without it until the flow stopped. Ending a track now parks its input: it is unlinked from its parser, the chain behind it gets the EOS, and a probe takes its buffers off the stream, keeping everything from the first keyframe on (up to 32 MiB). That keyframe marks a resume, and the watchdog starts a new file. It closes the current one (catching its EOS before the pipeline sees it), takes splitmuxsink to NULL, requests fresh pads for every track carrying data, and relinks them with new parsers and queues. Each track then replays what it kept before live data passes, on its own thread because splitmuxsink can hold one track's queue until another catches up. A new file is the only way back in: splitmuxsink keeps a pad's EOS across splits and state changes, and mp4mux refuses pads once a file has started. The parked input drops the RECONFIGURE that linking the new chain sends upstream. An encoder answers it with an ALLOCATION query that waits in the new queue, which then reads as drained to the watchdog. Past FILE_FINISH_TIMEOUT the switch waits until the file sink is idle before closing the file as it is. Stopping the muxer mid-write makes mp4mux fail the flow. A track that stays quiet, or whose source really ended, stays out of the new file. A real EOS still ends the track and the recording. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…during replay Starting a new file parked every track going over at once. A video track still being recorded was cut mid-GOP, and its part of the new file began at its next keyframe: the frames in between reached neither file. From a remote encoder with a GOP longer than the stall timeout, the new file then waited on that track until the watchdog ended it, and was left empty. Such a track is now parked at its next keyframe, after a key-unit request. The current file keeps it up to there, and the keyframe starts the next file. With no keyframe within KEYFRAME_CUT_TIMEOUT the track is ended instead, since splitmuxsink opens no file without its primary video, and a later file takes it back like any ended track. The stall check also runs while tracks replay into the new file. A replay larger than its queue waits on the other tracks, so one that died during the switch held the watchdog, and the recording, until it came back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e while waiting Parking, the keyframe cut and the stash take a frame without DELTA_UNIT as a keyframe. tsdemux output, which SRT passthrough feeds the recorder, flags no frame as a delta. A new file then started mid-GOP, its parser dropped everything up to the next IDR, and with a GOP over the stall timeout the track was ended into a parser that had output nothing, which fails the flow. Each video track now has an h264parse or h265parse that lives for the whole recording, ahead of a second identity where the recorder parks and cuts it. It marks keyframes and repeats SPS/PPS before each one, and is replaced if the codec changes. That parser drains on EOS, so a track that takes an EOS while parked can now have frames to replay. The parked input refuses to push its caps and segment once it is EOS, so the new chain gets them directly. start_next_file took the pipeline at the start and held it through every wait. A flow stopped during a switch then outlived the drop that the flow's leak check runs right after. It now takes the pipeline only to relink, and only while it is running. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A video input that changes codec is refused by its keyframe parser in the ACCEPT_CAPS query, so the caps event never reaches the probe that was to swap the parser. A codec change fails the flow as it did before; the recorder now handles only the first caps, and the parser is created once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A source that sent a keyframe just past the stall timeout, then nothing, started a new file on every keyframe. Each time the video then stopped, it froze the other tracks again for the stall timeout: 4.5 s of every 6 s, and a file per cycle. A parked input now counts as back only once its data spans 3 s with no gap longer than the stall timeout, so a source slow enough to be recorded before a stall is recorded after one, and one that sends less often stays out. A longer gap starts its stash over, so the new file starts on the first keyframe of the run and the wait loses no video. Once the switch has taken a track for the next file, a gap no longer drops its stash, which the new file starts with, and the run starts again. Frames that arrive while the stash is replayed are kept after it even if the first is not a keyframe, so the replay leaves no hole in the GOP. A full stash drops its oldest GOP instead of everything, and once it has dropped the frames a delta depends on, waits for a keyframe again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Noted as a draft, so I am holding off on a full review until it is marked ready for review. If you want one before then, say so here or request me as a reviewer. |
A stop that landed while start_next_file was relinking found the pipeline still held, tripping the leak check, and relinked a pipeline already going to NULL. Blocks can now register a pre-stop hook, which the pipeline manager runs before setting NULL. The recorder's closes a gate: a relink under way is waited for (10 s at most), and none starts after it. The watchdog and every wait in the switch give up once the gate is closed. Waiting for the relink exposed a splitmuxsink hang: opening a file overwrites the STOPPED state that PAUSED->READY sets, so a queue thread waits forever and setting NULL never returns. The hook sends FLUSH_START into splitmuxsink, which releases that wait. Before GStreamer 1.24.8 a probe that drops an event frees it twice and logs a CRITICAL. The park probe and the switch's EOS catch now discard events with Handled; buffers are still dropped. A replay whose segment push is refused now sends the stream's sticky events to the new chain directly, not only when the input is at EOS: buffers without a segment leave splitmuxsink unable to place them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Before GStreamer 1.24.8 a probe that drops an event frees it twice and logs a CRITICAL. The stand-ins for an encoder that ignores key-unit requests and a source whose EOS never arrives logged CI's remaining ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
srperens
left a comment
srperens
left a comment
There was a problem hiding this comment.
Verdict: Comment. The mechanism is right, the tests drive the real stop path and ran in CI, and the 5 s bound holds. What holds this at Comment is #927. It adds a second "block work before NULL" hook to stop() at the same line as this PR, and which order the two run in decides whether either one works. A maintainer has to choose that.
Requested changes (or a maintainer decision instead)
-
Settle the hook shape with #927 before either PR lands. This PR adds
run_stop_drains()where #927 addsrun_pre_stop_hooks(), both directly afterself.volume_ramps.clear();:- #927's hook sends
FLUSH_STARTinto every splitmuxsink sink pad. If it runs before this drain, the EOS this PR sends into the queue in front of those pads is discarded at a flushing pad. Every recorder stop would then wait out the 5 s and leave the file unfinished, as on main. - If the drain runs first, it can land in #927's relink. The target pad picked at
backend/src/blocks/builtin/recorder.rs:548—.unwrap_or(muxer_pad);comes frompeer()while #927 is replacing those queues.
Only one order works for both: close #927's gate, then drain, then flush. Choose between one ordered hook API, landing this first and having #927 slot its gate and flush around
run_stop_drains, or the reverse. - #927's hook sends
Claims
| Claim | Verdict | Evidence |
|---|---|---|
| A stuck recorder holds the stop at most 5 s | CONFIRMED |
One deadline is shared by every drain (citation below), and each receiver waits only for what remains of it |
| Nothing is sent before splitmuxsink has a file open | CONFIRMED |
file_opened checks only the first track that has a pad, which is the reference track in request order (citation below) |
| The tests exercise the changed code, and they ran | CONFIRMED |
recorder_stop_test builds flows through PipelineManager and calls stop(). Check (Linux) at 4227962 reported 8 passed; 0 ignored, finishing in 7.18 s, which is consistent with real recordings rather than early returns |
| A split in progress is not taken for the end | EXTERNAL |
Assumes splitmuxsink sends a split's EOS straight to the muxer pads, as the body says 1.28's gstsplitmuxsink.c does. CI ran the guarding test on 1.24.2 |
backend/src/gst/pipeline/lifecycle.rs:329 — let deadline = Instant::now() + STOP_DRAIN_TIMEOUT;
backend/src/blocks/builtin/recorder.rs:474 — .is_some_and(|(_, activity)| activity.last_muxed_ms.load(Ordering::Relaxed) != 0)
Lifetime: the EOS threads each hold a strong gst::Pad and nothing else. A pad does not reference its parent element, so a thread still blocked at teardown does not keep the pipeline alive for stop_flow's leak check. The probe on the muxer's src pad is EVENT_DOWNSTREAM and holds the muxer weakly. Drains run only in stop() and not in Drop. That is fine: every flow teardown goes through stop() in teardown_flow.
Radius: SHARED. A new BlockBuildContext API, and stop() now waits for every flow that has a recorder: milliseconds normally, 5 s when storage has failed, and 5 s for every Windows recording stop until #835 is fixed (the body says so).
Overlaps: #927: the same code path and design decision, described above. SPECULATIVE (not verified): file_opened reads pad names stored in a OnceLock, and #927 requests fresh pads on every file switch, so whichever PR lands second should re-check that rule after a switch. #944 also reports file boundaries on the same splitmuxsink.
Tests & CI: all Linux, WASM and API checks are green at 4227962. macOS and Windows were skipped. The Windows ignores are visible and cite #835.
Confidence: HIGH
|
Written by Claude. The stop order is settled on #927 (comment): close the relink gate, then drain, then flush. #927 lands first, and this PR is rebased onto it:
The rebase also covers the overlap you flagged. #927 requests new splitmuxsink pads at each file switch, so This PR doesn't change until #927 merges. |
A flow stop took the pipeline straight to NULL, so the muxer never saw EOS and never wrote its final index. An mp4 was left with an mdat of size 0 and the moov from its last periodic update, seconds short, with its tracks disagreeing; an mkv had no duration. Blocks can now register a stop drain on the build context. stop() runs them after the bus watch is removed and waits, at most 5 s for all of them, before going to NULL. The recorder's drain sends EOS into each track's queue and is done when EOS leaves the muxer with every muxer input ended, which a split's EOS does not satisfy. A recorder that never had data has nothing to drain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
splitmuxsink opens a file on the first buffer of its reference track, the primary video track or, with no video, the first audio track, and holds the other tracks until then. A track's caps unlock it earlier: an x264 encoder with lookahead spends about 250 ms there while audio already arrives. The stop drain's EOS in that window made splitmuxsink open a file just to end it, a 95 MB mp4 with an empty mdat that does not play, where a stop without the drain leaves nothing. The drain now runs only once the reference track has carried data. That replaces the locked-state check, which this case passed. Adds guards for two behaviours no test failed without: EOS goes in behind the buffers queued in front of the muxer, and one quiet track does not hold the other track's EOS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Windows the recorder never opens its file (Eyevinn#835), so the four tests that check a finished recording fail there with no file, or with a stop that waits out the drain timeout. They are ignored on Windows, like recorder_records_when_driven_through_pipeline_start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The recorder's pre-stop hook flushes every splitmuxsink sink pad so that going to NULL cannot deadlock on a parked queue. The stop drain sends EOS into those same pads, and it ran after the flush: the flushing pads refused the EOS, every stop waited out the 5 s drain timeout, and the file was left with an mdat size of 0. A stop now runs gate, drain, flush: - The drain closes the relink gate first, so no file switch is under way or starts while the EOS goes in. A switch relinking has the muxer in NULL, and an EOS sent into it left the new file unfinished. - `stop()` runs the stop drains before the pre-stop hooks. The flush still follows a drain that timed out, and a pipeline dropped without a stop still runs the hook alone. - `RelinkGate::close` waits only on its first call, so a relink that outlasts the drain's wait does not cost the hook a second one. `run_stop_drains` is public so the track-resume rig stops its recorder the way `stop()` does. The relink stop test now also checks that both files are finished; it fails without the gate in the drain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4227962 to
438dac2
Compare
|
Comment by Claude (AI), on behalf of @srperens. Thanks for this. Finishing the file on stop is the right idea, and your measurements (mdat size 0, up to ~2 s short) are what we saw too. The Live Recorder in #1034 does the same thing inside the block. Its sink sends EOS into each track's queue at its PLAYING→PAUSED step when the pipeline is headed for NULL, and waits up to 3 s, so it needs no stop hook in Where we're heading. We're aiming to deprecate the Recorder in favour of a new block, Live Recorder ( We'd really like your help with this:
|
2 participants
Important
This PR can probably be closed. Note added by Claude (Claude Code), on behalf of @wagenet, 2026-10-08.
Live Recorder (
builtin.liverecorder, #1034, merged 2026-10-08) is set to replace this block, and it does not have the problem this PR fixes: a stop sends EOS into its muxer and waits up to 3 s for it to finish, and its fragmented mp4 plays without a final index (a_stop_without_eos_still_writes_the_last_fragment,a_stop_before_a_dataless_track_is_released_still_writes_the_file).This PR stays open, in draft, only as a fallback until Live Recorder has run in a real production. It needs no review in the meantime.
Written by Claude (Claude Code), on behalf of @wagenet.
Depends on #927, and carries its commits until it merges. Review the last four commits, from
fix(recorder): finish the current file when the flow stops.No tracking issue.
Problem
POST /api/flows/:id/stoptakes the pipeline straight to NULL. No EOS reaches a recorder's muxer, so the muxer never writes its final index. Only a split (split-now) finished a file. With robust muxing (reserved-moov-update-period2 s) a stopped mp4 still plays, but:mdatbox has size 0 ("to end of file") and its moov is the last periodic snapshot;Standalone Strom, videotestsrc + audiotestsrc → videoenc/audioenc → recorder, MP4:
_00000_00001Delete after stop leaves the files unchanged. A client that needs finished files has to split, wait for every recorder's
RecorderFileChanged, then stop, which leaves a tiny tail file per recorder. A stopped mkv has no duration.Change
Stop drains (
BlockBuildContext::register_stop_drain). A block registers a closure that starts its work and returns a receiver for "done".PipelineManager::stop()runs every drain after removing the bus watch, waits for all of them together for at mostSTOP_DRAIN_TIMEOUT(5 s), then runs the pre-stop hooks (fix(recorder): record a stalled track again once it resumes #927), and then goes to NULL as before. TheNULL_STATE_TIMEOUTand wedge handling are unchanged.The recorder's drain sends EOS into the queue in front of each splitmuxsink pad, behind any buffers already waiting there. A pad with no queue linked (a connected track that never carried data) gets the EOS directly. Each send runs on its own thread, because a serialized event waits for the pad's stream lock, and the thread holding that lock can be parked inside splitmuxsink until another track's EOS arrives.
Only an open file is drained (
file_opened). splitmuxsink opens a file on the first buffer of its reference track (the primary video track, or with no video the first audio track) and holds the other tracks' data until then. Before that, EOS would make it open a file just to end it: a 95 MB mp4 with an emptymdatthat does not play. Caps unlock the sink before the first frame, and an x264 encoder with lookahead spends ~250 ms there while audio already arrives, so the sink's locked state is not enough.Done is EOS leaving the muxer while every muxer input's peer (splitmuxsink's internal queue) has carried EOS. At a split, splitmuxsink sends its EOS straight to the muxer pads, bypassing those queues, so a split in progress at stop is not mistaken for the end.
Gate, drain, flush with fix(recorder): record a stalled track again once it resumes #927. fix(recorder): record a stalled track again once it resumes #927's pre-stop hook flushes every splitmuxsink sink pad so that going to NULL cannot deadlock. Run before the drain, the flushing pads refuse its EOS, every stop waits out the 5 s, and the file keeps
mdatsize 0. So the drain first closes fix(recorder): record a stalled track again once it resumes #927's relink gate (ending the stall watchdog and waiting out a file switch that is relinking, up to 10 s; past that the drain is skipped), then sends its EOS, and the flush follows a finished or timed-out drain. A pipeline dropped without a stop runs the hook alone.RelinkGate::closewaits only on its first call, so a stuck relink does not cost the hook a second 10 s.Rejected:
PipelineEosas if the flow had ended on its own.Evidence
Each test below fails with the behaviour reverted, run on
20ea75df(27b5fbe1only sets the test encoder tospeed-preset=ultrafast):a_stopped_recording_is_finished_at_its_full_lengthrun_stop_drains()the stop did not finish the file: mdat size Some(0)a_recording_that_cannot_finish_does_not_hold_the_stop(blocks the file sink)stop() did not return within 15sa_recorder_that_never_had_data_does_not_delay_the_stopfile_openedchecka_stop_before_the_first_video_frame_leaves_no_file(video dropped after caps, audio flowing)a recorder stopped before its first video frame left [..._00000.mp4]a_stop_before_the_first_audio_track_has_data_leaves_no_file(track 0 silent, track 1 running)a recorder stopped before its first audio track had data left [..._00000.mp4]a_split_in_progress_is_not_taken_for_the_endmuxer_inputs_endedalways true..._00001.mp4 was not finished: mdat size Nonebuffers_queued_at_the_stop_are_recorded(30 video buffers queued)a_stop_with_one_track_quiet_is_not_held_upstop with one quiet track took 3.04sThe live sources in these tests never send EOS; it comes from the recorder, so source behaviour does not affect the stop.
Real server, macOS, GStreamer 1.28.6, release build, the flow from the table (720p videoenc + AAC, live test sources), on
b1b04073. The reference-track check added since changes nothing for a recording with data.af35c479Drain log:
1 stop drain(s) finished in 2.2ms. Split then stop after 0 / 0.3 / 0.8 / 1.5 / 3 s: both files finished each time, stop 0.03 s. mkv: durationN/Aupstream, 6.06 s here. mkv and mpegts drains finished in ~2 ms.With #927 (
438dac2b), each revert run once: the drain after the flush fails 4 of the 8 stop tests; no drain fails 5 of 8; no gate in the drain fails #927'sa_flow_stopped_while_the_switch_relinks_leaves_nothing_holding_its_pipeline3 of 3, now that it checks both files are finished.only_the_first_close_waits_for_a_relinkcovers the gate. No test guards the flush after a drain timeout.PipelineManager::stop()time with #927, 5 runs each against 1 run of the old order:mdat0mdat0, video 8.03 sTests
Ran on macOS 27 (arm64), GStreamer 1.28.6:
438dac2b:recorder_test30 passed,recorder_stop_test8,recorder_stall_test3,pipeline_lifecycle_test4, librecorderandpipeline::lifecycle17; clippy and fmt clean. The full suite, Linux CI and Windows have not run on this head.STROM_REQUIRE_GST_PLUGINS=1 cargo test --test recorder_stop_teston27b5fbe1: 8 passed, 2 runs, plus 4 runs undertaskpolicy -b(background QoS, close to a slow CI runner). On20ea75df,recorder_testandpipeline_lifecycle_testpassed.cargo test(all backend tests, 1090): passed on27d1c4c2, which differs from20ea75dfonly infile_openedand the audio-only test.cargo clippy --all-targets -- -D warnings,cargo fmt --all: clean.Skipped: none (
STROM_REQUIRE_GST_PLUGINS=1).Linux CI (GStreamer 1.24.2): passed on
27b5fbe1; all 8 new tests ran.Windows (fork run, the
ci:windowslabel needs maintainer access): the recorder never opens its file on Windows (#835, on main), so 4 of the new tests failed there with no recording.4227962ignores those 4 on Windows, asrecorder_records_when_driven_through_pipeline_startalready is.For the reviewer
last_muxed_msat the switch. Replay normally fills the new file within milliseconds.FLOW_EOSback for any buffer after the EOS. Its other branches (a tee into WHEP, say) keep running, and the pipeline goes to NULL right after.muxer_inputs_endedrelies on splitmuxsink closing a split fragment by sending EOS directly to the muxer pads, andfile_openedon it opening a file at the reference track's first buffer. That is how GStreamer 1.28 does it (gstsplitmuxsink.c). If either changes, the split or first-frame test fails.Known limits
Out of scope
tune=zerolatency, where this is negligible (11.97 s video vs 12.05 s audio above).🤖 Generated with Claude Code