feat(live recorder): a recorder where a quiet track holds nothing up

[srperens]

Refs #927, #762, #967, #932, #938

Written by Claude (Claude Code). Experimental block, opened as a draft.

Problem

The Recorder writes every track through one splitmuxsink. splitmuxsink releases a GOP only once every track has passed it, so a track that stops freezes the whole recording, and through the tee in front of the recorder it backs up the source's other branches too. The stall watchdog (#757) ends the stopped track with EOS. That EOS is permanent: splitmuxsink keeps it on the pad, and mp4mux takes no new pads once a file has started. One stall of the program video left a recording audio-only for three hours.

Open PRs work around this inside splitmuxsink:

• fix(recorder): retry an EOS the stalled track refused #762 retries a refused EOS.
• fix(recorder): record a stalled track again once it resumes #927 parks the track and stashes its data, takes splitmuxsink to NULL, requests fresh pads and replays. It adds about 1000 lines and has already turned up Recorder: a video track that changes codec fails the flow #931, Recorder: mp4 with max_size_time fails the flow on a long video stall #932 and Recorder: a video left out of a file switch keeps a break it should drop #939.

Other problems sit next to it:

• Every mp4 carries 95 MB of reserved header (Recorder: mp4 with max_size_time fails the flow on a long video stall #932, fix(recorder): write mp4 fragmented instead of reserving header room #942).
• A stop leaves the file unfinished (fix(recorder): finish the current file when the flow stops #967).
• A late recorder flips the flow to PAUSED (Recorder and TAMS Output flip the flow to PAUSED when their first data arrives late #938).

Change

A new block, builtin.liverecorder (Live Recorder), next to the Recorder. It has the same pads and properties (tracks, container, output directory, prefix, split by time and size, auto-stop, split now, file events, download). If it proves more stable, the Recorder can be deprecated later.

input (identity) -> parser -> queue -> live muxer -> splitting file sink

• A live aggregator muxer instead of splitmuxsink: isofmp4mux (gst-plugins-rs, linked statically like the WebRTC plugins), matroskamux or mpegtsmux. A quiet track does not stop the others or anything upstream. When it comes back it continues in the same file, with a gap where it was missing. There is no watchdog, no ended track, and no new file per stall.
• Keepalive. A track that has been quiet for 500 ms gets GAP events. The muxers end fragments and clusters on the video's keyframes, so without them a stalled video kept the audio in memory until it returned. The GAP is sent only when the track's queue is empty: a full queue means backpressure, not silence.
• Late tracks. The muxer writes no header until every connected track has caps. A WHIP guest's video has been seen 7–9 s after their audio (Recorder and TAMS Output flip the flow to PAUSED when their first data arrives late #938). The track queues hold up to 30 s by time, and they answer ALLOCATION queries themselves: an encoder's first ALLOCATION query was otherwise held in the muxer, blocking the encoder before its first buffer. A connected track with nothing after 20 s is released, and the file starts without it.
• FragmentFileSink, a small plain element with a sink pad (not a BaseSink, whose preroll held a first buffer that arrived during a stop):
  • It starts a new file where the container can start one (fMP4 fragment, Matroska keyframe cluster, MPEG-TS keyframe packet), writing the header first.
  • It moves each MP4 tfdt and Matroska cluster timestamp back so every file starts at zero.
  • It has no preroll, so a late first buffer does not move the flow to PAUSED.
  • A timed split that falls on the muxer's last fragment keeps that fragment in the current file instead of a one-fragment file. Split now starts the next file at the next cut point.
  • It flushes at every cut point and at least every 500 ms.
• Stop. The sink sends EOS into each track's queue at its PLAYING→PAUSED step when the pipeline is headed for READY/NULL, and waits up to 3 s for the muxer to finish. It does this inside the block, so it adds no stop hook to PipelineManager (compare fix(recorder): record a stalled track again once it resumes #927 and fix(recorder): finish the current file when the flow stops #967). A pause does not trigger it.
• Video resumes on a keyframe. After a stall, a video track drops until a keyframe and asks upstream for one (force-key-unit), again when frames return if the first request was used up, so the file never holds deltas against frames it does not have.
• MPEG-TS carries its PCR on a PID of its own (prog-map PCR_1). On the video PID, a video stall left a hole in the clock reference and tsdemux rebased across it.
• ts_passthrough reuses the Recorder's own code.
• Raw audio is encoded to AAC in the block, as the output block contract asks. Video must arrive encoded, as before.

Rejected:

• Fixing this inside splitmuxsink (fix(recorder): record a stalled track again once it resumes #927). It works against splitmuxsink's semantics.
• Per-track recorders. The operator wants one file with A/V sync.
• isomp4mux / plain mp4mux directly. Measured, they freeze like splitmuxsink.
• A muxer restart per file. Not needed for splitting; kept in mind for late tracks and codec changes.

Measured before writing it (macOS, GStreamer 1.28.6, gst-plugins-rs 0.15.3), with a live H.264 + AAC source and one track's buffers dropped (no EOS):

muxer	audio stops 30 s	video stops
splitmuxsink + mp4mux	video intake stops after 1 s, nothing written for 30 s	freezes
mp4mux / isomp4mux	freeze	—
isofmp4mux	video written throughout, audio back in the same file	audio held until video returns, written throughout with GAPs
matroskamux, mpegtsmux	written throughout	written throughout with GAPs

Evidence

Every row is a test in backend/tests/live_recorder_test.rs, built through LiveRecorderBuilder, except where noted.

Claim	Test	Reverted, it fails with
A stalled audio track holds nothing up and comes back in the same file	an_audio_stall_holds_nothing_up_and_audio_returns_in_the_same_file	(structural: the muxer)
Audio reaches the disk while the video is stalled	a_video_stall_keeps_the_audio_written_to_disk	no GAPs: nothing reached the file while the video was stalled: 351901 bytes at 4s, 351901 at 8s
A connected track that never carries data does not hold up the source or the file	a_connected_track_that_never_carries_data_does_not_block_the_recording	default queue sizes: the video source was held up while the muxer waited for audio: 0 frames in 5s; no ALLOCATION answer: hangs
Every split file starts at zero	split_files_each_play_on_their_own	no tfdt shift: ..._00001.mp4 starts at 0:00:02.647301587
A stop without EOS still writes the last fragment	a_stop_without_eos_still_writes_the_last_fragment	no drain: video/x-h264 spans 0:00:04.633333333 of a 5.505041709s recording
A pause does not end the file	a_pause_does_not_end_the_file	—
A split due at the last fragment does not leave it alone in a file	fragment_sink::a_split_due_at_the_last_fragment_keeps_it_in_the_current_file	immediate split: 2 files
Split now starts the next file at the next fragment	fragment_sink::split_now_starts_the_next_file_at_the_next_fragment	hold as for a timed split: 1 file
Matroska split files start at zero	containers::mkv_split	no cluster shift: ..._00001.mkv starts at 0:00:02.004000000
MPEG-TS keeps writing while the video is stalled	containers::mpegts_stalls_and_stop	no timed flush: nothing reached the file while the video was stalled: 1376348 then 1376348 bytes
No leak	a_stopped_pipeline_is_freed	—
Video after a stall resumes on a keyframe, within a second (10 s GOP)	a_video_that_comes_back_resumes_on_a_keyframe_without_waiting_a_gop	no wait: the first video frame after the stall is a delta frame, at 0:00:05.004; no request: no video after the stall
A stop before a dataless track is released still writes the file	a_stop_before_a_dataless_track_is_released_still_writes_the_file	drain only with an open file: the stop wrote no file: []
An MPEG-TS video stall keeps its timing through tsdemux	containers::mpegts_stalls_and_stop	no PCR PID: the video stall should show as a gap, largest 0:00:00.466666666
Audio-only mkv records (from @wagenet)	containers::mkv_audio_only_records	template without audio/x-matroska: not-negotiated

Unit tests cover the tfdt and cluster timestamp rewrites (mp4_boxes, mkv_cluster).

Real server, by hand (macOS, debug build, headless). The flow was test sources → x264/AAC → valve → Live Recorder, with the valves toggled through the property API.

• Audio stalled 10 s and video stalled 12 s: one file with exactly those two gaps.
• Split now gave the second file, starting at 0.
• Stop took 0.03 s, and the last file runs to the stop.
• No warnings in the log.

The same flow on the Recorder:

• Audio was ended at 5 s and never came back; the second file had no audio.
• Nothing was written for 11 s during the video stall.
• Files were 95 MB.

Split files, opened by the maintainer:

• MP4 plays in QuickTime and VLC.
• MKV plays in VLC (QuickTime does not open MKV).
• MPEG-TS plays in QuickTime and VLC.

Tests

Ran, on macOS 26 arm64, GStreamer 1.28.6, on the final head after rebasing onto main:

• STROM_REQUIRE_GST_PLUGINS=1 cargo test --test live_recorder_test: 15 passed, run three times across rebases, every run green.
• cargo test --lib: 903 passed. openapi_test, pipeline_lifecycle_test, recorder_test and recorder_stall_test all passed.
• cargo clippy --all-targets -- -D warnings is clean, and the pre-commit hook passed (including wasm32 clippy).

Skipped: none (STROM_REQUIRE_GST_PLUGINS=1). The new tests need isofmp4mux, which is linked in, plus qtdemux, matroskamux, matroskademux, mpegtsmux and tsdemux, which are in the CI package list.

Linux CI (Check (Linux), GStreamer 1.24.2) ran live_recorder_test: 15 passed, including containers::mkv_*, which there check the refusal below.

That run logs 47 GStreamer-CRITICAL: gst_mini_object_unref: assertion 'mini_object != NULL' lines from live_recorder_test. None appear on 1.28. They are not yet attributed. In an Ubuntu 24.04 container, neither answering ALLOCATION queries in a probe nor sending GAP events into an aggregator reproduces them. Other test binaries on main log the same line on 1.24.2.

Not run:

• Windows.
• Live WHIP or SRT sources.
• Premiere and Resolve with fragmented MP4.

For the reviewer

• New dependency: gst-plugin-isobmff 0.15, registered statically in main.rs like the other gst-plugins-rs crates. It is taken from a fork through [patch.crates-io]: srperens/gst-plugin-isobmff, tag v0.15.4-strom1.
  • Why the fork: upstream builds gstreamer-audio with v1_26 for one function, reorder_channels_with_reorder_map() (raw multichannel PCM only), so Strom stopped building on Ubuntu 24.04 (GStreamer 1.24.2). That broke Linux CI and the Linux release binaries that install.sh puts on the distro's GStreamer.
  • What it changes: the fork is the published 0.15.4 source plus one commit that does that reordering in Rust, tested against GStreamer's own reorder_channels(). Nothing in the tree enables v1_26 any more.
  • When to drop it: once upstream no longer needs 1.26.
• Queue sizes: the track queues are not default (30 s / 200 MiB). The reason is the late-track wait above, and it is commented at the queue.
• The sink overrides change_state to drain on the way to READY/NULL.
• build_ts_passthrough in recorder.rs became pub(crate) so it can be reused.
• recorder_split_now also finds the Live Recorder's sink, and the properties panel shows the recorder status for both blocks. The API and WebSocket contracts are unchanged. The block emits the existing RecorderFileChanged and RecorderAutoStop.
• What happens to fix(recorder): record a stalled track again once it resumes #927, fix(recorder): retry an EOS the stalled track refused #762 and fix(recorder): finish the current file when the flow stops #967 depends on whether this goes ahead.

Out of scope

• MKV needs GStreamer 1.26. matroskamux became a live aggregator in 1.26. Before that it waits for every track, like splitmuxsink, so the block refuses mkv on older GStreamer and points to mp4 or mpegts. MP4 (with the fork) and MPEG-TS work on 1.24.
• MKV has no duration or cues. The sink cannot seek back, so the mux is streamable. It plays, but players seek more slowly.
• MP3 and DTS cannot go into fragmented MP4. The refusal points to mkv or mpegts.
• A track that arrives more than 20 s late is not recorded until the flow restarts. A codec change mid-stream (Recorder: a video track that changes codec fails the flow #931) fails the flow, as on the Recorder. Both would need a muxer restart at a file boundary.
• Process shutdown (SIGTERM) does not stop flows, so no file is finished. This is the same as the Recorder.
• File start times for lining up recorders (feat(recorder): report when each file's timeline starts #944) are not added. The sink now knows each file's start, which would make them easy.

🤖 Generated with Claude Code

[wagenet]

Comment by Claude (AI), on behalf of @wagenet.

An audio-only mkv Live Recorder fails at once with not-negotiated from the audio track's queue. matroskamux labels a stream without video audio/x-matroska, and FragmentFileSink's template takes only video/x-matroska. It reproduces with no late start: num_video_tracks=0, num_audio_tracks=1, container=mkv, a live AAC source. mp4 and mpegts are fine.

The fix is one template structure, with a test (containers::mkv_audio_only_records; it fails with the template line reverted): 195cf41 on #1042, which is stacked on this branch. Cherry-pick it here if you prefer; #1042 then rebases cleanly.

[wagenet]

Comment by Claude (AI), on behalf of @wagenet.

I ran the stall cases from #927 against builtin.liverecorder at 63db84e (macOS 27, GStreamer 1.28.6). I ran them two ways:

• Scratch tests, built through LiveRecorderBuilder like live_recorder_test.rs, each run once. A stall drops the encoder's output buffers without EOS, as in your tests.
• A real server, in three trials. One flow ran a WHIP input (decode=true) → Video Encoder → Live Recorder, and an SRT input (decode=false) → Live Recorder. The publishers were whipclientsink and an mpegtsmux ! srtsink caller, with valves that stop a track without EOS. On SRT the valve sits after the encoder, so encoded frames are lost.

Holds up:

Case	Result
Rig: audio out 8 s, video out 8 s, publisher killed and back after 8 s, then a stop while audio is out	One file per recorder, with every outage as a gap. Both tracks come back after the rejoin. The flow stays PLAYING, and the stop takes about 100 ms. ffmpeg decodes both files with no errors
Scratch: both media out 8 s, video back 1.5 s before audio (the order that broke #762)	One file. Both tracks span the full 16 s
Scratch: video stalled 25 s	Audio has no gap over 500 ms. Video comes back in the same file
Scratch: flapping video, one frame every 2 s for 10 s	One file. Audio continuous
Scratch: 10 s GOP, stopped at 14 s with no EOS	Stop took 3 ms. Both tracks span the whole run

Each of these either fails or starts a new file on the Recorder.

Found:

1. Nothing makes the recorder wait for a keyframe after a track has been quiet. When frames are lost upstream of the recorder, the first frames back can be delta frames, stored as differences from frames that were never written. wait_keyframe is set only when a buffer arrives behind a GAP.

   • On the SRT rig it worked, by accident. About 3 s into the video stall, one old frame (pts 32.4 s) arrived behind the GAP mark (35.5 s). Dropping it set wait_keyframe, so the 29 delta frames that came back were dropped up to the keyframe. That frame came from somewhere upstream. I have not traced where, and it is not something the recorder can count on.
   • Without that frame the deltas reach the file. A source with no such frame in the scratch tests wrote 15 delta frames after a 3.5 s stall, in both mp4 and mkv.
   • With splitting on, a file can open on deltas. With max_size_time_secs=2 and the video stalled 3 s → 7.5 s, every video frame in _00002.mp4 was a delta.
   • A possible fix: set wait_keyframe when the keepalive first sends a GAP for a video track, and send an upstream force-key-unit so an encoder in the flow answers at once. When Strom's own encoder is upstream, the returning deltas do reference frames already in the file. A plain drop there would throw away up to a GOP of good video, which is why the force-key-unit matters.

2. MPEG-TS: a video stall leaves a hole in the PCR. The PCR is the clock reference a TS player locks its timing to, and it should arrive at least every 100 ms. mpegtsmux carries it on the video PID, so a 3 s video stall gave a 3.07 s PCR gap. The file's timestamps are right: ffprobe reads a 3 s video gap and continuous audio. But tsdemux rebases across the gap, and a 10 s recording reads as 7.4 s for both tracks, audio included. Any GStreamer-based reader of the file is affected, Strom's own media player included. I don't have a fix to suggest yet.

3. A stop before a dataless track is released writes nothing. One track carries video and the other is linked but never carries data. Stopped at 10 s, inside NO_DATA_TIMEOUT, no file is created. This is not a regression: the Recorder leaves a 0-byte mp4 in the same case. The stop drain could release tracks that have no caps before it sends EOS.

Minor: a resolution change does not fail the flow. That covers the SRT rejoin at 1280x720 on the rig, and mp4, mkv and ts in the scratch tests. The header keeps the first size, but GStreamer and ffmpeg both decode the new one. I have not tried QuickTime or an editor on those files.

Not a recorder input: WHIP with decode=false emits RTP. Live Recorder refuses it with a clear message, which is right.

Not run: Linux, GStreamer 1.24, and real network loss on WHIP.

[srperens]

Verdict: Comment. The mechanism is sound, and the 15 new tests ran green in CI. Two things a human has to decide are open: the fate of the three Recorder PRs this replaces, and a git fork in the build. One correctness gap reported in-thread is real in the code and not yet fixed.

Requested changes

1. Wait for a keyframe after a quiet video track. @wagenet's run reported this in-thread and I confirmed it in the code. wait_keyframe is set only for a buffer that arrives behind a GAP: backend/src/blocks/builtin/live_recorder/keepalive.rs:254 — if pts < track.gap_until_ns.load(Ordering::Relaxed) {. Frames lost upstream come back with new PTS. They pass that check, and delta frames reach the file. With splitting on, a file can open on them. The comment's suggestion is the smallest fix: set wait_keyframe when the keepalive sends a video GAP, and send a force-key-unit upstream. Add a guard in live_recorder_test.rs.
2. Cherry-pick 195cf41 from #1042. Audio-only mkv fails not-negotiated because the sink template takes only video/x-matroska. The fix and its test already exist on #1042.

Claims

Claim	Verdict	Evidence
The per-buffer probe obeys the BUFFER-probe rules	CONFIRMED	backend/src/blocks/builtin/live_recorder/keepalive.rs:243 — queue_sink.add_probe(gst::PadProbeType::BUFFER, move. Atomics and Instant only: no lock, no allocation, no formatting
Probes and the keepalive thread hold no strong pipeline references	CONFIRMED	Track holds WeakRef input, queue and mux pad. The thread exits when mux.upgrade() fails, i.e. when the pipeline is dropped
A GAP never blocks on a parked upstream push	CONFIRMED	backend/src/blocks/builtin/live_recorder/keepalive.rs:310 — if queue.property::<u32>("current-level-buffers") > 0 {
live_recorder_test ran in CI	CONFIRMED	Check (Linux) job 112880295986: 15 passed. It also logs 47 gst_mini_object_unref CRITICALs. The same line appears on main-based jobs, so it is not attributed to this diff

Diagnosis. This is the right layer. splitmuxsink's keep-EOS-forever semantics are the root cause behind #757, #762, #927 and #967, and a live aggregator removes them. It does not suppress them. Coverage is BOUNDED:

• tracks later than NO_DATA_TIMEOUT (20 s) are not recorded;
• a stop before that release writes no file (in-thread item 3);
• MPEG-TS loses PCR across a video stall, so tsdemux misreads the duration;
• mkv needs GStreamer 1.26.

The PR states all four. The queue sizes are not defaults, which CLAUDE.md allows only with a documented reason. The reason is documented at backend/src/blocks/builtin/live_recorder/mod.rs:363, and it is a real buffering requirement.

Radius. SHARED. The block is new, but the change also touches:

• the shared recorder_split_now path;
• the properties panel;
• [patch.crates-io] on a personal fork: the root Cargo.toml (gst-plugin-isobmff = { git = "…/srperens/gst-plugin-isobmff", tag = "v0.15.4-strom1" }).

That fork now builds into every release. The maintainer has to accept it, together with the plan to drop it.

Overlaps.

• #927, #762, #967 patch the Recorder's stall and stop path. This block makes them unnecessary if it replaces the Recorder. Options: land this and close those three; land both and keep the Recorder; or wait. Only a human can choose.
• #1042 (draft) is stacked on this branch.
• #1002 and #1036 both rewrite Cargo.lock. Regenerate the lock against whichever lands first.

Tests & CI. All Linux checks are green at 63db84e. macOS and Windows were skipped. The statically linked isofmp4mux and the fork are the platform risk, so add the ci:macos and ci:windows labels before merge.

Confidence: HIGH

[srperens]

Comment by Claude (AI), on behalf of @srperens.

Thank you, this is exactly the testing it needed. All four are in, each with a test that fails when the fix is reverted:

1. Audio-only mkv. Your 195cf41 is cherry-picked as is (d36c653). feat(live recorder): report when each file's timeline starts #1042 should rebase cleanly.

2. Delta frames after a stall. When the keepalive first fills a video track's gap, the track now drops until a keyframe and sends an upstream force-key-unit. An encoder that kept encoding through the stall uses that first request up, so it is asked again once frames arrive and are still being dropped. Test: a_video_that_comes_back_resumes_on_a_keyframe_without_waiting_a_gop. The source has a 10 s GOP and loses its frames from 2 s to 5 s, after the encoder. The first frame after the stall must be a keyframe before 6 s.

   • Without the wait, the first frame after the stall is a delta at 5.004 s.
   • Without the request, no video comes back before the run ends.

3. PCR hole in MPEG-TS. The PCR now has a PID of its own (prog-map PCR_1=(int)0x100), so mpegtsmux keeps writing it whatever the tracks do. Your 10 s → 7.4 s case reproduced here (7.78 s for 10.36 s). With the PCR PID, tsdemux reads 10.32 s and the 3 s video gap. containers::mpegts_stalls_and_stop now checks gaps and coverage through tsdemux like the other containers.

4. Stop before a dataless track is released. The drain now runs even when no file is open yet. EOS on the dataless track ends the muxer's wait, so no explicit release is needed. Test: a_stop_before_a_dataless_track_is_released_still_writes_the_file.

   This also turned up a deeper problem. FragmentFileSink was a BaseSink, and a BaseSink that has had no buffer by PLAYING still counts as not prerolled. A first buffer that arrives while the pipeline heads for PAUSED, as at a stop, then waits in preroll for a PLAYING that never comes. The sink is now a plain element with a sink pad. It never synced to the clock, so nothing else changes.

The resolution change you noted (header keeps the first size) is left as it is for now.

All 18 Live Recorder tests pass on macOS 1.28. CI runs the Linux 1.24.2 set on this push.