Skip to content

fix(mixer): keep an ended input's EOS off a force-live audio bus - #898

Open
wagenet wants to merge 2 commits into
Eyevinn:mainfrom
wagenet:wagenet/mixer-eos-runaway
Open

wagenet wants to merge 2 commits into
Eyevinn:mainfrom
wagenet:wagenet/mixer-eos-runaway

Conversation

@wagenet

@wagenet wagenet commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Opened by Claude (AI) on behalf of @wagenet.

Problem

A force-live audio bus (builtin.mixer buses, builtin.liveaudiorouter outputs) stops following the clock when the only input that ever carried audio sends EOS. It then emits silence as fast as the CPU allows.

audiomixer counts an EOS pad as ready and, with ignore-inactive-pads, skips pads that never had a buffer, so it never waits for the clock.

Real inputs that send EOS:

  • NDI input with default settings: ndisrc returns EOS after its receive timeout (5 s in Strom).
  • SRT inputs with keep_listening off, when the caller leaves.
  • Either of these through an inter-flow link, because intersink forwards EOS by default.

WHIP, media player and AES67 inputs do not send EOS.

Measured through the real blocks, 2.5 s after start with the input ending at 0.4 s:

free-running consumer: output PTS synced consumer: later source first heard
mixer main, before 11 min 14 s 1.04 s after it joined
router out, before 20 min 16 s never (1.5 s window)
both, after 2.42 s 0.09 s

A synced consumer, such as a WHEP viewer, sits behind up to a queue's worth of silence stamped ahead of the clock. Audio from a source that joins later is stamped behind the bus and dropped. A free-running consumer, such as a file writer, gets minutes of silence per second.

Fix

drop_input_eos puts an event probe on each block input: convert_N in the mixer, identity_in_N in the router. On a force-live bus it drops EOS, so the ended input reads as an idle one. The bus already covers an idle input with silence on its latency timeout.

Without force-live, EOS still passes, because that bus is meant to end with its inputs.

A force-live bus still never ends, even after every input has ended. It now emits silence on the clock rather than ahead of it. Keeping the bus up is what force-live is for, so that is left as it is.

Trade-off

With two inputs, one ended and one still playing, the ended input now reads as idle. Each output buffer then waits for the bus's latency timeout, as it already does for an idle WHIP seat. A free-running consumer receives the playing input's audio about 72 ms after its timestamp instead of 3 ms. A synced consumer is unaffected: 92 ms before and after. SRT, RTMP, AES67 and WHEP outputs sync; recorders do not, and are not latency-sensitive.

Tests

The tests are in backend/tests/mixer_input_eos_test.rs and build each block through its own builder.

  • Each block's bus follows the clock after its only input ends.
  • A source that joins afterwards is heard within 400 ms.
  • Without force-live, each block's bus still ends.

Run on GStreamer 1.28.6 (macOS), and 5 times each in Docker on 1.24.2 (the CI version) and 1.26.6 (the Strom image's version):

  • mixer_input_eos_test: 6/6 pass.
  • With the drop reverted, the 4 force-live tests fail.
  • With the force_live gate removed, the non-force-live test for that block fails. Checked for each block separately.
  • liveaudiorouter_test, pipeline_lifecycle_test and cargo test --lib mixer pass.

The test source does not pin a rate. On 1.24 a force-live bus can settle on its rate before any input links, and before #946 a mixer channel had no resampler, so a pinned 48 kHz source failed with not-negotiated in 3 of 5 runs.

CI's plugins-base/good/bad packages cover the elements these tests need.

Adversarial scratch tests (not committed) found the trade-off above and no other regression. Nothing in Strom depends on EOS passing a mixer input: the only pipeline-level consumer is the PipelineEos broadcast, which a force-live bus never produced before either.

Further testing possible

  • A real NDI source: ndisrc returning EOS on its timeout was read in its source, not run (no NDI runtime here).
  • A real SRT sender through a Strom flow. A standalone srtsrc with keep-listening off was confirmed to send EOS when the caller leaves; the tests use a live audiotestsrc with num-buffers instead.
  • An inter-flow link carrying the EOS into a mixer in another flow.
  • The vision mixer's compositors, for a video equivalent. They do not set ignore-inactive-pads.

🤖 Generated with Claude Code

srperens
srperens previously approved these changes Oct 5, 2026

@srperens srperens left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: Approve. This is the right layer: the drop sits on the block's own input, gated on the property that defines "this bus never ends". The new tests drive both real builders and ran in CI.

Claims

Claim Verdict Evidence
The drop is an event probe, not a buffer probe, and holds no element CONFIRMED backend/src/blocks/builtin/mixer/elements.rs:102 — gst::PadProbeType::EVENT_DOWNSTREAM,. The closure captures only a String label, and it is installed once per input at build time
Applied only on a force-live bus, in both blocks CONFIRMED backend/src/blocks/builtin/mixer/builder.rs:745 — drop_input_eos(&pad, format!("Mixer {} channel {}", instance_id, ch_num)); inside if force_live. Router: backend/src/blocks/builtin/liveaudiorouter.rs:496 — crate::blocks::builtin::mixer::drop_input_eos( under the same gate
Nothing else in Strom needs input EOS to cross a mixer CONFIRMED outside the mediaplayer/video-input bridges' private pipelines, the only bus-level EOS consumer is backend/src/gst/pipeline/bus.rs:151 — events.broadcast(StromEvent::PipelineEos { flow_id });. The only in-tree EOS producer is the recorder pushing on its own pad, which is downstream of any mixer. Draft #967 injects its stop EOS at the recorder too, so this does not affect it
audiomixer free-runs once its only active pad is EOS under ignore-inactive-pads EXTERNAL upstream aggregator behaviour. Assumed from the body's measurements; the force-live tests below assert the opposite outcome

Diagnosis: The stated cause explains both symptoms: the output runs ahead of the clock, and a later joiner is stamped behind the bus. Turning the ended input into an idle one reuses the latency-timeout path the bus already has for idle WHIP seats, so no new mechanism is added. Coverage is ABSOLUTE for input EOS on audio buses. The bound is the vision mixer, which the body names and which does not set ignore-inactive-pads. The disclosed cost: with one ended and one playing input, a free-running consumer sees audio ~72 ms late instead of 3 ms. Synced outputs are unchanged.

Radius: SHARED. force_live defaults to true in both blocks (backend/src/blocks/builtin/mixer/builder.rs:136 — let force_live = get_bool_prop(properties, "force_live", true);), so every default mixer and router changes. What changes is only what happens after an input ends: previously the bus raced ahead, now it stays on the clock. A non-force-live bus still ends; the two *_without_force_live_ends_with_its_input tests guard that.

Overlaps: #930 adds per-channel direct outputs downstream of the same convert_N. The textual merge should be easy. Once both land, a direct out on a force-live mixer also never sees an ended input's EOS, which matches "this block does not end". #933 (a consumer's EOS silencing a channel) is the opposite direction on a different pad, so neither PR settles the other.

Tests & CI: Check (Linux) at bbfcf1a: mixer_input_eos_test ran 6 passed in 2.93 s. require_elements() asserts instead of skipping, and each test runs ~2.5 s of real pipeline. pipeline_lifecycle_test passed. I could not attribute the 4 gst_mini_object_unref CRITICALs this new binary printed. liveaudiorouter_test printed none, both in the same job and on main's run today. The first two appear before any force-live test has finished, so they are not obviously the probe. It is worth one look, but it does not change the verdict. The run predates #971 (mixer latency on main). A local trial merge of this head into current main is clean; I did not build it. macOS/Windows skipped: nothing here is platform-specific.

Confidence: MED

wagenet and others added 2 commits October 9, 2026 09:02
A live source that gives up pushes EOS: ndisrc after its receive timeout
(5 s by default), srtsrc when its caller leaves with keep_listening off,
and either through an inter-flow link. audiomixer counts an EOS pad as
ready and, with ignore-inactive-pads, skips pads that never had a buffer.
When the ended input is the only one that ever carried audio, the bus
stops waiting for the clock and emits silence as fast as it can: the
mixer's main bus reached PTS 11 min after 2.5 s, the live audio router
20 min. A synced consumer then sits on up to a queue of silence stamped
ahead, and a source that joins later loses its first second (mixer) or
is not heard at all (router).

Drop EOS at each block input when the bus is force-live, so the ended
input reads as an idle one. Without force-live the bus is meant to end
with its inputs, so EOS still passes there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On GStreamer 1.24 a force-live bus can settle on its rate before any
input links. The mixer channel has no resampler, so a source pinned to
48 kHz failed with not-negotiated in Linux CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wagenet
wagenet force-pushed the wagenet/mixer-eos-runaway branch from bbfcf1a to 654dad6 Compare October 9, 2026 16:13
wagenet added a commit to wagenet/strom that referenced this pull request Oct 9, 2026
wagenet added a commit to wagenet/strom that referenced this pull request Oct 9, 2026
Eyevinn#1036 puts the element on every mixer channel; Eyevinn#898's tests build the
mixer without registering it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@srperens srperens left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: Comment — supersedes my Approve at bbfcf1a, which the push dismissed automatically. The diff still stands: it is a rebase, and the new test file ran green. The verdict moves to Comment only because Check (Linux) is red at this head, in vision mixer GL tests this PR does not touch.

Requested changes

  1. No code change. Rerun the failed job: gh run rerun 37957611576 --failed. A green Check (Linux) makes this approvable as it stands.

What moved since bbfcf1a: commit 2 is identical (git range-diff =). Commit 1 differs in two places. Its context now includes main's answer_latency_for_unfed_input from #1003, and the test switched from chunks_exact to as_chunks.

Claims

Claim Verdict Evidence
The drop is still an event probe, beside #1003's query probe on the same pad CONFIRMED backend/src/blocks/builtin/mixer/elements.rs:144 — gst::PadProbeType::EVENT_DOWNSTREAM, against backend/src/blocks/builtin/mixer/elements.rs:95 — ` pad.add_probe(gst::PadProbeType::QUERY_UPSTREAM,
Applied only on a force-live bus, in both blocks CONFIRMED backend/src/blocks/builtin/mixer/builder.rs:760 — drop_input_eos(&pad, format!("Mixer {} channel {}", instance_id, ch_num)); and backend/src/blocks/builtin/liveaudiorouter.rs:496 — crate::blocks::builtin::mixer::drop_input_eos(, both under if force_live
The new tests ran at this head CONFIRMED Job 113912053710: mixer_input_eos_test 6 passed in 2.92 s, with #1003's probe present
The red check is unrelated to the diff CONFIRMED gpu_i420 and gpu_gbr_10le panic at backend/tests/vision_mixer_input_format/mod.rs:202 — manager.start().expect("start vision mixer pipeline"); (current: Paused, pending: Playing). This PR touches only audio mixer and router code. Both tests passed at #809's and #935's heads on the same day

Diagnosis: unchanged from my review of 2026-10-05. Coverage is ABSOLUTE for input EOS on force-live audio buses. The video mixer is the stated bound.

Radius: SHARED, as before. force_live defaults to true (backend/src/blocks/builtin/mixer/builder.rs:136 — let force_live = get_bool_prop(properties, "force_live", true);), so every default mixer and router is affected.

Overlaps: #930 (direct outputs) and #1036 (voice isolation) both change mixer/builder.rs. Trial merges of this head with each of them are clean. #933 is unchanged from last time: it covers the opposite direction, on another pad.

Tests & CI: run 37957611576 at 654dad6. Both Linux builds, WASM and API Contract are green. Check (Linux) is red, as described above. The gst_mini_object_unref CRITICALs from my last review appear again, 4 of them. As before, I could not attribute them to the probe, and they do not change the verdict. macOS and Windows show skipping, which is fine because the diff has no platform code. I ran nothing locally.

Confidence: HIGH

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants