feat(vision-mixer): stinger transitions

[wagenet]

Problem

The vision mixer has no stinger: a keyed clip played over the program while the cut or transition happens beneath it, at a point in the clip chosen so the artwork covers the switch.

Change

A stinger is triggered through the existing transition endpoint as type stinger, naming a Media Player block that declares itself a stinger clip source and is wired to a keyed (DSK) input. Cut point, the transition beneath, and its duration are Media Player properties, since where a clip covers follows from the artwork; a take names only its clip. The clip is held on its first frame while idle, so a take only starts playback: 0.5 ms at 4K against 28.4 ms unparked. Declaration is opt-in, so a looping graphic on a keyed input is never parked or unlooped.

The cut is anchored to the clip, not timed. A one-shot probe holds the mixer's output on the frame before the clip's cut point until the take is applied, however far the output trails the clock. A wall-clock cut lands on whichever frame the mixer is on, which on a loaded machine is often before the clip is on air.

A cut the clip may not cover is reported as StingerFailed with still_running: true: the clip frame is late at the cut; the clip never reaches the mixer by cut point + 500 ms (the keyed input is taken off air and the cut made plainly); or the mixer output stalls for 5 s (the cut is made on wall clock).

A finished stinger releases the mixer only after the mixer's output passes the clip's last frame, because videoaggregator keeps showing an expired frame while the next take's first frame is queued. A take's mixer claim carries a token, so a take from a stopped flow cannot drive its replacement.

A declared source plays unpaced, and its video output waits instead of dropping. It ignores the Media Player's sync and playout_delay_ms, so the clip runs ahead of the mixer. Its video appsrc is non-leaky and blocking, so when the bridge's queues are full the clip waits rather than losing frames. Audio stays leaky: an audio output may be left unlinked, and a blocking audio appsrc that never drains stalled the player, and with it the next take's re-arm, for good.

A waiting push parks the Media Player's internal streaming thread, and pausing, seeking, switching file and shutting the player down all wait for that thread. So each first lifts the waiting appsrc's limits (changing max-bytes wakes the push) and the last overlapping call to return restores them: a take re-arms its clip from its own task while an operator may seek or stop it. Without this, a flow whose mixer stopped pulling the clip, or whose declared clip was played before its video was wired, could not be stopped.

Frames reaching the mixer, and how far the newest clip frame reaches past the cut frame at the cut check (5 takes each, macOS):

Feed	320x180	1080p	4K	margin
Paced, as #945 made every Media Player	all frames			-26 to +2 ms (24 takes); cut and repeat tests fail every run
Unpaced, leaky appsrc	all frames	14-16 of 45	8-11 of 30	erratic; most HD cuts reported
Unpaced, blocking video appsrc (this PR)	all frames	all	all	133 to 165 ms

Rejected: keep #945's pacing and give the clip a fixed head start

This PR opts stinger sources out of #945's pacing. The alternative stamps a paced clip a fixed lead ahead of its release. At 100 ms in a scratch build every frame arrived at all three sizes, with 72 to 166 ms at the check, but:

• the graphic goes on air one lead after the take: a full take took 1762 ms at 1080p against 1662 ms unpaced;
• the lead is a number to tune: large enough for a loaded machine, small enough that the frames held ahead fit the bridge's queues at 4K;
• it changes the timing module every Media Player uses, live streams included;
• that naive version cut 2 frames early on 2-3 of 12 takes; not investigated.

The unpaced route reuses the existing sync = false path, keeps the appsrc memory bound, and has no lead to tune. Switching to the paced route would touch only the Media Player builder and timing module.

Evidence

• Unpaced stinger source: reverting cc831ee's builder change (paced, with playout delay) fails the_cut_lands_on_the_frame_the_cut_point_names (take 1 landed on clip frame 1) and a_repeat_stinger_plays_forward_from_the_start ("coverage was [3, 3, 3, 3, 6, 10, …]"), 3 of 3 runs.
• Blocking video appsrc: making it leaky again fails a_clip_longer_than_the_output_queue_plays_whole: "every clip frame must reach the mixer", 160 of 240.
• Releasing a waiting clip: skipping the lift fails a_clip_the_mixer_stops_taking_can_still_be_stopped ("stopping the player must return while the mixer takes nothing") and an_unwired_stinger_clip_can_still_be_stopped ("seeking a clip nothing takes must return"). Both run on a runtime that is abandoned rather than joined, so a hung take fails instead of never finishing.
• Overlapping calls: letting each call save and restore the limits for itself fails overlapping_player_calls_keep_the_clip_queue_bounded ("the clip's video queue limits must be back as built"), 3 of 3 runs. Unguarded, the queue stayed unbounded: 630 MB for a 3 s 1080p take.
• Cut anchored to the mixer: replacing the anchor with a wall-clock sleep fails the_cut_lands_on_the_frame_the_cut_point_names (10 of 12 takes on clip frame 16, not 15).
• End-of-clip wait: skipping wait_for_mixer_to_pass fails a_repeat_stinger_plays_forward_from_the_start ("coverage was [100, 3, 6, …]": the previous clip's last frame on air).
• Opt-in: arming every Media Player fails undeclared_source_on_keyed_input_is_left_alone.
• Stale take leaves a restarted flow alone: a_stinger_outliving_its_flow_leaves_the_next_one_alone guards the claim released on stop. It still passes with the token check in run_take removed, so the token check itself is not guarded.
• Late-clip-frame report: not guarded by a deterministic test. Forcing it would need a test-only delay in the clip path.
• Scratch runs, not committed: 20 back-to-back takes each delivered every frame; a 10 s clip with an audio track delivered every frame with audio_out unlinked and with it on a synced sink; stopping a flow mid-clip at 1080p returned in 21-23 ms, 5 of 5; a 45-frame 1080p clip delivered 45 of 45 on 3 takes.

Tests

Linux CI will likely fail a_clip_longer_than_the_output_queue_plays_whole with 239 of 240 frames, in about half of runs (10 of 21 so far). The cause is a bug in CI's GStreamer 1.24.2, not in this change. avdec_ffv1 with frame threading drops a clip's last frame at end of stream when its output lags its input, and logs Couldn't get codec frame !. Upstream fixed it in 1.24.4, so production (1.26.6) and Windows and macOS CI are not affected. It reproduces on Ubuntu 24.04 with a plain gst-launch-1.0 pipeline, without Strom. This PR makes the lag more likely, because the stinger clip's output now waits for room instead of dropping frames. The test stays strict instead of excusing a missing last frame on old GStreamer; #964 proposes running Linux tests on 1.26.

Ran at 743c8b2, macOS, STROM_REQUIRE_GL=1 STROM_REQUIRE_GST_PLUGINS=1:

• cargo test --test stinger_test: 16/16 in 2 runs. On Linux, Check (Linux) passed at 743c8b2.
• cargo test --lib mediaplayer 30/30; cargo test --test pipeline_lifecycle_test 4/4.
• cargo test --test stinger_alpha_test 4/4, including alpha_survives_gl_videomixer with a real GL context; cargo test --test openapi_test.
• cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --check.

Skipped in CI: alpha_survives_gl_videomixer returns early in Check (Linux), which sets no STROM_REQUIRE_GL (the alpha binary finished in 0.07 s), and Build (macOS) is label-gated. The GL alpha half has run only locally.

Not run locally: cargo test --workspace.

For the reviewer

• Whether stinger sources should be exempt from feat(mediaplayer): play live streams, extra tracks and URLs #945's pacing (above).
• One segfault in the compositor's blend (blend_pads, ORC-generated code) appeared in about 80 heavy-load runs across both arms of an A/B, and not in 40 targeted runs. Not attributed to this change.

Out of scope

• Straight alpha only: premultiplied alpha composites about 40% too dark and cannot be detected from the file, so a binding that declares it is refused. No fill+key pairs.
• The stinger does not route the clip's audio. A clip's audio plays out of the Media Player's audio output if that is wired.
• One stinger source wired to two mixers (say program and a clean feed): the claim is per mixer, so a second take on the other mixer is accepted, and the first take's re-arm rewinds the shared clip under it.
• The take works out the clip's end one frame early: it takes the clip's timing from frame 1, because frame 0 is already waiting at the mixer before the take. The keyed input still comes off after the real end (the mixer was 1.6 to 2.8 frames past it at StingerCompleted in 11 CI runs), so nothing on air changes. A fix would change when every take ends, so it is left for a follow-up.
• A take, and StingerCompleted, run as late as the mixer's output does. On a machine where the mixer ran at about a quarter of real time, completion came several seconds after the clip ended on the clock.

🤖 Generated with Claude Code

[srperens]

Noted as a draft, so I am holding off on a full review until it is marked ready for
review. If you want one before then, say so here or request me as a reviewer.

[wagenet]

Written by Claude (AI).

Update this PR once #809 lands. #809 adds a per-DSK dsk_N_alpha_mode property on the vision mixer that composites premultiplied input correctly on both backends. The stinger clip reaches the mixer through the same dsk_in_N input, so the refusal here is no longer needed:

• Remove ALPHA_MODE_PROPERTY / ALPHA_MODE_PREMULTIPLIED and the PremultipliedUnsupported check in backend/src/gst/stinger.rs, along with the test that asserts the rejection. The source-side alpha_mode property is not exposed in the Media Player block's UI anyway.
• Update docs/STINGER_TRANSITIONS.md (lines 54–60 and 75): a premultiplied clip is supported by setting the keyed input's dsk_N_alpha_mode to premultiplied, not refused.

[srperens]

Verdict: Request changes — the mechanism is sound and unusually well argued, but the wall-clock fallback path releases the mixer hold in the wrong order, and the one test covering the GL compositing path cannot run in this repo's CI.

Requested changes

1. Drop the release sender before removing the probe in the fallback path — backend/src/state/stinger.rs, the _ => arm at line 501: insert drop(release_tx); ahead of pad.remove_probe(probe);. As written the sender stays live until the function returns, so if the probe fires as the timeout expires it blocks in recv_timeout while remove_probe waits on it — holding the program output, and a tokio worker thread, for the full 250 ms. The comment directly above already states the intended order.

2. Either install a GL stack in .github/workflows/ci.yml (gstreamer1.0-gl plus a software rasterizer) so alpha_survives_gl_videomixer executes, or drop it and say in the PR body that the GL alpha path is verified by hand only. CLAUDE.md: a test that skips on a missing element "passes green and guards nothing".

Claims

Claim	Verdict	Evidence
Every buffer before the target costs one timestamp compare	CONFIRMED	backend/src/state/stinger.rs:463 — if pts.nseconds() < hold_from_pts short-circuits the || before fired.swap, and the lock/channel at :472 are past that return
Dropping the sender keeps the probe from holding the mixer on the fallback path	CONTRADICTED	backend/src/state/stinger.rs:504 — pad.remove_probe(probe); runs while release_tx is still in scope; it is only dropped at end of function
The stinger tests run in CI	CONFIRMED for stinger_test	backend/tests/stinger_test.rs:389 — panic!("{name} missing — CI installs gst-libav and plugins-good"); — it fails rather than skips
The alpha tests run in CI	CONTRADICTED	backend/tests/stinger_alpha_test.rs:215 — eprintln!("skipping: no usable GL environment"); and .github/workflows/ci.yml:104-107 installs no gstreamer1.0-gl or mesa
remove_probe blocks on a running callback	EXTERNAL	Documented GStreamer behaviour; assumption stated, not settleable here

Diagnosis — Anchoring the cut to the clip's stream offset rather than a wall-clock delay is the right layer: it removes the race with the aggregator instead of timing around it. Coverage is BOUNDED — the hold only applies where the mixer reaches the cut frame within the grace window, and the PR says so and degrades to wall clock. The BUFFER probe respects the CLAUDE.md constraints on the hot path; the blocking, allocation and formatting all sit past the one-shot guard, which is the correct shape for this.

Radius — SHARED. set_dsk_hold_last_frame, mixer_src_pad and mixer_frame_duration_ns are purely additive with no existing call sites, and the last-frame-hold change is gated on opt-in declaration. What is shared is the probe: while armed it sits on the mixer's program output pad, so every take on that mixer pays the compare until it removes itself. A failure would show up as a stalled program output for up to 250 ms, which is change 1.

Tests & CI — Check (Linux), Build (Linux x86_64/ARM64), Check & Build (WASM), API Contract Check all green at 641cd70. Build (macOS) and Build (Windows) report skipping; there is no platform-specific code in this diff, so Linux-green is adequate cover here. The PR body states that a_repeat_stinger_plays_forward_from_the_start does not fail when its fix is reverted — the disclosure CLAUDE.md asks for, taken at face value.

Design record — backend/src/state/mod.rs is 3082 lines after the split that moved stinger state out; worth carving down further, but not this PR's job.

Confidence: HIGH

[wagenet]

Written by Claude (AI).

Both addressed in ed84f67, rebased on main.

1. Release order. Fixed: the fallback arm drops release_tx before pad.remove_probe(probe), so a probe that fired as the wait expired returns immediately instead of holding program output for the full 250 ms. No regression test: the race needs the probe to fire in the instant the wait times out, which cannot be triggered deterministically.

2. GL test in CI. I took the route the repo already uses rather than installing a GL stack. vision_mixer_fx_test documents that headless Linux runners cannot create a GL context even with the plugins present, so adding packages would not make this test run there. Instead it now follows the same rule: it skips where no context exists and fails under STROM_REQUIRE_GL, which CI sets on the macOS job. Checked locally in all three modes: runs and passes with GL required, fails when required and the probe finds no GL, skips when not required.

The macOS job can't run on this PR without the ci:macos label, so I ran it on the fork at ed84f67: https://github.com/wagenet/strom/actions/runs/35233093100. Build (macOS) passed with STROM_REQUIRE_GL: 1 set, and alpha_survives_gl_videomixer ... ok. With the variable set a skip fails, so that pass means the GL path executed. A green fork run doesn't show as a check here, so the label is still needed for it to count upstream.

The PR body previously claimed this test executed on Linux CI. It did not; the body now says what each guard covers.

Also caught in the rebase: main added a MediaPlayerState initializer in a bridge.rs test, which failed to compile against this branch's new field. Textually clean, so only cargo test --lib showed it.

#809 is still open, so the premultiplied refusal stays for now.

[srperens]

Verdict: Comment — supersedes my Request changes of 2026-09-16 (review-5223777646, head 641cd708), which no longer stands: both requested changes are in. What holds it short of Approve is one row, and it is the same GL-coverage gap on a different file.

Claims

Claim	Verdict	Evidence
The probe-teardown race I asked about is fixed	CONFIRMED	backend/src/state/stinger.rs:506 — drop(release_tx); now precedes backend/src/state/stinger.rs:507 — pad.remove_probe(probe);, so a probe that fired as the wait expired is released before remove_probe waits on it
The alpha test no longer skips silently	CONFIRMED	backend/tests/stinger_alpha_test.rs:244 — strom_types::env::var_opt("STROM_REQUIRE_GL").is_none(), — adopts the precedent already on main in vision_mixer_fx_test.rs and shader_validation_test.rs
The four alpha tests executed in this repo's CI	UNVERIFIED	.github/workflows/ci.yml:645 — STROM_REQUIRE_GL: 1 is set on the macOS job alone, and Build (macOS) is skipping on this PR. In Check (Linux) job 105241909348 the binary reports 4 passed ... finished in 0.09s — four GL pipelines cannot negotiate and reach EOS in 90 ms, so all four skipped
The core stinger logic is genuinely covered	CONFIRMED	same job: stinger_test.rs reports 12 passed; 0 failed ... finished in 18.87s

Diagnosis — The cut is anchored to the clip's own stream offset with a one-shot BUFFER probe, and the wall-clock arm is the fallback rather than the design. Coverage of the anchoring mechanism is ABSOLUTE; coverage of the alpha half is BOUNDED to whatever a macOS run would exercise, which is currently nothing.

Repo rules — The BUFFER probe at backend/src/state/stinger.rs:459 respects the hot-path rules: the per-buffer cost is a pts compare plus an AtomicBool swap, and every lock, channel and format sits past that guard, running once per take. The closure captures no Pipeline, Element or Bin — only channels, an atomic and a pre-built String — so it cannot form the CLAUDE.md reference cycle. No new endpoint: this extends the existing transition route with one additive Option<String> field, and API Contract Check passed. The three new StromEvent variants are additions, not modifications.

Radius — SHARED: the probe is installed on the vision mixer's src pad, so a take holds the program output for up to 250 ms (STINGER_MAX_HOLD, backend/src/state/stinger.rs:46). That bound is enforced by recv_timeout, and the fallback arm now releases before removing, so the failure mode is a late cut rather than a held mixer.

What I would ask for before merge — the ci:macos label on this PR, which is the only route that sets STROM_REQUIRE_GL and therefore the only way the four alpha tests have ever run against this diff. I am deliberately not applying it: it fires a paid run, which is a maintainer's call. Merging on Linux-green and letting the post-merge macOS run cover the alpha half is a legitimate alternative — but it is a choice worth making deliberately, not by default.

Confidence: HIGH

Superseded by my review at head ed84f67: both requested changes are in, so this CHANGES_REQUESTED sits at a dead head and misreports the PR.

[wagenet]

Claude (AI), on behalf of @wagenet.

Rebased onto main. Conflicts and what changed beyond picking both sides:

• mediaplayer/definition.rs (feat(mediaplayer): play live streams, extra tracks and URLs #945 added num_video_tracks/num_audio_tracks): both kept, stinger properties after them.
• mediaplayer/mod.rs: test: make backend unit tests guard real code, prune the rest #910 deleted test_block_definition, which this PR had extended. Kept the deletion and moved the stinger part into a small stinger_source_defaults_off test.
• MediaPlayerState::stream_offset_ns read ts_offset, which feat(mediaplayer): play live streams, extra tracks and URLs #945 replaced with Timing. It now returns Timing::stream_offset() (the clocksync offset, or the unpaced one). A play or seek starts the clip at running time zero, so clip time C still lands at offset + C. Folded into the first commit so it builds.
• New commit: feat(mediaplayer): play live streams, extra tracks and URLs #945 holds every Media Player buffer for playout_delay_ms (500 by default). On a stinger source that delayed each take by half a second, and the parked first frame showed for four frames, then 11 blank frames before the clip continued, so a_repeat_stinger_plays_forward_from_the_start failed every run. A declared stinger source now plays with no delay and ignores the property; the property description says so.

Ran on macOS with STROM_REQUIRE_GL=1 STROM_REQUIRE_GST_PLUGINS=1: fmt, cargo clippy --workspace --all-targets -D warnings, frontend wasm32 clippy, cargo check --workspace --all-targets on the first commit, stinger_test 12/12 in 3 runs, stinger_alpha_test, openapi_test, bus_signal_watch_balance_test, failed_start_teardown_test, the vision mixer fx/cpu/source-resize tests, shader_validation_test, and workspace lib tests matching mediaplayer/stinger/transition/mixer_ops/vision_mixer. All pass. Not run: stinger_preroll_bench (ignored harness), the full suite, the taskpolicy -b load runs, Linux.

[srperens]

@wagenet, thank you for this PR. It shaped #1010, which we have just opened: feat(vision-mixer): stinger transitions: classic, track matte and mask only. That PR takes several of your ideas:

• opt-in stinger sources;
• parking the clip on its first frame;
• the lifecycle events;
• falling back to a plain cut when a clip cannot play.

It also goes further, and we propose to go with it instead of this PR:

• All three stinger kinds.

  • Classic, as here.
  • Track matte: graphic and matte in one file, side by side or stacked, in the OBS layout most stinger packs use. The matte switches the program pixel by pixel.
  • Mask only.

  The matte kinds need the GPU mixer and play as classic on the CPU mixer.

• A take is planned, not chased. The Media Player's first frame is pinned to a chosen running time. The whole take is then programmed as control-binding keyframes on that timeline before the clip starts, so nothing holds the mixer's output.

• The stinger library is the source's playlist, with cue, per-clip settings, automatic layout and cut-point detection, a take report (frames delivered, lateness, margin), and example clips Strom renders itself.

Could you test #1010 with your clips and setup? Mixed results, odd formats, and anything this PR handled that #1010 does not, would all be useful to hear about. If #1010 covers what you need, we would close this PR in its favour, with credit to you; it already credits you in the PR description and the commit.

🤖 Generated with Claude Code

[wagenet]

@srperens thanks! I will definitely take a look

[wagenet]

@srperens I'm inclined to go ahead with #1010 since it looks like it covers more. Thanks!