Repository navigation
feat(vision-mixer): stinger transitions: classic, track matte and mask only - #1010
Conversation
srperens
left a comment
srperens
left a comment
There was a problem hiding this comment.
Code review (read and traced only, no tests run). One clear bug (removed/replaced cued clip stays parked, two call sites) plus three lower-risk issues. Details inline.
| if !playlist.is_empty() { | ||
| let target = match index.cmp(&cued) { | ||
| std::cmp::Ordering::Less => cued - 1, | ||
| std::cmp::Ordering::Equal => cued.min(playlist.len() - 1), |
There was a problem hiding this comment.
Bug (high): removing the cued clip leaves the removed clip parked.
stinger_remove_clip calls set_playlist, which keeps current_index and does not clear park.parked. In the Equal branch target == cued, so is_parked_on(target) is still true and stinger_cue → cue() returns Duration::ZERO without loading anything.
Example: library [a, b, c], a cued at 0, operator removes a. The panel shows b as cued and ready, but the player still holds a's first frame. The next take plays a (just removed) with b's analysis and settings (layout, cut point, crop, matte). Same when removing the last clip leaves one.
Root cause is is_parked_on (mediaplayer/state.rs) checking the playlist index rather than the loaded file. Suggested fix: clear park.parked whenever the file at the parked index changes, or have is_parked_on compare the loaded file.
There was a problem hiding this comment.
Fixed in a09cb5a. The player now records which file its pipeline holds, and is_parked_on compares that file with the playlist entry at the index, so a cue reloads when they differ. Guard: removing_the_cued_clip_parks_its_successor checks what airs; with the fix reverted the graphic never airs.
| // Only auto-start from the beginning if the player was stopped. A | ||
| // stinger clip source parks its first clip instead of playing it. | ||
| if player.stinger.enabled { | ||
| if player.playlist_len() > 0 && !player.is_parked_on(player.current_index()) { |
There was a problem hiding this comment.
Bug (medium): same root cause as the remove-clip issue in state/stinger.rs.
In stinger mode a PUT of a new playlist only re-cues when !player.is_parked_on(player.current_index()). If the player was parked on index i and the new playlist has a different file at i, the cue is skipped and the old file stays parked — it is what the next take plays. Same when set_playlist resets the index to 0 while parked there.
There was a problem hiding this comment.
Fixed by the same change (a09cb5a), since is_parked_on now compares files. Guard: a_playlist_put_over_the_parked_index_recues_it.
| // here. That is the stinger clip's parked first frame: it was already | ||
| // waiting inside its clocksync when the take armed them, so no | ||
| // clocksync probe sees it. | ||
| if paced && sync == UNSET && self.start_at.load(Ordering::Acquire) != UNSET { |
There was a problem hiding this comment.
Timing (low/medium): the pinned start can be taken from an audio buffer.
Audio and video share one Timing, and whichever stream's held frame reaches place() first sets sync_offset = pinned - rt from its own running time. play_at releases both clocksyncs at once, so audio can win. For a clip whose first audio and video timestamps differ (MP4 edit list, B-frames, audio priming), the first video frame then lands that much off start_at, and the mixer keyframes (graphic up, cut point, matte window) programmed for start_at no longer line up with the clip.
There was a problem hiding this comment.
Not addressed in this PR; listed as a follow-up under Out of scope.
There was a problem hiding this comment.
Fixed in 19ced5f after all. The parked frame's PTS (from the park probe, one-shot since 7bea68f) is turned into a running time before the clocksyncs are released, and Timing::pin_start takes it, so the first video frame lands on start_at and audio keeps its offset. Guards: a_pinned_start_lands_the_first_video_frame_when_audio_comes_first (unit) and a_clip_with_early_audio_lands_its_first_video_frame_on_the_take (audio 200 ms before video).
| // No graphic to hide a cut behind: mix across the span where the | ||
| // matte moves, or across the middle third without an analysis. | ||
| match settings.cut_point_ms { | ||
| Some(cut) => (Some(cut.min(last_ms)), settings.mix_ms), |
There was a problem hiding this comment.
Low: a mask-only clip downgraded on the CPU mixer ignores Cut and can mix past the clip's end.
With a user cut point this returns (cut, settings.mix_ms). It neither honours beneath == Cut nor clamps the mix to duration_ms - cut, as the classic branch further down does.
Example: cut at 1000 ms, 1000 ms mix, 1500 ms clip. The fade outlasts the clip, finish_stinger settles at end + 2 frames, and the incoming source jumps to full alpha mid-fade.
There was a problem hiding this comment.
Not addressed in this PR; listed as a follow-up under Out of scope.
There was a problem hiding this comment.
Fixed in 1014f86 after all: with a cut point, Cut gives no mix and Mix is clamped to duration_ms - cut. Guards: a_downgraded_mask_with_a_cut_point_honours_cut, a_downgraded_mask_mix_ends_with_the_clip.
| }; | ||
| let (watch, ftb_cancelled) = manager | ||
| .program_stinger(block, &take) | ||
| .map_err(|e| (e.to_string(), false))?; |
There was a problem hiding this comment.
Low: failure here leaves the mixer half-programmed.
If program_stinger fails after it has started changing pads, nothing calls abort_stinger. By then it may have cancelled a fade-to-black (ftb_active.swap(false)), reset pads, and keyed some alphas or z-orders. E.g. if enum_keyframes fails after B's alpha is keyed to 1 at start, program changes on air with no graphic, the error reports program_changed: false, and the FTB change is never broadcast.
There was a problem hiding this comment.
Fixed in ed896a7. program_stinger is split into argument checks, which change nothing, and the pad step. If the pad step fails, it calls abort_stinger and reports whether a fade-to-black was cancelled, and the take then broadcasts VisionMixerFtbChanged. Guard: a_take_that_fails_to_program_leaves_the_old_source_on_air, using a test-only hook to fail after every pad change.
| let start_at = (take.start + take.frame_ns / 2) as i64; | ||
| let played = tokio::task::spawn_blocking(move || player.play_at(start_at)) | ||
| .await | ||
| .map_err(|e| (e.to_string(), false))?; |
There was a problem hiding this comment.
Low: same as above. If the spawn_blocking join fails, this returns early without calling abort_stinger, unlike the played error path just below. The mixer stays programmed for a take whose clip never started.
There was a problem hiding this comment.
Fixed in ed896a7: a failed join now takes the same abort and cut-instead path as a play error. This path has no guard, since it would need a panic inside play_at.
|
I had my Claude to an analysis of this against #755:
|
wagenet
left a comment
wagenet
left a comment
There was a problem hiding this comment.
Review by Claude (AI), prepared at @wagenet's request.
I ran scratch tests against 4f20bfa2 for the points below (macOS 27, M4 Max, GStreamer 1.28.6). The tests are not pushed anywhere; I can share them. Inline comments have the details.
| Finding | Evidence | |
|---|---|---|
| 1 | A late classic clip cuts with no graphic over it, and the take still reports success | Clip frames held at the graphic pad. 700 ms: the cut frame airs uncovered. 1 s: 10 frames uncovered, graphic on air 5 of 30 frames. CPU and GPU mixers alike. Only StingerCompleted fires, with frames_arrived 30/30 |
| 2 | A flow restart during a take's inline analysis corrupts the next take on the new run | Reproduced 4 of 4. A still_ours check before program_stinger fixes it in the scratch test |
| 3 | Does not compile on current main |
Merges with no textual conflicts, but #1015 changed get_overlay_state to (flow_id, block_id): E0061 at state/stinger.rs:554, effects/stinger.rs:258, plus tests/stinger_test.rs:307 |
| 4 | An uncued take decodes its clip twice | Take-to-air for a clip added and taken at once: 157–189 ms at 1080p FFV1, 872–924 ms at 4K ProRes 4444 (45 frames), against 86–111 ms cued |
| 5 | A non-stinger transition without from_input/to_input now gets 400 instead of 422 |
Code reading; no test covers the new branch |
Two things that are fine:
- The leaky 20 MB bridge appsrc dropped nothing on a quiet machine in 48 takes. That included 4K ProRes 4444, which reaches it as AYUV64 at 66 MB per frame, larger than the cap. An empty queue still takes one oversized buffer, so drops need a starved process: under
taskpolicy -bplus CPU burners, it dropped 10–36 of 45 frames per 4K take. - The new BUFFER probes, signal closures and registries follow the CLAUDE.md rules. A subagent checked this by reading the code; it was not run.
Not done: Linux or Windows runs, and revert checks of the PR's own tests.
|
|
||
| match (variant, &pads.matte) { | ||
| (StingerVariant::Classic, _) => { | ||
| let cut = take.cut_at.unwrap_or(take.start); |
There was a problem hiding this comment.
The cut keyframes on a/b fire at cut_at whether or not the clip has reached the graphic pad. With max-last-buffer-repeat at two frames, a clip running late leaves nothing on the graphic pad at the cut, so the new source airs uncovered.
Measured with a scratch test that holds the first clip buffer after the take at the fill pad (cut point 500 ms, CPU mixer; the 700 ms and 1000 ms rows gave the same result on the GPU mixer):
| hold | graphic on air | uncovered frames after the cut | events |
|---|---|---|---|
| 0 ms | 30/30 | 0 | Started, Completed (30/30, 0 late) |
| 300 ms | 26/30 | 0 | Started, Completed (30/30, 6 late) |
| 700 ms | 14/30 | 1 (the cut frame) | Started, Completed (30/30, 18 late) |
| 1000 ms | 5/30 | 10 | Started, Completed (30/30, 27 late) |
The plan is fixed before the clip runs, so the cut cannot wait for the graphic. Reporting it is possible, though: see the comment on the counting probe.
There was a problem hiding this comment.
Fixed in 59e3b37. The probe records, with atomics only, whether the graphic frame due at the cut arrived in time. When it did not, the take report carries a warning and the take logs it. I kept StingerCompleted rather than StingerFailed, because the program did change. Guard: a_late_cut_frame_is_reported, which holds the first clip buffer for 1 s.
| let arrival = counted.epoch_rt + counted.epoch.elapsed().as_nanos() as i64; | ||
| let margin = pts as i64 + counted.latency_ns - arrival; | ||
| if counts_frames { | ||
| counted.arrived.fetch_add(1, Ordering::Relaxed); |
There was a problem hiding this comment.
arrived counts every frame that reaches the pad, including frames the mixer then discards as too late. In the 1 s case above the report says frames_arrived 30/30 while 5 frames aired.
Suggestion: this probe already knows each frame's margin. Count only frames that arrive on time, and remember whether the frame due at cut_at was on time. When it was not, finish_stinger_take could emit StingerFailed (program changed), or a warning, instead of only StingerCompleted. That gives a client something to react to.
There was a problem hiding this comment.
Done in 59e3b37: frames_arrived now counts only frames that arrive in time to air, and the cut frame is tracked as suggested.
| ) | ||
| .map_err(|e| (e, false))?; | ||
|
|
||
| // Program the mixer for the frame the clip lands on. |
There was a problem hiding this comment.
The take token is not re-checked after the cue and analysis awaits. A stop/start during them clears the claim (forget_flow), and pipelines.get(flow_id) then returns the new run's manager. This take then programs the new mixer. Its play_at fails on the unregistered old player ("Internal pipeline not created"), and the error path calls abort_stinger and "cut instead" on the new run.
Reproduced with a scratch test (CPU mixer). Take A uses an uncued 1080p ProRes 4444 clip, so its analysis takes 0.6–2 s. At 300 ms: stop_flow, then start_flow, then take B (classic, cut at 500 ms) on the new run.
- A ends during B (2/2): A's
program_stingerresets B's pads. B's graphic leaves at frame 14, the cut on frame 15 never airs, and the program changes only at frame 34, from A's "cut instead". B reportsCompleted 30/30. - A ends after B (2/2): A's "cut instead" changes the new run's program about 0.8 s after B finished, and A reports
StingerFailed(program_changed=true).
With this check added here, both cases are clean (B on air frames 2–31, cut on frame 15; A fails with program_changed=false):
if !still_ours(*flow_id, block, token) {
return Err(("the flow restarted during the take".to_string(), false));
}Doing the check under the pipelines read lock would close the remaining gap.
There was a problem hiding this comment.
Fixed in 18ab17d. The take re-checks its claim under the pipelines read lock before program_stinger, before the abort on a play error, and before finish_stinger. The check also confirms the clip source is still the registered player. Guard: a_take_outlived_by_a_restart_leaves_the_new_run_alone, made deterministic with a test-only hold before programming. Reverted, it fails with your symptom.
| } | ||
|
|
||
| if let Some(manager) = self.inner.pipelines.read().await.get(&flow) { | ||
| manager.finish_stinger(&mixer, &take); |
There was a problem hiding this comment.
The same gap exists here, but narrower. The loop breaks on the old pipeline's position, and then finish_stinger runs on whatever manager pipelines.get(&flow) returns. Re-checking still_ours under the read lock would close it. I did not reproduce this one: a restart would have to complete inside this one read().await.
There was a problem hiding this comment.
Covered by the same re-check in 18ab17d, before finish_stinger.
| let uri = ctx.uri(file); | ||
| let info = match analysis::cached(&uri) { | ||
| Some(info) => Some(info), | ||
| None => tokio::task::spawn_blocking(move || analysis::analyze_cached(&uri)) |
There was a problem hiding this comment.
This inline analyze_cached does not check ANALYSING, so a clip taken right after stinger_add_clip is decoded twice: once by the background run that the add started, and again here.
Take-to-air for a clip added and taken at once (45 frames, release build, three runs):
| clip | uncued | cued |
|---|---|---|
| FFV1 1080p | 157–189 ms | 86–111 ms |
| FFV1 4K | 363–456 ms | 86–111 ms |
| ProRes 4444 1080p | 407–425 ms | 86–111 ms |
| ProRes 4444 4K | 872–924 ms | 86–111 ms |
Waiting on the background run, instead of starting a second decode, would cut the cost.
There was a problem hiding this comment.
Not addressed in this PR; listed as a follow-up (wait on the background analysis instead of decoding again).
There was a problem hiding this comment.
Fixed in d64620d after all: analyze_cached runs one analysis per file at a time, and a take that arrives meanwhile waits for it and shares the result. Guard: a_clip_taken_straight_after_it_was_added_is_analysed_once counts analysis runs (2 without the fix).
| ) -> Result<StingerTakeResponse, PipelineError> { | ||
| let requested = Instant::now(); | ||
| let ctx = self.stinger_context(flow_id, block).await.map_err(err)?; | ||
| let state = crate::blocks::builtin::vision_mixer::overlay::get_overlay_state(block) |
There was a problem hiding this comment.
After #1015 this needs get_overlay_state(flow_id, block). The branch merges into current main without textual conflicts, but cargo check --workspace --tests fails with E0061 here and at effects/stinger.rs:258. tests/stinger_test.rs:307 uses the old signature too.
There was a problem hiding this comment.
Rebased onto main and fixed in ffcd218.
| "a matte stinger needs the GPU mixer".to_string(), | ||
| )); | ||
| } | ||
| let state = overlay::get_overlay_state(block_instance_id).ok_or_else(|| { |
There was a problem hiding this comment.
Same as state/stinger.rs:554: after #1015 this is overlay::get_overlay_state(&self.flow_id, block_instance_id).
| StatusCode::BAD_REQUEST, | ||
| Json(ErrorResponse::with_details( | ||
| "Failed to trigger transition", | ||
| "from_input and to_input are required (only a stinger may leave them out)", |
There was a problem hiding this comment.
Before this change, a request without from_input/to_input failed deserialisation and got 422. It now gets 400. Probably fine, but it is a visible change for API clients, and no test covers this branch.
There was a problem hiding this comment.
Intentional. It is now stated under API contract in the PR description.
7bea68f to
98ffc4f
Compare
…k only A stinger plays a clip over the program while the source changes beneath it. Three variants, after ATEM, vMix and OBS: - Classic: a graphic with alpha; the program cuts or mixes beneath it at a cut point (by default where the graphic covers most, found by analysing the clip). - Track matte: graphic and matte in one file, side by side or stacked (the OBS layout). The matte switches the program pixel by pixel. - Mask only: the clip is a matte, an animated wipe. The matte variants need the GPU mixer and play as classic on the CPU one. The source is a Media Player in stinger mode wired to the mixer's new stinger input; its playlist is the library and its block holds per-clip settings. A cue parks a clip on its first frame. A take pins the running time its first frame lands at, programs the whole take into the mixer as control-binding keyframes on that timeline, and only then lets the clip go, so nothing waits on wall clock or holds the mixer's output. The matte composites inside glvideomixer: the matte pad takes the matte off the destination alpha (reverse-subtract) and the incoming source blends by it for the clip's frames. Clip frames leave their clocksync ahead of time and are stamped half an output frame into their frame, so a millisecond-rounding container cannot move one into its neighbour's. A stinger source asks for system memory, so a hardware decoder keeps alpha (VideoToolbox gives AYUV64 for ProRes 4444), and the stinger input converts what GL upload cannot take. Operator panel: STING type, the clip library with cue, per-clip settings, take reports, and example clips Strom renders itself (one per variant). API: stinger state, cue, take, clip settings, examples; takes also run through the transition endpoint. Events: StingerCued, StingerStarted, StingerCompleted (with measurements), StingerFailed. Builds on ideas from #755 by wagenet: opt-in sources, parking, events, degrading to a cut when a clip cannot play. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The blade's glow and trails were drawn unclipped, so near the right edge they painted into the matte half of the side-by-side frame, and the trails stayed on screen after the blade had left. Clip the graphic to its half, and scale the trails with the blade's speed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What an outside client needs to drive stingers without knowing the Media Player behind them: - POST/DELETE .../stinger/clips: add a clip to the library (idempotent, a local file must exist) or remove one with its settings, on the mixer. - Calls that address a clip by index (cue, take, settings, remove) take an optional `file`; when the library has changed under the client, the call fails with 409 instead of acting on another clip. - Every take has an id: in the take response and in StingerStarted, StingerCompleted (report) and StingerFailed. The take response also names the file played. - from_input/to_input are optional in the transition request: a stinger always takes PGM to PVW; other transitions still require them (400). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Side by side and stacked were read against the program's aspect, so the same file could detect differently on a 4:3 or vertical production. A track-matte clip is two standard frames (16:9, 4:3 or 9:16) next to or above each other, whatever the program runs; the analysis cache no longer keys on the program either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every frame of the 3840x1080 example in a debug build took seconds of CPU beside the lib's timing-sensitive tests. The spill can only happen as the blade nears the right edge, so check those frames. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Linux CI (GStreamer 1.24.2, llvmpipe) a take that switched to a clip of another size showed the matte one frame late: the stinger branch renegotiated and the matte shader recompiled on the take's first frames. A cue now sends one blank frame in the clip's format through the consumer, so that happens at cue; the stinger pads are hidden between takes, so it never shows. The take report counted lateness on the graphic pad only; the matte pad, with a shader pass of its own, now counts too. The tests read the clip's frame count from its analysis, since 1.24.2's FFV1 decoder drops the last frame of a stream, and print the whole take when a frame is off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GStreamer 1.24.2's FFV1 decoder (Linux CI) drops a stream's last frame now and then, so a clip's analysis and its playback could disagree on its length and a take ran a frame longer than the test expected. PNG frames in QuickTime are lossless, keep alpha, and decode through libpng without frame threads. The FFV1 examples' own test allows the missing frame. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… pad dump Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… take Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
make_glshader now starts every glshader in passthrough (#1007). The matte is not an FX slot and must always render: in passthrough the clip reached the mixer as it is, and its own alpha became the matte. The GPU stinger test fails without this. Also report the take with the CPU test's cut assertion, and drop the pad-dump eprintln (the dump stays in the failure message). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The mixer stamps output frames from a rounded origin, so a frame's timestamp can be a nanosecond either side of FrameGrid's. A step key placed exactly on the frame then acted on that frame or the next by chance: depending on the take's phase, the graphic went on air a frame late while the cut landed on time. cpu_stingers_play_classic failed 6 of 8 local runs with 'cut at frame 13'; with the keys half a frame early it passed 8 of 8. The linear mix ramp keeps its exact times. The test rig now collects frames over a span of stream time, not wall time: on a slow runner the debug-build mixer falls behind real time and the window ended before the clip did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stinger source counted itself parked on a playlist index, not on a file. Removing the cued clip, or a playlist PUT that put another file at the parked index, left the old clip's first frame parked: the panel showed the new clip ready and the next take played the old one with the new clip's settings. The player now records the playlist entry its internal pipeline holds. `is_parked_on` compares it with the entry at the index, and a cue reloads when they differ, so both the library edit and the playlist endpoint re-cue the right file. Guard tests (stinger_api_test): removing_the_cued_clip_parks_its_successor and a_playlist_put_over_the_parked_index_recues_it check what airs, and both fail with the fix reverted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stop clears a take's claim, but the take did not look again after its cue
and analysis awaits. After a stop and start in that window,
`pipelines.get(flow)` returned the new run, and the old take programmed its
mixer, then (its clip source gone) aborted and cut the new run's program.
The finish step had the same gap after its wait loop.
The take now re-checks its claim, and that its clip source is still the
registered one, under the `pipelines` read lock before it programs the
mixer, before it aborts on a clip that would not play, and before it
settles the pads at the end. A stop clears every claim before it takes
the pipeline away, so a claim held under that lock means the run the take
started on. A take that lost its claim fails without touching anything and
without "cut instead".
Guard test (stinger_test):
a_take_outlived_by_a_restart_leaves_the_new_run_alone. Timing a restart
into a real cue or analysis is not reliable with the test's small clips,
so a test-only hook (`hold_takes_for_tests`, per flow) holds take A at the
point right before programming while the flow restarts. The test then
checks that A fails with program_changed false, the new run's PGM is
untouched, and a classic take on the new run lands frame by frame. With
the re-check removed it fails ("clip 1 could not play ..., cut instead").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A take's cut keyframes fire at the cut point whether or not the clip has reached the graphic pad. A clip running late airs the new source with no graphic over it, yet the take reported `frames_arrived` n/n (every frame that reached the pad, including those the mixer had already passed) and a plain StingerCompleted. The counting probe now counts a graphic frame as arrived only when it is in time, and records whether the classic clip's frame due at the cut point (the newest clip frame that starts before the cut's output frame ends) arrived in time. Still atomics only in the per-buffer probe: one range check and one fetch_or. When that frame was late, the take report carries a new `warning` field and the backend logs it. StingerTakeReport is new in this PR, so the optional field breaks no client; StingerCompleted and StingerFailed keep their fields. openapi.json updated. Guard test (stinger_test): a_late_cut_frame_is_reported holds the take's first clip frame for a second before the graphic pad, and requires a warning and fewer than n frames arrived. With the probe and report change reverted it fails (warning None, 30/30 arrived). The classic take helper now also requires a clean take to carry no warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`program_stinger` cancels a fade-to-black, resets the pads and then keys alphas, z-orders and blend modes one property at a time. A failure after the first of those left the mixer half programmed: the keyframes already set could change the program on air with no graphic, the error reported `program_changed: false`, and the ended fade-to-black was never broadcast. A failed `spawn_blocking` join around `play_at` returned early in the same state, without the abort the play error path does. `program_stinger` now aborts the take itself (old source alone on air, stinger pads down) when a pad change fails, and its error carries whether it ended a fade-to-black. The take broadcasts VisionMixerFtbChanged for that, and also when it aborts after a clip that would not play. A failed join now goes down the same abort and "cut instead" path as a play error. Guard test (stinger_test): a_take_that_fails_to_program_leaves_the_old_source_on_air. No real keyframe failure can be caused on demand, so a test-only hook (`fail_stinger_programming_for_tests`, per flow) fails programming after every pad change is made. From fade-to-black, the test requires the old source alone on air through where the clip and cut would have aired, an FTB-ended event, and a normal classic take afterwards. Without the abort it fails (the cut airs); without the broadcast it fails (no FTB event). The join-failure path has no guard: it needs a panic inside `play_at`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed mask A mask-only clip downgraded to classic on a CPU mixer took the operator's cut point but then mixed for the full mix length whatever `beneath` said. It now cuts when `beneath` is `cut`, and clamps the mix to the rest of the clip like the classic branch, so the fade never outlasts the clip. Guards: a_downgraded_mask_with_a_cut_point_honours_cut and a_downgraded_mask_mix_ends_with_the_clip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Audio and video of a stinger clip source share one Timing, and a take's pinned start was taken from whichever stream reached the bridge first. For a clip whose audio starts before or after its video (an MP4 edit list, B-frames, audio priming) the first video frame then aired off the frame the mixer was programmed for, by the difference. The park probe on the video clocksync now also keeps the last buffer's PTS (one more atomic store). play_at turns the parked frame's PTS into its running time from the clocksync's segment and pins the baseline to it, so the first video frame lands on start_at and audio keeps its offset from video. Without a parked frame the old behaviour stays. An ordinary player never pins, so its timing is unchanged. Guards: - a_pinned_start_lands_the_first_video_frame_when_audio_comes_first (timing.rs unit test, audio placed first). - a_clip_with_early_audio_lands_its_first_video_frame_on_the_take (stinger_test): a classic clip whose PCM audio starts 200 ms before its video, CPU mixer, checked frame by frame on the program output. Which stream reaches the bridge first is a race, so the test holds the take's first video sample for 30 ms through a new #[doc(hidden)] hold_bridge_for_tests hook; the bridge checks it with one relaxed load per sample. Fails 4/4 with the pin taken as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A clip taken straight after it was added decoded twice: once in the background run the add started, and again in the take, whose inline analyze_cached did not look at that run. analyze_cached now runs once at a time per file. A caller that comes while a run is going waits for it and shares its result (a per-file slot with a Condvar), and a guard hands an error to the waiters if the run stops without one, so nobody waits forever. With no run going it analyses as before. The take still re-checks that the flow was not restarted after the analysis await. Guard: a_clip_taken_straight_after_it_was_added_is_analysed_once (stinger_test) adds a 1080p clip and takes it at once, then checks a new #[doc(hidden)] per-file counter of analysis runs is 1. With the old analyze_cached it counts 2 (5/5 runs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The park probe on a stinger clip source's video clocksync stayed on the pad for good, counting every buffer of every take, though only the first frame after a cue matters: it tells the cue the clip is parked and gives the take that frame's PTS to pin the start to video. The probe is now one-shot. It records the parked frame's PTS, bumps the parked-frame count and returns Remove. StingerPlayback::arm_park_probe installs a fresh one each time a frame is expected to park: - The bridge arms it whenever it builds the first video chain of a stinger source. Loading a clip removes the bridge chains, so a cue that loads finds no chain to arm on; the first buffer into the new chain is the loaded clip's first frame. Arming on every new chain also covers the builder-seeded first clip, whose chain appears before or after its cue. A relinked chain (an HLS pad taking over) is not armed. - A cue that rewinds a loaded clip arms on the existing chain before the seek. The probe id is kept in the player with the parked-frame count it was armed at; re-arming removes a previous probe that has not fired yet. The lock is taken once per arm, never in the probe. last_pts is renamed parked_pts, and the take reads the segment from the named park clocksync. The bridge's test-hold check in new_sample is gated by a bool captured when the callback is built, so an ordinary Media Player never loads the BRIDGE_HOLDS_ARMED flag. Guard: a_take_plays_with_no_park_probe_on_its_path (stinger_test) takes a 30-frame clip and checks the parked-frame count moves at most twice over the take and the re-cue after it. With the probe returning Ok again it moves 31 and the test fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stinger cue also restarts a source output that stopped on an error, with a flush local to its branches, so a clip some consumer cannot take costs that clip and not every take after it. The GL input adaptation that made a ProRes 4444 clip after another format play at all is on main since #1033; the tests here cover the switch between formats on the stinger path and the recovery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Redraw the example stingers so they look like a stinger pack rather than test patterns, and add a fourth clip so every layout has an example. - strom-sweep (classic): pink, cyan and navy slanted panels whoosh in with glowing edges, light streaks and a lens flare; the navy panel covers the whole frame with an extruded, glowing STROM wordmark whose letters fly in with overshoot and leave after a small pull back. - strom-blade (track matte, side by side): a white-hot blade with a chromatic fringe, crackling arcs, speed lines and sparks; the matte's soft switch sits just behind the light. - strom-ribbons (track matte, stacked, new): seven brush ribbons with bristle texture, shading and drop shadows fan out from the middle; the matte reveals the new source behind their heads. - strom-shards (mask only): hexagonal tiles flip open in a staggered wave through grey to white. Moving clips average sub-frame samples for motion blur; mattes are drawn once per frame. Randomness comes from a seeded generator. The examples render in parallel, and examples.rs is split into one module per clip. Tests check full cover of the classic, a matte that is never painted into, goes strictly from 0 to 255 and never back, a clean frame at both ends, a monotonic mask, and the layout, cut point and matte marks of every encoded example, the stacked one included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p and ribbons examples The sweep's royal blue panel and the ribbons' rainbow of light gradient ends read as pastel. Both now share one palette: ink and graphite, an ultraviolet-to-magenta family and a single acid volt accent, dark at the tail and saturated at the head, with a lighter top highlight on the ribbons. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A clip file rewritten under the same name kept playing its old content: the player only compared file names. The stinger clip source now records the loaded file's modification time and length, and a parked clip whose file changed since is not parked (is_parked_on is false), so the panel shows it not ready and a plain cue or take loads it again. A file gone from disk keeps the parked clip. POST .../stinger/reload analyses every clip whose file has no analysis for its current content, loads and parks the cued clip again when its file changed, flags clips whose files are gone (new `missing` field on StingerClip) and returns the stinger state. Refused with 409 while a stinger is on air. The analysis cache keys on the same stamp, now at nanosecond resolution. Operator panel: clip tiles in up to four columns at two rows' height, a scrollbar that stays visible on macOS, a clip count, the cued clip kept in view, MISSING and analysing badges, and a RELOAD button. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stinger clip from a source that cannot seek (an HTTP server without range requests) played its first take, but the re-park after it failed with "Seek failed": cue() returned the error and left the clip neither parked nor reloaded, and every later cue and take failed the same way until the flow restarted. When pausing and rewinding fails, the cue now loads the clip again, which starts it from its first frame; the bridge arms the park probe on the new chain, and the parked-frame count is read again before the load. A seek that works behaves as before. The guard test serves a Matroska clip from a local HTTP server with `Accept-Ranges: none` and takes it twice; both takes must deliver every frame. CI installs libsoup-3.0-0 explicitly so souphttpsrc loads. The operator guide says stinger clips are meant to be local files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stinger source parks its first clip while the flow is still starting. The clocksync pacing probe drops every buffer that arrives before the flow has a running time (so a live stream is never paced against its raw timeline), and the internal pipeline decodes on meanwhile, so the frame that parked was whichever came after the flow reached PLAYING: frame 3, or frame 17 on a loaded machine. Every take of that clip then started that many frames in, and its report counted fewer frames than the clip has. A stinger clip's parked frame needs no offset from the flow: the take sets the clocksyncs and pins the start from that frame. Let it through and remove the probe instead of dropping it. An ordinary player keeps dropping as before. This was the flake in stinger_prores_test (frames_arrived 29 or 27 of 30, or the graphic on air for 16 frames, on the first take of a clip). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… download API Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he library - A mask-only clip downgraded to classic on the CPU mixer no longer raises its grey frames over the program; the cut-frame check skips a clip with no graphic. - Fade-to-black is refused while a stinger is on air, so the take's end no longer brings the program back under a flagged fade. - Classic takes and fade-to-black claim the mixer while they program the pads, closing the window between the on-air check and the pipeline lock. - The Media Player playlist, control, seek and goto endpoints answer 409 while the player is playing a stinger take. - Stinger library edits are serialised and refused during a take, so clips added together all land. - A mix beneath a classic take ends with the clip when the cut point moves up to the output grid (take timing moved to crate::stinger::take_times). - The cue's warm-up frame is skipped when the consumer already took one in the same format, and an unlinked output is not flushed on every cue. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
libsoup asks GIO for the system proxy, and GIO's proxy lookup aborts the whole test process when the org.gnome.system.proxy schema is missing: stinger_api_test died with SIGTRAP on Linux CI, twice on 038a224. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cache-apt-pkgs-action restores package files without running triggers, so gschemas.compiled never gains org.gnome.system.proxy on a cache hit, and the HTTP stinger test's process aborts in GIO's proxy lookup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 participants
Refs #755
Problem
The vision mixer has no stinger: a clip played over the program while the source changes beneath it. Broadcast switchers have two kinds. The classic stinger (ATEM, vMix) is a graphic that covers the frame while the program cuts beneath it. The track-matte stinger (OBS, and most stinger packs) adds a matte that switches the program pixel by pixel, so the change follows the graphic across the frame. There is also the mask-only variant, a matte with no graphic, which is an animated wipe.
Change
Credit. This builds on @wagenet's #755: opt-in stinger sources, parking the clip on its first frame, the lifecycle events, and degrading to a plain cut when a clip cannot play all come from there. This PR covers the matte variants as well, and times takes differently (below).
Model. A Media Player in the new Stinger Clip Source mode feeds the vision mixer's new
stinger_in. Its playlist is the stinger library. Per-clip settings are stored on the player block: layout, cut point, cut or mix beneath, premultiplied alpha, and matte inversion. A clip's layout is detected by decoding it once, from the clip alone. Two standard frames (16:9, 4:3 or 9:16) side by side, such as 32:9, mean side by side; one above the other, such as 16:18, mean stacked. One picture with alpha is classic, and grey without alpha is a mask. The program's format plays no part. The same analysis finds the classic cut point (the frame where the graphic covers most) and where the matte crosses half way. That crossing is the cut point when the CPU mixer plays a matte clip as classic.Timing: the take is planned, not chased.
Timing::pin_start).Nothing waits on wall clock, and nothing holds the mixer's output. The keyframes cover the graphic up and down, the cut or mix beneath, the matte window, the incoming source's z-order and blend functions, and the outgoing source down.
Step keys sit half a frame before the output frame they act on. The mixer stamps its frames from a rounded origin, so a frame's timestamp can be a nanosecond either side of the computed grid; a key placed exactly on it acted on that frame or the next by chance. The mix ramp keeps its exact times.
Rejected: holding the mixer's src pad with a probe until a task applies the cut, as #755 does. That pauses every output downstream, and it needs stall and wall-clock fallbacks.
Matte compositing happens inside
glvideomixer, with no new shader engine. Aglshaderturns the clip's matte region into alpha. It is not an FX slot, so it opts out of the idle-slot passthrough from #1007 and always renders.blend-equation-alpha=reverse-subtract).src·(1−dst.a) + dst·dst.a, which isB·m + A·(1−m).A matte frame that never arrives leaves the destination alpha at 1, so a late clip shows the old source, never a premature new one.
Delivery.
STINGER_RELEASE_AHEAD_MS), so they reach the mixer ahead of their deadline.Rejected: classic
uridecodebinwithautoplug-selectto keep hardware decoders away. Stopping one GPU flow then stalled stinger delivery in other flows in the same process.Operator panel (
vision-mixer.html):API (registered in
openapi.rs, snapshot updated). An outside client drives stingers on the mixer alone, without knowing the Media Player behind it:GET .../stinger: state, library, analysis, last take.POST .../stinger/clips: add a clip. Idempotent; a local file must exist; upload with the media API first.DELETE .../stinger/clips/{index}: remove a clip with its settings.PUT .../stinger/clips/{index}: clip settings.POST .../stinger/cue.POST .../stinger/take: returns atake_idand the file played.POST .../stinger/examples.POST .../stinger/reload: returns the same state as GET; 409 while a stinger is on air. A clip'smissingfield reports a file deleted since it was added.Every call that names a clip by index (cue, take, settings, remove) takes an optional
file. When the library has changed under the client, the call answers 409 instead of acting on another clip.A take also runs through the transition endpoint with
transition_type: "stinger"(new optional fieldstinger_clip).from_inputandto_inputare now optional in that request: a stinger always takes PGM to PVW, and other transitions still require them (400 when missing). Clients that send them are unaffected.Events: new
StromEventvariantsStingerCued,StingerStarted,StingerCompleted(with the take report) andStingerFailed. All three take events carry the take'stake_id. No existing variant changed.Files that change, and sources that cannot seek. The player remembers the modification time and length of the file it loaded, so a file rewritten under the same name is not taken as parked, and a plain cue loads it again. When a clip cannot seek back to its first frame (an
http://URL on a server without ranges), the cue loads it again instead of leaving the clip unplayable. Stinger clips are meant to be local files; a URL plays but is fetched again on every cue.A source output that stopped on an error (a consumer that refused a clip's format) is restarted at the next cue, so one bad clip costs that clip and not every take after it. The GL input adaptation that lets a ProRes 4444 clip after another format play at all is on main since #1033.
Measured per take: frames that reached the mixer, how many were late, the smallest margin on the deadline, take-to-air time and cue time.
Evidence
gpu_stingers_land_frame_accuratelyandcpu_stingers_play_classic. When the frame grid was computed in whole nanoseconds, the first failed withleft: [0, …, 30]against 30 frames. With step keys placed exactly on frame times, the second failed 6 of 8 runs withcut at frame 13(the graphic a frame late); with the half-frame lead it passed 8 of 8. That guard depends on the take's phase, so it catches the bug most runs, not every run.gpu_stingers_land_frame_accuratelyfailed on macOS and Linux with the old source on air for the whole take (the clip passed through, and its own alpha acted as the matte).gpu_stingers_land_frame_accuratelyreads the old/new boundary on every frame. Before the half-frame stamping it failed withprogram frame 0 shows matte edge 11, clip frame 0 has 0.prores_4444_keeps_its_alpha. With the system-memory request removed it fails withthe graphic never went on air. On Linux CI it decodes throughavdec_proresto A444_10LE and exercises the stinger input's conversion.a_clip_that_will_not_play_still_changes_the_program.stopping_mid_take_releases_the_pipeline.the_library_is_edited_on_the_mixer_and_guarded_by_file.StingerStartedandStingerCompleted; a stinger through/transitionneeds no inputs, and other transitions still do:a_take_is_followed_by_its_id.layout_follows_shape_and_alpha.the_blade_never_paints_into_its_matte,the_blade_leaves_the_frame_clean.stinger_clip_test.shader_validation_test(added toall_fragments).removing_the_cued_clip_parks_its_successor,a_playlist_put_over_the_parked_index_recues_it. Both check what airs. Without the fix both fail with the graphic never on air; the take response already named the right file, so a check on the response alone would not have caught it.a_take_outlived_by_a_restart_leaves_the_new_run_alone. Without the fix it fails withclip 1 could not play (Internal pipeline not created), cut instead, the symptom @wagenet reproduced.a_late_cut_frame_is_reported(the first clip buffer held 1 s). Without the fix: no warning,30/30arrived. Everyclassic_takein the suite also asserts no warning.a_take_that_fails_to_program_leaves_the_old_source_on_air. Without the abort the cut airs at about 0.67 s; without the broadcast the FTB event is missing.a_clip_taken_straight_after_it_was_added_is_analysed_once(counts analysis runs, no timing assertion). Without the fix it counted 2 in 5 of 5 runs. It relies on a 1080p analysis outlasting the cue, so a much faster machine could let the reverted code pass; the fixed code cannot fail because of that.a_pinned_start_lands_the_first_video_frame_when_audio_comes_first(unit, audio before and after video) anda_clip_with_early_audio_lands_its_first_video_frame_on_the_take(a clip whose audio starts 200 ms before its video, frame-exact on the program output). Without the fix both fail; the integration test 3 of 3.a_downgraded_mask_with_a_cut_point_honours_cut,a_downgraded_mask_mix_ends_with_the_clip. Without a cut point the clip still mixes across the matte's movement, as before.a_take_plays_with_no_park_probe_on_its_path. With a permanent probe it fails withsaw 31 frames over a take of 30.stinger_bench(ignored,--ignored --nocapture). macOS, 1080p30 GPU mixer, 3 takes per row, 54 takes in total:Every row held for preroll 120, 80 and 50 ms. Take-to-air was the preroll plus 0–30 ms. Analysis took 0.4–1.2 s per clip and runs in the background on cue.
Tests
Second review round (
038a224), on macOS (GStreamer 1.28.6):a_downgraded_mask_never_shows_its_matte: a mask clip on the CPU mixer kept its grey frames off air. Reverted:the mask is on air at 166666666 ns, x 0: [0, 0, 0].fade_to_black_waits_for_a_stinger: fade-to-black is refused during a take. Reverted: the fade went through.a_take_owns_its_clip_source: the Media Player's playlist and control endpoints answer 409 during a take, and so does a library addition. Reverted: the playlist edit answered 200.clips_added_together_all_land: six clips added concurrently all land. Reverted: 2 of 6 landed.a_mix_ends_with_the_clip_when_the_cut_point_moves_to_the_grid(unit,take_times). Reverted: the mix ran 10 ms past the clip.stinger_test(12),stinger_api_test(9),pipeline_lifecycle_test,media_player_control_test,openapi_test(snapshot updated for the four 409s),cargo test --lib(950): passed.fade_to_black_waits_for_a_stingerexercises the same claim.stinger_prores_test(4) andstinger_clip_test(3) withSTROM_REQUIRE_GL=1 STROM_REQUIRE_GST_PLUGINS=1: passed.stinger_bench, the operator panel by hand.After the rebase onto main with #1033 and #1043 (
f08da4e), on macOS (GStreamer 1.28.6) withSTROM_REQUIRE_GL=1 STROM_REQUIRE_GST_PLUGINS=1:stinger_prores_test(4): 10 of 10 full runs passed, and 12 of 12 with tracing on. Before86001ebit failed about half the runs: under load the source parked a frame other than frame 0 when the flow started, because the pacing probe dropped buffers until the flow had a running time. A stinger now keeps that frame (a_stinger_clip_keeps_its_first_frame_before_the_flow_runsfails with the fix reverted).stinger_test(10): 3 runs, all passed.stinger_api_test(7),stinger_clip_test(3),vision_mixer_input_format_test(14),pipeline_lifecycle_test(4),media_player_control_test,openapi_test,cargo test --workspace --lib: passed.reload_picks_up_a_clip_rewritten_on_disk,a_cue_loads_a_parked_clip_rewritten_on_disk,a_clip_that_cannot_seek_plays_on_every_take(needssouphttpsrc; CI now installslibsoup-3.0-0),a_prores_clip_after_another_format_still_airs,another_format_after_a_prores_clip_still_airs,the_stinger_source_recovers_after_a_clip_a_consumer_refused.stinger_bench, and the operator panel by hand. The new examples and RELOAD were exercised in a running server.Earlier runs, before these changes:
The integration tests build their clips as PNG frames in QuickTime (
pngenc,qtmux). The first version used FFV1 in Matroska, but GStreamer 1.24.2's FFV1 decoder (Linux CI) drops a stream's last frame now and then, which made frame counts flaky. The example clips the operator panel renders are still FFV1. The test rig collects program frames over a span of stream time, not wall time, so a slow runner's debug-build mixer cannot end the window before the clip.After the review fixes (
7bea68f, rebased onto main including #1015), on macOS withSTROM_REQUIRE_GL=1 STROM_REQUIRE_GST_PLUGINS=1:stinger_test(10): 7 runs, all passed.stinger_api_test(4),stinger_clip_test(3),stinger_prores_test(1).openapi_test,pipeline_lifecycle_test,vision_mixer_cpu_test,media_player_control_test.cargo test --workspace --lib: backend 916, frontend 28, strom-types 104 passed.stinger_bench, and the operator panel by hand.Before the review fixes, on
4f20bfa(macOS, Apple Silicon, GStreamer 1.28.6), rebased on main as of then:stinger_test(4): 8 consecutive runs, all passed.stinger_prores_test(1).stinger_api_test(2).stinger_clip_test(3).openapi_test,pipeline_lifecycle_test,vision_mixer_cpu_test,vision_mixer_fx_test,vision_mixer_source_resize_test,shader_validation_test,media_player_control_test,failed_start_teardown_test,bus_signal_watch_balance_test,block_properties_persist_test,api_tests.cargo test --lib: 901 passed.cargo clippy --workspace --all-targets -D warningsand the wasm32 frontend clippy (pre-commit), andcargo fmt --check.stinger_benchin release.Linux CI (GStreamer 1.24.2, llvmpipe from #977): green on
4f20bfa, Check (Linux) green oned896a7;f08da4eis pending.stinger_test(4),stinger_api_test(2),stinger_prores_test(1, ProRes throughavdec_prores),stinger_clip_test(3) andshader_validation_testall ran there. Windows and macOS CI builds are skipped for this PR.Not run:
cargo test --workspaceas a whole locally, and Windows.For the reviewer
Timinggainspin_startand a release-ahead, and an ordinary player's timing is unchanged (unit-tested).set_playliston a stinger source cues the first clip instead of playing it.TriggerTransitionRequest.from_inputandto_inputare no longer required in the schema. This is intentional; the handler still requires them for anything but a stinger. A non-stinger request without them now gets 400 where it used to fail deserialisation with 422.StingerTakeReporthas an optionalwarning, set when the graphic frame due at the cut point arrived late. The take still sendsStingerCompleted, because the program did change.frames_arrivedcounts only frames that arrived in time to air.hold_takes_for_testsandfail_stinger_programming_for_tests(#[doc(hidden)], per flow) make the restart race and a part-way programming failure deterministic. They lock once per take, never in a probe. Thespawn_blockingjoin-failure path takes the same abort as a play error but has no guard: it would need a panic insideplay_at. A third hook,hold_bridge_for_tests, holds a take's first video sample at the Media Player bridge so the early-audio test fails every run without its fix instead of one in four. It is behind aboolfixed when a stinger source's bridge is built, so an ordinary Media Player never reads it. These hooks are not a pattern for other code.Remove), armed when a chain is built and on a rewind cue. A playing clip has no probe on its path.#[doc(hidden)]statics (see above), not a cargo feature that every test run would have to enable.take.rs's classic pad reset becamereset_classic_take_pads, and the post-take bookkeeping inAppState::trigger_transitionbecameafter_vision_mixer_take. Both are shared with the stinger and change no behaviour.Out of scope
audio_outfor the operator to wire.state/mod.rs(about 3200 lines) andvision-mixer.html; the stinger code lives in its own modules.🤖 Generated with Claude Code