Skip to content

feat(stinger): web page stingers from an HTML Input - #1020

Closed
wagenet wants to merge 24 commits into
Eyevinn:feat/stingerfrom
wagenet:wagenet/web-stingers-on-1010
Closed

feat(stinger): web page stingers from an HTML Input#1020
wagenet wants to merge 24 commits into
Eyevinn:feat/stingerfrom
wagenet:wagenet/web-stingers-on-1010

Conversation

@wagenet

@wagenet wagenet commented Oct 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Written by Claude (AI) at @wagenet's request. Stacked on #1010: the base is feat/stinger.

Refs #1010

Problem

#1010 plays stingers from clips only. A stinger is the graphic that covers the frame while the program switches beneath it. Many stinger graphics are made as web pages, and Strom already renders pages through the HTML Input block. A clip take pins its first frame in advance. A page starts on a trigger, and Chromium paints its first frame some time after that, so the take cannot be planned before it starts.

Change

An HTML Input with Stinger Page on, wired to the mixer's stinger input, is a stinger source. Duration, cut point and mix are properties on the block. A take works in four steps:

  1. It raises the graphic pad before the trigger. The page is transparent at rest, so nothing shows yet.
  2. It sets the page's URL fragment to #strom-take-<id> through cef_pages::load_url. The page starts on hashchange.
  3. A probe on cefsrc's src pad finds the first frame of new content after the trigger. A repeat shares the previous buffer's memory and does not count. If that frame does not come within 100 ms, the take is timed from the trigger plus 30 ms instead (20 ms on a paint-only gstcefsrc).
  4. The take is programmed from the output frame that page frame goes on air in. It then reports and finishes as a clip take does.

The page plays as classic premultiplied: the GPU mixer blends it correctly, and the CPU mixer warns about dark edges, as #1010 already does.

The seam is 69afb01. The take can start with the fill pad already up (raise_stinger_fill, StingerTake::fill_raised), and stinger_take_claimed is split into take_on_grid and announce_stinger_take. The clip path is unchanged. The commit is self-contained, in case you would rather take it into #1010.

Which output frame a page frame lands on. The mixer drops queued buffers that end before an output frame and takes the oldest one left. So a page at or below the mixer's rate is on air in the frame its timestamp falls in, and a faster page from the next frame boundary. I rejected rounding to the nearest frame. cefsrc's phase against the mixer grid is fixed per process, and the nearest frame was one frame out in every run whose phase fell in the second half of a frame.

Also in this PR:

  • A take runs in its own task, so a dropped request cannot leave the mixer claimed.
  • A page URL with a #fragment of its own is refused, on the stored URL and on the one the page shows. The take's own fragment is accepted.
  • StingerState gains source_kind (clips or page). openapi.json is updated.
  • The panel shows a page source and hides the clip tools for it.

Evidence

Measured on Linux arm64 Docker with the production gstcefsrc pin (b633408, fixed rate), software CEF, and live sources on both mixer inputs. GPU runs used GLX/llvmpipe on Xvfb. The test page counts frames from its first frame on air to the cut.

Claim Evidence
The cut lands on the cut point's frame (500 ms = 15 frames) 15 frames on every take: CPU 30 fps page, GPU, and a 60 fps page into a 30 fps mixer. 30+ takes each over several processes. Reverting each of the following fails: the on-air rule (14 or 16), the novelty check (14), raising the fill before the trigger (14).
A page that paints nothing visible at first is timed from the trigger It cuts 7-46 ms early by the page's own clock. Reverted (anchor on the first frame however late): +294 to +360 ms.
A dropped take request frees the mixer a_dropped_take_request_still_frees_the_mixer. Reverted: "the dropped take never finished".
Too-early cut points, #fragment URLs, paint-only first frame Unit tests in stinger::web, each failing when reverted.
A page is the library's one entry, cue does nothing, library operations are refused stinger_page_api_test, no cefsrc needed. Reverting the source kind, the cue or the refusal each fails it.
Take to air 63-92 ms from the request to the page's first frame on air.
Clip takes unchanged #1010's stinger_test, stinger_clip_test and stinger_api_test pass.

Tests

For the reviewer

Out of scope

  • Paint-only gstcefsrc (build(gstcefsrc): CEF 154 and gstcefsrc master under the HTML Input block #955 / cef-154). The stinger pads drop a held frame after two frames, so a page that holds a still frame across the cut can drop off air. The take warns. livesync on HTML Input belongs to that upgrade.
  • A page that keeps drawing while at rest cuts 1-2 frames early. The block's help says a stinger page must stop drawing at rest, and there is no warning.
  • A fixed-rate gstcefsrc that sends nothing for over 100 ms across a take cuts one frame late if it resumes on a new paint. It is on time if it resumes on its rest frame.
  • Matte variants for a page: not supported.

🤖 Generated with Claude Code

srperens and others added 24 commits October 6, 2026 18:48
@srperens @claude
feat(vision-mixer): stinger transitions: classic, track matte and mas…
3d9c200 
…k only

A stinger plays a clip over the program while the source changes beneath
it. Three variants, after ATEM, vMix and OBS:

- Classic: a graphic with alpha; the program cuts or mixes beneath it at
  a cut point (by default where the graphic covers most, found by
  analysing the clip).
- Track matte: graphic and matte in one file, side by side or stacked
  (the OBS layout). The matte switches the program pixel by pixel.
- Mask only: the clip is a matte, an animated wipe.

The matte variants need the GPU mixer and play as classic on the CPU one.

The source is a Media Player in stinger mode wired to the mixer's new
stinger input; its playlist is the library and its block holds per-clip
settings. A cue parks a clip on its first frame. A take pins the running
time its first frame lands at, programs the whole take into the mixer as
control-binding keyframes on that timeline, and only then lets the clip
go, so nothing waits on wall clock or holds the mixer's output. The
matte composites inside glvideomixer: the matte pad takes the matte off
the destination alpha (reverse-subtract) and the incoming source blends
by it for the clip's frames.

Clip frames leave their clocksync ahead of time and are stamped half an
output frame into their frame, so a millisecond-rounding container cannot
move one into its neighbour's. A stinger source asks for system memory,
so a hardware decoder keeps alpha (VideoToolbox gives AYUV64 for ProRes
4444), and the stinger input converts what GL upload cannot take.

Operator panel: STING type, the clip library with cue, per-clip settings,
take reports, and example clips Strom renders itself (one per variant).
API: stinger state, cue, take, clip settings, examples; takes also run
through the transition endpoint. Events: StingerCued, StingerStarted,
StingerCompleted (with measurements), StingerFailed.

Builds on ideas from Eyevinn#755 by wagenet: opt-in sources, parking, events,
degrading to a cut when a clip cannot play.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@srperens @claude
feat(stinger): widen the example blade
9c0a8aa 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@srperens @claude
fix(stinger): keep the example blade's trails out of its matte
9eb3651 
The blade's glow and trails were drawn unclipped, so near the right edge
they painted into the matte half of the side-by-side frame, and the
trails stayed on screen after the blade had left. Clip the graphic to its
half, and scale the trails with the blade's speed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@srperens @claude
feat(stinger): library API on the mixer, file guards, take ids
4b1c87f 
What an outside client needs to drive stingers without knowing the Media
Player behind them:

- POST/DELETE .../stinger/clips: add a clip to the library (idempotent,
  a local file must exist) or remove one with its settings, on the mixer.
- Calls that address a clip by index (cue, take, settings, remove) take
  an optional `file`; when the library has changed under the client, the
  call fails with 409 instead of acting on another clip.
- Every take has an id: in the take response and in StingerStarted,
  StingerCompleted (report) and StingerFailed. The take response also
  names the file played.
- from_input/to_input are optional in the transition request: a stinger
  always takes PGM to PVW; other transitions still require them (400).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@srperens @claude
fix(stinger): detect a clip's layout from the clip alone
e6aac42 
Side by side and stacked were read against the program's aspect, so the
same file could detect differently on a 4:3 or vertical production. A
track-matte clip is two standard frames (16:9, 4:3 or 9:16) next to or
above each other, whatever the program runs; the analysis cache no
longer keys on the program either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@srperens @claude
test(stinger): render only the edge frames in the blade spill test
2db2645 
Every frame of the 3840x1080 example in a debug build took seconds of CPU
beside the lib's timing-sensitive tests. The spill can only happen as the
blade nears the right edge, so check those frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@srperens @claude
fix(stinger): warm the mixer up at cue; count late matte frames
2aa94f5 
On Linux CI (GStreamer 1.24.2, llvmpipe) a take that switched to a clip
of another size showed the matte one frame late: the stinger branch
renegotiated and the matte shader recompiled on the take's first frames.
A cue now sends one blank frame in the clip's format through the
consumer, so that happens at cue; the stinger pads are hidden between
takes, so it never shows.

The take report counted lateness on the graphic pad only; the matte pad,
with a shader pass of its own, now counts too. The tests read the clip's
frame count from its analysis, since 1.24.2's FFV1 decoder drops the
last frame of a stream, and print the whole take when a frame is off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@srperens @claude
test(stinger): test clips in PNG, not FFV1
f2ca310 
GStreamer 1.24.2's FFV1 decoder (Linux CI) drops a stream's last frame
now and then, so a clip's analysis and its playback could disagree on
its length and a take ran a frame longer than the test expected. PNG
frames in QuickTime are lossless, keep alpha, and decode through libpng
without frame threads. The FFV1 examples' own test allows the missing
frame.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@srperens @claude
test(stinger): show the mixer's pads mid-take when a matte frame is off
def809b 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@srperens @claude
test(stinger): add caps, geometry and shader uniforms to the mid-take…
5bb722d 
… pad dump

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@srperens @claude
test(stinger): show the program's middle row half way through a matte…
939f360 
… take

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@srperens @claude
fix(stinger): keep the matte shader out of FX passthrough
128b3cb 
make_glshader now starts every glshader in passthrough (Eyevinn#1007). The matte
is not an FX slot and must always render: in passthrough the clip reached
the mixer as it is, and its own alpha became the matte. The GPU stinger
test fails without this.

Also report the take with the CPU test's cut assertion, and drop the
pad-dump eprintln (the dump stays in the failure message).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@srperens @claude
fix(stinger): place step keyframes half a frame before their frame
4f20bfa 
The mixer stamps output frames from a rounded origin, so a frame's
timestamp can be a nanosecond either side of FrameGrid's. A step key
placed exactly on the frame then acted on that frame or the next by
chance: depending on the take's phase, the graphic went on air a frame
late while the cut landed on time. cpu_stingers_play_classic failed 6 of
8 local runs with 'cut at frame 13'; with the keys half a frame early it
passed 8 of 8. The linear mix ramp keeps its exact times.

The test rig now collects frames over a span of stream time, not wall
time: on a slow runner the debug-build mixer falls behind real time and
the window ended before the clip did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wagenet @claude
refactor(stinger): plan a take from a start reported at runtime
69afb01 
A clip take knows where its first frame lands before it starts: the Media
Player pins it. A source that starts on a trigger only says when it
started afterwards, so its take is programmed late, with the graphic pad
already up.

- `PipelineManager::raise_stinger_fill` stages the pads as a take does
  and raises the graphic pad. `program_stinger` with
  `StingerTake::fill_raised` skips the staging and keeps the pad up until
  the take's end.
- `stinger_take_claimed` builds its take with `take_on_grid` and hands it
  to `announce_stinger_take`, which reports and finishes any take. The
  re-park after a take is skipped when there is no clip to park.

The clip path does the same thing in the same order as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wagenet @claude
feat(stinger): find where a stinger page starts from its first frame
bf77f97 
A page starts its animation on a trigger, and Chromium paints its first
frame some time later. `stinger::web` holds what a page take needs to
time itself from that frame:

- `WebAnchor`: a probe on the page's cefsrc output that records the first
  frame of new content after the take. A GAP repeat, or a fixed-rate
  gstcefsrc re-sending its frame, shares the previous buffer's memory and
  does not count. Atomics only, removed once the take is planned.
- `web_stinger_start`: the first frame if it came within 100 ms of the
  take, otherwise the take plus the plugin's delivery delay: 30 ms on the
  fixed-rate production gstcefsrc (measured on cefsrc's own output), 20 ms
  on one that emits as it paints.
- `web_first_output_frame`: the output frame that page frame first goes
  on air in. The mixer drops queued buffers that end before an output
  frame and takes the oldest left, so a page at or below the mixer's rate
  is on air in the frame its timestamp falls in, and a faster one from
  the next frame boundary. Neither is the nearest frame: cefsrc's phase
  against the mixer's grid is fixed per run, and nearest is a frame out
  whenever that phase is in the second half of a frame.
- `WebStingerSettings`: duration, cut point and mix from the HTML Input
  block. A cut point earlier than the grace period plus two frames
  (167 ms at 30 fps) cannot be programmed in time and is refused, as is
  one at or past the end.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wagenet @claude
feat(stinger): take a stinger from an HTML Input page
9784e8a 
An HTML Input in stinger mode on the mixer's stinger input is a page
stinger. Its settings are properties of the block, and it is the one
entry of its library.

A take raises the graphic pad (the page is transparent at rest), triggers
the page by setting its URL fragment to #strom-take-<id> through
`cef_pages::load_url`, waits for the page's first frame, and programs the
take from the output frame that frame goes on air in. Reporting and
finishing are a clip take's: StingerStarted, StingerCompleted,
StingerFailed and the take report. A take that cannot program cuts
instead, as a clip take does.

The graphic pad goes up before the trigger because the take is
programmed after the page has started: a fill keyframe at the page's
start would already be behind the mixer, and the page's first frame
would not show.

- A page is a classic stinger with premultiplied alpha: the GPU mixer
  blends it correctly, the software mixer warns that edges come out dark.
- On a gstcefsrc that emits only as the page paints, the take warns: the
  stinger pads let a held frame go after two frames.
- Under remote control the trigger loads asynchronously; the take still
  anchors on the first frame, and logs that the fallback may be late.
- Library operations (add, remove, settings, examples) are refused for a
  page, so a page cannot ask for a matte variant. Cue is a no-op.
- The take report expects the page frames from when its frame count
  starts, which is after the page's first frame has reached the mixer.

`StingerState` gains `source_kind`, so a client can tell a clip library
from a page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wagenet @claude
feat(html_input): stinger page settings, and the panel's page source
0a12385 
The HTML Input block gains Stinger Page, Stinger Duration, Stinger Cut
Point and Stinger Mix. The vision mixer panel shows a page source as its
one entry, hides the clip tools for it, and stops polling for an analysis
a page never gets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wagenet @claude
test(stinger): page stingers against a real cefsrc
f5d3d17 
Gated by STROM_REQUIRE_CEF and run on Linux only, each over
STINGER_WEB_TAKES takes (default 10), with live sources on the mixer:

- the program cuts 15 frames (500 ms) after the page's first frame on
  air: on the software mixer, on the GPU mixer, and for a 60 fps page on
  a 30 fps mixer, whose takes start on both halves of an output frame;
- every page frame of a take reaches the mixer in time;
- a page that opens on 300 ms of unchanged pixels is timed from the take,
  and cuts within two frames of 500 ms by its own clock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wagenet @claude
fix(stinger): finish a take whose request is dropped
31d4ab4 
A take claims the mixer, and only its own ending releases the claim. A
page take waits for the page between the two, so a request dropped then
(the client went away, a proxy timed out) left the mixer claimed: every
later stinger and every cut was refused until the flow restarted, with
the graphic pad up. The take now runs in a task of its own that the
request awaits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wagenet @claude
fix(stinger): guard a stinger page's URL and first frame
db9b09e 
- A page URL with a fragment of its own is refused: the take replaces the
  fragment, which moves a page that routes by it off its route.
- On a gstcefsrc that sends only paints, the first buffer after the take
  is the page's start. It was spent as the baseline instead, which put
  every cut a frame late after a 100 ms wait for a buffer that never
  comes.
- A page that paints while at rest is noticed before the take and warned
  about: its first frame after the take can be a paint of its rest state,
  which puts the cut a frame or two early. The block's help says a
  stinger page must stop drawing at rest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wagenet @claude
fix(stinger): wait for one page frame; check paints at rest after a take
967676f 
- A take waits for one frame from a fixed-rate gstcefsrc again, not two,
  which cost about a page frame on every take.
- "A buffer with nothing before it is new" applies to a paint-only
  gstcefsrc only. On a fixed-rate one that sent nothing until the take,
  its first buffer is a repeat of the rest frame, and counting it as the
  page's start put the cut a frame or two early.
- Whether a page paints at rest is judged after the take instead. The
  probe stays on and counts the page's new frames from a page frame past
  its duration until the earliest the next take could trigger it (the
  mixer's latency past the take's finish point). Two or more warn; one
  is the page clearing itself as it ends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wagenet @claude
fix(stinger): refuse a take on a page showing a #fragment of its own
c6a894e 
The URL check covered only the URL stored on the block. A page moved live
through the element endpoint or MCP to a URL with a fragment still lost
it to the take. The take now checks the URL the page shows, which is
what it rewrites. A take's own fragment from an earlier take is
accepted, and surrounding space is trimmed first, as it is before the
page loads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wagenet @claude
refactor(stinger): drop the paints-at-rest warning
7913774 
The check guessed when the next take could start, and a take chained
straight after the last one beat the guess, so a well-behaved page was
warned about. For a log line, that is not worth a probe that outlives the
take and a mixer latency query. The block's help says a stinger page must
stop drawing at rest; a page that does not cuts a frame or two early.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wagenet @claude
test(stinger): the API on a page stinger source
b9d2b02 
A page is the library's one entry (classic, premultiplied, its cut
point), cueing it succeeds and does nothing, the clip-library operations
are refused with the reason, and a page URL with a #fragment of its own
is reported. Runs without cefsrc, so CI runs it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@srperens

srperens commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

Noted as a draft, so I am holding off on a full review until it is marked ready for review. If you want one before then, say so here or request me as a reviewer.

@srperens srperens mentioned this pull request Oct 7, 2026
Open
@srperens
srperens force-pushed the feat/stinger branch 2 times, most recently from 7bea68f to 98ffc4f Compare October 7, 2026 10:49
@srperens srperens mentioned this pull request Oct 7, 2026
Merged
@srperens
srperens deleted the branch Eyevinn:feat/stinger October 8, 2026 09:51
@srperens srperens closed this Oct 8, 2026
@wagenet wagenet mentioned this pull request Oct 8, 2026
Open
@wagenet

wagenet commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Claude (AI), for @wagenet: this closed when its base #1010 merged and the branch was deleted. The same commits, rebased onto main, are in #1056.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@wagenet @srperens