perf(whep): convert a video input once, not once per viewer

[srperens]

Supersedes #799. @wagenet's commit is included unchanged, with him as author. On top of it:

• A merge of main. It resolves the one conflict in backend/src/gst/mod.rs: fix(vision_mixer): accept CUDA-memory inputs on the GPU backend #948 added gl_input_front next to gl_bridge, and this PR renames gl_bridge to video_input_bridge, so both modules are kept.

• A fix from review: the converter that video_input_bridge builds from the convert mode now goes through gpu::configure_video_convert, as every other converter does. Without it, the RGBA to NV12 pass that every WHEP viewer depends on ran on one core on macOS, the platform this PR targets. It does nothing on other platforms.

• The converter is always videoconvert, never autovideoconvert. The bridge converts after any gldownload, so the frames are already in system memory and autovideoconvert has no GPU path to gain. Behind gldownload, autovideoconvert also crashes or hangs on GStreamer 1.24.2, the version CI runs. In an Ubuntu 24.04 container with surfaceless EGL, gltestsrc ! GL RGBA ! queue ! gldownload ! autovideoconvert ! NV12 failed 9 of 20 runs. videoconvert in the same place, and autovideoconvert with no GL in the pipeline, both passed 20 of 20. video_input_bridge_gl_test segfaulted on CI in 3 of 3 runs today, both on this branch and on the unchanged perf(whep): convert a video input once, not once per viewer #799 commit that passed yesterday, with the same image, packages and toolchain. The tests no longer have autovideoconvert cases. No test can guard the WHEP Output's choice of converter, because CI has no GPU and its convert mode already picks videoconvert.

Tests run on the merged branch (macOS): cargo test --lib video_input_bridge (19), cargo test --lib whep (30), gl_memory_link_test (7), vision_mixer_cuda_input_test (3), whip_whep_ice_transport_policy_test (9), pipeline_lifecycle_test (3): all passed. The pre-commit hook ran fmt and clippy, both clean. The full cargo test --workspace was not run locally.

---

The original description from #799 follows.

What this changes

webrtcsink builds a full encoding chain per consumer, and each chain starts with its own
videoconvert (PayloadChainBuilder::build → make_converter_for_video_caps). The WHEP Output
block pinned no raw format, so N viewers converted the same frames N times — each into whatever its
encoder's caps preferred, which measured as 16-bit RGBA64_LE for the VideoToolbox encoders and
4:4:4 Y444 for vp9enc.

The block now adapts its own video input from the negotiated caps, in the one-shot CAPS probe that
already downloaded GL memory. gl_bridge becomes video_input_bridge and makes both decisions in
one splice: gldownload first where the input is GL memory, then a converter and an NV12
capsfilter where the format is not already 8-bit 4:2:0. Left alone, deliberately:

• Encoded input. H.264/H.265/VP9/AV1 pass straight through; no converter can sit on that path.
• CUDA, NVMM, D3D11, VA and DMABuf memory. Those consumers encode the memory directly, and a
  videoconvert could not link to it.
• NV12 and I420. Converting one into the other costs a pass over every frame and at best moves
  the per-consumer conversion from one set of encoders to another.
• Formats deeper than 8 bits. Converting them would drop precision the encoder may keep.

CLAUDE.md named gst::gl_bridge as the example of deciding from negotiated caps; it now names
gst::video_input_bridge. devicesrc (upstream #842) takes GL_MEMORY_FEATURE from video_input_bridge, which now exports it.

Why NV12

NV12 is native to the hardware encoders: vtenc_h264/vtenc_h265, NVENC and the VA encoders, with
x264enc listing it alongside I420. The VP9 and AV1 software encoders (vp9enc, av1enc,
rav1enc, svtav1enc) do not accept NV12, so their consumers still convert once each — from NV12
to I420 instead of from RGBA. Per extra consumer, feeding the block each format directly:

fed to the block	VP9 viewer	H.265 viewer (vtenc)
RGBA	26.5% of a core	23.1%
NV12	19.6%	5.6%
I420	17.7%	3.6%

I420 measured slightly cheaper on both of these; NV12 is for NVENC and VA, which were not measured.
I420 input is passed through, not converted.

A mixer-side pin would serve several sinks with one conversion, and on the GPU backend costs nothing
(#797 makes output_format work there). This pin is conditional, so it is a no-op
behind such a mixer. Three WHEP sinks fed from one CPU mixer still convert three times.

Measured

Apple Silicon, 1080p30, RGBA in, two whepclientsrc consumers per run each pinned to one codec, CPU
of the process running the real block (backend/tests/whep_fanout_bench.rs), A/B interleaved, per
extra consumer:

consumer	before	after
H.265, vtenc_h265	23.7%, 39.2% of a core	1.4%, 1.3%
VP9, vp9enc	26.0%, 23.5%	17.2%, 14.9%

The one conversion costs 4 to 8 points of a core, so a single viewer already breaks even. Encoder
input caps confirm the mechanism: before, vtenc_h265 took RGBA64_LE and vp9enc took Y444;
after, NV12 and I420. These are debug builds, where webrtcsink is unoptimised, so absolute
figures may run high. H.264 consumers could not be measured: a whepclientsrc offering only H.264
gets a=inactive here, with or without this change.

Tests

• gst::video_input_bridge, 19 unit tests: 12 on the caps predicates (5 carried over from
  gl_bridge), 3 on what build_adapters returns and in what order, and 4 live pipelines through
  the bridge — RGBA converted, NV12 and I420 untouched, RGBA converted by autovideoconvert.
• backend/tests/video_input_bridge_gl_test.rs pushes real GL memory (gltestsrc, RGBA) through
  the bridge with videoconvert and autovideoconvert, and asserts the splice is
  gldownload ! <converter> ! capsfilter and the peer gets system-memory NV12. On Linux with no
  display it requests a surfaceless EGL context (Mesa software GL); under
  STROM_REQUIRE_GST_PLUGINS=1, as in CI, a missing element or GL context fails instead of
  skipping. An Ubuntu 24.04 container with CI's packages runs that pipeline headless.
• blocks::builtin::whep::tests::video_input_is_converted_before_the_sink builds the real block,
  feeds it RGBA, and asserts whepserversink's video_0 pad negotiates NV12.

Reverts: with needs_format_conversion stubbed to false, the GL test, the RGBA pipeline tests and
the block test fail; with needs_gl_download stubbed, the GL test fails.

Ran on macOS on this head: cargo test --lib (591 passed), the GL test with
STROM_REQUIRE_GST_PLUGINS=1, pipeline_lifecycle_test, openapi_test, clippy -D warnings,
fmt. Linux CI passed on the previous head, which lacked the GL test; this head's CI is its first
Linux run. Not run: Windows, CUDA hosts, the rest of the integration suite.

whep_fanout_bench.rs is #[ignore]d (it needs whepclientsrc and spawns consumer processes) and
guards nothing. It fails a run in which any consumer received no RTP buffers.

Not validated on the rig: no browser-level confirmation.

🤖 Generated with Claude Code