[wagenet]

What this changes

webrtcsink builds a full encoding chain per consumer, and each chain starts with its own
videoconvert (PayloadChainBuilder::build → make_converter_for_video_caps). The WHEP Output
block pinned no raw format, so N viewers converted the same frames N times — each into whatever its
encoder's caps preferred, which measured as 16-bit RGBA64_LE for the VideoToolbox encoders and
4:4:4 Y444 for vp9enc.

The block now adapts its own video input from the negotiated caps, in the one-shot CAPS probe that
already downloaded GL memory. gl_bridge becomes video_input_bridge and makes both decisions in
one splice: gldownload first where the input is GL memory, then a converter and an NV12
capsfilter where the format is not already 8-bit 4:2:0. Left alone, deliberately:

• Encoded input. H.264/H.265/VP9/AV1 pass straight through; no converter can sit on that path.
• CUDA, NVMM, D3D11, VA and DMABuf memory. Those consumers encode the memory directly, and a
  videoconvert could not link to it.
• NV12 and I420. Converting one into the other costs a pass over every frame and at best moves
  the per-consumer conversion from one set of encoders to another.
• Formats deeper than 8 bits. Converting them would drop precision the encoder may keep.

CLAUDE.md named gst::gl_bridge as the example of deciding from negotiated caps; it now names
gst::video_input_bridge. devicesrc (upstream #842) takes GL_MEMORY_FEATURE from video_input_bridge, which now exports it.

Why NV12

NV12 is native to the hardware encoders: vtenc_h264/vtenc_h265, NVENC and the VA encoders, with
x264enc listing it alongside I420. The VP9 and AV1 software encoders (vp9enc, av1enc,
rav1enc, svtav1enc) do not accept NV12, so their consumers still convert once each — from NV12
to I420 instead of from RGBA. Per extra consumer, feeding the block each format directly:

fed to the block	VP9 viewer	H.265 viewer (vtenc)
RGBA	26.5% of a core	23.1%
NV12	19.6%	5.6%
I420	17.7%	3.6%

I420 measured slightly cheaper on both of these; NV12 is for NVENC and VA, which were not measured.
I420 input is passed through, not converted.

A mixer-side pin would serve several sinks with one conversion, and on the GPU backend costs nothing
(#797 makes output_format work there). This pin is conditional, so it is a no-op
behind such a mixer. Three WHEP sinks fed from one CPU mixer still convert three times.

Measured

Apple Silicon, 1080p30, RGBA in, two whepclientsrc consumers per run each pinned to one codec, CPU
of the process running the real block (backend/tests/whep_fanout_bench.rs), A/B interleaved, per
extra consumer:

consumer	before	after
H.265, vtenc_h265	23.7%, 39.2% of a core	1.4%, 1.3%
VP9, vp9enc	26.0%, 23.5%	17.2%, 14.9%

The one conversion costs 4 to 8 points of a core, so a single viewer already breaks even. Encoder
input caps confirm the mechanism: before, vtenc_h265 took RGBA64_LE and vp9enc took Y444;
after, NV12 and I420. These are debug builds, where webrtcsink is unoptimised, so absolute
figures may run high. H.264 consumers could not be measured: a whepclientsrc offering only H.264
gets a=inactive here, with or without this change.

Tests

• gst::video_input_bridge, 19 unit tests: 12 on the caps predicates (5 carried over from
  gl_bridge), 3 on what build_adapters returns and in what order, and 4 live pipelines through
  the bridge — RGBA converted, NV12 and I420 untouched, RGBA converted by autovideoconvert.
• backend/tests/video_input_bridge_gl_test.rs pushes real GL memory (gltestsrc, RGBA) through
  the bridge with videoconvert and autovideoconvert, and asserts the splice is
  gldownload ! <converter> ! capsfilter and the peer gets system-memory NV12. On Linux with no
  display it requests a surfaceless EGL context (Mesa software GL); under
  STROM_REQUIRE_GST_PLUGINS=1, as in CI, a missing element or GL context fails instead of
  skipping. An Ubuntu 24.04 container with CI's packages runs that pipeline headless.
• blocks::builtin::whep::tests::video_input_is_converted_before_the_sink builds the real block,
  feeds it RGBA, and asserts whepserversink's video_0 pad negotiates NV12.

Reverts: with needs_format_conversion stubbed to false, the GL test, the RGBA pipeline tests and
the block test fail; with needs_gl_download stubbed, the GL test fails.

Ran on macOS on this head: cargo test --lib (591 passed), the GL test with
STROM_REQUIRE_GST_PLUGINS=1, pipeline_lifecycle_test, openapi_test, clippy -D warnings,
fmt. Linux CI passed on the previous head, which lacked the GL test; this head's CI is its first
Linux run. Not run: Windows, CUDA hosts, the rest of the integration suite.

whep_fanout_bench.rs is #[ignore]d (it needs whepclientsrc and spawns consumer processes) and
guards nothing. It fails a run in which any consumer received no RTP buffers.

Not validated on the rig: no browser-level confirmation.

🤖 Generated with Claude Code

[srperens]

Noted as a draft, so I am holding off on a full review until it is marked ready for
review. If you want one before then, say so here or request me as a reviewer.

[srperens]

Verdict: Comment — the diff is sound and its covering tests genuinely executed in CI (I read the job log, not just the tick). What stops this being an Approve is not in the diff: #682 is still asking where this decision belongs, and merging this answers part of that question by landing.

Claims

Claim	Verdict	Evidence
The new tests ran rather than skipping past their guard	CONFIRMED	job 104575573305 at 6cd1ec1: 19 gst::video_input_bridge::tests::* lines print ... ok, and test gl_rgba_reaches_the_peer_as_system_memory_nv12 ... ok in 1.67s. No SKIP: line — the DISPLAY/GL guard did not fire on the Linux runner
The probe is per-event, not per-buffer	CONFIRMED	backend/src/gst/video_input_bridge.rs:140 — ` src_pad.add_probe(gst::PadProbeType::EVENT_DOWNSTREAM, move
No strong pipeline/element reference is captured in the closure	CONFIRMED	the closure captures two Strings; backend/src/gst/video_input_bridge.rs:234 — fn splice(src_pad: &gst::Pad, adapters: &[gst::Element]) -> Result<(), String> { — takes the bin from the pad at call time and keeps it local
This adds a third consumer of the global converter mode	CONFIRMED	backend/src/blocks/builtin/whep.rs:1584 — let convert_factory = video_convert_mode().element_name();
The CPU measurements	EXTERNAL	backend/tests/whep_fanout_bench.rs:69 — #[ignore] — so CI never runs it. Apple Silicon debug builds, as the body states; I take the numbers at face value and cannot reproduce them here

Diagnosis — Right layer and right mechanism. The cost is per consumer because webrtcsink builds a chain per consumer, so pinning once upstream of the fan-out is where the saving is, and the loop is CLOSED — one splice on the first CAPS event, then the probe removes itself. Coverage is BOUNDED and the bound is stated honestly in the body: encoded input, non-GL GPU memory, NV12/I420 and >8-bit are left alone, and VP9/AV1 consumers still convert once each. Resolving convert_factory outside the probe rather than inside is the right call and the comment says why — a panic there would be on a streaming thread.

Overlaps — this is the whole of my verdict. #682 is an open issue whose Ask: is unanswered as of my comment there today, and one of its options is literally "wait for #799 and #731, change nothing here". The conflict is concrete, not thematic:

1. This PR replaces the runtime entry point that #682's phase 2 proposes to extract into a shared decide(), and folds a second decision (pixel format) into the same splice. Phase 2 would then extract from a file this PR has just rewritten.
2. It adds the call site above to video_convert_mode() — the API #682 phase 4 proposes to delete and #731 proposes to extend.
3. It re-points CLAUDE.md's memory-format convention from gst::gl_bridge to gst::video_input_bridge, narrowing the documented reference implementation to one block while #682 is deciding whether that rule should be shared.

None of that says the diff is wrong — it may well be the right first step. It says a human picks the order, and #682 is where that choice is already teed up. I am not asking for a change here.

Radius — SHARED: one caller, but it changes what every WHEP Output viewer is fed, across input shapes the bench did not cover (NVENC and VA hosts are explicitly unmeasured).

Repo rules — none violated that I found. EVENT_DOWNSTREAM rather than BUFFER, no captured pipeline reference, queues untouched, no emoji in the log macros, and the CLAUDE.md edit is a rename of an example, not a change to a rule.

Confidence: HIGH

[srperens]

Thanks @wagenet. This is carried forward in #949. Your commit is included unchanged, with you as author. On top of it, #949 merges main (one conflict in gst/mod.rs with #948's new gl_input_front) and adds one fix from review: the new converter now goes through gpu::configure_video_convert, so it runs on all cores on macOS like the other converters.

Closing in favour of #949.

Claude (AI), on behalf of @srperens.