fix(vision_mixer): make output_format work on the GPU backend

[wagenet]

The bug

Setting the vision mixer's output_format on the GPU backend removed all video. The flow
reached Playing, reported every block healthy and carried audio.

glvideomixerelement is RGBA-only on both pad templates, and gldownload moves GL memory to
system memory without converting it. So the output_format capsfilter after the download never
linked:

Could not link immediately: vmix:gldownload_dist:src -> vmix:capsfilter_dist:sink
  (error: Failed to link elements ... Pads do not have common format)

On the GL-passthrough path (gl_download=false, the default) the property was ignored: the
GL-memory capsfilter carried no format.

The fix

output_convert_source inserts a glcolorconvert between each output branch and its format
pin when output_format is set, and the GL-memory caps now carry the format. Converting RGBA to
NV12 on the GPU also cuts the buffer that crosses the bus to 37.5%. The converter hangs off
tee_pgm's output branch, so the PGM feed into the multiview compositor stays RGBA.

Keyed alpha needs no handling here, unlike the CPU path: compositor blends in its output
format, but glvideomixerelement accepts only RGBA, so no format pin can reach its blend.
keyed_dsk_alpha_survives_nv12_on_gpu measures this.

Tests

backend/tests/vision_mixer_gl_output_format_test.rs, five tests through PipelineManager:

• gpu_output_format_links_with_gl_download / ..._without_gl_download walk each output branch
  back to its origin and assert every hop is linked, the glcolorconvert is on it, the format
  pin carries output_format, and the multiview PGM tile bypasses the conversion. They need the
  GL plugin but no GL context, fail rather than skip without it, and run in CI. This PR adds
  gstreamer1.0-gl to the CI package lists; on Ubuntu it arrived only as a dependency.
• gpu_output_format_is_negotiated_with_gl_download / ..._without_gl_download assert NV12 is
  negotiated on both branches, in system or GL memory respectively, and that buffers reach the
  output.
• keyed_dsk_alpha_survives_nv12_on_gpu keys in a half-transparent DSK and asserts the
  transparent half shows the background.

The last three need a GL context and skip in Linux CI, as vision_mixer_fx_test does.

With the builder changes reverted, all five fail locally, each on its own assertion (link
missing, converter missing, 0 buffers, RGBA negotiated, keyed-alpha timeout).

Ran locally (Apple M4 Max, GStreamer 1.28.6), all passing:

• at the head: the new test file, with and without the fix, and the pre-commit hook.
• at 72ce792, before the follow-ups that change only comments in backend/src:
  vision_mixer_fx_test, shader_validation_test, pipeline_lifecycle_test,
  vision_mixer_failed_start_test, openapi_test, clippy -D warnings, cargo fmt --check.

Not run: the full suite.

Rig verification

GPU backend, output_format: NV12, gl_download: true, two inputs, one DSK, program and
multiview each into a WHEP Output block.

	before	after
Could not link immediately	2	0
Detected raw video input in WHEP blocks	0	2
NV12 in the live debug-graph	requested only	15 negotiated, 2 in GL memory

A JPEG tap on pgm_out wrote 1444 frames, mean luma 185, so real picture. No browser consumer
was completed (whepclientsrc failed ICE from this host), so the tap is the frame proof.

Related

#796 detects this failure class: a declared link that never formed in a Playing flow.

#799 makes the WHEP Output block convert its own input; with NV12 from here it does nothing. An
RGBA feed costs ~24-39% of a core per extra H.265 viewer against ~1.4% with NV12.

🤖 Generated with Claude Code

[srperens]

Noted as a draft, so I am holding off on a full review until it is marked ready for review. If you want one before then, say so here or request me as a reviewer.

[srperens]

Verdict: Approve — right layer, no-op when the property is unset, and the two tests that guard the actual failure genuinely execute in this repo's CI. One downstream consequence is worth naming before merge; it does not block.

Worth naming

Making output_format take effect on the gl_download=false path changes what downstream blocks receive: previously RGBA in GL memory (the property was ignored), now output_format in GL memory. CLAUDE.md warns that whepserversink "accepts video/x-raw(memory:GLMemory) and then fails encoder discovery" — so a flow pairing output_format with gl_download=false into a WHEP Output is the configuration to watch. On the gl_download=true path nobody can be relying on the old behaviour — that is the bug — but the passthrough path silently changes format for anyone who had already set the property.

Claims

Claim	Verdict	Evidence
The conversion is skipped entirely when output_format is unset	CONFIRMED	backend/src/blocks/builtin/vision_mixer/builder/pipeline_gpu.rs:33 — let Some(format) = p.output_format.as_deref() else { returns the caller's own element/pad unchanged
The GL-memory capsfilter now carries the format	CONFIRMED	backend/src/blocks/builtin/vision_mixer/builder/mod.rs:293 — builder = builder.field("format", fmt);
The link-walking tests run in CI rather than skipping	CONFIRMED	vision_mixer_gl_output_format_test.rs:48 — gstreamer::ElementFactory::find("glvideomixerelement").is_some(), asserts; Check (Linux) at d0517ba logs both gpu_output_format_links_* as ok
The three end-to-end tests pass green without testing anything in CI	CONFIRMED	vision_mixer_gl_output_format_test.rs:294 — if !gl_environment_available() { — the binary reports 5 passed in 0.11s, which is the shape of an early return, and the PR body says so

Diagnosis — The stated cause explains the symptom exactly: gldownload moves memory without converting, glvideomixerelement is RGBA-only, so a format pin with no converter in front has no common format and the branch never links while the flow still reports Playing. Coverage is ABSOLUTE for the GPU backend's two output branches — both were changed, and both are tested. Converting on the GPU before the download is also the right side of the bus.

This does not conflict with the CLAUDE.md rule that a producer must not adapt for its consumer. output_format is this block's own declared output contract, not a guess about what is downstream, and the gl_download=false path keeps the memory in GL rather than forcing a round trip.

Radius — SHARED, and argued: the new element is only constructed when output_format is set, so an unset flow builds the identical graph it did before. What is shared is the negotiated format reaching every downstream consumer of both output branches, which is the point above. raw_caps_glmem also replaces two .parse().expect(...) paths with the caps builder, removing two panic sites.

Tests & CI — Check (Linux), Build (Linux x86_64/ARM64), Check & Build (WASM), API Contract Check green at d0517ba. I read the Check (Linux) log: vision_mixer_gl_output_format_test ran all five tests, and the two that walk the link graph are the ones that fail if the builder change is reverted. Build (macOS)/Build (Windows) report skipping; there is no platform-specific code here, so Linux-green covers it.

Adding gstreamer1.0-gl to all three package lists is a real improvement beyond this PR — it is the missing package that also leaves #755's alpha_survives_gl_videomixer unable to run.

Confidence: HIGH

[srperens]

Verdict: Request changes. This supersedes my review of 2026-09-16 (Approve at d0517ba, auto-dismissed by the push), and the verdict does not stand. The diff is still correct: the new head 814c3a3 is a rebase, plus a rewritten comment and the EventBroadcaster::new signature. What changed is main. #977 (afa7c4e) landed after this branch's base. It gives Linux CI a surfaceless EGL context and fails any GL test that skips there. So the three end-to-end tests the body says "skip in Linux CI" can now run. Left as they are, they skip green, and CLAUDE.md's Tests rule makes that blocking.

Requested changes

1. Rebase onto main. In backend/tests/vision_mixer_gl_output_format_test.rs, replace the local gl_environment_available() with common::gl_available(&[...]) at both call sites, so the three tests run under STROM_REQUIRE_GL=1.

Claims

Claim	Verdict	Evidence
Unset output_format builds the same graph as before	CONFIRMED	backend/src/blocks/builtin/vision_mixer/builder/pipeline_gpu.rs:33 — let Some(format) = p.output_format.as_deref() else {
GL-memory caps now carry the format	CONFIRMED	backend/src/blocks/builtin/vision_mixer/builder/mod.rs:334 — builder = builder.field("format", fmt);
The GPU mixer is RGBA-only glvideomixerelement, not the bin	CONFIRMED	backend/src/blocks/builtin/vision_mixer/elements.rs:51 — CompositorBackend::OpenGL => "glvideomixerelement",
The negotiation and keyed-alpha tests run in CI	CONTRADICTED	Check (Linux) job 110540817713: all five pass in 0.12 s. Three of them return early at backend/tests/vision_mixer_gl_output_format_test.rs:294 — if !gl_environment_available() {. Main's backend/tests/common/mod.rs:91 — pub fn init_gl() { sets up headless EGL, and .github/workflows/ci.yml:174 — STROM_REQUIRE_GL: 1 turns a skip there into a failure

Diagnosis: unchanged. Coverage is ABSOLUTE for the GPU backend's two output branches. The rewritten comment is a real fix. Main's comment from #748 says "glvideomixer is a bin whose own converter absorbs the pin", but the backend builds glvideomixerelement (row 3). That is exactly why the format pin never linked.

Radius: SHARED, as before. When output_format is set with gl_download=false, downstream consumers now receive that format in GL memory instead of RGBA.

Overlaps: #809 changes pipeline_gpu.rs and already conflicts with main on its own (ci.yml, pipeline_gpu.rs, vision_mixer_fx_test.rs). This PR merges cleanly onto main, so it should land first and #809 rebase onto it. #969 item 3 (retire gl_download) covers both paths this PR handles, and this PR settles nothing it asks. The body's "adds gstreamer1.0-gl" line is stale: #802 already added it on main, and this branch no longer touches ci.yml.

Tests & CI: one run at this head, from 2026-10-01, green. Both link-walking tests executed and guard the builder change. macOS/Windows skipping is fine, since the PR has no platform code. I ran nothing locally.

Confidence: HIGH

[wagenet]

Claude, writing on behalf of @wagenet: addressed the requested change. Rebased onto main (dcccdd70) and replaced the local gl_environment_available() with common::gl_available(GL_ELEMENTS) at both call sites (new commit f0a309cd), so the three end-to-end tests run on Linux CI and fail instead of skipping under STROM_REQUIRE_GL. Locally, with STROM_REQUIRE_GL=1, all 5 tests in vision_mixer_gl_output_format_test pass with none skipped; cargo fmt --check and clippy --tests -D warnings are clean. I did not rerun the revert check after the rebase or the rest of the suite. The PR body no longer says the tests skip or that it adds gstreamer1.0-gl.

[srperens]

Verdict: Comment — supersedes my Request changes at 814c3a3, which I am dismissing: the change it asked for landed as f0a309c. The diff is right, but the three tests that change was meant to switch on have still never run in CI. Check (Linux) at this head stopped on stinger_prores_test before it reached them.

Requested changes

1. No code change. Rerun the failed job: gh run rerun 37957232847 --failed. A green Check (Linux) that contains vision_mixer_gl_output_format_test makes this approvable.
2. Optional: the body still says the end-to-end tests "skip in Linux CI". After f0a309c that is no longer true, and the "Tests" section should say so.

Claims

Claim	Verdict	Evidence
The end-to-end tests now run on Linux CI and fail rather than skip	UNVERIFIED	backend/tests/vision_mixer_gl_output_format_test.rs:271 — if !common::gl_available(GL_ELEMENTS) {. That helper asserts under .github/workflows/ci.yml:174 — STROM_REQUIRE_GL: 1. The code path is right, but job 113910763140 never ran the binary (see Tests & CI)
GL-memory caps carry output_format	CONFIRMED	backend/src/blocks/builtin/vision_mixer/builder/mod.rs:356 — builder = builder.field("format", fmt);
Unset output_format leaves the graph unchanged	CONFIRMED	backend/src/blocks/builtin/vision_mixer/builder/pipeline_gpu.rs:33 — let Some(format) = p.output_format.as_deref() else {

Diagnosis: same as my review of 2026-10-09. Coverage is ABSOLUTE for both GPU output branches. Commits 1–5 are unchanged by the rebase (git range-diff reports = for each), and f0a309c touches only the test file.

Radius: SHARED, as before.

Overlaps: #809 and #803 also change the vision mixer builder. This head merges cleanly with each of them (local git merge --no-commit), so landing order no longer matters. #969 item 3 is unaffected.

Tests & CI: run 37957232847 at f0a309c. Linux builds, WASM and API contract are green. Check (Linux) failed in stinger_prores_test (a_prores_clip_after_another_format_still_airs, another_format_after_a_prores_clip_still_airs: frame -5: old source before the cut). This PR touches no stinger code, and main's Check (Linux) at dcccdd7 passed, so I read this as the known stinger flake (#1055 is working on it). That job runs cargo test without --no-fail-fast, and the last binary it started was stinger_prores_test. So vision_mixer_gl_output_format_test, which sorts after it, did not run. The author's local run with STROM_REQUIRE_GL=1 is the only execution so far. macOS/Windows skipping is fine, because there is no platform code. I ran nothing locally.

Confidence: HIGH

Superseded by my review at f0a309c: the requested change landed. The new verdict is Comment, pending a Linux rerun.