fix(vision mixer): adapt GPU inputs to any pixel format, converting only while needed

[srperens]

Closes #674

Problem

A GPU Vision Mixer input starts with glupload, which uploads a fixed list of system-memory pixel formats. A source in any other format was refused:

• A flow link from a capsfilter pinned to such a format (AYUV64, GBR_10LE) was never made. The linker's caps check failed, the link stayed pending, and the source stopped with not-linked. The flow never reached PLAYING.
• A format change in the middle of a stream (a Media Player moving on to a ProRes 4444 clip, which decodes to AYUV64 on macOS) failed the producer with not-negotiated.

On the CPU backend, #674's cases already work on main: each input has its own converter. Nothing guarded that, though.

Change

GPU path (gst/gl_input_front.rs, gst/video_adapt.rs)

• decide() gets a rule for Consumer::GlUpload(&caps): raw video in system memory only, in no format glupload uploads, needs [VideoConvert]. The upload formats come from glupload's sink template (gl_upload_system_caps()), so they follow the installed GStreamer. This part comes from the stinger work on feat/stinger.
• The converter is transient. It goes in and comes out on the input's streaming thread, in an EVENT_DOWNSTREAM probe on the front queue's source pad. The probe acts on CAPS events only, never on buffers, and makes its change as the CAPS event leaves the queue. GStreamer reads the pad's peer after the probes have run, so the event and every frame after it go to the new chain. Frames queued in the old format have already gone through the old one. The converter comes out as soon as the caps are something glupload takes directly, or CUDA memory. Unlinking it, relinking around it, setting it to NULL and removing it from the bin all happen in that probe. Only that thread pushes on that link, so no IDLE wait is needed.
• No CPU round trip for CUDA memory. When CUDA memory follows a converted format, the same probe takes the converter out and then splices cudadownload (CUDA-GL interop) directly in front of glupload. It never lands in front of the converter. While the converter is in, a CUDA query does not trigger the query-time cudadownload splice. Inputs that never needed the converter keep the existing query-time cudadownload splice (IDLE probe), unchanged.
• Queries before the CAPS event are answered for the input as it will be after it:
  • An ACCEPT_CAPS for a format only the converter takes is accepted while the converter is out.
  • An ACCEPT_CAPS for caps the input takes without the converter is answered by the element after the converter while the converter is in. For CUDA memory with cudadownload not yet in, it is answered from cudadownload's sink template.
• CAPS-query answer. While the converter is in, the answer lists, in this order: what the element behind the converter takes directly (glupload's GL and uploadable system-memory entries), the CUDA alternative of those, and only then what the converter adds. While the converter is out, raw video in system memory in any format is appended last, and only when the query's filter offers nothing the input takes directly or there is no filter (the linker's link check). That append is what lets the flow link be made. A filtered query that offers something the input takes directly gets the same answer as before.
• GStreamer before 1.24.13. glupload there answers a CAPS query with only the caps of the upload method it is using. With the converter in, that is the GL memory method, so an unfiltered query got GL memory only: no system-memory formats, so no CUDA alternative. On 1.24.2, NVDEC into an input that had the converter in then decoded to system memory, one CPU round trip per frame. takes_directly() asks glupload a second time with a system-memory filter, which makes it list every method, and merges the answers. 1.24.13 and later list every method anyway, so nothing changes there.
• All state shared by the probes holds only weak references (WeakRef<Element>, WeakRef<Pad>).

CPU path: no code change. The per-input converter and capsfilter_in on main already handle #674's cases. The new tests guard them.

Rejected:

• Splicing the converter at query time and leaving it in, as on feat/stinger. A cudadownload spliced later lands in front of it, and the converter's CAPS answer offers every CPU format.
• Removing the converter at query time. Frames already queued in the old format would reach glupload directly and fail.
• A fixed BGRA capsfilter, as the triage suggested. It forces a CPU conversion on every input and drops GL/CUDA memory.
• autovideoconvert, per BLOCK_GUIDELINES.

Evidence

• GPU input takes AYUV64 and GBR_10LE from a pinned capsfilter, with the converter in: gpu_ayuv64, gpu_gbr_10le. Both fail with decide()'s VideoConvert rule removed (the main-equivalent behaviour) and with the staged feat/stinger front: start vision mixer pipeline: StateChange(... current: Paused, pending: Playing), from a link the linker never made. They also fail with the any-format entry left out of the CAPS-query answer.
• The converter comes out when the format switches back (I420, then GBR_10LE, NV12, AYUV64, I420), and glupload then receives the source's own format: gpu_converter_comes_out_when_the_format_switches_back. It fails with the removal disabled: NV12: glupload takes it, but the input still runs ["videoconvert"].
• CUDA memory after a converted format bypasses the converter (AYUV64, then CUDA, AYUV64, CUDA, through a cudadownload stand-in), and while the converter is in, CUDA memory is offered ahead of the converter's formats: cuda_after_a_converted_format_bypasses_the_converter. It fails:
  • with the removal disabled (adapters ["videoconvert", ...]);
  • with the answer reordering reverted to the staged behaviour (the CUDA entry after the converter's entries);
  • with the staged front.
• An unpinned source told only "raw video" while the converter is in settles on a format glupload uploads, and the converter comes out: second half of gpu_converter_answers_with_direct_formats_first. It fails with the removal disabled. Its first half (direct entries before converter-only ones) also passes with the reordering reverted. videoconvert's own answer already puts passthrough caps first, so for GL/system memory it documents the behaviour and does not guard it. The CUDA ordering is guarded by the CUDA test above.
• CPU backend takes Vision mixer inputs require BGRA, but the requirement isn't negotiated and videoformat can't select it #674's cases (direct, via videoconvert, I420, NV12, AYUV64, GBR_10LE): cpu_*. With the per-input converter removed from pipeline_cpu.rs, cpu_ayuv64 fails; the other five pass, because the compositor takes those formats itself.
• Answer ordering as a unit: gl_input_front::tests::with_the_converter_in_direct_paths_come_first.
• NVDEC into an input that had the converter in stays on the GPU: nvdec_cuda_after_a_converted_format_stays_on_the_gpu (#[ignore], NVIDIA only), run on an NVIDIA L4 with GStreamer 1.24.2. nvh264dec src: video/x-raw(memory:CUDAMemory), format=NV12; glupload sink: video/x-raw(memory:GLMemory), format=NV12; adapters ["cudadownload"], no videoconvert left. Without the 1.24 fix it fails: the decoder's frames stay in system memory.
• The 1.24 fix: with it reverted, on 1.24.2 gpu_converter_answers_with_direct_formats_first fails (the answer lists nothing only the converter takes) and so does cuda_after_a_converted_format_bypasses_the_converter (does not offer CUDA memory, the error the first CI run on this PR showed). On 1.28 it cannot be made to fail, since glupload there already lists every method; CI's 1.24.2 is what guards it.

Tests

Ran on macOS (GStreamer 1.28.6, native GL), with STROM_REQUIRE_GL=1 STROM_REQUIRE_GST_PLUGINS=1:

• vision_mixer_input_format_test: 14 passed (run 4 times)
• vision_mixer_input_format_cuda_test: 1 passed, 1 ignored (the NVDEC test)
• vision_mixer_cpu_test 7, vision_mixer_fx_test 2, vision_mixer_source_resize_test 8, pipeline_lifecycle_test 4, failed_start_teardown_test 3, shader_validation_test 2, gl_memory_link_test 7, thumbnail_tap_cuda_test 3, video_input_bridge_test 5, video_input_bridge_gl_test 4, vision_mixer_cuda_input_test 2, vision_mixer_cuda_missing_test 1, openapi_test 1: all passed
• cargo test --workspace --lib: 900 + 28 + 101 passed, 5 ignored
• cargo clippy --all-targets --features efp -- -D warnings, cargo fmt --check: clean

Skipped: none. Every GL test ran with a GL context.

On Linux with an NVIDIA L4 (GStreamer 1.24.2, headless EGL), after the 1.24 fix:

• vision_mixer_input_format_test: 14 passed
• vision_mixer_input_format_cuda_test: cuda_after_a_converted_format_bypasses_the_converter skips where the real cudadownload is installed; with nvcodec hidden (as on CI) it ran and passed
• vision_mixer_input_format_cuda_test -- --ignored nvdec: passed (encodes with nvcudah264enc; the legacy nvh264enc fails with Selected preset not supported on that driver)
• vision_mixer_cuda_input_test: 2 passed

Not run:

• The full cargo test --workspace integration suite.
• Windows. Linux without NVIDIA only through CI. The format choice was checked against 1.24.2's GST_GL_MEMORY_VIDEO_FORMATS_STR: AYUV64 and GBR_10LE are missing there too, while A444_10LE uploads on 1.24.2, so it is not used.

For the reviewer

• On a host with real nvcodec, cuda_after_a_converted_format_bypasses_the_converter skips. Its appsrc pushes system memory labelled as CUDA memory, which the real cudadownload cannot take. The ignored NVDEC test covers that host.
• Unfiltered CAPS queries on a GPU input now end with one extra system-memory entry. Producers that fixate on the first entry are unaffected. A producer that intersects in zig-zag order could now pick a format only the converter takes. No such case was seen; please consider it.
• Converter insertion and removal run on the input's streaming thread, including set_state(Null) and bin.remove() of the removed converter. This follows the dynamic-pipeline pattern, and pipeline_lifecycle_test passes.

Out of scope

• The CPU path's VideoConvertMode (autovideoconvert on NVIDIA hosts) and the other Retire the global VideoConvertMode and the remaining producer-side memory-format choices #969 items.
• cudadownload stays once it is in, as before.

🤖 Generated with Claude Code

[srperens]

Verdict: Comment. The design is right and follows BLOCK_GUIDELINES. I found no defect in the code. CI has not run at this head, which blocks approval.

Needed before merge

1. CI at 6581f3d. Run 37642337039 was created 2026-10-07T15:06Z, is still queued and has no jobs. Re-trigger: gh run cancel 37642337039 && gh run rerun 37642337039, or push an empty commit. The previous head dc4d16a was red in Check (Linux), on cuda_after_a_converted_format_bypasses_the_converter (does not offer CUDA memory, vision_mixer_input_format_cuda_test.rs:323). 6581f3d fixes that. cargo test stops at the first failing binary, so vision_mixer_input_format_test has never run in CI at any head. pipeline_lifecycle_test ran and passed at dc4d16a.

Claims

Claim	Verdict	Evidence
Probes fire per event or query, never per buffer	CONFIRMED	backend/src/gst/gl_input_front.rs:304 — front_src.add_probe(gst::PadProbeType::EVENT_DOWNSTREAM, move. No BUFFER probe is added
Shared probe state holds weak references only	CONFIRMED	backend/src/gst/gl_input_front.rs:221 — upload: glib::WeakRef<gst::Element>, and front_src likewise
An ACCEPT_CAPS for a convertible-only format is accepted before the converter is in	CONFIRMED	backend/src/gst/gl_input_front.rs:343 — q.set_result(convert_sink_caps().is_some());
The new tests guard the change	UNVERIFIED	Revert rows are on the author's macOS and an L4 rig. No CI run has executed them at this head (above)

Diagnosis. The cause is right: glupload uploads a fixed format list, and before this PR the input front never adapted pixel format. Coverage is BOUNDED to Vision Mixer GPU inputs, the only caller of install(). The deprecated Compositor still feeds glupload with no front: backend/src/blocks/builtin/compositor.rs:347 — let upload = gst::ElementFactory::make("glupload"). Putting the converter in and out from the queue's own src thread, as the CAPS event leaves, is sound. Nothing else pushes on that link, and the converter has no thread of its own to stop. That makes set_state(Null) and bin.remove() from inside the probe safe.

Behaviour change on every GPU input. Every Vision Mixer GPU input changes, including ones that never need a converter. An unfiltered CAPS query now ends with "any system-memory raw video": backend/src/gst/gl_input_front.rs:489 — if filter.is_none() || offers_only_convertible {. The author flags zig-zag intersection as the open risk. SPECULATIVE (not verified): a producer that intersects that way could now take a CPU conversion it did not need.

Radius. SHARED: one block's GPU inputs. video_adapt::decide() changes its Consumer::GlUpload signature, and gl_input_front.rs is its only caller. The new Adapter::VideoConvert variant breaks no exhaustive match: build_elements and installed both handle it, and no other open PR touches video_adapt.rs.

Overlaps.

• #674 owns this decision. My triage there recommended a fixed BGRA capsfilter (Option 1), and its Ask is still open. This PR rejects that option, and it is right to. A fixed BGRA capsfilter forces a CPU conversion on every input and drops GL/CUDA memory, which BLOCK_GUIDELINES' memory-format rule forbids. That rule postdates my triage. The PR comes from someone who may decide, so I read it as the answer to #674. Merging it closes that Ask.
• #969 keeps the CPU path's VideoConvertMode. It is out of scope here, as the PR says.

Tests & CI. No checks at 6581f3d (above). macOS and Windows are skipped on pull requests. The platform-specific evidence (ProRes 4444 on VideoToolbox) comes from the author's macOS run only. Merging on Linux green and letting main's macOS run cover it is reasonable.

Confidence: MED