perf(macos): convert colour through Accelerate/vImage

[wagenet]

Written by Claude (Claude Code) on behalf of @wagenet.

Description

Adds stromvimageconvert, a GStreamer element that converts colour through Accelerate/vImage on macOS, selected by a new macOS-only VideoConvertMode::VImage. It has videoconvert's n-threads property, so configure_video_convert() reaches it and no call site changed. STROM_DISABLE_VIMAGE=1 restores videoconvert.

Fallback. Anything vImage has no path for (an unlisted format pair, interlaced content, a matrix outside BT.601/BT.709, a chroma siting it does not write) runs on GstVideoConverter, the code videoconvert is built on. The read-only conversion-path property reports vimage / fallback / passthrough / unnegotiated. Like videoconvert, the element does not resize; in Video Format, videoscale ahead of it does, with borders. It accepts any caps features, as videoconvert does: field-per-buffer video converts on the fallback; GL memory only passes through.

vImage paths: packed RGB32 (RGBA/BGRA/ARGB/ABGR and x variants) to and from NV12, I420, YV12, UYVY, YUY2; RGB32 to RGB32; NV12 to and from I420/YV12.

Chroma siting. vImage writes centred chroma. Caps without chroma-site mean left-cosited at 720 lines and above, and x264enc and vtenc copy that label into the stream, so RGB-to-Y'CbCr output is labelled jpeg. Converted at 1080p and decoded back to RGB, the element's output is closer to the source than videoconvert's in all 8 cases measured (bars, two text overlays, a photo; NV12 and I420): PSNR 33.0-38.6 dB against 31.6-37.3 dB.

Format choice follows videoconvert's loss score, except that an 8-bit source is never widened when an 8-bit target exists: fed RGBA, vtenc_h264_hw offers AYUV64, which videoconvert takes at a cost of 15.7% end to end (t = −163, n = 40). Colour to grey outranks every other penalty, as in videoconvert.

Numbers (M4 Max, 1080p RGBA→NV12, 300 frames, 40 counterbalanced pairs, debug build, net of source): 3.4× the threaded videoconvert (186 ms → 54 ms), 13.9× stock. A/A control 1.01× (t = 0.60). The harness is #[ignore]d tests in bench.rs. Measured on an earlier head; the RGBA→NV12 path it times is unchanged.

Related Issue

None.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update

Checklist

• I have run cargo fmt --all
• I have run cargo clippy --workspace --all-targets -- -D warnings
• I have run cargo test --workspace
• I have updated documentation as needed
• I have added tests for new functionality

Tests

PR CI does not run these tests. The module is #[cfg(target_os = "macos")] and Build (macOS) needs the ci:macos label. A fork run of the macOS job: run 36618849668, on this head.

Run at a5a55d7b on macOS (M4 Max, Homebrew GStreamer): cargo fmt --check, cargo clippy --workspace --all-targets -D warnings, and cargo test --workspace --no-fail-fast (1094 passed, 0 failed, 7 ignored).

Each guard below was run with its behaviour reverted and fails:

• vimage_output_matches_videoconvert: RGB→Y'CbCr and the 4:2:0 shuffles, pixel for pixel against videoconvert, asserting the vImage path ran. Fails on a red/blue swap (by 149), YV12 planes swapped (149), limited range read as full (20).
• yuv_to_rgb_recovers_the_source_colours: Y'CbCr→RGB against flat source colours. It is not compared with videoconvert, which interpolates chroma for some formats and not others, so the two differ at hard edges while both are correct.
• a_portrait_source_is_letterboxed_when_the_format_also_changes (Video Format block): fails with 0 border pixels if the element offers to resize.
• hd_output_is_labelled_with_the_siting_vimage_writes, a_pinned_cosited_siting_takes_the_fallback, a_chroma_copy_keeps_the_input_siting.
• an_eight_bit_target_beats_every_deeper_one, dropping_colour_costs_more_than_any_colour_target, unsupported_pair_falls_back_and_still_converts.
• identical_caps_pass_straight_through, input_without_pixel_aspect_ratio_passes_through (camera caps omit the ratio), a_field_the_peer_requires_is_written_into_the_output (a consumer that requires a field the camera omits), interlaced_input_takes_the_fallback, caps_features_link_like_videoconvert (GL memory offered for conversion, or field-per-buffer video refused).

Not measured: vImage's thread pool with many converters running at once.

🤖 Generated with Claude Code

[srperens]

Noted as a draft, so I am holding off on a full review until it is marked ready for
review. If you want one before then, say so here or request me as a reviewer.

[wagenet]

@srperens I figured that this can't be merged until the PRs it is stacked on are merged. I'll bring it out of draft at that point. Though you can review it now if you want. (I really wish GitHub Stacked PRs worked with forks.)

[srperens]

Verdict: Comment. The design and fallback hold up. One thing blocks approval: none of the roughly 2,900 new lines has been compiled by this repo's CI. All of it sits behind cfg(target_os = "macos"), and Build (macOS) skipped on this PR.

Requested changes

1. Maintainer: add the ci:macos label so Build (macOS) runs cargo test on the PR merged with main. The job only runs on that label: .github/workflows/ci.yml:644 — && contains(github.event.pull_request.labels.*.name, 'ci:macos'))
2. Optional: a vImage error currently returns FlowError::Error and only writes a debug log, so the operator sees nothing beyond Internal data stream error. Post a gst::element_error! naming the format pair and the vImage code: backend/src/gst/vimage/imp.rs:400 — gst::error!(CAT, imp = self, "vImage conversion failed: error {}", err);

Claims

Claim	Verdict	Evidence
Frames vImage cannot handle fall back instead of failing	CONFIRMED	Interlaced input exits at backend/src/gst/vimage/plan.rs:97 — return None;. Odd sizes on subsampled formats also drop out (backend/src/gst/vimage/plan.rs:104 — !(even_w && !in_info.width().is_multiple_of(2)). None builds a GstVideoConverter in conversion_path
A later renegotiation is not stuck in passthrough	CONFIRMED	backend/src/gst/vimage/imp.rs:204 — const PASSTHROUGH_ON_SAME_CAPS: bool = true; makes BaseTransform recompute passthrough before every set_info (EXTERNAL: upstream configure_caps). The forced passthrough at backend/src/gst/vimage/imp.rs:366 — self.obj().set_passthrough(true); only fires when the two VideoInfo are equal
The tests guard the change and pass	UNVERIFIED here	They run and pass on the author's fork at this head: Build (macOS) job, gst::vimage::tests::*, gst::vimage::imp::tests::* and a_portrait_source_is_letterboxed_when_the_format_also_changes all ok, 754 passed. That run built the branch alone, not the branch merged with main. Main is 12 commits ahead; the only one touching these files is #910's gpu.rs test pruning, in a separate hunk
vImage's conversion state is safe to share across threads	EXTERNAL	Assumes Apple's documented thread-safety for generated vImage_*ToYpCbCr state

Diagnosis. The engine is picked once per negotiation, transform_frame makes a single vImage call or falls back to frame_ref, and nothing per frame spawns a thread or allocates. Coverage of the claimed speed-up is BOUNDED to RGB32 to and from NV12/I420/YV12/UYVY/YUY2, RGB32 shuffles and NV12/I420 interleaving. Everything else runs on the same code videoconvert runs.

Radius. GLOBAL on macOS, nothing elsewhere. Every converter built through video_convert_mode() switches, e.g. backend/src/blocks/builtin/videoenc.rs:122 — let convert_element_name = convert_mode.element_name(); and backend/src/blocks/builtin/devicesrc.rs:214 — let convert_element_name = video_convert_mode().element_name();, plus Video Format and four NDI paths. STROM_DISABLE_VIMAGE is the escape hatch: backend/src/gpu.rs:264 — return VideoConvertMode::Software;. The element holds no pipeline reference and no closures, and stop() drops its state.

Overlaps. None. #897 edits gpu.rs and backend/Cargo.toml in hunks that don't touch these. #682 (memory-format adaptation) is not settled here: GL memory passes through unchanged and is never converted.

Tests & CI. In this repo at a5a55d7: Check (Linux), both Linux builds, WASM, API Contract Check and Static JS Tests passed; Build (macOS) and Build (Windows) were skipped. The Linux jobs never compile gst::vimage. With this much new unsafe FFI, I would not merge on Linux-green and leave the problems for main's macOS run.

Confidence: HIGH

[wagenet]

Claude here, replying for @wagenet.

1. macOS CI. I can't apply ci:macos (no write access), so a maintainer needs to add it. Meanwhile I ran Build (macOS) on the fork against this branch rebased onto main: https://github.com/wagenet/strom/actions/runs/37016510548. It compiles and all gst::vimage tests pass. The job is red on two mediaplayer::bridge timing tests (a_live_timeline_with_a_stall_plays_on_time_in_the_main_pipeline, playback_carries_on_in_time_after_a_file_switch) that also fail in main's own macOS runs (37015055539, 37012267930), and they fail identically locally with STROM_DISABLE_VIMAGE=1. Because cargo stops at the first failing binary, the integration tests didn't run in CI. Locally on macOS the full suite on the rebased tree passed: 958 passed, 0 failed.
2. vImage error. dbbc21a posts gst::element_error! with the conversion kind, the format pair and the vImage code, and names STROM_DISABLE_VIMAGE=1 in the debug detail. No test covers it, because vImage can't be made to fail on demand.

The branch is rebased onto main to clear a conflict in gst/mod.rs.

[srperens]

Verdict: Comment. This supersedes my review of 2026-10-02 (at a5a55d7), and the verdict stands for the same single reason: none of the ~2,900 macOS-only lines has been compiled by this repo's CI. The optional ask from last time is done, and the new head merges cleanly with main@d2ae537 (git merge-tree reports no conflict).

Requested changes

1. Maintainer: add the ci:macos label so Build (macOS) builds and tests the PR merged with main. .github/workflows/ci.yml:644 — && contains(github.event.pull_request.labels.*.name, 'ci:macos')). I am not applying it myself, because it starts a paid run.

What changed since my last review

Compared with a5a55d7, the code delta is one hunk. A failed vImage call now posts an element error and no longer only logs: backend/src/gst/vimage/imp.rs:409 — gst::element_imp_error!(. The message names the plan, both formats and the cause (backend/src/gst/vimage/imp.rs:404 — let cause = if err == acc::K_PLANE_UNAVAILABLE {), and it points the operator at STROM_DISABLE_VIMAGE. This is what BLOCK_GUIDELINES.md asks for ("A refusal fails the flow, with a message the operator can act on"). The format! runs only on the error path, so the per-frame cost is unchanged.

The rest of the tree difference is the rebase. backend/Cargo.toml gains gstreamer-base.workspace = true for the subclass. The tower-sessions/cookie and gpu.rs test-module lines are main's own changes, not this PR's.

Claims

Claim	Verdict	Evidence
Frames vImage cannot handle fall back instead of failing	CONFIRMED	plan.rs and the fallback in conversion_path are unchanged since my review of 2026-10-02 at a5a55d7, where I confirmed this
The escape hatch restores videoconvert	CONFIRMED	backend/src/gpu.rs:259 — if std::env::var_os("STROM_DISABLE_VIMAGE").is_some() {
The tests guard the change and pass	UNVERIFIED	No run in this repo compiles gst::vimage. The fork run cited in the body built the branch alone, not the branch merged with main, and main has since changed gpu.rs (#1061)
element_imp_error! with this argument shape compiles against the pinned gstreamer-rs	UNVERIFIED	It is only compiled on macOS. The macOS build settles it

Radius. GLOBAL on macOS, nothing elsewhere, as before. Every converter built through video_convert_mode() switches to the new element.

Overlaps. None found. No open PR touches backend/src/gst/vimage/. #897 and #1002 edit backend/Cargo.toml in other hunks.

Tests & CI. At dbbc21a: Check (Linux), both Linux builds, WASM, API Contract Check and Static JS Tests pass. Build (macOS) and Build (Windows) were skipped. The Linux jobs do not compile this module, so they are no evidence for it. With this much new unsafe FFI, I would still not merge on Linux-green alone.

Confidence: HIGH