Skip to content

perf(video): thread the scaling step in the Video Format block - #731

Merged
srperens merged 1 commit into
Eyevinn:mainfrom
wagenet:wagenet/videoconvertscale
Sep 29, 2026
Merged

srperens merged 1 commit into
Eyevinn:mainfrom
wagenet:wagenet/videoconvertscale

Conversation

@wagenet

@wagenet wagenet commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

The Video Format block builds videoscale -> videoconvert -> capsfilter. configure_video_convert() raises n-threads on the videoconvert, but videoscale has the same property with the same default of 1 and nothing set it, so a resize to or from 4K ran on one core. The block now passes videoscale through the same helper.

Like the existing conversion threading, this is macOS-only: elsewhere the helper is a no-op and the block is unchanged.

Measurements

M4 Max (12 performance cores), gst-launch-1.0 with n-threads pinned to 4, 150 frames, 6 order-alternated pairs, videoscale ! videoconvert n-threads=4 against videoscale n-threads=4 ! videoconvert n-threads=4. Wall time includes the test source.

Shape Before After Total CPU
RGBA 1080p → I420 720p 0.74 s 0.36 s within 1%
UYVY 4K → I420 1080p 2.23 s 1.36 s +4%
I420 720p → UYVY 4K 1.49 s 0.53 s within 2%

6/6 pairs in every shape. On this machine the block itself uses video_convert_threads() = 12. Each resizing block adds that many threads; the CPU cost of many such blocks competing on a Mac with fewer cores was not measured.

For progressive video the output is byte-identical to the unthreaded chain. That covers the shapes above, I420 1080p → RGBA 720p, and every resolution in the block's dropdown from 1080p, 720p, 576p and 480p sources in I420, UYVY and NV12. The exception is SD upscaled to 4K, where chroma moves by 1–2 levels. 10-bit packed v210 output differs byte-wise; whether that reaches pixel values was not checked. Interlaced input can differ on the rows where one thread's stripe meets the next.

A single videoconvertscale doing both steps was also measured. It is ~5% faster than this on the first shape, but 1.1–2x slower on the other two. Single-threaded, as on Linux, it costs up to 2.5x the CPU of the two-element chain. It also changes output at sharp chroma edges.

Tests

scale_and_convert_are_threaded (macOS) builds the block and checks that every element with n-threads got video_convert_threads(). It fails with configure_video_convert did not reach 'vf0:videoscale' when the new call is removed.

The test is macOS-only, because the thread count is only raised there. The macOS CI job does not run on PRs without the ci:macos label, so it was run on the fork at this head, where the test passed: https://github.com/wagenet/strom/actions/runs/36247628073.

Ran locally on macOS at this head: the 11 videoformat and gpu lib tests, cargo fmt --all --check, and cargo clippy --workspace --all-targets -D warnings. The full suite runs in the Linux CI job, where the change compiles to nothing. Windows was not built.

Also corrects a doc comment in thumbnail.rs that names elements the module no longer uses.

🤖 Generated with Claude Code

@wagenet
wagenet force-pushed the wagenet/videoconvertscale branch 2 times, most recently from b656eb2 to d856a8a Compare August 31, 2026 19:32

@wagenet wagenet left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks reasonable.

@wagenet
wagenet force-pushed the wagenet/videoconvertscale branch from d856a8a to 60dc59a Compare September 1, 2026 04:46
@srperens srperens mentioned this pull request Sep 1, 2026
@srperens

Copy link
Copy Markdown
Collaborator

Noted as a draft, so I am holding off on a full review until it is marked ready for
review. If you want one before then, say so here or request me as a reviewer.

@wagenet

wagenet commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

@srperens it could be reviewed now, but it can't be merged until the PR it is stacked upon is merged.

@wagenet
wagenet force-pushed the wagenet/videoconvertscale branch 2 times, most recently from d13cbf2 to f8cb697 Compare September 26, 2026 04:21
@wagenet wagenet changed the title perf(video): convert and scale in one pass in the Video Format block perf(video): thread the scaling step in the Video Format block Sep 26, 2026
@wagenet
wagenet force-pushed the wagenet/videoconvertscale branch from f8cb697 to b1ca85a Compare September 26, 2026 05:03
@wagenet
wagenet marked this pull request as ready for review September 26, 2026 05:17
`videoscale` ships with `n-threads=1`, the same default that
`configure_video_convert()` already raises on the `videoconvert` beside
it, so a resize to or from 4K ran on one core. The block now passes its
`videoscale` through the same helper. For progressive video the output
in I420, UYVY and NV12 is byte-identical in every shape measured except
SD upscaled to 4K, where chroma moves by 1-2 levels. Interlaced input
can differ on the rows where one thread's stripe meets the next.

Like the conversion threading this is macOS-only. Elsewhere the helper
is a no-op and the block behaves as before.

M4 Max, gst-launch with n-threads pinned to 4, 150 frames, 6
order-alternated pairs, wall time including the test source:

- RGBA 1080p -> I420 720p: 0.74 s -> 0.36 s
- UYVY 4K -> I420 1080p: 2.23 s -> 1.36 s
- I420 720p -> UYVY 4K: 1.49 s -> 0.53 s

Total CPU stays within 4% in all three.

A single `videoconvertscale` doing both steps is ~5% faster on the first
shape, but 1.1-2x slower on the other two, and single-threaded, as on
Linux, costs up to 2.5x the CPU of the two-element chain.

Also corrects a doc comment in thumbnail.rs that names elements the
module no longer uses.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@wagenet
wagenet force-pushed the wagenet/videoconvertscale branch from b1ca85a to 4c60f2d Compare September 28, 2026 15:29

@srperens srperens left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified locally on macOS: scale_and_convert_are_threaded passes at this head and fails with configure_video_convert did not reach 'vf0:videoscale' when the new call is removed. Change is macOS-only and a no-op elsewhere. Thanks!

@srperens
srperens merged commit 6017ec9 into Eyevinn:main Sep 29, 2026
9 checks passed
srperens added a commit that referenced this pull request Sep 29, 2026
Add #731, #882 and #889, which landed on main after the bump was opened.

Audit every section from 0.4.10 on against the commits between its tags
and add the PRs that were never written up: #505-#507 and #514 in 0.4.11,
#533, #541 and #550 in 0.5.0, #577, #580, #581 and #584 in 0.6.0, #638 in
0.6.5 and #829 in 0.6.9. Only version-bump PRs are left unlisted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
srperens added a commit that referenced this pull request Sep 29, 2026
Add #731, #882 and #889, which landed on main after the bump was opened.

Audit every section from 0.4.10 on against the commits between its tags
and add the PRs that were never written up: #505-#507 and #514 in 0.4.11,
#533, #541 and #550 in 0.5.0, #577, #580, #581 and #584 in 0.6.0, #638 in
0.6.5 and #829 in 0.6.9. Only version-bump PRs are left unlisted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
srperens added a commit that referenced this pull request Sep 29, 2026
* chore(release): bump version to 0.6.11

Bump the workspace version to 0.6.11 (Cargo.toml, Cargo.lock, the OpenAPI
info.version in openapi.json) and add the 0.6.11 changelog entry for the
27 commits since 0.6.10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): add the three PRs since, and fill gaps in 0.4.11-0.6.9

Add #731, #882 and #889, which landed on main after the bump was opened.

Audit every section from 0.4.10 on against the commits between its tags
and add the PRs that were never written up: #505-#507 and #514 in 0.4.11,
#533, #541 and #550 in 0.5.0, #577, #580, #581 and #584 in 0.6.0, #638 in
0.6.5 and #829 in 0.6.9. Only version-bump PRs are left unlisted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): add #876 to 0.6.11

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants