⚡ export: screen-rect cursor plate, bounded unpremultiply, hwaccel decode - #3
Merged
Merged
Conversation
…cel decode - Cursor-only exports stream a screen-rect-sized RGBA plate over stdin instead of the full canvas: the renderer still draws into a canvas pixmap clipped to the fitted screen rect, then the rect is cropped out and composited back via overlay=x:y. Cuts per-frame rasterize, unpremultiply, pipe, and overlay work by the canvas padding fraction. - unpremultiply_rgba_bounded restricts the straight-alpha pass to the drawn bounding box (alpha-0 pixels are untouched by the conversion anyway, so output is identical). - Media inputs get -hwaccel auto when a hardware encoder is selected; auto falls back to software decode transparently, and the existing software-encoder retry still produces a clean software command. Co-Authored-By: Jean de Dieu HAGENIMANA <jdhagena77@gmail.com>
Contributor
Author
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
10 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📝 Description
Export profiling showed NVENC idle (~25% on RTX 3050) because every stage upstream is CPU-bound: software decode, a single-threaded
zoompan, and a full-canvas RGBA overlay plate re-rasterized per frame in Rust and piped to FFmpeg's stdin as rawvideo (8–16 MB/frame). This PR does the cheap, no-restructure wins; the bigger reworks (pre-rendered overlay inputs, cursor-spritesendcmd, chunked parallel export) stay as follow-ups.Cursor plate shrinks to the fitted screen rect (cursor-only exports): the renderer still rasterizes into a canvas-sized pixmap (every draw is already clipped to
video_screen), thencrop_rgba_regionextracts that rect and the graph composites it back viaoverlay=x={rx}:y={ry}. Per-frame rasterize→unpremultiply→pipe→composite work drops by the padding fraction (~10% at default padding, more for letterboxed/aspect-fit layouts). Dual-plane and items plates are unchanged — they draw in canvas coordinates.unpremultiply_rgba_boundedrestricts the premultiplied→straight-alpha pass to the bounding box of non-zero-alpha pixels. Pixels with alpha 0 are untouched by the conversion either way, so output is bit-identical while skipping ~90%+ of pixels on typical frames (a cursor sprite on a large canvas). Applied to both the cursor plane and the cached items plane.-hwaccel autoon media inputs when a hardware encoder is selected: moves decode off the CPU where a backend exists (D3D11VA/DXVA2 on Windows, VideoToolbox on macOS, VAAPI on Linux), auto-downloads frames for the software filter graph, and falls back to software decode when no backend applies (verified on a GPU-less machine). Software encodes — including the existing hardware-failure retry, which re-enters withExportEncoder::Software— still get a clean software command.🎯 Type of Change
🧪 Validation & Testing
cargo fmt --check— cleancargo clippy -- -D warnings— cleancargo check— clean (stable-x86_64-pc-windows-gnu; this VM has no MSVC toolchain, socargo testlinking hits the GNU ld ordinal limit — left to CI)Standalone equivalence harness:
unpremultiply_rgba_boundedproduces byte-identical output tounpremultiply_rgbaacross sparse / dense / fully-transparent / single-edge-pixel / mixed-alpha inputs;crop_rgba_regionverified pixel-exact-hwaccel autoverified to fall back to software decode cleanly on a GPU-less box (frame=60 ... rc=0)bun run typecheck— unchanged files are Rust-only; covered by CIbun run test— covered by CIcargo fmt --check/cargo clippy -D warnings/cargo checkpassTested on target operating systems:
📋 Checklist
cargo fmtcleanClipRectrounding, hwaccel gating rationale)Expected impact: removes a chunk of the CPU work feeding NVENC (smaller plate + dirty-bbox unpremultiply + hw decode). GPU utilization should rise but likely stays well under 100% —
zoompanremains single-threaded and the dual-plane path still ships full-canvas plates; raising GPU util substantially needs the chunked/sprite follow-ups.Link to Devin session: https://app.devin.ai/sessions/52c7ba7f829f4aecbefa2f889efc8dc5
Open in Devin Desktop: https://app.devin.ai/desktop/session/52c7ba7f829f4aecbefa2f889efc8dc5?variant=devin
Requested by: @jeandedieuH