Skip to content

⚡ export: chunked parallel timeline rendering - #5

Merged
jeandedieuH merged 2 commits into
mainfrom
devin/1790061993-chunked-parallel-export
Sep 22, 2026
Merged

jeandedieuH merged 2 commits into
mainfrom
devin/1790061993-chunked-parallel-export

Conversation

@devin-ai-integration

Copy link
Copy Markdown
Contributor

📝 Description

Splits timeline exports into frame-exact chunks rendered by parallel FFmpeg worker processes, then joins them with a stream-copying concat mux. Today a single composition pass serializes every filter stage — zoompan, the plate/camera/mask composites all run on one thread of one pipeline — which is exactly what starves a hardware encoder (NVENC sat at ~25% on an RTX 3050 in testing this pipeline). N chunk pipelines multiply compositing throughput up to the worker count and keep the encoder fed.

How it works

  • render_timeline_composition becomes a dispatcher: should_render_chunked (non-gif/webp, ≥20 s, ≥4 cores) routes to render_timeline_chunked, everything else takes the old single pass verbatim via render_composition_window(CompositionWindow::full, CompositionPass::standalone).
  • Each chunk = [first_frame, end_frame) on the output frame grid. A shifted pass applies setpts=PTS-STARTPTS+{win_start}/TB to the concatenated stream, so every absolute-time construct keeps working unmodified: between(t,…) enables, camera setpts=PTS+{offset}/TB, zoompan's it, and libass cue times.
  • Mid-segment cuts preserve the fps resampler's absolute grid: the chain anchors with setpts=PTS-STARTPTS+{cut*speed}/TB and the source window becomes src_in + cut*speed.
  • One subtlety found empirically: zoompan re-emits frames on a 0-based pts grid, so shifted passes apply a second setpts=PTS+{win_start}/TB after it to restore absolute timestamps for downstream pairing.
  • Window-disjoint camera overlays, masks, and caption burn-ins are skipped entirely — they can never fire inside the window.
  • Audio renders once as a single continuous AAC pass (muxed in at the end) — no per-chunk AAC priming discontinuity at seams. Embedded chapters and +faststart likewise move to the mux stage.
  • workers = cores/2 clamped to 2–4, ~2 chunks per worker, minimum 2 s per chunk. A shared halt flag kills sibling FFmpeg processes on first failure; cancel propagates through the existing flag. The overlay producer's worker/memory budgets divide by the worker count (parallel_divisor), keeping total RAM flat.
  • Each chunk renders first_frame_index + local plate frames — the cursor/items rasterizers are pure functions of absolute plan time, so plate content needs no changes.
  • Intermediates (chunk_*.mkv, audio.m4a, concat list) live in a per-export temp dir removed on success and failure.

Verified end-to-end on FFmpeg 8.1 with the exact generated graph shape (concat → tpad → zoompan + abs enables + camera overlay): single-pass vs 2 chunks at a mid-segment seam is bit-identical — 360/360 frames with matching framemd5, including a camera overlay spanning the boundary.

Closes #(issue)


🎯 Type of Change

  • ⚡ Performance optimization

🧪 Validation & Testing

Describe how you tested these changes:

  • bun run typecheck passes with zero errors — n/a for Rust changes; cargo clippy -- -D warnings clean
  • bun run test passes (all unit & integration tests) — cargo test cannot link under GNU ld on this Windows box (ordinal-limit error, pre-existing); added unit tests for chunk_boundaries + CompositionWindow::from_frames, CI covers the suite. All existing render tests use ≤2 s plans so they exercise the standalone path unchanged.
  • bun run format:check passes — cargo fmt --check clean
  • Tested on target operating systems: ffmpeg graph-level equivalence verified on Windows 11 (this box has no GPU; NVENC behavior untested)
    • Windows (10/11) — graph equivalence only
    • macOS
    • Linux

📋 Checklist

  • My code adheres to the style guidelines in CONTRIBUTING.md
  • I have commented complex or non-obvious algorithms (e.g. WASM / audio sync / FFmpeg jobs)
  • No raw video frames, audio buffers, or secrets are exposed across Tauri boundaries
  • No telemetry or external tracking was added

Notes for reviewers:

  • The redundant-decode trade-off is deliberate: chunks decode their sources from 0 rather than seeking (-ss would risk non-frame-exact landing). Decode is the cheap part; the serial filters were the bottleneck.
  • 4 workers → up to 4 concurrent NVENC sessions. Consumer GeForce cards allow ~5+ concurrent sessions on current drivers (older drivers capped at 3) — within budget, but worth knowing.
  • Expect ~2–3.5× wall-clock speedup on typical exports (bounded by the longest chunk and by decode redundancy), and GPU encoder utilization should rise substantially since the CPU compositor is now the parallelized part rather than the serial part.

Link to Devin session: https://app.devin.ai/sessions/52c7ba7f829f4aecbefa2f889efc8dc5
Open in Devin Desktop: https://app.devin.ai/desktop/session/52c7ba7f829f4aecbefa2f889efc8dc5?variant=devin
Requested by: @jeandedieuH

Split long exports into frame-exact chunks rendered by parallel FFmpeg
processes, then join them with a stream-copying concat mux. A single
composition pass is bottlenecked on serial filter stages (zoompan, the
overlay composites), which is what starves hardware encoders — N chunk
pipelines multiply throughput without changing emitted frames.

- render_timeline_composition splits into a dispatcher plus
  render_composition_window(window, pass): a shifted pass re-anchors the
  concatenated stream to absolute plan time so between(t,..), camera
  setpts offsets, zoompan 'it' and libass cues all keep working unmodified
- zoompan re-emits frames on a 0-based pts grid, so shifted passes
  re-shift its output back to absolute time for downstream pairing
- mid-segment cuts keep the fps resampler on the same absolute frame
  grid via a +cut*speed/TB phase on the chain anchor, and gap/trim
  clamps tile the window exactly
- camera overlays, masks and caption passes skip building chains that
  can never fire inside the window
- audio renders once as a single continuous AAC track and is spliced in
  at the mux — no per-chunk AAC priming at seams; embedded chapters and
  faststart move to the mux stage too
- workers = cores/2 (clamped 2..4), ~2 chunks per worker, chunk >= 2s,
  enabled only for non-gif/webp exports >= 20s on >=4 cores; a shared
  halt flag kills sibling passes on the first failure, and the
  overlay producer's worker/memory budgets divide by the worker count
- verified on FFmpeg 8.1: chunked vs single-pass output is bit-identical
  (360/360 frames, matching framemd5) including a mid-segment seam with
  zoom and a boundary-spanning camera overlay

Co-Authored-By: Jean de Dieu HAGENIMANA <jdhagena77@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

eof_action=repeat overlays fed by infinite looped secondaries keep emitting
clones of the last main frame past the stream's end; -t only trims them
approximately, so a repeat landing on an exact-millisecond boundary leaks
through as a duplicated seam frame (+1 frame, e.g. 901 vs 900 at 30s@30fps).
Add trim=end_frame on the windowed pass's final stream to bound output by
frame count instead of timestamp rounding.

Co-Authored-By: Jean de Dieu HAGENIMANA <jdhagena77@gmail.com>
@devin-ai-integration

devin-ai-integration Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor Author

E2E test: chunked parallel export — found & fixed a frame-exactness bug at camera-overlay seams

Tested on the real render path (temporary in-crate harness → render_timeline_composition → render_timeline_chunked, same serde boundary as export_timeline), 8-core box, ffmpeg 8.1.2, 30s@30fps plan → 8 chunks × 4 workers.

Confirmed working: export: rendering timeline in parallel chunks fires; recordforge_chunks_* holds 8 chunk_*.mkv + continuous audio.m4a + chapters.ffmeta; concat -c copy mux + faststart ✓; 3 embedded chapters ✓; h264+aac correct ✓; overlays/zoom/audio continuous across boundaries ✓; <20s export stays single-pass (300 frames) ✓; cancel aborts cleanly with no leftover temp/orphans ✓.

Bug found & fixed (5c6be54): with a camera/overlay element spanning a chunk boundary, that chunk emitted 114 frames instead of 113 — output 901 frames / 30.029s vs expected 900 / 30.000s, with the duplicated frame at the seam (source frame 338 appears twice consecutively, ~33ms visible freeze):

consecutive output frames both show source frame F338

Root cause (bisected on the captured chunk graph): the camera shadow/border layers are overlay eof_action=repeat fed by infinite loop=-1 secondaries — framesync keeps emitting clones of the last main frame past the main stream's end, and -t only trims the tail approximately (a repeat landing exactly on a millisecond boundary slips through, pts 3.7666667 < -t 3.766667). Fix: trim=end_frame={window.frame_count} on the windowed pass's final stream — bounds output by exact frame count instead of timestamp rounding. Verified on the same graph: now 113/113.

Also noted (not blockers for this PR)
  • Wall time chunked 8.18s vs single-pass 5.21s on this box for a 30s x264 export — x264 already saturates all 8 cores, so per-chunk overhead dominates at this duration. The parallelism targets GPU-encoder pipelines (NVENC etc.) where the CPU compositor is the bottleneck and the encoder has headroom; on longer exports / encoder-saturated pipelines the trade flips.
  • Pre-existing bug (not this PR — captions.rs untouched, single-pass fails identically): escape_filter_path double-escapes the drive colon → subtitles=filename='C\:/...' fails to parse on ffmpeg 8.1.2, so burn-in caption exports are broken on Windows paths.

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

Re-verified: boundary-dup fix confirmed (commit 5c6be54)

Re-ran the overlay-spanning-boundary repro through the real chunked path on the fix commit:

  • Overlay spanning the 7533ms boundary: 900 frames / 30.000s (was 901/30.029) ✓
  • chunk_00002.mkv = 113 frames (was 114 — the leaked eof_action=repeat clone) ✓
  • No F338 duplicate at the seam — consecutive frames at pts 11.265/11.298 now differ (PSNR ~17dB vs the previous ~74.5dB clone pair) ✓
  • Full spec (zoom + overlay + 2-segment + embedded chapters): 900/30.000, 3 chapters ✓
  • Regressions: overlay-inside-chunk → 900; single-pass → 900 (standalone path unchanged) ✓

Fix verified end-to-end.

@jeandedieuH
jeandedieuH merged commit 8dcbc21 into main Sep 22, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant