[feat] Add MiniMax-H3 disaggregated inference across two DGX Sparks

[Satyam-53]

Purpose

Enable MiniMax-H3 inference across two DGX Sparks by keeping the encoder and VAEs resident on one machine and the DiT on the other. This distributes model memory across the pair and avoids repeated component loading between stages.

Add payload and timing diagnostics to identify how much latency comes from computation versus inter-node handoffs.

Changes

• Add resident encoder/decoder and DiT pipelines for MiniMax-H3, including reference-conditioned variants.
• Integrate a two-node Ray executor with explicit node placement, topology validation, worker health checks, synchronous/asynchronous execution, and bounded request pipelining.
• Pass minimal, versioned tensor payloads directly between actors through Ray ObjectRefs. Remove explicit CUDA-to-CPU staging at stage boundaries.
• Add request-correlated payload logs and optional profiling through FASTVIDEO_H3_PROFILE_TRANSFERS=1, separating producer wait, object fetch, tensor materialization, and encode/denoise/decode time.
• Forward FASTVIDEO_FA4 to both actors before attention imports, log worker environments, and report actual backend import failures. Preserve VSA for the main H3 transformer.
• Add regression tests and a two-Spark setup guide covering deployment, memory configuration, pipelining, profiling, and FA4 troubleshooting.

Test Plan

CPU regression tests in the fastvideo conda environment:

python -m pytest \
  fastvideo/tests/worker/test_minimax_h3_ray_env.py \
  fastvideo/tests/worker/test_minimax_h3_transfer_timing.py \
  fastvideo/tests/api/test_flash_attention_import_selection.py \
  fastvideo/tests/api/test_attention_selector_resolution.py -q

Targeted pre-commit checks:

python -m pre_commit run --files \
  fastvideo/worker/minimax_h3_disaggregated.py \
  fastvideo/tests/worker/test_minimax_h3_ray_env.py \
  docs/getting_started/installation/minimax_h3_disaggregated_spark_pair.md

Also verified FA4 environment propagation with real local Ray actors, including an already initialized Ray instance, explicit runtime-environment overrides, and inherited uploaded job code.

Manual generation on the Spark pair used the local three-request script with transfer profiling enabled:

python /tmp/fasth3_req_video.py 2>&1 | tee /tmp/fasth3_req_video.log

The generation run used FastVideo/FastVideo-FastH3-4-step-Preview-v1-VSA-DataFree, with 832×480 output, 124 frames, and 5 inference steps.

Test Results

Test output and generation measurements

CPU regression output:

70 passed in 0.09s

Targeted pre-commit checks passed. The CUDA selector change also passed its targeted pre-commit checks.

Local Ray smoke-test output:

PASS: FA4 reaches actor imports on existing Ray; explicit override wins; uploaded job code and environment inherit.

All three manual Spark requests completed and saved videos:

Request	End-to-end latency	A → B receive	B → A receive	Combined receive
1	110.66 s	252.895 ms	93.287 ms	346.182 ms
2	84.95 s	68.728 ms	44.050 ms	112.778 ms
3	76.98 s	37.778 ms	83.382 ms	121.160 ms

Receive time includes Ray object fetching and tensor materialization, excluding producer wait. These measurements are not pure network wire time or a measured speedup against single-machine execution.

The generation measurements precede the final FA4 propagation fix. Full generation with FA4 and SSIM regression validation remain unverified.

Checklist

• I ran pre-commit run --all-files and fixed all issues
• I added or updated tests for my changes
• I updated documentation if needed
• I considered GPU memory impact of my changes

For model/pipeline changes, also check:

• I verified SSIM regression tests pass
• I updated the support matrix if adding a new model — N/A; this adds an execution mode for an existing model

[mergify]

Merge Protections

🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI

	Protection	Waiting on
🔴	PR merge requirements	👀 reviews and 🤖 CI

🔴 PR merge requirements

Waiting for

• #approved-reviews-by>=1
• check-success=fastcheck-passed
• check-success=full-suite-passed

This rule is failing.

• #approved-reviews-by>=1
• check-success=fastcheck-passed
• check-success=full-suite-passed
• check-success~=pre-commit
• title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]