[feat] Route FastWan 5B FullAttn through dense attention - #1888
Open
fusheng-ji wants to merge 5 commits into
Open
fusheng-ji wants to merge 5 commits into
fusheng-ji wants to merge 5 commits into
Conversation
Register the FullAttn checkpoint as a T2V-only Wan variant and reject incompatible sparse attention or image-conditioning requests at load time. Carry the resolved per-component attention backend through Wan block construction and denoising metadata. Add API regressions and inference documentation.
…config When a FullAttn checkpoint is trained with VSA sparsity, select the VSA-capable FastWan 5B pipeline config for the student while keeping dense FullAttn inference guards unchanged.
Replace the 1.3B basic_dmd example with the existing 5B DMD YAML and document the FullAttn-to-VSA LoRA training exception.
Explain metadata-first routing, dense attention propagation, and the FullAttn-to-VSA training exception. Keep manifest matching semantics while making the checks and regressions easier to read.
mergify
Bot
added
type: feat
Contributor
Merge Protections🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI
🔴 PR merge requirementsWaiting for
This rule is failing.
|
fusheng-ji
marked this pull request as ready for review
September 25, 2026 16:52
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+306
to
+308
| if workload != "t2v": | ||
| raise ValueError(f"FastWan2.2-TI2V-5B-FullAttn does not support workload type {workload!r}; " | ||
| "supported workload: 't2v'.") |
There was a problem hiding this comment.
image_path, but this guard checks only the workload label. Input validation loads the image, while WanDMDPipeline has no first-frame encoding stage and this config disables TI2V conditioning. The request therefore returns a text-only video without warning the caller that its image was ignored.
Prompt To Fix With AI
This is a comment left during a code review.
Path: fastvideo/models/wan/pipeline_config.py
Line: 306-308
Comment:
**Image input is silently ignored** A T2V request can still include `image_path`, but this guard checks only the workload label. Input validation loads the image, while `WanDMDPipeline` has no first-frame encoding stage and this config disables TI2V conditioning. The request therefore returns a text-only video without warning the caller that its image was ignored.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Comment on lines
+306
to
+309
| if workload != "t2v": | ||
| raise ValueError(f"FastWan2.2-TI2V-5B-FullAttn does not support workload type {workload!r}; " | ||
| "supported workload: 't2v'.") | ||
| if coerce_attn_backend(attention_backend) is AttentionBackendEnum.VIDEO_SPARSE_ATTN: |
There was a problem hiding this comment.
FastVideoArgs.from_kwargs(model_path=FULL, workload_type="t2v", ti2v_task=True) applies ti2v_task after this config’s __post_init__, but the guard does not check the resulting value. If that request includes an image, Wan denoising expects a first-frame latent that WanDMDPipeline never prepared and fails instead of rejecting the unsupported configuration up front.
Prompt To Fix With AI
This is a comment left during a code review.
Path: fastvideo/models/wan/pipeline_config.py
Line: 306-309
Comment:
**Override re-enables unsupported conditioning** `FastVideoArgs.from_kwargs(model_path=FULL, workload_type="t2v", ti2v_task=True)` applies `ti2v_task` after this config’s `__post_init__`, but the guard does not check the resulting value. If that request includes an image, Wan denoising expects a first-frame latent that `WanDMDPipeline` never prepared and fails instead of rejecting the unsupported configuration up front.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
The FastWan 2.2 5B FullAttn checkpoint needs dense attention and supports
text-to-video only. Its existing shared routing can select VSA or silently
accept image conditioning, while Wan block construction and denoising metadata
can resolve different attention backends for the same component.
Related to #864. This independently addresses the FullAttn runtime guards in
#1563 and overlaps backend-selection work in #1494. The implementation adds
metadata-first detection and per-component backend propagation; I am happy to
coordinate or consolidate if maintainers prefer.
Changes
model_index.jsonmetadata before name-based fallbacks.denoising metadata so construction and execution agree.
the existing VSA-capable 5B config only for that training case; the student
uses VSA while teacher/critic remain dense. FullAttn inference still rejects
VSA and I2V.
T2V-only listing, and make the inference guide invoke the real 5B CLI config.
Test Plan
GPU validation used the pinned official checkpoint revision
3e187042a324f6f5fb68fd22110a78725253de8f, a privateFlashAttention-2 2.8.1 overlay, and
CUDA_VISIBLE_DEVICESresource checks.The current independent quality comparison ran unchanged official
main@e90be598and candidateecd22f5bon the same B200 with the sameprompt, seed 1024, three DMD steps, 704×1280, 24 fps, FlashAttention-2 and
compile enabled. A historical
num_frames=104request on the previous8760eb7abaseline produced 101 native frames on both sides.Why the 104-frame deliverable is cropped
The current Wan path compresses time by a factor of four: a request for
Nframes prepares
floor((N - 1) / 4) + 1latent frames, and the VAE decodesthose latents to
4 * (latent_frames - 1) + 1frames. Native decoded lengthstherefore have the form
4k + 1. Requesting 104 gives 26 latent framesand 101 decoded frames; requesting 105 gives 27 latent frames and
105 decoded frames. There is no native 104-frame output in this unchanged path.
To deliver exactly 104 frames without changing Wan's VAE or the public
inference behavior, we generate the next valid native length (105) and keep
the first 104 decoded frames, dropping only the last frame. We apply this
identical local postprocessing to official
mainand the candidate, retainboth 105-frame originals, and verify that every retained frame has identical
decoded pixels to its source. This is a validation/output-format step, not a
claim that Wan natively supports 104 frames. The 104-frame and full 105-frame
MS-SSIM comparisons passed both on the historical
8760eb7abaseline andon the current rebased official
main@e90be598baseline.Test Results
render record binds clean HEAD
ecd22f5bagainst clean officialmain@e90be598; both use the same fixed checkpoint and environment.pre-commit run --all-files: all hooks passed on an underscore-namedtemporary worktree at clean HEAD
ecd22f5b, including mypy. The originalfastwan-fullattndirectory name alone trips mypy's invalid package-namecheck.
0 skipped, 14 warnings in 539.96s. Optional GPU kernels were
stubbed under
CUDA_VISIBLE_DEVICES=''for parent and fresh child Pythonprocesses; the stubs fail if actually called. The additional tests cover
case-insensitive manifest matching and strict rejection of integer
1as aboolean marker. New API and training tests are collected by the existing
Buildkite unit directories.
main@e90be598and candidateecd22f5beach generateda decodable native 105-frame, 1280×704, 24-fps video on the same B200
GPU 0 with identical pinned weights, prompt, seed 1024, three DMD steps,
FlashAttention-2 and Torch Compile. Both source trees were clean; effective
backend and checkpoint were confirmed. Sampled own-process peak GPU memory
was 37,236 MiB for main and 38,962 MiB for candidate, with no
pressure stop. The two native MP4 files have the same SHA-256.
to its first 104 frames using the same
libx264 -qp 0settings. Fulldecode and
ffprobeconfirmed 105/104 frames, 1280×704, 24 fps on bothsides; every retained frame's decoded-pixel checksum equals its own native
source frame. The two cropped MP4 files are byte-identical. Repository
MS-SSIM over all 104 cropped frames was mean/min/max 1.000000;
over all 105 native frames it was also mean/min/max 1.000000, including
frame 105. Both exceed the predeclared 0.93 mean threshold. CPU-hidden
comparison records retain every per-frame score; no public CI reference was
uploaded.
TORCH_SDPA and FlashAttention-2 renders; a synthetic FullAttn-to-VSA LoRA
forward/backward with finite loss and finite, nonzero trainable gate
gradients; and two-B200
sp_size=2FlashAttention-2 generation whose49-frame output matched the single-B200 render (MS-SSIM 1.000000). These
focused runs predate the current-main rebase; the full-size independent
comparison above uses the current clean candidate and official main.
Checklist
pre-commit run --all-filesand fixed all issuesunchanged; selecting dense FullAttn can use more memory than VSA
For model/pipeline changes:
main@e90be598: mean 1.000000 ≥ 0.93 for both the 104 cropped framesand all 105 native frames, including the final frame.
5B CLI guide.