feat: support torchao nvfp4 dynamic quantization

[DefTruth]

fixed #1089 #1061

Examples

python3 -m cache_dit.generate flux --compile --nvfp4 --cache --summary

[07-10 05:56:18] [Cache-DiT] FluxPipeline is officially supported by cache-dit. Use it's pre-defined BlockAdapter directly!
[07-10 05:56:18] [Cache-DiT] Applied patch functor FluxPatchFunctor for FluxTransformer2DModel, patched: True
[07-10 05:56:18] [Cache-DiT] Found transformer from diffusers: diffusers.models.transformers.transformer_flux enable check_forward_pattern by default.
[07-10 05:56:18] [Cache-DiT] Match Block Forward Pattern: ['FluxTransformerBlock', 'FluxSingleTransformerBlock'], ForwardPattern.Pattern_1
[07-10 05:56:18] [Cache-DiT] IN:('hidden_states', 'encoder_hidden_states'), OUT:('encoder_hidden_states', 'hidden_states'))
[07-10 05:56:18] [Cache-DiT] Use default 'enable_separate_cfg' from block adapter register: False, Pipeline: FluxPipeline.
[07-10 05:56:18] [Cache-DiT] Collected Context Config: DBCache_F1B0_W8I1M0MC3_R0.12_CFG0, Calibrator Config: None
[07-10 05:56:18] [Cache-DiT] Match Blocks: CachedBlocks_Pattern_0_1_2, for transformer_blocks, cache_context: transformer_blocks_139864774017920, context_manager: FluxPipeline_139864762830816.
[07-10 05:56:18] [Cache-DiT] Registered custom attn backends: native, flash, _sdpa_cudnn, sage, sage3, _native_npu, _npu_fia, flash_varlen.
[07-10 05:56:18] [Cache-DiT] Set attention backend to <native> for module: FluxTransformer2DModel.
[07-10 05:56:18] [Cache-DiT] Quantizing transformer module: FluxTransformer2DModel to nvfp4 ...
[07-10 06:00:36] [Cache-DiT] ---------------------------------------------------------------------------------
[07-10 06:00:36] [Cache-DiT] Quantized        Region: ['FluxTransformerBlock', 'FluxSingleTransformerBlock']  |
[07-10 06:00:36] [Cache-DiT] Quantized Linear Layers: 494    nvfp4              0 (skipped)                   |
[07-10 06:00:36] [Cache-DiT] Quantized Linear Layers: 494    (total)                                          |
[07-10 06:00:36] [Cache-DiT] Skipped   Linear Layers: 0      (total)                                          |
[07-10 06:00:36] [Cache-DiT] Linear           Layers: 494    (total)                                          |
[07-10 06:00:36] [Cache-DiT] ---------------------------------------------------------------------------------
[07-10 06:00:36] [Cache-DiT] Compiling repeated blocks in transformer: FluxTransformer2DModel ...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 28/28 [00:38<00:00,  1.37s/it]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 28/28 [00:03<00:00,  8.71it/s]
[07-10 06:01:20] [Cache-DiT] Can't find Cache Context Options for: FluxSingleTransformerBlock
[07-10 06:01:20] [Cache-DiT] Can't find Parallelism Config for: FluxSingleTransformerBlock
[07-10 06:01:20] [Cache-DiT] Can't find Quantization Config for: FluxSingleTransformerBlock
[07-10 06:01:20] [Cache-DiT] Can't find Cache Context Options for: FluxTransformerBlock
[07-10 06:01:20] [Cache-DiT] Can't find Parallelism Config for: FluxTransformerBlock
[07-10 06:01:20] [Cache-DiT] Can't find Quantization Config for: FluxTransformerBlock
[07-10 06:01:20] [Cache-DiT]
[07-10 06:01:20] [Cache-DiT] 🤗Cache Context Options: FluxTransformer2DModel
[07-10 06:01:20] [Cache-DiT]
[07-10 06:01:20] [Cache-DiT] {'cache_config': DBCacheConfig(cache_type=<CacheType.DBCache: 'DBCache'>, Fn_compute_blocks=1, Bn_compute_blocks=0, residual_diff_threshold=0.12, max_accumulated_residual_diff_threshold=None, max_warmup_steps=8, warmup_interval=1, max_cached_steps=-1, max_continuous_cached_steps=3, enable_separate_cfg=False, cfg_compute_first=False, cfg_diff_compute_separate=True, num_inference_steps=None, steps_computation_mask=None, steps_computation_policy='dynamic', force_refresh_step_hint=None, force_refresh_step_policy='once'), 'name': 'transformer_blocks_139864774017920'}
[07-10 06:01:20] [Cache-DiT] Can't find Parallelism Config for: FluxTransformer2DModel
[07-10 06:01:20] [Cache-DiT]
[07-10 06:01:20] [Cache-DiT] ⚡️Quantization Config: FluxTransformer2DModel: nvfp4
[07-10 06:01:20] [Cache-DiT]
[07-10 06:01:20] [Cache-DiT] ⚡️Cache Steps and Residual Diffs Statistics: FluxTransformer2DModel, Executed Steps: 28, Transformer Executed Steps: 28
[07-10 06:01:20] [Cache-DiT]
[07-10 06:01:20] [Cache-DiT] | Cache Steps | Diffs P00 | Diffs P25 | Diffs P50 | Diffs P75 | Diffs P95 | Diffs Min | Diffs Max |
[07-10 06:01:20] [Cache-DiT] |-------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
[07-10 06:01:20] [Cache-DiT] | 11          | 0.051     | 0.086     | 0.105     | 0.139     | 0.187     | 0.051     | 0.209     |
[07-10 06:01:20] [Cache-DiT]
[07-10 06:01:20] [Cache-DiT] 🤖 Example Init Config Summary:
[07-10 06:01:20] [Cache-DiT] - Model: /workspace/dev/vipdev/hf_models/FLUX.1-dev
[07-10 06:01:20] [Cache-DiT] - Task Type: T2I - Text to Image
[07-10 06:01:20] [Cache-DiT] - Torch Dtype: torch.bfloat16
[07-10 06:01:20] [Cache-DiT] - Warmup Seed: None
[07-10 06:01:20] [Cache-DiT] - Warmup Prompt: None
[07-10 06:01:20] [Cache-DiT] - LoRA Weights: None
[07-10 06:01:20] [Cache-DiT] 🤖 Example Input Summary:
[07-10 06:01:20] [Cache-DiT] - prompt: A cat holding a sign that says hello world
[07-10 06:01:20] [Cache-DiT] - height: 1024
[07-10 06:01:20] [Cache-DiT] - width: 1024
[07-10 06:01:20] [Cache-DiT] - num_inference_steps: 28
[07-10 06:01:20] [Cache-DiT] - generator: device cpu, seed 0
[07-10 06:01:20] [Cache-DiT] 🤖 Example Output Summary:
[07-10 06:01:20] [Cache-DiT] - Model: flux
[07-10 06:01:20] [Cache-DiT] - Optimization: C1_DBCache_F1B0_W8I1M0MC3_R0.12_CFG0_T0O0_nvfp4_S11
[07-10 06:01:20] [Cache-DiT] - Device: NVIDIA RTX PRO 5000 72GB Blackwell x 1
[07-10 06:01:20] [Cache-DiT] - Load Time: 0.28s
[07-10 06:01:20] [Cache-DiT] - Warmup Time: 38.86s
[07-10 06:01:20] [Cache-DiT] - Inference Time: 3.55s
[07-10 06:01:20] [Cache-DiT] Image saved to flux.1024x1024.C1_DBCache_F1B0_W8I1M0MC3_R0.12_CFG0_T0O0_nvfp4_S11.png