FLUX.2-klein's slow generation speed.

[daniandtheweb]

Closed #1214 by mistake.

I've been testing FLUX.2-klein in both the 4B and 9B variants and the generation speed is unusually slow (I expect generation to be slower than pytorch but this much seems wrong).

Here's some of the results:

FLUX.2-klein-4B: 1024x1024, 4 steps, cfg 1
ComfyUI: 5s
stable-diffusion.cpp: 25s (tested with Vulkan and ROCm)

[daniandtheweb]

During the generation there seems to be a big bottleneck that hits the GPU, the total power draw stops at 170W instead of going all the way up to 250W like the program does with any other model.

[HemanLin-cl]

Hi,

I've also tried FLUX.2-klein-4B model under Vulkan. However, under my platform, I cannot generate images 1024x1024 successfully.
Instead, I tried to add "--vae-conv-direct --vae-tiling" and set resolution as 512x512, it can generate successfully.

Platform Info:
PantherLake
CPU: Intel Core Ultra X7 358H
GPU: Intel Arc [0] GPU

This is my command:

sd-cli.exe --diffusion-model flux-2-klein-4b.safetensors --vae flux2-vae.safetensors --llm qwen_3_4b.safetensors -p "a lovely cat" --cfg-scale 1.0 --steps 4 -v --offload-to-cpu --diffusion-fa --vae-conv-direct --vae-tiling

And the time performance is like: FP16 (43.65 sec) , q8_0 (26.6 sec), q4_0 (23.35 sec)

By applying q8/q4, the resolution can be set as 768x768, and the time performance is q8 ( 70 sec)/q4(70.55 sec)

Is this number as expected?
Could we use llm as q8 and vae as tae to save time?

Thanks

[daniandtheweb]

The issue is actually about the unusual performance difference between ComfyUI and stable-diffusion.cpp. I can't really tell if your performance numbers are expected or not without a baseline to compare it against.

[wbruna]

I don't have a ComfyUI setup right now, but my 7600 XT (Linux, ROCm 6.4, radv) reaches:

	ROCm s/it	Vulkan s/it
FLUX.2 Klein 4B (CFG 1, 4 steps)	11.78	7.25
FLUX.1 Schnell (CFG 1, 4 steps)	19.45	13.67
Z-Image Turbo (CFG 1, 8 steps)	9.80	8.09
SDXL (CFG 1, 8 steps)	1.47	1.55

I also notice lower and less stable power draw with Klein 4B (~72W on ROCm, ~88W on Vulkan, several peaks reaching ~97W) compared with other models (almost flat 110W).

[daniandtheweb]

I've done some more accurate testing after updating everything and setting up correctly both stable-diffusion.cpp and ComfyUI. Unfortunately the performance difference isn't quite as close as what I thought it would be at first (ROCm is still a bit better in SDXL), however the power usage difference still stands.

All the tests are performed on a Radeon RX 7800XT on Arch Linux, using the latest mesa-git build.

Model	ComfyUI (cold start)	ComfyUI	ComfyUI Power	Vulkan	Vulkan Power	Speed difference
SDXL (CFG 4, 30 steps)	16.86s	12.36s	248W	29.63s	240W	2.39x slower
Z-Image Turbo (CFG 1, 9 steps)	38.13	20.84s	248W	34.57s	220W	1.65x slower
FLUX.2 Klein 4B (CFG 1, 4 steps)	16.22s	6.24s	246W	23.32s	175W (small peaks to 210W)	3.74x slower
FLUX.2 Klein 9B (CFG 1, 4 steps)	26.93s	17.32s	248W	33.50s	175W (small peaks to 220W)	1.93x slower

During generation it doesn't seem like there's any CPU offloading in FLUX.2 Klein so I think there's some kind of bottleneck slowing down the generation unrelated to the ggml backend.

[leejet]

@daniandtheweb I’m not sure whether you’re using --diffusion-fa; it can significantly improve performance on many devices.

[daniandtheweb]

Yes, i have tested all the settings to find the best configuration on my card: --diffusion-fa, conv direct for vae (when used on diffusion too it's slightly slower on my card).

[daniandtheweb]

Maybe some of the required ggml operations are just not well optimized?

[leejet]

On my RTX 4090 using the CUDA backend, sd.cpp is slightly slower than ComfyUI:
sd.cpp runs at about 6.7 it/s, while ComfyUI reaches 10.5 it/s.
So there is indeed a noticeable gap. I’ll look into how to optimize this later.

[JohnLoveJoy]

Let's do a backflip and implement FA4.

https://modal.com/blog/reverse-engineer-flash-attention-4

[leejet]

@daniandtheweb I’ve made some optimizations #1228 — you can try it and see if there’s any improvement.

[daniandtheweb]

The optimizations look amazing. The power usage is now also in line with any other model. Here's the results with --diffusion-fa --vae-conv-direct.

Model	Vulkan fa61ea7	Vulkan Power fa61ea7	Vulkan c7d4a60	Vulkan Power c7d4a60	Improvement
FLUX.2 Klein 4B (CFG 1, 4 steps)	23.32s	175W (small peaks to 210W)	15.90s	240W	31.8% faster
FLUX.2 Klein 9B (CFG 1, 4 steps)	33.50s	175W (small peaks to 220W)	23.68s	240W	29.3% faster

The generation speed improves in Klein 4B from 5.46 s/it to 3.62s/it and in the case of Klein 9B from 8.46s/it to 5.47s/it.

[daniandtheweb]

Here's also a performance comparison for ROCm on the same RX 7800XT card. In this case the performance settings have been changed to --diffusion-fa --diffusion-conv-direct, ROCm doesn't seem to like conv direct on VAE.

Model	ROCm fa61ea7	ROCm c7d4a60	Improvement
FLUX.2 Klein 4B (CFG 1, 4 steps)	22.09s - 4.83s/it	10.27s - 1.84s/it	53.5% faster
FLUX.2 Klein 9B (CFG 1, 4 steps)	44.62s - 10.42s/it	18.50s - 3.90s/it	58.5% faster