Inference time on Linux with openBLAS

[Scriptwonder]

I am running on Linux with OpenBLAS, but it is significantly slower than running via Python (256X256 with 2 steps). Is this intentional as the tradeoff of not using a GPU?

[antirez]

Hi! The problem is that the optimization effort was spent on MPS, but indeed OpenBLAS needs more love. May you tell me how much faster PyTorch is?

[antirez]

I added some BLAS speedup and the ability to tun the threads usage, maybe this will help.

[Scriptwonder]

Hi! The problem is that the optimization effort was spent on MPS, but indeed, OpenBLAS needs more love. May you tell me how much faster PyTorch is?

Sorry for the late reply! PyTorch costs around 10-12 seconds for a loaded 4B model from my usage.

I added some BLAS speedup and the ability to tun the threads usage, maybe this will help.

Sounds great! I will get back to it to try it when I'm free. Have you tested on linux machines with Python for the speedup?

[martin-frbg]

@Scriptwonder what's your hardware, please ?

[Scriptwonder]

@Scriptwonder what's your hardware, please ?

RTX4080 on Linux machine

[martin-frbg]

Thanks - I don't think the GPU is going to be involved in the calculation though, certainly not by OpenBLAS...

[Scriptwonder]

Thanks - I don't think the GPU is going to be involved in the calculation though, certainly not by OpenBLAS...

Oh yes, sorry I forgot that. It was an Intel i9 series CPU; I will need to look at the model number later.

[clort81]

OpenBLAS: Radeon 3500u thinkpad, AMD vega8 gpu [EDIT openblas not using gpu]

$ ./iris -d flux-klein-4b -p "A rustic painting in Norman ll style of a young midwestern boy in overalls named 'Tom' throwing a red brick at the 1800s small-town bank title over the image reads TomSawyer." -W 256 -H 256 -o output.png
BLAS: OpenBLAS | Zen | 8 threads / 8 procs
OpenBLAS 0.3.21 NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Zen MAX_THREADS=64
Seed: 1775591380
Loading VAE... done (0.2s)
Model: FLUX.2-klein-4B v1.0 (distilled, 4 steps, guidance 1.0)
Loading Qwen3 encoder... done (9.3s)
Encoding text... done (88.4s)
Loading FLUX.2 transformer... done (5.1s)
Denoising (d=double block, s=single blocks, F=final):
Step 1/4 dddddssssF
Step 2/4 dddddssssF
Step 3/4 dddddssssF
Step 4/4 dddddssssF
Decoding image... done (9.6s)
Saving... output.png 256x256 (0.3s)
Total generation time: 441.6 seconds

CPU: Radeon 3500u thinkpad, AMD vega8 gpu

$ ./iris -d flux-klein-4b -p "A rustic painting in Norman ll style of a young midwestern boy in overalls named 'Tom' throwing a red brick at the 1800s small-town bank title over the image reads TomSawyer." -W 256 -H 256 -o outputcpu.png
Generic: Pure C backend (no acceleration)
Seed: 1775592215
Loading VAE... done (2.7s)
Model: FLUX.2-klein-4B v1.0 (distilled, 4 steps, guidance 1.0)
Loading Qwen3 encoder... done (9.2s)
Encoding text... done (674.0s)
Loading FLUX.2 transformer... done (2.3s)
Denoising (d=double block, s=single blocks, F=final):
Step 1/4 dddddssssF
Step 2/4 dddddssssF
Step 3/4 dddddssssF
Step 4/4 dddddssssF
Decoding image... done (2268.5s)
Saving... outputcpu.png 256x256 (0.0s)
Total generation time: 5941.3 seconds

13x Speedup with openblas. Seems to have defaulted to 8-threads? 720x720 kinda painful. Curious how we'd get some gpu support for the maths. Someday maybe.

Might be fun to patch-in gguf support to get access to smaller quants for better speed.

Thanks for the awesome program!