CUDA codepath !

[ServeurpersoCom]

I had a lot of fun creating and properly optimizing a codepath for CUDA with Claude Opus 4.5 would you like me to PR it (then I rebase it)?

I use an agentic workflow with Claude Opus 4.5 via MCP (Model Context Protocol). The setup:

• Local infrastructure: dedicated container server equipped with NVIDIA RTX PRO 6000 Blackwell
• Agentic loop: implement -> compile -> benchmark -> compare images (same testing script) -> human review & commit ....
• Observed max_diff=9 for 1024 vs reference image (from the python testing script).
• Speedup 1024 x 1024 from 118.4s on CPU+BLAS to 11.4s on CUDA (10.4x)

WDYT?

CPU+BLAS :

CUDA :

CPU+BLAS :

CUDA :

CPU+BLAS :

CUDA :

[antirez]

Great support to have, the problem is: will it make the building process more complicated, or could the "make cuda" path work well? Is "cuBLAS" a better route? And, given I don't have the hardware to test with CUDA, how to continue the development without breaking it?

[ServeurpersoCom]

Yes it's cuBLAS! Opus has found the right compromise: cuBLAS matmuls for heavy matrix multiplications (QKV projections, FFN, attention scores) and custom CUDA kernels for model-specific operations (RMSNorm, RoPE, AdaLN modulation, softmax, activations).

I could maintain it. Since I work with containers, starting from a basic Debian Minimal system, only make cuda (and not the other backends) requires the cuda keyring (dpkg -i cuda-keyring_1.1-1_all.deb) and apt install cuda-toolkit-13-1.
And if it's not in the path, add LD_LIBRARY_PATH=/usr/local/cuda/lib64.

Look at :
main...ServeurpersoCom:flux2.c:cuda rebased (Slight (invisible) loss of accuracy to be checked)
main...ServeurpersoCom:flux2.c:cuda-backup non rebased (maximum GPU accuracy)

It's a good agentic challenge, I'm having fun benchmarking its performance, finding bottlenecks, the compromise (speed vs memory consumption vs simplicity of code) of the transformer, and the accuracy compared to the reference image.

Continue development until you like the architecture (again, it's a compromise, to have it in C -> That's minimalism, because otherwise it would be better to use GGML like stable-diffusion.cpp, but what is the better minimalism compromise in C? I don't know, it's up to you to choose!) and I would reimplement CUDA on it, it's just well-placed ifdefs, a few subtleties to avoid CPU<->GPU exchanges, and a make cuda path.

[antirez]

Thank yiu, I would like to merge it: it does not make the other builds more complicated, nor the code is more complex, there are just a few ifdefs like the Metal backend, so why not? Especially if from time to time you can refresh it. Do you want me to merge ASAP, and then you can continue working in the rebase code to check what the mismatch is (if rounding errors or what) and to gain more speed? Or you want to work more in your branch and later update the PR with what you achieved? Thanks.

[ServeurpersoCom]

Perfect, so I'll rebase to properly integrate it into the Makefile (which has changed), retest, refine, and send a PR

[ServeurpersoCom]

I did this for educational purposes, because I now want to try integrating GGML to see how much it simplifies multi-backend support; normally, if I'm not mistaken, it's also possible to use it in C. I'm going to make Opus sweat over the subject! In addition, in terms of memory and bandwidth, the GGUF is so much better! In addition, we would clearly see the model graph appear in a single file, which allows for supporting other models later, but ultimately we're just having fun rewriting stable-diffusion in C lol

[antirez]

Please tell us your results! Thanks.

[ServeurpersoCom]

Hey antirez! Here's an update on the GGML experiments.

I ran several tests to see if GGML could replace the manual backends
(BLAS/CUDA/Metal) in flux2.c with a unified abstraction.

What works fine with GGML:

• Using it from pure C is no problem at all, it's just a C lib
• Simple components port easily (Qwen3 text encoder, VAE decoder)
• The graph-based approach (build once, compute once) is elegant
• Multi-backend for free (CUDA + Metal + Vulkan + CPU)

Where I got stuck on the DiT transformer:

• stable-diffusion.cpp implements the exact same FLUX Klein model
  via GGML and it works perfectly, so GGML is clearly capable
• But reproducing their approach turned out to be really hard
  in practice, for several compounding reasons:

1. GGML has almost zero documentation. The only way to learn the
   API patterns (backend buffer management, tensor lifecycle,
   cross-context references) is to read sd.cpp source, which is
   ~3000 lines of dense C++ for the FLUX part alone

2. We hit subtle bugs that each required deep GGML internals
   knowledge to diagnose:

   • Cross-context backend buffer mismatch (GGUF weights in one
     context, compute graph in another -> weights silently read
     as zeros)
   • BF16 matmul not properly supported on CPU backend (our GGUF
     had modulation weights in BF16 instead of quantized)
   • Weight transposition confusion between GGUF and safetensors
     layout conventions in ggml_mul_mat

3. FLUX's custom 2D RoPE (4 axes, [32,32,32,32], separate
   text/image positions, repeat_interleave) was particularly
   painful to express through GGML's tensor ops, which don't
   handle strided non-contiguous views well

4. I use Claude Opus as my agentic coding partner, and its
   context window kept filling up during debug sessions, losing
   state across conversations and re-diagnosing already-found
   issues

Bottom line: GGML works (sd.cpp proves it), but the learning curve
is brutal without proper docs, and the compound complexity of
FLUX's architecture (AdaLN + custom RoPE + joint attention) makes
it a poor fit for a first GGML project. The current approach with
clean C and targeted backend ifdefs remains simpler and more
maintainable for flux2.c's minimalist philosophy.

I'm happy to keep maintaining the CUDA backend on my side!

[ServeurpersoCom]

The existing CPU/BLAS/Metal C code was almost cheating to write CUDA, because the Claude Opus model had a low-level mathematical description of everything that needed to be done and I just optimized in CUDA. These are still concepts firmly rooted in the data; GGML is on another level!

[ServeurpersoCom]

Okay, I just managed to port ACE-step 1.5, the music generator, to pure C++/CUDA with Opus 4.6 Extended. That means GGML is just a matter of documentation

[ServeurpersoCom]

Please tell us your results! Thanks.

Quick GGML update: I ported ACE-Step 1.5 (music generator, Qwen3 > detokenizer > 24L DiT > VAE > 48kHz WAV) to GGML.

It works great (I tested on CUDA/CPU/ARM!). The only real gotcha is that ggml_backend_sched reuses input tensor buffers for intermediates, so if you call graph_compute in a loop your inputs get overwritten silently. Not documented, bit me twice, trivial fix once you know. Everything else was found quickly with layer-by-layer cossim dumps against the Python reference. Had a lot of fun, will definitely do more GGML ports!