Difference of gguf conversion ?

[razvanab]

Are there any differences in the GGUF conversion between stable-diffusion.cpp and the rest?

Almost any GGUF I tried doesn't work in stable-diffusion.cpp and needs the main model to be converted with the sd.exe specifically to work.

[stduhpf]

As far as I know only Stable Diffusion 3/3.5 GGUFs are incompatible between ComfyUI GGUF and stable-diffusion.cpp, because ComfyUI-GGUF slightly modifies some tensor's shape for an obscure reason).

The main difference for all other models is that the GGUFs made with sd.exe convert all tensors to the same quant type, except a few that are kept at full precision, while GGUFs made with other tools tend to use a mixture of different quant types within a same file. One of the reason some GGUFs might not work well is that the version of GGML used by stable-diffusion.cpp is a bit out of date, and some types are poorly supported by most backends (Example: BF16 on Vulkan).

[razvanab]

I see.

I tried with Chroma and T5, and I got some errors.
chroma-unlocked-v40-Q8_0.gguf

Both
flan_t5_xxl_TE-only_Q6_K.gguf

and
t5-v1_1-xxl-encoder-Q6_K.gguf

log.txt

[stduhpf]

Ah yes, I forgot about T5. The T5xxl quants you're trying to use are made for llama.cpp. Llama.cpp renames all tensors to their own convention, while stable-diffusion.cpp uses the original naming convention of the model.

It should work with https://huggingface.co/Green-Sky/flux.1-schnell-GGUF/blob/main/t5xxl_q4_k.gguf or https://huggingface.co/Green-Sky/flux.1-schnell-GGUF/blob/main/t5xxl_q8_0.gguf, for example, or there's also this patch: #654 I made earlier that would help support the one you tried.

[razvanab]

Nice.
Thank you.

[Green-Sky]

@razvanab #654 has been merged.

[advanced-lvl-up]

I talked about this in the Discussions thread also, and it's probably been mentioned before but can we please drop the bulky T5 requirement and make it optional? for simple testing it is not needed for much of anything? thank you and nice work on the project so far!

[razvanab]

I don't think that Flux and other more recent image-to-text models will work without T5. Please correct me if I am wrong.

[Green-Sky]

Also referencing #645 here.
Leaving out embedders is very model dependent.

[github-actions]

This issue has had no activity for one year. The latest version of the code may already have fixed the problem.
If the issue still exists in the latest version, you can reopen this issue at any time with updated reproduction details.