clip related crash, segmentation fault

[kuesji]

it seems like 48956ff introduced a bug and it causes crash on my laptop.
i tried to compile latest release but it was crashing. i was able the trace the issue back to that commit.

let me know if anything else is needed. i'm going to stay at ddc4a18 for now.

output of execution with -v arg

Option:
n_threads: 8
mode: img_gen
model_path: /data/kuesji/archive/models/image_generation/models/sdxl_illustrious:hassakuxl:v2.2.gguf
wtype: unspecified
clip_l_path:
clip_g_path:
clip_vision_path:
t5xxl_path:
diffusion_model_path:
high_noise_diffusion_model_path:
vae_path:
taesd_path:
esrgan_path:
control_net_path:
embedding_dir:
stacked_id_embed_dir:
input_id_images_path:
style ratio: 20.00
normalize input image: false
output_path: output.png
init_image_path:
end_image_path:
mask_image_path:
control_image_path:
ref_images_paths:
increase_ref_index: false
offload_params_to_cpu: false
clip_on_cpu: false
control_net_cpu: false
vae_on_cpu: true
diffusion flash attention: false
diffusion Conv2d direct: false
vae_conv_direct: false
control_strength: 0.90
prompt: purple cat
negative_prompt:
clip_skip: -1
width: 576
height: 1024
sample_params: (txt_cfg: 3.50, img_cfg: 3.50, distilled_guidance: 3.50, slg.layer_count: 3, slg.layer_start: 0.01, slg.layer_end: 0.20, slg.scale: 0.00, scheduler: karras, sample_method: euler_a, sample_steps: 8, eta: 0.00)
high_noise_sample_params: (txt_cfg: 7.00, img_cfg: 7.00, distilled_guidance: 3.50, slg.layer_count: 3, slg.layer_start: 0.01, slg.layer_end: 0.20, slg.scale: 0.00, scheduler: default, sample_method: euler_a, sample_steps: -1, eta: 0.00)
moe_boundary: 0.875
flow_shift: inf
strength(img2img): 0.75
rng: cuda
seed: 1329308695
batch_count: 1
vae_tiling: false
upscale_repeats: 1
chroma_use_dit_mask: true
chroma_use_t5_mask: false
chroma_t5_mask_pad: 1
video_frames: 1
fps: 16
System Info:
SSE3 = 1
AVX = 1
AVX2 = 1
AVX512 = 0
AVX512_VBMI = 0
AVX512_VNNI = 0
FMA = 1
NEON = 0
ARM_FMA = 0
F16C = 1
FP16_VA = 0
WASM_SIMD = 0
VSX = 0
[DEBUG] stable-diffusion.cpp:151 - Using Vulkan backend
[DEBUG] ggml_extend.hpp:62 - ggml_vulkan: Found 1 Vulkan devices:
[DEBUG] ggml_extend.hpp:62 - ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
[INFO ] stable-diffusion.cpp:201 - loading model from '/data/kuesji/archive/models/image_generation/models/sdxl_illustrious:hassakuxl:v2.2.gguf'
[INFO ] model.cpp:1035 - load /data/kuesji/archive/models/image_generation/models/sdxl_illustrious:hassakuxl:v2.2.gguf using gguf format
[DEBUG] model.cpp:1052 - init from '/data/kuesji/archive/models/image_generation/models/sdxl_illustrious:hassakuxl:v2.2.gguf'
[INFO ] stable-diffusion.cpp:267 - Version: SDXL
[INFO ] stable-diffusion.cpp:301 - Weight type: q4_K
[INFO ] stable-diffusion.cpp:302 - Conditioner weight type: q4_K
[INFO ] stable-diffusion.cpp:303 - Diffusion model weight type: q4_K
[INFO ] stable-diffusion.cpp:304 - VAE weight type: f32
[DEBUG] stable-diffusion.cpp:306 - ggml tensor size = 400 bytes
[WARN ] stable-diffusion.cpp:311 - !!!It looks like you are using SDXL model. If you find that the generated images are completely black, try specifying SDXL VAE FP16 Fix with the --vae parameter. You can find it here: https://huggingface.co/madebyollin/sdxl-vae-fp16-fix/blob/main/sdxl_vae.safetensors
[DEBUG] clip.hpp:171 - vocab size: 49408
[DEBUG] clip.hpp:182 - trigger word img already in vocab
[DEBUG] ggml_extend.hpp:1609 - clip params backend buffer size = 66.61 MB(VRAM) (196 tensors)
[DEBUG] ggml_extend.hpp:1609 - clip params backend buffer size = 380.09 MB(VRAM) (517 tensors)
[DEBUG] ggml_extend.hpp:1609 - unet params backend buffer size = 1960.49 MB(VRAM) (1680 tensors)
[INFO ] stable-diffusion.cpp:455 - VAE Autoencoder: Using CPU backend
[DEBUG] ggml_extend.hpp:1609 - vae params backend buffer size = 94.47 MB(RAM) (140 tensors)
[DEBUG] stable-diffusion.cpp:564 - loading weights
[DEBUG] model.cpp:1998 - loading tensors from /data/kuesji/archive/models/image_generation/models/sdxl_illustrious:hassakuxl:v2.2.gguf
|==================================================| 2641/2641 - 1586.19it/s
[INFO ] model.cpp:2216 - loading tensors completed, taking 1.67s (process: 0.00s, read: 0.74s, memcpy: 0.00s, convert: 0.01s, copy_to_backend: 0.77s)
[INFO ] stable-diffusion.cpp:645 - total params memory size = 2501.66MB (VRAM 2407.19MB, RAM 94.47MB): text_encoders 446.70MB(VRAM), diffusion_model 1960.49MB(VRAM), vae 94.47MB(RAM), controlnet 0.00MB(VRAM), pmid 0.00MB(VRAM)
[INFO ] stable-diffusion.cpp:713 - running in eps-prediction mode
[DEBUG] stable-diffusion.cpp:724 - finished loaded file
[DEBUG] stable-diffusion.cpp:2180 - generate_image 576x1024
[INFO ] stable-diffusion.cpp:736 - running with Karras scheduler
[INFO ] stable-diffusion.cpp:2313 - TXT2IMG
[INFO ] stable-diffusion.cpp:858 - attempting to apply 0 LoRAs
[INFO ] stable-diffusion.cpp:878 - apply_loras completed, taking 0.00s
[DEBUG] stable-diffusion.cpp:879 - prompt after extract and remove lora: "purple cat"
[DEBUG] conditioner.hpp:345 - parse 'purple cat' to [['purple cat', 1], ]
[DEBUG] clip.hpp:311 - token length: 77
[DEBUG] ggml_extend.hpp:1437 - clip compute buffer size: 1.40 MB(VRAM)
Segmentation fault (core dumped) ./build/bin/sd -m "$models/image_generation/models/sdxl_illustrious:hassakuxl:v2.2.gguf" --seed -1 --steps 8 --sampling-method euler_a --scheduler karras --cfg-scale 3.5 -W 576 -H 1024 -p "purple cat" --vae-on-cpu -v

[stduhpf]

does --type f32 fix it?

[kuesji]

it doesn't have enough memory to test--type f32 but f16 worked fine. also q4_0 and q4_1 works too. problem is q4_k.

[stduhpf]

Maybe it's somehow related to that?
https://github.com/leejet/stable-diffusion.cpp/blob/master/docs/lora.md#support-matrix

Does --clip-on-cpu work as a workaround to avoid the crash?

[wbruna]

Passing --embd-dir (pointing to any directory) should work around the crash, too.

[kuesji]

both "--clip-on-cpu" and "--embd-dir" with empty directory avoids crash. also i don't use any lora.

[stduhpf]

I know you're not using loras, but my guess is that it's crashing for the same underlying reason lora crashes with some types on cuda (with get_rows op).

[wbruna]

I was able to reproduce it here on Vulkan, with an SD1.5 model converted to Q4_K. Probably for the same reason: https://github.com/ggml-org/ggml/blob/5fdc78fff274094e2a1b155928131983362d8a71/src/ggml-vulkan/ggml-vulkan.cpp#L11940

[kuesji]

I know you're not using loras, but my guess is that it's crashing for the same underlying reason lora crashes with some types on cuda (with get_rows op).

i see. it looks like previous versions always used f32 for clip but this changed after that commit with force_clip_f32 check. that explains why it causes problems when embd-dir is not provided.

stable-diffusion.cpp/conditioner.hpp

Line 66 in 79426d5

bool force_clip_f32 = embd_dir.size() > 0;

solution?

• providing empty embd-dir
• keeping models with non-K quantization
• using clip on cpu
• implementing guardrail to pick nearest supported quantization in the case of unsupported quantization (?)

i'm not sure what to do about crash by default or crash if not supplied with an argument behaviour though.

[wbruna]

Oh, it definitely shouldn't crash 🙂 Giving a bit of context: passing an embeddings directory turns on the embd_dir.size() > 0 check, which replicates at runtime the reason for the upcast to f32 in the first place ( fbd42b6 ). But it turns out that change helps to avoid other quant issues, like this one.

@stduhpf , I guess the more appropriate fix would be calling ggml_backend_supports_op to check for get_rows availability?

[stduhpf]

@wbruna yes, but the backend isn't passed to init_params(), so it will be a bit annoying to implement there.

Checking during the forward pass could work, but I'm not sure what you could do instead if get_rows is not supported.

[leejet]

I think it's best to keep token_embedding.weight in f16 or f32 without quantization. In fact, it's generally not recommended to quantize token_embedding.weight, as quantization may severely affect the model's accuracy.

[leejet]

You can check out this PR #840

[stduhpf]

I think it's best to keep token_embedding.weight in f16 or f32 without quantization. In fact, it's generally not recommended to quantize token_embedding.weight, as quantization may severely affect the model's accuracy.

On the other hand, token_embedding.weight can make up a significant amount of the total parameters of the model. If I remember correctly, llama.cpp quantizes it to Q6_K wtype for Q4_0 ftype, and runs the get_rows on CPU.