Maybe T5 model on the CPU black image fix

[phil2sat]

i dont know if im right, but building with:
-DSD_HIPBLAS=ON
for me flux chroma Q8_0 runs slow ~38s/it, 832x1216, dont laugh.

all fine cant run the model in f16 because of VRAM only 16GB
clip on gpu no issue all quants i.O
get_learned_condition completed, taking 1428 ms (nice)

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon Pro WX 9100, gfx900:xnack- (0x900), VMM: no, Wave Size: 64

but if i build with:
-DSD_HIPBLAS=ON -DGGML_CUDA_FORCE_CUBLAS=ON
the inference gets about 40% faster ~21s/it, same settings but black picture.
clip on cpu fixes the issue
get_learned_condition completed, taking 45811 ms, again dont laugh it a stoneage CPU (bad)

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: yes
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon Pro WX 9100, gfx900:xnack- (0x900), VMM: no, Wave Size: 64

the information about i got from this table
https://github.com/user-attachments/files/18832391/MI60-MI25-MI100_MAT_MUL.xlsx
ggml-org/llama.cpp#11931

what is exact what i could replicate.

my idea maybe its possible to deactivate cublas for t5xxl or other text encoders if HIPBLAS=ON

maybe someone else tries to test on different rocm platforms and gpu's or maybe also for nvidia if its not only a amd issue

[Green-Sky]

Does --diffusion-fa make the generation slower on your device. Upstream code changed since the last time it was tested (or reported to me), I think.

[phil2sat]

yes with fa its about from 1.5s/it in SDXL to 8s/it, maybe i fetch latest git and rebuild

[Green-Sky]

yes with fa its about from 1.5s/it in SDXL to 8s/it

I see.

, maybe i fetch latest git and rebuild

I do not think this will change anything. :(

Does vulkan perform better on your device?

[phil2sat]

vulkan, not really rather worse.
dont get me wrong thats a 40%+
but Clip works T5 not if i only could seperate it to disable cublas for only T5, then im fine.
i guess it has something to do with gfx900 not supporting WMMA cause its gcn5 and not rdna or cdna.
had to fight alot angainst pytorch / triton and many other py packages on my comfyui venv to get it working.

[phil2sat]

Funny thing... WAN 2.1 UMT5-XXL works without clip-on-cpu.

[stduhpf]

Funny thing... WAN 2.1 UMT5-XXL works without clip-on-cpu.

Yes, I believe that's the reason clip-on-cpu isn't forced when a t5 models is used anymore.

[leejet]

Funny thing... WAN 2.1 UMT5-XXL works without clip-on-cpu.

Yes, I believe that's the reason clip-on-cpu isn't forced when a t5 models is used anymore.

Yes, this is the reason why I don’t force T5 to run on the CPU.

[phil2sat]

T5-XXL and UMT5-XXL, both run fine on GPU and CPU.
problem it takes 45sec. on CPU for me

and if i enable -DSD_HIPBLAS=ON -DGGML_CUDA_FORCE_CUBLAS=ON (CUBLAS is the part)
T5-XXL goes black image, UMT5-XXL keeps generating Images

enabling cublas gives me a speed boost of around 40% in iference but for flux it gets eaten up by the extra 45sec. text encoder.

Maybe its also related to how FLUX is implemented and there is a bug and since WAN is another implementation, there it works

[leejet]

This PR should be able to completely resolve the issue of generating black images when running T5 on the GPU. #881

[phil2sat]

It does, just tested
Thanks

Chroma-8step 1024x1024
GGML_CUDA_FORCE_CUBLAS: no ~40s/it
GGML_CUDA_FORCE_CUBLAS: yes ~25s/it
both produce an image now