Got Qwen3.8-Flash-Next running on 4x MI50 (gfx906, 32GB each, ROCm 6.3.3) with this fork — thanks for carrying the qwen4exp code. Sharing the exact working recipe and two things that cost me a load cycle each, in case it helps the next gfx906 person.
Build (in a rocm-gfx906 container, since the host toolchain is minimal):
cmake -B build -G Ninja -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx906 -DGPU_TARGETS=gfx906 \
-DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF && cmake --build build -j8
Run (unsloth UD-Q4_K_XL, 4 shards, TP4):
llama-server -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
-ngl 99 -ot per_layer_token_embd=CPU --split-mode tensor -tps 4 \
-c 32768 -np 1 -fa on --jinja --no-mmap
Two gotchas:
--split-mode tensor requires -fa on. With -fa off it aborts at load with SPLIT_MODE_TENSOR requires flash_attn to be enabled. Most gfx906 guidance says "start with -fa off," so this bites — worth a note in the README, or defaulting FA on when tensor-split is set.- The PLE table is
per_layer_token_embd (not ple_ngram_embd), so -ot per_layer_token_embd=CPU is the right override to keep the ~51B n-gram table in host RAM (needs ~51GB+ free).
Measured: single-stream decode ~20-21 tok/s at 256-token gen, coherent output. Keeping the PLE on-GPU instead of CPU (dropping the -ot) gave no speedup (~19 vs ~21) and more VRAM, so the CPU offload is the right call. Loads to ~22GB/card.
Also FWIW common_fit_params ... not implemented for SPLIT_MODE_TENSOR, abort prints as scary but is just a warning — load continues.
Got Qwen3.8-Flash-Next running on 4x MI50 (gfx906, 32GB each, ROCm 6.3.3) with this fork — thanks for carrying the qwen4exp code. Sharing the exact working recipe and two things that cost me a load cycle each, in case it helps the next gfx906 person.
Build (in a rocm-gfx906 container, since the host toolchain is minimal):
Run (unsloth UD-Q4_K_XL, 4 shards, TP4):
Two gotchas:
--split-mode tensorrequires-fa on. With-fa offit aborts at load withSPLIT_MODE_TENSOR requires flash_attn to be enabled. Most gfx906 guidance says "start with -fa off," so this bites — worth a note in the README, or defaulting FA on when tensor-split is set.per_layer_token_embd(notple_ngram_embd), so-ot per_layer_token_embd=CPUis the right override to keep the ~51B n-gram table in host RAM (needs ~51GB+ free).Measured: single-stream decode ~20-21 tok/s at 256-token gen, coherent output. Keeping the PLE on-GPU instead of CPU (dropping the
-ot) gave no speedup (~19 vs ~21) and more VRAM, so the CPU offload is the right call. Loads to ~22GB/card.Also FWIW
common_fit_params ... not implemented for SPLIT_MODE_TENSOR, abortprints as scary but is just a warning — load continues.