Qwen4 next flash multi gpu & buffer size issues solved - #28623
Draft
gopinath87607 wants to merge 2 commits into
Draft
Qwen4 next flash multi gpu & buffer size issues solved#28623gopinath87607 wants to merge 2 commits into
gopinath87607 wants to merge 2 commits into
Conversation
score and mask each query chunk against the compacted cache instead of materializing [n_kv, n_tokens] tensors, and hoist the per-chunk mask buffer so the scratch no longer grows with the chunk count. enable the CUDA top_k radix select for any row width and route flash attention to the mask-compacting sparse path for the QSA head shapes. rebuild the causal mask on device from cell positions, with future and foreign cells saturating to -inf. Assisted-by: Claude Code
Checkpoint of the working tree on qwen4exp-qsa-chunked. Root cause of the QSA output corruption: prefill ubatch self-attends up to `ub` in-batch cells, so corruption occurs iff ub > width (top_k + r - 1). Keep LLAMA_QSA_TOPK >= ub to avoid it. Fix is env-gated and behavior-identical to stock when unset. Also carries the chunked-QSA prefill with bounded scratch from HEAD plus remaining experimental/bisect toggles (QSA_OFF, NOIDXW, fresh-mask, fattn/cu/test-backend-ops hacks) marked "revert me". Cleanup of those is a follow-up before any upstream PR. Co-Authored-By: Claude Code <noreply@anthropic.com>
gopinath87607
requested review from
a team,
CISC and
ggerganov
as code owners
September 9, 2026 02:40
github-actions
Bot
added
model
|
Hi @gopinath87607, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
github-actions
Bot
removed
the
draft
gopinath87607
changed the title
gopinath87607
changed the title
gopinath87607
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
this pr is not complete one just sharing what i find you can close it it any time
Additional information
here how you can run
export LLAMA_QSA_TOPK=9000
echo 0 | sudo tee /proc/sys/kernel/numa_balancing
CUDA_VISIBLE_DEVICES=2,3,0,1
numactl --interleave=all
~/llama.cpp/build/bin/llama-server
--model /mnt/nvme/Qwen3.8-Flash-Next-Q8_0-00001-of-00006.gguf
--chat-template-kwargs '{"reasoning_effort":"xhigh"}'
--tensor-split 3,4,1,1
--mlock
--lazy-mode off
-cmoe
--numa distribute
-c 260000
--batch-size 9000
--ubatch-size 9000
--top-p 0.95
--top-k 20
--min-p 0.0
--presence-penalty 0.0
--repeat-penalty 1.0
--parallel 1
--threads 42
--threads-batch 42
-ngl 100
--host 127.0.0.1
--port 8082
--jinja
Requirements