fix(qwen3_5_moe): accept a per-channel fp8 weight_scale in the dense loader - #415
Open
salekseev wants to merge 1 commit into
Open
fix(qwen3_5_moe): accept a per-channel fp8 weight_scale in the dense loader#415salekseev wants to merge 1 commit into
salekseev wants to merge 1 commit into
Conversation
…loader
_per_row_scale unconditionally did scale.reshape(1), which is only valid for a
per-tensor scale. compressed-tensors also emits strategy: "channel", one scalar
per output row stored as [rows, 1], and on such a checkpoint the reshape raises
RuntimeError: shape '[1]' is invalid for input of size 248320
Reshape to [-1] and branch on the element count instead: 1 broadcasts as before,
rows passes through, anything else raises rather than being broadcast -- a
mis-shaped scale that loaded would apply one row's factor to every output row and
serve fluent, wrong tokens.
Fp8PerTensorLinear.weight_scale is already declared per-output-row, so nothing
downstream changes and no kernel work follows.
This was referenced Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hit this while trying to load a compressed-tensors FP8 checkpoint that uses per-channel weight scales.
_per_row_scaleassumes the scale is always a single scalar:That's right for
strategy: "tensor", but llm-compressor also emitsstrategy: "channel"— one scalar per output row, stored as[rows, 1]. On that shape the reshape blows up:Since
Fp8PerTensorLinear.weight_scaleis already declared per output row, there's nothing to change downstream — a genuine per-tensor weight just stores the same scalar in every row. So this reshapes to[-1]and branches on the element count: 1 broadcasts exactly as before,rowspasses straight through.The part I'd push back on if I were reviewing: the third branch raises instead of doing something lenient. That's deliberate. If a mis-shaped scale broadcast row 0's factor across all 248320 output rows, the model would load happily and generate fluent nonsense. I'd much rather it fail at load with a message naming both counts.
Reachability, since it affects how you want to review this
You can't trigger this on
maintoday. The only checkpoints that reach_iter_weights_attn_fp8are ModelOpt ones, and those use a per-tensor scale. So onmainthis is hardening plus a prerequisite — it becomes load-bearing once a per-channel checkpoint can actually route there, which needs the compressed-tensors detection from #390 plus a one-token widening to accept"channel".I mention it so you can weigh it as "small safe change that unblocks a checkpoint class" rather than "fixes a bug users are hitting", because the latter would be overselling it.
Tests
Adds
tests/models/test_qwen3_5_moe_weight.py, modelled ontests/models/test_glm5_next_config.py. Covers both on-disk granularities, fp32 promotion from every storage dtype, and the mismatch raising. The per-channel case asserts row order with distinct values rather than a sum or a set — permuting the scales would keep every aggregate identical while silently scaling each row by the wrong factor.There were no
test_qwen3_5_moe_*files before this, so nothing is replaced.Testing done
Verified against
unsloth/Qwen3.6-35B-A3B-NVFP4-Fastandnvidia/Qwen3.6-35B-A3B-NVFP4on an RTX 4080 SUPER (16 GiB, sm_89, TP=1). Full background and measurements in #252.