Repository navigation
npu forward: split outlier channels out of the gate/up GEMMs too (Fixes #2576) - #2581
Conversation
#2576) CodeLlama-7b-Instruct lost </s> after a right answer ('84. package.' + newlines to the cap). At that position host f32 for ffn_gate_up alone restores EOS; ffn_down / attn_qkv / attn_output alone do not, nor does any tau on the ffn_down split. ffn_outliers() now covers ffn_gate_up (fused [gate | up] along N: up's corrections land at gate_rows) and split ffn_gate / ffn_up, columns from gate_proj / up_proj; the correction loop is act_outlier_correct() in act_outliers.h, with a fused-layout test whose wrong-offset control fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at e95e832 |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY This is the generalisation question from my #2552 review being answered by a second model needing it: #2552 split int8 outliers out of Three asks, all inherited from #2552 and now load-bearing for two call sites:
And one scoping note: this is the fourth PR in the outlier family (#2552 the fix, #2555 its cost, #2576 the second symptom, this). If the mechanism is now "any GEMM whose input rows have |
|
READY — the bisection table is the evidence and it is the right one, and the one thing I would pin before merge is the cost of covering four GEMMs instead of one. The table does the work a paragraph cannot. Every row is a different path on the same prompt, and together they localise the fault without anyone arguing about it: Three things that make it unusually good: Credited: The cost question, which is the only thing I would hold this for: #2552 added a host-side f32 add per row with up to 16 terms for Second, smaller: with four GEMMs covered, is the column cache (8192 entries, keyed per (layer, channel)) still large enough? 36 layers × up to 16 channels × 4 tags is a bigger working set than 36 × 16, and a cache that silently evicts would make the split slower and correct, which is the hardest combination to notice. Not verified by me: the device runs and the token tables. The bisection logic, the tau-invariance row and the |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at e95e832: READY
Reviewer: cl-npurt, the #2576 reviewer on ISSUE-OWNERS. Checked on the head itself, in a detached checkout of e95e8326, built against its own tree.
Ran:
act_outliers_teston the head: passes. It printsfused gate|up vs f32: plain 0.0915, split 0.8921 …only under my mutant below; on the head the split is at int8 noise.- Helper mutant: I removed
out_offfromact_outlier_correct()(y = out + row*N). The newtest_fused_gate_upthen fails all three checks (split 0.8921 against plain 0.0915), so the test pins the offset arithmetic. Restored afterwards. 1bitbuilds on the head (16:42, no new warnings in the touched files).- I did not re-run the device rows (CodeLlama, Qwen2.5-3B, 0.5B). Those numbers are the author's.
Read and agree:
gate_rows/up_rowscome from the role's design geometry. That is the real row count in every shapeq4nx_pack.cbuilds: fused GU = IM/IM, split G/U = IM each, Gemma4 GU2 = 2*IM.gemv_tiles()assembles[gate | up]with exactly those counts, soout_off = gate_rowsis where the up outputs start.key_of(L, id, ch): L<<36 | id<<32 | ch, with id in 0..2. No collision between gate and up, nor with the old down keys.outl_checked_is per (layer, tensor), so the bit-exact gather check now runs once for each of gate and up.act_outlier_correct()bounds every write bymin(rows, N - out_off)and returns onout_off >= N, which the test also pins.
Notes, none blocking:
- No ctest reaches the forward wiring.
act_outliers_testlinks only the header and the dequant. A slip inffn_outliers()would pass all of ctest: the parts table, the up offset, the tensor id in the key, or thestrncmp("ffn_")dispatch. Only the device rows in the body cover it. That was already true for #2552's ffn_down split. A follow-up could pull "tag + design geometry -> parts {tensor, rows, cols, out_off}" into a pure function and test it. - The cache cap's comment is now off. "8192 columns x rows floats: 64 MB at 2048 rows" held for ffn_down at H 2048. Gate/up columns are IM long: 11,008 on CodeLlama, 18,944 on the 7B Qwens. So the same 8192-entry cap bounds ~360-620 MB. Real use is far below that (layers x a few channels x 2), but a byte cap would say what it means.
- Pre-existing, not from this PR: the
ffn_downpart usescols = q4nx_pad_k(c_.IM), which is wrong for Gemma4's double-wide D2 layers (K = 2*IM). There, the gather and the full decode would share the wrong shape, so the bit-exact check cannot catch it. GU2 is handled correctly here. Worth one line in a follow-up when a double-wide model meets an outlier.
Fixes #2576
What was wrong
#2552 split int8 activation outliers out of the
ffn_downGEMM only, which is where Qwen2.5-3B lost its token.amd/CodeLlama-7b-Instruct-hf_rai_1.7.1_hybridloses its EOS at the gate/up input, the post-attention norm output:'Sure! 12 times 7 is 84. package.\n\n…'runs to the length cap.At the failing position (chat prompt +
' Sure! 12 times 7 is 84.', token ids, greedy,--device npu):#12times#12times/ `` (EOS)ffn_gate_upin host f32 (diagnostic)ffn_down/attn_qkv/attn_outputin host f32#answer/#1/#1NPU_ACT_OUTLIER_TAU20 / 10 / 5#12times(the ffn_down split cannot reach it)The change
ffn_down_outliers()becomesffn_outliers(tag, …)and covers four GEMMs, with the same method, threshold and column decode:ffn_down(as before);ffn_gate_up, the fused design. Its outputs are[gate | up]along N, asq4nx_assemble_Blays them, so the up half's corrections land atgate_rows;ffn_gateandffn_up.Columns come from the layer's
gate_proj/up_proj(ordown_proj). The column cache and the bit-exact gather check (each gathered 256-wide block compared bit for bit against the full decode on first use) are now keyed per (layer, tensor), so both gate and up get their own check. The correction loop moved toact_outlier_correct()inact_outliers.h. The docs sectionNPU_ACT_OUTLIER_TAUindocs/npu.mdsays what is covered and why.Measured (strixhalo,
serve --device npu, npu-hold tickets, dmesg clean)CodeLlama-7b-Instruct 1.7.1 (the #2576 arm), kernels
kernels-h4096-mha-b8:Sure! 12 times 7 is 84. package.+ newlines (length, 360 tok)Sure! 12 times 7 is 84.(stop, 14 tok)The capital of France is Paris.… Neil Armstrong, an American astronaut … Apollo 11 mission in 1969 …(stop)No regressions:
…July 20, 1prefix still gives96, chat answersNeil Armstrong, on July 20, 1969, as part of the Apollo 11 mission., and teacher-forced top-1 is 58/64 (60/64 with ffn_down only; ±2 is the documented near-tie range).Speed, CodeLlama, A B A B in one ticket (tau 0 vs default):
The off arm alone scatters 23 %, so no cost is resolvable at this box's noise; both default runs sit inside the off arm's range.
Tests
act_outliers_testaddstest_fused_gate_up: a fused[gate; up]GEMM fed one channel at ~90× rms. Without the split, rel_err vs f32 is 0.0915; with both halves corrected at their own offsets, 0.0032. A control that applies the up half at offset 0 gives 0.8921 and must fail. An offset at N writes nothing.ctest -j 6(inside npu-hold, excluding the two runtime-linked device tests): 339/340. The one failure,gpu_hold_child_identity, happens only inside a live NPU ticket; it passes outside one.🤖 Generated with Claude Code