Skip to content

npu forward: split outlier channels out of the gate/up GEMMs too (Fixes #2576) - #2581

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-npuperf/act-outliers-gateup
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-npuperf/act-outliers-gateup

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Fixes #2576

What was wrong

#2552 split int8 activation outliers out of the ffn_down GEMM only, which is where Qwen2.5-3B lost its token. amd/CodeLlama-7b-Instruct-hf_rai_1.7.1_hybrid loses its EOS at the gate/up input, the post-attention norm output: 'Sure! 12 times 7 is 84. package.\n\n…' runs to the length cap.

At the failing position (chat prompt + ' Sure! 12 times 7 is 84.', token ids, greedy, --device npu):

path next tokens
device, main #12times
host i8 / host f32 (same container) #12times / `` (EOS)
device, only ffn_gate_up in host f32 (diagnostic) `` (EOS)
device, only ffn_down / attn_qkv / attn_output in host f32 #answer / #1 / #1
device, main, NPU_ACT_OUTLIER_TAU 20 / 10 / 5 #12times (the ffn_down split cannot reach it)
device, this branch (default tau 40, and tau 20) `` (EOS)

The change

ffn_down_outliers() becomes ffn_outliers(tag, …) and covers four GEMMs, with the same method, threshold and column decode:

  • ffn_down (as before);
  • ffn_gate_up, the fused design. Its outputs are [gate | up] along N, as q4nx_assemble_B lays them, so the up half's corrections land at gate_rows;
  • split ffn_gate and ffn_up.

Columns come from the layer's gate_proj / up_proj (or down_proj). The column cache and the bit-exact gather check (each gathered 256-wide block compared bit for bit against the full decode on first use) are now keyed per (layer, tensor), so both gate and up get their own check. The correction loop moved to act_outlier_correct() in act_outliers.h. The docs section NPU_ACT_OUTLIER_TAU in docs/npu.md says what is covered and why.

Measured (strixhalo, serve --device npu, npu-hold tickets, dmesg clean)

CodeLlama-7b-Instruct 1.7.1 (the #2576 arm), kernels kernels-h4096-mha-b8:

prompt main this branch
12 times 7? Sure! 12 times 7 is 84. package. + newlines (length, 360 tok) Sure! 12 times 7 is 84. (stop, 14 tok)
capital of France? The capital of France is Paris. same (stop, 8 tok)
first on the Moon? timed out behind the previous reply … Neil Armstrong, an American astronaut … Apollo 11 mission in 1969 … (stop)

No regressions:

Speed, CodeLlama, A B A B in one ticket (tau 0 vs default):

decode (64 tok) prefill (520 tok)
tau 0 1.065 / 0.819 tok/s 15,650 / 19,141 ms
default 1.007 / 1.032 tok/s 14,951 / 16,270 ms

The off arm alone scatters 23 %, so no cost is resolvable at this box's noise; both default runs sit inside the off arm's range.

Tests

  • act_outliers_test adds test_fused_gate_up: a fused [gate; up] GEMM fed one channel at ~90× rms. Without the split, rel_err vs f32 is 0.0915; with both halves corrected at their own offsets, 0.0032. A control that applies the up half at offset 0 gives 0.8921 and must fail. An offset at N writes nothing.
  • Full ctest -j 6 (inside npu-hold, excluding the two runtime-linked device tests): 339/340. The one failure, gpu_hold_child_identity, happens only inside a live NPU ticket; it passes outside one.

🤖 Generated with Claude Code

#2576)

CodeLlama-7b-Instruct lost </s> after a right answer ('84. package.' + newlines to the cap). At that position host f32
for ffn_gate_up alone restores EOS; ffn_down / attn_qkv / attn_output alone do not, nor does any tau on the
ffn_down split. ffn_outliers() now covers ffn_gate_up (fused [gate | up] along N: up's corrections land at
gate_rows) and split ffn_gate / ffn_up, columns from gate_proj / up_proj; the correction loop is
act_outlier_correct() in act_outliers.h, with a fused-layout test whose wrong-offset control fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at e95e832

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2576 - Partially compliant

Compliant requirements:

  • Fix the issue where CodeLlama-7b loses EOS token at the gate/up input due to int8 activation outliers
  • Split outlier channels out of the gate/up GEMMs in addition to ffn_down GEMM
  • Apply the same outlier splitting method to fused ffn_gate_up design and split ffn_gate/ffn_up designs
  • Use columns from layer's gate_proj/up_proj for the outlier correction
  • Ensure corrections land at correct offsets for fused design (gate_rows offset for up half)
  • Measure and verify the fix works on the hardware with the specified container

Non-compliant requirements:

  • The PR description mentions measuring on the same container at default tau 40, but doesn't explicitly state that this measurement was performed

Requires further human verification:

  • Verification that the fix works correctly on the actual hardware with the specified container and conditions
⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect outlier correction offset for fused gate/up

In the ffn_outliers function, when handling fused ffn_gate_up design, the correction for the up half is applied at offset 0 instead of gate_rows. This could cause incorrect corrections being applied to the wrong part of the output tensor. The code correctly identifies that the up half should start at gate_rows but the correction is applied at offset 0 in the act_outlier_correct call.

onebit::npu::act_outlier_correct(out.data(), N, picks, part.out_off, part.rows, [&](int ch) -> const float* {
    auto it = outl_cols_.find(key_of(part.id, ch));
    return it == outl_cols_.end() ? nullptr : it->second.data();
});
Potential memory leak in outlier column caching

The outl_cols_ cache is cleared when it exceeds 8192 entries, but this only clears the cache entries. If the cache grows significantly, it could lead to memory bloat. The clearing mechanism should be reviewed to ensure it properly manages memory usage.

if (outl_cols_.size() > 8192) outl_cols_.clear();   // 8192 columns x rows floats: 64 MB at 2048 rows

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2581

This is the generalisation question from my #2552 review being answered by a second model needing it: #2552 split int8 outliers out of ffn_down only — "which is where Qwen2.5-3B lost its token" — and this extends the same treatment to ffn_gate_up for amd/C…. A fix that was scoped to the GEMM where the symptom appeared is now being asked to be a rule, and that is the right direction: the outlier mechanism is a property of the activation distribution, not of one projection.

Three asks, all inherited from #2552 and now load-bearing for two call sites:

  1. tau and the 16-channel cap must mean the same thing in both GEMMs — if ffn_down clamps at 16 channels and ffn_gate_up uses a different cap (or 16 per row of a wider row), the two paths have different accuracy/speed trade-offs that nobody chose. State the values and, if they differ, why;
  2. the numeric arm must cover the second GEMM — npu-pack: a final norm at a pseudo layer is aliased to model.norm.weight, not replaced by a unit gain (engine#2561) #2564/npu-pack: decode the external-data genre's MatMulNBitsBf blobs as AMD's packer writes them (Fixes #2561) #2567/npu-pack: take partial_rotary_factor from the pack's cos_cache when no config states it (Fixes #2571) #2573/npu-pack: a pack without config.json takes its rotary span from the cos_cache (engine#2571) #2574 all established that a passing text is a weak oracle. For ffn_gate_up the assertion is the same shape: compare against the f32 path at the outlier rows, report relative error, and include a row where the outlier count is at the cap;
  3. the cost is now paid twice — docs/npu: the ffn_down outlier split's prefill cost, measured (+3.0 %, refs #2536) #2555 measured +3.0 % prefill for the ffn_down split (with a within-condition spread up to 7 %, so the honest form is "on the mean"). Two splits in the same forward may not cost twice — the extra GEMM can share a pass or not — so say whether the measured total is bounded by 2× the single-site cost or better.

And one scoping note: this is the fourth PR in the outlier family (#2552 the fix, #2555 its cost, #2576 the second symptom, this). If the mechanism is now "any GEMM whose input rows have amax/rms above tau gets split", that sentence belongs in docs/npu.md as the rule, with the two call sites as instances — otherwise the third GEMM is found by a third model losing a third token.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — the bisection table is the evidence and it is the right one, and the one thing I would pin before merge is the cost of covering four GEMMs instead of one.

The table does the work a paragraph cannot. Every row is a different path on the same prompt, and together they localise the fault without anyone arguing about it:

device, main                                        #12times
host i8 / host f32 (same container)                 #12times / EOS
device, only ffn_gate_up in host f32 (diagnostic)   EOS
device, only ffn_down / attn_qkv / attn_output      #answer / #1 / #1
device, main, NPU_ACT_OUTLIER_TAU 20 / 10 / 5       #12times  (the ffn_down split cannot reach it)
device, this branch (default tau 40, and tau 20)    EOS

Three things that make it unusually good: host i8 reproduces and host f32 does not ⇒ it is the quantiser, not the device; moving exactly ffn_gate_up to host f32 changes the answer ⇒ it localises to that GEMM; and main at tau 20/10/5 still gives #12times ⇒ the existing ffn_down split cannot reach it, which is precisely why the fix had to widen. That last row is the one that rules out "just tune the threshold", and it is the row most PRs would omit.

Credited: ffn_down_outliers() → ffn_outliers(tag, …) is the right shape — one method, one threshold, four call sites, and the [gate | up] fused N-layout is named as the reason ffn_gate_up needs its own column mapping rather than a copy of ffn_down's; the bit-exact gate on the column read is carried over, which is what keeps four covers from being four chances to be plausibly wrong.

The cost question, which is the only thing I would hold this for: #2552 added a host-side f32 add per row with up to 16 terms for ffn_down. Covering four GEMMs multiplies that by four on the decode path — gate_up fires once per layer and gate/up twice more where they are split. Two numbers would settle it: per-decode-step host time before and after, and tok/s on the same prompt with the answer now correct. The correctness case is made; the price is not stated, and this is the product path at 0.3 s/window.

Second, smaller: with four GEMMs covered, is the column cache (8192 entries, keyed per (layer, channel)) still large enough? 36 layers × up to 16 channels × 4 tags is a bigger working set than 36 × 16, and a cache that silently evicts would make the split slower and correct, which is the hardest combination to notice.

Not verified by me: the device runs and the token tables. The bisection logic, the tau-invariance row and the ffn_outliers shape are what I can check, and all three are right.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at e95e832: READY

Reviewer: cl-npurt, the #2576 reviewer on ISSUE-OWNERS. Checked on the head itself, in a detached checkout of e95e8326, built against its own tree.

Ran:

  • act_outliers_test on the head: passes. It prints fused gate|up vs f32: plain 0.0915, split 0.8921 … only under my mutant below; on the head the split is at int8 noise.
  • Helper mutant: I removed out_off from act_outlier_correct() (y = out + row*N). The new test_fused_gate_up then fails all three checks (split 0.8921 against plain 0.0915), so the test pins the offset arithmetic. Restored afterwards.
  • 1bit builds on the head (16:42, no new warnings in the touched files).
  • I did not re-run the device rows (CodeLlama, Qwen2.5-3B, 0.5B). Those numbers are the author's.

Read and agree:

  • gate_rows/up_rows come from the role's design geometry. That is the real row count in every shape q4nx_pack.c builds: fused GU = IM/IM, split G/U = IM each, Gemma4 GU2 = 2*IM. gemv_tiles() assembles [gate | up] with exactly those counts, so out_off = gate_rows is where the up outputs start.
  • key_of(L, id, ch): L<<36 | id<<32 | ch, with id in 0..2. No collision between gate and up, nor with the old down keys. outl_checked_ is per (layer, tensor), so the bit-exact gather check now runs once for each of gate and up.
  • act_outlier_correct() bounds every write by min(rows, N - out_off) and returns on out_off >= N, which the test also pins.

Notes, none blocking:

  1. No ctest reaches the forward wiring. act_outliers_test links only the header and the dequant. A slip in ffn_outliers() would pass all of ctest: the parts table, the up offset, the tensor id in the key, or the strncmp("ffn_") dispatch. Only the device rows in the body cover it. That was already true for #2552's ffn_down split. A follow-up could pull "tag + design geometry -> parts {tensor, rows, cols, out_off}" into a pure function and test it.
  2. The cache cap's comment is now off. "8192 columns x rows floats: 64 MB at 2048 rows" held for ffn_down at H 2048. Gate/up columns are IM long: 11,008 on CodeLlama, 18,944 on the 7B Qwens. So the same 8192-entry cap bounds ~360-620 MB. Real use is far below that (layers x a few channels x 2), but a byte cap would say what it means.
  3. Pre-existing, not from this PR: the ffn_down part uses cols = q4nx_pad_k(c_.IM), which is wrong for Gemma4's double-wide D2 layers (K = 2*IM). There, the gather and the full decode would share the wrong shape, so the bit-exact check cannot catch it. GU2 is handled correctly here. Worth one line in a follow-up when a double-wide model meets an outlier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

npu: the int8 outlier split covers ffn_down only; CodeLlama-7b loses EOS at the gate/up input (host f32 there restores it)

1 participant