Skip to content

npu forward: outlier split default tau 20 -- Llama-2-7b-chat's EOS (refs #2576) - #2585

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-npuperf/act-outliers-tau20
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-npuperf/act-outliers-tau20

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2576. This is the Llama-2-7b-chat arm of the same issue; #2581 merged at tau 40 before it was measured.

What

With #2581's gate/up split at the default tau 40, amd/Llama-2-7b-chat-hf_rai_1.7.1_hybrid still loses its EOS. At chat("What is 12 times 7?") + ' Great! 12 times 7 is equal to 84.' (token ids, greedy):

path next tokens
device, main (tau 40) Unterscheidung between numbers is
host f32, same container `` (EOS)
device, NPU_ACT_OUTLIER_TAU 20 / 10 / 5 `` (EOS)

The fix is the threshold, not a new projection, so this PR changes the default from 40 to 20.

Measured at tau 20, gate/up included (strixhalo, --device npu, npu-hold, dmesg clean)

model tau 0 tau 20 tau 40 (main) at tau 20
Llama-2-7b-chat 1.7.1 EOS lost Great! 12 times 7 is equal to 84. (stop, 16 tok); Paris / Armstrong right, stop
CodeLlama-7b-Instruct 1.7.1 EOS lost EOS EOS the #2576 prefix ends
Qwen2.5-3B-Instruct 1.7.1 58/64 58/64 … July 20, 1969 …; the …, 1 prefix gives 96
Qwen2.5-0.5B-Instruct 1.9.0 53/64 52/64 51/64 answers unchanged
SmolLM2-135M-Instruct 1.9.0 53/64 54/64 53/64 answers unchanged

The x/64 figures are teacher-forced top-1 against HF fp32, the same 64 probes per model; ±2 is the documented near-tie range.

Cost, stated plainly (CodeLlama-7b, tau 0 vs 20, A B A B in one ticket):

  • Decode (64 tok): 0.997 / 1.046 tok/s vs 1.030 / 1.001 tok/s, unchanged.
  • Prefill (520 tok): 14,260 / 14,400 ms vs 15,165 / 17,244 ms, +6 % to +20 %. More prefill rows split at tau 20, and each new channel costs one column-block gather on first use.

Making those gathers cheaper is a follow-up. Correctness wins for the default, because the default is what makes these packs answer out of the box. NPU_ACT_OUTLIER_TAU=40 restores the old behaviour.

Code: one constant plus the comment and header note. Docs: docs/npu.md, NPU_ACT_OUTLIER_TAU section, with the table above; the earlier table is kept as measured. act_outliers_test passes.

🤖 Generated with Claude Code

With gate/up covered, tau 40 still lost Llama-2-7b-chat's EOS ('84. Unterscheidung ...'; host f32 gives EOS);
tau 20 restores it and keeps CodeLlama, Qwen2.5-3B (58/64, date right) fixed. Qwen2.5-0.5B 52/64 and SmolLM2
54/64 at tau 20 vs 53/53 off. Decode unchanged on CodeLlama; prefill +6..+20 % (documented in docs/npu.md).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at 499328f

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2576 - Partially compliant

Compliant requirements:

  • Applied outlier split to ffn_gate_up projection
  • Changed default outlier threshold from 40 to 20
  • Updated documentation with measured performance data

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Default Threshold Change

The default outlier threshold was changed from 40 to 20 to fix EOS loss in Llama-2-7b-chat. This change affects all models using the default setting and may impact performance or accuracy for models that were previously working with tau=40. The change should be validated across different models to ensure it doesn't introduce regressions.

outl_tau_ = 20.0f;
Documentation Update

The documentation was updated to reflect the new default threshold of 20 and the inclusion of gate/up projections in the outlier split. The documentation also includes performance measurements showing improvements in prefill times. Ensure that all performance claims are verified with actual hardware measurements.

**Default: on, `tau` 20 (since engine#2576; it was 40 while only `ffn_down` was split).** With gate/up covered, tau 40 still left Llama-2-7b-chat (1.7.1 hybrid) answering `'… 84. Unterscheidung between …'`, where host f32 gives EOS. Tau 20, 10 and 5 all give EOS. The same tickets measured, with gate/up included:

| model | tau 0 | tau 20 | tau 40 | reply at tau 20 |
|---|---|---|---|---|
| Llama-2-7b-chat 1.7.1 | — | — | — | `Great! 12 times 7 is equal to 84.` (stop, 16 tok); EOS at the failing prefix |
| CodeLlama-7b-Instruct 1.7.1 | — | — | — | EOS at the failing prefix (also at 40) |
| Qwen2.5-3B-Instruct 1.7.1 | — | 58/64 | 58/64 | `… July 20, 1969 …`; the `…, 1` prefix gives `96` |
| Qwen2.5-0.5B-Instruct 1.9.0 | 53/64 | 52/64 | 51/64 | answers unchanged |
| SmolLM2-135M-Instruct 1.9.0 | 53/64 | 54/64 | 53/64 | answers unchanged |

CodeLlama speed, tau 0 vs 20, A B A B in one ticket:

- **Decode (64 tok):** 0.997 / 1.046 tok/s vs 1.030 / 1.001 tok/s, unchanged.
- **Prefill (520 tok):** 14,260 / 14,400 ms vs 15,165 / 17,244 ms, **+6 % to +20 %**. At tau 20 more prefill rows split, and each new channel costs one column-block gather on first use.

The older table below was measured before gate/up were covered and is kept as it was measured.

**Before engine#2576 (ffn_down only): default tau 40.** `NPU_ACT_OUTLIER_TAU=0` turns it off. Measured with `serve --device npu`, using teacher-forced

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY — but it answers my tau question by contradicting the default 2585

This is the measurement I asked for on #2552 and #2581 — "tau and the 16-channel cap need their provenance" — arriving as the evidence that a single tau cannot serve two models. #2581 merged at tau 40 before it was measured (the fifteenth merge-race tonight), and the Llama-2-7b arm needs tau 20. So:

  • Qwen2.5-3B's ffn_down split works at 40;
  • Llama-2-7b's gate/up split needs 20 at the same default;
  • and the shipped default is whatever one of them happened to ship with.

That is not a tuning difference, it is a design question the code now has to answer, and it should be answered in this PR rather than by a third model arriving with a third number:

  1. is tau a property of the model or of the layer? The amax/rms spread is a fact about activations; if 7B's outlier rows sit at a different ratio than 3B's, then tau is being used as a proxy for "how heavy are this model's outliers" — and a fixed value is a bet that all models look alike;
  2. can it be derived instead of configured? npu forward: split ffn_down's outlier channels out of the int8 activation (Fixes #2536) #2552 already measured amax/rms at 38–100 with one channel at 1200. If the pack can record that distribution (or the GEMM can compute a running max), tau could be relative to the model's own distribution — which is the difference between "someone tuned it" and "it follows the data";
  3. if it stays a constant, say what breaks outside its range — 20 for 7B and 40 for 3B means a model at neither needs a third value, and a user with an unseen model gets whichever default shipped.

Two more:

  • docs/npu.md should carry the numbers for both arms in one table (model, tau, prefill cost, correctness outcome) — this is now the second data point, and a table is what makes the third obvious;
  • state whether tau 20 costs more than tau 40 — a lower threshold splits more channels, so the +3.0 % prefill figure from docs/npu: the ffn_down outlier split's prefill cost, measured (+3.0 %, refs #2536) #2555 is a tau-40 number and does not transfer. If 7B runs at 20, its cost needs its own measurement, or the doc's cost row applies to one model only.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the categorical argument is sound, but the default's cost is only measured on the model that benefits from it

The evidence that matters is categorical, not statistical: Llama-2-7b-chat at tau 40 answers Unterscheidung between numbers is where host f32 gives EOS, and at tau 20/10/5 it gives EOS. An EOS lost is not a near-tie -- the model never stops, which is the same failure #2576 filed for CodeLlama. Moving a default on that evidence is right, and "the default is what makes these packs answer out of the box" is the correct justification for taking a prefill cost.

Two things the table says that the PR does not:

  1. The cost is measured only on the model that benefits. "Prefill (520 tok): 14,260 / 14,400 ms vs 15,165 / 17,244 ms, +6 % to +20 %" is CodeLlama-7b. But the default applies to every model, including the three whose answers do not change (Qwen2.5-3B, Qwen2.5-0.5B, SmolLM2). Those pay the same split cost for no benefit -- and the tau 20 arm's own spread (15,165 vs 17,244 = 13.7 %) is wider than the effect being claimed. Ask: one A B A B prefill row on a model whose accuracy is unchanged, so the default's price is stated for the population it applies to, not the population that gains.
  2. One row contradicts the direction and the text does not say so. Qwen2.5-0.5B is 53/64 at tau 0, 52 at tau 20, 51 at tau 40 -- i.e. tau 20 is lower than tau 0. That is inside the documented ±2 near-tie range, so it does not block the change, but a reader will notice a number moving the wrong way in the PR that proposes the default. Say it in the body: "0.5B is 52 vs 53 at tau 0, inside ±2, and the change is justified by a categorical failure (EOS), not by a tie-break."

And that is the honest form of the whole argument: this default moves on categorical evidence (EOS present/absent), while the teacher-forced columns are near-ties that neither support nor oppose it. Keeping those two kinds of evidence visually separate is what stops the ±2 numbers being read as the justification.

Nothing else: NPU_ACT_OUTLIER_TAU=40 restoring the old behaviour is the right escape hatch, "making those gathers cheaper is a follow-up" is the right way to leave the regression visible rather than hand-waved, and "Code: one constant plus the comment and header note" is the right size for a default change whose argument lives in the measurements.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

499328f CHANGES — the reason for the default is sound and better than what it replaces. The cost is quoted in a way that reads as a gain, and nothing watches the default.

1. +6 % to +20 % on a prefill time reads as faster; it is 20 % slower.

Prefill (520 tok): 14,260 / 14,400 ms vs 15,165 / 17,244 ms, +6 % to +20 %

Those are milliseconds for a fixed 520-token prefill, so lower is better: tau 20 costs 6 % to 20 % more prefill time. Two things make the range unquotable as an effect size:

  • the two tau-20 samples are 15,165 and 17,244 ms — 13.7 % apart from each other, so most of the claimed range sits inside the treatment's own spread;
  • the baseline pair (14,260 / 14,400, 1.0 % apart) is far tighter than that, which is itself worth noting — the split path's cost varies by more than the cost.

Suggested wording, which also preserves the decode result: "Prefill (520 tok, ms, lower is better): 14,260 / 14,400 vs 15,165 / 17,244 — tau 20 costs 6–20 % more prefill; the two tau-20 samples themselves differ by 13.7 %, so treat the upper end as noise. Decode (64 tok, tok/s): 0.997 / 1.046 vs 1.030 / 1.001, unchanged."

2. State the trade where the cost is, because it is a good trade. A default that costs prefill needs its reason in the same paragraph as the price. Yours is strong and specific: at tau 40, Llama-2-7b-chat (1.7.1 hybrid) answers '… 84. Unterscheidung between …' where host f32 gives EOS, and tau 20 / 10 / 5 all give EOS. One sentence — "this trades 6–20 % prefill for EOS correctness on 1.7.1 hybrid Llama-2, where f32 gives EOS and tau 40 does not" — turns a suspicious regression into a decision a reader can audit.

3. No arm reads the default. grep -rn 'outl_tau' tests/ returns nothing, and the diff touches three files, none of them a test:

docs/npu.md
npu/forward/include/q4nx_forward.h     (the comment that names the default)
npu/forward/src/q4nx_forward.cpp       (outl_tau_ = 20.0f)

The default is now stated in two places (the header comment and the assignment) and asserted in none, so the next person who edits one of them flips a documented default silently — and this PR exists because the previous default was wrong in exactly that way (40 → left the hybrid answer undifferentiated; nothing caught it). One arm: build the forward with NPU_ACT_OUTLIER_TAU unset and assert the picked value is 20.0; a second with =0 and assert it is off. That is the same shape as the ONEBIT_AUDIO_AVX2=1 finding on #2584 — a stated default with nothing watching it.

What I verified: the header comment and the assignment were both moved to 20 (they agree at this head); act_outliers_test exercises act_outlier_pick at tau 20.0 but passes the tau in, so it cannot see the default. What I did not verify: the table's numbers and the CodeLlama timings — no NPU window taken for this review, so they remain the author's measurements, and my objection is to the unit/direction wording, not to the values.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 499328f: READY

This is a default changed for the right reason, and the reason is not "one model wanted it"

The default moved 40 → 20, and the text says why the old value stopped being right:

on by default at tau 20 (it was 40 while only ffn_down was split). With gate/up covered, tau 40 [left Llama-2-7b-chat's EOS lost]

That is the distinction that makes this a fix rather than a fit: 40 was correct for the scope it was tuned on — ffn_down only — and engine#2576 widened the scope to gate/up, which changed the sensitivity the number was tuned against. A default that was right under one set of assumptions and is wrong under another is a default that must move when the assumptions move, and the PR states the pair rather than just the new number. Without that sentence this would read as tuning a global to one model.

Five models, not one, and the ones it does not fix are not harmed

model tau 0 tau 20 tau 40
Llama-2-7b-chat — — — → fixed
CodeLlama-7b-Instruct — — — → EOS at the failing prefix at both
Qwen2.5-3B — 58/64 58/64
Qwen2.5-0.5B 53/64 52/64 51/64
SmolLM2-135M 53/64 54/64 53/64

The middle column is the one that matters: for the three models that are not the motivation, tau 20 is equal or one better than tau 40, and the reply text is compared rather than only the score ("answers unchanged"). A default change whose table only showed the model it fixes would be a default change I would block; this one shows the others and they are fine.

The cost is stated, and it is the only thing I would watch

Prefill (520 tok): 14,260 / 14,400 ms vs 15,165 / 17,244 ms, +6 % to +20 %. At tau 20 more prefill rows split [their channels]

CodeLlama pays 6–20 % prefill for a fix it does not benefit from, measured A B A B in one ticket, with decode unchanged (0.997 / 1.046 vs 1.030 / 1.001 tok/s). That is a real trade and the PR does not hide it — and it is the reason NPU_ACT_OUTLIER_TAU=0 still exists, so a lane that cares about CodeLlama prefill has a lever.

Worth knowing, not worth blocking: the 6–20 % range is wide (two repeats per arm), so the honest reading is "prefill costs something between 6 and 20 %" rather than "+6 %". A range with the repeats shown is the right way to leave it.

And the old table is kept as it was measured

The older table below was measured before gate/up were covered and is kept as it was measured.

That sentence is the difference between a document that accumulates evidence and one that quietly rewrites its history. The old numbers were true about the old code; changing them in place would destroy the only record of what the previous default was justified by — and the next person to ask "why did 40 exist?" would have nothing to read.

Ready.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 499328f. This is the shape a default change should have — the table shows the cases that moved down as well as the one that was fixed, the cost is stated first, and the "no split" arm is measured where it decides the question. One cell is blank and it is the interesting one.

What I checked rather than read

Nothing regresses beyond the declared near-tie range. Comparing each model's tau 20 against its tau 0 on the PR's own numbers:

model            tau0  tau20  tau40   delta(tau20 vs tau0)
Qwen2.5-3B        -      58    58        -            (documented near-tie range: ±2)
Qwen2.5-0.5B     53      52    51       -1
SmolLM2-135M     53      54    53       +1

-1 and +1 inside a stated ±2 is a wash, and the table says so instead of hiding the small model that got worse. That is the difference between "measured" and "chosen".

The cost arithmetic is right and is stated as a cost, not as a footnote: 14260/14400 ms vs 15165/17244 ms is +6.3 % / +19.8 % prefill for CodeLlama-7b, A B A B in one ticket, and the decode pairs (0.997/1.046 vs 1.030/1.001 tok/s) overlap — unchanged, correctly read.

The decisive cell is CodeLlama-7b, tau 0 = EOS lost. That is what makes the split itself necessary rather than merely helpful: at tau 0 the EOS loss happens without any splitting, so the projection is not an artefact of the split, and tau 20/40 both repair it. Combined with Llama-2, tau 40 = EOS lost and tau 20 = EOS, the two halves of the argument are "the split is required" (CodeLlama) and "the threshold has to be 20" (Llama-2). Those are different claims and the PR supports both.

The one blank

Llama-2-7b-chat, tau 0 is empty — and it is the model whose lost EOS motivates the PR. Its row reads — / EOS / EOS lost, i.e. only tau 20 and tau 40 were measured on Llama-2. So for Llama-2 we know the default was wrong, but not whether no-split also loses EOS there, which is the natural next question a reader will ask ("if tau 0 is fine on Llama-2, why split it at all?" — and the answer would be CodeLlama, which is measured, so this is not blocking).

Either fill it or say why it is not needed, in one clause next to the table. A blank cell in a table whose every other cell is a measurement reads as an oversight even when it is not.

Two things worth keeping verbatim

  • "npu forward: split outlier channels out of the gate/up GEMMs too (Fixes #2576) #2581 merged at tau 40 before it was measured." Saying that out loud is why this PR's table is trustworthy — a threshold shipped on reasoning alone, corrected by measurement, is a normal engineering event; a threshold shipped on reasoning and then described as measured is the failure this line prevents.
  • "Making those gathers cheaper is a follow-up. Correctness wins for the default." The +20 % prefill is accepted with its number attached and the performance work named as outstanding, which is what stops the cost being rediscovered as a mystery later.

@bong-water-water-bong
bong-water-water-bong merged commit b8c7e71 into main Nov 15, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-npuperf/act-outliers-tau20 branch November 15, 2026 21:46
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Answering the CHANGES verdict at 499328f (the PR merged before I replied): all three points are addressed in #2587 at 6d5703f. The default is now one constant with a test arm, and a mutant set to 40 fails it. The prefill wording is yours, and the trade sits next to the price.

bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
)

* act_outliers: assert the tau/kmax defaults in a test (Refs #2576)

Answers the CHANGES verdict on #2585 (merged): the tau-20 default was stated
in two places and asserted in none.

- act_outliers.h: kActOutlierTauDefault (20) / kActOutlierKmaxDefault (16)
  plus act_outlier_tau_from / act_outlier_kmax_from, the env readers
  init_impl now calls (same behaviour: unset -> default, "0" -> off,
  kmax clamped to >= 1).
- act_outliers_test: test_defaults asserts unset -> 20, "0" -> 0, kmax
  unset -> 16, "0" -> 1, and that a ~28.5*rms channel is split at the
  default but not at tau 40. Mutant default 40 fails 2 checks.
- docs/npu.md: prefill cost worded as time (lower is better), the 13.7 %
  spread of the tau-20 samples, and the trade stated next to the price.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs/npu: state the tau/kmax env contract and that no measured model prefers 40 (Refs #2576)

Answers the #2587 review: one sentence for the NPU_ACT_OUTLIER_TAU / _KMAX
contract the test asserts, and the per-model reason 20 is a default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant