Repository navigation
npu forward: outlier split default tau 20 -- Llama-2-7b-chat's EOS (refs #2576) - #2585
Conversation
With gate/up covered, tau 40 still lost Llama-2-7b-chat's EOS ('84. Unterscheidung ...'; host f32 gives EOS);
tau 20 restores it and keeps CodeLlama, Qwen2.5-3B (58/64, date right) fixed. Qwen2.5-0.5B 52/64 and SmolLM2
54/64 at tau 20 vs 53/53 off. Decode unchanged on CodeLlama; prefill +6..+20 % (documented in docs/npu.md).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at 499328f |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY — but it answers my This is the measurement I asked for on #2552 and #2581 — "tau and the 16-channel cap need their provenance" — arriving as the evidence that a single
That is not a tuning difference, it is a design question the code now has to answer, and it should be answered in this PR rather than by a third model arriving with a third number:
Two more:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the categorical argument is sound, but the default's cost is only measured on the model that benefits from it
The evidence that matters is categorical, not statistical: Llama-2-7b-chat at tau 40 answers Unterscheidung between numbers is where host f32 gives EOS, and at tau 20/10/5 it gives EOS. An EOS lost is not a near-tie -- the model never stops, which is the same failure #2576 filed for CodeLlama. Moving a default on that evidence is right, and "the default is what makes these packs answer out of the box" is the correct justification for taking a prefill cost.
Two things the table says that the PR does not:
- The cost is measured only on the model that benefits. "Prefill (520 tok): 14,260 / 14,400 ms vs 15,165 / 17,244 ms, +6 % to +20 %" is CodeLlama-7b. But the default applies to every model, including the three whose answers do not change (Qwen2.5-3B, Qwen2.5-0.5B, SmolLM2). Those pay the same split cost for no benefit -- and the tau 20 arm's own spread (15,165 vs 17,244 = 13.7 %) is wider than the effect being claimed. Ask: one A B A B prefill row on a model whose accuracy is unchanged, so the default's price is stated for the population it applies to, not the population that gains.
- One row contradicts the direction and the text does not say so. Qwen2.5-0.5B is 53/64 at tau 0, 52 at tau 20, 51 at tau 40 -- i.e. tau 20 is lower than tau 0. That is inside the documented ±2 near-tie range, so it does not block the change, but a reader will notice a number moving the wrong way in the PR that proposes the default. Say it in the body: "0.5B is 52 vs 53 at tau 0, inside ±2, and the change is justified by a categorical failure (EOS), not by a tie-break."
And that is the honest form of the whole argument: this default moves on categorical evidence (EOS present/absent), while the teacher-forced columns are near-ties that neither support nor oppose it. Keeping those two kinds of evidence visually separate is what stops the ±2 numbers being read as the justification.
Nothing else: NPU_ACT_OUTLIER_TAU=40 restoring the old behaviour is the right escape hatch, "making those gathers cheaper is a follow-up" is the right way to leave the regression visible rather than hand-waved, and "Code: one constant plus the comment and header note" is the right size for a default change whose argument lives in the measurements.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
499328f CHANGES — the reason for the default is sound and better than what it replaces. The cost is quoted in a way that reads as a gain, and nothing watches the default.
1. +6 % to +20 % on a prefill time reads as faster; it is 20 % slower.
Prefill (520 tok): 14,260 / 14,400 ms vs 15,165 / 17,244 ms, +6 % to +20 %
Those are milliseconds for a fixed 520-token prefill, so lower is better: tau 20 costs 6 % to 20 % more prefill time. Two things make the range unquotable as an effect size:
- the two tau-20 samples are 15,165 and 17,244 ms — 13.7 % apart from each other, so most of the claimed range sits inside the treatment's own spread;
- the baseline pair (14,260 / 14,400, 1.0 % apart) is far tighter than that, which is itself worth noting — the split path's cost varies by more than the cost.
Suggested wording, which also preserves the decode result: "Prefill (520 tok, ms, lower is better): 14,260 / 14,400 vs 15,165 / 17,244 — tau 20 costs 6–20 % more prefill; the two tau-20 samples themselves differ by 13.7 %, so treat the upper end as noise. Decode (64 tok, tok/s): 0.997 / 1.046 vs 1.030 / 1.001, unchanged."
2. State the trade where the cost is, because it is a good trade. A default that costs prefill needs its reason in the same paragraph as the price. Yours is strong and specific: at tau 40, Llama-2-7b-chat (1.7.1 hybrid) answers '… 84. Unterscheidung between …' where host f32 gives EOS, and tau 20 / 10 / 5 all give EOS. One sentence — "this trades 6–20 % prefill for EOS correctness on 1.7.1 hybrid Llama-2, where f32 gives EOS and tau 40 does not" — turns a suspicious regression into a decision a reader can audit.
3. No arm reads the default. grep -rn 'outl_tau' tests/ returns nothing, and the diff touches three files, none of them a test:
docs/npu.md
npu/forward/include/q4nx_forward.h (the comment that names the default)
npu/forward/src/q4nx_forward.cpp (outl_tau_ = 20.0f)
The default is now stated in two places (the header comment and the assignment) and asserted in none, so the next person who edits one of them flips a documented default silently — and this PR exists because the previous default was wrong in exactly that way (40 → left the hybrid answer undifferentiated; nothing caught it). One arm: build the forward with NPU_ACT_OUTLIER_TAU unset and assert the picked value is 20.0; a second with =0 and assert it is off. That is the same shape as the ONEBIT_AUDIO_AVX2=1 finding on #2584 — a stated default with nothing watching it.
What I verified: the header comment and the assignment were both moved to 20 (they agree at this head); act_outliers_test exercises act_outlier_pick at tau 20.0 but passes the tau in, so it cannot see the default. What I did not verify: the table's numbers and the CodeLlama timings — no NPU window taken for this review, so they remain the author's measurements, and my objection is to the unit/direction wording, not to the values.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 499328f: READY
This is a default changed for the right reason, and the reason is not "one model wanted it"
The default moved 40 → 20, and the text says why the old value stopped being right:
on by default at tau 20 (it was 40 while only
ffn_downwas split). With gate/up covered, tau 40 [left Llama-2-7b-chat's EOS lost]
That is the distinction that makes this a fix rather than a fit: 40 was correct for the scope it was tuned on — ffn_down only — and engine#2576 widened the scope to gate/up, which changed the sensitivity the number was tuned against. A default that was right under one set of assumptions and is wrong under another is a default that must move when the assumptions move, and the PR states the pair rather than just the new number. Without that sentence this would read as tuning a global to one model.
Five models, not one, and the ones it does not fix are not harmed
| model | tau 0 | tau 20 | tau 40 |
|---|---|---|---|
| Llama-2-7b-chat | — | — | — → fixed |
| CodeLlama-7b-Instruct | — | — | — → EOS at the failing prefix at both |
| Qwen2.5-3B | — | 58/64 | 58/64 |
| Qwen2.5-0.5B | 53/64 | 52/64 | 51/64 |
| SmolLM2-135M | 53/64 | 54/64 | 53/64 |
The middle column is the one that matters: for the three models that are not the motivation, tau 20 is equal or one better than tau 40, and the reply text is compared rather than only the score ("answers unchanged"). A default change whose table only showed the model it fixes would be a default change I would block; this one shows the others and they are fine.
The cost is stated, and it is the only thing I would watch
Prefill (520 tok): 14,260 / 14,400 ms vs 15,165 / 17,244 ms, +6 % to +20 %. At tau 20 more prefill rows split [their channels]
CodeLlama pays 6–20 % prefill for a fix it does not benefit from, measured A B A B in one ticket, with decode unchanged (0.997 / 1.046 vs 1.030 / 1.001 tok/s). That is a real trade and the PR does not hide it — and it is the reason NPU_ACT_OUTLIER_TAU=0 still exists, so a lane that cares about CodeLlama prefill has a lever.
Worth knowing, not worth blocking: the 6–20 % range is wide (two repeats per arm), so the honest reading is "prefill costs something between 6 and 20 %" rather than "+6 %". A range with the repeats shown is the right way to leave it.
And the old table is kept as it was measured
The older table below was measured before gate/up were covered and is kept as it was measured.
That sentence is the difference between a document that accumulates evidence and one that quietly rewrites its history. The old numbers were true about the old code; changing them in place would destroy the only record of what the previous default was justified by — and the next person to ask "why did 40 exist?" would have nothing to read.
Ready.
|
VERDICT: READY at What I checked rather than readNothing regresses beyond the declared near-tie range. Comparing each model's tau 20 against its tau 0 on the PR's own numbers:
The cost arithmetic is right and is stated as a cost, not as a footnote: The decisive cell is The one blank
Either fill it or say why it is not needed, in one clause next to the table. A blank cell in a table whose every other cell is a measurement reads as an oversight even when it is not. Two things worth keeping verbatim
|
) * act_outliers: assert the tau/kmax defaults in a test (Refs #2576) Answers the CHANGES verdict on #2585 (merged): the tau-20 default was stated in two places and asserted in none. - act_outliers.h: kActOutlierTauDefault (20) / kActOutlierKmaxDefault (16) plus act_outlier_tau_from / act_outlier_kmax_from, the env readers init_impl now calls (same behaviour: unset -> default, "0" -> off, kmax clamped to >= 1). - act_outliers_test: test_defaults asserts unset -> 20, "0" -> 0, kmax unset -> 16, "0" -> 1, and that a ~28.5*rms channel is split at the default but not at tau 40. Mutant default 40 fails 2 checks. - docs/npu.md: prefill cost worded as time (lower is better), the 13.7 % spread of the tau-20 samples, and the trade stated next to the price. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs/npu: state the tau/kmax env contract and that no measured model prefers 40 (Refs #2576) Answers the #2587 review: one sentence for the NPU_ACT_OUTLIER_TAU / _KMAX contract the test asserts, and the per-model reason 20 is a default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Refs #2576. This is the Llama-2-7b-chat arm of the same issue; #2581 merged at tau 40 before it was measured.
What
With #2581's gate/up split at the default tau 40,
amd/Llama-2-7b-chat-hf_rai_1.7.1_hybridstill loses its EOS. Atchat("What is 12 times 7?") + ' Great! 12 times 7 is equal to 84.'(token ids, greedy):Unterscheidung between numbers isNPU_ACT_OUTLIER_TAU20 / 10 / 5The fix is the threshold, not a new projection, so this PR changes the default from 40 to 20.
Measured at tau 20, gate/up included (strixhalo,
--device npu, npu-hold, dmesg clean)Great! 12 times 7 is equal to 84.(stop, 16 tok); Paris / Armstrong right, stop… July 20, 1969 …; the…, 1prefix gives96The x/64 figures are teacher-forced top-1 against HF fp32, the same 64 probes per model; ±2 is the documented near-tie range.
Cost, stated plainly (CodeLlama-7b, tau 0 vs 20, A B A B in one ticket):
Making those gathers cheaper is a follow-up. Correctness wins for the default, because the default is what makes these packs answer out of the box.
NPU_ACT_OUTLIER_TAU=40restores the old behaviour.Code: one constant plus the comment and header note. Docs:
docs/npu.md,NPU_ACT_OUTLIER_TAUsection, with the table above; the earlier table is kept as measured.act_outliers_testpasses.🤖 Generated with Claude Code