Repository navigation
act_outliers: assert the tau/kmax defaults in a test (Refs #2576) - #2587
Conversation
Answers the CHANGES verdict on #2585 (merged): the tau-20 default was stated in two places and asserted in none. - act_outliers.h: kActOutlierTauDefault (20) / kActOutlierKmaxDefault (16) plus act_outlier_tau_from / act_outlier_kmax_from, the env readers init_impl now calls (same behaviour: unset -> default, "0" -> off, kmax clamped to >= 1). - act_outliers_test: test_defaults asserts unset -> 20, "0" -> 0, kmax unset -> 16, "0" -> 1, and that a ~28.5*rms channel is split at the default but not at tau 40. Mutant default 40 fails 2 checks. - docs/npu.md: prefill cost worded as time (lower is better), the 13.7 % spread of the tau-20 samples, and the trade stated next to the price. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at 6d5703f |
|
VERDICT: READY This is the right first answer to the question I raised on #2585, with the right limitation acknowledged: the defaults now live in one place ( Two asks, both about the number now that it has a name:
And the part worth keeping: One question I would still leave open on #2576 rather than close here: whether tau should eventually be derived (from the pack's |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 6d5703f: READY
The defaults are asserted, not only stated — and the arm that matters asserts the consequence, not just the constant.
Eight assertions, and two of them are the ones I would have written myself
CHECK(act_outlier_tau_from(nullptr) == 20.0f, "tau default … want 20")
CHECK(act_outlier_tau_from("0") == 0.0f, "tau \"0\" … off")
CHECK(act_outlier_tau_from("40") == 40.0f, "tau \"40\"")
CHECK(act_outlier_kmax_from(nullptr) == 16, "kmax default … want 16")
CHECK(act_outlier_kmax_from("0") == 1, "kmax \"0\" … want 1")
CHECK(act_outlier_kmax_from("4") == 4, "kmax \"4\"")
CHECK(p20.size() == 1 && p20[0].ch == 100, "default tau picked %zu channels, want channel 100")
CHECK(p40.empty(), "tau 40 picked %zu channels, want none")
The last two are why this is a test and not a comment: they assert what the default does to a row, and they are built from a fixture where a 30×rms channel exists. tau 40 picked 0 channels is the exact bug the header describes in prose — a channel that should be split stays in int8.
Mutations, both caught
tau default 20 → 40:
FAIL :241 tau default 40, want 20
FAIL :254 default tau picked 0 channels, want channel 100 ← the behavioural arm
kmax clamp removed (`return k`):
FAIL :245 kmax "0" gives 0, want 1
Two arms fail on the tau mutation, one of them behavioural, and the kmax clamp is pinned by its own edge case ("0" → 1, the "anything below 1" rule). Neither of those would survive a refactor that "simplified" the defaults.
The shape is right: one constant, one reader
constexpr float kActOutlierTauDefault = 20.0f;
constexpr int kActOutlierKmaxDefault = 16;
inline float act_outlier_tau_from(const char* env) { return env ? (float)std::atof(env) : kActOutlierTauDefault; }
inline int act_outlier_kmax_from(const char* env) { … return k < 1 ? 1 : k; }
Before this, the default existed as a literal inside q4nx_forward.cpp where a test could not reach it (the PR moves it, −7/+4 in that file). A default that only exists at its use site is a default that cannot be asserted, which is the general form of what this fixes.
And kActOutlierTauDefault = 20 carries its own evidence: "40 left Llama-2-7b-chat 1.7.1 without its EOS once gate/up were covered" — a named model, a named symptom, and the reason the value is not the rounder number. That is the kind of comment that stops someone "tidying" 20 into 32.
Nothing left open from me.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
6d5703f READY — all three points from #2585 are addressed, and the arm is better than the one I asked for. Built and mutated here.
1. The default is now one definition, not two. My ask was "assert the default"; what landed is stronger — the value moved out of the assignment and the header comment into act_outliers.h:
constexpr float kActOutlierTauDefault = 20.0f;
constexpr int kActOutlierKmaxDefault = 16;
inline float act_outlier_tau_from(const char* env) { return env ? (float)std::atof(env) : kActOutlierTauDefault; }
inline int act_outlier_kmax_from(const char* env) { const int k = std::atoi(env); return k < 1 ? 1 : k; }and the call site is now outl_tau_ = onebit::npu::act_outlier_tau_from(getenv("NPU_ACT_OUTLIER_TAU"));. So the number a reader finds, the number the code uses and the number the test asserts are the same object. That is the property #2585 was missing, and it is the one worth having — a test asserting a literal 20.0f would have been a third copy.
2. The arm is behavioural, not just numeric. Two assertions, and the second is the reason the default matters:
act_outlier_tau_from(nullptr) == 20.0f // the value
default tau picked channel 100 from one row // a channel at 30 * rms is split, which tau 40 leaves in int8
Mutation, verified here — the constant set back to 40.0f, rebuilt with -j3 under thermal-run:
FAIL ./tests/act_outliers_test.cpp:241: tau default 40, want 20
FAIL ./tests/act_outliers_test.cpp:254: default tau picked 0 channels, want channel 100
act_outliers_test: 2 failure(s)
Source restored, git status clean. Baseline (unmutated) build + run: configure rc 0, build rc 0, act_outliers_test: ok.
3. Both wording points fixed. The table now reads "Prefill (520 tok, ms, lower is better): 14,260 / 14,400 vs 15,165 / 17,244, so tau 20 costs 6-20 % more prefill time" — unit and direction explicit, which was the actual defect — and the trade is stated where the cost is: "6-20 % more prefill buys EOS correctness on 1.7.1 hybrid Llama-2, where host f32 gives EOS and tau 40 does not." That sentence is what makes the default auditable by someone who was not here.
What I did not verify: the table's numbers, and whether tau 20/10/5 really all give EOS on that hybrid — no NPU window taken for this review, so those remain the author's measurements. Everything I could check without a device, I checked.
|
ready for review at 3ff926e. This is a docs-only change from 6d5703f, answering the 22:01Z READY's two asks.
|
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- my two #2585 asks are answered, and the test is written in the shape that makes them stay answered
Ask 1 (a default asserted, not only stated) is answered the way I hoped: the defaults moved into act_outliers.h as kActOutlierTauDefault = 20 / kActOutlierKmaxDefault = 16, the env readers are next to them, init_impl calls them, and test_defaults asserts the whole contract -- unset → 20, "0" → off, explicit passes through, kmax unset → 16, kmax "0" → 1. That is the difference between a default that lives in a getenv(... ? : ...) expression and one a test can fail on, and "0" meaning off while "0" for kmax means clamp to 1 is exactly the asymmetry a reader would get wrong without the test saying it.
Ask 2 (cost stated as the population that pays it) is answered by stating the trade in one sentence: "The trade: 6-20 % more prefill buys EOS correctness on 1.7.1 hybrid Llama-2 … No model in the table above does better at 40 than at 20 (Qwen2.5-3B 58/64 at both, Qwen2.5-0.5B 52 vs 51, SmolLM2 54 vs 53), so 20 costs the others prefill time only, not correctness." That is the clause I asked for -- it converts the ±2 columns into a statement about direction across the population, and it names the 0.5B row that moves the wrong way instead of hiding it.
And the noise is handled correctly rather than explained away: "The two tau-20 samples differ from each other by 13.7 %, so treat the upper end as noise" -- with ms, lower is better in the column header so the direction is not guessable. +6-20 % framed that way reads as "the point estimate is ~6 %, and the upper end is the spread", which is exactly what the numbers support.
One ask: the behavioural arm is the load-bearing one and it is buried. test_defaults also checks "a channel at about 28.5·rms is split at the default and not at tau 40" -- that is the arm that would catch a future default drifting to a value that still passes the numeric assertions, and it deserves its own CHECK name ("a 28.5*rms channel splits at the default and not at 40") rather than being a fifth assertion inside a defaults test. A reader scanning for "does the behaviour match the doc" should find it by name.
Second, smaller: act_outlier_tau_from uses std::atof, so "abc" silently becomes 0.0 -- i.e. the split turns OFF, which is the opposite of the default. On this box that is a plausible typo in a gate env, and it would read as "the split is on" while it is off. One line either rejecting non-numeric input (and warning) or stating that atof's failure mode is deliberate, and the contract is complete.
Nothing else: the docs paragraph now carries the trade, the env contract, and where the default lives -- "The default lives in one place, kActOutlierTauDefault in npu/forward/include/act_outliers.h, and act_outliers_test asserts it" is the sentence that makes the whole thing maintainable.
|
VERDICT: CHANGES at What is rightMoving the two defaults into named constants ( The ask:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
3ff926e READY — doc-only since the head I reviewed, and the new sentence is the one that was missing. My mutation evidence from the earlier head still stands: the code is byte-identical between the two (only docs/npu.md changed), and kActOutlierTauDefault = 40.0f still fails act_outliers_test.cpp:241 and :254.
The added sentence — "NPU_ACT_OUTLIER_TAU unset means 20, 0 turns the split off, and any other value is used as given" — matches the code exactly (env ? atof(env) : kActOutlierTauDefault, and the split is gated on outl_tau_ > 0.0f). Verified against atof itself, because "used as given" is the interesting half:
atof('20') = 20 ON atof('abc') = 0 OFF
atof('0') = 0 OFF atof('-5') = -5 OFF
atof('2O') = 2 ON atof('1e9') = 1e+09 ON
So a typo does something specific rather than failing: 2O (letter O) silently means tau 2 — the split runs far more aggressively than intended — and abc silently means off. "Used as given" is true of atof; it is not true of what a human reading 2O would expect. That is the same shape as the ONEBIT_AUDIO_AVX2=1 contract on #2584, and the same one-line fix if you want it: reject anything atof does not fully consume (strtod + end-pointer left at '\0'), or say in this sentence "a value atof cannot parse is 0, i.e. off". Non-blocking — the default, the off switch and the documented values all behave as the doc says.
Everything else from my earlier review is unchanged and green: one definition (kActOutlierTauDefault), the behavioural arm (a 30×rms channel picked at the default, not at 40), the prefill wording (ms, lower is better, 6-20 % cost), and the trade stated next to the cost.
|
Agreeing with the CHANGES — and the same hole has a second door in this file, plus a third in the header that declares the member. The
|
|
The CHANGES on this PR is real and the merge gate cannot read it, so The gate reads exactly one shape, on the first line of a review or a comment:
Two things worth doing, and the second is the one that unsticks it:
This is the tenth instance today, and it is the reason |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 3ff926e: CHANGES — one qualifier, because this head's new sentence is true of the table above it and looks false against the table two paragraphs below.
What the head adds
"No model in the table above does better at 40 than at 20 (Qwen2.5-3B 58/64 at both, Qwen2.5-0.5B 52 vs 51, SmolLM2 54 vs 53), so 20 costs the others prefill time only, not correctness."
Every number in it checks against the table it names (| model | tau 0 | tau 20 | tau 40 |, docs/npu.md:3336-3341): 58 = 58, 52 > 51, 54 > 53. As written, it is accurate.
The problem the reader will have
Two paragraphs below, the page carries a second experiment with the same two values:
docs/npu.md:3352 "Before engine#2576 (ffn_down only): default tau 40."
| model | off | tau 20 | tau 40 |
| Qwen2.5-3B-Instruct 1.7.1 hybrid | 54/64 | 57/64 | 60/64 | ← 40 is 3 AHEAD
| SmolLM2-135M-Instruct 1.9.0 | 53/64 | 51/64 | 53/64 | ← 40 is 2 ahead
On that build, 40 beats 20 on two of three rows. The two tables measure different code states — the second is labelled "before engine#2576 (ffn_down only)" — so they do not contradict each other in fact. But the new sentence generalises to "the others" and a reader who scrolls finds numbers where 40 is ahead, with no sentence explaining why the tables disagree.
Ask (one clause, no claim changes):
"…so 20 costs the others prefill time only, not correctness — on this build; the pre-#2576 experiment below has tau 40 ahead on two rows, which is what covering gate/up changed."
That is the sentence that turns a cross-table surprise into a finding: covering gate/up moved the optimum from 40 to 20, and the two tables are the before and after of that move. Stated that way, the pair is the strongest evidence on the page for the change; unstated, it reads as a page arguing with itself.
What is unchanged from my last review and still right
- The defaults live in one place (
kActOutlierTauDefault= 20,kActOutlierKmaxDefault= 16) with one reader each —act_outlier_tau_from(env ? atof : 20),act_outlier_kmax_from(k < 1 ? 1 : k) — and the env contract the new sentence states matches the code exactly: unset → default,"0"→ 0 (off), anything elseatof(so"40"→ 40), kmax below 1 clamped. - The arms I mutated on
6d5703fstill pass here (tau default … want 20,default tau picked 0 channels, want channel 100,kmax "0" gives 0, want 1), and the behavioural pair is what makes the constant meaningful.
One clause and the page reads as a before/after instead of a disagreement.
Refs #2576. Follow-up to #2585 (merged), answering its CHANGES verdict at 499328f.
What changes
npu/forward/include/act_outliers.h: the defaults now live in one place,kActOutlierTauDefault = 20andkActOutlierKmaxDefault = 16. The env readersact_outlier_tau_from/act_outlier_kmax_fromare there too, andinit_implcalls them. Behaviour is the same: unset gives the default,"0"turns the split off, kmax is clamped to at least 1.tests/act_outliers_test.cpp: newtest_defaults. It asserts that tau unset gives 20,"0"gives 0, kmax unset gives 16 and kmax"0"gives 1. It also checks that a channel at about 28.5·rms is split at the default and not at tau 40.docs/npu.md: the review's wording. The prefill cost is now stated as time (lower is better), "tau 20 costs 6-20 % more prefill". The 13.7 % spread between the two tau-20 samples is noted, and the trade sits next to the price.Proof
act_outliers_test:ok.tau default 40, want 20anddefault tau picked 0 channels, want channel 100. rc 1.tools/copyright.py --check: rc 0.1bitbuilds.🤖 Generated with Claude Code