Repository navigation
npu pack: the blocked int8 packer vectorises and parallelises its amax pass, bit-identical (#1783) - #2359
Conversation
…n parallel, bit-identical (#1783) q4nx_pack_B_int8_blocked packs every weight into the NPU's b8 image, and on the encoder route it also packs the activation operands of the attention products on every call. Two things kept it slow: the column amax pass was serial, and std::roundf / std::isfinite kept the quantise loop scalar. - amax runs over row chunks in parallel, per-thread arrays merged after (a max is exact and order-independent); - the code is computed by truncation plus a step on the exact fraction, which is roundf's half-away-from-zero for |y| <= 127 and vectorises; `v - v == 0` replaces isfinite; - one blocked_index per 8-column microtile run instead of per element. Output is unchanged: codes, scales and the scale sum equal the previous packer's on 20 shapes x 2 magnitudes, 1/8/16 threads. A new arm of npu_blocked_layout_test pins the rounding against the row-major reference on every k +- 0.5 tie and its float neighbours, the +-127 clamp and NaN/inf; a ties-toward-zero mutant passes the old test and fails this one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY at The asymmetry, stated exactly
So one function serves two very different lifetimes: a weight packed once and cached, and an activation packed per call. Optimising the second is pure win — no cache invalidation, no memory growth — and the numbers say why it matters: 1536×1536 per projection per call on the encoder route. Two asks
And the scope line is the right one"Part of #1783 (Whisper encoder on the NPU). This is the host side." — naming which side means the measurement that matters (the 0.6 s bar from #2356) is attributable to host work, and the device side follows separately. With #2356 parallelising Conv/Softmax/LayerNorm and this parallelising the packer, the host remainder of a 30 s window is being attacked from both of its biggest terms — which is the right order given #2356's split (Conv 3.74 s, Softmax 2.56 s, attention products 2.76 s). |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 2add34c: READY
The rounding is the part I came to check and it is exactly right — you avoided the trap that a vectorised roundf usually falls into.
const float y = (v - v == 0.0f) ? v * tis : 0.0f;
int x = (int)y;
const float f = y - (float)x;
x += (f >= 0.5f) - (f <= -0.5f);roundfrounds half away from zero; every SSE conversion (_mm_cvtps_epi32,lrintfunder the default mode) rounds half to even. A vectorisation that reaches for the instruction instead of the expression changes the answer on exactly the inputs a fixture is least likely to contain — exact.5values. Trunc-plus-fraction reproducesroundfincluding those cases:-2.5 → t = -2, f = -0.5 → -3,2.5 → 3.- And the fraction is exact, for the reason your comment gives:
|y| <= 127bounds the truncation intoint, andy - twitht = trunc(y)is a difference of two floats within a factor of two of each other (ort == 0, where it is trivially exact), so no rounding enters before the comparison. (v - v == 0.0f)is a branch-freeisfiniteand it covers both cases the old guard covered: for±inf,inf - infis NaN; for NaN it is NaN; both take the0.0fbranch, exactly asif (!std::isfinite(v)) v = 0.0fdid. Nice that it is also the vectorisable form.
The amax pass is order-free by construction — per-thread mine, merged under one critical, and max is associative and commutative on exact values, so nowait and any merge order give the same amax. ssum staying serial is the right call for the same reason in reverse: a sum is not order-free in floating point, so leaving it alone is what makes "the output is unchanged" true rather than approximately true.
The equivalence proof is the right shape: the old packer compiled side by side under another name, 10 shapes × 2 magnitudes at 1/8/16 threads, including padded images (KD > K, ND > N) — the padded cases matter because bpos() with a padded image is where an index refactor would go wrong, and one blocked_index per 8-column microtile run is exactly that kind of refactor.
One edge the equivalence proof cannot reach, and it is not this PR's fault
tis[j] = 127.0f / am with am = 0 (a column entirely zero, or entirely non-finite) gives +inf. Then for a finite v = 0:
y = v * tis = 0 * inf = NaN (the ternary's condition is true, so the multiply happens)
int x = (int)NaN ← undefined behaviour; x86 yields INT_MIN
x < -127 → x = -127 ← the clamp catches it, so the code is -127, not 0
main has the identical path — (int)std::roundf(0 * inf) — so this is pre-existing and this PR leaves the observable output unchanged, which is what it promised. It is also numerically benign only by accident: col_scales[j] = am / 127 = 0, so every code dequantises to 0 * anything, and both versions therefore agree.
Two consequences worth a line somewhere, in the PR or a note:
- it is reachable UB, so a UBSan/UBSan-instrumented run on an all-zero column would flag
(int)NaN. Fixing it is one token —tis[j] = am > 0.0f ? 127.0f / am : 0.0f— and it would make the zero column actually encode zeros instead of relying on the scale to erase them; - your equivalence harness would not catch a change here even if it regressed, because both sides take the same UB path and produce the same
INT_MIN. That is a useful thing to know about the proof: it compares implementations, not correctness, on inputs where both are wrong.
Not a blocker — the change is exact, the proof is strong, and this is the same behaviour as main. But this PR is what put the expression in one place with a comment claiming "a non-finite v as 0", and the honest addendum is that a finite v with an infinite tis falls outside that claim.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- sign-aware truncation, ties and ulps in the test, and the one decision that had to go the other way
Replacing std::roundf with truncation is where this kind of PR usually breaks, and the handling here is exactly right:
x += (f >= 0.5f) - (f <= -0.5f); // f = y - trunc(y), the SIGNED fractiontrunc(-2.7) is -2 and the fraction is -0.7, so t - 1 = -3 -- the sign-aware step that trunc(x + 0.5f) gets wrong for every negative value. The comment states the rule ("rounding half away from zero is then t + 1 for a fraction >= 0.5 and t - 1 for one <= -0.5") so the next reader does not have to reconstruct it, and v - v == 0 for the finite check is the neat version of isfinite.
And the test is the only one that can actually prove "roundf exactly": "Columns whose amax is 127 make the scale exactly 1, so each value below reaches the rounding unchanged: every k + 0.5 and k - 0.5 tie, the floats one ulp either side of a tie..." Ties are where half-away-from-zero and half-to-even diverge, and one ulp either side is where a fraction comparison against 0.5f can flip. Testing the ties alone would pass with a subtly wrong offset; testing the neighbours is what makes the claim real.
The pair of decisions worth naming, because they are opposite and both correct:
- amax is parallelised -- "A max is exact and order-independent", so parallel chunks merged per thread cannot change the result;
- the scale sum is left serial -- "so
ssumis order-independent as before".
The same person choosing to parallelise one reduction and not the other, for the right reason each time, is the thing this week's NPU PRs kept getting wrong in the other direction (#2356's per-row split, #2349's guard): max can be split, sum cannot, and a float sum over 16 threads would have made the packed image depend on scheduling.
The numbers carry the useful separation too: 3.01 -> 0.89 ms on K768×N3072 at 16 threads is 3.4x, but the 1-thread row is 7.61 -> 4.88 ms, so the vectorisation alone is ~1.6x and the threads add the rest. Worth one sentence in the body, because it tells the next person which lever to reach for on a shape where 16 threads are not available.
Nothing else: the padded-image, row-major and odd-N shapes are in the 10×2 matrix, which is where a blocked packer actually goes wrong.
Part of #1783 (Whisper encoder on the NPU). This is the host side.
q4nx_pack_B_int8_blockedpacks every weight into the NPU's blocked int8 image (I8Ctx::packB, the weight-cache fill). On the encoder route it would also pack the activation operand of each attention product on every call: Kᵀ, and 1536×1536 for the probabilities. At that size the packer cost more than the device product it feeds.What changes (
npu/forward/src/q4nx_pack_blocked.cpp)std::roundfandstd::isfiniteper element, which kept it scalar. The code is now truncation plus a step on the fraction:+1for a fraction ≥ 0.5,−1for one ≤ −0.5. That is exactlyroundf's round-half-away-from-zero here, because the fraction is exact and |y| ≤ 127 (every value is bounded by its column's amax, and non-finite values are zeroed).v - v == 0replacesisfinite.blocked_indexper 8-column microtile run, instead of one per element (8 consecutive bytes in the blocked layout).The output is unchanged. The scale-sum loop is untouched and stays serial, so
ssumis order-independent as before.Proof
Equivalence against main's packer, compiled side by side under another name:
blocked=0), an N that is not a multiple of 8, and NaN, +inf and −inf, plus an all-zero column.ssum: 0 differing cases.Timing. Median of 25 runs, 8 OpenMP threads, not pinned, box load ~9:
For the encoder's 144 attention calls per 30 s window, packing the operands drops from about 0.37 s to 0.135 s.
Tests:
test_packer_rounding_edgesinnpu_blocked_layout_test. Columns with amax 127 give a scale of exactly 1. The arm then checks 775 values against the row-major reference'sroundf: every k ± 0.5 tie and the floats one ulp either side, the ±127 clamp, ±0, denormals, NaN and ±inf.f > 0.5) passes the existing test and fails the new arm.tools/copyright.py --check: rc 0.🤖 Generated with Claude Code