Skip to content

npu pack: the blocked int8 packer vectorises and parallelises its amax pass, bit-identical (#1783) - #2359

Merged
bong-water-water-bong merged 1 commit into
mainfrom
1bit/1783-fast-blocked-pack
Nov 13, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
1bit/1783-fast-blocked-pack

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Part of #1783 (Whisper encoder on the NPU). This is the host side.

q4nx_pack_B_int8_blocked packs every weight into the NPU's blocked int8 image (I8Ctx::packB, the weight-cache fill). On the encoder route it would also pack the activation operand of each attention product on every call: Kᵀ, and 1536×1536 for the probabilities. At that size the packer cost more than the device product it feeds.

What changes (npu/forward/src/q4nx_pack_blocked.cpp)

  • The column amax pass runs in parallel over row chunks, into per-thread arrays merged afterwards. A max is exact and order-independent, so the scales are unchanged. Before, this pass was serial.
  • The quantise loop vectorises. It used std::roundf and std::isfinite per element, which kept it scalar. The code is now truncation plus a step on the fraction: +1 for a fraction ≥ 0.5, −1 for one ≤ −0.5. That is exactly roundf's round-half-away-from-zero here, because the fraction is exact and |y| ≤ 127 (every value is bounded by its column's amax, and non-finite values are zeroed). v - v == 0 replaces isfinite.
  • One blocked_index per 8-column microtile run, instead of one per element (8 consecutive bytes in the blocked layout).

The output is unchanged. The scale-sum loop is untouched and stays serial, so ssum is order-independent as before.

Proof

Equivalence against main's packer, compiled side by side under another name:

  • 10 shapes × 2 magnitudes, at 1, 8 and 16 threads. The shapes include padded images (KD > K, ND > N), row-major (blocked=0), an N that is not a multiple of 8, and NaN, +inf and −inf, plus an all-zero column.
  • Codes, column scales and ssum: 0 differing cases.

Timing. Median of 25 runs, 8 OpenMP threads, not pinned, box load ~9:

shape main this PR
K64 × N1536 (Kᵀ per head) 0.165 ms 0.074 ms
K1536 × N1536 (Pᵀ per head) 2.44 ms 0.86 ms
K768 × N3072 (a weight) 3.01 ms 0.89 ms
K1536 × N1536, 1 thread 7.61 ms 4.88 ms

For the encoder's 144 attention calls per 30 s window, packing the operands drops from about 0.37 s to 0.135 s.

Tests:

  • New arm, test_packer_rounding_edges in npu_blocked_layout_test. Columns with amax 127 give a scale of exactly 1. The arm then checks 775 values against the row-major reference's roundf: every k ± 0.5 tie and the floats one ulp either side, the ±127 clamp, ±0, denormals, NaN and ±inf.
  • Mutants:
    • Ties-toward-zero (f > 0.5) passes the existing test and fails the new arm.
    • Dropping the negative step fails both.
  • ctest: 12/12 pass (npu_blocked_layout, npu_pb16i/t2s/t2_layout, npu_padded_grid, npu_design_grid, q4nx_diff_grid, repack_tile_order, npu_repack_path, npu_pack_components, audio_pack, audio_exec).
  • tools/copyright.py --check: rc 0.

🤖 Generated with Claude Code

…n parallel, bit-identical (#1783)

q4nx_pack_B_int8_blocked packs every weight into the NPU's b8 image, and on the encoder route it also packs the
activation operands of the attention products on every call. Two things kept it slow: the column amax pass was
serial, and std::roundf / std::isfinite kept the quantise loop scalar.

- amax runs over row chunks in parallel, per-thread arrays merged after (a max is exact and order-independent);
- the code is computed by truncation plus a step on the exact fraction, which is roundf's half-away-from-zero for
  |y| <= 127 and vectorises; `v - v == 0` replaces isfinite;
- one blocked_index per 8-column microtile run instead of per element.

Output is unchanged: codes, scales and the scale sum equal the previous packer's on 20 shapes x 2 magnitudes, 1/8/16
threads. A new arm of npu_blocked_layout_test pins the rounding against the row-major reference on every k +- 0.5 tie and
its float neighbours, the +-127 clamp and NaN/inf; a ties-toward-zero mutant passes the old test and fails this one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

1783 - Partially compliant

Compliant requirements:

  • Correctness bar: bit-exact transcripts verified with new test cases
  • Measure and close the gap: performance improvements achieved through vectorization and parallelization

Non-compliant requirements:

  • Baseline harness not implemented in this PR
  • Encoder route on our runtime not implemented in this PR
  • Row not yet completed in this PR

Requires further human verification:

  • Baseline harness implementation
  • Full encoder route on our runtime
  • Final performance measurements and memory usage verification
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Rounding Precision

The new quant_code function uses truncation followed by a step based on the fraction to achieve vectorization, which is intended to be equivalent to std::roundf for round-half-away-from-zero behavior. However, the implementation assumes that the fraction is exact and that values are bounded by their column's amax, which may not hold in all edge cases. This could lead to subtle differences in rounding behavior that might affect correctness, especially in scenarios involving very small or very large values.

inline int8_t quant_code(float v, float tis) {
    const float y = (v - v == 0.0f) ? v * tis : 0.0f;
    int x = (int)y;
    const float f = y - (float)x;
    x += (f >= 0.5f) - (f <= -0.5f);
    x = x > 127 ? 127 : x;
    x = x < -127 ? -127 : x;
    return (int8_t)x;
}
Parallelization Strategy

The column amax calculation is now parallelized using OpenMP, with each thread maintaining its own array and merging results via a critical section. While this improves performance, the use of #pragma omp critical can introduce a bottleneck and may not scale well with a large number of threads or very large matrices. The critical section could become a performance bottleneck, especially for large N values.

#pragma omp parallel
    {
        std::vector<float> mine((size_t)N, 0.0f);
#pragma omp for schedule(static) nowait
        for (int i = 0; i < K; i++) {
            const float* row = w + (size_t)i * (size_t)N;
            for (int j = 0; j < N; j++) {
                const float a = std::fabs(row[j]);
                if (std::isfinite(a) && a > mine[(size_t)j]) mine[(size_t)j] = a;
            }
        }
#pragma omp critical(q4nx_pack_amax)
        for (int j = 0; j < N; j++)
            if (mine[(size_t)j] > amax[(size_t)j]) amax[(size_t)j] = mine[(size_t)j];
    }

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 2add34c — the packer is doing per-call work that the weight path does once, and the distinction between the two is the whole PR.

The asymmetry, stated exactly

"q4nx_pack_B_int8_blocked packs every weight into the NPU's blocked int8 image (I8Ctx::packB, the weight-cache fill). On the encoder route it would also pack the activation operand of each attention product on every call: Kᵀ, and 1536×1536 for the p[rojection]"

So one function serves two very different lifetimes: a weight packed once and cached, and an activation packed per call. Optimising the second is pure win — no cache invalidation, no memory growth — and the numbers say why it matters: 1536×1536 per projection per call on the encoder route.

Two asks

  1. say whether the packing is now vectorised over the same layout or a different one — the PR title says "vectorises and parallelises"; if the output bytes must remain identical to I8Ctx::packB's (which the weight-cache path depends on), that equality is the test the change needs, and it is the same ask I made on gpu-kernels#75's q4_0 encoder. If the layout changes, say so, because then the weight and activation paths would diverge;
  2. state the threading contract — "parallelises" over what, and is the accumulation order per output element unchanged? For a packing loop (pure rearrangement of bytes) that is trivially true; if any arithmetic is inside, it is not, and the difference decides whether the encoder's output is bit-stable.

And the scope line is the right one

"Part of #1783 (Whisper encoder on the NPU). This is the host side." — naming which side means the measurement that matters (the 0.6 s bar from #2356) is attributable to host work, and the device side follows separately. With #2356 parallelising Conv/Softmax/LayerNorm and this parallelising the packer, the host remainder of a 30 s window is being attacked from both of its biggest terms — which is the right order given #2356's split (Conv 3.74 s, Softmax 2.56 s, attention products 2.76 s).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 2add34c: READY

The rounding is the part I came to check and it is exactly right — you avoided the trap that a vectorised roundf usually falls into.

const float y = (v - v == 0.0f) ? v * tis : 0.0f;
int x = (int)y;
const float f = y - (float)x;
x += (f >= 0.5f) - (f <= -0.5f);
  • roundf rounds half away from zero; every SSE conversion (_mm_cvtps_epi32, lrintf under the default mode) rounds half to even. A vectorisation that reaches for the instruction instead of the expression changes the answer on exactly the inputs a fixture is least likely to contain — exact .5 values. Trunc-plus-fraction reproduces roundf including those cases: -2.5 → t = -2, f = -0.5 → -3, 2.5 → 3.
  • And the fraction is exact, for the reason your comment gives: |y| <= 127 bounds the truncation into int, and y - t with t = trunc(y) is a difference of two floats within a factor of two of each other (or t == 0, where it is trivially exact), so no rounding enters before the comparison.
  • (v - v == 0.0f) is a branch-free isfinite and it covers both cases the old guard covered: for ±inf, inf - inf is NaN; for NaN it is NaN; both take the 0.0f branch, exactly as if (!std::isfinite(v)) v = 0.0f did. Nice that it is also the vectorisable form.

The amax pass is order-free by construction — per-thread mine, merged under one critical, and max is associative and commutative on exact values, so nowait and any merge order give the same amax. ssum staying serial is the right call for the same reason in reverse: a sum is not order-free in floating point, so leaving it alone is what makes "the output is unchanged" true rather than approximately true.

The equivalence proof is the right shape: the old packer compiled side by side under another name, 10 shapes × 2 magnitudes at 1/8/16 threads, including padded images (KD > K, ND > N) — the padded cases matter because bpos() with a padded image is where an index refactor would go wrong, and one blocked_index per 8-column microtile run is exactly that kind of refactor.

One edge the equivalence proof cannot reach, and it is not this PR's fault

tis[j] = 127.0f / am with am = 0 (a column entirely zero, or entirely non-finite) gives +inf. Then for a finite v = 0:

y = v * tis = 0 * inf = NaN      (the ternary's condition is true, so the multiply happens)
int x = (int)NaN                 ← undefined behaviour; x86 yields INT_MIN
x < -127 → x = -127              ← the clamp catches it, so the code is -127, not 0

main has the identical path — (int)std::roundf(0 * inf) — so this is pre-existing and this PR leaves the observable output unchanged, which is what it promised. It is also numerically benign only by accident: col_scales[j] = am / 127 = 0, so every code dequantises to 0 * anything, and both versions therefore agree.

Two consequences worth a line somewhere, in the PR or a note:

  • it is reachable UB, so a UBSan/UBSan-instrumented run on an all-zero column would flag (int)NaN. Fixing it is one token — tis[j] = am > 0.0f ? 127.0f / am : 0.0f — and it would make the zero column actually encode zeros instead of relying on the scale to erase them;
  • your equivalence harness would not catch a change here even if it regressed, because both sides take the same UB path and produce the same INT_MIN. That is a useful thing to know about the proof: it compares implementations, not correctness, on inputs where both are wrong.

Not a blocker — the change is exact, the proof is strong, and this is the same behaviour as main. But this PR is what put the expression in one place with a comment claiming "a non-finite v as 0", and the honest addendum is that a finite v with an infinite tis falls outside that claim.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- sign-aware truncation, ties and ulps in the test, and the one decision that had to go the other way

Replacing std::roundf with truncation is where this kind of PR usually breaks, and the handling here is exactly right:

x += (f >= 0.5f) - (f <= -0.5f);      // f = y - trunc(y), the SIGNED fraction

trunc(-2.7) is -2 and the fraction is -0.7, so t - 1 = -3 -- the sign-aware step that trunc(x + 0.5f) gets wrong for every negative value. The comment states the rule ("rounding half away from zero is then t + 1 for a fraction >= 0.5 and t - 1 for one <= -0.5") so the next reader does not have to reconstruct it, and v - v == 0 for the finite check is the neat version of isfinite.

And the test is the only one that can actually prove "roundf exactly": "Columns whose amax is 127 make the scale exactly 1, so each value below reaches the rounding unchanged: every k + 0.5 and k - 0.5 tie, the floats one ulp either side of a tie..." Ties are where half-away-from-zero and half-to-even diverge, and one ulp either side is where a fraction comparison against 0.5f can flip. Testing the ties alone would pass with a subtly wrong offset; testing the neighbours is what makes the claim real.

The pair of decisions worth naming, because they are opposite and both correct:

  • amax is parallelised -- "A max is exact and order-independent", so parallel chunks merged per thread cannot change the result;
  • the scale sum is left serial -- "so ssum is order-independent as before".

The same person choosing to parallelise one reduction and not the other, for the right reason each time, is the thing this week's NPU PRs kept getting wrong in the other direction (#2356's per-row split, #2349's guard): max can be split, sum cannot, and a float sum over 16 threads would have made the packed image depend on scheduling.

The numbers carry the useful separation too: 3.01 -> 0.89 ms on K768×N3072 at 16 threads is 3.4x, but the 1-thread row is 7.61 -> 4.88 ms, so the vectorisation alone is ~1.6x and the threads add the rest. Worth one sentence in the body, because it tells the next person which lever to reach for on a shape where 16 threads are not available.

Nothing else: the padded-image, row-major and odd-N shapes are in the 10×2 matrix, which is where a blocked packer actually goes wrong.

@bong-water-water-bong
bong-water-water-bong merged commit 6689366 into main Nov 13, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the 1bit/1783-fast-blocked-pack branch November 13, 2026 11:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant