Skip to content

npu: the i8 host arm indexes a K*N buffer with the blocked tile grid, so an N that is not 128-aligned reads and writes past it (merged in #1888; blocked_fits() has no caller) #1994

Closed

Description

@bong-water-water-bong

host_gemm_i8's weight image is sized K*N but indexed with the blocked tile grid, so any N that is not a multiple of 128 writes and reads past the buffer (merged in #1888; the blocked_fits() guard it needs still has no caller).

Filed because my review of #1888 raised this three times and the PR merged anyway, so the finding should not die with a comment thread. Nothing here is new information — it is the same arithmetic, restated with the line numbers on main:

// npu/forward/src/q4nx_forward.cpp, on main today
:3145   t.codes.assign((size_t)K * (size_t)N, 0);                       // exactly K*N bytes
:3203   acc += (int32_t)((int)t.codes[q4nx_layout::blocked_index((size_t)k, (size_t)n, K, N)] * ...
:5370   acc += (int32_t)((int)c.codes[q4nx_layout::blocked_index((size_t)k, (size_t)n, H, Tn)] *  // LM head

blocked_index() addresses a tile grid: ceil(N/128) * ceil(K/64) * 8192 bytes. That equals K*N only when N % 128 == 0 and K % 64 == 0. Otherwise the write (the packer uses the same index) and the read both run past the end of a std::vector<int8_t>:

path K N buffer blocked needs
LM head's last vocabulary block, phi-3-mini (vocab 32064 → last block 32064 % 8192 = 7488) 3072 7488 23,003,136 23,199,744 +196,608 B
a body GEMM at N = 8512 2048 8512 17,432,576 17,563,648 +131,072 B
LM head's last block, vocab 150144 (the measured case) 4096 2688 11,010,048 11,010,048 fits
body at N = 11008 (the measured family) 4096 11008 45,088,768 45,088,768 fits

Why it survived review and measurement: every geometry this arm has been exercised on is aligned. 150144 % 8192 = 2688 = 21 × 128 and 11008 = 86 × 128, so the documented bit-exact results are true on those shapes. The failure needs a vocabulary whose last block is not 128-aligned — 32064 % 8192 = 7488 = 58.5 × 128 is phi-3-mini, i.e. exactly the family this importer is for — or a body width that is not, 8512 = 66.5 × 128.

What it would look like: not a crash. The packer writes past the vector (heap corruption of whatever follows), then the MAC reads the same out-of-range offsets, so logits come back as whatever is in those bytes. For an arm whose entire purpose is to be the reference a kernel family is gated against, silently wrong numbers that still look like an answer are the worst available failure mode — and the gate would then "pass" while the device was right and the reference was wrong.

The fix is one expression at each of the two sites (q4nx_layout::blocked_fits() exists, is tested in tests/npu_blocked_layout_test.cpp, and has no caller in the forward):

const int blocked = q4nx_layout::blocked_fits(K, N) ? 1 : 0;   // rowmajor_index is i*ND + j: safe for any geometry

The 174.5 s vs 32.2 s argument for blocked is real (measured in the PR), so refusing or falling back on unaligned geometry keeps the fast path where it is valid and is slower only where the alternative is corruption. An arm with N = 32064, K = 3072 under NPU_INFER_HOST_GEMM=i8 would make this executable — ASan turns it into a hard failure, and the arithmetic above makes it a deterministic one.

Related: #1992 documents this arm (and its LM head) as the reference that can be bit-exact. That is true on aligned shapes and I am not disputing the measurements; the constraint simply needs to be enforced, or stated on the page.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claimed (cl-1077). Plan: gate the blocked layout on q4nx_layout::blocked_fits(K, N) at the packer and at both MAC sites, so the body GEMM and the LM head's last block use row-major when the tile grid would not fit. Proof: a crafted arm at an unaligned N (e.g. 7488) that is bit-exact against row-major and runs clean under ASan.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Fix is up as #1999 (a579da51), and I built its packers and ran the new test: the unaligned shapes come out row-major with the forced blocked index reaching 23,199,232 of 23,003,136 B (phi-3-mini's last block) and 17,563,136 of 17,432,576 B (N=8512), while 4096x2688 — the measured, aligned head block — stays blocked and lands exactly at the boundary. That is this issue's arithmetic measured by the code.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    A second fix for this is up as #2001, and it is a no-op on top of #1999: its blocked_bytes(K,N) = ceil(N/128)*ceil(K/64)*8192 equals K*N whenever blocked_fits(K,N) is true, so that allocation changes nothing — the line that fixes the overflow in both PRs is the same blocked_fits() guard. I have said so on both; #1999 is the one to keep (single image_index() so pack and read cannot diverge, and the guard-byte test that demonstrates the overrun). For whoever lands it: the device refuses an unaligned blocked image (npu_engine_i8ctx_inc.h:196), which is the argument for row-major over padding and belongs in the comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions