You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
npu: the i8 host arm indexes a K*N buffer with the blocked tile grid, so an N that is not 128-aligned reads and writes past it (merged in #1888; blocked_fits() has no caller) #1994
host_gemm_i8's weight image is sized K*N but indexed with the blocked tile grid, so any N that is not a multiple of 128 writes and reads past the buffer (merged in #1888; the blocked_fits() guard it needs still has no caller).
Filed because my review of #1888 raised this three times and the PR merged anyway, so the finding should not die with a comment thread. Nothing here is new information — it is the same arithmetic, restated with the line numbers on main:
blocked_index() addresses a tile grid: ceil(N/128) * ceil(K/64) * 8192 bytes. That equals K*N only when N % 128 == 0 and K % 64 == 0. Otherwise the write (the packer uses the same index) and the read both run past the end of a std::vector<int8_t>:
path
K
N
buffer
blocked needs
LM head's last vocabulary block, phi-3-mini (vocab 32064 → last block 32064 % 8192 = 7488)
3072
7488
23,003,136
23,199,744
+196,608 B
a body GEMM at N = 8512
2048
8512
17,432,576
17,563,648
+131,072 B
LM head's last block, vocab 150144 (the measured case)
4096
2688
11,010,048
11,010,048
fits
body at N = 11008 (the measured family)
4096
11008
45,088,768
45,088,768
fits
Why it survived review and measurement: every geometry this arm has been exercised on is aligned. 150144 % 8192 = 2688 = 21 × 128 and 11008 = 86 × 128, so the documented bit-exact results are true on those shapes. The failure needs a vocabulary whose last block is not 128-aligned — 32064 % 8192 = 7488 = 58.5 × 128 is phi-3-mini, i.e. exactly the family this importer is for — or a body width that is not, 8512 = 66.5 × 128.
What it would look like: not a crash. The packer writes past the vector (heap corruption of whatever follows), then the MAC reads the same out-of-range offsets, so logits come back as whatever is in those bytes. For an arm whose entire purpose is to be the reference a kernel family is gated against, silently wrong numbers that still look like an answer are the worst available failure mode — and the gate would then "pass" while the device was right and the reference was wrong.
The fix is one expression at each of the two sites (q4nx_layout::blocked_fits() exists, is tested in tests/npu_blocked_layout_test.cpp, and has no caller in the forward):
constint blocked = q4nx_layout::blocked_fits(K, N) ? 1 : 0; // rowmajor_index is i*ND + j: safe for any geometry
The 174.5 s vs 32.2 s argument for blocked is real (measured in the PR), so refusing or falling back on unaligned geometry keeps the fast path where it is valid and is slower only where the alternative is corruption. An arm with N = 32064, K = 3072 under NPU_INFER_HOST_GEMM=i8 would make this executable — ASan turns it into a hard failure, and the arithmetic above makes it a deterministic one.
Related: #1992 documents this arm (and its LM head) as the reference that can be bit-exact. That is true on aligned shapes and I am not disputing the measurements; the constraint simply needs to be enforced, or stated on the page.
Claimed (cl-1077). Plan: gate the blocked layout on q4nx_layout::blocked_fits(K, N) at the packer and at both MAC sites, so the body GEMM and the LM head's last block use row-major when the tile grid would not fit. Proof: a crafted arm at an unaligned N (e.g. 7488) that is bit-exact against row-major and runs clean under ASan.
Fix is up as #1999 (a579da51), and I built its packers and ran the new test: the unaligned shapes come out row-major with the forced blocked index reaching 23,199,232 of 23,003,136 B (phi-3-mini's last block) and 17,563,136 of 17,432,576 B (N=8512), while 4096x2688 — the measured, aligned head block — stays blocked and lands exactly at the boundary. That is this issue's arithmetic measured by the code.
A second fix for this is up as #2001, and it is a no-op on top of #1999: its blocked_bytes(K,N) = ceil(N/128)*ceil(K/64)*8192 equals K*N whenever blocked_fits(K,N) is true, so that allocation changes nothing — the line that fixes the overflow in both PRs is the same blocked_fits() guard. I have said so on both; #1999 is the one to keep (single image_index() so pack and read cannot diverge, and the guard-byte test that demonstrates the overrun). For whoever lands it: the device refuses an unaligned blocked image (npu_engine_i8ctx_inc.h:196), which is the argument for row-major over padding and belongs in the comment.
Activity
bong-water-water-bong commentedon Nov 10, 2026
Claimed (cl-1077). Plan: gate the blocked layout on
q4nx_layout::blocked_fits(K, N)at the packer and at both MAC sites, so the body GEMM and the LM head's last block use row-major when the tile grid would not fit. Proof: a crafted arm at an unaligned N (e.g. 7488) that is bit-exact against row-major and runs clean under ASan.bong-water-water-bong commentedon Nov 10, 2026
Fix is up as #1999 (
a579da51), and I built its packers and ran the new test: the unaligned shapes come outrow-majorwith the forced blocked index reaching 23,199,232 of 23,003,136 B (phi-3-mini's last block) and 17,563,136 of 17,432,576 B (N=8512), while4096x2688— the measured, aligned head block — staysblockedand lands exactly at the boundary. That is this issue's arithmetic measured by the code.bong-water-water-bong commentedon Nov 10, 2026
A second fix for this is up as #2001, and it is a no-op on top of #1999: its
blocked_bytes(K,N) = ceil(N/128)*ceil(K/64)*8192equalsK*Nwheneverblocked_fits(K,N)is true, so that allocation changes nothing — the line that fixes the overflow in both PRs is the sameblocked_fits()guard. I have said so on both; #1999 is the one to keep (singleimage_index()so pack and read cannot diverge, and the guard-byte test that demonstrates the overrun). For whoever lands it: the device refuses an unaligned blocked image (npu_engine_i8ctx_inc.h:196), which is the argument for row-major over padding and belongs in the comment.