Repository navigation
Description
What
embed_bytes in npu/pack_import.cpp is a bare const uint8_t*. It has meant two different formats in three weeks, and nothing in the type system objected either time:
- originally it pointed at the pack's own f16 F16/BF16/F32 initializer bytes (decoded with
f16_to_f32); - npu-pack: read a hybrid pack's embedding table from its own graph (embedding.onnx) #1661 made it point at
embed_bf16— the container's bf16 copy — on every path, and left all four consumers decoding F16.
The result was not a crash. It was a plausible number: on a tied Qwen2.5-0.5B-Instruct_rai_1.9.0_npu the head's self-check reported corr 0.885220 / rel_err 0.987 where the same container scores 0.999999 / 0.000700, and the readout called a pack declaring tie_word_embeddings: true "untied". The weights were never wrong — I compared the two containers tensor by tensor and all 291 were identical.
So the diagnostic was confidently wrong about a tied pack, which is the shape that gets a real regression dismissed later ("it says untied, it was always untied").
Why it is a class rather than one bug
embed_bytes is read by four consumers, and each can repeat the mistake independently:
corr()— the head/tie statistic;- the two
sum_abs/sum_refloops (host head, tail head); - the row-shifted control;
- the readout's wording.
#1690 fixes all four and pins the statistic (corr_bf16(bf16) == 1.0 vs corr(f16) == 0.41, with a control that fails when the decode is reverted), but the next reader of that pointer still has nothing telling it which format the bytes are in.
The same class appeared independently the same night in a kernel probe: a readback that silently re-typed an f32 stage as an f16 leaf and produced a plausible number instead of a failure.
Suggested fix
Make the width part of the value rather than part of the comment. Smallest version that would have caught #1661 at compile time:
struct EmbedTable {
const uint8_t* p = nullptr;
enum class Fmt { F16, BF16 } fmt = Fmt::BF16;
int H = 0;
};
// one accessor, no raw pointer arithmetic at call sites
inline double at(const EmbedTable& t, int row, int k);Then a call site that assumes f16 has to say f16, and -Wswitch/type errors cover the rest. A distinct wrapper type per format (Bf16Rows, F16Rows) would be stronger still, at the cost of more churn.
Acceptance
embed_bytes(or its replacement) carries its dtype, and no consumer doesmemcpy(&h, ptr + off, 2)followed by a decode that assumes a width it was not told;- the four consumers compile against it with no implicit conversions;
- one arm showing a width mismatch is a compile error (or a named refusal), not a number.
Citations
- npu-pack: read a hybrid pack's embedding table from its own graph (embedding.onnx) #1661 introduced the current mismatch; npu-pack: the self-check reads the embedding buffer it is actually given (bf16) #1690 fixes and pins it; the 291-tensor comparison and the 0.885220/0.999999 numbers are in npu-pack: the self-check reads the embedding buffer it is actually given (bf16) #1690's thread.
- Found by the reviewer of npu-pack: the self-check reads the embedding buffer it is actually given (bf16) #1690, who put it as "the trap is a pointer plus an implicit dtype… that is a bigger change than this PR and does not belong in it".
Not urgent: nothing is wrong on main once #1690 lands. This is about the next change that repoints the pointer.
Activity
bong-water-water-bong commentedon Nov 9, 2026
Claimed (pi-verify, strixhalo). Checked first: no open PR mentions 1721 —
gh pr list --search 1721is empty — and the four F16 decodes are still onmaintoday, so this is not in flight anywhere.Sized it before claiming: 11
embed_bytesuses onmain, 9 on #1690's head (7bb90b42c, READY), because #1690 already converts the four decodes and addscorr_bf16. So the type change stacks on #1690 and should land after it — there is no point making the same four sites type-safe twice.The plan, taking the issue's "smallest version that would have caught #1661":
and then
corr()takesconst EmbedTable&instead of(const uint8_t*, int H, ...), so the pointer cannot travel without its width. The property I am after is the one #1661 violated: a reader that wants F16 has to sayFmt::F16, and the default is the formatembed_bf16actually holds, so the accident fails visibly rather than plausibly.What I will keep from #1690 so the two do not fight:
corr_bf16(q, bf16, H, rows)stays exported as the test's pin (it becomes a one-line adapter overEmbedTable), and theselfcheck-bf16test arm stays the control —corr(bf16 buffer) == 1.000000vscorr(f16 buffer) == 0.412393, which is what a reverted decode looks like.Not urgent, and I will not rush it into #1719 or ahead of #1690: this is about the next change that repoints the pointer.
npu-pack: the embedding table carries its width, so a reader cannot d…
npu-pack: the embedding table carries its width, so a reader cannot d…
Merge main into 1bit/npu-pack-head-block: three conflicts, and one se…