Skip to content

npu-pack: a pointer plus an implicit dtype is how #1661 repointed embed_bytes under four readers without a compile error #1721

Closed

Description

@bong-water-water-bong

What

embed_bytes in npu/pack_import.cpp is a bare const uint8_t*. It has meant two different formats in three weeks, and nothing in the type system objected either time:

The result was not a crash. It was a plausible number: on a tied Qwen2.5-0.5B-Instruct_rai_1.9.0_npu the head's self-check reported corr 0.885220 / rel_err 0.987 where the same container scores 0.999999 / 0.000700, and the readout called a pack declaring tie_word_embeddings: true "untied". The weights were never wrong — I compared the two containers tensor by tensor and all 291 were identical.

So the diagnostic was confidently wrong about a tied pack, which is the shape that gets a real regression dismissed later ("it says untied, it was always untied").

Why it is a class rather than one bug

embed_bytes is read by four consumers, and each can repeat the mistake independently:

  1. corr() — the head/tie statistic;
  2. the two sum_abs/sum_ref loops (host head, tail head);
  3. the row-shifted control;
  4. the readout's wording.

#1690 fixes all four and pins the statistic (corr_bf16(bf16) == 1.0 vs corr(f16) == 0.41, with a control that fails when the decode is reverted), but the next reader of that pointer still has nothing telling it which format the bytes are in.

The same class appeared independently the same night in a kernel probe: a readback that silently re-typed an f32 stage as an f16 leaf and produced a plausible number instead of a failure.

Suggested fix

Make the width part of the value rather than part of the comment. Smallest version that would have caught #1661 at compile time:

struct EmbedTable {
    const uint8_t* p = nullptr;
    enum class Fmt { F16, BF16 } fmt = Fmt::BF16;
    int H = 0;
};
// one accessor, no raw pointer arithmetic at call sites
inline double at(const EmbedTable& t, int row, int k);

Then a call site that assumes f16 has to say f16, and -Wswitch/type errors cover the rest. A distinct wrapper type per format (Bf16Rows, F16Rows) would be stronger still, at the cost of more churn.

Acceptance

  • embed_bytes (or its replacement) carries its dtype, and no consumer does memcpy(&h, ptr + off, 2) followed by a decode that assumes a width it was not told;
  • the four consumers compile against it with no implicit conversions;
  • one arm showing a width mismatch is a compile error (or a named refusal), not a number.

Citations

Not urgent: nothing is wrong on main once #1690 lands. This is about the next change that repoints the pointer.

Activity

  1. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claimed (pi-verify, strixhalo). Checked first: no open PR mentions 1721 — gh pr list --search 1721 is empty — and the four F16 decodes are still on main today, so this is not in flight anywhere.

    Sized it before claiming: 11 embed_bytes uses on main, 9 on #1690's head (7bb90b42c, READY), because #1690 already converts the four decodes and adds corr_bf16. So the type change stacks on #1690 and should land after it — there is no point making the same four sites type-safe twice.

    The plan, taking the issue's "smallest version that would have caught #1661":

    // pack_import.h
    struct EmbedTable {
        const uint8_t* p = nullptr;
        enum class Fmt { F16, BF16 } fmt = Fmt::BF16;   // default is what the importer actually holds
        int H = 0;
    };
    double embed_at(const EmbedTable& t, int row, int k);   // the ONLY decode, width taken from t.fmt

    and then corr() takes const EmbedTable& instead of (const uint8_t*, int H, ...), so the pointer cannot travel without its width. The property I am after is the one #1661 violated: a reader that wants F16 has to say Fmt::F16, and the default is the format embed_bf16 actually holds, so the accident fails visibly rather than plausibly.

    What I will keep from #1690 so the two do not fight: corr_bf16(q, bf16, H, rows) stays exported as the test's pin (it becomes a one-line adapter over EmbedTable), and the selfcheck-bf16 test arm stays the control — corr(bf16 buffer) == 1.000000 vs corr(f16 buffer) == 0.412393, which is what a reverted decode looks like.

    Not urgent, and I will not rush it into #1719 or ahead of #1690: this is about the next change that repoints the pointer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions