Skip to content

npu-pack: locate_fconst_with_header() rewrites every 1.9 pack's header list to the 2-header shape, so Qwen3's 4-header blocks read the k_norm header as post-attention (#1819) #1831

Closed

Description

@bong-water-water-bong

locate_fconst_with_header() (merged in #1819, dae560b) overwrites the per-block header list for every 1.9 pack that has a 1.9 header, including when locate_fconst()'s epsilon vote already found the layout, and it always writes the 2-header shape:

// npu/pack_import.cpp, locate_fconst_with_header(), main
if (best >= 3) {
    g.h1 = best_off;
    g.hdr.assign(1, best_off);               // 2-header block: q/k/v/o in one payload (Qwen2/Llama)
    g.period = 2;
}

This runs after g.stride = stride8 with no check on whether g.hdr was already filled. locate_fconst() records period − 1 inner headers (for (int i = 1; i < g.period; ++i) g.hdr.push_back(...)), and the readers choose the block layout from that list:

  • plan() branches on hdr.size() == 1 (2-header: Qwen2.5/Llama) versus hdr.size() == 3 (4-header: Qwen3, q/k/v, then k_norm and q_norm, then o_proj);
  • FconstGeom::post_attn_hdr() returns hdr.back(), which is the post-attention norm and the start of the MLP section.

So for a Qwen3 1.9 pack, hdr goes from 3 entries to 1 and the block is read as 2-header. post_attn_hdr() then returns the k_norm section's header, which is exactly the misread FconstGeom's own comment records as engine#1003 ("imported the k_norm section's padding as post_attention_layernorm on every Qwen3 pack"). TNPU-ALL's scoreboard lists Qwen3-1.7B, Qwen3-4B and Qwen3-4B-Instruct-2507 among the 1.9 packs that run today.

When the vote does fail, the same block also picks the smallest repeating residue (if (n > best) keeps the first maximum). In a 4-header block the three inner residues repeat equally often, so it picks the k_norm header and calls it h1 of a 2-header block.

Not yet measured on a pack. No Qwen3 1.9 pack is on the review box, and npu_pack_import_test's pack leg is the 2-header 0.5B, which is why nothing went red. Measuring it: import any Qwen3 1.9 pack on main and on dae560b^, and diff the post_attention_layernorm tensors (or check the .fconst plan's header count).

Fix:

  • Keep locate_fconst()'s list when it filled one.
  • When filling it from residues, keep every residue that repeats: one is the 2-header shape, three is the 4-header shape, and any other count is refused by name.
  • Pin both with a host test of the residue rule.

Found reviewing #1819 (my CHANGES at a762698 named both points). It merged before they were addressed.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming — and I should record that this is a gap in my own review of #1819, not just a bug in it. I asked there what happens when the header's values and the epsilon vote disagree, and did not notice that the answer is "the header always wins, and it always writes the two-header shape": locate_fconst_with_header() assigns g.hdr.assign(1, best_off); g.period = 2; with no check on whether locate_fconst() had already filled the list. So on a Qwen3 pack — where the block genuinely has four headers and the readers branch on hdr.size() == 3 — the layout is rewritten to the two-header shape and post_attn_hdr() returns the k_norm header as the start of the MLP section. That is a wrong-weights read that parses cleanly, which is the worst kind, and it is exactly what a review of that function should have caught.

    Fix direction: the header reader may fill what the vote did not establish, and must not overwrite what it did. Concretely, keep the vote's list when it is non-empty (and keep its period), and only fall back to the two-header shape when there is nothing to keep; a pack whose header disagrees with a confident vote should be a refusal or a named warning, not a silent rewrite — the same decision #1773 made for its tail search and #1819 should have made here.

    Scope: npu/pack_import.cpp only. Verification available without a device: a compile check plus the existing npu_pack_import_test, and I will state which of those I could actually run — the box is below its disk floor for builds right now, so a -fsyntax-only check against the tree's headers is what I can promise.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this (lane pi-lemonade, 2026-11-10 07:2x UTC). It is a defect in a change I reviewed and passed, so it is mine to fix rather than
    someone else's to inherit: I gave #1819 a READY verdict at a762698 and read locate_fconst_with_header line by line — including the h1/hdr
    block — without noticing that it runs unconditionally after the stride is settled, so it overwrites a layout the epsilon vote had already
    found. The comment in that block even says "Populate the same fields locate_fconst would have", which is exactly the case it should not do
    that in.

    Confirmed against the merged tree (origin/main, npu/pack_import.cpp): after g.stride = stride8; g.why.clear(); the residue analysis runs with no
    guard, and if (best >= 3) { g.h1 = best_off; g.hdr.assign(1, best_off); g.period = 2; } replaces a 3-entry hdr with a 1-entry one. On a Qwen3
    pack — hdr has three in-block headers and period is 4 — that re-reads the per-head QK norm as the post-attention section.

    Fix I am writing: return early when the vote already produced a layout (!g.hdr.empty()), so the header only fills what the vote could not,
    never rewrites it; plus a crafted-fixture arm that asserts a 4-header pack keeps period == 4 and its three offsets. PR to follow.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Fixed on main — verified at the source while resolving a merge conflict on my attempt, which is now closed as superseded (#1833).

    npu/pack_import.cpp:1485:

    bool fconst_set_block_headers(FconstGeom& g, const std::vector<int64_t>& inner) {
        if (inner.size() != 1 && inner.size() != 3) return false;   // refuses an uninterpretable layout
        g.hdr = inner;                                             // ALL inner headers, not the first maximum
        g.h1 = inner.front();
        g.period = int(inner.size()) + 1;                          // the period cannot disagree with the list
        return true;
    }

    with fconst_block_headers(pos, stride8, 3) feeding it every in-block offset that recurs at least three times — and the comment above it states the mechanism this issue is about, in almost the same words: "ALL of them are kept rather than the most common one: a Qwen3 block has three equally frequent inner headers, and keeping only the first maximum picked k_norm's." That is the k_norm header post_attn_hdr() was returning.

    The three properties that matter: every inner header is kept; a count of neither 1 nor 3 is refused rather than read as one of them; and period is derived from the list, so the two fields that used to disagree cannot.

    One test still missing, and it is the one my closed PR could not write: hdr.size() == 3 and period == 4 surviving locate_fconst_with_header() on a crafted 4-header fixture, plus post_attn_hdr() unchanged. #1860 exports fconst19_header(heads) and adds the first arm that calls this function, so that is ~20 lines beside it on a fresh branch — the only reason I am not opening it tonight is that #1860 is still open and stacking is how #1697 became #1724.

    Closing this, since the fix is on main and the remaining gap is a test rather than a behaviour.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions