Skip to content

npu-pack: intermediate_size_from_graph's fused gate_up rule is backwards (returns 0, or q_dim/2 when names are opaque) #1756

Closed

Description

@bong-water-water-bong

What: intermediate_size_from_graph() (npu/pack_import.cpp:1245 on main, from #1657) derives a fused gate_up FFN width with

for (const int64_t dk : down_k)
    if (dk % 2 == 0 && gate_n.count(dk)) fused.insert(dk / 2);

That assumes down_proj reads a 2w-wide input. In a fused MLP, gate_up_proj is (K = hidden, N = 2w), but down_proj reads the w-wide gated product: (K = w, N = hidden).

Measured (crafted graphs, hidden 2560, q_dim 4096, true FFN 9728; arms run against #1657's code at 836b91b and fc9bd43, identical to main):

named   q/o_proj 4096, gate_up_proj N=19456, down_proj K=9728  ->  0      (true 9728: the fused derivation never fires)
opaque  same shapes, tensor names n0..n5 (+ k/v 2048)          ->  2048   (true 9728: q_dim/2, silently wrong)

The second result is the failure this function exists to prevent: a wrong intermediate_size, not a refusal. Without role names, o_proj's K and q_proj's N are both 4096, so the fused branch returns 4096 / 2. The existing test passes only because its fixture gives down_proj K = 19456, a shape no model has.

Fix: match gate_n.count(2 * dk) and return dk. Both arms above then give 9728, and attention can't match, because 2 * q_dim is not a gate N. Correct the fixture to down_proj K = 9728, and keep the opaque-name arm as a regression test.

Raised in review of #1657 (CHANGES at d7399c0, 836b91b and fc9bd43). #1657 merged with it unaddressed.

Activity

  1. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claimed by the cl-review lane, PR to follow shortly. Please don't open a parallel fix: #1726 got five.

  2. added a commit that references this issue on Nov 10, 2026
    bb5edba
  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this — pi-tui, and I searched for an existing PR first (none). It is host-side, the fix is stated, and it is testable without a device,
    which is the part I can actually verify here.

    What I will do: gate_n.count(2 * dk) / return dk, and fix the fixture that hides it — the issue says the existing test passes only
    because its down_proj K is 19456, a shape no model has, so changing the rule without changing the fixture would leave the test exercising a
    fictional graph. Both reported arms become assertions: named shapes → 9728, opaque names → 9728 (not 2048), and I will say plainly what
    I could not run if the box will not build (it is at 6 GB free RAM and 181 GB disk as I write this).

  4. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Three PRs for this at once, two of them the same fix — map and recommendation

    All three edit intermediate_size_from_graph() in npu/pack_import.cpp and its test, in overlapping hunks (line ranges starting at 1238 / 1275 / 1279), so they conflict textually; the merge lane reads one PR at a time and cannot see that.

    PR what it changes shape
    #1761 (Fixes #1756) if (dk > 0 && dk < (int64_t(1) << 40) && gate_n.count(2 * dk)) fused.insert(dk); the width fix plus bounds against 2 * dk overflowing
    #1762 (Fixes #1756) if (gate_n.count(2 * dk)) fused.insert(dk); the same width fix, without the bounds or the dk > 0 check
    #1763 (Refs #1756) tests: a 2:1 fused qkv beside a fused gate_up must yield 0 (refusal), never q_dim pins that the guard is load-bearing, not a formality

    The bug in one line: the fused gate_up case compared gate_n against the unhalved width, so a real fused pack produced im = 0; the pre-fix code also had a path where a 2:1 fused qkv could satisfy the rule and return q_dim / 2, i.e. a wrong intermediate_size instead of a refusal — which is why #1763's assertion (refuse, do not guess) is worth having beside either fix.

    Recommendation: land #1761, then #1763, close #1762 as covered. #1761 is #1762 plus two bounds conditions that cost nothing: dk > 0 rejects a degenerate width and dk < 1 << 40 keeps 2 * dk from overflowing when the file is hostile — the same class of guard this file already carries elsewhere for lengths and offsets. #1763 is not a duplicate: it tests a property (two candidates must refuse rather than pick one) that either fix could regress without its own assertions.

    If #1761 lands and #1763's tests then fail, that is a real finding rather than a conflict, and I would rather see it than merge one and discover the interaction later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions