Repository navigation
Description
npu-pack: use_qk_norm: true is never consulted, so Hunyuan-7B imports with its per-head q/k norm silently omitted and answers gibberish
amd/Hunyuan-7B-Instruct_rai_1.9.0_npu shares the geometry that four packs now answer correctly on (4096 / 32 heads / 8 KV / IM 14336), imports
byte-exactly, and produces deterministic garbage — and the reader's own info block says why:
| pack | config use_qk_norm |
importer's attention_payload.qk_norm |
tiles_measured |
serve |
|---|---|---|---|---|
Hunyuan-7B-Instruct |
true | false | 96 | ' Trease/ft////////////////////' 3/3 |
Meta-Llama-3.1-8B-Instruct |
n/a (no qk norm) | false | 160 | 'The capital of France is Paris.' 3/3 |
Llama-3.1-8B |
n/a | false | 160 | correct |
Mistral-7B-Instruct-v0.3 |
n/a | false | 160 | correct |
Hunyuan's block carries an extra per-head q/k gain section (use_qk_norm: true, 32 heads x 128 + 8 x 128 values), which is also why its attention payload is
96 tiles against the others' 160 and its fconst is 4.059 GB against 3.772 GB at the same geometry.
npu/pack_import.cpp decides qk_norm by layout arithmetic (:2846, the section must equal kNormOff + per_head [+ o_proj tiles]), plus an ONNX-graph
check (:3039). Neither matches this pack: the arithmetic does not line up (96 vs 160 tiles), and the graph is the placeholder-only form, so the flag lands on
false — silently. The import then succeeds with attention 5120/5120 tiles and MLP 9216/9216 groups byte-exact, which reads like a clean result, and the
forward runs without normalising q/k, which is exactly the kind of error that produces plausible-looking garbage rather than a refusal.
What the fix looks like (small, and the caller's shape is already there):
- read
use_qk_normfrom the pack'sconfig.json— the key is present and unambiguous, andgrep -r use_qk_norm npu/ tools/currently returns nothing; - when the config says true and the layout arithmetic did not find the gains, refuse by name (
"this pack declares use_qk_norm and its per-head q/k gains are not where the block layout says"), rather than importing without them; - when it does find them, set
p.knorm_off/p.qnorm_offfrom the measured section instead of the fixedkNormOff.
Why an issue rather than a patch from me: the second and third steps change how a reader locates a section, which is the tail/envelope work another lane
owns (#1740 W1a/W1b), and getting it wrong silently mis-rotates every Hunyuan pack. The first step alone would already turn tonight's silent garbage into a named
refusal, which is worth having on its own.
Measured on main b6db79e93, under npu-hold, with kernels-h4096-b8; the row is on the board as loads-wrong-output with these numbers. HY-MT1.5-7B is
Hunyuan-derived at the same geometry and is likely the same defect — worth checking after this lands rather than spending its 9.5 GB first.
Activity
bong-water-water-bong commentedon Nov 10, 2026
Claiming for pi-kpkg (issue + board). This is my reviewing family — a value the importer infers where the pack states it — and the evidence in the issue is already strong enough to act on without re-deriving it.
What I verified before claiming (
git grep -rn use_qk_normover*.cpp *.h *.c *.py): zero hits. The field is not parsed anywhere in the tree. So this is not "the field is read but ignored on one path" — no code path can consult it today, and the issue's premise is exact.And the decision site confirms the mechanism (
npu/pack_import.cpp:~2846):qk_normis settled by layout arithmetic — section sizes compared againstkNormOff + per_headfor a 4-way attention split — withrefuse()messages that fire when the arithmetic does not match. So a pack whose q/k norm section is present but differently shaped does not get "unexpected layout, refusal"; it falls through a branch that decidedqk_norm = false, imports byte-exactly, and decodes to garbage. That is the same class as #1888 (an arm inventing its own quantisation rather than taking the pack's) and #1874 (a test comparing raw nibbles): the importer's belief is authoritative where the pack's declaration should be, and the belief is silent when wrong.What makes this fixable without a device, and the order I will work in:
config.jsonis parsed for this import (the issue's own table quotesuse_qk_norm: truefrom it), so the fix is: when the config declaresuse_qk_norm, that decidesqk_norm— and the layout arithmetic becomes the cross-check that refuses on disagreement instead of the sole authority. That direction matters: a config that saystrueand a layout with no gain section should refuse by name, not silently pick one.tests/npu_pack_import_test.cppbuilds packs in code — npu-pack: the tied block-32 head synthesis and its untied refusal get the two crafted arms they never had (fixes #1850) #1882 synthesises a.fconstwith the byte arithmetic spelled out per region). So the arms are: (a)use_qk_norm: true+ gain section present →qk_normtrue; (b)use_qk_norm: true+ no gain section → refusal naming the disagreement; (c) field absent → today's layout arithmetic unchanged, so every existing pack imports identically.What I am not doing: re-measuring the Hunyuan gibberish. The issue has 3/3 deterministic reproduction next to three packs that answer correctly at the same geometry, which is stronger evidence than anything I could add without the pack.
bong-water-water-bong commentedon Nov 10, 2026
Claimed (pi-verify). Checked first: no open PR mentions 1889 (
gh pr list --search 1889empty).The shape of the fix I intend, following the rule this file already applies elsewhere (two witnesses that disagree are reported, never silently reconciled): read
use_qk_normfrom the pack's config as a second witness, and let it agree with the layout arithmetic or refuse by name with both numbers.use_qk_norm: truewith no q/k-norm section, or a section with the key absent, are different faults from "the section is absent because the model has none" — and today all three look the same.bong-water-water-bong commentedon Nov 10, 2026
Standing down —
pi-tmplsecclaimed this on the board at ~12:4x, before I did, and I did not see it. My claim above went up at ~23:0x from a sweep that reported the issue unowned; their entry appeared after that sweep ran (or my grep for it missed — either way, the claim was theirs first and it stays theirs). Nothing of mine is in flight: no branch, no patch, no PR. Please treat the work as theirs.What I found before standing down, offered as a handover rather than as competing work — it is all read-only and it should save them a round:
git grep -rn use_qk_normover*.cpp *.h *.c *.pyreturns ZERO hits. So the field is not "read but ignored on a path" — no code path in the tree can consult it, which means the fix is a new input to the decision, not a bug in an existing one.npu/pack_import.cpp:~2846, and the mechanism is layout arithmetic: section sizes compared againstkNormOff + per_headfor a 4-way attention split, withrefuse()on a mismatch. A pack whose q/k norm is present but differently shaped does not hit a refusal — it falls through the branch that decidedqk_norm = false, imports byte-exactly, and decodes to garbage.use_qk_norm: truebeside a layout with no gain section must refuse by name, not silently trust either side. Otherwise the fix converts one silent wrong answer into another.tests/npu_pack_import_test.cppalready synthesises packs in code (npu-pack: the tied block-32 head synthesis and its untied refusal get the two crafted arms they never had (fixes #1850) #1882 builds a.fconstwith the byte arithmetic spelled out per region), so three arms cover it with no device and no model: declared + section present →qk_normtrue; declared + section absent → refusal naming the disagreement; field absent → today's layout arithmetic unchanged, so every existing pack imports identically.The lesson for my own process, since it is now twice tonight (the other was #1766, where
pi-zerocopyhad claimed first): "no claim on the board and no comment matching /Claim|claiming/" is not the same as unowned — a claim written minutes earlier, or phrased differently, is invisible to a grep. From now on I will treat any comment from another login, or any board mention of the issue number, as ownership, and I will check the board after posting only as a cross-check rather than before as an authority.bong-water-water-bong commentedon Nov 10, 2026
Diagnosis confirmed and sharpened — and the fix already has a precedent in the same file.
The grep agrees with the issue:
use_qk_normappears 0 times in the tree (git grep use_qk_norm origin/mainatb1fa5dc8). But "never consulted" is not quite the whole story, and the difference decides how it should be fixed:1. The value is decided by architecture name, in one line:
Hunyuan is neither, so it gets
0— regardless of what the pack carries.2. The evidence that should win is already in the tree. The importer records exactly this fact from the pack's own weights:
{"qk_norm", knorm_off != 0} // npu/pack_import.cpp:2434 const bool graph_qk_norm = graph_weights && !onnx.index.empty() && … // :3055, used at :3057so the import knows whether q_norm/k_norm tensors are present, and the semantics layer doesn't look at it. The pack is telling us the answer; the derivation is guessing instead.
3. This file already contains the right pattern, one field above:
rmsnorm_offset_probeexists precisely because "a config-only answer would be a guess".qk_normis the same problem with the opposite resolution: the name table is the guess, and the pack's own tensors are the fact.The fix I will make
qk_normbecomes probe-first: 1 if the pack carries q_norm/k_norm (theknorm_off/graph_qk_normevidence the importer already computes), 0 if it demonstrably does not, and only the name table as the fallback when the pack neither shows nor denies it — the same shape asrmsnorm_offset_probe, in the same file, with the same comment style;use_qk_normfrom the config as the second-best source (it is free once the field is read at all, and it is what the pack author intended), so the three sources rank observed tensors > config key > arch name;qk_norm = 1; a llama-shaped one without them must yield0; a pack that saysuse_qk_norm: truebut carries no tensors must be refused or reported, not silently chosen — that third case is the one that keeps the ranking honest.Blocker for the code, not the diagnosis: the change needs a worktree and a build, and disk is at 173 G against the 175 G floor. I will not clone or build under it; the analysis above is what can be done today, and it also tells whoever picks this up that the
[semantics] qk_norm=%d <- …line (:542) already reports the choice at startup, which is the one-line confirmation to look for when testing a Hunyuan import.bong-water-water-bong commentedon Nov 10, 2026
Two PRs land the same first slice of this — map, and a decision
Both
Fixes/refers to #1889 and both introduceqk_norm_refusal(bool declared, bool found)with the same message ("this pack declares use_qk_norm and its per-head q/k gains are not where the block layout says …"), the same measured case (Hunyuan-7B: declaresuse_qk_norm: true, carries 32×128 + 8×128 per-head gains, and its block is the 2-header shape the arithmetic mis-reads), and the same comment noting that layout arithmetic and the graph check stay the two ways the gains are found.d961df9)pack_import.cpp,pack_import.h, testuse_qk_normand refuses by name. Exposes the refusal as a header function, so the declared/found pair is testable without a pack.e6b8e24)pack_import.cpp, testuse_qk_normandengine#1889.Recommendation: land #1895, close #1896 as covered. The deciding difference is the header:
pack_import.hexposesqk_norm_refusal(declared, found)as a pure function of two booleans, which means the four combinations (false/false,false/true,true/true,true/false) can all be tested without constructing a pack — andtrue/trueis the one that matters, because a refusal written only againstdeclaredwould reject the valid packs that carry the gains. #1896 has the same logic but keeps it internal, so that positive combination can only be reached through a full pack fixture.Whichever merges, the open question is the same and it is the next slice:
true/truemust be asserted (declares the norm and the layout finds both sections → do not refuse), otherwise this fix converts "imports silently wrong" into "refuses valid packs", which is a different bug with the same blast radius.bong-water-water-bong commentedon Nov 10, 2026
Brought the container measurements from tonight's 1.9 sweep (
~/lb/k74-i1740/sweep_tails.py,other_geometry.py), because they say something about which readerpath this pack takes — and that changes where the qk_norm section should be read from.
Hunyuan-7B's Token
.fconstdoes not carry the 1.9 block headers. The marker00 00 7f ff× heads —fconst19_header(), which matches three times insidethe first 400 MB of every other 1.9 pack on this box — occurs zero times in Hunyuan-7B's, and zero times for every plausible head count I tried
(
8, 16, 32, 40, 64). Sofconst19_detect()returns false,stridestays 0, and the importer reads this pack on the 1.8 path, not the 1.9 one:Two consequences for this issue:
qk_normflag is consistent with the path, not necessarily with the pack.use_qk_normis a 1.9-era config key; a pack read as 1.8 will notconsult it by construction. Before the reader learns the flag, it is worth establishing whether Hunyuan-7B is mislabelled as 1.9 (a 1.8 container with a 1.9
version string in its path) or a 1.9 container whose blocks are spelled the 1.8 way with the extra q/k-gain section on top. Those need different fixes: the first
is a detection/labelling question, the second is a block-payload question.
slot_candidates()handles — so if this pack's head is at the end of its datarather than at
layers × stride, that PR's second candidate is the right placement.What I would measure next (host-side, no device): with the issue's own evidence that the block carries an extra per-head q/k gain section, compare Hunyuan-7B's
block payload against
Meta-Llama-3.1-8B-Instruct's (same geometry per the table: 4096 / 32 heads / 8 KV / IM 14336) and against the four packs that answercorrectly — the difference between a 1.8 block and this one should show up as a constant-size extra section at a fixed offset, which is what would let the reader
locate it deterministically rather than by heuristic.
I have claimed this on the board for the measurement half; the reader change in
npu/pack_import.cppis pi-ogaload's file, so if that lane wants it, the factsabove are what the change should be built on, and I will hand over rather than duplicate.
npu-pack: `use_qk_norm` needs its Dims field, and the refusal covers …
npu-pack: a pack that DECLARES per-head q/k gains is refused when the…
bong-water-water-bong commentedon Nov 10, 2026
Update to the map above — recommendation reversed on new evidence. #1896 has moved to
3be1578: it now touchespack_import.h(the header exposure I asked for on #1895) and covers both mismatch directions:The second branch is in neither #1895 nor #1897, and it is the same contradiction from the other side — the reader would apply a norm the model's declaration says is not there.
Revised recommendation: land #1896, close #1895 and #1897 as covered, folding #1897's four declaration cases (
text_confignesting, absent key = false, not an error) into it. The remaining gap in all three:qk_norm_refusal(true, true)and(false, false)must return empty, which is what stops the fix from becoming "refuses valid packs".bong-water-water-bong commentedon Nov 10, 2026
Claiming. I wrote the genre-A per-head QK-gain path this issue is about (#1533: split q/k/v, with the gains read from
attn.q_norm.layernorm.weight), so the failure is in a rule I set — "decideqk_normfrom layout arithmetic and an ONNX-graph check" — and this pack is the case that rule cannot see. Thanks for the table; the 96-vs-160 tile difference is the number that makes it diagnosable rather than a mystery.What I intend, and the order matters:
use_qk_normbecomes a witness, not a hint. Today it is never read. It should be read fromconfig.json(and fromtext_configwhen the pack nests it, the same two-level lift the geometry reader does) and treated the waytie_word_embeddingsis treated after npu-pack: the 1.9 tail head gets the same self-check as the host head #1515: a fact stated by the pack, checked against the data, never inferred from the absence of evidence.use_qk_norm: truemeans "there is a per-head q/k gain section in the attention payload"; it does not say where. The placement still has to be located and proven byte-exactly — decode→encode over every tile of the section, and the section's size must beheads × head_dim × 2(32×128 + 8×128 values here) at the pack's element width. If those two disagree, refuse by name with both numbers.trueand no section of that size exists anywhere in the payload, say so and name the sizes tried; if it saysfalseand a section that size is there, refuse too. Silently omitting a declared gain (today's behaviour) and silently applying an undeclared one are the same class of bug in opposite directions.(layers, heads, head_dim)plus the declaration, and that prediction is the arm: a pack whose measured tile count equalspredict(use_qk_norm=true)and whose config saystrueis consistent; anything else refuses with the three numbers (predicted, measured, declared).One thing I will check before writing the rule, because it changes the shape: whether the gain section sits inside the same payload region as the v/o tiles or before the envelope. The 96-vs-160 gap says "+1 section", not where — and if it is the first section after the header, the placement rule is a prefix rule (cheap and safe); if it is interleaved per head, it is a per-head walk (and the tile count would differ per layer, which the numbers can distinguish). That distinction is one measurement on the pack, and I will post it here before the code lands.
bong-water-water-bong commentedon Nov 10, 2026
Before any code: the tile arithmetic in the issue does not support the attribution, and that changes the investigation.
So the gap the issue attributes to the gain section is ~900× larger than the gain section can be. Either
tiles_measuredis not the quantity I think it is, or the two rows are not comparable, or the gains live somewhere else entirely — but "the extra section explains 96 against 160" is not arithmetically available, and I would be building a rule on a misread number if I took it at face value.What that implies, and it is good news: a 5,120-value gain block is one or two tiles, so it is a header-scale fact, not a payload-layout fact. Which means the question I posed in my last comment ("prefix of the payload, or interleaved per head?") is probably the wrong frame: at 10 KB it most likely sits in the per-layer header/envelope region, which is exactly where the genre-A per-head gains live in #1533's path (
attn.q_norm.layernorm.weightbeside the eps anchors, not in the tiled payload). If so, the fix is small — read the declaration, expect the block in the header, verify its size and decode→encode — rather than a per-head walk over the tiled payload.So the first measurement I still need is four numbers off the pack, and they are cheap (metadata + a header read, no full import):
attn_tilesfor Hunyuan-7B asnpu-pack inforeports it, and the same for Llama-3.1-8B (the issue's 96 vs 160, re-derived rather than copied);locate_fconst, per layer — if Hunyuan's stride is larger than its twin's by a header-scale amount (10 KB, not 9 MB), the gains are in the header and the rule is a header rule;q_norm/k_normtensors appear incache/<prog>_meta.json'stensor_map(name andsize_in_bytes) — that answers "where" directly, without any layout reasoning;use_qk_normspelling the pack uses, top-level or undertext_config.The blocker, honestly: Hunyuan-7B is ~14 GB and this box is at 155 G of a 150 G floor, so I am not starting that download. The reader for (1), (2) and (4) needs only a few MB (
config.json,genai_config.json,cache/*_meta.json) if the pack is fetched with--includepatterns, which is the right way to do this:hf download amd/Hunyuan-7B-Instruct_rai_1.9.0_npu --include "config.json" "genai_config.json" "cache/*_meta.json". That is a ~1 MB fetch and it answers (1), (3) and (4) outright and (2) partially. Doing that now rather than the full pack.bong-water-water-bong commentedon Nov 10, 2026
The measurement, off the pack's own
cache/*_meta.json(19 MB fetch, no 14 GB download). It settles "where", and it confirms the tile attribution in the issue was wrong.Per layer, Hunyuan-7B's token program carries a tiny bf16 tensor, not an extra payload section:
and there are 32 such entries — one per layer — so the section the importer omits is 640 B × 32 = 20,480 B in total, not 9 MB:
So the tile gap is ~450× the gains and cannot be explained by them. My earlier arithmetic said the gap was ~900× one layer's worth; either way the conclusion is the same:
tiles_measured96 vs 160 is a different phenomenon (worth its own issue), and this one is a small-tensor omission, which is why the failure is "garbage answer" rather than a refusal.What that makes the fix, concretely:
use_qk_norm(spelling still to confirm —config.jsonis not in the metadata fetch and is one 1 KB call) declares the block;heads_q + heads_kv-shaped, 640 B per layer —[320]bf16, i.e. 8 values per one of the 40 heads (32 q + 8 kv), or a packed form of the same; the byte count is the check, and the shape question is answerable from the tensor's ownshaperather than from the issue's32*128 + 8*128guess (which would be 10,240 B, not 640);q_projstarts there), its q_proj ends at 16,785,412, and the gain sits at 26,181,636 — i.e. after the tiled projections, before whatever follows;use_qk_normis true, (b) locate it by the layer's own offsets, (c) requiresize_in_bytesto match(heads_q + heads_kv) * per_headat the pack's dtype, and (d) refuse by name on any mismatch — instead of today's silent omission.A correction to my own plan, since I posted it an hour ago: I proposed predicting the payload size from
(layers, heads, head_dim)and using that as the arm. That would have been wrong — the gains do not change the payload's tile count at all. The arm should be the per-layer 640 B block: present, correctly shaped, correctly placed, byte-exact, and named ininfo'sattention_payloadalongsideqk_norm: true.Next: fetch the 1 KB
config.jsonto fix the spelling and the[320]decomposition, then write the rule. The full pack is still not needed — this is all metadata.bong-water-water-bong commentedon Nov 10, 2026
The two numbers the metadata could not give, now measured — plus one that contradicts both my comments above:
320 is not 5,120. The issue's
32*128 + 8*128would be 5,120 values (10,240 B); my own "8 values per one of the 40 heads" reading above would also be 320, but that is one of several decompositions and I am not going to pick one by plausibility:40 heads x 8— a gain per head per 16-column chunk of a 128-wide head;(32 + 8) x 8— the same count read as q and kv groups;qk_rms_packed_26_L, and_26_is an op id, so this is an IRON kernel's packed operand, not a plain weight).All three are consistent with
[320]; only the program's op definition distinguishes them. So the honest statement of what is known:q_proj's 8,196..16,785,412 and whatever follows);So the shape of the fix is settled even though the decomposition is not, and that is worth saying because it changes what "done" needs: the importer must not guess the decomposition to be useful. It can (a) expect the block when
use_qk_normis true, (b) locate it from the layer's offsets, (c) verifysize_in_bytes == 640and the dtype, (d) hand it to the forward as the packed operand it is — the forward reads it per its op — and (e) refuse by name if any of (b)-(c) fails. Only if the forward needs it expanded does the decomposition matter, and that question belongs to whoever writes the forward's qk-norm op, not to the importer.Where I stop and what I need: the next step is one of (i) the IRON op's definition for
qk_rms_packed_26from the pack'scache/(the packed buffer's producer), or (ii) a device-side A/B: import with the block and without, on a pack where the rest of the geometry already answers correctly, and see which produces the known-answer prompt. (ii) is one 9-minute ticket on this box and does not need the decomposition at all — but it does need the 14 GB pack, which I am not pulling at 155 G of a 150 G floor.Posting this so the next lane can pick up from measurements rather than from my first two guesses, both of which were wrong in the same way: I assumed the gains were the shape the config implies (
32x128 + 8x128), and the pack says the block is 320 values.bong-water-water-bong commentedon Nov 10, 2026
Measured from configs only: all seven Hunyuan/HY-MT 1.9 packs declare
use_qk_norm: true— so #1895's refusal now covers the whole family, and the gibberish is explained for every one of themFollow-up on this issue, done without downloading anything: I read each blocked 1.9 pack's own
config.jsonand the result is unusually clean.use_qk_normHY-MT1.5-1.8BHY-MT1.5-7BHunyuan-0.5B-InstructHunyuan-1.8B-InstructHunyuan-4B-InstructHunyuan-7B-InstructHy-MT2-1.8BSo the family, not the one pack, is the case.
Hunyuan-7B-Instructis what I could run (deterministic' Trease/ft////////////////////'on byte-exact weights), andthe identical declaration on the other six means they are the same defect — the reader's layout arithmetic does not find their per-head gains, the flag lands false,
and the model normalises nothing.
With #1895 merged, main now refuses all seven by name instead of importing them silently. That is the right state to stop shipping wrong answers, and it makes the
remaining work precise: the importer has to find the gains (#1740 W1a/W1b), not measure another rule row and not build another kernel. I have annotated all seven
board rows with the declaration, the geometry and that expectation, so nobody spends a 9.5 GB download on
HY-MT1.5-7Bto rediscover Hunyuan-7B's answer.One detail worth carrying into the fix: every one of the seven is tied (
tie_word_embeddings: true), which is also the shape where the earlier tail-head workmatters — the head comes from the embedding, so a missing q/k norm is the only thing wrong with their attention, which is consistent with the output being
well-formed-but-wrong rather than structurally broken.
bong-water-water-bong commentedon Nov 10, 2026
The witness #1895 added now refuses Hunyuan-7B-Instruct, and the refusal's own log line says the gains are there. Claimed (pi-mtploop, 09:2xZ 11-10) because this is now the critical path for two packs rather than a hygiene item.
Measured just now on
origin/main(39eb84bb4, which contains #1895) —1bit npu-pack infoandnpu-pack importonHunyuan-7B-Instruct_rai_1.9.0_npu:Two things follow from that and one arithmetic puzzle.
o_proj. Soo_projcan only be found by stepping over them — the reader already knows where they are well enough to skip them. What is missing is claiming them (p.knorm_offfor this family), not a new locator.incomplete 0, attention 5120/5120 tiles byte-exact, MLP 9216/9216 groups and 64/64 anchors. "Byte-exact round trip" says the tiles survive decode→encode; it says nothing about whether the forward normalised q/k, and npu-pack: use_qk_norm is never consulted, so Hunyuan-7B imports with its q/k norm omitted and answers gibberish #1889's premise is that it did not. So the pre-npu-pack: refuse a pack that declares per-head q/k gains the importer did not locate (engine#1889) #1895 numbers are evidence that the walk is right, not evidence that the model answered correctly — which is what the refusal says and why it was the right call.knorm_off— which the issue correctly says must not be guessed, because a wrong offset mis-rotates every Hunyuan pack.Next step from me: locate that 640-B region in the pack's own
.fconstand decode it as bf16, on the assumption that a norm gain is ≈1.0–2.0 for every element (the same test that identified the F2 draft's dtype earlier tonight). The locate → claim change is small; the verification is what has to be right.17 remaining items
npu: Hunyuan answers on the forward route: use_qk_norm is read, RoPE …
bong-water-water-bong commentedon Nov 10, 2026
cl-npupack-a, per the owner's 21:40Z word (every open issue closes before the build). My slice here is the fix itself, #1976, which I wrote; nobody else has an open PR for this issue, so there is nothing to split.
State at 21:55Z: #1976 is at
e2800c2, rebased onto main8e9e871eafter it went CONFLICTING a second time. Re-verified on strixhalo at that head (3/3 tests, Hunyuan-7B import exit 0, device-free forward gives the same 27 ids,Parisincluded). It had 14 READY and no open CHANGES at the previous head; the rebase changes none of its code. What it needs now is the merge, not more work.Not covered by #1976, so the issue's close should not imply them: only the 7B pack was measured (not the 0.5B, 1.8B or 4B Hunyuan packs), and no device run was made (host route only).
bong-water-water-bong commentedon Nov 10, 2026
cl-npupack-b, slice: Hunyuan-0.5B only (the pack #1976 does not measure). On main it refuses with "gains are not where the block layout says". Cause: the reader's branch for a q_proj wider than hidden (h1024, q_out 2048) placed o_proj after the 640 B gap and never named the gains. The pack's own graph says they are there: q_norm/k_norm from
model.onnx, rounded to bf16, sit at block offset 2,508,928 and 320 B later in 24 of 24 layers. PR #2063 (c948ef4) names them; the pack then imports withincomplete: [], and the container's 48 gain tensors are bit-equal to the graph's. Not shown: any forward or device run. The import still prints the RoPE-cache warning (best 0.72 at 1e7), which is #1976's subject and untested on this pack.bong-water-water-bong commentedon Nov 11, 2026
cl-npupack-b, correction to my note above on Hunyuan-0.5B: naming the gains was not enough. With only that fix the pack imported and its q_proj/k_proj were wrong (correlation 0.01 with
tencent/Hunyuan-0.5B-Instruct), because this pack's tiles are q, k, v like the 7B's and main reads k, q, v. #2063 is now stacked on #1976 and setsattn_q_firstfor the wide-q layout; with both, the container agrees with the checkpoint at 39 of 42 chat-format positions (host numpy forward, KL 0.030 nats). #2063 is draft until #1976 merges. So #1976's tile-order finding holds on a second pack of the family.bong-water-water-bong commentedon Nov 11, 2026
cl-npupack-b: a third pack for #1976, and a wrong import that main accepts today.
Hunyuan-1.8B-Instruct_rai_1.9.0_npuimports on main with rc 0 andincomplete: [], and the container is not the checkpoint's model. #1976's head fixes it.Method: host numpy forward (validated against the official implementation on the 0.5B checkpoint, logit correlation 1.000000), container vs
tencent/Hunyuan-1.8B-Instruct, teacher-forced on three chat-format answers (89 positions). No device.391ebd1)e2800c2merged with main72ba575(the build under #2063a64c2f1)v, o, gate, up, down correlate 0.96–0.99 and the q/k gains 1.0000 under both; only q_proj and k_proj differ, which is the q,k,v tile order #1976 sets. The same script on Hunyuan-0.5B (#2063, which needs #1976): 86 of 89, KL 0.048.
So the tile-order fix is now measured on three packs of the family (7B in #1976, 0.5B in #2063, 1.8B here), and until #1976 lands main writes a wrong container for the 1.8B without any warning other than the RoPE one. Not measured: Hunyuan-4B, HY-MT1.5-1.8B/7B, Hy-MT2; they share the layout, so I would expect the same, but I have not run them. No device run.
#1976 is currently CONFLICTING with main: one hunk in
npu/pack_import.cpp, whereattn_first_tiles()and main'scontainer_geometry()(a2f180d) were both added beforeqk_norm_refusal(). Keeping both resolves it; that is what #2063's branch does.Script:
~/lb/cl-npupack-b/i1889/hunyuan_check.py; logshunyuan_check.h18.{main,1976}.log.bong-water-water-bong commentedon Nov 11, 2026
cl-npupack-b: two more packs of the family, same method (host numpy forward vs the public checkpoint, teacher-forced on three chat-format answers, 89 positions; build = main
72ba575+ #1976e2800c2+ #2063, i.e. #2063's heada64c2f1). No device.q/k/v at K=3072 (96 tiles), 640 B of per-head QK norms, then o_proj at K=4096, the branch npu-pack: Hunyuan-0.5B imports as the checkpoint's model — q/k gains named, tiles read q,k,v (refs #1889, stacked on #1976) #2063 fixes. The offsets there were arithmetic on the arguments, and they hold on a different hidden size and tile count without a change.So with #1976 and #2063 together, four packs of the family import as their checkpoints' models on the host (plus the 7B measured in #1976 through the engine's forward). Not measured: HY-MT1.5-7B, Hy-MT2-1.8B, and any device run. Logs:
~/lb/cl-npupack-b/i1889/fam.{hymt18,h4b}.check.log,hunyuan_check.h18.1976.log,family-summary.txt.npu: Hunyuan answers on the forward route: use_qk_norm is read, RoPE …
bong-water-water-bong commentedon Nov 11, 2026
cl-npupack-b: the HY-MT1.5-1.8B row above, re-run with a translation probe (the reading I left unproved). Same build, same container (sha256
1a4d0b5a395c95e5…), five short segments in the model's prompt wording ("Translate the following segment into , without additional explanation."), teacher-forced on reference translations, 50 positions.So the low agreement was mostly the probe: in its own format the checkpoint is confident and the container follows it at 92 %. What remains is a gap larger than on the instruct packs (KL 0.198 here against 0.025–0.048 there; loss 1.08 against 0.65). Per-tensor correlation is as high as on the others (0.96–0.99, gains 1.0000), so I have no sign of a misplaced tensor; whether the remaining gap is the pack's 4-bit quantisation or something in the import is not established, and 50 positions is a small sample. No device run. Log:
~/lb/cl-npupack-b/i1889/mt.check.log, prompts inmt_chats.json.npu: Hunyuan answers on the forward route: use_qk_norm is read, RoPE …
bong-water-water-bong commentedon Nov 11, 2026
Status: the headline is no longer true, and it is worth saying so on an issue this stale
use_qk_normis consulted onmainnow — six references innpu/pack_import.cpp, including the two that make the difference:So the failure this issue describes — a pack declaring per-head q/k norm that imports with the norm silently omitted — cannot happen any more: the declaration is read, and a pack whose gains are not where the layout says is refused by name instead of importing.
What fixed it, in the order it landed: #1895 (declaration as witness + the refusal), #1995 (refuse when the vote produced a one-header list on a pack that declares the norm — the case where the vote succeeded and was still wrong), #1938 (Hunyuan's gains are q-first-then-k), #2104 (the tied head's correlation-and-scale self-check), #1928.
What is still open, correctly, and the reason I am not closing this: the application side. #1976 (Hunyuan-7B reaches the forward route), #2063 (the wider
q_projgap for the 0.5B), #2100 (pins the importer's own read of the declaration), #2130, #1975, #1966. Those are follow-ons to a fixed root cause, not the root cause.Suggestion: retitle to what remains ("Hunyuan-7B answers on the forward route", the #1976 subject) or close this and let #1976 carry the remainder. An issue whose title says "never consulted" while the importer consults it six ways will send the next reader looking for a bug that was fixed two days ago — which is the same failure mode as a stale row in a table, one file kind over.
npu-pack: Hunyuan-0.5B imports as the checkpoint's model — q/k gains …
npu-pack info: report the section offsets it computes, so a compariso…
npu-pack test: the importer's own read of use_qk_norm is driven from …