Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions docs/npu.md
Original file line number Diff line number Diff line change
Expand Up @@ -1419,6 +1419,26 @@ graph: `model.onnx` carries `attn.q_norm.layernorm.weight` and `attn.k_norm.laye
FP16 `[128]` initializers, and on Hunyuan-7B-Instruct their bf16 rounding is the first and the
second block bit for bit in 32 of 32 layers. `qk_gain_offsets()` holds that arithmetic.

The same gap exists when q_proj is wider than hidden. Hunyuan-0.5B-Instruct (hidden 1024, 16 heads
of 128, so q_proj has 2,048 rows) has `[64 q/k/v tiles at K=1024][640 B][16 o_proj tiles at
K=2048]`, and its graph's gains are in that gap in the same order in 24 of 24 layers (q_norm at
block offset 2,508,928, k_norm 320 B later). `wide_q_attn_section()` holds that division and names
the gains; before it did, this pack was refused as declaring gains the layout did not hold.

Where the gains are, by layout, in one place: a 4-header block keeps them in their own two sections,
k first (Qwen3, above); a 2-header block keeps them in the 640 B gap between q/k/v and o_proj, q
first, whether o_proj's K is hidden (Hunyuan-7B, Hy-MT2-1.8B) or q_proj's width (Hunyuan-0.5B). A
pack that declares `use_qk_norm` and has them in none of these is refused (`qk_norm_refusal()`).
The gap's offsets are block arithmetic, one number for every layer, so the import checks each
layer's bytes to be `[64 B zeros][finite gain][64 B zeros][finite gain]` (`qk_gap_block_why()`)
and refuses a layer that is not, by name, instead of writing whatever is there.

Hunyuan-0.5B's first payload is in the order q, k, v, like Hunyuan-7B's (`Plan::attn_q_first`).
Checked against `tencent/Hunyuan-0.5B-Instruct` on the host: read as k, q, v, q_proj and k_proj
correlate 0.01 with the checkpoint in layers 0, 11 and 23; read as q, k, v they correlate 0.96 to
0.99, and in chat format the container picks the checkpoint's next token at 39 of 42 positions
(mean KL 0.030 nats).

**Every tile read is its own check.** Before anything is written, each tile is decoded and
re-encoded; the encoder has to reproduce the 34,816 bytes it came from, bias slots included. That
one comparison holds every field, index and nibble order at once, and it is why a tile whose
Expand Down
97 changes: 81 additions & 16 deletions npu/pack_import.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -3340,12 +3340,63 @@ QkGainOffsets qk_gain_offsets(int64_t gap_start, int64_t head_dim, int64_t gap_b
return o;
}

AttnFirstTiles attn_first_tiles(int n_kv, int n_q, bool q_first) {
AttnFirstTiles f;
f.k = q_first ? n_q : 0;
f.q = q_first ? 0 : n_kv;
f.v = n_kv + n_q;
return f;
std::string qk_gap_block_why(const uint8_t* gap, int64_t head_dim) {
const int64_t per_head = head_dim * 2;
for (int b = 0; b < 2; ++b) {
const uint8_t* blk = gap + int64_t(b) * (kNormOff + per_head);
const char* nm = b == 0 ? "q_norm" : "k_norm";
for (int64_t i = 0; i < kNormOff; ++i)
if (blk[i])
return std::string("the 64 B before ") + nm + " are not zero (byte " + std::to_string(i) + " is " +
std::to_string(int(blk[i])) + "), so this is not the [header][gain] pair";
bool any = false;
for (int64_t i = 0; i < head_dim; ++i) {
const uint16_t v = uint16_t(blk[kNormOff + 2 * i]) | uint16_t(uint16_t(blk[kNormOff + 2 * i + 1]) << 8);
if ((v & 0x7F80) == 0x7F80)
return std::string(nm) + " element " + std::to_string(i) + " is not finite (bf16 bits " +
std::to_string(int(v)) + "), so these bytes are not a gain";
any = any || (v & 0x7FFF) != 0;
}
if (!any) return std::string(nm) + " is all zero, so these bytes are not a gain";
}
return std::string();
}

WideQAttnSection wide_q_attn_section(int64_t attn_off, int64_t payload, int64_t qkv_tiles, int64_t tile_bytes,
int64_t o_tiles, int64_t o_tile_bytes, int64_t head_dim) {
WideQAttnSection s;
if (payload <= 0 || qkv_tiles <= 0 || tile_bytes <= 0 || o_tiles <= 0 || o_tile_bytes <= 0) return s;
const int64_t qkv_bytes = qkv_tiles * tile_bytes;
s.gap = payload - qkv_bytes - o_tiles * o_tile_bytes;
// The ceiling is a derived bound, not a measurement: two bf16 [head_dim] gains plus 256 B of headers and padding
// (768 at head_dim 128, 512 at 64). Only the offsets below are measured.
if (s.gap < 0 || s.gap > int64_t(2) * head_dim * 2 + 256) return s;
s.ok = true;
s.o_off = attn_off + qkv_bytes + s.gap;
// Measured on Hunyuan-0.5B-Instruct_rai_1.9.0_npu against the q_norm/k_norm FP16 initializers of its own model.onnx:
// rounded to bf16, q_norm is at block offset 2,508,928 and k_norm 320 B later in 24 of 24 layers, each behind 64
// zero bytes -- the same [header][q][header][k] gap as Hunyuan-7B's, at attn_off + 64 tiles.
const QkGainOffsets g = qk_gain_offsets(attn_off + qkv_bytes, head_dim, s.gap);
s.q_norm = g.q_norm;
s.k_norm = g.k_norm;
return s;
}

bool apply_wide_q_section(Plan& p, const WideQAttnSection& wq, int64_t n_kv, int64_t n_q, int q_out, int64_t hidden) {
if (int64_t(q_out) == hidden || !wq.ok) return false;
p.attn_tiles = n_kv * 2 + n_q;
p.o_off = wq.o_off;
p.o_k = q_out;
// engine#1889: the gap IS the per-head gains when it has their shape; 0,0 otherwise.
p.qnorm_off = wq.q_norm;
p.knorm_off = wq.k_norm;
if (wq.k_norm) p.qk_gap_off = wq.o_off - wq.gap;
// The Hunyuan block's first payload is q, k, v here too (see attn_q_first). Measured on Hunyuan-0.5B-Instruct
// against the public checkpoint: read as k, q, v, the container's k_proj is q_proj's first 1,024 rows and its q_proj
// is q's last 1,024 rows followed by k (per-tensor correlation 0.01 for both in three layers); read as q, k, v all
// three correlate above 0.95.
if (wq.k_norm) p.attn_q_first = true;
return true;
}

std::string header_witness_refusal(int vote_hdrs, bool declared_qk) {
Expand Down Expand Up @@ -3406,6 +3457,14 @@ ContainerGeometry container_geometry(const json& meta, const json& config) {
return g;
}

AttnFirstTiles attn_first_tiles(int n_kv, int n_q, bool q_first) {
AttnFirstTiles f;
f.k = q_first ? n_q : 0;
f.q = q_first ? 0 : n_kv;
f.v = n_kv + n_q;
return f;
}

int64_t head_stored_rows(int64_t vocab, bool from_fconst_tail) {
if (!from_fconst_tail || vocab <= 0) return vocab;
return (vocab + kTileRows - 1) / kTileRows * kTileRows;
Expand Down Expand Up @@ -3786,7 +3845,7 @@ Plan plan(const std::string& pack_dir) {
const int64_t n_kv = kv_out / 64, n_q = q_out / 64;
auto refuse = [&](const std::string& s) {
p.attn_note = s;
p.knorm_off = p.qnorm_off = p.o_off = 0; // an unread section is not a half-read one
p.knorm_off = p.qnorm_off = p.o_off = p.qk_gap_off = 0; // an unread section is not a half-read one
p.o_k = 0;
};
if (g.hdr.size() == 1) {
Expand Down Expand Up @@ -3829,6 +3888,7 @@ Plan plan(const std::string& pack_dir) {
qk_gain_offsets(attn_off + p.attn_tiles * tile_bytes, int64_t(d.head_dim), rem);
p.knorm_off = qkg.k_norm;
p.qnorm_off = qkg.q_norm;
if (qkg.k_norm) p.qk_gap_off = attn_off + p.attn_tiles * tile_bytes;
static bool said2 = false;
if (!said2) {
said2 = true;
Expand All @@ -3849,15 +3909,11 @@ Plan plan(const std::string& pack_dir) {
}
}
const int64_t o_tiles_h = H / kTileRows;
const int64_t qkv_bytes_c = int64_t(n_kv * 2 + n_q) * tile_bytes;
const int64_t o_bytes_c = o_tiles_h * tile_o;
const int64_t gap_c = payload - qkv_bytes_c - o_bytes_c;
const int64_t norm_ceiling = int64_t(2) * int64_t(d.head_dim) * 2 + 256;
if (!handled && payload > 0 && q_out != H && o_tiles_h > 0 && gap_c >= 0 && gap_c <= norm_ceiling) {
const WideQAttnSection wq = wide_q_attn_section(attn_off, payload, int64_t(n_kv * 2 + n_q), tile_bytes,
o_tiles_h, tile_o, int64_t(d.head_dim));
const int64_t gap_c = wq.gap;
if (!handled && apply_wide_q_section(p, wq, n_kv, n_q, q_out, H)) {
handled = true;
p.attn_tiles = n_kv * 2 + n_q;
p.o_off = attn_off + qkv_bytes_c + gap_c;
p.o_k = int(q_out);
// NOT p.attn_note: that field is the refusal channel -- a non-empty value there refuses the
// attention biases by name (it means "why this section could not be read"). This is the
// opposite: the section WAS read this way. Reported on stderr once instead.
Expand Down Expand Up @@ -5651,7 +5707,16 @@ Report import_pack(const std::string& pack_dir, const std::string& out_dir, cons
}
// per-head QK norms (Qwen3 and later): plain bf16 [head_dim] gains that the
// container already carries for families that have them.
if (p.knorm_off)
// The gap families: the offsets are one number for every layer, so each layer's bytes are checked
// to be the [header][gain] pair before a gain is written; a layer that differs is refused by name.
std::string gap_why;
if (p.qk_gap_off)
gap_why = qk_gap_block_why(
at(fc, block + p.qk_gap_off, 2 * (kNormOff + int64_t(p.dims.head_dim) * 2)),
int64_t(p.dims.head_dim));
if (!gap_why.empty())
late.push_back("model.layers." + std::to_string(L) + ".self_attn.q_norm.weight: " + gap_why);
else if (p.knorm_off)
for (const auto& [off, nm] : {std::pair<int64_t, const char*>{p.knorm_off, "k_norm"},
std::pair<int64_t, const char*>{p.qnorm_off, "q_norm"}}) {
const int64_t n = p.dims.head_dim;
Expand Down
29 changes: 29 additions & 0 deletions npu/pack_import.h
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,26 @@ struct QkGainOffsets {
QkGainOffsets qk_gain_offsets(int64_t gap_start, int64_t head_dim, int64_t gap_bytes);
std::string qk_norm_refusal(bool declared, bool found);

// The one-header attention section of a pack whose q_proj is WIDER than hidden (Hunyuan-0.5B: hidden 1024, 16 heads x
// head_dim 128, so q_out = 2048): [q,k,v tiles at K=hidden][gap][o_proj tiles at K=q_out]. `payload` is the bytes from
// `attn_off` to the block's header. `ok` is false when the remainder is not a norms-sized gap (negative, or larger than
// two gains plus 256 B), which is a layout this reader has not measured. When the gap has the gains' shape
// (qk_gain_offsets()), `q_norm`/`k_norm` are their offsets; otherwise both are 0 and the section is still read
// (engine#1889: this branch placed o_proj and never named the gains, so a pack that declares use_qk_norm was refused
// with "gains are not where the block layout says" although they sit in the gap it had just computed).
struct WideQAttnSection {
bool ok = false;
int64_t gap = 0, o_off = 0, q_norm = 0, k_norm = 0;
};
// Why one block's gap is NOT `[64 B zeros][bf16 [head_dim] gain][64 B zeros][bf16 [head_dim] gain]`, or "" when it is.
// `gap` points at 2 * (64 + head_dim*2) bytes. The offsets come from the block arithmetic, which is the same number
// for every layer; this is the per-layer witness that the bytes there are gains: both headers all zero, and each gain
// finite and not all zero. A gain window displaced by a section (320 B at head_dim 128) lands on a header or on tile
// bytes and fails one of these. The import refuses the layer by name on a non-empty answer (review of engine#2063).
std::string qk_gap_block_why(const uint8_t* gap, int64_t head_dim);
WideQAttnSection wide_q_attn_section(int64_t attn_off, int64_t payload, int64_t qkv_tiles, int64_t tile_bytes,
int64_t o_tiles, int64_t o_tile_bytes, int64_t head_dim);

// How many rows a head read from the .fconst tail occupies in the container. The pack stores the head in whole 64-row
// tiles, so a vocabulary that is not a multiple of 64 is carried with the pack's own padding rows (Hunyuan: 120,818 ->
// 120,832) while `vocab_size` and the plan's tensor shape stay the vocabulary. A head that is a graph initializer is
Expand Down Expand Up @@ -225,6 +245,9 @@ struct Plan {
// none. Identified and verified byte-exact against the public Qwen3-1.7B/4B checkpoints (0 of 128
// values differ on either pack, and the swap differs on 127-128 of 128).
int64_t knorm_off = 0, qnorm_off = 0;
// Where the 2-header families' gap `[64 B zeros][q_norm][64 B zeros][k_norm]` starts in the block, 0 when the gains
// are not in that form. The import checks the form in EVERY layer (qk_gap_block_why()) before it writes a gain.
int64_t qk_gap_off = 0;
// The order of the three projections in the block's first payload. Every family measured before Hunyuan writes
// k, q, v (identified on Qwen2.5-0.5B by the tiles' bias subvolumes). The Hunyuan block -- q/k/v at K=hidden, the
// packed per-head gains, o_proj -- writes q, k, v (engine#1889): with k,q,v the container's k_proj was q_proj's
Expand Down Expand Up @@ -342,6 +365,12 @@ ContainerGeometry container_geometry(const nlohmann::json& meta, const nlohmann:

// Reads the pack (model.onnx, its external data, both .fconst headers, config.json).
// Throws only when the pack is structurally unreadable.
// engine#1889 (review of #2063): plan()'s wide-q branch (q_out != hidden), as one function so its effect on the plan is
// testable without a fabricated 1.9 pack. When the section was read (`wq.ok`) and q is wider than hidden, it places the
// q/k/v run, o_proj and the gains, and sets attn_q_first when the gap held the gains: that block is the Hunyuan writer's,
// whose first payload is q, k, v (read as k, q, v, the container's k_proj was q_proj's first rows, correlation 0.01).
// Returns false and leaves `p` untouched otherwise.
bool apply_wide_q_section(Plan& p, const WideQAttnSection& wq, int64_t n_kv, int64_t n_q, int q_out, int64_t hidden);
Plan plan(const std::string& pack_dir);

// Whether the canonical Token .fconst already on the scratch path holds exactly the bytes this run would write,
Expand Down
Loading
Loading