Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions docs/npu.md
Original file line number Diff line number Diff line change
Expand Up @@ -1414,6 +1414,24 @@ stride the `.fconst` attention tile already carries (2,176 B subvolume = 32 rows
from the node's own `N`/`K` (`gate_proj`/`up_proj` → N, `down_proj` → K), because the weight is a 1-D byte blob with no
shape to read it from (engine#1503).

**How the `.packed` blob decodes (engine#2561).** It is the same 2,176-byte subvolume layout as the
[`.fconst` attention payload](#the-fconst-attention-payload), with the same rules:

- **Codes:** the w8 interleave.
- **Zero points:** one per output row in the 16 bytes at `+2048`.
- **Scales:** 32 bf16 values at `+2112`.
- **Row order:** subvolume `2 + 2*kblock + half` holds rows `half*32 + r` of the 64-row tile. The halves are **not** swapped.
- **Values:** the nibbles are the ONNX MatMulNBits fields as given, read **unsigned**: `w = (c - z) * s`.

This was established offline with a known-answer write. AMD's packer (`matmulnbits_pack_const_float32`, mladf v2, K 5120 x N 1024, group 128) was run on chosen weights only as an oracle; the engine neither calls nor ships it. Its output has the blob's own geometry (2,854,912 B = 16 tiles x 82 x 2,176). This layout recovers 100 % of the planted codes, zero points and scales. The earlier reading recovered 6 %, 9 % and 0 %, because it swapped the halves, read four zero points per row from 64 bytes, and sign-extended. That earlier reading had been checked only against this importer's own container.

On `phi-4` the corrected decode reproduces HF `microsoft/phi-4` layer-0 `qkv_proj` times the pack's AWQ fold ratio (`ln_HF / ln_pack`) at corr **+0.992**, with the same row norm (14.79 vs 14.89). The old decode gave corr 0.00 and rows 49x too large, and the model answered `void void ...` on every prompt. With the fix it answers "Paris.", "hydrogen ..." and "Neil Armstrong." on `--device npu`.

Two more facts this genre needs:

- **Final norm:** the graph hangs it on a pseudo layer one past the last (`model.layers.40.final_norm_layernorm.weight`, equal to HF `model.norm.weight`). The importer writes it as `model.norm.weight`, not as a unit gain.
- **RoPE base:** `config.json` can be `{}`, so the base comes from the pack's own `cos_cache` initializer (`rope_cache_best_theta`). On phi-4 that adopts 250000 at corr 1.0000, where the default was 10000.

**The 1.8 hybrid line (`*_rai_1.8.0_hybrid`) is not importable, and that is a decision, not a gap still to be filled (engine#1503).** These packs pair a `model.onnx` with a `model.bin`: 1.3 GB on `Llama-3.2-1B-Instruct_rai_1.8.0_hybrid`, alongside a 542 MB `model.onnx.data` that holds only the embedding and the norms. In the graph, every `.MatMulNBits.*` initializer is a `[0]` placeholder with no bytes, and so is its `.packed` sibling (162 of 162 on that pack). The weights exist only in `model.bin`, a compiled container that the vendor's runtime reads. Writing a reader for it means decoding that container, which this engine does not do, so `npu-pack info` and `npu-pack import` refuse these packs by name before reading anything:

```
Expand Down
93 changes: 70 additions & 23 deletions npu/pack_import.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -2391,30 +2391,28 @@ Q4 decode_external_tile(const uint8_t* data, size_t bytes, int N, int K, std::st
const uint8_t* zeros = subv + 2048;
const uint8_t* scales = subv + 2112;
for (int r = 0; r < kExtSubvRows; ++r) {
// MEASURED, not assumed: a 64-row block's two subvolume halves are stored in the order (half 1,
// half 0) -- the block's FIRST 32 rows live in the pair's SECOND subvolume. Checked row for row against
// the imported container: with `half * 32 + r` exactly one row in 64 matched (R mod 64 == 0), and with the
// swap the sampled rows match (engine#1503).
const int64_t row = tile * kExtTileRows + (1 - half) * kExtSubvRows + r;
// engine#2561, proven by a known-answer write through AMD's own packer (matmulnbits_pack_const_float32,
// mladf v2, K 5120 x N 1024, group 128: 2,854,912 B = 16 tiles x 82 subvolumes x 2,176 B, the eager
// blob's own geometry): this layout recovers 100 % of the planted codes, zero points and scales.
// Subvolume `2 + 2*kblock + half` holds output rows `half*32 + r` of the 64-row tile -- NOT swapped (the
// earlier swap was measured against this importer's own container, so it agreed with itself).
const int64_t row = tile * kExtTileRows + half * kExtSubvRows + r;
const float s = bf16_to_f32(uint16_t(scales[r * 2] | (scales[r * 2 + 1] << 8)));
// four int4 zero points per row, one per 32-column group of the 128-column block
const uint8_t z0 = zeros[r * 2], z1 = zeros[r * 2 + 1];
// Both fields are int4 in TWO'S COMPLEMENT, not 0..15 (the pack's own writer asserts -8 <= x <= 7,
// FACTS-oga-pack-weight-layout.md §7.13). Reading them unsigned is only equivalent when the code and
// its zero point happen to share a sign: for one negative and one positive the difference is off by
// 16 * scale, which is ~16 codes wide and dominates the weight. That is what left phi-4 answering
// '都都都都': every row where the two differed in sign became a near-constant matrix (engine#1503).
const int zp[4] = {int8_t(z0 << 4) >> 4, int8_t(z0) >> 4, int8_t(z1 << 4) >> 4, int8_t(z1) >> 4};
// ONE zero point per output row per 128-column block: 16 bytes at +2048, nibble r (low nibble = even r).
// Reading 64 bytes as four zero points per row recovered 9 % of them.
const uint8_t zb = zeros[r / 2];
const int z = (r % 2 == 0) ? int(zb & 0xF) : int(zb >> 4);
for (int g = 0; g < 4; ++g) {
q.scale[size_t(row) * (K / 32) + (kk - 1) * 4 + g] = s;
// The container's form is w = code*d + m with code in 0..15 and m the group MIN, so the signed
// code c becomes c + 8 and m becomes (-8 - z) * scale: (c+8)*s + (-8-z)*s == (c - z)*s exactly.
q.zero[size_t(row) * (K / 32) + (kk - 1) * 4 + g] = float(-8 - zp[g]) * s;
// The fields are the ONNX MatMulNBits nibbles as given, UNSIGNED: w = (c - z) * s. In the container's
// form w = code*d + m that is code = c, d = s, m = -z * s. (The packer stores the nibbles it is
// handed unchanged; the earlier two's-complement reading was a correction for the other two bugs.)
q.zero[size_t(row) * (K / 32) + (kk - 1) * 4 + g] = -float(z) * s;
}
for (int c = 0; c < 128; ++c) {
// Row-major: a row's 128 codes are 64 contiguous bytes, even column in the low nibble.
// The .fconst formula: one byte per COLUMN holds the row PAIR 2cp (low) / 2cp+1 (high), so a
// row's codes are strided by 4 through a 512-byte block -- no column pairing at all.
// w8 interleave (the packer's wh_w8_index): one byte per K column holds the output-row pair
// 2cp (low nibble) / 2cp+1 (high), strided by 4 through a 512-byte block. Row-major is kept only
// as the alternative the unit test holds to its own spec.
uint8_t nib;
if (row_major) {
const uint8_t b = quants[r * 64 + c / 2];
Expand All @@ -2424,10 +2422,7 @@ Q4 decode_external_tile(const uint8_t* data, size_t bytes, int N, int K, std::st
const uint8_t b = quants[(cp % 4) + c * 4 + (cp / 4) * 512];
nib = (r % 2 == 0) ? uint8_t(b & 0xF) : uint8_t(b >> 4);
}
// signed int4 -> the container's 0..15 offset from the group minimum: c_signed = nib-16 for a
// negative nibble, then +8. A no-op would have been `nib`, which is the bug this replaces.
const int cs = (nib >= 8) ? int(nib) - 16 : int(nib);
q.codes[size_t(row) * K + (kk - 1) * 128 + c] = uint8_t(cs + 8);
q.codes[size_t(row) * K + (kk - 1) * 128 + c] = nib;
}
}
}
Expand Down Expand Up @@ -5063,6 +5058,32 @@ std::string kernel_set_refusal(const std::vector<std::string>& file_names, const
// Copies the tokenizer/chat files the container needs beside model.q4nx; defined past kTextFiles' declaration.
static int copy_text_artifacts(const std::string& pack_dir, const std::string& out_dir);

double rope_cache_best_theta(const std::vector<float>& cache, int64_t rows, int64_t half,
const std::vector<double>& cands, double* best_theta) {
// ONE Pearson r of the cache against cos(p * theta^(-2i/d)) over the whole matrix (the per-column mean is noise
// for the flat high columns; see the .fconst path's note on Qwen3-1.7B). Invariant to a per-pack cache scale.
double best = -2.0;
if (best_theta) *best_theta = 0.0;
const int64_t pmax = std::min<int64_t>(rows, 4096);
const double d = double(half) * 2.0;
for (const double th : cands) {
if (!(th > 0)) continue;
double sx = 0, sy = 0, sxx = 0, syy = 0, sxy = 0, n = 0;
for (int64_t i = 0; i < half; ++i) {
const double inv = 1.0 / std::pow(th, 2.0 * double(i) / d);
for (int64_t p = 0; p < pmax; ++p) {
const double y = cache[size_t(p) * size_t(half) + size_t(i)], x = std::cos(double(p) * inv);
sx += x; sy += y; sxx += x * x; syy += y * y; sxy += x * y; n += 1;
}
}
const double cx = sxx - sx * sx / n, cy = syy - sy * sy / n;
if (cx <= 1e-12 || cy <= 1e-12) continue;
const double r = (sxy - sx * sy / n) / std::sqrt(cx * cy);
if (r == r && r > best) { best = r; if (best_theta) *best_theta = th; }
}
return best;
}

Report import_external(const std::string& pack_dir, const std::string& out_dir, const Options& opt) {
const Plan p = plan(pack_dir);
const std::string graph = graph_filename(pack_dir);
Expand Down Expand Up @@ -5260,6 +5281,32 @@ Report import_external(const std::string& pack_dir, const std::string& out_dir,
c["head_dim"] = p.dims.head_dim;
c["vocab_size"] = p.dims.vocab;
c["rope_theta"] = p.dims.rope_theta;
// engine#2561: this genre's config.json can be `{}` (phi-4), so rope_theta above is a default (10000); the graph's
// GroupQueryAttention reads a precomputed `cos_cache` initializer, which is the pack's own statement of its base.
if (onnx.index.count("cos_cache")) {
const OnnxTensor& cc = onnx.tensors[onnx.index.at("cos_cache")];
if (cc.dims.size() == 2 && cc.dims[0] > 1 && cc.dims[1] > 0 && cc.type == kF32) {
const int64_t rows = cc.dims[0], half = cc.dims[1];
uint64_t need = 0;
if (mul_ok(uint64_t(rows) * uint64_t(half), 4u, need)) {
std::vector<float> cache(size_t(rows) * size_t(half));
std::memcpy(cache.data(), bytes_of(cc, need), size_t(need));
const std::vector<double> cands{double(p.dims.rope_theta), 1e4, 5e4, 1e5, 2.5e5, 5e5, 1e6, 5e6, 1e7};
double best_theta = 0.0;
const double best_corr = rope_cache_best_theta(cache, rows, half, cands, &best_theta);
if (best_corr > 0.99 && best_theta > 0.0) {
if (std::fabs(best_theta - double(p.dims.rope_theta)) > 1e-6)
std::fprintf(stderr, "note: rope_theta %.6g declared, %.6g adopted (correlation %.4f against the "
"pack's own cos_cache)\n", double(p.dims.rope_theta), best_theta, best_corr);
c["rope_theta"] = best_theta;
} else {
std::fprintf(stderr, "WARNING: no candidate rope_theta explains the pack's cos_cache (best %.4f at %.6g); "
"keeping %.6g, attention may be wrong\n", best_corr, best_theta,
double(p.dims.rope_theta));
}
}
}
}
c["q4nx_parameters"] = "fp32 scale/zero per 32-column group";
if (!opt.weight_layout.empty() && opt.weight_layout != "rm") c["q4nx_weight_layout"] = opt.weight_layout;
c["q4nx_source"] = {{"kind", "oga-npu-pack external data"}, {"pack", p.pack},
Expand Down
4 changes: 4 additions & 0 deletions npu/pack_import.h
Original file line number Diff line number Diff line change
Expand Up @@ -893,6 +893,10 @@ bool external_container_tensors(const std::string& role, int nh, int nkv, int hd

Q4 decode_external_tile(const uint8_t* data, size_t bytes, int N, int K, std::string& err,
bool row_major = kExtQuantsRowMajor);
// engine#2561: the candidate theta whose cos(p * theta^(-2i/d)) best explains a pack's precomputed cos_cache
// ([rows, half] row-major, d = 2*half), as one Pearson r over the whole matrix; writes the winner to *best_theta.
double rope_cache_best_theta(const std::vector<float>& cache, int64_t rows, int64_t half,
const std::vector<double>& cands, double* best_theta);
Q4 decode_matmulnbits(const uint8_t* qweight, const float* scales, const uint8_t* zeros,
int N, int K, int block, bool swap_nibbles = false, bool drop_zero = false);

Expand Down
Loading
Loading