From 3d183af19652df4853b6b7738df119e8b513da5c Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Wed, 11 Nov 2026 10:04:30 -0400 Subject: [PATCH 1/2] npu: read the 4,608-byte symmetric chunk, so Hy-MT2-1.8B-NPU2 serves FastFlowLM's Hy-MT2 pack stores its projections in a tile no reader knew: 256 bf16 scales, then 4,096 code bytes in column order, value = two's complement nibble * scale. The size names the kind. Settled by the tie test against the pack's own bf16 embedding; the pack then answers on the per-op route with kernel shapes that already existed. Co-Authored-By: Claude Fable 5.2 --- docs/npu.md | 23 ++++++++++- npu/forward/include/pad_head_rows.h | 2 +- npu/forward/include/q4nx_dequant.h | 10 +++++ npu/forward/src/q4nx_dequant.c | 30 +++++++++++--- npu/forward/src/q4nx_forward.cpp | 2 +- npu/model.cpp | 2 +- tests/npu_dequant_bf16_test.c | 63 ++++++++++++++++++++++++++--- 7 files changed, 117 insertions(+), 15 deletions(-) diff --git a/docs/npu.md b/docs/npu.md index baedf4d8e..7100a4d4c 100644 --- a/docs/npu.md +++ b/docs/npu.md @@ -1843,7 +1843,7 @@ engine#587 measured the route on it): | `qwen35` | `FastFlowLM/D3-*-NPU2` (Edge, Flash, Mini, Nano, Lite, Turbo, Ultra), `FastFlowLM/Qwen3.8-27B-NPU2`, `FastFlowLM/Qwen3.5-0.8B-NPU2` | hrx, npu | **served, and measured** (engine#587): the same `Qwen3.8-27B-NPU2` family runs on the NPU route — 0.2902 s/token (3.45 tok/s) and 19.55 tok/s of prefill on the sealed single-ELF ternary rung, 22.811 s to a first token with the weight load in it (`bench/npu-leaderboard/27b-t2s2-*-20261106.json`) — int8 Q4NX decodes at 63.1 s/token and takes 145.7 s to a first token (`bench/npu-leaderboard/27b-*.json`, 2026-10-27), a 5-token prefill chunk is 58.14 s, and the ternary sealed-ELF rung on the same container is 1.72 s/token with 3/3 text-identical output (engine#697, closed by #728). Quality on the same route: **GSM8K 97.5 % (39/40)** with greedy decoding, **6 of the 40 answers cut off by the 512-token budget and exactly 1 of those 6 also wrong** — so among answers that finished, 34/34 were right (`27b-gsm8k-20261107`); and the per-token cost at those answer lengths is 0.3857 s/token against 0.2902 on a 13-token prompt, i.e. it grows ~1.33x with context (`27b-decode-cot-20261107`). What a stock install still needs is a kernel set for the model's geometry — none ships in this repository — so the row read `not served` until the route itself was measured | | `dflash` | `FastFlowLM/Nemotron-3.5-Bolt-E4B-DFlash-NPU2` | hrx | not served: a draft head, not a decoder; "The draft loop, and whether to serve it" below covers what the HRX route does with it | | `gemma4` | `FastFlowLM/Nemotron-3.5-Bolt-E4B-NPU2`, `FastFlowLM/Gemma4-12B-IT-NPU2` | hrx | not served: no gemma4 model type or kernels in the route | -| `hunyuan-dense` | `FastFlowLM/Hy-MT2-1.8B-NPU2` | hrx | not served | +| `hunyuan-dense` | `FastFlowLM/Hy-MT2-1.8B-NPU2` | hrx, npu | **served and measured** on the per-op route ("FastFlowLM's own containers" below) | | `gemma3` | `FastFlowLM/medgemma-1.5-4b-it-NPU2` | hrx | not served | | `lfm2` | `FastFlowLM/LFM2-2.6B-NPU2`, `FastFlowLM/LFM2-1.2B-NPU2`, `FastFlowLM/LFM2.5-1.2B-NPU2` | hrx, npu | `LFM2.5-1.2B-NPU2` **served and measured** on the per-op route ("FastFlowLM's own containers" below); `LFM2-1.2B-NPU2` serves too (one paired ticket, same section); `LFM2-2.6B-NPU2` was not run | | `gpt-oss` | `FastFlowLM/GPT-OSS-20B-NPU2`, `FastFlowLM/GPT-OSS-Safeguard-20b-NPU2` | hrx | not served | @@ -1900,6 +1900,7 @@ does on a geometry that has a sealed set: `Qwen3-1.7B-NPU2` is 3.0 tok/s here an | `Llama-3.2-3B-NPU2` | `llama` | identical, `Paris.` | 1.7 to 1.8 tok/s | | `Phi4-mini-Instruct-NPU2` | `phi3` | identical, `Paris`, once the turn end is read from the template (below) | 1.4 to 1.5 tok/s | | `Qwen3-VL-4B-Instruct-NPU2`, text tower only | `qwen3` | identical, `Paris` | 1.4 to 1.5 tok/s | +| `Hy-MT2-1.8B-NPU2` | `hunyuan_v1_dense` | identical, `Paris`; five translations against FastFlowLM's runtime below | 2.0 to 2.8 tok/s | | `LFM2.5-1.2B-NPU2` | `lfm2`, hybrid: 10 conv layers of 16 | identical, `Paris`; against FastFlowLM's runtime below | 3.5 to 4.6 tok/s | | `LFM2-1.2B-NPU2` | `lfm2`, hybrid: 10 conv layers of 16 | `Paris`; one request per prompt, against FastFlowLM's runtime below | 3.7 tok/s | @@ -1964,6 +1965,26 @@ does not carry the three-identical-answers condition. reasoning, so there is no answer text to compare and no row; the reasoning differs from FastFlowLM's from its first words. +**`Hy-MT2-1.8B-NPU2`: a third tile.** This pack's projections are stored 4,608 bytes to a tile where every +other pack on the box uses 5,120, and no reader knew the size, so it was refused at the first dequant. The tile is +256 bf16 scales (index `g*32 + r`) followed by 4,096 code bytes in column order (`c*16 + r/2`, even row in the low +nibble), and a weight is the two's-complement nibble times its scale: the symmetric reading of the LFM2 packs with +the zero-point block left out and the codes in the Q4K chunk's order. It was settled the same way, against the +pack's own bf16 embedding (its `lm_head` is the quantised copy): mean 0.244 of a step, maximum 1.000, and the +eleven other combinations of scale order, code order and code sign are 3 to 8 steps off. Here the size names the +kind, so nothing is read off the data. The kernel directory is four shapes that were already built for other +packs (`QKV K2048 N3072`, `O K2048 N2048`, `GU K2048 N12288`, `D K6144 N2048`); q/k norm and the template +(`chat_template.jinja`) are read from the pack. + +Measured 2026-11-11, same `model.q4nx` (sha256 `a992683e04c2110e…`) for both runtimes, one ticket: five +translation requests (into German, French, Spanish, Chinese and English), all five answered with a correct +translation by both; one byte-identical (`Où se trouve la gare de train la plus proche ?`), four worded +differently (`Me gustaría una taza de café…` here, `Quisiera una taza de café…` there). The host float run of the +first request gives this route's wording, so the array agrees with float arithmetic on these weights as it does +for LFM2; no reference independent of both runtimes was available for this model, so the row does not say whose +wording is the model's. Three requests of the known-answer prompt returned `Paris` three times. Decode 2.0 to +2.8 tok/s (32 layers on the per-op route) against FastFlowLM's 50 to 53. + Not established by any row here: that a pack goes through the importer (`npu-pack` still reports these containers as recognised, not importable), anything about a vision or audio tower, and any pack whose geometry has no kernel set on the box. Of the 57 `*-NPU2` repos with a readable geometry on 2026-11-11, 19 had all four diff --git a/npu/forward/include/pad_head_rows.h b/npu/forward/include/pad_head_rows.h index 89e0f77f7..91bb7794d 100644 --- a/npu/forward/include/pad_head_rows.h +++ b/npu/forward/include/pad_head_rows.h @@ -45,7 +45,7 @@ // tiles gives 64,064, which fails the bound and falls through to the row spelling (review of #2026 at 63a47be). // Anything the three spellings cannot place returns NV, and the caller's own dequant then refuses it by name. inline bool head_chunk_bytes(int64_t b) { - return b == Q4NX_TILE_BYTES || b == 6144 /* q4_1f, the imported-container tile */ || b == Q4NX_CHUNK_Q4K || b == Q4NX_CHUNK_Q8; + return b == Q4NX_TILE_BYTES || b == 6144 /* q4_1f, the imported-container tile */ || b == Q4NX_CHUNK_Q4K || b == Q4NX_CHUNK_Q8 || b == Q4NX_CHUNK_Q4S; } inline int padded_head_rows(const int64_t* shape, int ndim, int kpad, int NV) { if (!shape || kpad <= 0 || kpad % Q4NX_TILE_COLS != 0 || NV <= 0) return NV; diff --git a/npu/forward/include/q4nx_dequant.h b/npu/forward/include/q4nx_dequant.h index adf2736c3..9476d9dbc 100644 --- a/npu/forward/include/q4nx_dequant.h +++ b/npu/forward/include/q4nx_dequant.h @@ -90,6 +90,16 @@ extern "C" { * the cross-validation against the checkpoint's own `.bf16` twin. */ #define Q4NX_CHUNK_Q4K 4736 #define Q4NX_CHUNK_Q8 8704 +/* The symmetric 4-bit chunk with no zero-point block (FastFlowLM lists the pack as "Q4_0"; Hy-MT2-1.8B-NPU2): + * scales [256] bf16 @[0,512) index g*32 + r (group-major, as in the 5120-byte tile) + * codes [256][16] @[512,4608) byte c*16 + r/2, even row = low nibble (column-major, as in the Q4K chunk) + * value[r][c] = (code >= 8 ? code - 16 : code) * scale[g][r] + * Settled by the tie test on that pack (2026-11-11): its lm_head is the quantised copy of its own bf16 + * embed_tokens, and over the first 1024 rows |dequant - embedding| / |scale| has mean 0.244 and maximum 1.000 with + * this reading; the eleven other combinations of scale order, code order and code sign are 3 to 8 steps off. + * The size names the kind (no tile of another kind is 4608 bytes), so nothing is inferred from the data. */ +#define Q4NX_CHUNK_Q4S 4608 +void q4nx_dequant_tile_q4s(const uint8_t* chunk, float* out, int ld); /* Q4_K chunk: 32 rows x 256 K, 4736 B. * scales[8][32] uint8 @[0,256) index g*32 + r diff --git a/npu/forward/src/q4nx_dequant.c b/npu/forward/src/q4nx_dequant.c index 25acac18b..dca7810ce 100644 --- a/npu/forward/src/q4nx_dequant.c +++ b/npu/forward/src/q4nx_dequant.c @@ -136,12 +136,26 @@ void q4nx_dequant_tile_q8(const uint8_t* t, float* out, int ld) { } } +void q4nx_dequant_tile_q4s(const uint8_t* t, float* out, int ld) { + const uint8_t* sc_b = t; /* [256] bf16, index g*32 + r */ + const uint8_t* qs = t + 512; /* [256][16], byte c*16 + r/2 */ + for (int r = 0; r < Q4NX_TILE_ROWS; r++) { + for (int c = 0; c < Q4NX_TILE_COLS; c++) { + const int g = c / Q4NX_GROUP_SIZE; + const uint8_t byte = qs[c * 16 + r / 2]; + int code = (r % 2 == 0) ? (byte & 0xF) : (byte >> 4); + if (code >= 8) code -= 16; /* two's complement nibble */ + out[(size_t)r * ld + c] = (float)code * bf16_to_f32(le16(sc_b + 2 * (g * Q4NX_TILE_ROWS + r))); + } + } +} + int q4nx_dequant_tensor_chunked(const uint8_t* data, size_t nbytes, int rows, int cols, int chunk_bytes, float* out) { if (!data || !out || rows <= 0 || cols <= 0) return -1; if (rows % Q4NX_TILE_ROWS || cols % Q4NX_TILE_COLS) return -1; if (chunk_bytes != Q4NX_TILE_BYTES && chunk_bytes != Q4NX_CHUNK_Q4K && - chunk_bytes != Q4NX_CHUNK_Q8) + chunk_bytes != Q4NX_CHUNK_Q8 && chunk_bytes != Q4NX_CHUNK_Q4S) return -1; const int ntr = rows / Q4NX_TILE_ROWS, ntc = cols / Q4NX_TILE_COLS; if (nbytes < (size_t)ntr * (size_t)ntc * (size_t)chunk_bytes) return -1; @@ -161,6 +175,8 @@ int q4nx_dequant_tensor_chunked(const uint8_t* data, size_t nbytes, int rows, q4nx_dequant_tile_q4k(c, tmp, Q4NX_TILE_COLS); else if (chunk_bytes == Q4NX_CHUNK_Q8) q4nx_dequant_tile_q8(c, tmp, Q4NX_TILE_COLS); + else if (chunk_bytes == Q4NX_CHUNK_Q4S) + q4nx_dequant_tile_q4s(c, tmp, Q4NX_TILE_COLS); else q4nx_dequant_tile_as(c, tmp, Q4NX_TILE_COLS, sym); for (int r = 0; r < Q4NX_TILE_ROWS; r++) @@ -181,7 +197,7 @@ int q4nx_dequant_tile_row(const uint8_t* data, size_t nbytes, int rows, int cols if (r < 0 || r >= rows) return -1; if (rows % Q4NX_TILE_ROWS || cols % Q4NX_TILE_COLS) return -1; if (chunk_bytes != Q4NX_TILE_BYTES && chunk_bytes != Q4NX_CHUNK_Q4K && - chunk_bytes != Q4NX_CHUNK_Q8) + chunk_bytes != Q4NX_CHUNK_Q8 && chunk_bytes != Q4NX_CHUNK_Q4S) return -1; const int ntr = rows / Q4NX_TILE_ROWS, ntc = cols / Q4NX_TILE_COLS; if (nbytes < (size_t)ntr * (size_t)ntc * (size_t)chunk_bytes) return -1; @@ -199,6 +215,8 @@ int q4nx_dequant_tile_row(const uint8_t* data, size_t nbytes, int rows, int cols q4nx_dequant_tile_q4k(c, tmp, Q4NX_TILE_COLS); else if (chunk_bytes == Q4NX_CHUNK_Q8) q4nx_dequant_tile_q8(c, tmp, Q4NX_TILE_COLS); + else if (chunk_bytes == Q4NX_CHUNK_Q4S) + q4nx_dequant_tile_q4s(c, tmp, Q4NX_TILE_COLS); else q4nx_dequant_tile_as(c, tmp, Q4NX_TILE_COLS, sym); memcpy(out + (size_t)tc * Q4NX_TILE_COLS, tmp + (size_t)rr * Q4NX_TILE_COLS, @@ -210,8 +228,8 @@ int q4nx_dequant_tile_row(const uint8_t* data, size_t nbytes, int rows, int cols int q4nx_tile_row_kind(size_t nbytes, int rows, int cols) { if (rows <= 0 || cols <= 0 || rows % Q4NX_TILE_ROWS || cols % Q4NX_TILE_COLS) return 0; const size_t tiles = (size_t)(rows / Q4NX_TILE_ROWS) * (size_t)(cols / Q4NX_TILE_COLS); - const int kinds[3] = {Q4NX_TILE_BYTES, Q4NX_CHUNK_Q4K, Q4NX_CHUNK_Q8}; - for (int i = 0; i < 3; i++) + const int kinds[4] = {Q4NX_TILE_BYTES, Q4NX_CHUNK_Q4K, Q4NX_CHUNK_Q8, Q4NX_CHUNK_Q4S}; + for (int i = 0; i < 4; i++) if (nbytes == tiles * (size_t)kinds[i]) return kinds[i]; return 0; } @@ -262,7 +280,7 @@ int q4nx_dequant_tensor_packed(const uint8_t* data, size_t nbytes, int rows, int if (rows_packed < rows || cols_packed < cols) return -1; if (rows_packed % Q4NX_TILE_ROWS || cols_packed % Q4NX_TILE_COLS) return -1; if (chunk_bytes != Q4NX_TILE_BYTES && chunk_bytes != Q4NX_CHUNK_Q4K && - chunk_bytes != Q4NX_CHUNK_Q8) + chunk_bytes != Q4NX_CHUNK_Q8 && chunk_bytes != Q4NX_CHUNK_Q4S) return -1; if (rows_packed == rows && cols_packed == cols) return q4nx_dequant_tensor_chunked(data, nbytes, rows, cols, chunk_bytes, out); @@ -285,6 +303,8 @@ int q4nx_dequant_tensor_packed(const uint8_t* data, size_t nbytes, int rows, int q4nx_dequant_tile_q4k(c, tmp, Q4NX_TILE_COLS); else if (chunk_bytes == Q4NX_CHUNK_Q8) q4nx_dequant_tile_q8(c, tmp, Q4NX_TILE_COLS); + else if (chunk_bytes == Q4NX_CHUNK_Q4S) + q4nx_dequant_tile_q4s(c, tmp, Q4NX_TILE_COLS); else q4nx_dequant_tile_as(c, tmp, Q4NX_TILE_COLS, sym); for (int r = 0; r < Q4NX_TILE_ROWS; r++) diff --git a/npu/forward/src/q4nx_forward.cpp b/npu/forward/src/q4nx_forward.cpp index 9dca11690..a2b7f3733 100644 --- a/npu/forward/src/q4nx_forward.cpp +++ b/npu/forward/src/q4nx_forward.cpp @@ -187,7 +187,7 @@ static int dequant_any(const void* data, size_t nbytes, int rows, int cols, floa const size_t n = (size_t)(rows / Q4NX_TILE_ROWS) * (size_t)(cols / Q4NX_TILE_COLS); if (n == 0) return -1; const int cb = (int)(nbytes / n); - if (cb == Q4NX_TILE_BYTES || cb == Q4NX_CHUNK_Q4K || cb == Q4NX_CHUNK_Q8) + if (cb == Q4NX_TILE_BYTES || cb == Q4NX_CHUNK_Q4K || cb == Q4NX_CHUNK_Q8 || cb == Q4NX_CHUNK_Q4S) return q4nx_dequant_tensor_chunked((const uint8_t*)data, nbytes, rows, cols, cb, out); // The container may store this weight on a wider grid than the model's dimensions (engine#1097). // Read it on the container's grid and write only the model's [rows, cols], so buffers, row diff --git a/npu/model.cpp b/npu/model.cpp index fb8a5fed6..2b1beb5e2 100644 --- a/npu/model.cpp +++ b/npu/model.cpp @@ -67,7 +67,7 @@ bool table_holds_vocab(const Tensor& t, int64_t vocab, int64_t hidden, std::stri } if (t.dtype != "I8") return true; if (t.bytes == uint64_t(vocab) * uint64_t(hidden)) return true; // plain int8 + per-32-column scale (Gemma4) - const auto is_chunk = [](int64_t b) { return b == Q4NX_TILE_BYTES || b == Q4NX_CHUNK_Q4K || b == Q4NX_CHUNK_Q8; }; + const auto is_chunk = [](int64_t b) { return b == Q4NX_TILE_BYTES || b == Q4NX_CHUNK_Q4K || b == Q4NX_CHUNK_Q8 || b == Q4NX_CHUNK_Q4S; }; int64_t r = -1; if (t.shape.size() == 3 && is_chunk(t.shape[2]) && t.shape[1] == col_tiles) r = t.shape[0] * Q4NX_TILE_ROWS; else if (t.shape.size() == 2 && is_chunk(t.shape[1]) && t.shape[0] % col_tiles == 0) diff --git a/tests/npu_dequant_bf16_test.c b/tests/npu_dequant_bf16_test.c index 0840e21a9..86626d70c 100644 --- a/tests/npu_dequant_bf16_test.c +++ b/tests/npu_dequant_bf16_test.c @@ -121,11 +121,11 @@ static void embed_row_arms(void) { static void chunked_row_arms(void) { const int rows = 64, cols = 512; /* 2x2 tiles: several rows per tile, several tiles per row */ const int ntc = cols / Q4NX_TILE_COLS; - const int kinds[3] = {Q4NX_TILE_BYTES, Q4NX_CHUNK_Q4K, Q4NX_CHUNK_Q8}; + const int kinds[4] = {Q4NX_TILE_BYTES, Q4NX_CHUNK_Q4K, Q4NX_CHUNK_Q8, Q4NX_CHUNK_Q4S}; float* whole = malloc((size_t)rows * cols * sizeof(float)); float* row = malloc((size_t)cols * sizeof(float)); if (!whole || !row) { check(0, "chunked row: the arms allocate"); free(whole); free(row); return; } - for (int k = 0; k < 3; k++) { + for (int k = 0; k < 4; k++) { const int cb = kinds[k]; const size_t nbytes = (size_t)(rows / Q4NX_TILE_ROWS) * (size_t)ntc * (size_t)cb; uint8_t* q = malloc(nbytes); @@ -141,7 +141,8 @@ static void chunked_row_arms(void) { } check(bad == 0, cb == Q4NX_TILE_BYTES ? "chunked row: every row matches the whole-table decode (5,120 B tiles)" : cb == Q4NX_CHUNK_Q4K ? "chunked row: every row matches the whole-table decode (Q4K chunks)" - : "chunked row: every row matches the whole-table decode (Q8 chunks)"); + : cb == Q4NX_CHUNK_Q8 ? "chunked row: every row matches the whole-table decode (Q8 chunks)" + : "chunked row: every row matches the whole-table decode (Q4S chunks, 4,608 B)"); check(q4nx_dequant_tile_row(q, nbytes, rows, cols, cb, rows, row) == -1, "chunked row: row == rows is refused"); check(q4nx_dequant_tile_row(q, nbytes, rows, cols, cb, -1, row) == -1, "chunked row: row -1 is refused"); check(q4nx_dequant_tile_row(q, nbytes - 1, rows, cols, cb, 0, row) == -1, @@ -152,7 +153,7 @@ static void chunked_row_arms(void) { free(q); } free(whole); free(row); - printf(" chunked row: 3 kinds x 64 rows bit-exact + 5 refusal arms\n"); + printf(" chunked row: 4 kinds x 64 rows bit-exact + 5 refusal arms\n"); } /* engine#1842: the forward picks the per-row path for a chunked embedding with q4nx_tile_row_kind(). It used to pick it @@ -165,8 +166,8 @@ static int old_rule(size_t esz, int nv, int kpad) { return tiles > 0 && esz % tiles == 0; } static void tile_row_kind_arms(void) { - const int kinds[3] = {Q4NX_TILE_BYTES, Q4NX_CHUNK_Q4K, Q4NX_CHUNK_Q8}; - for (int k = 0; k < 3; k++) + const int kinds[4] = {Q4NX_TILE_BYTES, Q4NX_CHUNK_Q4K, Q4NX_CHUNK_Q8, Q4NX_CHUNK_Q4S}; + for (int k = 0; k < 4; k++) check(q4nx_tile_row_kind((size_t)4 * (size_t)kinds[k], 64, 512) == kinds[k], "tile-row kind: each supported chunk kind on a 2x2 grid is accepted as itself"); struct { size_t esz; int nv, kpad; const char* what; } bad[] = { @@ -329,7 +330,57 @@ static void tile_kind_is_per_tensor(void) { free(t); free(out); free(row); } +/* The 4,608-byte symmetric chunk (Hy-MT2-1.8B-NPU2): 256 bf16 scales at g*32 + r, then 4,096 code bytes at + * c*16 + r/2 (even row = low nibble), value = two's-complement nibble * scale. The expected values below index the + * bytes from that description, not through the decoder; and each of the three ways the layout was NOT (the 5,120 + * tile's code order, row-major scales, offset-binary codes) is shown to give a different table, so the fixture can + * tell them apart. */ +static void q4s_chunk(void) { + uint8_t* t = (uint8_t*)calloc(1, Q4NX_CHUNK_Q4S); + float* out = (float*)malloc(sizeof(float) * Q4NX_TILE_ROWS * Q4NX_TILE_COLS); + if (!t || !out) { check(0, "q4s: the buffers allocate"); free(t); free(out); return; } + for (int i = 0; i < 256; i++) put_bf16(t, (size_t)i, ((i % 5 == 0) ? -1.0f : 1.0f) * 0.125f * (float)(1 + i % 7)); + for (int i = 0; i < 4096; i++) t[512 + i] = (uint8_t)((i * 113 + 29) & 0xFF); + #define Q4S_U(r, c) (((r) % 2 == 0) ? (t[512 + (c) * 16 + (r) / 2] & 0xF) : (t[512 + (c) * 16 + (r) / 2] >> 4)) + #define Q4S_SC(i) ((((i) % 5 == 0) ? -1.0f : 1.0f) * 0.125f * (float)(1 + (i) % 7)) + q4nx_dequant_tile_q4s(t, out, Q4NX_TILE_COLS); + int bad = 0, as_5120_order = 0, as_row_major = 0, as_offset = 0, neg = 0; + for (int r = 0; r < Q4NX_TILE_ROWS; r++) + for (int c = 0; c < Q4NX_TILE_COLS; c++) { + const int g = c / 32, u = Q4S_U(r, c), sgn = u >= 8 ? u - 16 : u; + const float got = out[r * Q4NX_TILE_COLS + c]; + if (u >= 8) neg = 1; + if (got != (float)sgn * Q4S_SC(g * 32 + r)) bad++; + /* the layouts it is not */ + const uint8_t b5 = t[512 + ((r / 16) * 8 + g) * 256 + 8 * (c % 32) + (r % 16) / 2]; + const int u5 = ((r % 16) % 2 == 0) ? (b5 & 0xF) : (b5 >> 4); + if (got != (float)(u5 >= 8 ? u5 - 16 : u5) * Q4S_SC(g * 32 + r)) as_5120_order++; + if (got != (float)sgn * Q4S_SC(r * 8 + g)) as_row_major++; + if (got != (float)(u - 8) * Q4S_SC(g * 32 + r)) as_offset++; + } + check(neg, "q4s: the fixture has codes of 8 and above"); + check(bad == 0, "q4s: every element is (two's-complement nibble at c*16 + r/2) * (scale at g*32 + r)"); + check(as_5120_order > 1000, "q4s: the 5,120 tile's code order would give a different table"); + check(as_row_major > 1000, "q4s: row-major scales would give a different table"); + check(as_offset > 1000, "q4s: offset-binary codes would give a different table"); + float* whole = (float*)malloc(sizeof(float) * Q4NX_TILE_ROWS * Q4NX_TILE_COLS); + if (whole) { + check(q4nx_dequant_tensor_chunked(t, Q4NX_CHUNK_Q4S, Q4NX_TILE_ROWS, Q4NX_TILE_COLS, Q4NX_CHUNK_Q4S, whole) == 0 && + memcmp(whole, out, sizeof(float) * Q4NX_TILE_ROWS * Q4NX_TILE_COLS) == 0, + "q4s: the tensor reader takes the kind by its size and decodes the same values"); + check(q4nx_dequant_tensor_chunked(t, Q4NX_CHUNK_Q4S - 1, Q4NX_TILE_ROWS, Q4NX_TILE_COLS, Q4NX_CHUNK_Q4S, whole) == -1, + "q4s: one byte short is refused"); + } else check(0, "q4s: the reader buffer allocates"); + check(q4nx_tile_row_kind((size_t)Q4NX_CHUNK_Q4S, 32, 256) == Q4NX_CHUNK_Q4S && + q4nx_tile_row_kind((size_t)Q4NX_TILE_BYTES, 32, 256) == Q4NX_TILE_BYTES, + "q4s: 4,608 and 5,120 bytes per tile are told apart by size alone"); + #undef Q4S_U + #undef Q4S_SC + free(whole); free(t); free(out); +} + int main(void) { + q4s_chunk(); tile_kinds(); tile_kind_is_per_tensor(); tile_row_kind_arms(); From f94f8b02565a6e7fef3477eef054741976c8653f Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Wed, 11 Nov 2026 11:31:19 -0400 Subject: [PATCH 2/2] docs: Hy-MT2 on the tile reader alone is 1 of 5 against FastFlowLM, 4 of 5 with #1976's RoPE base Measured with #1976 merged locally. main runs hunyuan_v1_dense at RoPE base 10000 where alpha makes it about 1.116e7, so the row says the pack serves and is not yet the model. Co-Authored-By: Claude Fable 5.2 --- docs/npu.md | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/docs/npu.md b/docs/npu.md index 7100a4d4c..4ce3fe04d 100644 --- a/docs/npu.md +++ b/docs/npu.md @@ -1843,7 +1843,7 @@ engine#587 measured the route on it): | `qwen35` | `FastFlowLM/D3-*-NPU2` (Edge, Flash, Mini, Nano, Lite, Turbo, Ultra), `FastFlowLM/Qwen3.8-27B-NPU2`, `FastFlowLM/Qwen3.5-0.8B-NPU2` | hrx, npu | **served, and measured** (engine#587): the same `Qwen3.8-27B-NPU2` family runs on the NPU route — 0.2902 s/token (3.45 tok/s) and 19.55 tok/s of prefill on the sealed single-ELF ternary rung, 22.811 s to a first token with the weight load in it (`bench/npu-leaderboard/27b-t2s2-*-20261106.json`) — int8 Q4NX decodes at 63.1 s/token and takes 145.7 s to a first token (`bench/npu-leaderboard/27b-*.json`, 2026-10-27), a 5-token prefill chunk is 58.14 s, and the ternary sealed-ELF rung on the same container is 1.72 s/token with 3/3 text-identical output (engine#697, closed by #728). Quality on the same route: **GSM8K 97.5 % (39/40)** with greedy decoding, **6 of the 40 answers cut off by the 512-token budget and exactly 1 of those 6 also wrong** — so among answers that finished, 34/34 were right (`27b-gsm8k-20261107`); and the per-token cost at those answer lengths is 0.3857 s/token against 0.2902 on a 13-token prompt, i.e. it grows ~1.33x with context (`27b-decode-cot-20261107`). What a stock install still needs is a kernel set for the model's geometry — none ships in this repository — so the row read `not served` until the route itself was measured | | `dflash` | `FastFlowLM/Nemotron-3.5-Bolt-E4B-DFlash-NPU2` | hrx | not served: a draft head, not a decoder; "The draft loop, and whether to serve it" below covers what the HRX route does with it | | `gemma4` | `FastFlowLM/Nemotron-3.5-Bolt-E4B-NPU2`, `FastFlowLM/Gemma4-12B-IT-NPU2` | hrx | not served: no gemma4 model type or kernels in the route | -| `hunyuan-dense` | `FastFlowLM/Hy-MT2-1.8B-NPU2` | hrx, npu | **served and measured** on the per-op route ("FastFlowLM's own containers" below) | +| `hunyuan-dense` | `FastFlowLM/Hy-MT2-1.8B-NPU2` | hrx, npu | loads and answers on the per-op route once its tile is read; its RoPE base and q/k-norm order are engine#1976 ("FastFlowLM's own containers" below) | | `gemma3` | `FastFlowLM/medgemma-1.5-4b-it-NPU2` | hrx | not served | | `lfm2` | `FastFlowLM/LFM2-2.6B-NPU2`, `FastFlowLM/LFM2-1.2B-NPU2`, `FastFlowLM/LFM2.5-1.2B-NPU2` | hrx, npu | `LFM2.5-1.2B-NPU2` **served and measured** on the per-op route ("FastFlowLM's own containers" below); `LFM2-1.2B-NPU2` serves too (one paired ticket, same section); `LFM2-2.6B-NPU2` was not run | | `gpt-oss` | `FastFlowLM/GPT-OSS-20B-NPU2`, `FastFlowLM/GPT-OSS-Safeguard-20b-NPU2` | hrx | not served | @@ -1900,7 +1900,7 @@ does on a geometry that has a sealed set: `Qwen3-1.7B-NPU2` is 3.0 tok/s here an | `Llama-3.2-3B-NPU2` | `llama` | identical, `Paris.` | 1.7 to 1.8 tok/s | | `Phi4-mini-Instruct-NPU2` | `phi3` | identical, `Paris`, once the turn end is read from the template (below) | 1.4 to 1.5 tok/s | | `Qwen3-VL-4B-Instruct-NPU2`, text tower only | `qwen3` | identical, `Paris` | 1.4 to 1.5 tok/s | -| `Hy-MT2-1.8B-NPU2` | `hunyuan_v1_dense` | identical, `Paris`; five translations against FastFlowLM's runtime below | 2.0 to 2.8 tok/s | +| `Hy-MT2-1.8B-NPU2` | `hunyuan_v1_dense` | identical, `Paris`; serves, and needs engine#1976 to be the model (below) | 2.0 to 2.8 tok/s | | `LFM2.5-1.2B-NPU2` | `lfm2`, hybrid: 10 conv layers of 16 | identical, `Paris`; against FastFlowLM's runtime below | 3.5 to 4.6 tok/s | | `LFM2-1.2B-NPU2` | `lfm2`, hybrid: 10 conv layers of 16 | `Paris`; one request per prompt, against FastFlowLM's runtime below | 3.7 tok/s | @@ -1976,14 +1976,24 @@ kind, so nothing is read off the data. The kernel directory is four shapes that packs (`QKV K2048 N3072`, `O K2048 N2048`, `GU K2048 N12288`, `D K6144 N2048`); q/k norm and the template (`chat_template.jinja`) are read from the pack. -Measured 2026-11-11, same `model.q4nx` (sha256 `a992683e04c2110e…`) for both runtimes, one ticket: five -translation requests (into German, French, Spanish, Chinese and English), all five answered with a correct -translation by both; one byte-identical (`Où se trouve la gare de train la plus proche ?`), four worded -differently (`Me gustaría una taza de café…` here, `Quisiera una taza de café…` there). The host float run of the -first request gives this route's wording, so the array agrees with float arithmetic on these weights as it does -for LFM2; no reference independent of both runtimes was available for this model, so the row does not say whose -wording is the model's. Three requests of the known-answer prompt returned `Paris` three times. Decode 2.0 to -2.8 tok/s (32 layers on the per-op route) against FastFlowLM's 50 to 53. +Measured 2026-11-11, same `model.q4nx` (sha256 `a992683e04c2110e…`) for both runtimes, five translation requests +(into German, French, Spanish, Chinese and English), FastFlowLM 1.0.12's runtime then this route in one ticket, +twice: + +| this route's code | byte-identical to FastFlowLM's answer | +|---|---| +| the tile reader alone (this change on `main`) | 1 of 5 | +| the tile reader plus the Hunyuan RoPE base and q/k-norm order of engine#1976 (merged locally for the measurement) | 4 of 5 | + +So reading the tiles makes the pack serve, and it is **not yet this model as its authors define it**: `main` runs +every `hunyuan_v1_dense` pack at RoPE base 10000 where the family's `rope_scaling.alpha` of 1000 makes it +10000 x 1000^(128/126), about 1.116e7, at every position, and normalises q/k before RoPE where the family does it +after (engine#1889). All five answers were correct translations either way, which is why fluency was no evidence: +`Das Wetter ist heute schön. Wir werden zum Spazierengehen gehen.` on the wrong base, +`Das Wetter ist heute schön, und wir gehen spazieren.` on the right one and from FastFlowLM. The one request still +different with both changes is `Me gustaría una taza de café…` here against `Quisiera una taza de café…` there. +Three requests of the known-answer prompt returned `Paris` three times, and decode was 2.0 to 2.8 tok/s (32 layers +on the per-op route) against FastFlowLM's 50 to 53; both of those were taken on the tile reader alone. Not established by any row here: that a pack goes through the importer (`npu-pack` still reports these containers as recognised, not importable), anything about a vision or audio tower, and any pack whose geometry