Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 32 additions & 1 deletion docs/npu.md
Original file line number Diff line number Diff line change
Expand Up @@ -1843,7 +1843,7 @@ engine#587 measured the route on it):
| `qwen35` | `FastFlowLM/D3-*-NPU2` (Edge, Flash, Mini, Nano, Lite, Turbo, Ultra), `FastFlowLM/Qwen3.8-27B-NPU2`, `FastFlowLM/Qwen3.5-0.8B-NPU2` | hrx, npu | **served, and measured** (engine#587): the same `Qwen3.8-27B-NPU2` family runs on the NPU route — 0.2902 s/token (3.45 tok/s) and 19.55 tok/s of prefill on the sealed single-ELF ternary rung, 22.811 s to a first token with the weight load in it (`bench/npu-leaderboard/27b-t2s2-*-20261106.json`) — int8 Q4NX decodes at 63.1 s/token and takes 145.7 s to a first token (`bench/npu-leaderboard/27b-*.json`, 2026-10-27), a 5-token prefill chunk is 58.14 s, and the ternary sealed-ELF rung on the same container is 1.72 s/token with 3/3 text-identical output (engine#697, closed by #728). Quality on the same route: **GSM8K 97.5 % (39/40)** with greedy decoding, **6 of the 40 answers cut off by the 512-token budget and exactly 1 of those 6 also wrong** — so among answers that finished, 34/34 were right (`27b-gsm8k-20261107`); and the per-token cost at those answer lengths is 0.3857 s/token against 0.2902 on a 13-token prompt, i.e. it grows ~1.33x with context (`27b-decode-cot-20261107`). What a stock install still needs is a kernel set for the model's geometry — none ships in this repository — so the row read `not served` until the route itself was measured |
| `dflash` | `FastFlowLM/Nemotron-3.5-Bolt-E4B-DFlash-NPU2` | hrx | not served: a draft head, not a decoder; "The draft loop, and whether to serve it" below covers what the HRX route does with it |
| `gemma4` | `FastFlowLM/Nemotron-3.5-Bolt-E4B-NPU2`, `FastFlowLM/Gemma4-12B-IT-NPU2` | hrx | not served: no gemma4 model type or kernels in the route |
| `hunyuan-dense` | `FastFlowLM/Hy-MT2-1.8B-NPU2` | hrx | not served |
| `hunyuan-dense` | `FastFlowLM/Hy-MT2-1.8B-NPU2` | hrx, npu | loads and answers on the per-op route once its tile is read; its RoPE base and q/k-norm order are engine#1976 ("FastFlowLM's own containers" below) |
| `gemma3` | `FastFlowLM/medgemma-1.5-4b-it-NPU2` | hrx | not served |
| `lfm2` | `FastFlowLM/LFM2-2.6B-NPU2`, `FastFlowLM/LFM2-1.2B-NPU2`, `FastFlowLM/LFM2.5-1.2B-NPU2` | hrx, npu | `LFM2.5-1.2B-NPU2` **served and measured** on the per-op route ("FastFlowLM's own containers" below); `LFM2-1.2B-NPU2` serves too (one paired ticket, same section); `LFM2-2.6B-NPU2` was not run |
| `gpt-oss` | `FastFlowLM/GPT-OSS-20B-NPU2`, `FastFlowLM/GPT-OSS-Safeguard-20b-NPU2` | hrx | not served |
Expand Down Expand Up @@ -1900,6 +1900,7 @@ does on a geometry that has a sealed set: `Qwen3-1.7B-NPU2` is 3.0 tok/s here an
| `Llama-3.2-3B-NPU2` | `llama` | identical, `Paris.` | 1.7 to 1.8 tok/s |
| `Phi4-mini-Instruct-NPU2` | `phi3` | identical, `Paris`, once the turn end is read from the template (below) | 1.4 to 1.5 tok/s |
| `Qwen3-VL-4B-Instruct-NPU2`, text tower only | `qwen3` | identical, `Paris` | 1.4 to 1.5 tok/s |
| `Hy-MT2-1.8B-NPU2` | `hunyuan_v1_dense` | identical, `Paris`; serves, and needs engine#1976 to be the model (below) | 2.0 to 2.8 tok/s |
| `LFM2.5-1.2B-NPU2` | `lfm2`, hybrid: 10 conv layers of 16 | identical, `Paris`; against FastFlowLM's runtime below | 3.5 to 4.6 tok/s |
| `LFM2-1.2B-NPU2` | `lfm2`, hybrid: 10 conv layers of 16 | `Paris`; one request per prompt, against FastFlowLM's runtime below | 3.7 tok/s |

Expand Down Expand Up @@ -1964,6 +1965,36 @@ does not carry the three-identical-answers condition.
reasoning, so there is no answer text to compare and no row; the reasoning differs from FastFlowLM's from its
first words.

**`Hy-MT2-1.8B-NPU2`: a third tile.** This pack's projections are stored 4,608 bytes to a tile where every
other pack on the box uses 5,120, and no reader knew the size, so it was refused at the first dequant. The tile is
256 bf16 scales (index `g*32 + r`) followed by 4,096 code bytes in column order (`c*16 + r/2`, even row in the low
nibble), and a weight is the two's-complement nibble times its scale: the symmetric reading of the LFM2 packs with
the zero-point block left out and the codes in the Q4K chunk's order. It was settled the same way, against the
pack's own bf16 embedding (its `lm_head` is the quantised copy): mean 0.244 of a step, maximum 1.000, and the
eleven other combinations of scale order, code order and code sign are 3 to 8 steps off. Here the size names the
kind, so nothing is read off the data. The kernel directory is four shapes that were already built for other
packs (`QKV K2048 N3072`, `O K2048 N2048`, `GU K2048 N12288`, `D K6144 N2048`); q/k norm and the template
(`chat_template.jinja`) are read from the pack.

Measured 2026-11-11, same `model.q4nx` (sha256 `a992683e04c2110e…`) for both runtimes, five translation requests
(into German, French, Spanish, Chinese and English), FastFlowLM 1.0.12's runtime then this route in one ticket,
twice:

| this route's code | byte-identical to FastFlowLM's answer |
|---|---|
| the tile reader alone (this change on `main`) | 1 of 5 |
| the tile reader plus the Hunyuan RoPE base and q/k-norm order of engine#1976 (merged locally for the measurement) | 4 of 5 |

So reading the tiles makes the pack serve, and it is **not yet this model as its authors define it**: `main` runs
every `hunyuan_v1_dense` pack at RoPE base 10000 where the family's `rope_scaling.alpha` of 1000 makes it
10000 x 1000^(128/126), about 1.116e7, at every position, and normalises q/k before RoPE where the family does it
after (engine#1889). All five answers were correct translations either way, which is why fluency was no evidence:
`Das Wetter ist heute schön. Wir werden zum Spazierengehen gehen.` on the wrong base,
`Das Wetter ist heute schön, und wir gehen spazieren.` on the right one and from FastFlowLM. The one request still
different with both changes is `Me gustaría una taza de café…` here against `Quisiera una taza de café…` there.
Three requests of the known-answer prompt returned `Paris` three times, and decode was 2.0 to 2.8 tok/s (32 layers
on the per-op route) against FastFlowLM's 50 to 53; both of those were taken on the tile reader alone.

Not established by any row here: that a pack goes through the importer (`npu-pack` still reports these
containers as recognised, not importable), anything about a vision or audio tower, and any pack whose geometry
has no kernel set on the box. Of the 57 `*-NPU2` repos with a readable geometry on 2026-11-11, 19 had all four
Expand Down
2 changes: 1 addition & 1 deletion npu/forward/include/pad_head_rows.h
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
// tiles gives 64,064, which fails the bound and falls through to the row spelling (review of #2026 at 63a47be).
// Anything the three spellings cannot place returns NV, and the caller's own dequant then refuses it by name.
inline bool head_chunk_bytes(int64_t b) {
return b == Q4NX_TILE_BYTES || b == 6144 /* q4_1f, the imported-container tile */ || b == Q4NX_CHUNK_Q4K || b == Q4NX_CHUNK_Q8;
return b == Q4NX_TILE_BYTES || b == 6144 /* q4_1f, the imported-container tile */ || b == Q4NX_CHUNK_Q4K || b == Q4NX_CHUNK_Q8 || b == Q4NX_CHUNK_Q4S;
}
inline int padded_head_rows(const int64_t* shape, int ndim, int kpad, int NV) {
if (!shape || kpad <= 0 || kpad % Q4NX_TILE_COLS != 0 || NV <= 0) return NV;
Expand Down
10 changes: 10 additions & 0 deletions npu/forward/include/q4nx_dequant.h
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,16 @@ extern "C" {
* the cross-validation against the checkpoint's own `.bf16` twin. */
#define Q4NX_CHUNK_Q4K 4736
#define Q4NX_CHUNK_Q8 8704
/* The symmetric 4-bit chunk with no zero-point block (FastFlowLM lists the pack as "Q4_0"; Hy-MT2-1.8B-NPU2):
* scales [256] bf16 @[0,512) index g*32 + r (group-major, as in the 5120-byte tile)
* codes [256][16] @[512,4608) byte c*16 + r/2, even row = low nibble (column-major, as in the Q4K chunk)
* value[r][c] = (code >= 8 ? code - 16 : code) * scale[g][r]
* Settled by the tie test on that pack (2026-11-11): its lm_head is the quantised copy of its own bf16
* embed_tokens, and over the first 1024 rows |dequant - embedding| / |scale| has mean 0.244 and maximum 1.000 with
* this reading; the eleven other combinations of scale order, code order and code sign are 3 to 8 steps off.
* The size names the kind (no tile of another kind is 4608 bytes), so nothing is inferred from the data. */
#define Q4NX_CHUNK_Q4S 4608
void q4nx_dequant_tile_q4s(const uint8_t* chunk, float* out, int ld);

/* Q4_K chunk: 32 rows x 256 K, 4736 B.
* scales[8][32] uint8 @[0,256) index g*32 + r
Expand Down
30 changes: 25 additions & 5 deletions npu/forward/src/q4nx_dequant.c
Original file line number Diff line number Diff line change
Expand Up @@ -136,12 +136,26 @@ void q4nx_dequant_tile_q8(const uint8_t* t, float* out, int ld) {
}
}

void q4nx_dequant_tile_q4s(const uint8_t* t, float* out, int ld) {
const uint8_t* sc_b = t; /* [256] bf16, index g*32 + r */
const uint8_t* qs = t + 512; /* [256][16], byte c*16 + r/2 */
for (int r = 0; r < Q4NX_TILE_ROWS; r++) {
for (int c = 0; c < Q4NX_TILE_COLS; c++) {
const int g = c / Q4NX_GROUP_SIZE;
const uint8_t byte = qs[c * 16 + r / 2];
int code = (r % 2 == 0) ? (byte & 0xF) : (byte >> 4);
if (code >= 8) code -= 16; /* two's complement nibble */
out[(size_t)r * ld + c] = (float)code * bf16_to_f32(le16(sc_b + 2 * (g * Q4NX_TILE_ROWS + r)));
}
}
}

int q4nx_dequant_tensor_chunked(const uint8_t* data, size_t nbytes, int rows,
int cols, int chunk_bytes, float* out) {
if (!data || !out || rows <= 0 || cols <= 0) return -1;
if (rows % Q4NX_TILE_ROWS || cols % Q4NX_TILE_COLS) return -1;
if (chunk_bytes != Q4NX_TILE_BYTES && chunk_bytes != Q4NX_CHUNK_Q4K &&
chunk_bytes != Q4NX_CHUNK_Q8)
chunk_bytes != Q4NX_CHUNK_Q8 && chunk_bytes != Q4NX_CHUNK_Q4S)
return -1;
const int ntr = rows / Q4NX_TILE_ROWS, ntc = cols / Q4NX_TILE_COLS;
if (nbytes < (size_t)ntr * (size_t)ntc * (size_t)chunk_bytes) return -1;
Expand All @@ -161,6 +175,8 @@ int q4nx_dequant_tensor_chunked(const uint8_t* data, size_t nbytes, int rows,
q4nx_dequant_tile_q4k(c, tmp, Q4NX_TILE_COLS);
else if (chunk_bytes == Q4NX_CHUNK_Q8)
q4nx_dequant_tile_q8(c, tmp, Q4NX_TILE_COLS);
else if (chunk_bytes == Q4NX_CHUNK_Q4S)
q4nx_dequant_tile_q4s(c, tmp, Q4NX_TILE_COLS);
else
q4nx_dequant_tile_as(c, tmp, Q4NX_TILE_COLS, sym);
for (int r = 0; r < Q4NX_TILE_ROWS; r++)
Expand All @@ -181,7 +197,7 @@ int q4nx_dequant_tile_row(const uint8_t* data, size_t nbytes, int rows, int cols
if (r < 0 || r >= rows) return -1;
if (rows % Q4NX_TILE_ROWS || cols % Q4NX_TILE_COLS) return -1;
if (chunk_bytes != Q4NX_TILE_BYTES && chunk_bytes != Q4NX_CHUNK_Q4K &&
chunk_bytes != Q4NX_CHUNK_Q8)
chunk_bytes != Q4NX_CHUNK_Q8 && chunk_bytes != Q4NX_CHUNK_Q4S)
return -1;
const int ntr = rows / Q4NX_TILE_ROWS, ntc = cols / Q4NX_TILE_COLS;
if (nbytes < (size_t)ntr * (size_t)ntc * (size_t)chunk_bytes) return -1;
Expand All @@ -199,6 +215,8 @@ int q4nx_dequant_tile_row(const uint8_t* data, size_t nbytes, int rows, int cols
q4nx_dequant_tile_q4k(c, tmp, Q4NX_TILE_COLS);
else if (chunk_bytes == Q4NX_CHUNK_Q8)
q4nx_dequant_tile_q8(c, tmp, Q4NX_TILE_COLS);
else if (chunk_bytes == Q4NX_CHUNK_Q4S)
q4nx_dequant_tile_q4s(c, tmp, Q4NX_TILE_COLS);
else
q4nx_dequant_tile_as(c, tmp, Q4NX_TILE_COLS, sym);
memcpy(out + (size_t)tc * Q4NX_TILE_COLS, tmp + (size_t)rr * Q4NX_TILE_COLS,
Expand All @@ -210,8 +228,8 @@ int q4nx_dequant_tile_row(const uint8_t* data, size_t nbytes, int rows, int cols
int q4nx_tile_row_kind(size_t nbytes, int rows, int cols) {
if (rows <= 0 || cols <= 0 || rows % Q4NX_TILE_ROWS || cols % Q4NX_TILE_COLS) return 0;
const size_t tiles = (size_t)(rows / Q4NX_TILE_ROWS) * (size_t)(cols / Q4NX_TILE_COLS);
const int kinds[3] = {Q4NX_TILE_BYTES, Q4NX_CHUNK_Q4K, Q4NX_CHUNK_Q8};
for (int i = 0; i < 3; i++)
const int kinds[4] = {Q4NX_TILE_BYTES, Q4NX_CHUNK_Q4K, Q4NX_CHUNK_Q8, Q4NX_CHUNK_Q4S};
for (int i = 0; i < 4; i++)
if (nbytes == tiles * (size_t)kinds[i]) return kinds[i];
return 0;
}
Expand Down Expand Up @@ -262,7 +280,7 @@ int q4nx_dequant_tensor_packed(const uint8_t* data, size_t nbytes, int rows, int
if (rows_packed < rows || cols_packed < cols) return -1;
if (rows_packed % Q4NX_TILE_ROWS || cols_packed % Q4NX_TILE_COLS) return -1;
if (chunk_bytes != Q4NX_TILE_BYTES && chunk_bytes != Q4NX_CHUNK_Q4K &&
chunk_bytes != Q4NX_CHUNK_Q8)
chunk_bytes != Q4NX_CHUNK_Q8 && chunk_bytes != Q4NX_CHUNK_Q4S)
return -1;
if (rows_packed == rows && cols_packed == cols)
return q4nx_dequant_tensor_chunked(data, nbytes, rows, cols, chunk_bytes, out);
Expand All @@ -285,6 +303,8 @@ int q4nx_dequant_tensor_packed(const uint8_t* data, size_t nbytes, int rows, int
q4nx_dequant_tile_q4k(c, tmp, Q4NX_TILE_COLS);
else if (chunk_bytes == Q4NX_CHUNK_Q8)
q4nx_dequant_tile_q8(c, tmp, Q4NX_TILE_COLS);
else if (chunk_bytes == Q4NX_CHUNK_Q4S)
q4nx_dequant_tile_q4s(c, tmp, Q4NX_TILE_COLS);
else
q4nx_dequant_tile_as(c, tmp, Q4NX_TILE_COLS, sym);
for (int r = 0; r < Q4NX_TILE_ROWS; r++)
Expand Down
2 changes: 1 addition & 1 deletion npu/forward/src/q4nx_forward.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,7 @@ static int dequant_any(const void* data, size_t nbytes, int rows, int cols, floa
const size_t n = (size_t)(rows / Q4NX_TILE_ROWS) * (size_t)(cols / Q4NX_TILE_COLS);
if (n == 0) return -1;
const int cb = (int)(nbytes / n);
if (cb == Q4NX_TILE_BYTES || cb == Q4NX_CHUNK_Q4K || cb == Q4NX_CHUNK_Q8)
if (cb == Q4NX_TILE_BYTES || cb == Q4NX_CHUNK_Q4K || cb == Q4NX_CHUNK_Q8 || cb == Q4NX_CHUNK_Q4S)
return q4nx_dequant_tensor_chunked((const uint8_t*)data, nbytes, rows, cols, cb, out);
// The container may store this weight on a wider grid than the model's dimensions (engine#1097).
// Read it on the container's grid and write only the model's [rows, cols], so buffers, row
Expand Down
2 changes: 1 addition & 1 deletion npu/model.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ bool table_holds_vocab(const Tensor& t, int64_t vocab, int64_t hidden, std::stri
}
if (t.dtype != "I8") return true;
if (t.bytes == uint64_t(vocab) * uint64_t(hidden)) return true; // plain int8 + per-32-column scale (Gemma4)
const auto is_chunk = [](int64_t b) { return b == Q4NX_TILE_BYTES || b == Q4NX_CHUNK_Q4K || b == Q4NX_CHUNK_Q8; };
const auto is_chunk = [](int64_t b) { return b == Q4NX_TILE_BYTES || b == Q4NX_CHUNK_Q4K || b == Q4NX_CHUNK_Q8 || b == Q4NX_CHUNK_Q4S; };
int64_t r = -1;
if (t.shape.size() == 3 && is_chunk(t.shape[2]) && t.shape[1] == col_tiles) r = t.shape[0] * Q4NX_TILE_ROWS;
else if (t.shape.size() == 2 && is_chunk(t.shape[1]) && t.shape[0] % col_tiles == 0)
Expand Down
Loading
Loading