Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1643,6 +1643,14 @@ target_include_directories(q4nx_mrope_test PRIVATE npu/forward/include)
target_link_libraries(q4nx_mrope_test PRIVATE m)
add_test(NAME q4nx_mrope COMMAND q4nx_mrope_test)
add_dependencies(npu_sampling_test q4nx_mrope_test)
# Which models get the per-head q/k RMSNorm: config.json's use_qk_norm when the config states it, else the
# model_type table (engine#1889: Hunyuan declares it under a model_type the table has no row for, so its gains were
# loaded and never applied). Plain C, links the semantics TU directly, no model and no device.
add_executable(q4nx_qk_norm_test tests/q4nx_qk_norm_test.c npu/forward/src/q4nx_semantics.c)
target_include_directories(q4nx_qk_norm_test PRIVATE npu/forward/include)
target_link_libraries(q4nx_qk_norm_test PRIVATE m)
add_test(NAME q4nx_qk_norm COMMAND q4nx_qk_norm_test)
add_dependencies(npu_sampling_test q4nx_qk_norm_test)
add_executable(q4nx_layer_kinds_test tests/q4nx_layer_kinds_test.c npu/forward/src/q4nx_semantics.c)
target_include_directories(q4nx_layer_kinds_test PRIVATE npu/forward/include)
target_link_libraries(q4nx_layer_kinds_test PRIVATE m)
Expand Down Expand Up @@ -1950,6 +1958,17 @@ if(ONEBIT_NPU_HOST AND NOT ONEBIT_NPU AND NOT WIN32 AND Python3_Interpreter_FOUN
COMMAND ${Python3_EXECUTABLE} ${CMAKE_SOURCE_DIR}/tests/batch_check_state_ci.py
$<TARGET_FILE:batch_check_state_test>)
set_tests_properties(batch_check_state PROPERTIES SKIP_RETURN_CODE 77 TIMEOUT 120)
# The same test on an LFM2-shaped hybrid the script fabricates (conv, attention, conv): the only container with
# short-convolution layers CI loads through the forward (engine#1786).
add_test(NAME batch_check_state_lfm2
COMMAND ${Python3_EXECUTABLE} ${CMAKE_SOURCE_DIR}/tests/batch_check_state_ci.py
$<TARGET_FILE:batch_check_state_test> lfm2)
set_tests_properties(batch_check_state_lfm2 PROPERTIES SKIP_RETURN_CODE 77 TIMEOUT 120)
# A fabricated qwen3_5 hybrid (a linear-attention layer, then attention): load only, see the script's header.
add_test(NAME batch_check_state_qwen35_load
COMMAND ${Python3_EXECUTABLE} ${CMAKE_SOURCE_DIR}/tests/batch_check_state_ci.py
$<TARGET_FILE:batch_check_state_test> qwen3_5)
set_tests_properties(batch_check_state_qwen35_load PROPERTIES SKIP_RETURN_CODE 77 TIMEOUT 120)
endif()

# Private NPU routes (docs/npu.md, "Private routes"): closed-source add-ons in a checkout of
Expand Down
50 changes: 49 additions & 1 deletion docs/npu.md
Original file line number Diff line number Diff line change
Expand Up @@ -877,6 +877,25 @@ forward route, which reads the block from `config.json`: `rope_type: "llama3"`
declared-but-unusable `llama3` block, and every other declared type, at `1bit serve` /
`forward-serve` load time rather than at request time.

**Hunyuan (`hunyuan_v1_dense`) differs from the other q/k-norm families in four places: three in
the forward, which reads them from the model rather than from a family list where it can, and one in
the importer**
([engine#1889](https://github.com/1bit-MONSTER/engine/issues/1889)). Each was measured against
`Hunyuan-7B-Instruct`'s own pack:

| what | rule | where it comes from |
|---|---|---|
| whether q and k are normalised | `use_qk_norm` in `config.json` decides when the config states it; the `qwen3*`/`gemma*` table answers only for configs that do not | the config key. The table has no Hunyuan row, so its gains were loaded and never applied |
| the RoPE base | `rope_scaling: {type: "dynamic", alpha: A}` is a **fixed** base `theta * A^(dim/(dim-2))` at every position (11,158,840 for theta 10000, alpha 1000, dim 128). Without `alpha` the block is the position-dependent NTK rule | the graph's `cos_cache`/`sin_cache` in `model.onnx.data` match that base to 0.00024; plain theta 10000 is off by up to 1.99. `1bit npu-pack import` tests this base against the pack's cache first and prints `note: RoPE base ... verified`; before, it warned that no plain theta explained the cache |
| the order of norm and RoPE | RoPE first, then the per-head RMSNorm (Qwen3 and Gemma normalise first) | the graph: `q_proj -> RotaryEmbedding -> q_norm -> GroupQueryAttention` in every layer |
| the order of the attention tiles in the pack (importer) | q tiles first, then k, then v. Every other family's block is k, q, v. `1bit npu-pack import` picks the order with the block shape that carries the packed q/k gains | measured: with k, q, v the imported model predicts 26 of 77 next tokens of a fixed text, with q, k, v 59 (the two projections were being read as each other) |

A model whose config says q and k are normalised and whose container has no
`self_attn.q_norm.weight` is refused at load: the gain would read as zeros and zero every head's q.
That is the shape of a Hunyuan container imported before the importer carried the gains, so the
answer is to re-import it. A Hunyuan container imported before the tile order was right has
the gains and still answers filler words; re-import that one too.

| GGUF type | To q4_1 | Error |
|---|---|---|
| Q4_0 | scale `d`, zero `-8d`, codes unchanged | bf16 rounding of scale and zero |
Expand Down Expand Up @@ -1843,7 +1862,7 @@ engine#587 measured the route on it):
| `qwen35` | `FastFlowLM/D3-*-NPU2` (Edge, Flash, Mini, Nano, Lite, Turbo, Ultra), `FastFlowLM/Qwen3.8-27B-NPU2`, `FastFlowLM/Qwen3.5-0.8B-NPU2` | hrx, npu | **served, and measured** (engine#587): the same `Qwen3.8-27B-NPU2` family runs on the NPU route — 0.2902 s/token (3.45 tok/s) and 19.55 tok/s of prefill on the sealed single-ELF ternary rung, 22.811 s to a first token with the weight load in it (`bench/npu-leaderboard/27b-t2s2-*-20261106.json`) — int8 Q4NX decodes at 63.1 s/token and takes 145.7 s to a first token (`bench/npu-leaderboard/27b-*.json`, 2026-10-27), a 5-token prefill chunk is 58.14 s, and the ternary sealed-ELF rung on the same container is 1.72 s/token with 3/3 text-identical output (engine#697, closed by #728). Quality on the same route: **GSM8K 97.5 % (39/40)** with greedy decoding, **6 of the 40 answers cut off by the 512-token budget and exactly 1 of those 6 also wrong** — so among answers that finished, 34/34 were right (`27b-gsm8k-20261107`); and the per-token cost at those answer lengths is 0.3857 s/token against 0.2902 on a 13-token prompt, i.e. it grows ~1.33x with context (`27b-decode-cot-20261107`). What a stock install still needs is a kernel set for the model's geometry — none ships in this repository — so the row read `not served` until the route itself was measured |
| `dflash` | `FastFlowLM/Nemotron-3.5-Bolt-E4B-DFlash-NPU2` | hrx | not served: a draft head, not a decoder; "The draft loop, and whether to serve it" below covers what the HRX route does with it |
| `gemma4` | `FastFlowLM/Nemotron-3.5-Bolt-E4B-NPU2`, `FastFlowLM/Gemma4-12B-IT-NPU2` | hrx | not served: no gemma4 model type or kernels in the route |
| `hunyuan-dense` | `FastFlowLM/Hy-MT2-1.8B-NPU2` | hrx | not served |
| `hunyuan-dense` | `FastFlowLM/Hy-MT2-1.8B-NPU2` | hrx, npu | **served and measured** on the per-op route ("FastFlowLM's own containers" below) |
| `gemma3` | `FastFlowLM/medgemma-1.5-4b-it-NPU2` | hrx | not served |
| `lfm2` | `FastFlowLM/LFM2-2.6B-NPU2`, `FastFlowLM/LFM2-1.2B-NPU2`, `FastFlowLM/LFM2.5-1.2B-NPU2` | hrx, npu | `LFM2.5-1.2B-NPU2` **served and measured** on the per-op route ("FastFlowLM's own containers" below); `LFM2-1.2B-NPU2` serves too (one paired ticket, same section); `LFM2-2.6B-NPU2` was not run |
| `gpt-oss` | `FastFlowLM/GPT-OSS-20B-NPU2`, `FastFlowLM/GPT-OSS-Safeguard-20b-NPU2` | hrx | not served |
Expand Down Expand Up @@ -1900,6 +1919,7 @@ does on a geometry that has a sealed set: `Qwen3-1.7B-NPU2` is 3.0 tok/s here an
| `Llama-3.2-3B-NPU2` | `llama` | identical, `Paris.` | 1.7 to 1.8 tok/s |
| `Phi4-mini-Instruct-NPU2` | `phi3` | identical, `Paris`, once the turn end is read from the template (below) | 1.4 to 1.5 tok/s |
| `Qwen3-VL-4B-Instruct-NPU2`, text tower only | `qwen3` | identical, `Paris` | 1.4 to 1.5 tok/s |
| `Hy-MT2-1.8B-NPU2` | `hunyuan_v1_dense` | identical, `Paris`; 4 of 5 translations byte-identical to FastFlowLM's runtime (below) | 2.3 to 2.8 tok/s |
| `LFM2.5-1.2B-NPU2` | `lfm2`, hybrid: 10 conv layers of 16 | identical, `Paris`; against FastFlowLM's runtime below | 3.5 to 4.6 tok/s |
| `LFM2-1.2B-NPU2` | `lfm2`, hybrid: 10 conv layers of 16 | `Paris`; one request per prompt, against FastFlowLM's runtime below | 3.7 tok/s |

Expand Down Expand Up @@ -1964,6 +1984,34 @@ does not carry the three-identical-answers condition.
reasoning, so there is no answer text to compare and no row; the reasoning differs from FastFlowLM's from its
first words.

**`Hy-MT2-1.8B-NPU2`: a third tile.** This pack's projections are stored 4,608 bytes to a tile where every
other pack on the box uses 5,120, and no reader knew the size, so it was refused at the first dequant. The tile is
256 bf16 scales (index `g*32 + r`) followed by 4,096 code bytes in column order (`c*16 + r/2`, even row in the low
nibble), and a weight is the two's-complement nibble times its scale: the symmetric reading of the LFM2 packs with
the zero-point block left out and the codes in the Q4K chunk's order. It was settled the same way, against the
pack's own bf16 embedding (its `lm_head` is the quantised copy): mean 0.244 of a step, maximum 1.000, and the
eleven other combinations of scale order, code order and code sign are 3 to 8 steps off. Here the size names the
kind, so nothing is read off the data. The kernel directory is four shapes that were already built for other
packs (`QKV K2048 N3072`, `O K2048 N2048`, `GU K2048 N12288`, `D K6144 N2048`); q/k norm and the template
(`chat_template.jinja`) are read from the pack.

Measured 2026-11-11, same `model.q4nx` (sha256 `a992683e04c2110e…`) for both runtimes, five translation requests
(into German, French, Spanish, Chinese and English), FastFlowLM 1.0.12's runtime then this route in one ticket:

| this route's code | byte-identical to FastFlowLM's answer |
|---|---|
| the tile reader alone, on the semantics `main` had for this family | 1 of 5 |
| the tile reader with the family's RoPE base and q/k-norm order (engine#1889), which is this change | 4 of 5 |

Reading the tiles made the pack serve; it did not make it this model. `hunyuan_v1_dense` was run at RoPE base 10000
where the family's `rope_scaling.alpha` of 1000 makes it 10000 x 1000^(128/126), about 1.116e7, at every position,
and q/k were normalised before RoPE where the family does it after. All five answers were correct translations
either way, which is why fluency was no evidence: `Das Wetter ist heute schön. Wir werden zum Spazierengehen gehen.`
on the wrong base, `Das Wetter ist heute schön, und wir gehen spazieren.` on the right one and from FastFlowLM. The
one request still different is `Me gustaría una taza de café…` here against `Quisiera una taza de café…` there. At
the same code three requests of the known-answer prompt returned `Paris` three times, and decode was 2.3 to 2.8
tok/s (32 layers on the per-op route) against FastFlowLM's 50 to 53.

Not established by any row here: that a pack goes through the importer (`npu-pack` still reports these
containers as recognised, not importable), anything about a vision or audio tower, and any pack whose geometry
has no kernel set on the box. Of the 57 `*-NPU2` repos with a readable geometry on 2026-11-11, 19 had all four
Expand Down
2 changes: 1 addition & 1 deletion npu/forward/include/pad_head_rows.h
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
// tiles gives 64,064, which fails the bound and falls through to the row spelling (review of #2026 at 63a47be).
// Anything the three spellings cannot place returns NV, and the caller's own dequant then refuses it by name.
inline bool head_chunk_bytes(int64_t b) {
return b == Q4NX_TILE_BYTES || b == 6144 /* q4_1f, the imported-container tile */ || b == Q4NX_CHUNK_Q4K || b == Q4NX_CHUNK_Q8;
return b == Q4NX_TILE_BYTES || b == 6144 /* q4_1f, the imported-container tile */ || b == Q4NX_CHUNK_Q4K || b == Q4NX_CHUNK_Q8 || b == Q4NX_CHUNK_Q4S;
}
inline int padded_head_rows(const int64_t* shape, int ndim, int kpad, int NV) {
if (!shape || kpad <= 0 || kpad % Q4NX_TILE_COLS != 0 || NV <= 0) return NV;
Expand Down
10 changes: 10 additions & 0 deletions npu/forward/include/q4nx_dequant.h
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,16 @@ extern "C" {
* the cross-validation against the checkpoint's own `.bf16` twin. */
#define Q4NX_CHUNK_Q4K 4736
#define Q4NX_CHUNK_Q8 8704
/* The symmetric 4-bit chunk with no zero-point block (FastFlowLM lists the pack as "Q4_0"; Hy-MT2-1.8B-NPU2):
* scales [256] bf16 @[0,512) index g*32 + r (group-major, as in the 5120-byte tile)
* codes [256][16] @[512,4608) byte c*16 + r/2, even row = low nibble (column-major, as in the Q4K chunk)
* value[r][c] = (code >= 8 ? code - 16 : code) * scale[g][r]
* Settled by the tie test on that pack (2026-11-11): its lm_head is the quantised copy of its own bf16
* embed_tokens, and over the first 1024 rows |dequant - embedding| / |scale| has mean 0.244 and maximum 1.000 with
* this reading; the eleven other combinations of scale order, code order and code sign are 3 to 8 steps off.
* The size names the kind (no tile of another kind is 4608 bytes), so nothing is inferred from the data. */
#define Q4NX_CHUNK_Q4S 4608
void q4nx_dequant_tile_q4s(const uint8_t* chunk, float* out, int ld);

/* Q4_K chunk: 32 rows x 256 K, 4736 B.
* scales[8][32] uint8 @[0,256) index g*32 + r
Expand Down
16 changes: 16 additions & 0 deletions npu/forward/include/q4nx_rope.h
Original file line number Diff line number Diff line change
Expand Up @@ -69,10 +69,26 @@ typedef struct {
int on; /* config declared rope_type "dynamic" with usable numbers */
double factor; /* 1.0 on Hunyuan-0.5B -- a factor of 1 is a no-op until pos > original */
double original_max_position; /* 262144 on Hunyuan-0.5B */
/* engine#1889: Hunyuan's own form of "dynamic". Its rope_scaling block also carries `alpha` (1000.0 on every
* Hunyuan/HY-MT 1.9 pack), and when alpha is set the model's rotary embedding does not move the base with the
* position at all: it uses the FIXED base
*
* base = theta * alpha ^ (dim / (dim - 2))
*
* (HunYuanDenseV1RotaryEmbedding, "DynamicNTKAlphaRotary"). Measured against the pack rather than taken from the
* reference: Hunyuan-7B's model.onnx.data holds the graph's cos_cache/sin_cache (FP16 [32768, 64]), and they match
* base 10000 * 1000^(128/126) = 11,158,840 to 0.00024 at positions 1, 2, 7, 50 and 300, where plain theta 10000 is
* off by up to 1.99. Reading the block as factor-only made it a no-op, i.e. every position rotated by the wrong
* angle. 0 = the block has no alpha, and the factor rule above applies. */
double alpha;
} Q4nxRopeDynamic;

static inline double q4nx_rope_dynamic_base(double theta, const Q4nxRopeDynamic* d, double pos, int dim) {
if (!d || !d->on) return theta;
if (d->alpha > 0.0) { /* the alpha form: one fixed base, every position */
if (dim <= 2) return theta;
return theta * pow(d->alpha, (double)dim / (double)(dim - 2));
}
if (!(d->factor > 0.0) || !(d->original_max_position > 0.0) || dim <= 2) return theta;
if (!(pos >= d->original_max_position)) return theta; /* inside the window: plain rope, bit for bit */
const double scaled = d->factor * pos / d->original_max_position - (d->factor - 1.0);
Expand Down
20 changes: 19 additions & 1 deletion npu/forward/include/q4nx_semantics.h
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,16 @@ typedef struct {
* Source: model_type/architectures == qwen3_5*, resolved against the weights. */
int rmsnorm_offset_probe;
/* Per-head RMSNorm applied to q and k before RoPE.
* Source: presence of self_attn.q_norm.weight / k_norm.weight. */
* Source: config.json's `use_qk_norm` when the config states it (engine#1889: Hunyuan), else the
* model_type/architectures table (qwen3*, gemma*). The forward refuses a model whose flag is set and whose
* layers carry no self_attn.q_norm.weight, since normalising with an absent gain zeroes q. */
int qk_norm;
/* The q/k RMSNorm runs AFTER RoPE instead of before it (engine#1889). Qwen3 and Gemma normalise, then rotate;
* Hunyuan rotates, then normalises. With a per-dimension gain the two orders differ wherever a rotary pair's two
* gains differ, so the order is part of the model. Source: model_type/architectures (hunyuan*) -- no config key
* states it; the evidence is the pack's own graph, where every layer is
* q_proj -> RotaryEmbedding -> q_norm (SimplifiedLayerNormalization) -> GroupQueryAttention. */
int qk_norm_after_rope;
/* RoPE base. Source: rope_theta. */
float rope_theta;
/* Channels actually rotated; 0 means the full head_dim. Partial rotary
Expand Down Expand Up @@ -254,6 +262,16 @@ int q4nx_rmsnorm_offset_from_norm(const float* w, int n);
* off: a family the table knows keeps its answer whatever the tensors are. Exported so the rule is testable. */
int q4nx_semantics_qk_norm_from_tensors(Q4nxSemantics* s, const unsigned char* has_q, const unsigned char* has_k, int layers);

/* engine#1889's refusal, as a rule: with q/k norm on, a layer whose q gain is absent would normalise q to zero, so the
* container is refused. That is a statement about layers with SOFTMAX ATTENTION. A layer of another kind carries no
* q_norm because it has no q to normalise: ten of LFM2's sixteen layers are gated short convolutions, and 18 of
* Qwen3.5-2B's 24 are linear-attention (GDN) layers, layer 0 in both. Applied to every layer the rule refused every
* LFM2 pack and every qwen3_5 hybrid at load with "layer 0 has no self_attn.q_norm.weight".
* Returns 1 when layer l must be refused. Exported so the rule is testable with no model present. */
static inline int q4nx_qk_gain_missing(int qk_norm, int layer_has_softmax_attention, int q_gain_present) {
return qk_norm && layer_has_softmax_attention && !q_gain_present;
}

/* attention.mode for one container, from the two places that can state it (engine#796).

* A d3 pack declares the mask it was trained with in its own `decision_config.json`
Expand Down
Loading
Loading