Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1748,6 +1748,14 @@ target_include_directories(q4nx_gemma3_semantics_test PRIVATE npu/forward/includ
target_link_libraries(q4nx_gemma3_semantics_test PRIVATE m)
add_test(NAME q4nx_gemma3_semantics COMMAND q4nx_gemma3_semantics_test)
add_dependencies(npu_sampling_test q4nx_gemma3_semantics_test)
# LFM3 (engine#1786): "conv" in layer_types, the sandwich, the per-head gate, the zero-tail RoPE, and model_load's
# renaming of the upstream tensor names. Links the two C sources directly, no model and no device.
add_executable(q4nx_lfm3_semantics_test tests/q4nx_lfm3_semantics_test.cpp npu/forward/src/model.c
npu/forward/src/q4nx_semantics.c)
target_include_directories(q4nx_lfm3_semantics_test PRIVATE npu/forward/include)
target_link_libraries(q4nx_lfm3_semantics_test PRIVATE m)
add_test(NAME q4nx_lfm3_semantics COMMAND q4nx_lfm3_semantics_test)
add_dependencies(npu_sampling_test q4nx_lfm3_semantics_test)
add_dependencies(npu_sampling_test q4nx_qk_norm_test)
add_executable(q4nx_layer_kinds_test tests/q4nx_layer_kinds_test.c npu/forward/src/q4nx_semantics.c)
target_include_directories(q4nx_layer_kinds_test PRIVATE npu/forward/include)
Expand Down Expand Up @@ -2115,6 +2123,11 @@ if(ONEBIT_NPU_HOST AND NOT ONEBIT_NPU AND NOT WIN32 AND Python3_Interpreter_FOUN
COMMAND ${Python3_EXECUTABLE} ${CMAKE_SOURCE_DIR}/tests/batch_check_state_ci.py
$<TARGET_FILE:batch_check_state_test> lfm2)
set_tests_properties(batch_check_state_lfm2 PROPERTIES SKIP_RETURN_CODE 77 TIMEOUT 120)
# LFM3 (engine#1786): the upstream tensor names, "conv" in layer_types, sandwich norms and the per-head gate.
add_test(NAME batch_check_state_lfm3
COMMAND ${Python3_EXECUTABLE} ${CMAKE_SOURCE_DIR}/tests/batch_check_state_ci.py
$<TARGET_FILE:batch_check_state_test> lfm3)
set_tests_properties(batch_check_state_lfm3 PROPERTIES SKIP_RETURN_CODE 77 TIMEOUT 120)
# A fabricated qwen3_5 hybrid (a linear-attention layer, then attention), off the kernel's N grid on purpose.
add_test(NAME batch_check_state_qwen35
COMMAND ${Python3_EXECUTABLE} ${CMAKE_SOURCE_DIR}/tests/batch_check_state_ci.py
Expand Down
31 changes: 31 additions & 0 deletions docs/npu.md
Original file line number Diff line number Diff line change
Expand Up @@ -1998,6 +1998,37 @@ and against the pack's own embedding (`lm_head`: 0.9963, at most 0.57 of a step
the known-answer prompt `Paris` and explains Rayleigh scattering in coherent sentences. Before, it failed at load
with `chunk 4266 B is not a kind`. No kernel set for its shapes was built or run (the NPU is off).

**LFM3-1.7B-NPU2: the forward reads LFM3 (engine#1786).** LiquidAI's LFM3 is LFM2's hybrid of gated short
convolutions and attention, plus four rules from its `modeling_lfm3.py`. The forward now runs all four:

- `layer_types` names the conv layers `"conv"`, beside `full_attention` and `sliding_attention`. Each `"conv"` entry
has to match a layer whose conv tensors are in the container, and a mismatch either way is refused by name.
`"conv"` is an exact match: `conv_attention` is still an unknown kind.
- `use_sandwich_norm`: a norm on the conv or attention output before the residual add, and one on the MLP output.
It is the same pair the Gemma3 path runs (`extra_layer_norms` 2), and here the conv layers take it too.
- A per-head output gate. Every attention layer has `self_attn.gate_proj` `[heads, hidden]` (bf16 in the pack), and
each head's context is multiplied by `sigmoid(gate_proj . x)` before `o_proj`, `x` being the block's normed input.
- Zero-tail partial RoPE on the full layers. With `partial_rotary_factor` 0.5 the first 32 frequencies follow the
progression of the whole 128-channel head and the other 32 are zero, with `rotate_half` pairing across the whole
head. That is the rule Gemma4's "proportional" RoPE already had. The vendor GGUF writes the same rule as
`rope_freqs` (1 for 32 factors, 1e30 for the rest).

The pack keeps the upstream tensor names (`conv.*`, `operator_norm`, `ffn_norm`, `feed_forward.w1/w2/w3`,
`post_operator_norm`, `post_ffn_norm`, `embedding_norm`). `model_load` renames them to the canonical ones, and only
where the canonical name is absent (`model_alias_lfm_names`, `model.h`). `ffn_norm` is the MLP input's norm, so it
becomes `pre_feedforward_layernorm` in a sandwich layer and `post_attention_layernorm` otherwise.

Checked on the host, 2026-11-12, against a numpy transcription of `modeling_lfm3.py`. Run on the bf16 checkpoint,
the reference's last-position logits correlate 0.939 with ours. Run on the dequantized weights of the vendor's
`LFM3-1.7B-QAD-Q4_0` GGUF, which this pack's I8 tensors come from, they correlate 0.9995, with the same top 10.
The two references differ from each other by the same 0.94, so that gap is the quantization-aware weights, not the
arithmetic. Each rule matters on this prompt: with the gate off the bf16 reference falls to 0.907 against ours,
with the usual partial RoPE to 0.822 (and its top token changes), and without the sandwich to 0.134.
`batch_check_state_test` on the pack: batched equals sequential, rel-L1 0. The known-answer prompt, with thinking
off (the template's `<think></think>`), gives `Paris</think>Paris<|im_end|>`. The reference makes the same three
choices: `</think>` after the first `Paris`, then `Paris` again. Asked why the sky is blue, it explains Rayleigh
scattering in correct sentences (80 tokens, greedy). No kernel set for its shapes was built or run (the NPU is off).

**Per-head q/k normalisation is read from the container (engine#1788).** Whether the forward normalises q
and k per head used to be decided by `config.json`'s `model_type` alone: on for `qwen3*` and `gemma*`.
A pack of the same architecture under another name got it wrong in silence. FastFlowLM's MiniCPM-V 4.7
Expand Down
9 changes: 9 additions & 0 deletions npu/forward/include/model.h
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,15 @@ void* model_tensor_data(ModelWeights* mw, TensorDesc* desc);
// Used for architecture-specific tensors the fixed fields do not cover.
// (Distinct from the legacy model_find_tensor(name, mw), which returns an index.)
TensorDesc* model_tensor_by_name(ModelWeights* mw, const char* name);

// LFM3 packs (FastFlowLM/LFM3-1.7B-NPU2) keep the upstream Liquid names -- conv.*, operator_norm, ffn_norm,
// feed_forward.w1/w2/w3, post_operator_norm, post_ffn_norm, embedding_norm -- where the LFM2 packs carry the
// canonical ones (shortconv.*, input_layernorm, mlp.*, post_attention_layernorm, model.norm). This renames each
// to the canonical name, in place, and only where that name is not already taken. ffn_norm is the norm in front
// of the MLP, so it becomes pre_feedforward_layernorm in a layer that also has post_operator_norm (the sandwich
// form, where post_attention_layernorm is the attention/conv output's norm) and post_attention_layernorm
// otherwise. Returns the number of tensors renamed.
int model_alias_lfm_names(TensorDesc* tensors, int count);
// engine#1786 (item F1): three FastFlowLM NPU2 model sources spell the per-layer prefix `model.layer.` (singular)
// where this forward addresses `model.layers.`. Writes the OTHER spelling into `out` and returns 1 when `name` carries
// either prefix, 0 when it does not. Only the leading prefix is swapped, once: a caller that wants a fallback retries
Expand Down
4 changes: 4 additions & 0 deletions npu/forward/include/q4nx_forward.h
Original file line number Diff line number Diff line change
Expand Up @@ -647,6 +647,10 @@ class Q4nxNpuForward {
// Layout and maths: docs/qwen35-4b-weight-format.md.
int lin_kd_ = 0, lin_vd_ = 0; // key_dim = nkh*khd, value_dim = nvh*vhd
int attn_gate_ = 0; // qwen3_5 attn_output_gate
// LFM3's per-head output gate, [NH][H] per attention layer (empty for conv layers and other models):
// ctx of head h *= sigmoid(head_gate_[l][h] . xn), xn the operator-normed layer input.
std::vector<std::vector<float>> head_gate_;
void apply_head_gate(int l, const float* xn, float* ctx, int hd) const;
int lin_nkh_ = 0, lin_nvh_ = 0; // head counts
int lin_khd_ = 0, lin_vhd_ = 0; // head dims
int lin_ck_ = 4; // depthwise conv kernel
Expand Down
15 changes: 12 additions & 3 deletions npu/forward/include/q4nx_semantics.h
Original file line number Diff line number Diff line change
Expand Up @@ -42,12 +42,15 @@ typedef enum {

/* Attention layer kinds, from layer_types. FULL and SLIDING run the attention block, LINEAR runs the
* GDN recurrence (lin_kd_/conv_state_); OTHER is anything else, which nothing implements - the forward
* refuses a container whose layer_types contains one, by name (unknown_layer_type). */
* refuses a container whose layer_types contains one, by name (unknown_layer_type). CONV is LFM3's
* spelling of LFM2's gated short convolution ("conv"); the forward runs it only where the container holds
* that layer's conv tensors, and refuses a mismatch either way. */
typedef enum {
Q4NX_LAYER_FULL = 0,
Q4NX_LAYER_SLIDING,
Q4NX_LAYER_LINEAR,
Q4NX_LAYER_OTHER
Q4NX_LAYER_OTHER,
Q4NX_LAYER_CONV
} Q4nxLayerType;

typedef struct {
Expand Down Expand Up @@ -153,8 +156,14 @@ typedef struct {
int num_kv_shared_layers;
/* Gemma4 adds pre/post feedforward norms and a post-PLE norm that the plain
* layer does not have (1). Gemma3 has the same sandwich around attention and
* MLP but no post-PLE norm and no layer scalar (2). Source: model_type. */
* MLP but no post-PLE norm and no layer scalar (2). LFM3's use_sandwich_norm is the same pair of norms
* around both blocks (post_operator_norm, post_ffn_norm), so it is 2 as well. Source: model_type, and
* use_sandwich_norm for lfm3 (default true, as configuration_lfm3.py). */
int extra_layer_norms;
/* LFM3: every attention layer carries self_attn.gate_proj [heads, hidden] (bf16), and each head's
* context is multiplied by sigmoid(gate_proj . x) before o_proj, x being the operator-normed input.
* Source: model_type. */
int attn_head_gate;
/* Gemma3 states the sliding layers' RoPE base on its own, as
* `rope_local_base_freq`, and its `rope_scaling` applies to the full layers
* only. 1 when that key was read into rope_theta_sliding. */
Expand Down
58 changes: 58 additions & 0 deletions npu/forward/src/model.c
Original file line number Diff line number Diff line change
Expand Up @@ -850,6 +850,10 @@ ModelWeights* model_load(const char* path, ModelConfig config) {
}

LOG_INFO("Found %d tensors in metadata", num_tensors);
{
const int renamed = model_alias_lfm_names(tensors, num_tensors);
if (renamed > 0) LOG_INFO("Renamed %d LFM tensor names to the canonical ones", renamed);
}

// engine#1185: a container imported from an OGA pack stores its I8 weights as 6,144-byte q4_1f
// chunks (fp32 scale/zero) instead of the 5,120-byte q4_1 the kernels here dequantise in-kernel.
Expand Down Expand Up @@ -1162,6 +1166,60 @@ void model_free(ModelWeights* mw) {
// where every other tensor matches by name. Swapping one prefix is a spelling fallback, not a rename: it happens only
// after an exact match has failed, so a container that carries both is unaffected.

static int name_taken(const TensorDesc* t, int count, const char* name) {
for (int i = 0; i < count; i++)
if (strcmp(t[i].name, name) == 0) return 1;
return 0;
}

int model_alias_lfm_names(TensorDesc* t, int count) {
static const char* const kLayerAlias[][2] = {
{"conv.in_proj.weight", "shortconv.in_proj.weight"},
{"conv.conv.weight", "shortconv.conv.weight"},
{"conv.out_proj.weight", "shortconv.out_proj.weight"},
{"operator_norm.weight", "input_layernorm.weight"},
{"feed_forward.w1.weight", "mlp.gate_proj.weight"},
{"feed_forward.w3.weight", "mlp.up_proj.weight"},
{"feed_forward.w2.weight", "mlp.down_proj.weight"},
{"post_operator_norm.weight", "post_attention_layernorm.weight"},
{"post_ffn_norm.weight", "post_feedforward_layernorm.weight"},
};
if (!t || count <= 0) return 0;
int renamed = 0;
char want[sizeof(t[0].name)];
for (int i = 0; i < count; i++) {
int l = -1, off = 0;
if (sscanf(t[i].name, "model.layers.%d.%n", &l, &off) != 1 || off <= 0) {
if (strcmp(t[i].name, "model.embedding_norm.weight") == 0 &&
!name_taken(t, count, "model.norm.weight")) {
snprintf(t[i].name, sizeof(t[i].name), "%s", "model.norm.weight");
renamed++;
}
continue;
}
const char* suffix = t[i].name + off;
const char* to = NULL;
if (strcmp(suffix, "ffn_norm.weight") == 0) {
// decided from the layer's own tensors: the sandwich form also has post_operator_norm
char probe[sizeof(t[0].name)];
snprintf(probe, sizeof(probe), "model.layers.%d.post_operator_norm.weight", l);
const int sandwich = name_taken(t, count, probe);
snprintf(probe, sizeof(probe), "model.layers.%d.post_attention_layernorm.weight", l);
to = (sandwich || name_taken(t, count, probe)) ? "pre_feedforward_layernorm.weight"
: "post_attention_layernorm.weight";
} else {
for (size_t a = 0; a < sizeof(kLayerAlias) / sizeof(kLayerAlias[0]); a++)
if (strcmp(suffix, kLayerAlias[a][0]) == 0) { to = kLayerAlias[a][1]; break; }
}
if (!to) continue;
snprintf(want, sizeof(want), "model.layers.%d.%s", l, to);
if (name_taken(t, count, want)) continue;
snprintf(t[i].name, sizeof(t[i].name), "%s", want);
renamed++;
}
return renamed;
}

TensorDesc* model_tensor_by_name(ModelWeights* mw, const char* name) {
if (!mw || !mw->all_tensors || !name) return NULL;
for (int i = 0; i < mw->all_count; i++) {
Expand Down
45 changes: 45 additions & 0 deletions npu/forward/src/q4nx_forward.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1112,6 +1112,19 @@ bool Q4nxNpuForward::init_impl(xrt::device* dev, const char* model_path,
"position; file the kind you need instead of converting it";
return false;
}
// LFM3 names its gated short-convolution layers in layer_types ("conv"); LFM2 does not, and the container
// decides there (model_shortconv_layers, above). When both speak they have to agree layer for layer: a
// "conv" entry the container has no conv tensors for would run as attention with no attention weights,
// and a conv layer listed as attention would be given a KV cache and the wrong RoPE.
for (int l = 0; l < sem_.n_layer_types && l < c_.NL; l++) {
const bool said_conv = sem_.layer_types[l] == Q4NX_LAYER_CONV;
if (said_conv == is_conv_layer(l) || (!said_conv && !shortconv_)) continue;
err_ = "config.json layer_types[" + std::to_string(l) + "] is \"" +
(said_conv ? std::string("conv") : std::string("attention")) + "\" but model.q4nx " +
(is_conv_layer(l) ? "holds" : "has no") + " model.layers." + std::to_string(l) +
".shortconv (conv) tensors for that layer";
return false;
}
// Refuse rather than silently apply full-attention math to a model whose
// config declares layer types (or RoPE variants) this forward lacks.
// Refusal is now precise. Only layers whose attention MATH we do not
Expand Down Expand Up @@ -1828,6 +1841,23 @@ bool Q4nxNpuForward::load_norms() {
layer_attn_dims(l, hd_l, nkv_l);
if (!read_bf16(&mw_->layers[l].input_layernorm_weight, in_norm_[l].data(), H)) return false;
if (!read_bf16(&mw_->layers[l].post_attention_layernorm_weight, post_norm_[l].data(), H)) return false;
// LFM3: the per-head sigmoid output gate, self_attn.gate_proj [heads, hidden] in bf16, on every attention
// layer. Missing, it would leave the context ungated, so a layer without one is refused by name.
if (sem_.attn_head_gate) {
if (l == 0) head_gate_.assign((size_t)NL, std::vector<float>());
const TensorDesc* gd = &mw_->layers[l].self_attn_gate_proj_weight;
if (!is_conv_layer(l)) {
if (absent(gd) || strcmp(gd->dtype, "BF16") != 0 || gd->ndim != 2 || gd->shape[0] != c_.NH ||
gd->shape[1] != H) {
err_ = "config.json's model_type gates each attention head (LFM3) and model.layers." +
std::to_string(l) + ".self_attn.gate_proj.weight is not bf16 [num_attention_heads, "
"hidden_size]";
return false;
}
head_gate_[(size_t)l].resize((size_t)c_.NH * (size_t)H);
if (!read_bf16(gd, head_gate_[(size_t)l].data(), c_.NH * H)) return false;
}
}
// engine#1889: the config says q and k are normalised and this layer has no gain to normalise with. read_bf16
// would hand back zeros for the absent tensor and rmsnorm(q, 0) is 0 for every head, so the layer would attend
// uniformly and answer fluent nonsense. A Hunyuan container imported before the importer claimed the gains is
Expand Down Expand Up @@ -1923,6 +1953,18 @@ bool Q4nxNpuForward::load_norms() {
return true;
}

void Q4nxNpuForward::apply_head_gate(int l, const float* xn, float* ctx, int hd) const {
if (l < 0 || (size_t)l >= head_gate_.size() || head_gate_[(size_t)l].empty()) return;
const int H = c_.H;
const float* w = head_gate_[(size_t)l].data();
for (int h = 0; h < c_.NH; h++) {
double z = 0.0;
for (int k = 0; k < H; k++) z += (double)w[(size_t)h * H + k] * (double)xn[k];
const float g = 1.0f / (1.0f + std::exp(-(float)z));
for (int i = 0; i < hd; i++) ctx[(size_t)h * hd + i] *= g;
}
}

void Q4nxNpuForward::layer_attn_dims(int l, int& hd, int& nkv) const {
const int lt = (l < sem_.n_layer_types) ? sem_.layer_types[l] : Q4NX_LAYER_FULL;
if (lt != Q4NX_LAYER_SLIDING && sem_.global_head_dim > 0) {
Expand Down Expand Up @@ -4316,6 +4358,8 @@ bool Q4nxNpuForward::step_batch_impl(const std::vector<int>& tokens, int start_p
for (int m = 0; m < M; m++)
if (!row_attn(m, CTX.data() + (size_t)m * ctx_w, true, 0)) return false;
}
for (int m = 0; m < M; m++)
apply_head_gate(l, XN.data() + (size_t)m * H, CTX.data() + (size_t)m * ctx_w, l_hd);
const Q4nxDesignGeom* go = &designs_[i_o_l];
Q4nxProjections po{};
po.o = wo_.data();
Expand Down Expand Up @@ -5493,6 +5537,7 @@ bool Q4nxNpuForward::step_impl(int token, int pos, std::vector<float>& logits, i
if (gate_ptr)
for (int i = 0; i < nhd; i++)
ctx[i] *= 1.0f / (1.0f + expf(-gate_ptr[i]));
apply_head_gate(l, xn.data(), ctx.data(), l_hd);
}
{
const Q4nxDesignGeom* g = &designs_[i_o_l];
Expand Down
Loading
Loading