Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions docs/npu.md
Original file line number Diff line number Diff line change
Expand Up @@ -909,6 +909,25 @@ What converts, and how faithfully:
| `model.norm` (final) | `.fconst`, the tail envelope at `layers x stride` | bf16 | byte-exact |
| `sin_cache_cos_cache` | `model.onnx.data` | not imported | the lane computes RoPE from `rope_theta` (`fill_rope`) |

**A multi-component pack's per-layer embedding** (engine#1725). gemma-4's embedding component
carries, beside the token table, `embed_tokens_per_layer.weight` (`vocab x layers*P` f32; 262,144 x
8,960 on e2b, 9.4 GiB), a `P`-wide `per_layer_projection_norm.weight` shared by every layer, and
the context projection `per_layer_model_projection.weight_t_Q4` quantised at block 32. The import
writes them under the names the forward loads:

| container tensor | from | form |
|---|---|---|
| `model.per_layer_token_embd.weight` `[vocab, layers*P]` | the f32 table | BF16, round to nearest even |
| `model.per_layer_proj_norm.weight` `[P]` | the norm | BF16 |
| `model.per_layer_model_proj.weight_layer{l}` `[P, hidden]` | the projection, dequantised, one per layer | BF16 |

The table is never held in memory, in either form: the reader points into the component's mapped
external data (its extent is checked once, when it is located), and the container writer converts
it 32 rows at a time while it writes the file. A projection with no zero-point tensor takes the
implicit zero point 8, the same rule as every other MatMulNBits here. `config.json` gets
`hidden_size_per_layer_input` and `vocab_size_per_layer_input` only when all three parts went into
the container, because those two keys are what switch the block on in the forward.

An OGA pack keeps its tensors inside the `If` branches: the top-level graph has **no
initializers** and four nodes (`Shape`/`Constant`/`Equal`/`If`). The importer walks the
subgraphs. The 4-bit decode is `w = (code - zp) * scale` with two codes per byte (low nibble
Expand Down
33 changes: 32 additions & 1 deletion npu/forward/src/q4nx_forward.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -2770,7 +2770,38 @@ bool Q4nxNpuForward::load_ple_block(int row) {
ple_block_.assign((size_t)Q4NX_TILE_ROWS * (size_t)cols, 0.0f);
const size_t plain_bytes =
(size_t)sem_.vocab_size_per_layer_input * (size_t)cols;
if ((size_t)td->data_size == plain_bytes) {
// engine#1725: a RAW BF16 band. The multi-component packs state this table as f32 and the importer converts it to BF16
// when it writes the container (a converting copy of the whole V x NL*P table would be 9.4 GiB), so the band here is
// 32 rows of BF16 -- the shape the plain-int8 branch handles, with a 2-byte element instead of a code.
//
// The branch is chosen by the container's DECLARED dtype, not by arithmetic on data_size (review of #1825): a size test
// needs `% bf16_bytes == 0 && != plain_bytes` to separate two layouts, and every multiple of the band satisfies it, so a
// plain table whose byte count happens to divide would be read as BF16 bands -- plausible and wrong. This tree already
// decides exactly this way elsewhere (`model.c`: `strcmp(dtype,"I8")`), and `q4nx_dequant.h`'s rule is that an
// ambiguous layout is refused rather than resolved by which test happens to pass.
const size_t bf16_bytes = (size_t)Q4NX_TILE_ROWS * (size_t)cols * 2;
const char* pl_dtype = td->dtype;
const bool declared_bf16 = pl_dtype[0] != '\0' && std::strcmp(pl_dtype, "BF16") == 0;
const bool declared_i8 = pl_dtype[0] != '\0' && (std::strcmp(pl_dtype, "I8") == 0 || std::strcmp(pl_dtype, "Q4") == 0);
if (!declared_bf16 && !declared_i8) {
// Some other declared format, or none at all: refuse rather than fall through to a layout guess. A container
// written by this importer always states one (`w.add()` writes dtype for every tensor), so an empty dtype means a
// foreign or truncated container, not a layout this loader can infer.
err_ = pl_dtype[0] != '\0'
? std::string("per_layer_token_embd has dtype ") + pl_dtype +
", which this loader does not read (BF16 or plain I8 only)"
: std::string("per_layer_token_embd declares no dtype, so its layout cannot be decided");
return false;
}
if (declared_bf16) {
if ((size_t)td->data_size < ((size_t)block + 1) * bf16_bytes ||
(size_t)td->data_size % bf16_bytes != 0) {
err_ = "per_layer_token_embd is declared BF16 but its size is not whole 32-row bands";
return false;
}
const uint16_t* band = reinterpret_cast<const uint16_t*>(raw + (size_t)block * bf16_bytes);
for (size_t i = 0; i < (size_t)Q4NX_TILE_ROWS * (size_t)cols; ++i) ple_block_[i] = bf16_to_f32(band[i]);
} else if ((size_t)td->data_size == plain_bytes) {
// PLAIN int8 + per-32-column scale (Gemma4): only the requested band of
// 32 rows is needed, so it is dequantized in place rather than the whole
// [vocab, NL*ple_dim] table.
Expand Down
Loading
Loading