Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 33 additions & 2 deletions docs/npu.md
Original file line number Diff line number Diff line change
Expand Up @@ -1993,8 +1993,39 @@ again, the known-answer prompt answered `*,,,,,,,`; with the plain form it answe
form, and gemma2 keeps `(1 + w)` (not measured here). `tests/q4nx_gemma3_semantics_test.c` pins each rule with
Medgemma's own keys.

A prefill runs batched (`step_batch`) with the same three extra norms; only Gemma4 still falls back to one
`step()` per prompt token. `batch_check_state_test` on the Medgemma pack: batched equals sequential, rel-L1 0.
A prefill runs batched (`step_batch`) with the same three extra norms. `batch_check_state_test` on the Medgemma
pack: batched equals sequential, rel-L1 0.

**Gemma4 runs batched too (engine#1780).** It used to fall back to one `step()` per row, so verifying k drafted
tokens cost k weight streams, and a DFlash pair on a Gemma4 target could not beat plain decode. The batched body
now does what `step()` does for each row:
- it builds the row's per-layer inputs;
- it runs the PLE gate and projection as M-row GEMMs, then the layer scalar;
- on a KV-shared layer it reads the owner layer's cache and appends nothing (the owner already appended every row
of the batch earlier in the same pass);
- a shared layer's double-wide MLP uses `GU2`/`D2`.

A batch with injected embeddings is still refused on a PLE model, because the per-layer inputs need token ids.

The KV cache's row width is now per layer, from `kv_widths()`. A model with two attention geometries holds more
floats per position on its full layers: on Gemma4-E4B, 2 × 512 against 2 × 256 on the sliding layers. Snapshot,
restore, a verify's rollback and `extend_context` used to count every layer at `NKV * HD`. That reported twice the
positions and cut the full layers at half on a snapshot or a partial accept, which corrupts the prefix cache and
any verify on such a model.

`batch_check_state_test` on `Nemotron-3.5-Bolt-E4B` (host GEMM) checks all of this:
- chunks [0,3), [3,5), [5,6): every chunk's argmax matches; the last chunk's logits equal the sequential ones
(rel-L1 0); the positions held are exact after each chunk;
- a verify arm: a 4-row `step_batch_all` is bit-identical to 4 sequential steps (the oracle reports
`identical=4`); after `commit_verify_rows(2)` exactly 5 positions are held, and the next step equals a forward
that saw only those rows.

Before the width fix, the first arm failed 4 of 4. With the old single width for the rollback, the verify arm fails.
`q4nx_state_planes_test` holds the per-layer cut to Gemma4-E4B's 42-layer geometry.

The verify arm also runs on the CI's fabricated containers. There it found that `step_batch_all()` on a model that
still falls back to one `step()` per row (gpt-oss, a MoE with a leading dense layer) returned no per-row logits: the
fallback filled only the last row and left `logits_all` as it found it. The fallback now appends every step's row.

The serve also needs the BOS. Medgemma's `config.json` states neither `bos_token_id` nor `add_bos_token`, and its
`tokenizer_config.json` states both (`<bos>`, true), as HF reads them. forward-serve now takes what `config.json`
Expand Down
6 changes: 6 additions & 0 deletions npu/forward/include/q4nx_forward.h
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,9 @@ class Q4nxNpuForward {
// Copy the state of the first `tokens` positions out. False when the forward is not ready or
// `tokens` exceeds what it holds.
bool snapshot_state(int tokens, StateSnapshot& out) const;
// Floats one position adds to each layer's K (and V) cache: NKV * HD, or the full layers' own width on a model
// with two attention geometries (engine#1780).
std::vector<size_t> kv_widths() const;
// Replace the state with the first `tokens` positions of `in`: the caller then continues at
// position `tokens`. Restoring fewer positions than the snapshot holds is the normal case (a
// request resumes to just short of its own prompt and re-runs the last token), and restoring
Expand Down Expand Up @@ -323,6 +326,9 @@ class Q4nxNpuForward {
// The same elementwise gate*up on caller memory, so a batched pass can read its ACT row and write its HH
// row where they already are instead of staging both through a vector (a copy on each side).
bool mlp(const float* gu, float* h, int im);
// engine#1780: the end of a Gemma-family layer for M rows in the batched body, as step() runs it for one: the
// per-layer embedding block (when the model has one; `ple_rows` is [M][NL * ple_dim]) and then layer_scalar.
bool gemma4_tail_rows(int l, int M, const std::vector<float>& ple_rows, std::vector<float>& X);
// `tag` is the NPU_INFER_DUMP_OPS label (ops-dump.h); the per-head q/k norms pass their own.
float rmsnorm(const float* x, const float* w, int n, float* out, const char* tag = "rmsnorm") const;
float rmsnorm_noweight(const float* x, int n, float* out) const;
Expand Down
51 changes: 51 additions & 0 deletions npu/forward/include/q4nx_state_planes.h
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,57 @@ inline bool give_whole(const std::vector<std::vector<float>>& src, std::vector<s
return true;
}

// engine#1780: the same per-position cut where the layers' KV rows differ in width. A model with two attention
// geometries (Gemma4: sliding layers 2 x 256 = 512 floats per position, full layers 2 x 512 = 1024) holds a different
// number of floats per position in each plane, so one `want` cut the wide planes at half their positions -- the
// snapshot kept the prefix's first half on those layers, and a restore then resumed from a cache that was short.
// `per[l]` is plane l's floats per position; a plane past the end of `per` is cut at per.back().
inline size_t per_of(const std::vector<size_t>& per, size_t l) {
return per.empty() ? 0 : (l < per.size() ? per[l] : per.back());
}

// The positions the planes hold: the longest plane, counted in its own row width (a shared-KV plane holds none).
inline size_t held_positions(const std::vector<std::vector<float>>& src, const std::vector<size_t>& per) {
size_t held = 0;
for (size_t l = 0; l < src.size(); l++) {
const size_t w = per_of(per, l);
if (w) held = std::max(held, src[l].size() / w);
}
return held;
}

inline void take_per_pos(const std::vector<std::vector<float>>& src, std::vector<std::vector<float>>& dst,
const std::vector<size_t>& per, size_t tokens) {
dst.resize(src.size());
for (size_t l = 0; l < src.size(); l++) {
const size_t n = std::min(tokens * per_of(per, l), src[l].size());
dst[l].assign(src[l].begin(), src[l].begin() + long(n));
}
}

inline bool give_per_pos(const std::vector<std::vector<float>>& src, std::vector<std::vector<float>>& dst,
const std::vector<size_t>& per, size_t tokens) {
if (dst.size() < src.size()) return false;
for (size_t l = 0; l < src.size(); l++) {
if (src[l].empty()) {
dst[l].clear();
continue;
}
const size_t want = tokens * per_of(per, l);
if (src[l].size() < want) return false;
dst[l].assign(src[l].begin(), src[l].begin() + long(want));
}
return true;
}

// Keep the first `tokens` positions of every non-empty plane (a verify's rollback): planes are cut, never grown.
inline void keep_positions(std::vector<std::vector<float>>& planes, const std::vector<size_t>& per, size_t tokens) {
for (size_t l = 0; l < planes.size(); l++) {
const size_t keep = tokens * per_of(per, l);
if (!planes[l].empty() && planes[l].size() > keep) planes[l].resize(keep);
}
}

} // namespace q4nx_state

#endif // Q4NX_STATE_PLANES_H
Loading
Loading