From 8577203ac3ecd0767d3fbe69e0fa9d4296427b3f Mon Sep 17 00:00:00 2001 From: t Date: Tue, 10 Nov 2026 02:13:49 -0400 Subject: [PATCH] npu: only take the tile-row path for an embedding table it has been shown to decode (engine#1807 review) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The review found the real hole: init took the cheap path on arithmetic alone -- `esz % ((NV/32)*(kpad_/256)) == 0` -- and set the chunk size without checking that it is one of the three decodable chunk kinds, or that `NV` is a multiple of the 32-row tile. In either case `q4nx_dequant_tile_row()` returns -1 for every token, `embed_at()` answers 0.0f, and the model runs on an all-zero embedding **with no message** -- where the whole-table path this replaces refused at load with "embedding dtype … but dequant failed". So the optimisation could turn a loud refusal into silent wrong output, which is the worst trade in this file. Fix, as the review asked: at init, gather row 0 and row `NV-1` through `q4nx_dequant_tile_row()` before committing to the path. A failure leaves `embed_tile_` null and the existing whole-table buffer runs and reports its own failure. The cheap path therefore only ever runs where it has been shown to decode on this container. Re-measured at this head, with the knob pinned (`NPU_FFN_CACHE=0`: its default flips with free RAM, which changed the warm-up between arms and was the reason a first attempt never became ready inside a 9-minute ticket): before (materialised) RssAnon after load = 9932 MiB after (probe + row) RssAnon after load = 2619 MiB -> 7.14 GiB returned Unchanged from the pre-probe figure (9773 -> 2658 MiB, 7.15 GiB): the probe allocates one `kpad_`-sized temporary and reuses the row buffer, so it costs the win nothing. Suite 168/168. (cherry picked from commit 0c6e163e9d780562a9e5e6fce2cfa7c5cc80f65e) --- npu/forward/src/q4nx_forward.cpp | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/npu/forward/src/q4nx_forward.cpp b/npu/forward/src/q4nx_forward.cpp index 95ec35e94..25310d23b 100644 --- a/npu/forward/src/q4nx_forward.cpp +++ b/npu/forward/src/q4nx_forward.cpp @@ -744,14 +744,25 @@ bool Q4nxNpuForward::init_impl(xrt::device* dev, const char* model_path, * Same tile grid and ordering as q4nx_dequant_tensor_chunked(), pinned by the row test. */ const int ntr = NV / 32, ntc = kpad_ / 256; const size_t tiles = (size_t)ntr * (size_t)ntc; + /* Take the cheap path only where it has been SHOWN to decode: read the first and the last row through it + * now (engine#1807 review). The arithmetic above can be satisfied by a container this cannot read -- + * a chunk size that is not one of the three decodable kinds, or an NV that is not a multiple of the 32-row + * tile -- and embed_at() would then answer 0.0f for every token, i.e. the model would run on an all-zero + * embedding with no message, where the whole-table path below refuses at load with the reason. A failed + * probe leaves embed_tile_ null and the buffer path (unchanged) reports it. */ if (tiles > 0 && esz % tiles == 0) { - embed_tile_ = (const uint8_t*)e; - embed_tile_bytes_ = esz; - embed_tile_rows_ = NV; - embed_tile_cols_ = kpad_; - embed_tile_cb_ = int(esz / tiles); - embed_row_buf_.assign((size_t)kpad_, 0.0f); - embed_row_tok_ = -1; + const int cb = int(esz / tiles); + std::vector probe((size_t)kpad_, 0.0f); + if (q4nx_dequant_tile_row((const uint8_t*)e, esz, NV, kpad_, cb, 0, probe.data()) == 0 && + q4nx_dequant_tile_row((const uint8_t*)e, esz, NV, kpad_, cb, NV - 1, probe.data()) == 0) { + embed_tile_ = (const uint8_t*)e; + embed_tile_bytes_ = esz; + embed_tile_rows_ = NV; + embed_tile_cols_ = kpad_; + embed_tile_cb_ = cb; + embed_row_buf_.assign((size_t)kpad_, 0.0f); + embed_row_tok_ = -1; + } } rc = 0; // the buffer below is the fallback, and it reports its own failure }