From 4e5e99d457d76fe4c83088f4149a0fb0bee8996b Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 8 Nov 2026 19:23:54 -0400 Subject: [PATCH] npu: the batch phase line was fed by the decode accumulator only, so a device prefill read as host engine#1582 reports a 219-row prefill printing n=0 launches and 93 s of host time, while the very next decode step on the same target reports npu_gemm 656 ms. It is not a device fallback and not a lost launch: it is this condition. Every launch site accounts M == 1 to prof_dec_ and M > 1 to prof_pre_ (xdna_target.cpp:900-906, 969-974), but prof_add only fed the step's phase snapshot for prof_dec_: if (q4nx_lh::phprof_on() && acc == prof_dec_ && !q4nx_lh::g_in_head) { g_ph... ; launches++ } so every batched launch was invisible to g_ph. And the [bs] line derives host as "layer time minus the device slices" (q4nx_forward.cpp:3606, g_ph.layer - gsum), which means a prefill that launched perfectly on the device reported exactly what a host fallback would: n=0, host = the whole layer time. Measurement only: g_ph is the per-step snapshot nothing reads back, so this changes what the line says, not what the engine does. A step is either batched or per-token, so counting both accumulators cannot double-count a launch; the g_in_head exclusion is unchanged. The 93 s wall itself is untouched by this - if the prefill is genuinely slow, the fixed line will now show where it went (ql/wait/rb instead of all-host), which is the number gate 3 (#1498) needs. Compile-checked at this head with the CI compiler and flags: g++-16 -std=c++26 -fsyntax-only -DONEBIT_NPU_RUNTIME_OWN plus the npu include paths, rc 0. Runtime confirmation needs hybrid_prompt_probe under npu-hold and is stated as pending on the issue, not claimed here. Fixes #1582 --- npu/target/xdna/xdna_target.cpp | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/npu/target/xdna/xdna_target.cpp b/npu/target/xdna/xdna_target.cpp index def6a0534..4e4172068 100644 --- a/npu/target/xdna/xdna_target.cpp +++ b/npu/target/xdna/xdna_target.cpp @@ -447,7 +447,14 @@ int64_t XdnaTarget::m16_check_gemm_rows(I8Ctx& ref, I8Ctx& fast, const float* A, void XdnaTarget::prof_add(ProfAcc* acc, int design, double pack, double quant, double wait, double rb) { - if (q4nx_lh::phprof_on() && acc == prof_dec_ && !q4nx_lh::g_in_head) { + // Both accumulators feed the step's phase line. `acc == prof_dec_` here was wrong for the batch half: + // every launch with M > 1 is accounted to prof_pre_ (see the M == 1 branch at each launch site), so a + // batched pass reported n=0 launches and, because the [bs] line derives host as "layer time minus the + // device slices", 100 % of a device prefill as HOST work (engine#1582: 219 rows, n=0, host=93113 of + // wall=97588, while the very next decode step reported npu_gemm 656 ms on the same target). Measurement + // only - g_ph is a per-step snapshot nothing reads back - so this changes what the line *says*, not what + // the engine does. A step is either batched or per-token, so counting both cannot double-count one launch. + if (q4nx_lh::phprof_on() && !q4nx_lh::g_in_head) { q4nx_lh::g_ph.g_pack += pack; q4nx_lh::g_ph.g_ql += quant; q4nx_lh::g_ph.g_wait += wait; q4nx_lh::g_ph.g_rb += rb; q4nx_lh::g_ph.launches++;