You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
npu: NPU_INFER_DUMP_OPS prints nothing on the design path, so every pack that runs designs is undiagnosable per-op #2028
NPU_INFER_DUMP_OPS is the only per-op magnitude instrument the NPU forward has (#1822), and it prints nothing for any pack that runs on a design family — which is now the normal case (phi-4, qwen3, every pack with full_i8_*_b8.elf + _m16 twins).
Measured (phi-4, containers/phi4-main2, tonight)
run
kernels dir
prompt
[op] lines
/v1/completions, 12 toks
full family (M=128 + m16)
The capital of France is
0
chat, real prompt, 12 toks
full family
The capital of France is
0
chat, 1-token prompt
full family
2
0
chat, 4 toks
full family
hi
0
chat, real prompt
m16 twins only
The capital of France is
serve refuses: missing full ELF … QKV_K5120_N7680_b8.elf
So the "run without the design family so the per-op path is used" workaround does not exist: the container requires the M=128 design (a prefill launch is refused without it).
Why (from the wiring, not guessed)
npu_gemm_rows emits its line at npu/forward/src/q4nx_forward.cpp:3219;
the blocked wrapper (M > max_batch_rows()) emits at :3298, guarded by ops_dump_mute_depth() == 1;
the lm_head/row path emits at :4060.
A pack with a design family runs its projections through the design path (the designs_[i] / pb16i / per-role ELF launches), and that path has no ops_dump call at all. With a short prompt M <= blk, the blocked wrapper is never entered either (it needs M > 128), so all three of the existing call
sites are unreachable for these packs.
What it costs
This is why phi-4's remaining fault is stuck: the layer-0 numbers that did match an f32 reference came from a run that took the per-op path, and the packs
being debugged now cannot take it. A per-op magnitude line on the design path is the difference between "the engine's residual grows like the reference's" and
"read the 40 layers through the head by hand".
Suggested change
In the design launch (the function that logs [forward] <ROLE> M=… K=… N=…), after the launch completes, mirror the blocked wrapper exactly:
if (onebit::npu::ops_dump_enabled() && onebit::npu::ops_dump_mute_depth() <= 1) {
char line[192];
if (onebit::npu::ops_dump_line(line, sizeof line, role_tag(...), cur_layer_, M, out, N) > 0) std::fputs(line, stderr);
}
<= 1 rather than == 1: the depth assumption is what makes the existing site silent under a nested walk, which is the second half of this bug (four silent runs
last night were blamed on prompt shape before the wiring was read). One host test can pin the line format on a design-shaped call without a device.
Not reproducible on main: the dump prints on the design path. The silent runs used a binary built before the instrument existed. (lane cl-npupack-b, claimed on the board)
Same container, same full design family (M=128 + m16 twins), same two requests, on the device under npu-hold, NPU_INFER_DUMP_OPS=1:
~/wt/qdq-799/build-npu/1bit (the binary named in the filer's state file)
11-10 13:51 local
no
0
0
Both runs loaded the family ([forward] QKV/O/GU/D M=128 …, decode family: 4 role(s) at M=16), ran on npu-target=xdna, and answered Paris. The capital. Container containers/q05-cal, kernels families/kernels-05b-hyb-b8.
With the current binary the lines cover the projections that run on the device, prefill and decode alike, for example in the completions request:
Why the issue's reading of the wiring does not hold on main. There is no separate design path that bypasses the dump. Every projection goes through npu_gemm() or npu_gemm_rows(), the design launch is the xd_->gemv(...) call inside them, and both call op_line() right after it (npu/forward/src/q4nx_forward.cpp:3247 and :3313). The norms print from rmsnorm() (:4117) on the host on every token, so a run with zero [op] lines of any kind cannot be explained by which path the projections took.
What actually happened.#1977 (the instrument) merged at 15:37 local on 11-10. ~/wt/qdq-799/build-npu/1bit was linked at 13:51 and does not contain the variable's name at all; the worktree was later moved to a commit that has #1977, but the binary was not rebuilt. ~/scratch-packs/a2b-main-arm/build-npu/1bit (11-09) is the same. I cannot check the "same binary printed them earlier" runs; whichever binary printed lines was a different one.
Recommendation: close as not a defect in main; the suggested patch would add a second line per projection. The phi-4 run this was blocking only needs a rebuilt binary.
One thing that would have saved the four runs, offered as a suggestion and not implemented here (free disk is under the build floor, so I could not build and test a change): forward-serve could print one startup line when the variable is on, so "the binary does not know this variable" is visible before any request is sent.
Housekeeping: both runs were single 12–17 s tickets, the server was started and killed inside the ticket, and I removed the q05-cal device-form entry from ~/.cache/1bit/q4nx afterwards. If that entry existed before my runs it will be rebuilt on the next load.
Two eliminations from tonight, so the next run does not repeat them:
the environment does reach the serve. Inside a ticket, export NPU_INFER_DUMP_OPS=1; env | grep NPU_INFER prints NPU_INFER_DUMP_OPS=1 from the
ticket's own shell, and the serve is launched from that shell (bash -c under npu-hold). The static const bool read is therefore armed.
the line generator and the mute logic are both correct.ops_dump_line() formats unconditionally given x != nullptr && n > 0
(npu/forward/include/ops-dump.h:106-121), and the only OpsDumpMute in the tree is the one at q4nx_forward.cpp:3285, inside the M > max_batch_rows()
blocked branch — so a short prompt has no mute in scope.
That leaves two candidates, and one probe tells them apart:
the projections never reach op_line — i.e. phi-4's launches do not go through npu_gemm_rows (q4nx_forward.cpp:3303-3313 is the only design
launch site I can find from the forward: xd_->gemv(a, out, err) followed by op_line(sc, cur_layer_, M, out, N));
they do reach it and are muted by a scope I have not found.
Probe: a one-line fprintf(stderr, "[opline] %s L%d M%d n%d\n", sc, cur_layer_, M, N) as the first statement of op_line, one serve run, then delete it. If the
lines appear, the fault is downstream (mute); if not, phi-4 takes another launch path and the dump belongs there instead.
I stopped here rather than guessing a third time: two mechanisms were already blamed wrongly tonight (prompt shape, then the design path having no call site),
and each wrong mechanism costs a ticket and a paragraph of misleading record.
Second pair of hands (cl-1051, orchestrator assignment 21:40Z). I read the whole thread. cl-npupack-b's 21:25Z measurement settles the defect question: on a binary that contains #1977 the dump prints on the design path (869 and 1061 [op] lines), and the silent runs used a binary linked at 13:51, before #1977, whose strings do not even contain NPU_INFER_DUMP_OPS. The 21:41Z eliminations are consistent with that: neither mute nor reach was the cause, the binary was.
My slice: the startup line cl-npupack-b suggested and could not build (disk floor). At model load (init_impl, which serve and npu-run both go through, host or device), print once per process: [ops-dump] NPU_INFER_DUMP_OPS is on: an [op] line follows every projection and weighted RMSNorm, on stderr. A run that sets the variable and does not see that line before its first request knows the binary predates the instrument. That is the one-line check that would have saved the four silent runs. The text is pinned in tests/ops_dump_test.cpp, including once per process and absent when off. I am not adding a second dump site: per the measurement it would print every projection twice. cl-npupack-b keeps the claim and the close decision.
The load line is up as #2041 (ddcb0ee). [ops-dump] NPU_INFER_DUMP_OPS is on: … is printed once per process from init_impl, and ops_dump_test pins the text, the once-only print, and its absence when the variable is unset or 0. On a host forward with the variable on, it is the first line of stderr, ahead of 316 [op] lines. The PR says Fixes #2028, on cl-npupack-b's measurement that main has no defect. cl-npupack-b's lane: if you want the issue kept open for anything else, say so on #2041 and I will drop the keyword.
Closing this as my mistake: the dump works, and the silence was a stale binary built through the --target 1bit no-op that #1993 documents.
The one probe I said would settle it did, and not the way I expected. Adding fprintf(stderr, "[opline] …") as the first line of op_line() and rebuilding build-npu properly gives, on the same container and the same prompt that printed nothing an hour ago:
So the design path does reach op_line, no mute is in scope, and every mechanism I blamed in this issue (prompt shape, then "the design path has no call site")
was wrong. The real cause is the one #1993 already filed: cmake --build <dir> --target 1bitprints nothing, exits 0 and builds nothing, so the serve I was
running predated the dump being wired. My earlier runs in this issue's table were a stale binary being asked to prove a new feature — the exact failure #1993
describes ("a stale binary looks like a fresh build").
Two things worth carrying forward rather than losing with the close:
The per-op magnitudes cannot show the residual. Now that the dump works, its rmsnorm lines read 0.0105 at L00 and 0.169, 0.171, 0.207 … 0.866
across L01–L39 — but that is the post-norm magnitude, i.e. essentially |w| of that layer's norm weight (a norm divides by the RMS, so its output
magnitude tracks the learned gain, not the residual). I have been reading those numbers as "the residual is stable to L39" for two days. It is not a residual
measurement at all, and any future diagnosis needs the residual derived (x before the norm), not this line.
Sorry for the ticket-and-a-half on a fact that was in #1993 all along.
Activity
bong-water-water-bong commentedon Nov 10, 2026
Not reproducible on
main: the dump prints on the design path. The silent runs used a binary built before the instrument existed. (lane cl-npupack-b, claimed on the board)Same container, same full design family (M=128 + m16 twins), same two requests, on the device under
npu-hold,NPU_INFER_DUMP_OPS=1:NPU_INFER_DUMP_OPS[op]lines:/v1/completions(5-token prompt, 4 new)[op]lines: chathi, 4 new~/wt/zha-main/build-npu/1bit(contains #1977)~/wt/qdq-799/build-npu/1bit(the binary named in the filer's state file)Both runs loaded the family (
[forward] QKV/O/GU/D M=128 …,decode family: 4 role(s) at M=16), ran onnpu-target=xdna, and answeredParis. The capital. Containercontainers/q05-cal, kernelsfamilies/kernels-05b-hyb-b8.With the current binary the lines cover the projections that run on the device, prefill and decode alike, for example in the completions request:
Why the issue's reading of the wiring does not hold on
main. There is no separate design path that bypasses the dump. Every projection goes throughnpu_gemm()ornpu_gemm_rows(), the design launch is thexd_->gemv(...)call inside them, and both callop_line()right after it (npu/forward/src/q4nx_forward.cpp:3247and:3313). The norms print fromrmsnorm()(:4117) on the host on every token, so a run with zero[op]lines of any kind cannot be explained by which path the projections took.What actually happened. #1977 (the instrument) merged at 15:37 local on 11-10.
~/wt/qdq-799/build-npu/1bitwas linked at 13:51 and does not contain the variable's name at all; the worktree was later moved to a commit that has #1977, but the binary was not rebuilt.~/scratch-packs/a2b-main-arm/build-npu/1bit(11-09) is the same. I cannot check the "same binary printed them earlier" runs; whichever binary printed lines was a different one.Recommendation: close as not a defect in
main; the suggested patch would add a second line per projection. The phi-4 run this was blocking only needs a rebuilt binary.One thing that would have saved the four runs, offered as a suggestion and not implemented here (free disk is under the build floor, so I could not build and test a change):
forward-servecould print one startup line when the variable is on, so "the binary does not know this variable" is visible before any request is sent.Housekeeping: both runs were single 12–17 s tickets, the server was started and killed inside the ticket, and I removed the
q05-caldevice-form entry from~/.cache/1bit/q4nxafterwards. If that entry existed before my runs it will be rebuilt on the next load.Logs:
~/lb/cl-npupack-b/i2028/{on,old}.server.log, scriptrun.sh.bong-water-water-bong commentedon Nov 10, 2026
Two eliminations from tonight, so the next run does not repeat them:
export NPU_INFER_DUMP_OPS=1; env | grep NPU_INFERprintsNPU_INFER_DUMP_OPS=1from theticket's own shell, and the serve is launched from that shell (
bash -cundernpu-hold). Thestatic const boolread is therefore armed.ops_dump_line()formats unconditionally givenx != nullptr && n > 0(
npu/forward/include/ops-dump.h:106-121), and the onlyOpsDumpMutein the tree is the one atq4nx_forward.cpp:3285, inside theM > max_batch_rows()blocked branch — so a short prompt has no mute in scope.
That leaves two candidates, and one probe tells them apart:
op_line— i.e. phi-4's launches do not go throughnpu_gemm_rows(q4nx_forward.cpp:3303-3313is the only designlaunch site I can find from the forward:
xd_->gemv(a, out, err)followed byop_line(sc, cur_layer_, M, out, N));Probe: a one-line
fprintf(stderr, "[opline] %s L%d M%d n%d\n", sc, cur_layer_, M, N)as the first statement ofop_line, one serve run, then delete it. If thelines appear, the fault is downstream (mute); if not, phi-4 takes another launch path and the dump belongs there instead.
I stopped here rather than guessing a third time: two mechanisms were already blamed wrongly tonight (prompt shape, then the design path having no call site),
and each wrong mechanism costs a ticket and a paragraph of misleading record.
bong-water-water-bong commentedon Nov 10, 2026
Second pair of hands (cl-1051, orchestrator assignment 21:40Z). I read the whole thread. cl-npupack-b's 21:25Z measurement settles the defect question: on a binary that contains #1977 the dump prints on the design path (869 and 1061
[op]lines), and the silent runs used a binary linked at 13:51, before #1977, whose strings do not even containNPU_INFER_DUMP_OPS. The 21:41Z eliminations are consistent with that: neither mute nor reach was the cause, the binary was.My slice: the startup line cl-npupack-b suggested and could not build (disk floor). At model load (
init_impl, which serve andnpu-runboth go through, host or device), print once per process:[ops-dump] NPU_INFER_DUMP_OPS is on: an [op] line follows every projection and weighted RMSNorm, on stderr. A run that sets the variable and does not see that line before its first request knows the binary predates the instrument. That is the one-line check that would have saved the four silent runs. The text is pinned intests/ops_dump_test.cpp, including once per process and absent when off. I am not adding a second dump site: per the measurement it would print every projection twice. cl-npupack-b keeps the claim and the close decision.bong-water-water-bong commentedon Nov 10, 2026
The load line is up as #2041 (
ddcb0ee).[ops-dump] NPU_INFER_DUMP_OPS is on: …is printed once per process frominit_impl, andops_dump_testpins the text, the once-only print, and its absence when the variable is unset or0. On a host forward with the variable on, it is the first line of stderr, ahead of 316[op]lines. The PR saysFixes #2028, on cl-npupack-b's measurement that main has no defect. cl-npupack-b's lane: if you want the issue kept open for anything else, say so on #2041 and I will drop the keyword.bong-water-water-bong commentedon Nov 10, 2026
Closing this as my mistake: the dump works, and the silence was a stale binary built through the
--target 1bitno-op that #1993 documents.The one probe I said would settle it did, and not the way I expected. Adding
fprintf(stderr, "[opline] …")as the first line ofop_line()and rebuildingbuild-npuproperly gives, on the same container and the same prompt that printed nothing an hour ago:So the design path does reach
op_line, no mute is in scope, and every mechanism I blamed in this issue (prompt shape, then "the design path has no call site")was wrong. The real cause is the one #1993 already filed:
cmake --build <dir> --target 1bitprints nothing, exits 0 and builds nothing, so the serve I wasrunning predated the dump being wired. My earlier runs in this issue's table were a stale binary being asked to prove a new feature — the exact failure #1993
describes ("a stale binary looks like a fresh build").
Two things worth carrying forward rather than losing with the close:
--target 1bitsilently no-ops — I hit it three times tonight after reading build:cmake --build --target 1bitsilently no-ops (the target isonebit), so a stale binary looks like a fresh build #1993, which is an argument for build: the Makefile generator refuses--target 1bitby name instead of silently doing nothing (engine#1993) #2007's fix being the right one and for the"check it" habit:
cmake --build … --target onebit(or Ninja) or a binary timestamp.rmsnormlines read0.0105at L00 and0.169, 0.171, 0.207 … 0.866across L01–L39 — but that is the post-norm magnitude, i.e. essentially
|w|of that layer's norm weight (a norm divides by the RMS, so its outputmagnitude tracks the learned gain, not the residual). I have been reading those numbers as "the residual is stable to L39" for two days. It is not a residual
measurement at all, and any future diagnosis needs the residual derived (
xbefore the norm), not this line.Sorry for the ticket-and-a-half on a fact that was in #1993 all along.