You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
hrx: GGML_HRX_LOG_DISPATCH is not behaviour-neutral — the 18.8k request fails without it and succeeds with it #2143
GGML_HRX_LOG_DISPATCH=1changes behaviour, not just logging. Same binary, same model, same request, mask kernels on:
arm
NaN logits
the 18,782-token /completion
LOG_DISPATCHunset
2
fails — "Remote end closed connection without response"
LOG_DISPATCH=1
0
succeeds, prompt_n = 18782
A logging environment variable must not decide whether a request completes. Two consequences:
Every A/B and every rate taken with LOG_DISPATCH=1 is suspect. I have one of my own: a plain 18.6k prefill measured 271.75 t/s with the flag unset and 118.86 t/s with it set — I reported that as a 2.3× logging cost an hour ago and I am now withdrawing the interpretation, because the flag demonstrably changes what runs. Any lane that A/B'd dispatch selection with this flag has the same problem.
The logging path calls tensor_is_readback_request(value->tensor) (see dispatch-registry.cpp, first_covered_requested_output / dispatch_log_requested_output). If marking a value as a readback changes how it is bound or kept resident, then logging is not observational — and that is the first thing to check.
Suggested first step
GGML_HRX_LOG_DISPATCH=1 with the request from #1536/#1925 (18,782 tokens, Flash-Next, mask kernels on) and a diff of the accepted dispatch set against the same run without the flag. If the sets differ, the flag is selection-affecting by definition; if they are identical, the difference is downstream (binding/readback) and the check moves there.
Kernel internals stay private; this issue needs no kernel source.
Read at the engine pin (fork 7a1f2b81ce17, read only, no device run): in the fork, the flag gates only two stderr prints. The flag does not decide which registration is selected, how values are bound, or what is read back.
Every reader of GGML_HRX_LOG_DISPATCH in the fork outside *.md, at the pin and also at c16424f5620d (the off-pin checkout #2141 found on this box):
dispatch-registry.cpp:138 dispatch_log_requested_output() static const bool enabled = getenv(...) != nullptr; if (!enabled) return; -> fprintf only, capped at 8 lines
dispatch-registry.cpp:161 dispatch_log_match() same gate -> fprintf once per registration name (a std::set under a mutex)
The HRX runtime (hrx-system at 82804197bc) does not read the flag. On engine main it appears only in docs and in tools/bench-preflight.sh.
The readback test does not depend on the flag. In DispatchRegistry::match, first_covered_requested_output() → tensor_is_readback_request() runs on every match whether the flag is set or not. Its decision depends only on GGML_TENSOR_FLAG_OUTPUT together with a zha-readback./hrx-readback. name prefix. The decline (continue) also happens whether or not the flag is set; the flag only decides whether the decline is printed. So the hypothesis in the body ("the logging path calls tensor_is_readback_request, which might change binding") does not hold at the pin. The call is in the match path, not the log path, and it runs in both arms.
So by the source, the accepted dispatch set is identical in both arms. The only side effects the flag adds are a handful of fprintf(stderr) calls during graph planning, and one mutex plus a set lookup per match. That gives two consequences for the table:
The 271.75 vs 118.86 t/s pair does not look like a cost of this code. A few dozen stderr lines and one lock per planned node are microseconds per graph, not a 2.3× prefill. Something else differed between those two runs: box load (see the correction in ~/briefs/DEMO-NPU-06B.md, where a quiet box against load 20-49 moved a rate by about 2×), the binary, or GTT state left by the run before. The withdrawal in the body is right, but for a different reason: the flag did not change what ran.
What would settle it (GPU, one gpu-hold ticket, about 6 × the 18.8k request):
Use the same binary, recording ldd … | grep hrx and the fork commit, and alternate the arms A B A B A B with the box load recorded beside each run.
Count NaN logits and completion per run.
Run GGML_HRX_LOG_DISPATCH=1 in both arms with stderr going to /dev/null in one of them. That separates "the flag" from "the writes", if anyone still suspects the writes.
Cross-reference: #1536's close test (08:17Z, cl-hrxnan) ran with GGML_HRX_LOG_DISPATCH=1 set, with 0 guard lines in 6 of 6 identical 18.8k requests on 1703bfea. If the flag changed behaviour, as this issue's body supposes, that close would rest on the arm this issue calls clean.
My reading above says it does not. At the pin the flag only switches on two fprintf calls, and the readback and decline check runs in both arms. So the close stands on the source.
The only thing that would tighten it is the unset arm with the same build and command: six requests, about 40 min of one gpu-hold ticket at the rates in that run (334-561 s each). I am not taking that ticket today, because it would hold the GPU for 40 minutes on release day while #856 and #1962 need it. If the #1536 or #1962 row lead wants it before the release, it can go on the same script that made the close test, with only the env line changed.
Proposed close condition for this issue: either that unset run, or the row lead accepting the source read. Either way, the 2-vs-0 and 271.75/118.86 rows in the body should not be cited as a flag effect.
Activity
bong-water-water-bong commentedon Nov 11, 2026
Read at the engine pin (fork
7a1f2b81ce17, read only, no device run): in the fork, the flag gates only two stderr prints. The flag does not decide which registration is selected, how values are bound, or what is read back.Every reader of
GGML_HRX_LOG_DISPATCHin the fork outside*.md, at the pin and also atc16424f5620d(the off-pin checkout #2141 found on this box):The HRX runtime (
hrx-systemat82804197bc) does not read the flag. On engine main it appears only in docs and intools/bench-preflight.sh.The readback test does not depend on the flag. In
DispatchRegistry::match,first_covered_requested_output()→tensor_is_readback_request()runs on every match whether the flag is set or not. Its decision depends only onGGML_TENSOR_FLAG_OUTPUTtogether with azha-readback./hrx-readback.name prefix. The decline (continue) also happens whether or not the flag is set; the flag only decides whether the decline is printed. So the hypothesis in the body ("the logging path callstensor_is_readback_request, which might change binding") does not hold at the pin. The call is in the match path, not the log path, and it runs in both arms.So by the source, the accepted dispatch set is identical in both arms. The only side effects the flag adds are a handful of
fprintf(stderr)calls during graph planning, and one mutex plus a set lookup per match. That gives two consequences for the table:=1run comes up with probability about 1/3. A 1-1 split is what chance gives one time in three.~/briefs/DEMO-NPU-06B.md, where a quiet box against load 20-49 moved a rate by about 2×), the binary, or GTT state left by the run before. The withdrawal in the body is right, but for a different reason: the flag did not change what ran.What would settle it (GPU, one
gpu-holdticket, about 6 × the 18.8k request):ldd … | grep hrxand the fork commit, and alternate the arms A B A B A B with the box load recorded beside each run.GGML_HRX_LOG_DISPATCH=1in both arms withstderrgoing to/dev/nullin one of them. That separates "the flag" from "the writes", if anyone still suspects the writes.I have not run it. The read is all I claim.
bong-water-water-bong commentedon Nov 11, 2026
Cross-reference: #1536's close test (08:17Z, cl-hrxnan) ran with
GGML_HRX_LOG_DISPATCH=1set, with 0 guard lines in 6 of 6 identical 18.8k requests on1703bfea. If the flag changed behaviour, as this issue's body supposes, that close would rest on the arm this issue calls clean.My reading above says it does not. At the pin the flag only switches on two
fprintfcalls, and the readback and decline check runs in both arms. So the close stands on the source.The only thing that would tighten it is the unset arm with the same build and command: six requests, about 40 min of one
gpu-holdticket at the rates in that run (334-561 s each). I am not taking that ticket today, because it would hold the GPU for 40 minutes on release day while #856 and #1962 need it. If the #1536 or #1962 row lead wants it before the release, it can go on the same script that made the close test, with only the env line changed.Proposed close condition for this issue: either that unset run, or the row lead accepting the source read. Either way, the 2-vs-0 and 271.75/118.86 rows in the body should not be cited as a flag effect.
docs: GGML_HRX_LOG_DISPATCH changes behaviour, so say it where the pa…
docs: GGML_HRX_LOG_DISPATCH -- say what the source shows, and what to…
tests: GGML_HRX_LOG_DISPATCH stays a logging switch (engine #2143) (#…