Skip to content

hrx: GGML_HRX_LOG_DISPATCH is not behaviour-neutral — the 18.8k request fails without it and succeeds with it #2143

Closed

Description

@bong-water-water-bong

Summary

GGML_HRX_LOG_DISPATCH=1 changes behaviour, not just logging. Same binary, same model, same request, mask kernels on:

arm NaN logits the 18,782-token /completion
LOG_DISPATCH unset 2 fails — "Remote end closed connection without response"
LOG_DISPATCH=1 0 succeeds, prompt_n = 18782

A logging environment variable must not decide whether a request completes. Two consequences:

  1. Every A/B and every rate taken with LOG_DISPATCH=1 is suspect. I have one of my own: a plain 18.6k prefill measured 271.75 t/s with the flag unset and 118.86 t/s with it set — I reported that as a 2.3× logging cost an hour ago and I am now withdrawing the interpretation, because the flag demonstrably changes what runs. Any lane that A/B'd dispatch selection with this flag has the same problem.
  2. It explains part of hrx: intermittent all-NaN logits at n_slots=1 and 18.8k context on the pin that carries #227 (count=248320 of 248320, 4 sightings in 6 long-context runs) #1536's intermittency. That issue has 4 sightings in 6 long-context runs and has resisted a deterministic repro; a debugging flag that changes the outcome is exactly the kind of variable that turns a reproducible failure into a flaky one — and the runs that carried it would have been clean.

What dispatch_log_requested_output does

The logging path calls tensor_is_readback_request(value->tensor) (see dispatch-registry.cpp, first_covered_requested_output / dispatch_log_requested_output). If marking a value as a readback changes how it is bound or kept resident, then logging is not observational — and that is the first thing to check.

Suggested first step

GGML_HRX_LOG_DISPATCH=1 with the request from #1536/#1925 (18,782 tokens, Flash-Next, mask kernels on) and a diff of the accepted dispatch set against the same run without the flag. If the sets differ, the flag is selection-affecting by definition; if they are identical, the difference is downstream (binding/readback) and the check moves there.

Kernel internals stay private; this issue needs no kernel source.

Activity

  1. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Read at the engine pin (fork 7a1f2b81ce17, read only, no device run): in the fork, the flag gates only two stderr prints. The flag does not decide which registration is selected, how values are bound, or what is read back.

    Every reader of GGML_HRX_LOG_DISPATCH in the fork outside *.md, at the pin and also at c16424f5620d (the off-pin checkout #2141 found on this box):

    dispatch-registry.cpp:138  dispatch_log_requested_output()  static const bool enabled = getenv(...) != nullptr; if (!enabled) return;  -> fprintf only, capped at 8 lines
    dispatch-registry.cpp:161  dispatch_log_match()             same gate -> fprintf once per registration name (a std::set under a mutex)
    

    The HRX runtime (hrx-system at 82804197bc) does not read the flag. On engine main it appears only in docs and in tools/bench-preflight.sh.

    The readback test does not depend on the flag. In DispatchRegistry::match, first_covered_requested_output() → tensor_is_readback_request() runs on every match whether the flag is set or not. Its decision depends only on GGML_TENSOR_FLAG_OUTPUT together with a zha-readback./hrx-readback. name prefix. The decline (continue) also happens whether or not the flag is set; the flag only decides whether the decline is printed. So the hypothesis in the body ("the logging path calls tensor_is_readback_request, which might change binding") does not hold at the pin. The call is in the match path, not the log path, and it runs in both arms.

    So by the source, the accepted dispatch set is identical in both arms. The only side effects the flag adds are a handful of fprintf(stderr) calls during graph planning, and one mutex plus a set lookup per match. That gives two consequences for the table:

    1. One run per arm cannot separate the arms. hrx: intermittent all-NaN logits at n_slots=1 and 18.8k context on the pin that carries #227 (count=248320 of 248320, 4 sightings in 6 long-context runs) #1536 fails 4 of 6 times. If the flag has no effect, a single unset run fails with probability about 2/3, and a single clean =1 run comes up with probability about 1/3. A 1-1 split is what chance gives one time in three.
    2. The 271.75 vs 118.86 t/s pair does not look like a cost of this code. A few dozen stderr lines and one lock per planned node are microseconds per graph, not a 2.3× prefill. Something else differed between those two runs: box load (see the correction in ~/briefs/DEMO-NPU-06B.md, where a quiet box against load 20-49 moved a rate by about 2×), the binary, or GTT state left by the run before. The withdrawal in the body is right, but for a different reason: the flag did not change what ran.

    What would settle it (GPU, one gpu-hold ticket, about 6 × the 18.8k request):

    I have not run it. The read is all I claim.

  2. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Cross-reference: #1536's close test (08:17Z, cl-hrxnan) ran with GGML_HRX_LOG_DISPATCH=1 set, with 0 guard lines in 6 of 6 identical 18.8k requests on 1703bfea. If the flag changed behaviour, as this issue's body supposes, that close would rest on the arm this issue calls clean.

    My reading above says it does not. At the pin the flag only switches on two fprintf calls, and the readback and decline check runs in both arms. So the close stands on the source.

    The only thing that would tighten it is the unset arm with the same build and command: six requests, about 40 min of one gpu-hold ticket at the rates in that run (334-561 s each). I am not taking that ticket today, because it would hold the GPU for 40 minutes on release day while #856 and #1962 need it. If the #1536 or #1962 row lead wants it before the release, it can go on the same script that made the close test, with only the env line changed.

    Proposed close condition for this issue: either that unset run, or the row lead accepting the source read. Either way, the 2-vs-0 and 271.75/118.86 rows in the body should not be cited as a flag effect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions