From 24912bb6633fd287e87922efe405f55f7ff1705e Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Wed, 11 Nov 2026 06:51:08 -0400 Subject: [PATCH] docs: GGML_HRX_LOG_DISPATCH -- say what the source shows, and what to do either way (#2143) #2162 told readers the flag is "a different code path". At the engine's pin it only gates fprintf calls in dispatch-registry.cpp; the readback check and its decline run with or without it. The two measurements it quoted are one run per arm (4-in-6 base failure rate for #1536) and one rate pair with box load unrecorded. Replace the claim with the advice that holds either way: set the flag in both arms and record load. Also moves the zyphra.md blockquote inside its list item, so it no longer ends the list, and drops the two blank lines #2162 added. Co-Authored-By: Claude Opus 5.5 --- docs/determinism.md | 13 +++++++------ docs/zyphra.md | 13 +++++++------ 2 files changed, 14 insertions(+), 12 deletions(-) diff --git a/docs/determinism.md b/docs/determinism.md index 7789b47df..f4c6039bd 100644 --- a/docs/determinism.md +++ b/docs/determinism.md @@ -244,11 +244,13 @@ comparators and `logprob_agree.py`'s prompts, so it measures what this page docu ### What the `hrx` width gate is, by name -> **It is not a behaviour-neutral instrument (engine#2143).** With this flag set, an 18,782-token request -> *completes* where the same binary, model and request **fails** without it, and NaN-logit incidents went from 2 -> to 0; a plain 18.6k prefill measured **271.75 t/s** unset against **118.86 t/s** set. A 2.3x gap is not -> "logging cost" — it is a different code path. So a rate taken with the flag is not comparable to one taken -> without it, and an A/B that sets it on one arm only is not an A/B. +> **Set it in both arms of an A/B, and record box load beside any rate (engine#2143).** #2143 asked whether this +> flag changes behaviour: one 18,782-token request failed without it and completed with it, and one 18.6k prefill +> measured 271.75 t/s unset against 118.86 t/s set. At the engine's pin the flag only gates `fprintf` calls +> (`dispatch_log_match()` and `dispatch_log_requested_output()` in `dispatch-registry.cpp`): the readback check and +> its decline run in both arms, so the dispatches selected are the same either way. Each of those measurements is +> one run per arm, #1536 then failed in 4 of 6 runs, and box load moves a rate by about 2x here, so neither shows +> a flag effect. Setting it in both arms is right whichever way #2143 settles. `GGML_HRX_LOG_DISPATCH=1` prints each dispatch registration the first time it matches (`dispatch_registry/dispatch-registry.cpp`), so the set of registrations a serve logs is the set of @@ -262,7 +264,6 @@ step splits into two disjoint sets: | 1 (solo, and any sequential history) | `hip.qkv_rope`, `kquant.mul_mat.decode_f32`, `hip.dense_kquant_swiglu`, `hip.gemv_add_norm`, `common.mul_mat.f32_f32_decode` | | 2 (concurrent) | `hip.kquant_tokens.mm`, `hip.kquant_tokens.mm_add`, `hip.kquant_tokens.swiglu`, `common.rmsnorm_binary_f32` | - Seventeen more registrations match at both widths, including the attention kernel (`common.flash_attention_decode_split_next_q8`): on `hrx`, unlike on `cpu`, attention is *not* where the width dependence comes from. What changes is the **fusion boundaries**. The residual add and diff --git a/docs/zyphra.md b/docs/zyphra.md index 2995a5cce..38329c045 100644 --- a/docs/zyphra.md +++ b/docs/zyphra.md @@ -264,7 +264,6 @@ have been measured in three windows and answered differently every time. prefill drop noted below, and it is why a throughput row here is inseparable from *which quantizer built it*. **Measured here on 2026-11-08**: two arms that differ *only* in the adapter tensors' type (F16 ×26 vs Q6_K ×13 + Q8_0 ×13), benched back to back in one ticket and one window — - | shape | F16 pairs | block-quantized pairs | cost | |---|---|---|---| | `pp8` | 114.19 tok/s | 14.03 tok/s | **8.1×** | @@ -281,11 +280,13 @@ have been measured in three windows and answered differently every time. `input_size = 128` and HRX's block-format admission needs `>= 256` while the float predicate needs only `>= 4`. So the pair leaves the device for *any* block type, including the substitute — the same "the label is not evidence" rule this page applies to arm names, applied to the tensors inside them. -> **It is not a behaviour-neutral instrument (engine#2143).** With this flag set, an 18,782-token request -> *completes* where the same binary, model and request **fails** without it, and NaN-logit incidents went from 2 -> to 0; a plain 18.6k prefill measured **271.75 t/s** unset against **118.86 t/s** set. A 2.3x gap is not -> "logging cost" — it is a different code path. So a rate taken with the flag is not comparable to one taken -> without it, and an A/B that sets it on one arm only is not an A/B. + > **Set it in both arms of an A/B, and record box load beside any rate (engine#2143).** #2143 asked whether this + > flag changes behaviour: one 18,782-token request failed without it and completed with it, and one 18.6k prefill + > measured 271.75 t/s unset against 118.86 t/s set. At the engine's pin the flag only gates `fprintf` calls + > (`dispatch_log_match()` and `dispatch_log_requested_output()` in `dispatch-registry.cpp`): the readback check and + > its decline run in both arms, so the dispatches selected are the same either way. Each of those measurements is + > one run per arm, #1536 then failed in 4 of 6 runs, and box load moves a rate by about 2x here, so neither shows + > a flag effect. Setting it in both arms is right whichever way #2143 settles. - **A control on 2026-11-08 shows how far a single reading can be off, and it retire the A/B this page would need.** One file whose form cause 2 did **not** touch (`Zamba-7B-v1` `Q6_K`) was re-benched on a current build: `pp512 211.16 (sd 170.84)`, `pp4096 47.14 (sd 12.73)`, `tg128 4.26