Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions docs/determinism.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,11 +244,13 @@ comparators and `logprob_agree.py`'s prompts, so it measures what this page docu

### What the `hrx` width gate is, by name

> **It is not a behaviour-neutral instrument (engine#2143).** With this flag set, an 18,782-token request
> *completes* where the same binary, model and request **fails** without it, and NaN-logit incidents went from 2
> to 0; a plain 18.6k prefill measured **271.75 t/s** unset against **118.86 t/s** set. A 2.3x gap is not
> "logging cost" — it is a different code path. So a rate taken with the flag is not comparable to one taken
> without it, and an A/B that sets it on one arm only is not an A/B.
> **Set it in both arms of an A/B, and record box load beside any rate (engine#2143).** #2143 asked whether this
> flag changes behaviour: one 18,782-token request failed without it and completed with it, and one 18.6k prefill
> measured 271.75 t/s unset against 118.86 t/s set. At the engine's pin the flag only gates `fprintf` calls
> (`dispatch_log_match()` and `dispatch_log_requested_output()` in `dispatch-registry.cpp`): the readback check and
> its decline run in both arms, so the dispatches selected are the same either way. Each of those measurements is
> one run per arm, #1536 then failed in 4 of 6 runs, and box load moves a rate by about 2x here, so neither shows
> a flag effect. Setting it in both arms is right whichever way #2143 settles.

`GGML_HRX_LOG_DISPATCH=1` prints each dispatch registration the first time it matches
(`dispatch_registry/dispatch-registry.cpp`), so the set of registrations a serve logs is the set of
Expand All @@ -262,7 +264,6 @@ step splits into two disjoint sets:
| 1 (solo, and any sequential history) | `hip.qkv_rope`, `kquant.mul_mat.decode_f32`, `hip.dense_kquant_swiglu`, `hip.gemv_add_norm`, `common.mul_mat.f32_f32_decode` |
| 2 (concurrent) | `hip.kquant_tokens.mm`, `hip.kquant_tokens.mm_add`, `hip.kquant_tokens.swiglu`, `common.rmsnorm_binary_f32` |


Seventeen more registrations match at both widths, including the attention kernel
(`common.flash_attention_decode_split_next_q8`): on `hrx`, unlike on `cpu`, attention is *not* where
the width dependence comes from. What changes is the **fusion boundaries**. The residual add and
Expand Down
13 changes: 7 additions & 6 deletions docs/zyphra.md
Original file line number Diff line number Diff line change
Expand Up @@ -264,7 +264,6 @@ have been measured in three windows and answered differently every time.
prefill drop noted below, and it is why a throughput row here is inseparable from *which quantizer built it*. **Measured here on 2026-11-08**: two arms
that differ *only* in the adapter tensors' type (F16 ×26 vs Q6_K ×13 + Q8_0 ×13), benched back to back in one ticket and one window —


| shape | F16 pairs | block-quantized pairs | cost |
|---|---|---|---|
| `pp8` | 114.19 tok/s | 14.03 tok/s | **8.1×** |
Expand All @@ -281,11 +280,13 @@ have been measured in three windows and answered differently every time.
`input_size = 128` and HRX's block-format admission needs `>= 256` while the float predicate needs only `>= 4`. So the pair leaves the device for *any*
block type, including the substitute — the same "the label is not evidence" rule this page applies to arm names, applied to the tensors inside them.

> **It is not a behaviour-neutral instrument (engine#2143).** With this flag set, an 18,782-token request
> *completes* where the same binary, model and request **fails** without it, and NaN-logit incidents went from 2
> to 0; a plain 18.6k prefill measured **271.75 t/s** unset against **118.86 t/s** set. A 2.3x gap is not
> "logging cost" — it is a different code path. So a rate taken with the flag is not comparable to one taken
> without it, and an A/B that sets it on one arm only is not an A/B.
> **Set it in both arms of an A/B, and record box load beside any rate (engine#2143).** #2143 asked whether this
> flag changes behaviour: one 18,782-token request failed without it and completed with it, and one 18.6k prefill
> measured 271.75 t/s unset against 118.86 t/s set. At the engine's pin the flag only gates `fprintf` calls
> (`dispatch_log_match()` and `dispatch_log_requested_output()` in `dispatch-registry.cpp`): the readback check and
> its decline run in both arms, so the dispatches selected are the same either way. Each of those measurements is
> one run per arm, #1536 then failed in 4 of 6 runs, and box load moves a rate by about 2x here, so neither shows
> a flag effect. Setting it in both arms is right whichever way #2143 settles.

- **A control on 2026-11-08 shows how far a single reading can be off, and it retire the A/B this page would need.** One file whose form cause 2
did **not** touch (`Zamba-7B-v1` `Q6_K`) was re-benched on a current build: `pp512 211.16 (sd 170.84)`, `pp4096 47.14 (sd 12.73)`, `tg128 4.26
Expand Down
Loading