From f8d433829784c5905ec28a04defc064e2668592e Mon Sep 17 00:00:00 2001 From: pi-zquant Date: Sun, 8 Nov 2026 19:01:42 -0400 Subject: [PATCH 1/2] docs(hrx): what makes a measurement on HRX0 citable - the pool budget, the load, the blind histogram docs/hrx.md documents the device, the patches and the fixed bugs but says nothing about how to measure on it, and this week four separate attempts at one A/B died on a precondition rather than on the code. The section added here is the set of rules that would have prevented them, each with the measurement behind it: 1. the guardrail's pool budget (ceiling 104 GiB of 120, in use subtracted) and the two lanes' independent thresholds for refusing to measure (12 GiB budget for a 5.8 GiB model; 3.0x weights + 2 GiB) - over one evening the budget sat at 14-17 GiB while long-lived services held the rest; 2. budget the ticket for the LOAD: a 24 GiB file spent 35 min at 88% of one core with GTT unmoved, so a -t 40 expires before the first arm (happened four times); 3. GGML_HRX_LOG_DISPATCH lists what RAN, so a fallback leaves no line - measured on two arms with near-identical registration sets and 5.4x different pp512; 4. two windows are not a comparison: the same file measured 2-2.9x apart with 71% sd, so rows are compared inside one ticket with the GTT figure per arm and a control; 5. thermal-run freezes at 78 C while gpu-hold starts below 75 C, so a CPU job can park the die in a band where 22 tickets wait an hour; 6. every row states its binary, pool budget and the tensor types read from the file - a build can be deleted before its table is read, and a name is not a format (UD-Q4_K_XL here was six Q4_K tensors). --- docs/hrx.md | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/docs/hrx.md b/docs/hrx.md index 0dd2f8a38..a2c2edcff 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -668,6 +668,47 @@ It registered the file mapping as a KFD userptr, which the kernel always write-f its mtime, and never finished on large models while writeback kept invalidating it (engine#1339). HRX's repacked copy of the dense weights still sits next to the device buffer in GTT. +### Measuring on `HRX0`: what makes a number citable + +Numbers measured on this device swing far more between *runs* than between *files*, so the harness matters as much as the arm. Everything below is +either a guardrail the engine prints or an effect measured on this box on 2026-11-07/08; where another lane measured it first, they are named. + +**1. The GTT pool is finite and it is shared.** Every launch prints the guardrail line: + +``` +ggml_hrx: HRX0 guardrail: GTT total 120.0 GiB, in use 40.7 GiB, ceiling 104.0 GiB (reserve 16.0 GiB), + pool budget 63.3 GiB - a launch that needs more is refused here rather than faulting the device (engine #680) +``` + +`pool budget = ceiling - in use`, and a launch needing more than that is **refused**, not faulted. So the first thing a measurement needs is the budget, +not the file: two lanes independently arrived at the same rule from opposite sides — the Zyphra lane refuses below a 12 GiB budget for a 5.8 GiB model, and +the architecture lane refuses below `3.0 x weights + 2 GiB` (an 8B row was refused at budgets of 13.6 / 13.5 / 10.8 GiB). Over eight hours of this box's +evening the budget sat at **14-17 GiB** while long-lived services held the rest, which is why runs "that worked yesterday" crawl today. + +**2. Budget the ticket for the *load*, not for the arms.** Loading is single-core-bound when other jobs share the four pinned cores: a 24 GiB GGUF spent +**35 minutes at 88 % of one core with GTT unmoved** — it was still loading, having measured nothing. A `gpu-hold -t 40` therefore expires before the first +arm, which happened four times in one night. Either take a long ticket on a quiet box, or reduce the **number of loads** (one process per arm means one +load per arm). + +**3. A fallback is invisible in the dispatch histogram.** `GGML_HRX_LOG_DISPATCH=1` lists the registrations that **ran**; an op that never dispatched on +the device leaves **no line at all**. Measured: two arms differing only in one tensor type's format had a **near-identical** registration set (one extra +line, none missing) while `pp512` differed **5.4x** and `pp8` **8.1x**. So the histogram answers *"which kernel won"*, never *"did something fall back"* — +that shows up in the **timing**, against a known-answer test (`test-backend-ops -o -b HRX0`). + +**4. Two windows are not a comparison.** The same unchanged file measured `211.16 / 47.14 / 4.26 tok/s` (pp512 / pp4096 / tg128) in one hour and +`108.07 / 92.28 / 12.20` in another — **2-2.9x apart, with sd 71 % on decode**. Rows may only be compared inside **one ticket**, with the guardrail's GTT +figure recorded at **each arm** and an unchanged-file control in the same window; otherwise the report is of the window, not of the file. + +**5. A hot die starves the queue, and the thresholds do not line up.** `~/lb/thermal-run.sh` freezes a CPU job at **78 C** and thaws at 68 C, while +`gpu-hold` only starts below **75 C**. A CPU job under thermal-run can therefore park the die in the 75-78 C band indefinitely — too hot for the GPU +queue to start, too cool to be paused — and a queue of 22 tickets waited an hour behind exactly that. CPU-heavy measurement belongs under thermal-run +*and* out of a window with a deep device queue. + +**6. State the binary, the pool budget and the tensor types with every row.** The build a row came from may be deleted before the row is read (it +happened: a 12-row table outlived its build by a day), and a file's *name* is not its format — a GGUF called `UD-Q4_K_XL` measured here was Q8_0 x279 / +F32 x246 / Q5_K x129 / IQ4_XS x69 / BF16 x47 / Q4_K x6, i.e. six tensors of the type its name advertises. Read the types out of the produced file, and +quote them, before quoting a number. + ### Known issues on `HRX0` - **Several sequences per batch fail.** `llama-perplexity` with `n_seq` > 1 stops on an From fc33f63214379f1c877034dedbf5df07355b27c7 Mon Sep 17 00:00:00 2001 From: pi-zquant Date: Sun, 8 Nov 2026 19:20:41 -0400 Subject: [PATCH 2/2] docs(hrx): take the reviewer's two additions, and the co-tenant measurement Reviewer's additions, both from their own runs tonight: - rule 3 now has the counter-example I was missing: the histogram shows what WON, but GGML_HRX_REPORT=2 prints the placement summary that names what fell back ("994 nodes, 1 off HRX (GET_ROWS x 1)"), =1 prints the once-at-load fallback, and -lv 4 adds the matcher's rejection reason and the offending node. A three-instrument table replaces the single-tool advice. - rule 6 adds: digest the binary you INVOKE, not the one you built - a gate run against a snapshot bin/ from hours earlier printed a clean 484/484 from the pre-change library and read like a no-op. Plus their corroboration for rule 1: two resident llama-server processes held 94.3 GiB of 120 while arms died with 'unable to allocate HRX0 buffer' at a pool budget 42.6 GiB - the co-tenant is a service that is supposed to be resident, so 'take a quiet box' is scheduling, not courtesy. --- docs/hrx.md | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/docs/hrx.md b/docs/hrx.md index a2c2edcff..9947b8c54 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -683,7 +683,9 @@ ggml_hrx: HRX0 guardrail: GTT total 120.0 GiB, in use 40.7 GiB, ceiling 104.0 Gi `pool budget = ceiling - in use`, and a launch needing more than that is **refused**, not faulted. So the first thing a measurement needs is the budget, not the file: two lanes independently arrived at the same rule from opposite sides — the Zyphra lane refuses below a 12 GiB budget for a 5.8 GiB model, and the architecture lane refuses below `3.0 x weights + 2 GiB` (an 8B row was refused at budgets of 13.6 / 13.5 / 10.8 GiB). Over eight hours of this box's -evening the budget sat at **14-17 GiB** while long-lived services held the rest, which is why runs "that worked yesterday" crawl today. +evening the budget sat at **14-17 GiB** while long-lived services held the rest, which is why runs "that worked yesterday" crawl today. Two `llama-server` +processes from a resident embed install have been measured holding **94.3 GiB of 120** while arms died with `unable to allocate HRX0 buffer` at a +`pool budget 42.6 GiB` — so the co-tenant is a service that is **supposed to be resident**, and "take a quiet box" is a scheduling problem, not a courtesy. **2. Budget the ticket for the *load*, not for the arms.** Loading is single-core-bound when other jobs share the four pinned cores: a 24 GiB GGUF spent **35 minutes at 88 % of one core with GTT unmoved** — it was still loading, having measured nothing. A `gpu-hold -t 40` therefore expires before the first @@ -692,8 +694,17 @@ load per arm). **3. A fallback is invisible in the dispatch histogram.** `GGML_HRX_LOG_DISPATCH=1` lists the registrations that **ran**; an op that never dispatched on the device leaves **no line at all**. Measured: two arms differing only in one tensor type's format had a **near-identical** registration set (one extra -line, none missing) while `pp512` differed **5.4x** and `pp8` **8.1x**. So the histogram answers *"which kernel won"*, never *"did something fall back"* — -that shows up in the **timing**, against a known-answer test (`test-backend-ops -o -b HRX0`). +line, none missing) while `pp512` differed **5.4x** and `pp8` **8.1x**. So the histogram answers *"which kernel won"*, never *"did something fall back"*. + +Use the right instrument for each question — three are needed and the histogram is only the first: + +| question | instrument | +|---|---| +| which kernel **won** | `GGML_HRX_LOG_DISPATCH=1` (one line per registration that ran) | +| which node **fell back** | `GGML_HRX_REPORT=2` — `placement: 994 nodes, 1 off HRX (GET_ROWS x 1), 2 graph splits`, and `GGML_HRX_REPORT=1` for the once-at-load fallback line. Add `-lv 4` and it prints the matcher's own rejection reason and the node that broke the graph | +| was the result **right** | `test-backend-ops -o -b HRX0`, known-answer against the CPU backend | + +And when a fallback is suspected but all three say nothing, the cost still shows up in the **timing** — against a control arm in the same window. **4. Two windows are not a comparison.** The same unchanged file measured `211.16 / 47.14 / 4.26 tok/s` (pp512 / pp4096 / tg128) in one hour and `108.07 / 92.28 / 12.20` in another — **2-2.9x apart, with sd 71 % on decode**. Rows may only be compared inside **one ticket**, with the guardrail's GTT @@ -707,7 +718,9 @@ queue to start, too cool to be paused — and a queue of 22 tickets waited an ho **6. State the binary, the pool budget and the tensor types with every row.** The build a row came from may be deleted before the row is read (it happened: a 12-row table outlived its build by a day), and a file's *name* is not its format — a GGUF called `UD-Q4_K_XL` measured here was Q8_0 x279 / F32 x246 / Q5_K x129 / IQ4_XS x69 / BF16 x47 / Q4_K x6, i.e. six tensors of the type its name advertises. Read the types out of the produced file, and -quote them, before quoting a number. +quote them, before quoting a number. **And digest the binary you actually *invoke*, not the one you built**: a gate run against a snapshot `bin/` +taken hours before a rebuild printed a clean `484/484` from the *pre-change* library, which reads exactly like a no-op. The citable form is the arm's own +`md5sum` of the invoked binary plus `libggml-hrx.so`, printed by the run that produced the row. ### Known issues on `HRX0`