diff --git a/docs/hrx.md b/docs/hrx.md index 0dd2f8a38..9947b8c54 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -668,6 +668,60 @@ It registered the file mapping as a KFD userptr, which the kernel always write-f its mtime, and never finished on large models while writeback kept invalidating it (engine#1339). HRX's repacked copy of the dense weights still sits next to the device buffer in GTT. +### Measuring on `HRX0`: what makes a number citable + +Numbers measured on this device swing far more between *runs* than between *files*, so the harness matters as much as the arm. Everything below is +either a guardrail the engine prints or an effect measured on this box on 2026-11-07/08; where another lane measured it first, they are named. + +**1. The GTT pool is finite and it is shared.** Every launch prints the guardrail line: + +``` +ggml_hrx: HRX0 guardrail: GTT total 120.0 GiB, in use 40.7 GiB, ceiling 104.0 GiB (reserve 16.0 GiB), + pool budget 63.3 GiB - a launch that needs more is refused here rather than faulting the device (engine #680) +``` + +`pool budget = ceiling - in use`, and a launch needing more than that is **refused**, not faulted. So the first thing a measurement needs is the budget, +not the file: two lanes independently arrived at the same rule from opposite sides — the Zyphra lane refuses below a 12 GiB budget for a 5.8 GiB model, and +the architecture lane refuses below `3.0 x weights + 2 GiB` (an 8B row was refused at budgets of 13.6 / 13.5 / 10.8 GiB). Over eight hours of this box's +evening the budget sat at **14-17 GiB** while long-lived services held the rest, which is why runs "that worked yesterday" crawl today. Two `llama-server` +processes from a resident embed install have been measured holding **94.3 GiB of 120** while arms died with `unable to allocate HRX0 buffer` at a +`pool budget 42.6 GiB` — so the co-tenant is a service that is **supposed to be resident**, and "take a quiet box" is a scheduling problem, not a courtesy. + +**2. Budget the ticket for the *load*, not for the arms.** Loading is single-core-bound when other jobs share the four pinned cores: a 24 GiB GGUF spent +**35 minutes at 88 % of one core with GTT unmoved** — it was still loading, having measured nothing. A `gpu-hold -t 40` therefore expires before the first +arm, which happened four times in one night. Either take a long ticket on a quiet box, or reduce the **number of loads** (one process per arm means one +load per arm). + +**3. A fallback is invisible in the dispatch histogram.** `GGML_HRX_LOG_DISPATCH=1` lists the registrations that **ran**; an op that never dispatched on +the device leaves **no line at all**. Measured: two arms differing only in one tensor type's format had a **near-identical** registration set (one extra +line, none missing) while `pp512` differed **5.4x** and `pp8` **8.1x**. So the histogram answers *"which kernel won"*, never *"did something fall back"*. + +Use the right instrument for each question — three are needed and the histogram is only the first: + +| question | instrument | +|---|---| +| which kernel **won** | `GGML_HRX_LOG_DISPATCH=1` (one line per registration that ran) | +| which node **fell back** | `GGML_HRX_REPORT=2` — `placement: 994 nodes, 1 off HRX (GET_ROWS x 1), 2 graph splits`, and `GGML_HRX_REPORT=1` for the once-at-load fallback line. Add `-lv 4` and it prints the matcher's own rejection reason and the node that broke the graph | +| was the result **right** | `test-backend-ops -o -b HRX0`, known-answer against the CPU backend | + +And when a fallback is suspected but all three say nothing, the cost still shows up in the **timing** — against a control arm in the same window. + +**4. Two windows are not a comparison.** The same unchanged file measured `211.16 / 47.14 / 4.26 tok/s` (pp512 / pp4096 / tg128) in one hour and +`108.07 / 92.28 / 12.20` in another — **2-2.9x apart, with sd 71 % on decode**. Rows may only be compared inside **one ticket**, with the guardrail's GTT +figure recorded at **each arm** and an unchanged-file control in the same window; otherwise the report is of the window, not of the file. + +**5. A hot die starves the queue, and the thresholds do not line up.** `~/lb/thermal-run.sh` freezes a CPU job at **78 C** and thaws at 68 C, while +`gpu-hold` only starts below **75 C**. A CPU job under thermal-run can therefore park the die in the 75-78 C band indefinitely — too hot for the GPU +queue to start, too cool to be paused — and a queue of 22 tickets waited an hour behind exactly that. CPU-heavy measurement belongs under thermal-run +*and* out of a window with a deep device queue. + +**6. State the binary, the pool budget and the tensor types with every row.** The build a row came from may be deleted before the row is read (it +happened: a 12-row table outlived its build by a day), and a file's *name* is not its format — a GGUF called `UD-Q4_K_XL` measured here was Q8_0 x279 / +F32 x246 / Q5_K x129 / IQ4_XS x69 / BF16 x47 / Q4_K x6, i.e. six tensors of the type its name advertises. Read the types out of the produced file, and +quote them, before quoting a number. **And digest the binary you actually *invoke*, not the one you built**: a gate run against a snapshot `bin/` +taken hours before a rebuild printed a clean `484/484` from the *pre-change* library, which reads exactly like a no-op. The citable form is the arm's own +`md5sum` of the invoked binary plus `libggml-hrx.so`, printed by the run that produced the row. + ### Known issues on `HRX0` - **Several sequences per batch fail.** `llama-perplexity` with `n_seq` > 1 stops on an