Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions docs/hrx.md
Original file line number Diff line number Diff line change
Expand Up @@ -668,6 +668,60 @@ It registered the file mapping as a KFD userptr, which the kernel always write-f
its mtime, and never finished on large models while writeback kept invalidating it (engine#1339). HRX's repacked copy of the
dense weights still sits next to the device buffer in GTT.

### Measuring on `HRX0`: what makes a number citable

Numbers measured on this device swing far more between *runs* than between *files*, so the harness matters as much as the arm. Everything below is
either a guardrail the engine prints or an effect measured on this box on 2026-11-07/08; where another lane measured it first, they are named.

**1. The GTT pool is finite and it is shared.** Every launch prints the guardrail line:

```
ggml_hrx: HRX0 guardrail: GTT total 120.0 GiB, in use 40.7 GiB, ceiling 104.0 GiB (reserve 16.0 GiB),
pool budget 63.3 GiB - a launch that needs more is refused here rather than faulting the device (engine #680)
```

`pool budget = ceiling - in use`, and a launch needing more than that is **refused**, not faulted. So the first thing a measurement needs is the budget,
not the file: two lanes independently arrived at the same rule from opposite sides — the Zyphra lane refuses below a 12 GiB budget for a 5.8 GiB model, and
the architecture lane refuses below `3.0 x weights + 2 GiB` (an 8B row was refused at budgets of 13.6 / 13.5 / 10.8 GiB). Over eight hours of this box's
evening the budget sat at **14-17 GiB** while long-lived services held the rest, which is why runs "that worked yesterday" crawl today. Two `llama-server`
processes from a resident embed install have been measured holding **94.3 GiB of 120** while arms died with `unable to allocate HRX0 buffer` at a
`pool budget 42.6 GiB` — so the co-tenant is a service that is **supposed to be resident**, and "take a quiet box" is a scheduling problem, not a courtesy.

**2. Budget the ticket for the *load*, not for the arms.** Loading is single-core-bound when other jobs share the four pinned cores: a 24 GiB GGUF spent
**35 minutes at 88 % of one core with GTT unmoved** — it was still loading, having measured nothing. A `gpu-hold -t 40` therefore expires before the first
arm, which happened four times in one night. Either take a long ticket on a quiet box, or reduce the **number of loads** (one process per arm means one
load per arm).

**3. A fallback is invisible in the dispatch histogram.** `GGML_HRX_LOG_DISPATCH=1` lists the registrations that **ran**; an op that never dispatched on
the device leaves **no line at all**. Measured: two arms differing only in one tensor type's format had a **near-identical** registration set (one extra
line, none missing) while `pp512` differed **5.4x** and `pp8` **8.1x**. So the histogram answers *"which kernel won"*, never *"did something fall back"*.

Use the right instrument for each question — three are needed and the histogram is only the first:

| question | instrument |
|---|---|
| which kernel **won** | `GGML_HRX_LOG_DISPATCH=1` (one line per registration that ran) |
| which node **fell back** | `GGML_HRX_REPORT=2` — `placement: 994 nodes, 1 off HRX (GET_ROWS x 1), 2 graph splits`, and `GGML_HRX_REPORT=1` for the once-at-load fallback line. Add `-lv 4` and it prints the matcher's own rejection reason and the node that broke the graph |
| was the result **right** | `test-backend-ops -o <OP> -b HRX0`, known-answer against the CPU backend |

And when a fallback is suspected but all three say nothing, the cost still shows up in the **timing** — against a control arm in the same window.

**4. Two windows are not a comparison.** The same unchanged file measured `211.16 / 47.14 / 4.26 tok/s` (pp512 / pp4096 / tg128) in one hour and
`108.07 / 92.28 / 12.20` in another — **2-2.9x apart, with sd 71 % on decode**. Rows may only be compared inside **one ticket**, with the guardrail's GTT
figure recorded at **each arm** and an unchanged-file control in the same window; otherwise the report is of the window, not of the file.

**5. A hot die starves the queue, and the thresholds do not line up.** `~/lb/thermal-run.sh` freezes a CPU job at **78 C** and thaws at 68 C, while
`gpu-hold` only starts below **75 C**. A CPU job under thermal-run can therefore park the die in the 75-78 C band indefinitely — too hot for the GPU
queue to start, too cool to be paused — and a queue of 22 tickets waited an hour behind exactly that. CPU-heavy measurement belongs under thermal-run
*and* out of a window with a deep device queue.

**6. State the binary, the pool budget and the tensor types with every row.** The build a row came from may be deleted before the row is read (it
happened: a 12-row table outlived its build by a day), and a file's *name* is not its format — a GGUF called `UD-Q4_K_XL` measured here was Q8_0 x279 /
F32 x246 / Q5_K x129 / IQ4_XS x69 / BF16 x47 / Q4_K x6, i.e. six tensors of the type its name advertises. Read the types out of the produced file, and
quote them, before quoting a number. **And digest the binary you actually *invoke*, not the one you built**: a gate run against a snapshot `bin/`
taken hours before a rebuild printed a clean `484/484` from the *pre-change* library, which reads exactly like a no-op. The citable form is the arm's own
`md5sum` of the invoked binary plus `libggml-hrx.so`, printed by the run that produced the row.

### Known issues on `HRX0`

- **Several sequences per batch fail.** `llama-perplexity` with `n_seq` > 1 stops on an
Expand Down
Loading