Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion docs/hrx.md
Original file line number Diff line number Diff line change
Expand Up @@ -516,6 +516,16 @@ the first sampled token, which is [engine#1536](https://github.com/1bit-MONSTER/
5.4K-token prompt in a new process 4.46 -> 3.43 s, first short prompt 0.47 -> 0.22 s, with
identical output. `GGML_HRX_JIT_CACHE=0` turns it off; it is off while a Loom sanitizer is set.
Deleting the directory is always safe.
- **Routed-expert MoE on the HRX route (flint-40b-a2.5b).** `flint-UD-Q4_K_XL`, a 6-of-256 MoE that needs
31.01 GiB at its own context, serves with `--device hrx --no-strict-device`: 64-token completions at
**50.66 tok/s**, median of **6 runs** (45.69–50.92, spread 1.114x). A step reads 2.213 GB at this depth, so
that is **~112 GB/s effective**, **44 % of this box's 256 GB/s peak and 51 % of the 220 GB/s goal**. (A 2-run
sample before this read 42.61 tok/s with a 1.418x spread; six runs move the median up 19 %, which is the
argument for not quoting two.) The flag is required because `embd` is `GET_ROWS` f32 `ne=[2048,2,1,1]` and strict placement refuses
the model over those 2 of 3,576 nodes; see `dispatch-get-rows.cpp`, whose demand list is only ever given
`Q8_1`, so a tied f32 embedding never matches and lands on the CPU. Two rows would finish it: demand `F32`
for a non-Qwen f32 gather, or accept a named off-device op instead of refusing the model. Measured
2026-11-14 under one gpu-hold window, `dmesg` clean before and after.
- **Fused residual scale (llama.cpp #35).** ZAYA scales both residual branches twice per layer,
`(x + bx) * sx + (r + br) * sr`, which ran as up to nine ADD / MUL dispatches per layer. Each side
now runs as one dispatch (`common.res_scale_pair.*`, `ops/res_scale_pair_f32.loom`): 3,366 -> 3,165
Expand Down Expand Up @@ -1069,7 +1079,9 @@ The text is identical. Over six interleaved arms (trace off), prompt eval was 32
(or `GGML_HRX_PREFILL_FOLLOWS_DECODE=0`) restores the fused kernel for Q4_K, `GGML_HRX_Q6K_PACKED_ROWS=0` the stored Q6_K binding.
Before either, making decode step aside instead was measured and does not help:
the same duplicates, and decode 8.5 % slower. flint UD-Q4_K_XL (routed experts) reads every weight stored and holds one copy per
weight with or without the default.
weight with or without the default. (That sentence is about **resident copies**, not bytes read per decode step: a step reads
only the routed experts, `expert_used_count / expert_count` = 6 of 256 for this model. Had it read all 256, flint's measured
50.66 tok/s would imply 1,269 GB/s, above this box's 256 GB/s peak.)

**Since pin `962507f8e` a short prompt holds no weight twice either** (llama.cpp #324, #325). At 6 to 8 tokens the few-token
kernels took the batch and read the stored layout of weights that prefill reads in the packed rows, so a short prompt or a
Expand Down
Loading