From 3409e9b9725c45eeecd93152d928d4f52be6d1c8 Mon Sep 17 00:00:00 2001 From: bcloud Date: Fri, 13 Nov 2026 21:46:30 -0400 Subject: [PATCH 1/3] =?UTF-8?q?docs/hrx:=20the=20flint=20routed-expert=20M?= =?UTF-8?q?oE=20row=20=E2=80=94=2042.61=20tok/s,=20~94.3=20GB/s,=20and=20w?= =?UTF-8?q?hy=20--no-strict-device=20is=20required?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/hrx.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/hrx.md b/docs/hrx.md index 140e8b0d8..d8f274534 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -516,6 +516,15 @@ the first sampled token, which is [engine#1536](https://github.com/1bit-MONSTER/ 5.4K-token prompt in a new process 4.46 -> 3.43 s, first short prompt 0.47 -> 0.22 s, with identical output. `GGML_HRX_JIT_CACHE=0` turns it off; it is off while a Loom sanitizer is set. Deleting the directory is always safe. +- **Routed-expert MoE on the HRX route (flint-40b-a2.5b).** `flint-UD-Q4_K_XL`, a 6-of-256 MoE that needs + 31.01 GiB at its own context, serves with `--device hrx --no-strict-device`: 64-token completions at + **42.61 tok/s** (2 runs, 35.24 and 49.97, spread 1.418x — a sample, not a rate). A step reads 2.213 GB at + this depth, so that is **~94.3 GB/s effective**, 37 % of this box's 256 GB/s peak and 43 % of the 220 GB/s + goal. The flag is required because `embd` is `GET_ROWS` f32 `ne=[2048,2,1,1]` and strict placement refuses + the model over those 2 of 3,576 nodes; see `dispatch-get-rows.cpp`, whose demand list is only ever given + `Q8_1`, so a tied f32 embedding never matches and lands on the CPU. Two rows would finish it: demand `F32` + for a non-Qwen f32 gather, or accept a named off-device op instead of refusing the model. Measured + 2026-11-14 under one gpu-hold window, `dmesg` clean before and after. - **Fused residual scale (llama.cpp #35).** ZAYA scales both residual branches twice per layer, `(x + bx) * sx + (r + br) * sr`, which ran as up to nine ADD / MUL dispatches per layer. Each side now runs as one dispatch (`common.res_scale_pair.*`, `ops/res_scale_pair_f32.loom`): 3,366 -> 3,165 From 539bf661a7d6d38d1c644a7b76972b21e8fdcd8f Mon Sep 17 00:00:00 2001 From: bcloud Date: Fri, 13 Nov 2026 21:56:13 -0400 Subject: [PATCH 2/3] docs/hrx: say that flint's "reads every weight stored" is about resident copies, not per-step reads (6 of 256 experts; 42.61 tok/s would be 1,067 GB/s otherwise) --- docs/hrx.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/hrx.md b/docs/hrx.md index d8f274534..0c1e834ee 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -1078,7 +1078,9 @@ The text is identical. Over six interleaved arms (trace off), prompt eval was 32 (or `GGML_HRX_PREFILL_FOLLOWS_DECODE=0`) restores the fused kernel for Q4_K, `GGML_HRX_Q6K_PACKED_ROWS=0` the stored Q6_K binding. Before either, making decode step aside instead was measured and does not help: the same duplicates, and decode 8.5 % slower. flint UD-Q4_K_XL (routed experts) reads every weight stored and holds one copy per -weight with or without the default. +weight with or without the default. (That sentence is about **resident copies**, not bytes read per decode step: a step reads +only the routed experts, `expert_used_count / expert_count` = 6 of 256 for this model. Had it read all 256, flint's measured +42.61 tok/s would imply 1,067 GB/s, above this box's 256 GB/s peak.) **Since pin `962507f8e` a short prompt holds no weight twice either** (llama.cpp #324, #325). At 6 to 8 tokens the few-token kernels took the batch and read the stored layout of weights that prefill reads in the packed rows, so a short prompt or a From 3563d726f30aabc22cc6493896d7ae13fc4c4af7 Mon Sep 17 00:00:00 2001 From: bcloud Date: Fri, 13 Nov 2026 22:09:42 -0400 Subject: [PATCH 3/3] =?UTF-8?q?docs/hrx:=20the=20flint=20row=20at=20its=20?= =?UTF-8?q?6-run=20median=20(50.66=20tok/s,=20spread=201.114x)=20=E2=80=94?= =?UTF-8?q?=20112=20GB/s,=2044%=20of=20peak?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/hrx.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/hrx.md b/docs/hrx.md index 0c1e834ee..55e33b81a 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -518,9 +518,10 @@ the first sampled token, which is [engine#1536](https://github.com/1bit-MONSTER/ Deleting the directory is always safe. - **Routed-expert MoE on the HRX route (flint-40b-a2.5b).** `flint-UD-Q4_K_XL`, a 6-of-256 MoE that needs 31.01 GiB at its own context, serves with `--device hrx --no-strict-device`: 64-token completions at - **42.61 tok/s** (2 runs, 35.24 and 49.97, spread 1.418x — a sample, not a rate). A step reads 2.213 GB at - this depth, so that is **~94.3 GB/s effective**, 37 % of this box's 256 GB/s peak and 43 % of the 220 GB/s - goal. The flag is required because `embd` is `GET_ROWS` f32 `ne=[2048,2,1,1]` and strict placement refuses + **50.66 tok/s**, median of **6 runs** (45.69–50.92, spread 1.114x). A step reads 2.213 GB at this depth, so + that is **~112 GB/s effective**, **44 % of this box's 256 GB/s peak and 51 % of the 220 GB/s goal**. (A 2-run + sample before this read 42.61 tok/s with a 1.418x spread; six runs move the median up 19 %, which is the + argument for not quoting two.) The flag is required because `embd` is `GET_ROWS` f32 `ne=[2048,2,1,1]` and strict placement refuses the model over those 2 of 3,576 nodes; see `dispatch-get-rows.cpp`, whose demand list is only ever given `Q8_1`, so a tied f32 embedding never matches and lands on the CPU. Two rows would finish it: demand `F32` for a non-Qwen f32 gather, or accept a named off-device op instead of refusing the model. Measured @@ -1080,7 +1081,7 @@ Before either, making decode step aside instead was measured and does not help: the same duplicates, and decode 8.5 % slower. flint UD-Q4_K_XL (routed experts) reads every weight stored and holds one copy per weight with or without the default. (That sentence is about **resident copies**, not bytes read per decode step: a step reads only the routed experts, `expert_used_count / expert_count` = 6 of 256 for this model. Had it read all 256, flint's measured -42.61 tok/s would imply 1,067 GB/s, above this box's 256 GB/s peak.) +50.66 tok/s would imply 1,269 GB/s, above this box's 256 GB/s peak.) **Since pin `962507f8e` a short prompt holds no weight twice either** (llama.cpp #324, #325). At 6 to 8 tokens the few-token kernels took the batch and read the stored layout of weights that prefill reads in the packed rows, so a short prompt or a