From 7adc17def7c80de1a7a90289c870b183ab6ae7f4 Mon Sep 17 00:00:00 2001 From: agent Date: Sat, 7 Nov 2026 17:59:14 -0400 Subject: [PATCH] docs: the Zyphra page named the wrong cause for the CPU-side input GET_ROWS (Refs #1316) The paragraph I added in #1345 said the token-embedding GET_ROWS was on the CPU "after the host-pointer weight import landed" and filed it under engine #1316. Both halves were measured wrong, and #233 landed a fix while this PR was open, so the page now says what is true at the pin: - it is **not** the #856 import: `GGML_HRX_WEIGHT_HOST_PLACEMENT=0` reproduces the placement identically, and #228 only adds a readback prefix for fused matches; - the cause is **#170's deliberate default** - the input layer stays on the CPU (`src/llama-model.cpp`, "very little benefit to offloading the input layer"); my clean "before" arm was simply a base from before #170; - `LLAMA_INPUT_LAYER_ON_DEV=1` **works** for a tied embedding since fork #233 (in the pin, 67d8c42ec): a tied embedding is gathered by GET_ROWS and contracted in a packed layout by the output MUL_MAT, a program cannot hold both, so the dispatch now declines the packed match for a weight another node gathers - measured 0 nodes off HRX 3/3 with the switch on against a refusal 3/3 with the gate disabled; - it stays off by default because device input measured ~6% slower than CPU input on a 0.6B, so at default settings `1 of 3024` nodes with `GET_ROWS x 1` and 2 splits is the floor for every model at `-ngl 99`, Zamba2 forms included, and the switch is how you leave it. Refs #1316 --- docs/zyphra-models.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/docs/zyphra-models.md b/docs/zyphra-models.md index edc74bce7..a5a912d5e 100644 --- a/docs/zyphra-models.md +++ b/docs/zyphra-models.md @@ -112,6 +112,16 @@ The adapter path adds two `MUL_MAT`s and an `ADD` per hybrid layer and **all of on both forms at prefill and decode. There is no throughput cost to report: pp512 441.98 → 440.57 and tg64 22.87 → 22.60 on one binary with both forms (0.997 / 0.988). -One caveat for anyone running `-ngl 99` today: the token-embedding `GET_ROWS` is currently **not** on HRX -(`1 of 3024` nodes, `GET_ROWS x 1`, 2 graph splits) after the host-pointer weight import landed — that is -base-level and affects **both** forms equally, tracked as engine **#1316**, not in this family's changes +One caveat for anyone running `-ngl 99` today: the token-embedding `GET_ROWS` sits on the CPU **by default**, and +that is #170's policy rather than this family's doing — the input layer stays on the CPU +(`src/llama-model.cpp`: "very little benefit to offloading the input layer"). `LLAMA_INPUT_LAYER_ON_DEV=1` is the +opt-in, and it works for a tied embedding since fork **#233** (in the pin): a tied embedding is gathered natively by +`GET_ROWS` and contracted in a packed layout by the output `MUL_MAT`, which a command program cannot hold twice, so +the dispatch now declines the packed match for a weight another node gathers — measured as **0 nodes off HRX** with +the switch on, 3/3 runs, against a refusal in 3/3 with that gate disabled. It stays **off by default** because device +input measured ~6 % slower than CPU input on a 0.6B while removing one node and one split per graph. + +So at default settings a census of `1 of 3024` nodes with `GET_ROWS x 1` and 2 graph splits is the floor for **every** +model at `-ngl 99`, Zamba2 forms included, and it is not residue these changes added; `LLAMA_INPUT_LAYER_ON_DEV=1` +is how you leave it. (Engine **#1316** is where both halves were measured — it is not the #856 host-pointer import, +which `GGML_HRX_WEIGHT_HOST_PLACEMENT=0` shows changes nothing here.)