Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 69 additions & 2 deletions docs/oga-drafter-path.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,76 @@ limitations under the License.
it asks the drafter lane for acceptance per pass against a bar. Two of the three arms are measured and published
(`docs/amd-drafters.md`: PARD over GGUF, **HRX 1.012×** at 3.095 tokens/round and 0.524 acceptance per offered draft; **CPU
0.508×**, which is structural — a round can only return `1 + accepted ≤ 1 + drafted` while a batched verify costs ~M
token-equivalents). This page is the third arm: **the OGA pack itself**, and why it cannot carry a drafter measurement yet.
token-equivalents). This page is the third arm: **the OGA pack itself**. On 2026-11-08 no pack could carry a drafter measurement, and the
gate table further down records why; on 2026-11-10 one pack serves and the row is measured, in the next section.

## The route has three gates, and a pack dies at a different one each time
## 2026-11-10: a pack serves, and the drafter row is measured

The table below is from 2026-11-08, when no OGA pack reached a logit. Since then the 1.9 reader and the b8 kernel
families landed, and `Qwen2.5-7B-Instruct_rai_1.9.0_npu` imports complete and serves. So the row #1371 was opened for
exists now: **a real drafter, on an OGA pack, against that pack's own costs**.

| | measured |
|---|---|
| target | `Qwen2.5-7B-Instruct_rai_1.9.0_npu`, imported with `--weight-layout b8` (incomplete 0; attention 3584/3584 tiles and MLP 9856/9856 groups byte for byte), served by `forward-serve` on the NPU |
| one plain decode step | **1021.2 ms** (median of 33; 1005.9-1065.5), 0.83-0.94 tok/s per request |
| one batched pass, 5 rows / 8 rows / 16 rows | **1049.1 / 1056.1 / 1073.4 ms** (medians of 4): a verify pass of up to 16 rows costs 1.03-1.05 steps |
| drafter | `qwen2.5-0.5b-instruct-q4_k_m.gguf` on the CPU (`--device none -ngl 0`), 5.8-6.4 ms per drafted token |
| acceptance, k = 4 | **2.963 tokens/round** (160 tokens in 54 rounds; 108 of 216 offered drafts accepted) |
| acceptance, k = 7 | **3.40-3.48 tokens/round** (160 tokens in 46-47 rounds over two replays; 116-117 of 322-329 offered) |

Per prompt at k = 4 the rate is 2.22 (prose), 3.33 (code), 3.33 (a list), 3.33 (a short answer); at k = 7 it is 2.50,
4.44, 4.00 and 3.33-3.64. No prompt accepted everything or nothing, so neither of the tool's two warnings applies.

What that acceptance is worth, from those measured parts:

| round shape | k | round cost | bar (tokens/round to break even) | measured tokens/round | projected |
|---|---:|---:|---:|---:|---:|
| one pass (verify only) | 4 | 1072 ms | **1.05** | 2.963 | 2.82x |
| one pass (verify only) | 7 | 1101 ms | **1.08** | 3.404 | 3.16x |
| two passes (`mtp::run`: verify, then the step that consumes the committed token) | 4 | 2093 ms | **2.05** | 2.963 | 1.45x |
| two passes | 7 | 2122 ms | **2.08** | 3.404 | 1.64x |

**The acceptance column is measured; the "projected" column is arithmetic and is not a measured loop.** No speculative
loop is wired into `forward-serve` (engine#1781 carries that), so nobody has run these rounds end to end on this pack.
What the measurement does settle is the question #1371 asks: on this route the bar is **1.05-1.08 tokens/round for a
one-pass round**, not the 1.17 measured on the 0.5B pack, because here a batched pass of up to 16 rows costs what one
step costs (the device wait is 854-872 ms of either). A 0.5B drafter of the same family clears that bar by 2.8-3.2x on
these prompts, and still clears the two-pass bar.

Three limits of the row, so it is not read as more than it is:

- four prompts of 40 tokens each. It is a first row, not a distribution; code and list text draft better than prose,
as on the other arms.
- acceptance is counted on the draft tokenizer's ids over the target's recorded **text** (the pair shares the Qwen2.5
vocabulary, and the tool refuses a recording whose prompt does not tokenize to a prefix of prompt + continuation).
The drafter is not bit-reproducible between replays: one prompt's k = 7 accept list moved between two runs, which is
the 3.40-3.48 range above.
- the pass costs were taken at the start of a context (rows 0-15). The host share of a pass is 34-53 ms of ~1050, so
a longer context moves it little, but that is an expectation and was not measured.

One thing a verify loop on this pack must respect: **a pass of 17 rows costs 6992 ms, 6.5 times a 16-row pass**, because
a chunk wider than 16 rows runs the 128-row kernel family. A verify batch must stay at 16 rows or fewer here.

How it was taken (`amdxdna` error count unchanged across every ticket):

```sh
1bit npu-pack import <pack> -o <out> --weight-layout b8 --kernels <the pack family's kernel set>
# ticket 1 (npu-hold): the target's greedy continuations, through the endpoint forward-serve has
tools/spec_decode_acceptance.py --record-openai http://127.0.0.1:18172 --prompts prompts.json --tokens 40 --out target.json
# no NPU: the drafter proposes from every committed prefix of that recording
tools/spec_decode_acceptance.py --target-recorded target.json --draft http://127.0.0.1:18173 --k 4 \
--pass-ms 1049.1 --step-ms 1021.2
```

The replay rests on greedy decode being deterministic, and that was checked rather than assumed: the four
continuations were recorded twice, on two separate serves in two tickets, and are identical text for text (4 of 4).

`--pass-ms` and `--step-ms` are the serve's own `[bs] M=<rows>` and `[ph] tok` walls under `NPU_PHASE_PROF=1`; a prompt
of n+1 token ids produces one n-row pass. `tools/oga_pack_probe.sh` still imports without `--weight-layout b8` and
without a kernel set, so it reports this pack as stopping at gate 2; the commands above are the ones that serve it.

## The route has three gates, and a pack dies at a different one each time (2026-11-08)

`tools/oga_pack_probe.sh <pack dir> [port]` runs gates 1-2 inside one `npu-hold` ticket (the server it starts is killed by a trap on
every exit path) and prints which gate bit. Measured on this box, 2026-11-08, kernel `7.4.0-rc1-1bit`:
Expand Down
15 changes: 14 additions & 1 deletion tests/spec_decode_acceptance_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,21 @@ def check(cond, what):
check(r.returncode == 0, f"the tool's own selftest passes (rc={r.returncode})")
for want in ("a shared prefix of 2 is accepted", "a first-token miss accepts nothing",
"a draft longer than the target's continuation is capped by the target",
"tokens/round is emitted/rounds", "accepted drafts/round is a different number"):
"tokens/round is emitted/rounds", "accepted drafts/round is a different number",
"replay: a perfect drafter at k=4", "replay: a drafter that is always wrong",
"economics: bars are round cost / step cost",
"economics: a drafter that accepts nothing is a slow-down"):
check(want in r.stdout, f"selftest covers: {want}")
check("FAIL" not in r.stdout, "no selftest arm fails")

# The recorded-target mode (engine#1371) refuses half-specified invocations in the tool, not in the caller's head.
rr = subprocess.run([sys.executable, TOOL, "--record-openai", "http://127.0.0.1:1"], capture_output=True, text=True)
check(rr.returncode == 2 and "--prompts and --out" in rr.stderr, f"record mode without --prompts/--out exits 2 (rc={rr.returncode})")
rr = subprocess.run([sys.executable, TOOL, "--target-recorded", "x.json"], capture_output=True, text=True)
check(rr.returncode == 2 and "needs --draft" in rr.stderr, f"replay mode without --draft exits 2 (rc={rr.returncode})")
rr = subprocess.run([sys.executable, TOOL, "--target-recorded", "x.json", "--draft", "http://127.0.0.1:1", "--pass-ms", "5"],
capture_output=True, text=True)
check(rr.returncode == 2 and "go together" in rr.stderr, f"--pass-ms without --step-ms exits 2 (rc={rr.returncode})")

# No server: the tool must fail, not report 0 acceptance as if it were a result.
p = os.path.join(ROOT, "tests", ".spec_decode_prompt.txt")
Expand Down
163 changes: 163 additions & 0 deletions tools/spec_decode_acceptance.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,23 @@
tools/spec_decode_acceptance.py --target http://127.0.0.1:18120 --draft http://127.0.0.1:18121 \\
--prompt-file p.txt --tokens 32 --k 4
tools/spec_decode_acceptance.py --selftest # the accept-prefix rule and the units, no server needed

A target that is not a llama-server (engine#1371: an OGA pack served by `1bit forward-serve` on the NPU has no
`/completion`, and its ticket should not be held while a drafter runs) is measured in two steps. Greedy decode is
deterministic, so the target's continuation is recorded once and the loop is replayed against it:

# inside the device ticket: the target's own greedy text for each prompt, through the OpenAI-shaped endpoint
tools/spec_decode_acceptance.py --record-openai http://127.0.0.1:18172 --prompts prompts.json --tokens 40 \
--out target.json
# afterwards, no device: the draft proposes k tokens from every committed prefix of that recording
tools/spec_decode_acceptance.py --target-recorded target.json --draft http://127.0.0.1:18173 --k 4 \
--pass-ms 1037.5 --step-ms 1020.8

`prompts.json` is `{"name": "prompt text", ...}`. Token ids come from the DRAFT server's tokenizer, so the pair must
share a vocabulary (the tool refuses a recording whose prompt does not tokenize to a prefix of prompt + continuation).
With `--pass-ms` (one batched verify pass of k+1 rows) and `--step-ms` (one plain decode step), both measured on the
target, the row also carries the break-even bar and the projected speed-up for a one-pass round and for a two-pass
round. Those two lines are ARITHMETIC over measured parts, and are labelled so: they are not a measured loop.
"""
import argparse
import json
Expand Down Expand Up @@ -73,6 +90,35 @@ def accept_prefix(draft_ids, truth_ids):
return n


def replay_rounds(truth_ids, draft_fn, k):
"""The verify loop against a FIXED greedy continuation `truth_ids` (the target's own, recorded once).

Each round asks `draft_fn(i, k)` for up to k draft ids continuing the committed prefix of length i, accepts the
longest common prefix with truth_ids[i:], and commits accepted + 1 tokens: the accepted drafts plus the target's
own token at the first mismatch, which the same verify pass produced. Returns (per-round accepts, offered).
Pure: `draft_fn` is the only thing that touches a server."""
i, per_round, offered = 0, [], 0
while i < len(truth_ids):
d = list(draft_fn(i, k))[:k]
offered += len(d)
a = accept_prefix(d, truth_ids[i:])
per_round.append(a)
i += a + 1
return per_round, offered


def loop_economics(tokens_per_round, k, pass_ms, step_ms, draft_ms_per_token):
"""Break-even and projected speed-up from measured parts. A one-pass round is one batched verify pass plus k
draft tokens; a two-pass round (mtp::run's shape) adds the plain step that consumes the committed token. The bar
is the tokens/round at which a round costs what the same tokens would cost as plain steps."""
out = {}
for name, cost in (("one_pass", pass_ms + k * draft_ms_per_token),
("two_pass", pass_ms + step_ms + k * draft_ms_per_token)):
out[name] = {"round_ms": round(cost, 1), "bar_tokens_per_round": round(cost / step_ms, 3),
"projected_speedup": round(tokens_per_round * step_ms / cost, 3)}
return out


def selftest():
"""The parts that can be wrong without a model: the prefix rule and the unit arithmetic."""
fails = 0
Expand All @@ -93,9 +139,103 @@ def check(cond, what):
tpr = emitted / rounds
check(abs(tpr - 2.667) < 0.001, f"tokens/round is emitted/rounds ({tpr:.3f})")
check(abs(accepted / rounds - 2.111) < 0.001, "accepted drafts/round is a different number, printed beside it")
# the replay against a recorded continuation (engine#1371): a + 1 tokens per round, the target's own at the miss
truth = [10, 11, 12, 13, 14, 15, 16]
pr, off = replay_rounds(truth, lambda i, k: truth[i:i + k], 4)
check(pr == [4, 2] and off == 6, f"replay: a perfect drafter at k=4 takes 7 tokens in 2 rounds ({pr}, offered {off})")
pr, off = replay_rounds(truth, lambda i, k: [0] * k, 4)
check(pr == [0] * 7 and off == 28, "replay: a drafter that is always wrong commits exactly 1 token per round")
pr, _ = replay_rounds(truth, lambda i, k: [truth[i], 99, 99, 99][:k] if i < len(truth) else [], 4)
# the last round has no "+1" when its accepted drafts reach the end of the recording, so the identity is
# accepted + rounds == tokens or tokens + 1 -- which is why tokens/round divides the TOKENS, not that sum
check(pr == [1, 1, 1, 1] and sum(pr) + len(pr) in (len(truth), len(truth) + 1),
f"replay: accepted + rounds is the tokens committed, give or take the final round's own token ({pr})")
pr, _ = replay_rounds(truth[:6], lambda i, k: [truth[i], 99, 99, 99][:k], 4)
check(pr == [1, 1, 1] and sum(pr) + len(pr) == 6, f"replay: and exactly the tokens when the recording ends on a miss ({pr})")
pr, off = replay_rounds(truth, lambda i, k: truth[i:i + 50], 3)
check(max(pr) <= 3 and off <= 3 * len(pr), "replay: a draft longer than k is cut to k before it is counted")
# the economics: a verify pass that costs one step puts the one-pass bar just above 1 and the two-pass bar above 2
e = loop_economics(3.0, 4, 1000.0, 1000.0, 5.0)
check(e["one_pass"]["bar_tokens_per_round"] == 1.02 and e["two_pass"]["bar_tokens_per_round"] == 2.02,
f"economics: bars are round cost / step cost ({e['one_pass']['bar_tokens_per_round']}, "
f"{e['two_pass']['bar_tokens_per_round']})")
check(e["one_pass"]["projected_speedup"] == 2.941 and e["two_pass"]["projected_speedup"] == 1.485,
"economics: projected speed-up is tokens/round over the bar")
e = loop_economics(1.0, 4, 1200.0, 300.0, 50.0)
check(e["one_pass"]["projected_speedup"] < 1.0, "economics: a drafter that accepts nothing is a slow-down, not 1.0x")
return fails


def _post_openai(url, prompt, n):
r = _post(url, "/v1/completions", {"prompt": prompt, "max_tokens": n, "temperature": 0})
return r["choices"][0].get("text", ""), r.get("usage", {})


def record_openai(url, prompts_path, n, out_path):
"""Step 1 of the recorded mode: the target's greedy continuation of each prompt, as text."""
prompts = json.load(open(prompts_path))
if not isinstance(prompts, dict) or not prompts:
print("spec_decode_acceptance: --prompts must be a non-empty JSON object {name: prompt}", file=sys.stderr)
return EXIT_BAD_INPUT
rec = {}
for name, prompt in prompts.items():
text, usage = _post_openai(url, prompt, n)
rec[name] = {"prompt": prompt, "text": text, "usage": usage}
print(f"recorded {name}: {usage.get('completion_tokens', '?')} tokens", file=sys.stderr)
with open(out_path, "w") as fh:
json.dump(rec, fh, indent=1)
return EXIT_OK


def recorded(a):
"""Step 2: replay the loop against the recording, the draft server proposing from every committed prefix."""
rec = json.load(open(a.target_recorded))
rows, tot_tokens, tot_rounds, tot_acc, tot_off, d_ms, d_n = {}, 0, 0, 0, 0, 0.0, 0

def tok(text):
return _post(a.draft, "/tokenize", {"content": text, "add_special": False}).get("tokens") or []

for name, r in rec.items():
p_ids = tok(r["prompt"])
t_ids = tok(r["prompt"] + r["text"])
if not p_ids or t_ids[:len(p_ids)] != p_ids:
print(f"spec_decode_acceptance: {name}: the prompt does not tokenize to a prefix of prompt + continuation "
"under the draft's tokenizer, so token positions cannot be compared; refusing this recording",
file=sys.stderr)
return EXIT_BAD_INPUT
truth = t_ids[len(p_ids):]
timing = {"ms": 0.0, "n": 0}

def draft_fn(i, k, _ids=t_ids, _base=len(p_ids), _t=timing):
rr = _post(a.draft, "/completion", {"prompt": _ids[:_base + i], "n_predict": k, "temperature": 0.0,
"top_k": 1, "top_p": 1.0, "repeat_penalty": 1.0, "seed": 0,
"cache_prompt": True, "stream": False, "return_tokens": True,
"stop": []})
tm = rr.get("timings") or {}
_t["ms"] += tm.get("predicted_ms", 0.0)
_t["n"] += tm.get("predicted_n", 0)
return rr.get("tokens") or []

per_round, offered = replay_rounds(truth, draft_fn, a.k)
rows[name] = {"tokens": len(truth), "rounds": len(per_round),
"tokens_per_round": round(len(truth) / len(per_round), 3) if per_round else 0.0,
"accepted_tokens_per_round": round(sum(per_round) / len(per_round), 3) if per_round else 0.0,
"per_round_accept_tokens": per_round}
tot_tokens += len(truth); tot_rounds += len(per_round); tot_acc += sum(per_round); tot_off += offered
d_ms += timing["ms"]; d_n += timing["n"]
out = {"mode": "recorded target, replayed", "k": a.k, "prompts": rows, "tokens": tot_tokens, "rounds": tot_rounds,
"accepted_tokens": tot_acc, "offered_tokens": tot_off,
"tokens_per_round": round(tot_tokens / tot_rounds, 3) if tot_rounds else 0.0,
"accepted_per_offered": round(tot_acc / tot_off, 3) if tot_off else 0.0,
"draft_ms_per_token": round(d_ms / d_n, 2) if d_n else None,
"note": "token ids are the draft tokenizer's over the recorded TEXT; a round commits accepted + 1 tokens."}
if a.pass_ms and a.step_ms and tot_rounds:
out["economics_from_measured_parts_not_a_measured_loop"] = loop_economics(
tot_tokens / tot_rounds, a.k, a.pass_ms, a.step_ms, out["draft_ms_per_token"] or 0.0)
print(json.dumps(out, indent=2))
return EXIT_OK


def main(argv):
ap = argparse.ArgumentParser()
ap.add_argument("--target", help="llama-server base URL of the TARGET model")
Expand All @@ -105,9 +245,32 @@ def main(argv):
ap.add_argument("--k", type=int, default=4)
ap.add_argument("--rounds", type=int, default=0, help="0 = until --tokens emitted")
ap.add_argument("--selftest", action="store_true", help="check the prefix rule and the units; needs no server")
ap.add_argument("--record-openai", help="record mode: base URL of a target that serves /v1/completions")
ap.add_argument("--prompts", help="record mode: JSON object {name: prompt}")
ap.add_argument("--out", help="record mode: where the recording is written")
ap.add_argument("--target-recorded", help="replay mode: a recording written by --record-openai")
ap.add_argument("--pass-ms", type=float, default=0.0, help="replay mode: one batched verify pass of k+1 rows, ms")
ap.add_argument("--step-ms", type=float, default=0.0, help="replay mode: one plain decode step on the target, ms")
a = ap.parse_args(argv)
if a.selftest:
return EXIT_BAD_INPUT if selftest() else EXIT_OK
if a.record_openai or a.target_recorded:
if a.record_openai and not (a.prompts and a.out):
print("spec_decode_acceptance: --record-openai needs --prompts and --out", file=sys.stderr)
return EXIT_BAD_INPUT
if a.target_recorded and not a.draft:
print("spec_decode_acceptance: --target-recorded needs --draft", file=sys.stderr)
return EXIT_BAD_INPUT
if a.target_recorded and bool(a.pass_ms) != bool(a.step_ms):
print("spec_decode_acceptance: --pass-ms and --step-ms go together (the bar is their ratio)", file=sys.stderr)
return EXIT_BAD_INPUT
try:
if a.record_openai:
return record_openai(a.record_openai, a.prompts, a.tokens, a.out)
return recorded(a)
except (urllib.error.URLError, OSError) as e:
print(f"spec_decode_acceptance: a server did not answer: {e}", file=sys.stderr)
return EXIT_SERVER
if not (a.target and a.draft and a.prompt_file):
print(__doc__)
return EXIT_BAD_INPUT
Expand Down
Loading