diff --git a/docs/oga-drafter-path.md b/docs/oga-drafter-path.md index 0a27024ab..888aaf54f 100644 --- a/docs/oga-drafter-path.md +++ b/docs/oga-drafter-path.md @@ -20,9 +20,76 @@ limitations under the License. it asks the drafter lane for acceptance per pass against a bar. Two of the three arms are measured and published (`docs/amd-drafters.md`: PARD over GGUF, **HRX 1.012×** at 3.095 tokens/round and 0.524 acceptance per offered draft; **CPU 0.508×**, which is structural — a round can only return `1 + accepted ≤ 1 + drafted` while a batched verify costs ~M -token-equivalents). This page is the third arm: **the OGA pack itself**, and why it cannot carry a drafter measurement yet. +token-equivalents). This page is the third arm: **the OGA pack itself**. On 2026-11-08 no pack could carry a drafter measurement, and the +gate table further down records why; on 2026-11-10 one pack serves and the row is measured, in the next section. -## The route has three gates, and a pack dies at a different one each time +## 2026-11-10: a pack serves, and the drafter row is measured + +The table below is from 2026-11-08, when no OGA pack reached a logit. Since then the 1.9 reader and the b8 kernel +families landed, and `Qwen2.5-7B-Instruct_rai_1.9.0_npu` imports complete and serves. So the row #1371 was opened for +exists now: **a real drafter, on an OGA pack, against that pack's own costs**. + +| | measured | +|---|---| +| target | `Qwen2.5-7B-Instruct_rai_1.9.0_npu`, imported with `--weight-layout b8` (incomplete 0; attention 3584/3584 tiles and MLP 9856/9856 groups byte for byte), served by `forward-serve` on the NPU | +| one plain decode step | **1021.2 ms** (median of 33; 1005.9-1065.5), 0.83-0.94 tok/s per request | +| one batched pass, 5 rows / 8 rows / 16 rows | **1049.1 / 1056.1 / 1073.4 ms** (medians of 4): a verify pass of up to 16 rows costs 1.03-1.05 steps | +| drafter | `qwen2.5-0.5b-instruct-q4_k_m.gguf` on the CPU (`--device none -ngl 0`), 5.8-6.4 ms per drafted token | +| acceptance, k = 4 | **2.963 tokens/round** (160 tokens in 54 rounds; 108 of 216 offered drafts accepted) | +| acceptance, k = 7 | **3.40-3.48 tokens/round** (160 tokens in 46-47 rounds over two replays; 116-117 of 322-329 offered) | + +Per prompt at k = 4 the rate is 2.22 (prose), 3.33 (code), 3.33 (a list), 3.33 (a short answer); at k = 7 it is 2.50, +4.44, 4.00 and 3.33-3.64. No prompt accepted everything or nothing, so neither of the tool's two warnings applies. + +What that acceptance is worth, from those measured parts: + +| round shape | k | round cost | bar (tokens/round to break even) | measured tokens/round | projected | +|---|---:|---:|---:|---:|---:| +| one pass (verify only) | 4 | 1072 ms | **1.05** | 2.963 | 2.82x | +| one pass (verify only) | 7 | 1101 ms | **1.08** | 3.404 | 3.16x | +| two passes (`mtp::run`: verify, then the step that consumes the committed token) | 4 | 2093 ms | **2.05** | 2.963 | 1.45x | +| two passes | 7 | 2122 ms | **2.08** | 3.404 | 1.64x | + +**The acceptance column is measured; the "projected" column is arithmetic and is not a measured loop.** No speculative +loop is wired into `forward-serve` (engine#1781 carries that), so nobody has run these rounds end to end on this pack. +What the measurement does settle is the question #1371 asks: on this route the bar is **1.05-1.08 tokens/round for a +one-pass round**, not the 1.17 measured on the 0.5B pack, because here a batched pass of up to 16 rows costs what one +step costs (the device wait is 854-872 ms of either). A 0.5B drafter of the same family clears that bar by 2.8-3.2x on +these prompts, and still clears the two-pass bar. + +Three limits of the row, so it is not read as more than it is: + +- four prompts of 40 tokens each. It is a first row, not a distribution; code and list text draft better than prose, + as on the other arms. +- acceptance is counted on the draft tokenizer's ids over the target's recorded **text** (the pair shares the Qwen2.5 + vocabulary, and the tool refuses a recording whose prompt does not tokenize to a prefix of prompt + continuation). + The drafter is not bit-reproducible between replays: one prompt's k = 7 accept list moved between two runs, which is + the 3.40-3.48 range above. +- the pass costs were taken at the start of a context (rows 0-15). The host share of a pass is 34-53 ms of ~1050, so + a longer context moves it little, but that is an expectation and was not measured. + +One thing a verify loop on this pack must respect: **a pass of 17 rows costs 6992 ms, 6.5 times a 16-row pass**, because +a chunk wider than 16 rows runs the 128-row kernel family. A verify batch must stay at 16 rows or fewer here. + +How it was taken (`amdxdna` error count unchanged across every ticket): + +```sh +1bit npu-pack import -o --weight-layout b8 --kernels +# ticket 1 (npu-hold): the target's greedy continuations, through the endpoint forward-serve has +tools/spec_decode_acceptance.py --record-openai http://127.0.0.1:18172 --prompts prompts.json --tokens 40 --out target.json +# no NPU: the drafter proposes from every committed prefix of that recording +tools/spec_decode_acceptance.py --target-recorded target.json --draft http://127.0.0.1:18173 --k 4 \ + --pass-ms 1049.1 --step-ms 1021.2 +``` + +The replay rests on greedy decode being deterministic, and that was checked rather than assumed: the four +continuations were recorded twice, on two separate serves in two tickets, and are identical text for text (4 of 4). + +`--pass-ms` and `--step-ms` are the serve's own `[bs] M=` and `[ph] tok` walls under `NPU_PHASE_PROF=1`; a prompt +of n+1 token ids produces one n-row pass. `tools/oga_pack_probe.sh` still imports without `--weight-layout b8` and +without a kernel set, so it reports this pack as stopping at gate 2; the commands above are the ones that serve it. + +## The route has three gates, and a pack dies at a different one each time (2026-11-08) `tools/oga_pack_probe.sh [port]` runs gates 1-2 inside one `npu-hold` ticket (the server it starts is killed by a trap on every exit path) and prints which gate bit. Measured on this box, 2026-11-08, kernel `7.4.0-rc1-1bit`: diff --git a/tests/spec_decode_acceptance_test.py b/tests/spec_decode_acceptance_test.py index 566cef86f..d77151aad 100644 --- a/tests/spec_decode_acceptance_test.py +++ b/tests/spec_decode_acceptance_test.py @@ -41,8 +41,21 @@ def check(cond, what): check(r.returncode == 0, f"the tool's own selftest passes (rc={r.returncode})") for want in ("a shared prefix of 2 is accepted", "a first-token miss accepts nothing", "a draft longer than the target's continuation is capped by the target", - "tokens/round is emitted/rounds", "accepted drafts/round is a different number"): + "tokens/round is emitted/rounds", "accepted drafts/round is a different number", + "replay: a perfect drafter at k=4", "replay: a drafter that is always wrong", + "economics: bars are round cost / step cost", + "economics: a drafter that accepts nothing is a slow-down"): check(want in r.stdout, f"selftest covers: {want}") +check("FAIL" not in r.stdout, "no selftest arm fails") + +# The recorded-target mode (engine#1371) refuses half-specified invocations in the tool, not in the caller's head. +rr = subprocess.run([sys.executable, TOOL, "--record-openai", "http://127.0.0.1:1"], capture_output=True, text=True) +check(rr.returncode == 2 and "--prompts and --out" in rr.stderr, f"record mode without --prompts/--out exits 2 (rc={rr.returncode})") +rr = subprocess.run([sys.executable, TOOL, "--target-recorded", "x.json"], capture_output=True, text=True) +check(rr.returncode == 2 and "needs --draft" in rr.stderr, f"replay mode without --draft exits 2 (rc={rr.returncode})") +rr = subprocess.run([sys.executable, TOOL, "--target-recorded", "x.json", "--draft", "http://127.0.0.1:1", "--pass-ms", "5"], + capture_output=True, text=True) +check(rr.returncode == 2 and "go together" in rr.stderr, f"--pass-ms without --step-ms exits 2 (rc={rr.returncode})") # No server: the tool must fail, not report 0 acceptance as if it were a result. p = os.path.join(ROOT, "tests", ".spec_decode_prompt.txt") diff --git a/tools/spec_decode_acceptance.py b/tools/spec_decode_acceptance.py index ecbba5710..e9fc3cee5 100755 --- a/tools/spec_decode_acceptance.py +++ b/tools/spec_decode_acceptance.py @@ -35,6 +35,23 @@ tools/spec_decode_acceptance.py --target http://127.0.0.1:18120 --draft http://127.0.0.1:18121 \\ --prompt-file p.txt --tokens 32 --k 4 tools/spec_decode_acceptance.py --selftest # the accept-prefix rule and the units, no server needed + +A target that is not a llama-server (engine#1371: an OGA pack served by `1bit forward-serve` on the NPU has no +`/completion`, and its ticket should not be held while a drafter runs) is measured in two steps. Greedy decode is +deterministic, so the target's continuation is recorded once and the loop is replayed against it: + + # inside the device ticket: the target's own greedy text for each prompt, through the OpenAI-shaped endpoint + tools/spec_decode_acceptance.py --record-openai http://127.0.0.1:18172 --prompts prompts.json --tokens 40 \ + --out target.json + # afterwards, no device: the draft proposes k tokens from every committed prefix of that recording + tools/spec_decode_acceptance.py --target-recorded target.json --draft http://127.0.0.1:18173 --k 4 \ + --pass-ms 1037.5 --step-ms 1020.8 + +`prompts.json` is `{"name": "prompt text", ...}`. Token ids come from the DRAFT server's tokenizer, so the pair must +share a vocabulary (the tool refuses a recording whose prompt does not tokenize to a prefix of prompt + continuation). +With `--pass-ms` (one batched verify pass of k+1 rows) and `--step-ms` (one plain decode step), both measured on the +target, the row also carries the break-even bar and the projected speed-up for a one-pass round and for a two-pass +round. Those two lines are ARITHMETIC over measured parts, and are labelled so: they are not a measured loop. """ import argparse import json @@ -73,6 +90,35 @@ def accept_prefix(draft_ids, truth_ids): return n +def replay_rounds(truth_ids, draft_fn, k): + """The verify loop against a FIXED greedy continuation `truth_ids` (the target's own, recorded once). + + Each round asks `draft_fn(i, k)` for up to k draft ids continuing the committed prefix of length i, accepts the + longest common prefix with truth_ids[i:], and commits accepted + 1 tokens: the accepted drafts plus the target's + own token at the first mismatch, which the same verify pass produced. Returns (per-round accepts, offered). + Pure: `draft_fn` is the only thing that touches a server.""" + i, per_round, offered = 0, [], 0 + while i < len(truth_ids): + d = list(draft_fn(i, k))[:k] + offered += len(d) + a = accept_prefix(d, truth_ids[i:]) + per_round.append(a) + i += a + 1 + return per_round, offered + + +def loop_economics(tokens_per_round, k, pass_ms, step_ms, draft_ms_per_token): + """Break-even and projected speed-up from measured parts. A one-pass round is one batched verify pass plus k + draft tokens; a two-pass round (mtp::run's shape) adds the plain step that consumes the committed token. The bar + is the tokens/round at which a round costs what the same tokens would cost as plain steps.""" + out = {} + for name, cost in (("one_pass", pass_ms + k * draft_ms_per_token), + ("two_pass", pass_ms + step_ms + k * draft_ms_per_token)): + out[name] = {"round_ms": round(cost, 1), "bar_tokens_per_round": round(cost / step_ms, 3), + "projected_speedup": round(tokens_per_round * step_ms / cost, 3)} + return out + + def selftest(): """The parts that can be wrong without a model: the prefix rule and the unit arithmetic.""" fails = 0 @@ -93,9 +139,103 @@ def check(cond, what): tpr = emitted / rounds check(abs(tpr - 2.667) < 0.001, f"tokens/round is emitted/rounds ({tpr:.3f})") check(abs(accepted / rounds - 2.111) < 0.001, "accepted drafts/round is a different number, printed beside it") + # the replay against a recorded continuation (engine#1371): a + 1 tokens per round, the target's own at the miss + truth = [10, 11, 12, 13, 14, 15, 16] + pr, off = replay_rounds(truth, lambda i, k: truth[i:i + k], 4) + check(pr == [4, 2] and off == 6, f"replay: a perfect drafter at k=4 takes 7 tokens in 2 rounds ({pr}, offered {off})") + pr, off = replay_rounds(truth, lambda i, k: [0] * k, 4) + check(pr == [0] * 7 and off == 28, "replay: a drafter that is always wrong commits exactly 1 token per round") + pr, _ = replay_rounds(truth, lambda i, k: [truth[i], 99, 99, 99][:k] if i < len(truth) else [], 4) + # the last round has no "+1" when its accepted drafts reach the end of the recording, so the identity is + # accepted + rounds == tokens or tokens + 1 -- which is why tokens/round divides the TOKENS, not that sum + check(pr == [1, 1, 1, 1] and sum(pr) + len(pr) in (len(truth), len(truth) + 1), + f"replay: accepted + rounds is the tokens committed, give or take the final round's own token ({pr})") + pr, _ = replay_rounds(truth[:6], lambda i, k: [truth[i], 99, 99, 99][:k], 4) + check(pr == [1, 1, 1] and sum(pr) + len(pr) == 6, f"replay: and exactly the tokens when the recording ends on a miss ({pr})") + pr, off = replay_rounds(truth, lambda i, k: truth[i:i + 50], 3) + check(max(pr) <= 3 and off <= 3 * len(pr), "replay: a draft longer than k is cut to k before it is counted") + # the economics: a verify pass that costs one step puts the one-pass bar just above 1 and the two-pass bar above 2 + e = loop_economics(3.0, 4, 1000.0, 1000.0, 5.0) + check(e["one_pass"]["bar_tokens_per_round"] == 1.02 and e["two_pass"]["bar_tokens_per_round"] == 2.02, + f"economics: bars are round cost / step cost ({e['one_pass']['bar_tokens_per_round']}, " + f"{e['two_pass']['bar_tokens_per_round']})") + check(e["one_pass"]["projected_speedup"] == 2.941 and e["two_pass"]["projected_speedup"] == 1.485, + "economics: projected speed-up is tokens/round over the bar") + e = loop_economics(1.0, 4, 1200.0, 300.0, 50.0) + check(e["one_pass"]["projected_speedup"] < 1.0, "economics: a drafter that accepts nothing is a slow-down, not 1.0x") return fails +def _post_openai(url, prompt, n): + r = _post(url, "/v1/completions", {"prompt": prompt, "max_tokens": n, "temperature": 0}) + return r["choices"][0].get("text", ""), r.get("usage", {}) + + +def record_openai(url, prompts_path, n, out_path): + """Step 1 of the recorded mode: the target's greedy continuation of each prompt, as text.""" + prompts = json.load(open(prompts_path)) + if not isinstance(prompts, dict) or not prompts: + print("spec_decode_acceptance: --prompts must be a non-empty JSON object {name: prompt}", file=sys.stderr) + return EXIT_BAD_INPUT + rec = {} + for name, prompt in prompts.items(): + text, usage = _post_openai(url, prompt, n) + rec[name] = {"prompt": prompt, "text": text, "usage": usage} + print(f"recorded {name}: {usage.get('completion_tokens', '?')} tokens", file=sys.stderr) + with open(out_path, "w") as fh: + json.dump(rec, fh, indent=1) + return EXIT_OK + + +def recorded(a): + """Step 2: replay the loop against the recording, the draft server proposing from every committed prefix.""" + rec = json.load(open(a.target_recorded)) + rows, tot_tokens, tot_rounds, tot_acc, tot_off, d_ms, d_n = {}, 0, 0, 0, 0, 0.0, 0 + + def tok(text): + return _post(a.draft, "/tokenize", {"content": text, "add_special": False}).get("tokens") or [] + + for name, r in rec.items(): + p_ids = tok(r["prompt"]) + t_ids = tok(r["prompt"] + r["text"]) + if not p_ids or t_ids[:len(p_ids)] != p_ids: + print(f"spec_decode_acceptance: {name}: the prompt does not tokenize to a prefix of prompt + continuation " + "under the draft's tokenizer, so token positions cannot be compared; refusing this recording", + file=sys.stderr) + return EXIT_BAD_INPUT + truth = t_ids[len(p_ids):] + timing = {"ms": 0.0, "n": 0} + + def draft_fn(i, k, _ids=t_ids, _base=len(p_ids), _t=timing): + rr = _post(a.draft, "/completion", {"prompt": _ids[:_base + i], "n_predict": k, "temperature": 0.0, + "top_k": 1, "top_p": 1.0, "repeat_penalty": 1.0, "seed": 0, + "cache_prompt": True, "stream": False, "return_tokens": True, + "stop": []}) + tm = rr.get("timings") or {} + _t["ms"] += tm.get("predicted_ms", 0.0) + _t["n"] += tm.get("predicted_n", 0) + return rr.get("tokens") or [] + + per_round, offered = replay_rounds(truth, draft_fn, a.k) + rows[name] = {"tokens": len(truth), "rounds": len(per_round), + "tokens_per_round": round(len(truth) / len(per_round), 3) if per_round else 0.0, + "accepted_tokens_per_round": round(sum(per_round) / len(per_round), 3) if per_round else 0.0, + "per_round_accept_tokens": per_round} + tot_tokens += len(truth); tot_rounds += len(per_round); tot_acc += sum(per_round); tot_off += offered + d_ms += timing["ms"]; d_n += timing["n"] + out = {"mode": "recorded target, replayed", "k": a.k, "prompts": rows, "tokens": tot_tokens, "rounds": tot_rounds, + "accepted_tokens": tot_acc, "offered_tokens": tot_off, + "tokens_per_round": round(tot_tokens / tot_rounds, 3) if tot_rounds else 0.0, + "accepted_per_offered": round(tot_acc / tot_off, 3) if tot_off else 0.0, + "draft_ms_per_token": round(d_ms / d_n, 2) if d_n else None, + "note": "token ids are the draft tokenizer's over the recorded TEXT; a round commits accepted + 1 tokens."} + if a.pass_ms and a.step_ms and tot_rounds: + out["economics_from_measured_parts_not_a_measured_loop"] = loop_economics( + tot_tokens / tot_rounds, a.k, a.pass_ms, a.step_ms, out["draft_ms_per_token"] or 0.0) + print(json.dumps(out, indent=2)) + return EXIT_OK + + def main(argv): ap = argparse.ArgumentParser() ap.add_argument("--target", help="llama-server base URL of the TARGET model") @@ -105,9 +245,32 @@ def main(argv): ap.add_argument("--k", type=int, default=4) ap.add_argument("--rounds", type=int, default=0, help="0 = until --tokens emitted") ap.add_argument("--selftest", action="store_true", help="check the prefix rule and the units; needs no server") + ap.add_argument("--record-openai", help="record mode: base URL of a target that serves /v1/completions") + ap.add_argument("--prompts", help="record mode: JSON object {name: prompt}") + ap.add_argument("--out", help="record mode: where the recording is written") + ap.add_argument("--target-recorded", help="replay mode: a recording written by --record-openai") + ap.add_argument("--pass-ms", type=float, default=0.0, help="replay mode: one batched verify pass of k+1 rows, ms") + ap.add_argument("--step-ms", type=float, default=0.0, help="replay mode: one plain decode step on the target, ms") a = ap.parse_args(argv) if a.selftest: return EXIT_BAD_INPUT if selftest() else EXIT_OK + if a.record_openai or a.target_recorded: + if a.record_openai and not (a.prompts and a.out): + print("spec_decode_acceptance: --record-openai needs --prompts and --out", file=sys.stderr) + return EXIT_BAD_INPUT + if a.target_recorded and not a.draft: + print("spec_decode_acceptance: --target-recorded needs --draft", file=sys.stderr) + return EXIT_BAD_INPUT + if a.target_recorded and bool(a.pass_ms) != bool(a.step_ms): + print("spec_decode_acceptance: --pass-ms and --step-ms go together (the bar is their ratio)", file=sys.stderr) + return EXIT_BAD_INPUT + try: + if a.record_openai: + return record_openai(a.record_openai, a.prompts, a.tokens, a.out) + return recorded(a) + except (urllib.error.URLError, OSError) as e: + print(f"spec_decode_acceptance: a server did not answer: {e}", file=sys.stderr) + return EXIT_SERVER if not (a.target and a.draft and a.prompt_file): print(__doc__) return EXIT_BAD_INPUT