Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -363,6 +363,10 @@ add_python_test(hf_tensor_self ${CMAKE_SOURCE_DIR}/tools/hf_tensor.py --self-tes
# `tools/ci/ctest_skips.txt`, and `(Not Run)` (a missing binary) as a build-list mistake that can never be baselined.
# Self-test only here: wiring it into ci.yml's Test step needs the log plus a `workflow`-scoped token (engine#1817).
add_python_test(ctest_skips_self ${CMAKE_SOURCE_DIR}/tools/ci/ctest_skips.py --self-test)
# engine#2106 / #2102: the release gate's NPU serve check asserts the fast route, a decode floor, the first token and a
# non-empty answer, because a serve on the slow per-op route still answers. --self-test drives the judging on the shape
# of each failure (the silent fallback at 1.9 tok/s, thinking left on, a degenerate reply); no device, no binary.
add_python_test(npu_serve_smoke_self ${CMAKE_SOURCE_DIR}/tools/npu_serve_smoke.py --self-test)
add_python_test(moe_placement ${CMAKE_SOURCE_DIR}/tools/moe_placement.py --self-test)
# engine#856: one HRX process's GTT split into the loader's copy and HRX's packed second copy, from its own -v log.
add_python_test(hrx_gtt_split_self ${CMAKE_SOURCE_DIR}/tools/hrx_gtt_split.py --self-test)
Expand Down
40 changes: 40 additions & 0 deletions docs/hw-gate.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,46 @@ The settings are environment variables at the top of the script: `HW_GATE_STORE`
`$HW_GATE_OUT` (default `~/.cache/hw-gate/runs/<time>`). Exit code: 0 pass, 1 a check failed, 2
the box could not run the gate (lock timeout, a model or base file missing, memory refused).

## The NPU serve check (`tools/npu_serve_smoke.py`)

The `npu` phase above checks the NPU engine through a private checker. It does not start `1bit serve`,
so it cannot see what engine#2102 and #2106 are about: a shipped binary whose serve quietly takes the
slow per-op route. That serve still answers, correctly, at about 2 tok/s where the fast route gives 80
to 100, so a check that asks "did it answer" passes on the wrong route.

`tools/npu_serve_smoke.py` starts the serve, sends three prompts with thinking off, stops the serve,
and asserts the four things the fallback fails:

| assertion | passes when |
|---|---|
| `route` | the serve log names the fast route and does not name the per-op forward |
| `rate` | the median decode rate is at least `--floor` tok/s (default 60; the two routes are 45× apart) |
| `first-token` | the median prompt time is at most `--first-token-ms` (default 600) |
| `content` | every reply has non-empty content that is not one character repeated |

```sh
npu-hold -l npu-smoke -t 10 -- tools/npu_serve_smoke.py --bin <1bit> --model <model dir> [--out <dir>]
tools/npu_serve_smoke.py --self-test # the judging on the shape of each failure; no device, no binary
tools/npu_serve_smoke.py --judge <dir> # judge a finished run's serve.log and reply_*.json again
```

The two route texts are what the serve prints today (`on the private route dx`, `per-op forward`);
`--fast-route` and `--slow-route` change them when the serve's wording does, and a log that names
neither is a failure, not a pass. It prints one line per assertion and `RESULT: PASS` or `RESULT: FAIL` (exit 0 or 1). It takes no device
ticket of its own: run it under `npu-hold`, and the serve ends with the script. Logs and replies go to
`--out` (default `~/.cache/1bit/npu-serve-smoke`).

Measured on strixhalo, 2026-11-11, Qwen3-0.6B, one ticket each:

| binary | route taken | result |
|---|---|---|
| the 2026-11-07 release candidate, with the kernel set that ships beside it | fast | `PASS`: median 84.4 tok/s (98.4, 84.4, 82.1), first token 277 ms |
| a build of main without the private route, model-generic directory | per-op | `FAIL`: `route`, `rate` (median 1.9 tok/s) and `first-token` (1,063 ms) fail; `content` passes |

The second row is the point: its three answers were fine, and only the other three assertions say
the release would have shipped the slow route. The check is not wired into `scripts/hw-gate.sh` or the
release script yet; it is the instrument, and the release lane decides where it runs.

## Measured

Strix Halo, 2026-10-12, at the release pin (main d2a56a4, llama.cpp 0e255d77, hrx-system
Expand Down
280 changes: 280 additions & 0 deletions tools/npu_serve_smoke.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,280 @@
#!/usr/bin/env python3
# Copyright 2026 bong-water-water-bong
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""npu_serve_smoke: does `1bit serve --device npu` serve a small model on the FAST route, at speed, with an answer?

The release gate's check for engine#2106 and engine#2102. A serve that falls back to the slow per-op forward still
answers, so "did it answer" passes on the wrong route (1.9 tok/s measured against 87-101). This asserts the four things
the fallback fails, as ~/briefs/DEMO-SMOKE.md specified them:

route the serve log names the fast route (and does not name the per-op forward)
rate median decode rate >= --floor tok/s (default 60: the two routes are 45x apart, so this is not a tolerance)
first-token median prompt time <= --first-token-ms (default 600)
content every reply has non-empty content that is not one character repeated

npu-hold -l npu-smoke -t 10 -- tools/npu_serve_smoke.py --bin <1bit> --model <model dir> [--out <dir>]

It starts the serve, waits for it, sends three prompts with thinking off, stops the serve, and prints one line per
assertion and `RESULT: PASS` or `RESULT: FAIL` (exit 0 / 1). It takes no device ticket itself: run it under npu-hold,
so the serve ends with the ticket. Logs go to --out (default ~/.cache/1bit/npu-serve-smoke), never to /tmp.

tools/npu_serve_smoke.py --self-test the judging on recorded shapes of each failure; no device, no binary
tools/npu_serve_smoke.py --judge <dir> judge a finished run's serve.log and reply_*.json again
"""
import argparse
import json
import os
import signal
import statistics
import subprocess
import sys
import time
import urllib.request

FAST_ROUTE = "on the private route dx"
SLOW_ROUTE = "per-op forward"
PROMPTS = [
"What is the capital of France? Answer in one sentence.",
"Write a Python function that returns the n-th Fibonacci number.",
"What is 17 times 23? Reply with the number only.",
]


def decode_rate(reply):
"""tok/s of one reply from the server's own timings, or None when it reports none."""
t = reply.get("timings") or {}
if isinstance(t.get("predicted_per_second"), (int, float)) and t["predicted_per_second"] > 0:
return float(t["predicted_per_second"])
n = (reply.get("usage") or {}).get("completion_tokens")
ms = t.get("predicted_ms")
if isinstance(n, int) and n > 0 and isinstance(ms, (int, float)) and ms > 0:
return 1000.0 * n / ms
return None


def content_of(reply):
try:
c = reply["choices"][0]["message"].get("content")
except (KeyError, IndexError, TypeError, AttributeError):
return None
return c if isinstance(c, str) else None


def degenerate(text):
"""One character repeated (the '//////' symptom of engine#1810), ignoring whitespace."""
s = "".join(text.split())
return len(s) >= 4 and len(set(s)) == 1


def judge(log_text, replies, floor=60.0, first_token_ms=600.0, fast_route=FAST_ROUTE, slow_route=SLOW_ROUTE):
"""[(name, ok, detail)] for the four assertions. Pure: a log and the replies in, verdicts out."""
out = []
fast, slow = fast_route in log_text, slow_route in log_text
if fast and not slow:
out.append(("route", True, "the serve log names the fast route"))
elif slow:
out.append(("route", False, "the serve log names the %s: this run is on the slow route" % slow_route))
else:
out.append(("route", False, "the serve log never says '%s'" % fast_route))

rates = [decode_rate(r) for r in replies]
if not replies or any(r is None for r in rates):
out.append(("rate", False, "%d of %d replies carry no decode timing" % (sum(r is None for r in rates), len(replies))))
else:
med = statistics.median(rates)
out.append(("rate", med >= floor, "median %.1f tok/s (floor %.1f; each: %s)" % (med, floor, ", ".join("%.1f" % r for r in rates))))

firsts = [(r.get("timings") or {}).get("prompt_ms") for r in replies]
if not replies or any(not isinstance(f, (int, float)) for f in firsts):
out.append(("first-token", False, "a reply carries no prompt_ms"))
else:
med = statistics.median(firsts)
out.append(("first-token", med <= first_token_ms, "median %.0f ms (limit %.0f)" % (med, first_token_ms)))

bad = []
for i, r in enumerate(replies):
c = content_of(r)
if c is None or not c.strip():
bad.append("reply %d: empty content%s" % (i, " (the tokens went to reasoning_content: is thinking off?)"
if (r.get("choices") or [{}])[0].get("message", {}).get("reasoning_content") else ""))
elif degenerate(c):
bad.append("reply %d: one character repeated (%r)" % (i, c[:12]))
if not replies:
bad.append("no replies")
out.append(("content", not bad, "; ".join(bad) if bad else "%d replies, all with content" % len(replies)))
return out


def report(verdicts):
ok = all(v[1] for v in verdicts)
for name, good, detail in verdicts:
print("%-12s %s %s" % (name, "ok " if good else "FAIL", detail))
print("RESULT: %s" % ("PASS" if ok else "FAIL"))
return 0 if ok else 1


def run(args):
out = os.path.expanduser(args.out)
os.makedirs(out, exist_ok=True)
log_path = os.path.join(out, "serve.log")
cmd = [args.bin, "serve", "-m", args.model, "--device", "npu", "--host", "127.0.0.1", "-p", str(args.port)]
replies = []
with open(log_path, "wb") as log:
serve = subprocess.Popen(cmd, stdout=log, stderr=subprocess.STDOUT)
try:
deadline = time.time() + args.ready_seconds
ready = False
while time.time() < deadline and serve.poll() is None:
try:
with urllib.request.urlopen("http://127.0.0.1:%d/health" % args.port, timeout=2) as r:
if r.status == 200:
ready = True
break
except OSError:
time.sleep(0.5)
if ready:
for i, prompt in enumerate(PROMPTS):
body = json.dumps({"model": "m", "max_tokens": args.max_tokens, "temperature": 0,
"chat_template_kwargs": {"enable_thinking": False},
"messages": [{"role": "user", "content": prompt}]}).encode()
req = urllib.request.Request("http://127.0.0.1:%d/v1/chat/completions" % args.port, data=body,
headers={"Content-Type": "application/json"})
try:
with urllib.request.urlopen(req, timeout=args.request_seconds) as r:
reply = json.loads(r.read().decode())
except (OSError, ValueError) as e:
reply = {"error": str(e)}
with open(os.path.join(out, "reply_%d.json" % i), "w") as f:
json.dump(reply, f)
replies.append(reply)
finally:
# the serve must not outlive this script (and so not outlive the ticket it runs under)
if serve.poll() is None:
serve.send_signal(signal.SIGTERM)
try:
serve.wait(timeout=20)
except subprocess.TimeoutExpired:
serve.kill()
serve.wait()
with open(log_path, errors="replace") as f:
log_text = f.read()
if not replies:
print("serve FAIL it never became ready within %d s (exit %s); see %s" % (args.ready_seconds, serve.returncode, log_path))
return report(judge(log_text, replies, args.floor, args.first_token_ms, args.fast_route, args.slow_route))


def rejudge(args):
d = os.path.expanduser(args.judge)
with open(os.path.join(d, "serve.log"), errors="replace") as f:
log_text = f.read()
replies = []
i = 0
while os.path.exists(os.path.join(d, "reply_%d.json" % i)):
with open(os.path.join(d, "reply_%d.json" % i)) as f:
replies.append(json.load(f))
i += 1
return report(judge(log_text, replies, args.floor, args.first_token_ms, args.fast_route, args.slow_route))


def self_test():
fails = []

def reply(content, tps, prompt_ms=220.0, reasoning=None, n=20):
m = {"role": "assistant", "content": content}
if reasoning is not None:
m["reasoning_content"] = reasoning
return {"choices": [{"index": 0, "finish_reason": "stop", "message": m}],
"timings": {"prompt_ms": prompt_ms, "predicted_ms": 1000.0 * n / tps, "predicted_per_second": tps},
"usage": {"completion_tokens": n}}

def expect(name, verdicts, want):
got = {v[0]: v[1] for v in verdicts}
if got != want:
fails.append("%s: got %s, want %s" % (name, got, want))

fast_log = "1bit unified: on the private route dx: dense Qwen3 on the dx kernels, one design per layer (/x/1024x3072x16x8)\n"
slow_log = "1bit unified: loading on the NPU\n[forward] decode family: 0 role(s) at M=16, 4 on M=128\n1bit serve: per-op forward\n"
good = [reply("The capital of France is Paris.", 101.4, 224), reply("def fib(n): ...", 87.5, 225), reply("391", 93.0, 263)]
all_ok = {"route": True, "rate": True, "first-token": True, "content": True}

# the measured demo row passes
expect("the measured fast-route row", judge(fast_log, good), all_ok)
# engine#2102 as it was seen: the slow route answers correctly, at 1.9 tok/s. Three of four assertions catch it.
slow = [reply("The capital of France is Paris.", 1.9, 4100), reply("def fib(n): ...", 1.87, 4200), reply("391", 1.9, 4000)]
expect("the silent fallback", judge(slow_log, slow), {"route": False, "rate": False, "first-token": False, "content": True})
# a log that names neither route is not a pass: the evidence has to be there
expect("a log that names no route", judge("1bit serve: ready\n", good), dict(all_ok, route=False))
# ... and a log that names BOTH (a retry that fell back) is the slow route
expect("a log naming both routes", judge(fast_log + slow_log, good), dict(all_ok, route=False))
# the fast route with thinking left on: every token in reasoning_content, content empty
thinking = [reply("", 95.0, 220, reasoning="Okay, the user asks ...") for _ in range(3)]
expect("thinking left on", judge(fast_log, thinking), dict(all_ok, content=False))
# one degenerate reply among good ones (engine#1810's '//////')
expect("one degenerate reply", judge(fast_log, good[:2] + [reply("//////", 90.0)]), dict(all_ok, content=False))
# the rate is the median: one slow reply of three does not fail it, two do
expect("one slow reply of three", judge(fast_log, [good[0], good[1], reply("391", 20.0)]), all_ok)
# ... and it IS the median: (70, 70, 1.9) has a mean of 47, under the floor, and still passes
expect("the median, not the mean", judge(fast_log, [reply("a b", 70.0), reply("c d", 70.0), reply("391", 1.9)]), all_ok)
expect("two slow replies of three", judge(fast_log, [good[0], reply("x y", 20.0), reply("391", 20.0)]), dict(all_ok, rate=False))
# just under and at the floor
expect("just under the floor", judge(fast_log, [reply("a b", 59.9)] * 3), dict(all_ok, rate=False))
expect("at the floor", judge(fast_log, [reply("a b", 60.0)] * 3), all_ok)
# a slow first token alone
expect("slow first token", judge(fast_log, [reply("a b", 90.0, 900.0)] * 3), dict(all_ok, **{"first-token": False}))
# replies without timings cannot pass the rate or the first-token assertion
bare = [{"choices": [{"message": {"content": "Paris."}}]}] * 3
expect("no timings", judge(fast_log, bare), {"route": True, "rate": False, "first-token": False, "content": True})
# timings without predicted_per_second: the rate comes from tokens and milliseconds
derived = [{"choices": [{"message": {"content": "Paris."}}], "timings": {"prompt_ms": 200.0, "predicted_ms": 100.0}, "usage": {"completion_tokens": 9}}] * 3
expect("rate from tokens / ms", judge(fast_log, derived), all_ok)
# an error reply, and no replies at all (the serve never came up)
expect("an error reply", judge(fast_log, good[:2] + [{"error": "timed out"}]), {"route": True, "rate": False, "first-token": False, "content": False})
expect("no replies", judge(fast_log, []), {"route": True, "rate": False, "first-token": False, "content": False})
if not degenerate("//////") or degenerate("391") or degenerate("aa") or not degenerate("a a a a"):
fails.append("degenerate(): wrong on a fixture")

for f in fails:
print("FAIL " + f)
print("npu_serve_smoke self-test: %d case(s) failed" % len(fails) if fails else "npu_serve_smoke self-test: 16 cases ok")
return 1 if fails else 0


def main():
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--self-test", action="store_true")
p.add_argument("--judge", metavar="DIR")
p.add_argument("--bin")
p.add_argument("--model")
p.add_argument("--out", default="~/.cache/1bit/npu-serve-smoke")
p.add_argument("--port", type=int, default=18173)
p.add_argument("--floor", type=float, default=60.0)
p.add_argument("--first-token-ms", type=float, default=600.0)
p.add_argument("--fast-route", default=FAST_ROUTE, help="the text the serve log prints on the fast route")
p.add_argument("--slow-route", default=SLOW_ROUTE, help="the text the serve log prints on the slow route")
p.add_argument("--max-tokens", type=int, default=96)
p.add_argument("--ready-seconds", type=int, default=120)
p.add_argument("--request-seconds", type=int, default=120)
a = p.parse_args()
if a.self_test:
return self_test()
if a.judge:
return rejudge(a)
if not a.bin or not a.model:
p.error("give --bin and --model (or --self-test, or --judge <dir>)")
return run(a)


if __name__ == "__main__":
sys.exit(main())
Loading