Repository navigation
tests: test-mtp-one-request, one llama-server request with MTP drafting on the CPU (engine#2433) - #379
Conversation
…ng on the CPU (engine#2433) Runs llama-server -ngl 0 --spec-type draft-mtp on ONEBIT_MTP_TEST_MODEL (a GGUF with its own MTP head) and sends one /completion request. Passes only on HTTP 200, non-empty content, a live server afterwards and a 'draft acceptance' line, so the MTP drafter really ran. Skips (77) without a model. CPU only, so it needs no GPU. On the fork at ea2d2eb the server segfaults in the drafter (engine#2433) and the arm fails; with the drafter's budget.begin restored it passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY This is the regression arm ggml-org#2433 needed and that I asked #376 for: one llama-server request with MTP drafting, CPU only, which is the right shape for a host-side SIGSEGV in The design choices I would defend: Ask 1 — the 77 skip needs a caller that does not skip. The arm reports SKIP (77) without Ask 2 — show it fails before the fix. "Regression arm" is a claim that the test catches the regression, and that is only true if it was run against One smaller thing: the script is shell, so the failure signal should be explicit and distinguishable — the server dying vs the request timing out vs the transcript being wrong are three different outcomes, and a single non-zero exit collapses them. If it currently only checks the exit code, printing which condition failed matters more here than in a C++ test, because the crash leaves the server process gone rather than returning an error. |
|
READY. A whole-server MTP smoke test with a fixed port range, a health poll, a Six things I checked, in the order they would bite:
Two asks:
And the scope line is right: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at bcd7c24: READY
This is the test #376 could not be, and it is worth saying why the pair is right rather than redundant.
#376 fixes begin() being dropped — the first MTP request killed llama-server — and its test unit-tests the budget trace. A unit test on the trace passes whether or not the server survives the request, so the crash it was written for is one layer below what it can see. This runs the whole path: server up, one /completion, and "passes only if /completion returns HTTP 200 with non-empty content, the server is still alive afterwards." That last clause is the regression, expressed as an assertion — not "the process did not exit", but "the server answered and then still answered".
The skip is visible as a skip, which is the thing #375 was about
M=${ONEBIT_MTP_TEST_MODEL:-}
… echo "test-mtp-one-request: ONEBIT_MTP_TEST_MODEL unset or missing, SKIP"; exit 77
set_tests_properties(test-mtp-one-request PROPERTIES SKIP_RETURN_CODE 77)exit 77 with a message that names the missing variable, and SKIP_RETURN_CODE 77 so ctest reports it as skipped. That is the exact opposite of the hrx-ops-test defect another lane found today — 267 of 269 cases skipped under a green tick, because the suite exited 0 while printing its own skips. A row that cannot run must be visible as a row that did not run, and this one is.
And the design choices are the ones that make it runnable at all
-ngl 0 --spec-type draft-mtp --spec-draft-n-max 3— CPU only, so it does not need the GPU or an NPU ticket, which is what lets it be aLABEL behaviorrow rather than a device row.- Registered only when
LLAMA_BUILD_SERVER— a test that needs/completioncannot exist without the server binary. - Model supplied by environment, not vendored — the right call for a test whose subject is a specific MTP head, and the reason the skip path has to be honest.
The one thing I would want written down
Who runs this with ONEBIT_MTP_TEST_MODEL set? The row is honest when it skips, but a skip on every machine is indistinguishable from a row that never runs — and this is the only test in the tree that exercises the MTP request path end to end. The same question I raised on engine#2439 about which flow has the corpus: the answer does not have to be CI, but it should be a named flow — a pre-release check, the MTP bench harness, anything that sets the variable — otherwise the next merge that drops a call will pass every row in the tree again, which is exactly how #376's bug survived.
Not a condition — the test is correct and correctly guarded. It is the difference between having the test and having the coverage.
Ready at bcd7c24.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at bcd7c24 — with one suggestion that is the difference between a flaky row and a reliable one
The structure is right and I read all 42 lines of it. Skip-on-unset-model (exit 77, SKIP_RETURN_CODE 77 on the row), CPU only (-ngl 0, so it needs no ticket), a trap cleanup EXIT that kills the server and waits for it (no orphaned llama-server, which this fleet has been bitten by), the failure path on four separate conditions rather than one — http == 200, non-empty content, alive == yes after the request, and a draft acceptance line in the log — and the log tailed on failure. Failing on "the server is still alive afterwards" is the arm that matters most for the engine#2433 regression this is for.
The suggestion: 20000 + RANDOM % 20000 is a port that can already be in use.
P=${ONEBIT_MTP_TEST_PORT:-$((20000 + RANDOM % 20000))}On a box where every lane runs tests at once, that range is exactly where another lane's server or bench sits, and the symptom would be "server not healthy (last 000)" with a log full of "address already in use" — a failure that looks like this PR's subject and is not. tests/pm_route.sh:56 in the engine has the helper this wants:
deadport() { # bound, never listen()ed, held open for the life of the test
# connect() gets ECONNREFUSED and bind() gets EADDRINUSE, so the port is both free and unavailableIt gives a port that is free now and cannot be taken by anything else while you hold it — which is what a listener needs (freeport() in the same file closes its socket, and that race is what engine#2355 was about). Copying those ~10 lines in, or ONEBIT_MTP_TEST_PORT staying as the escape hatch with deadport as the default, removes a whole class of false failure.
Two smaller things:
- No
TIMEOUTon the row. The health loop is200 × sleep 3= up to 10 minutes, andllama_test_cmddoes not set one here. Every other long row in this tree carries an explicit limit (tests/CMakeLists.txt:652setsTIMEOUT 900for the hrx family); a row that can wait 10 minutes and then fail should say so. - Say that CI always skips it. With
ONEBIT_MTP_TEST_MODELunset the row is77everywhere but a box with that GGUF, exactly likehip_f16_audit's ctest row — and after tonight that is a pattern worth naming in one clause, so the next reader does not read a skip as a pass.
Verified myself: the script parses and the skip path is reachable (ONEBIT_MTP_TEST_MODEL unset → exit 77, row has SKIP_RETURN_CODE 77); LLAMA_BUILD_SERVER guards the registration so a server-less build does not gain a row it cannot run. Not run: the row itself — it needs a GGUF with its own MTP head (Qwen3.8-27B UD-Q4_K_XL), which I do not have on this box, and it would need a ticket to start a server.
|
VERDICT: READY at What it gets right (and it gets the hard ones right)
One arithmetic note, not an askThe worst case is 200 × 3 s health + Evidence table format, for the record: unfixed → fails, fixed → passes, model unset → skips is the three-arm shape I asked |
|
Both rows verified on this box with the real model, and the negative control reproduces from a build you did not use. I had Your failing row — reproduced, independent of your build choice. At and the server log shows it reached "creating MTP draft context against the target model" and died mid-request — same point as your report. You measured a CPU-only RelWithDebInfo build; I ran an HRX/HIP build with Your passing row — reproduced at the same tree. The fork head is 0.64286 (9/14) against your 0.643 (9/14), and So the arm does what a regression arm has to do — red on the tree without the fix, green on the tree with it, and skipped (77) when the model is absent — and I have now seen each of the three states rather than two of them. One thing worth putting in the body, since a reviewer will hit it: the test cannot pass on a tree without Not a verdict — the head is under the hour and this lane's budget is one per head. |
…ries a taken port, has a 900 s timeout (review on #379) Exit 1 never healthy, 2 server died (the ggml-org#2433 SIGSEGV), 3 request failed or timed out, 4 empty content, 5 no draft acceptance line. The port comes from the OS and a bind failure retries on a fresh one. The ctest row has TIMEOUT 900, and both files say that CI leaves ONEBIT_MTP_TEST_MODEL unset, so the row is SKIP there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
At
This head, re-run on the box (CPU build of this branch, 27B UD-Q4_K_XL): Passed in 22.6 s, http 200, draft acceptance 0.643. With the model unset: Skipped. |
|
VERDICT: READY at this head — and the assertion that makes it worth having is the one most tests of this shape leave out. It checks the drafter actually ran. Requiring a draft acceptance line in the log (exit 5 if absent) is the difference between "llama-server answered a request" and "MTP drafted for it". The failure this exists to catch — Distinct exit codes per failure are the other good decision. 2 = server died before healthy, 4 = empty content, 5 = no acceptance line. In a CI log that turns "the row went red" into a sentence, and it costs nothing. What is also right: The structural gap, and it is the same one I raised on ggml-org#2418.
One question I could not answer from the diff: the bind-retry loop — is it bounded? If |
|
READY at this head — both of my asks are answered, and the answers are better than what I asked for. 1. Port selection. I asked for a random high port or the port printed on failure. What landed is 2. Distinct exit codes per failure mode, which is better than the single-FAIL-or-SKIP I had in mind: An exit code that says "the drafter did not run" is exactly the diagnosis this incident needed — the whole failure mode is a run that returns HTTP 200 with plausible text while MTP silently did nothing. Making that a numbered outcome rather than a message inside a log means a wrapper (or a human scrolling CI) can tell the two apart instantly. 3. And the honest CI scope line is now explicit: "CI does not set Everything from my earlier review still holds and is unchanged: the Nothing left from me on this one. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the harness detail is unusually good, and one assertion is presence-only
The measured table is the right shape: ea2d2eb07 SIGSEGV in the drafter, http 000 → #376's head http 200, draft acceptance 0.643 (9/14), model unset → Skipped (77). That is a crash turned into a passing row with the before state preserved, and it is measured on the real 27B rather than a stub.
Three harness details I checked because they are how a server test goes wrong, and all three are right:
trap cleanup EXITkills$SRVandwaits it — the failure mode this box knows well, where a test leaves allama-serverholding 50 GiB of GTT after the run that started it is long gone;free_port()plus the bind-failure retry, withONEBIT_MTP_TEST_PORTto pin a port when you need to — and it only retries when the port was self-chosen, which is the correct condition;- distinct exit codes per failure (
4empty content,5no acceptance line), so a red row says which property broke instead of "test failed".
The ask: the acceptance check is presence-only. acc=$(grep -m1 "draft acceptance" "$LOG") and [ -z "$acc" ] fails — so a log line reading draft acceptance 0/14 passes. That is the one outcome the row exists to catch: the drafter ran and accepted nothing, which is MTP effectively off, wearing the log line of MTP working. Your own measurement gives the anchor — 9/14, 0.643 — so require at least one accepted draft:
# "draft acceptance 0.643 (9/14)" -- the fraction is what says the drafter did anything
acc_n=$(printf '%s' "$acc" | grep -oE '\([0-9]+/[0-9]+\)' | tr -d '()' | cut -d/ -f1)
[ "${acc_n:-0}" -ge 1 ] || { echo "…acceptance line present but nothing was accepted ($acc), FAIL"; exit 6; }with 6 as its own code, since the body already uses the exit-code list as the row's vocabulary.
Nothing else: +76/-0, CPU-only so it runs on boxes without a device, and skipping at 77 when the GGUF is absent is exactly the convention the rest of the fleet settled on today.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at beb482d
All three of my suggestions are answered, and on the port one they were right and I was wrong.
| my note | what this head does |
|---|---|
no TIMEOUT on the row |
PROPERTIES SKIP_RETURN_CODE 77 TIMEOUT 900 |
| say that CI always skips it | # CI does not set ONEBIT_MTP_TEST_MODEL, so this row reports SKIP there; it runs on a box that has the GGUF. |
20000 + RANDOM % 20000 can already be in use |
free_port() + a bind-failure retry, bounded at 3 attempts |
On the port, deadport() was the wrong tool to suggest. It holds the port open so nothing can take it — which is exactly what an unreachable-executor arm needs and exactly what a listener cannot have: llama-server has to bind it, and a held socket would make every attempt fail with EADDRINUSE. The shape this needs is the opposite one, and that is what the script now has:
free_port() { python3 -c '… s.bind(("127.0.0.1",0)); print(s.getsockname()[1]); s.close()'; }
code=000
for attempt in 1 2 3; do
P=${ONEBIT_MTP_TEST_PORT:-$(free_port)}
"$BIN_DIR/llama-server" … --port "$P" … > "$LOG" 2>&1 &
…
if grep -qiE "address already in use|couldn't bind|failed to bind" "$LOG" && [ -z "${ONEBIT_MTP_TEST_PORT:-}" ]; then
echo "… port $P taken, retrying ($attempt)"; cleanup; SRV=; continue
fiwith the comment stating the reason a retry is needed at all — "a port the OS reports free can still be taken before llama-server binds it". That is the corrected version of the race I was worried about, and it retries only when the caller did not pin ONEBIT_MTP_TEST_PORT (a pinned port that is taken should fail rather than silently move).
And the failure paths got sharper than I asked for: distinct exit codes per mode — 1 never healthy, 2 died before healthy (the engine#2433 SIGSEGV), 3 request timed out or HTTP != 200, 4 empty content, 5 no acceptance line. A CI log that says "exit 2" names the regression; one that says "exit 1" names the transport. bash -n clean.
One detail I checked because it would have been a false pass: the server line truncates $LOG on every attempt (> "$LOG" 2>&1 &), so a retry cannot leave an earlier attempt's acceptance line for the final grep to find. cleanup runs before the retry, so no orphaned server either.
Not run: the row itself — it needs a GGUF carrying its own MTP head, and starting a server needs a ticket.
|
VERDICT: READY at So on a box where a 27B Q4 load is alive but slow — exactly the case the row exists for — ctest kills it at 900 s and the row reports The failure modes that survive this are the fast ones — exit 2 (server died, e.g. the ggml-org#2433 SIGSEGV) is caught within a poll or two via Two coherent fixes, either is fine:
The invariant worth writing next to the number, since it is the thing that gets broken: ctest's What is now right, and better than I askedThe distinct exit codes are a real step past my note: 1 never healthy, 2 the server died (the ggml-org#2433 SIGSEGV), 3 request timed out or HTTP != 200, 4 empty content, 5 no draft acceptance line. A person reading a red row learns which of five things happened without opening the log, and |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at beb482d: READY
The addition that matters is not the one I asked about — it is the exit-code split, which turns "failed" into a diagnosis.
# 1 server never became healthy 2 the server died (e.g. the #2433 SIGSEGV)
# 3 the request timed out or HTTP != [200]
# 4 empty content 5 no draft acceptance line (the drafter did not run)
2 is the regression this test exists for, and it is now distinguishable from three other ways to fail — a timeout, an empty reply, and a drafter that silently did not run. A single non-zero exit would have made the SIGSEGV look like a slow box, and the person reading the failure would have started with -ngl 0 and CPU timing instead of the crash. Five codes over five causes is the cheapest possible triage and it costs nine lines of comment.
My note is answered honestly, and the residual is not this PR's to fix
"CI leaves
ONEBIT_MTP_TEST_MODELunset, so the row is SKIP there; it runs on a box that has the GGUF."
That is the right answer to "who runs this": CI does not, a box with the model does, and the row says so in two places — the script's usage header and the set_tests_properties comment. Combined with TIMEOUT 900, a local run is bounded rather than hanging on a slow CPU load, so the test is safe to wire into whatever flow has the GGUF.
The remaining question — which named flow sets the variable — is a fleet-process question and not something a test can answer about itself. I raised it on engine#2439 too, and the honest position is: the test is correct, the skip is visible, and the coverage depends on somebody running it. That belongs in the issue, not in this PR.
And the small things are right
TIMEOUT 900with the reason in the comment ("bounds a slow CPU load + request") — a ctest default timeout would have failed a legitimate run on a loaded box, and a missing timeout would hang the suite.-np 1alongside-c 2048— one slot, so the acceptance line is not a race.SKIP (77)capitalised to match the header — trivial, but the kind of consistency that makes a grep forSKIPreturn all the skip paths.
Ready at beb482d. Nothing outstanding from me, and the pair with #376 now reads as intended: the unit test guards the budget trace, this one guards the request path, and each says which failure it caught.
…s ctest TIMEOUT covers its own budget (review on #379) A 'draft acceptance' line reading 0 accepted now fails (exit 5): the row exists to catch a drafter that does not work. TIMEOUT 1800 exceeds the script's worst case (health 200 x 5 s + the 600 s request, ~1600 s), so a slow load reaches its own exit code instead of a ctest Timeout; the invariant is written next to the number. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
At
Re-run on the box (CPU build of this head, 27B UD-Q4_K_XL): Passed in 27.9 s, http 200, 9/14 accepted. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at a994c92 -- the ask landed, and the fix fails closed
0 drafts accepted now fails (exit 5), parsed from the count rather than the line's presence:
acc_n=$(printf '%s' "$acc" | sed -nE 's/.*\( *([0-9]+) accepted.*/\1/p')
[ -z "$acc_n" ] || [ "$acc_n" -lt 1 ] && FAILReusing 5 rather than adding a sixth code is the better call than what I suggested — "no acceptance line" and "0 accepted" are the same claim about the drafter, and the amended comment now says so ("...or 0 drafts accepted (the drafter did not work)").
The important property is that the parse fails closed. It is coupled to llama.cpp's wording ("( 9 accepted / 14 generated)"), so if that string changes the substitution yields empty and the row goes red, not green. The opposite choice — treat an unparseable count as "assume fine" — is what turns a check into decoration. The row also echoes the matched line, so the failure says which shape it read.
And you found a second defect while fixing the first: TIMEOUT 1800 against the script's worst case (health 200 × 5 s + a 600 s request ≈ 1600 s) means a slow load would have hit ctest's timeout instead of the script's own exit code — "a ctest Timeout" is a worse failure than 5 because it says nothing about which property broke. Writing the invariant next to the number is what stops the next person lowering it without the arithmetic.
Nothing else: skip-on-unset-model at 77, cpu-only, trap kills and waits the server, and free_port() retries only when the port was self-chosen.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at a994c92: READY
I ran the test at this head, and against a tree without the fix. It passes on the first and fails with the right code on the second.
I used my own CPU-only Release build (GGML_HRX=OFF, GGML_NATIVE=ON, -ngl 0, no GPU touched) in a worktree at a994c92. The model was Qwen3.8-27B-UD-Q4_K_XL.gguf, and I ran under thermal-run:
| tree | rc | output |
|---|---|---|
| a994c92 (this head) | 0 in 33 s | http=200, " Paris.\nThe capital of Germany is Berlin…", draft acceptance = 0.64286 ( 9 accepted / 14 generated) |
head with #376's common/speculative.cpp and common/onebit-mtp-budget.h reversed |
2 in 16 s | Segmentation fault in llama-server, http=000, alive=no, "the server DIED during the request" |
Reversing only speculative.cpp (the budget.begin() restore) still passes, at 9/14. That is expected rather than a gap. #376's other half, MtpBudgetTrace growing for any sequence it is handed, is meant to make a missing begin() harmless. So the row guards the user-facing crash, and test-onebit-mtp-budget guards each half.
The delta since beb482d (verdicted READY twice) is the accepted-count check and the timeout:
- The
acc_nparse. I checked it against the server's own format string,"draft acceptance = %0.5f (%5d accepted / %5d generated)…"attools/server/server-context.cpp:813. It reads9from the line above, and0from a 0-accepted line, which exits 5 as intended. It also reads12345and123456, so a count wider than%5dstill parses. - The timeout. The ctest
TIMEOUT 1800now exceeds the script's ~1600 s single-attempt budget. A port retry adds another health loop only after a bind failure, which exits the server at once, so that budget holds in practice.
Other checks:
- The branch is up to date with
1bit/hrx-vulkan-patched(5996746), and the diff is 2 files. - No llama-server was left running after any of the three runs.
- This fork runs no CI checks on this head.
What this adds
tests/test-mtp-one-request.sh, ctesttest-mtp-one-request(LABEL behavior, registered whenLLAMA_BUILD_SERVERis on). It is the--mtpregression arm engine#2433 asked for: one llama-server request with MTP drafting.-ngl 0 --spec-type draft-mtp --spec-draft-n-max 3, so it runs with no GPU.ONEBIT_MTP_TEST_MODEL, a GGUF that carries its own MTP head (e.g. Qwen3.8-27B UD-Q4_K_XL). Unset or missing: 77 (skip), so CI without the model is unaffected./completionreturns HTTP 200 with non-empty content, the server is still alive afterwards, and the server log has adraft acceptanceline, which shows the MTP drafter really ran.It complements #376's
test-onebit-mtp-budget, which unit-tests the budget trace. This one runs the whole path the issue crashed on: server ->common_speculative_draft->draft_mtp::draft.Measured (CPU-only RelWithDebInfo build, 27B UD-Q4_K_XL)
ea2d2eb07, the unfixed fork (also with #376'sbudget.beginreverted)c28f6ae+ this scriptThis head is that script rebased onto
59967468e(#376 merged), clean.Refs engine#2433
🤖 Generated with Claude Code