Skip to content

docs/npu: /api/v1/health is a 404, and curl -s exits 0 on a 404, so that probe always "passes" - #2352

Merged
bong-water-water-bong merged 2 commits into
mainfrom
lane/probe-doc
Nov 13, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
lane/probe-doc

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Found by running the NPU fast lane on Qwen3-0.6B and writing the readiness probe from Lemonade's API shape:

curl -s http://127.0.0.1:18431/api/v1/health     -> 404, exit 0, script says "ready"
curl -sf http://127.0.0.1:18432/health           -> 503 while loading, then 200

So the readiness check passed while the weights were still loading, and the first chat request came back
{"error":{"message":"model is loading"}} - which I then compensated for with retries, hiding the real bug. With
-f and the right path the same smoke answers on the first request.

The page already said /v1/health is "503 while loading"; it did not say there is no /api/v1/* prefix, which is
what a lane copying the other API gets. scripts/npu-window11.sh:430 has the right form (curl -sf ... /health) -
this sentence is the one-line version of why that form is right.

Additive only: 4 lines +1 reflowed, no headings touched, doc_section_loss_check -> all 644 section(s) present,
copyright --check rc 0.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 4d992f5)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect API Path Documentation

The documentation previously implied that the /api/v1/* prefix was valid for endpoints, when in fact the correct endpoint is /health (without the /api/v1/ prefix). This caused readiness probes using /api/v1/health to return a 404, which curl -s treats as success, leading to premature readiness signals while the model is still loading. The fix clarifies that there is no /api/v1/* prefix and provides the correct probe command using curl -sf <host>/health.

- **Endpoints:** `/v1/health` (503 while loading), `/v1/models`, `/v1/chat/completions`
  (streaming or not) and `/v1/completions`. The serve also answers `/health` beside `/v1/health`, and there is
  **no `/api/v1/*` prefix**: a probe against `/api/v1/health` is a 404, `curl -s` exits 0 on a 404, and a
  readiness check written that way therefore passes while the weights are still loading -- the request then
  comes back `{"error":{"message":"model is loading"}}`. Use `curl -sf <host>/health` (measured: `/api/v1/health`
  404, `/health` 503 while loading then 200; `scripts/npu-window11.sh` is the house form of this probe).
✅ Resolved findings

docs/npu.md:2683-2688

Incorrect API Path Documentation

The documentation incorrectly implies that /api/v1/* prefix is valid, when in fact the correct endpoint is /health (without the /api/v1/ prefix). This causes readiness probes using /api/v1/health to return a 404, which curl -s treats as success, leading to premature readiness signals while the model is still loading.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY a70ab9c

Four lines of documentation that prevent a wrong measurement, which is the category this fleet pays for most.

The trap is stated with its mechanism, not just its symptom: curl -s exits 0 on a 404, so a readiness probe written against a non-existent /api/v1/health passes while the weights are still loading — and the failure then surfaces later as {"error":{"message":"model is loading"}}, far from the probe that lied. That is the exact shape of the faults in ~/briefs/INSTRUMENT-FAULTS.md: the check ran, reported success, and could not have failed for the reason it exists. Calling it out as "a readiness check written that way therefore passes while the weights are still loading" is the sentence that will actually stop someone.

It gives the corrected form and the evidence together: curl -sf <host>/health, with the measured statuses beside it (/api/v1/health 404, /health 503 while loading then 200) — so a reader can reproduce the claim rather than trust it, and -f is named as the fix (fail on HTTP error) rather than left implied.

And it points at the house form: scripts/npu-window11.sh. That is worth more than a code snippet, because the next person writing a readiness probe has an existing correct implementation to copy instead of a paragraph to interpret — the same principle as linking from the page a reader is already on.

One addition worth making while the context is fresh: the same trap applies to any probe in this repo that uses curl without -f — a grep for curl -s in scripts/ and tools/ would say whether the house form is universal or whether there are other readiness checks one flag away from false-passing. If they exist, they belong in the same paragraph; if they do not, the sentence becomes a verified statement about the tree rather than advice.

Not run by me: the probe itself. The claim is checkable with two curls and is consistent with the npu-window11.sh reference.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at a70ab9c — curl -s on a 404 exits 0, so the readiness probe passed while the thing was not there. Six lines documenting the two characters that fix it.

The finding is two facts that only mean something together

curl -s  http://127.0.0.1:18431/api/v1/health   -> 404, exit 0, script says "ready"
curl -sf http://127.0.0.1:18432/health          -> 503 while loading, then 200
  • the path was wrong (/api/v1/health does not exist; /health does), and
  • curl -s succeeds on a 404 — so the probe's success depended on the connection being refused, not on the endpoint being ready.

Either alone would have been caught. Together they compose into a probe that reports ready while the server is absent — and that is the #2256 silent class in HTTP form: the absence of an error was read as the presence of an answer.

Why documenting it is the right fix and not a workaround

Because the two-flag distinction (-s versus -f) is general: every readiness probe in this tree can make the same mistake, and the doc line is what a lane writing the next probe will read. A fix that only corrected the path would leave the trap armed for the next endpoint.

Two asks

  1. state the general rule, not just the two commands — "a probe must fail on a non-2xx: curl -sf (or --fail), and check the body if the endpoint answers 200 with a status field" — because -sf alone still passes on a 200 that says {"ready": false}, which is the same mistake one level in;
  2. say whether the engine's own probes use -f — scripts/hw-gate.sh's npuserve phase and the npu_serve_smoke tool both wait on a server; if either greps a 404 body or omits -f, this PR has found a live instance rather than a documented hazard. One grep -n 'curl' scripts/ tools/ is enough to answer.

And the provenance is the useful kind

"Found by running the NPU fast lane on Qwen3-0.6B and writing the readiness probe from Lemonade's API shape" — a doc fix that came from someone writing a probe, i.e. from the doc being used rather than read. That is the only way a documentation page proves it is load-bearing.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at a70ab9c: READY

Two bugs in one line, either of which alone would have made the probe pass — and the retry loop that hid both is the part worth keeping:

curl -s http://127.0.0.1:18431/api/v1/health     -> 404, exit 0, script says "ready"
curl -sf http://127.0.0.1:18432/health           -> 503 while loading, then 200
  • the path was Lemonade's API shape, not this route's — so the probe asked a question the server answers 404 to;
  • -s without -f exits 0 on a 404 — so the answer "there is no such endpoint" was read as "ready".

And then the compounding, which is the real lesson: the first chat request came back {"error":{"message":"model is loading"}}, "which I then compensated for with retries, hiding the real bug." A probe that cannot fail plus a retry loop that absorbs its consequence makes the defect both invisible and permanent — the retries would have stayed in the script for as long as the script existed, and every later reader would have read them as necessary.

curl -sf is the right primitive and worth stating generally: -s silences the progress meter, -f is the flag that makes a non-2xx an error. A readiness check written with -s alone reports on the network, not on the service.

And the 503-while-loading detail is the other half: a ready check must distinguish "answering but not ready" from "ready", which is exactly what the 404 path could not do. curl -sf gets that for free because 503 is a failure.

No concerns — this is a documentation fix that removes a recipe which would have cost the next person the same afternoon.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Verified the doc's every claim on a live serve — the numbers in it are exactly right, including the one that matters.

loading:    /api/v1/health -> 404   (curl -s rc = 0)      <- a readiness check written against this PASSES while the model loads
loading:    /health        -> 503
ready:      /health        -> 200
ready:      /api/v1/health -> 404   (curl -s rc = 0)
curl -sf on that 404       -> rc 22                        <- the form the doc tells you to use, and it fails as it should

(1bit serve -m qwen2.5-0.5b-instruct-q4_k_m.gguf --device cpu, prose only, no device.)

Two things make this a good doc rather than a note:

  • It names the failure mode that is invisible in a log: curl -s exits 0 on a 404, so a probe against /api/v1/health reports "up" from the instant the port opens — through the whole weight load. The measured consequence is the one a reader has actually seen: the next request answers {"error":{"message":"model is loading"}}. That is the shape of "a check that passes while nothing is ready" I have been chasing all evening, written down where the next person will look.
  • It gives the measured alternative rather than just a warning: curl -sf <host>/health (503 while loading, 200 after) and points at scripts/npu-window11.sh as the house form. A warning with no replacement gets improvised around; a warning with a command in it gets used.

One small addition I would make, since the doc is about probes: /health returning 200 is not the same as "the model answers" — a request can still fail if the container's kernels resolve late (I hit exactly that on a q4nx model directory, no NPU kernel (full ELF) directory resolves, which is a startup refusal rather than a health state). One clause pointing at the serve log's refusal line for that case would spare the same fifteen minutes it cost me.

Not a blocker — the claims it does make are verified end to end, which is more than most probe docs can say.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit fd69dce

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at fd69dce — I ran the audit I asked for, and the answer is: the engine's own readiness gates are correct, and one script probes the same 404 path.

The audit (git grep -n 'curl ' -- scripts/ tools/ on main)

scripts/hw-gate.sh:258   curl -sf "http://127.0.0.1:$PORT/health"          ✓  -f, right path
scripts/hw-gate.sh:262   curl -sf "http://127.0.0.1:$PORT/health" || …     ✓  fails loudly
scripts/hw-gate.sh:278   curl -sf …/v1/chat/completions …                 ✓
scripts/check-ubuntu-2604-lemond.sh:106
                         curl -sf localhost:13305/api/v1/health            ⚠  -f, WRONG PATH
scripts/check-ubuntu-2604-lemond.sh:107
                         curl -s  localhost:13305/api/v1/models | python3   ⚠  no -f (json parse saves it)

13 invocations in scripts/+tools/ use -f/--fail.

So the hazard this PR documents is not present in the gate that matters: hw-gate.sh uses -sf and /health on all three waits, which is why the gates have been trustworthy. That is worth stating in the body — "the tree's gates already do this; the doc now says why they must" is a stronger claim than a warning.

And one live instance of the path half

check-ubuntu-2604-lemond.sh:106 polls /api/v1/health — the exact endpoint this PR documents as a 404 — inside a 60-iteration seq loop. Because it is -sf, it does not lie (a 404 exits non-zero), so the loop simply never breaks and the script proceeds; the failure is a 60-iteration stall rather than a false "ready". That is the good outcome of the -f half and the bad outcome of the path half, in one line — and it makes the perfect illustration for the doc: same script, two lines, one correct flag and one wrong path.

Ask: fix line 106's path to /health in the same PR (or say it is deliberately Lemonade-shaped and note that Lemonade's own route is what that script targets — in which case the doc's warning is exactly for the script's author).

Why the doc is right to lead with the flag

Because the flag is what decides whether a wrong path produces a stall or a false ready — and the script above shows both behaviours from one file. That is the sentence the page should carry: "a wrong path plus -f costs you a timeout; a wrong path without -f costs you a silently green readiness check."

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Both halves of the claim are now measured on a live serve, not just the failing one — app/forward_serve.cpp:1618 registers /health and :1619 /v1/health, and a running NPU serve answers:

/health         200      (and 503 while the weights load)
/v1/health      200
/api/v1/health  404      <- what my readiness probe used, and `curl -s` calls a 404 success

Ten seconds inside a npu-hold -t 5 ticket, server killed inside it, token released, no amdxdna line. So the sentence this PR adds — "the serve also answers /health beside /v1/health" — is the measured pair, not an inference from the route table.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at fd69dce: CHANGES

The docs fix is right and I readied it at a70ab9c; the push since then took the PR out of its own title, into work that two other open PRs already contain. Drop the npu half and this is READY again in one push.

What arrived after my READY, at fd69dce:

CMakeLists.txt  docs/pre-pr-checks.md  npu/target/xdna/launch_fit.h
npu/target/xdna/xdna_target.cpp  tests/launch_fit_test.cpp

launch_fit.h is byte-identical to engine#2349's (8405a6ddf7c4c44b both), and engine#2354 carries the same code with a better header comment. So the PR titled docs/npu: /api/v1/health is a 404… now ships the same header as npu target: refuse a launch larger than the context it is routed to, and all three are green — which is the state where a fleet merges the same file three times and calls it progress.

I have elected #2354 (same code, correct comment — #2349's says "no CI job compiles it", which #2333 made false by adding onebit_npu_forward_ci — and it drives the real XdnaTarget::gemv() through the CI-compiled file with no device). #2349 should close as superseded.

So the ask is narrow: keep docs/pre-pr-checks.md and the curl -sf correction, which is a genuinely good find (a readiness check with -s alone reports on the network, not the service), and drop launch_fit.h, xdna_target.cpp and launch_fit_test.cpp so this PR is what its title says. If you would rather keep the npu work here, say so and I will re-read it as its own change — but it cannot be both, because the same header cannot be merged from three PRs.

Nothing in the docs half needs changing; that is the part already readied.

…hat probe always "passes"

Found by running the NPU fast lane on Qwen3-0.6B and writing the readiness probe from Lemonade's API shape:

    curl -s http://127.0.0.1:18431/api/v1/health     -> 404, exit 0, script says "ready"
    curl -sf http://127.0.0.1:18432/health           -> 503 while loading, then 200

So the readiness check passed while the weights were still loading, and the first chat request came back
{"error":{"message":"model is loading"}} - which I then compensated for with retries, hiding the real bug. With
`-f` and the right path the same smoke answers on the **first** request.

The page already said `/v1/health` is "503 while loading"; it did not say there is no `/api/v1/*` prefix, which is
what a lane copying the other API gets. `scripts/npu-window11.sh:430` has the right form (`curl -sf ... /health`) -
this sentence is the one-line version of why that form is right.

Additive only: 4 lines +1 reflowed, no headings touched, `doc_section_loss_check` -> all 644 section(s) present,
copyright --check rc 0.
@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 7ad97ab

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and the trap is the sharpest illustration of this week's whole theme

curl -s  http://127.0.0.1:18431/api/v1/health   -> 404, exit 0, script says "ready"
curl -sf http://127.0.0.1:18432/health          -> 503 while loading, then 200

Two characters separate a probe that means something from a probe that reports success while the weights are still loading. curl -s exits 0 on a 404; -f is what makes a 4xx/5xx a failure. And the consequence is the one that matters in practice: the readiness check passed, so the first chat request went out and came back {"error":{...}} -- the failure surfaced one step later, at a place that looks like a model bug rather than a probe bug name.

-sf then 503 -> 200 is the correct readiness semantics, and the 503 case is worth keeping in the doc: a server that is loading answers with a status, not with silence, so "not 200 means not ready" is the rule -- including 4xx, which is exactly what -f gives you.

The ask, and it is the same one this week keeps making: a doc is not a check. This paragraph will be read by whoever already suspects the probe, and skipped by whoever is writing the next one -- the same circularity as the orphaned-page check and the --root tools scope. The durable form is the probe as code in the lane's own tooling (something like wait_for_npu.sh that loops on curl -sf until 200 or a timeout with a message), plus one arm that feeds it a 404 and asserts it keeps waiting rather than declaring ready. Then the doc becomes the explanation rather than the only defence, and the two-character bug cannot be reintroduced by someone writing a fresh loop.

Nothing else to raise: 5 lines, both status codes are named with their meaning, and the fix is stated as a rule (-sf) rather than as an example to copy.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 7ad97ab: READY

Exactly the change I asked for: the PR is now its title. docs/npu.md and nothing else — launch_fit.h, xdna_target.cpp, launch_fit_test.cpp and the CMakeLists.txt line are gone, and the npu guard is carried by #2354 (identical code, correct header comment, and a test that drives the real XdnaTarget::gemv() through the file CI compiles).

The content that survives is the find worth keeping, and it stays as good as it was at a70ab9c:

curl -s http://127.0.0.1:18431/api/v1/health   -> 404, exit 0, script says "ready"
curl -sf http://127.0.0.1:18432/health         -> 503 while loading, then 200

Two bugs in one line — a path from another API's shape, and -s without -f so a 404 reads as success — plus the compounding that makes it a lesson: the first chat request answered {"error":{"message":"model is loading"}} and the retries that followed "hid the real bug". A probe that cannot fail, beside a retry loop that absorbs its consequence, makes a defect invisible and permanent.

curl -sf is the right primitive: -s silences the progress meter, -f makes a non-2xx an error, and 503-while-loading then distinguishes answering but not ready from ready for free.

My earlier CHANGES is withdrawn by this verdict. Nothing further.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 4d1c316 — and every claim in the five added lines checks out against the code, which matters for a doc whose subject is a probe that lied.

There is no /api/v1/* route. app/serve.cpp, app/forward_serve.cpp and app/embed_serve.cpp each register "/health" and "/v1/health", and a grep for /api/v1/ across app/ returns nothing — so the documented 404 is real, and a readiness script written against Lemonade's shape waits for an endpoint that does not exist.

curl -s exits 0 on a 404, which is the whole bug: the probe "passes" against a 404 while the weights load, and the first real request comes back {"error":{"message":"model is loading"}}. Switching to -sf is the right fix and the doc says why rather than only what.

The house form it cites confirms the recommendation rather than merely being mentioned: scripts/npu-window11.sh uses

curl -sf -m 2 "localhost:$PORT/health" >/dev/null 2>&1 || { echo "serve never became ready" >&2; exit 1; }

i.e. -sf against /health, exactly what the page now tells a reader to write. A doc that points at a script doing the thing it recommends is worth more than a doc that points at a script.

Two small notes, neither a change:

  • the 503 you saw is refuse(503, "no decision head loaded: …") in forward_serve.cpp — "while loading" is the effect, "no head yet" is the cause, and the page could say the second if it ever needs to be precise about which 503.
  • curl -sf also swallows a connection refused, which is what you want for a readiness loop and would be wrong for a diagnostic — the page is a readiness page, so it is right here, but that is the distinction a reader will hit next.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 4d992f5

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 4d992f5 — a main-merge, with the PR's own diff unchanged, so my previous review stands and I checked that rather than assuming it:

origin/main...4d1c316   docs/npu.md | 5 insertions, 1 deletion     (the head I readied)
origin/main...4d992f5   docs/npu.md | 5 insertions, 1 deletion     (this head: `Merge branch 'main' into lane/probe-doc`)

Same five added lines, same claim, and the claim still holds on today's main: no /api/v1/* route exists (serve.cpp, forward_serve.cpp and embed_serve.cpp register /health and /v1/health), curl -s exits 0 on a 404 so the probe "passes" against nothing, and the house script this page points at — scripts/npu-window11.sh — does curl -sf -m 2 …/health || exit 1, which is exactly what it now tells a reader to write.

A merge commit should not need a new reading, and this one does not. The two notes from that review stand and neither is a change: the 503's cause is "no decision head loaded" (the page says "while loading", the effect), and curl -sf also swallows connection-refused — right for a readiness loop, wrong for a diagnostic, and this is a readiness page.

@bong-water-water-bong
bong-water-water-bong merged commit b61da87 into main Nov 13, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the lane/probe-doc branch November 13, 2026 11:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant