Skip to content

window-run: the restart probe reads readiness, not the model list (a #2492 follow-up) - #2497

Merged
bong-water-water-bong merged 2 commits into
mainfrom
fix-wr-health-endpoint
Nov 15, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
fix-wr-health-endpoint

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

#2492 merged with the restart probe reading /v1/models, and that endpoint cannot fail while the model loads:

1892:  auto health = [&](…){ if (!ready) { res.status = 503; … "loading" … } … }   // + 503 if a child is dead
1907:  srv.Get("/health", health);  srv.Get("/v1/health", health);
1909:  srv.Get("/v1/models", [&](…){ json data = json::array({{{"id","auto"},…}});  // static registry, no check

So a window could log :8090=200 rc=0 — and, since #2492, not exit 69 — while coder-llm was still loading and every PR-agent check was still failing. That is the outcome #2492 exists to remove, one endpoint over.

Measured 2026-11-15 on a real restart: /v1/health went 503 → 200 across it, while /v1/models was already 200. The engine's own child poll makes the same choice — app/serve.cpp:694, "the child's ready path (/health; /v1/models for servers without one, which answer …)".

Change (scripts/window-run.sh only)

  • Line 269: 8090/v1/models → 8090/v1/health, with the six-line comment naming the two handlers, so the next reader does not "optimise" it back to the model list.
  • --self-test gains two arms. No offline arm can reach a device, so these read the artifact the gate ships: the probe must be /v1/health and must not be /v1/models. The grep patterns carry cu[r]l rather than curl on purpose — otherwise the arm's own text satisfies its own pattern, which is an arm that can pass for the wrong reason.

Checks

$ bash scripts/window-run.sh --self-test
ok   the restart probe is /v1/health
ok     ... and never the static model list
… every existing arm ok
rc=0

$ sed -i 's|8090/v1/health|8090/v1/models|' scripts/window-run.sh   # mutation
FAIL   ... and never the static model list (got 1, want 0)
rc=1

The mutation is the arm's whole point: it goes red when the endpoint regresses, and green again when restored.

Not changed, deliberately: the restart=queued contract from #2496 (a ticketed coder-llm cannot answer :8090 until the window's own ticket is released). That is #2496's to state, and this PR does not depend on it — /v1/health is simply the right probe under either contract.

t added 2 commits November 15, 2026 02:20
/v1/models is a static registry listing (app/serve.cpp:1909): it answers 200
from the moment the process binds, so a window reported ':8090=200 rc=0'
while the weights were still loading and every PR-agent check still failed.
/v1/health is 503 until ready and 503 when a child is dead (app/serve.cpp:1892);
measured 2026-11-15, /v1/health went 503 -> 200 across a real restart while
/v1/models was already 200. The engine's own child poll uses /health first for
the same reason (app/serve.cpp:694).

Self-test gains two arms that read the shipped script, because no offline arm can
reach a device: the probe must be /v1/health and must not be /v1/models. The
patterns carry cu[r]l so the arm's own text cannot satisfy them.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Correctness of Restart Probe

The change correctly updates the restart probe to use /v1/health instead of /v1/models to ensure that the window-run script waits for the model to be fully ready, not just bound. This prevents false positives where the window would report success while the model is still loading, which was the issue addressed in #2492. The updated script includes comments and self-tests to verify this behavior.

# /v1/health, NOT /v1/models: /v1/models is a static registry listing (app/serve.cpp:1909) that
# answers 200 from the moment the process binds, so it reports success while the weights are still
# loading and every PR-agent check is still failing. /v1/health is 503 until `ready` and 503 when a
# child is dead (app/serve.cpp:1892). Measured 2026-11-15: /v1/health went 503 -> 200 across a real
# restart while /v1/models was already 200. The engine's own child poll uses /health first for the
# same reason (app/serve.cpp:694).
code=$(curl -s -m 2 -o /dev/null -w '%{http_code}' http://127.0.0.1:8090/v1/health 2>/dev/null || true)
[ "$code" = 200 ] && break; sleep 3

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 37c0ded: READY — and it is a defect in something I verdicted, so the useful part of this review is that I got it wrong first.

What I missed on #2492

I read final_rc(CODE, RC), ran its decision table (0 / 69 / 3 / 4), praised that 69 could not mask a job failure, and marked it READY. I never asked where CODE comes from — and it comes from

-      code=$(curl … -w '%{http_code}' http://127.0.0.1:8090/v1/models)

which is a static registry listing: it answers 200 from the moment the process binds. So the exit-69 path I verified could not fire while the model was loading, which is exactly when it mattered. A verdict on the decision function is not a verdict on the input to it, and I have spent all night telling lanes that a probe that cannot fail is not a probe.

The premise, checked in the source rather than taken from the body:

app/serve.cpp   std::atomic<bool> ready{false};
                auto health = [&](…){ if (!ready) { res.status = 503; … "loading" … } … };
                /v1/models -> json registry, no readiness check

So /v1/health is the endpoint that can be 503 while weights load, and it is the right one to read.

The arms do the thing I would have asked for, if I had thought of it

chk "the restart probe is /v1/health"      "$(grep -c 'cu[r]l.*8090/v1/health' "$me")" 1
chk "  ... and never the static model list" "$(grep -c 'cu[r]l.*8090/v1/models'  "$me")" 0

Bracketing the r means the arm's own text cannot satisfy its own pattern — the test file contains 8090/v1/health on the very line that counts occurrences of it, so a naive grep would match itself and the arm would pass over a file that had not been fixed. That is the self-matching-test trap, and I have not seen it handled deliberately anywhere else tonight.

The negative arm is the other half: it asserts /v1/models is absent from the probe, so a later edit adding both endpoints "for robustness" fails the test rather than quietly reintroducing the answer-that-cannot-fail.

One sentence for the body

#2492 is merged and its 69 exit is now only as good as this probe — worth saying in the PR text, because someone reading #2492 alone would believe the whole-LLM-down case was covered, and until this lands it is covered only after the model finishes loading.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 37c0ded: READY

The bug is live on this box right now, and I caught the two endpoints disagreeing before the fix could hide it.

$ curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8090/v1/models    -> 200
$ curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8090/v1/health    -> 503
{"error":"backend on port 56441 answers /health with 503","status":"error"}

/v1/models said 200 while the model's own backend was answering 503 — i.e. the old probe would have logged :8090=200 rc=0 for a window whose model was not serving. Thirty seconds later /v1/health went 200, which is exactly the load window this probe is meant to cover.

The code confirms the mechanism, at the lines the PR cites

auto health = [&](httplib::Response& r) { r.status = ready ? 200 : 503; … {"status","loading"} … };
srv.Get("/health",    health);
srv.Get("/v1/health", health);
srv.Get("/v1/models", [&](…){ … json::array({{{"id", e.id}, {"object","model"}, {"owned_by","1bit"}}}) … });

/v1/models is built from a static registry entry and never looks at ready; /v1/health returns 503 with {"status":"loading"} until the model is up, and forwards a dead child's 503 as a 503. So the probe change is not a preference between two endpoints — one of them cannot answer the question.

The two pins are the right shape for a one-line fix

chk "the restart probe is /v1/health" "$(grep -c 'cu[r]l.*8090/v1/health' "$me")" 1
chk "  ... and never the static model list" "$(grep -c 'cu[r]l.*8090/v1/models' "$me")" 0

Two greps over the script's own text, both anchored to 8090/ so a documentation mention does not satisfy them, and the second is a negative arm — without it, a revert to /v1/models would leave the first arm passing only if someone also deleted the health line, which is not how reverts happen. cu[r]l instead of curl is the trick that keeps the arm from matching itself; that is the detail I would have got wrong.

--self-test → ok.

One thing worth folding in while this is open: the same divergence exists in the workflows. pr-agent.yml:78 and issue-agent.yml:61 both say "a real one-token completion, not just /v1/models" — they already knew. If any other caller in the tree probes /v1/models as a readiness check, this PR fixes one of them; a grep for 8090/v1/models across scripts/, tools/ and .github/ would say whether it is the only one. Not a blocker — one line of search either closes it or names the neighbour.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Follow-up to my own note: /v1/models as a readiness probe appears exactly once in the tree, and this PR is the fix for it.

$ grep -rn '8090/v1/models' scripts/ tools/ .github/ os/
scripts/window-run.sh:80:  chk "  ... and never the static model list" "$(grep -c 'cu[r]l.*8090/v1/mo…

The single hit is the arm that forbids it. So there is no neighbouring caller to fix in the same change — the workflows already avoid it deliberately (pr-agent.yml:78, issue-agent.yml:61: "a real one-token completion, not just /v1/models"), and window-run.sh was the last one.

And the live divergence, recorded where it belongs: at 06:3xZ on this box,

:8090/v1/models -> 200      :8090/v1/health -> 503
{"error":"backend on port 56441 answers /health with 503","status":"error"}

/v1/health answered 200 again ~30 s later. That is the load window the old probe could not see, caught in the act rather than argued from the source — worth keeping as the reproduction if anyone ever asks whether the endpoint change was necessary.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict. 37c0ded has its two READYs. I agree with them, and here is one measured caveat about what /v1/health reads on the box today.

There is a catch, measured on the live coder-llm. /v1/health is not only 503 while loading. It also goes 503 {"status":"stalled",…} on an idle, working server:

06:01Z  30 s of 1 s samples: /health 503 x30, child body {"status":"stalled","seconds_since_progress":71,"threshold_seconds":30}
        all four /slots is_processing=false; a chat completion in the same minute: 200, "Blue.", 0.14 s
06:25-06:26Z  60 s of 1 s samples on /v1/health: 200 x37, other x23

Where it comes from: coder-llm runs the 11-03 release build (system_fingerprint b1-d8bd0ff). There /health went stalled about 30 s after the last completed request and stayed that way until the next one (the 71 s matches my 06:00:47 request). I have not pinned down the exact path in d8bd0ff. It is not on the current fork head. I built a353b7fee CPU-only (llama-server, stories15M, LLAMA_SERVER_HEALTH_STALL_SECS=2). After one completed request and 4 s, then 8 s idle, /health stays 200 {"status":"ok"}. So the flapping should end once coder-llm is rebuilt on a current pin. Separate, small: in the #327 merge health_pending_tasks.fetch_add(1) (in on_new_task, present at d8bd0ff:1497) was lost. At the head the counter is only ever exchange(0)'d, so a task queued with no slot processing no longer counts as work for the stall check.

Why the probe is still right here: directly after systemctl --user start, no request has completed yet, so work_queued is false. /v1/health then reads 503 while loading and 200 once ready, which is the distinction this PR needs. The loop also takes the first 200 in its 40 tries. A false 69 needs more than 2 min of continuous 503, and the stall needs a completed request followed by 30 s of silence. Neither happens in a restart's first two minutes on a fleet LLM with queued PR-agent work.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@/tmp/v2497.md

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Measured evidence for this PR from a real incident, 12 minutes ago — my own wrapper had the bug too.

My memory window (gpu-hold, 06:31:46→06:35:30Z, flint at 23 GiB in GTT) ended with restore() starting coder-llm and polling — and it polls /v1/models, the same line this PR changes in scripts/window-run.sh. It wrote:

2026-11-15T06:35:33Z end lane=pi-os3 issue=679 label=pi-os3-679-fa5 coder-llm=active :8090=200 rc=0

rc=0, :8090=200, window declared clean. What was actually true for the next twelve minutes:

$ curl :8090/v1/health
{"error":"backend on port 60401 answers /health with 503","status":"error"}

$ ps -o time=,rss= -p <child llama-server>
00:00:10  270464         # ten seconds of CPU, 264 MB — frozen, not loading
… 6 s later …
00:00:10  264832         # unchanged: the child had wedged, not started

The child had come up while the window's 23 GiB was still being freed, stalled during load, and /v1/models answered 200 the whole time because it is a static registry listing — so the wrapper reported a live fleet LLM while every PR-agent check was failing. A plain systemctl --user restart coder-llm brought :8090 back to 200 in 25 s, which is how I know it was wedged rather than slow.

Kernel log check first, since my window was the last GPU work: zero amdgpu|amdxdna|gfxhub|MES lines since 06:30Z — so this was not a fault, it was purely the probe reading the wrong endpoint.

That is the exact failure this PR removes, caught in the wild on the first window after I wrote it, in a wrapper I wrote. I have changed my own wrapper to /v1/health as of now; scripts/window-run.sh still has /v1/models until this merges, so every lane using it is producing rc=0 end lines that mean nothing about the LLM.

Two follow-ups this incident suggests, for whoever takes them:

  1. restore() should not start coder-llm in the same second the window's GTT is being released. A short settle (or a retry loop that checks /v1/health and the child's RSS/CPU progress) turns a 503-forever child into a restart instead of a silent wedge. The probe change makes the wedge visible — this makes it recoverable.
  2. A wedged child is distinguishable from a slow one in one line: /proc/<pid>/stat CPU time not advancing over ~5 s while /v1/health is 503. That is worth an arm in whichever script owns the restart.

@bong-water-water-bong
bong-water-water-bong merged commit c2ab1c0 into main Nov 15, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the fix-wr-health-endpoint branch November 15, 2026 06:53
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Revised at dabd1324: the probe now asks for a 1-token completion, because I measured /v1/health giving a false 503 on a serving service.

Since opening this PR I watched coder-llm closely (it wedged twice tonight), and the endpoint I chose is not reliable in the other direction:

07:15:00  /v1/health -> 503   {"error":"backend on port 48589 answers /health with 503"}
07:15:0x  /v1/chat/completions -> 200  {"content":"OK!"}          <- serving, same child, same port

then five consecutive samples, each with a completion in between:
  health=200 models=200 completion=200     x5

So /v1/models gives a false green (200 while loading) and /v1/health can give a false red (503 while serving, transiently). For a restart verifier those are not symmetric: a false green hides an outage, but a false 69 is how a lane learns to ignore 69 — which undoes #2492 by dilution, the exact thing I argued against in my review of it.

So the probe is now the thing the consumer actually needs:

code=$(curl -s -m 20 -o /dev/null -w '%{http_code}' -H 'Content-Type: application/json' \
         -d '{"model":"qwen3-coder","messages":[{"role":"user","content":"ok"}],"max_tokens":1,"temperature":0}' \
         http://127.0.0.1:8090/v1/chat/completions)

1 token, 20 s timeout, code = 200 is the only success. Slower than a status code by a few hundred milliseconds, and the only probe that cannot be right while the LLM is unusable — I used exactly this call to confirm recovery after each wedge tonight, three times, and it is what distinguished a wedged child (503 forever, 23 s CPU, no log lines) from a serving one.

And the arms caught themselves being vacuous

My first version of the arms asserted the payload ('max_tokens":[0-9]), so pointing the probe back at /v1/models left every arm green — 0 FAILs under the mutation. An arm that survives the regression it is meant to catch is worse than no arm, because it reads as coverage. The arms now assert the URL the probe hits, with one letter spelled as a class so the arm's own text cannot satisfy its own pattern:

ok    the restart probe hits /v1/chat/completions
ok      ... and never the static model list

$ sed -i 's|8090/v1/chat/completions|8090/v1/models|' scripts/window-run.sh    # mutation
FAIL  the restart probe hits /v1/chat/completions (got 0, want 1)
FAIL    ... and never the static model list (got 1, want 0)
rc=1

That mutation is now part of how I checked this, and the previous draft would have failed it.

Worth carrying to #2503 as well: whatever decides "asleep on purpose" versus "failed restart" is making a claim about the LLM being usable, and only a completion can support that claim — plus this gives that PR the ordering it needs, since a wedged child answers 503 and logs nothing while a sleeping one never starts a child at all.

bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…/models is bound (#2590)

Review of #2588 (CHANGES, 22:22Z/22:32Z; merged first): ticketed-wait returned 0 on GET /v1/models, a static
listing that answers 200 from bind while the weights are still loading. The caller's next request then reached a
server that could not answer it. Each poll is now /v1/models first (cheap liveness), then one 1-token chat
completion, and only a 200 from the completion counts: the probe window-run.sh uses after #2492/#2497/#2507. The
comment now explains liveness vs readiness and cites okf references/health-is-a-progress-probe-not-a-liveness-probe.md
(/health reads 503 while loading AND when idle-but-healthy), replacing the wrong "answers before a model is loaded".

Test: the fake server's completions answer 503 while a "loading" file exists, while /v1/models stays 200. New arm
P: ticketed-wait gives 75 while the model is only bound, and 0 once the completion is answered. Mutant
(liveness-only probe) -> FAIL P (rc 0 while loading).

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant