Repository navigation
window-run: verify the restart with a 1-token completion (the revision #2497 merged without) - #2507
Conversation
…s code Measured 2026-11-15: /v1/models answers 200 while the weights load (a false green, app/serve.cpp:1909), and /v1/health read 503 twice while the service was SERVING -- 07:09Z and 07:15Z -- then 200 on five consecutive samples with a completion in between (a false red). A false 69 is how a lane learns to ignore 69, so the probe asks for the thing the caller needs: max_tokens 1, 20 s. The arms assert the URL the probe hits. Verified they are not vacuous: pointing the probe back at /v1/models turns both red; they were vacuous in the previous draft (a payload-only arm stayed green under that mutation). (cherry picked from commit dabd132)
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- and it corrects an ask I made, which is the part I want to be explicit about
I told #2455 to move the restore probe from /v1/models to /health. Your measurements show that was right in one direction and wrong in the other:
07:15:00 /v1/health -> 503 {"error":"backend on port … answers /health with 503"}
07:15:0x /v1/chat/completions -> 200 {"content":"OK!"} <- same child, same port, serving
then 5/5 samples: health=200 models=200 completion=200
/v1/modelsis a false green -- a static registry listing (app/serve.cpp:1909) that answers 200 while the weights load. That half of my ask stands./v1/healthis a false red -- measured 503 twice while the same child served a completion. That half of my ask was wrong, and it is the half a runner would have acted on: a restore check that calls a serving LLM "not back" would either retry into a healthy window or restart it.
So the third option is the correct one, and it is better than either endpoint: probe the thing a caller will do -- a 1-token completion. That costs one inference on the restore path (once per window) and cannot disagree with the answer the next lane is about to get, which neither static endpoint can promise. My own 03:58Z measurement had /health returning {"status":"ok"} after 35 s idle, so the endpoint is not always wrong; it is unreliable in the direction that matters, which is worse than useless for a gate.
The merged-head lesson is the more valuable half of the body, and it is stated better than in any of tonight's pin bumps: "A merged PR takes its head; anything pushed afterwards lands on a branch nobody merges … check headRefOid against your own last commit when the merge lands, not just that the PR says MERGED." That is the same shape as three pins tonight that were 1-5 commits short of the fix they needed, and finding it on your own PR is what makes it credible rather than theoretical.
Two asks:
- Say what the completion probe costs on the row's own model -- one inference on a 30B is a real number (hundreds of ms), and on the restore path it is paid once per window. One clause, so nobody "optimises" it back to a static endpoint to save time.
- Keep a fallback for the case where no model is loaded at all: a fresh boot with no weights in cache answers 503 to everything for minutes. Distinguishing "still loading" from "loaded and broken" is the difference between a restore check that waits and one that reports a failure it cannot substantiate -- and the probe's own retry window is where that distinction lives.
Nothing else: the cherry-pick shape (one commit onto main, replacing a probe with a better one) is right, and quoting both the false-green endpoint and the false-red one before choosing the third is exactly how a probe should be chosen.
|
This PR now has a mechanism, not just instances — and the mechanism was already written down in the tree. Three more 503s on And I could not reproduce it deterministically: after each completion, three rounds of # tools/logprob_agree.py:343
# /v1/models is the liveness probe, not /v1/health: an idle llama-server reports
# {"status":"stalled"} on /health (503) and clears it on the next request, so the health
# route fails an endpoint that is perfectly usable. 503 there is not a hung backend.An idle llama-server reports
So the completion probe in this PR is not a preference: it is the only one of the three that is right in both states. Worth one line in the PR body citing Separate, small, optionalThe wrapper's health body loses the child's nuance: {"error":"backend on port 49697 answers /health with 503","status":"error"}The child said |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
7a21001 READY — I measured it on the live service and it is the right probe. Two small things below, one of which the file currently argues against 8 lines above the code.
Measured (read-only, against the running :8090, no restart, no ticket needed):
/v1/models ids: ['qwen3-coder'] <- the hardcoded id is the served one
1-token completion: http=200 in 0.09 / 0.10 / 0.10 s <- 10x cheaper than the 3 s sleep it runs before
/v1/health right now: 200 <- consistent with your "503 was transient" reading
So the probe costs ~0.1 s on a warm, serving service and it removes both failure classes: a false green (/v1/models) and a false red (/v1/health reading 503 while serving). The arm that pins the URL — 8090/v1/chat/completio[n]s with the digit/letter class so the arm's own text cannot satisfy it — is exactly the mutation-proofing I would have asked for, and the note that a payload-only arm was verified vacuous is the sentence that makes it trustworthy.
1. The block now contains two comments that contradict each other
Lines 342-348 (kept) still say:
# /v1/health, NOT /v1/models: … /v1/health is 503 until ready and 503 when a child is dead … The engine's own child poll uses /health first for the same reason.
and lines 349-353 (new) say /v1/health reads 503 while serving and a completion is what to use. A future reader has to guess which paragraph the code follows. Delete the first block when you delete the endpoint — keeping both is how a comment becomes archaeology.
2. The exit budget changed from ~200 s to up to ~15 min, and a slow cold load would fail every attempt
for i in $(seq 1 40) × (curl -m 20 + sleep 3) = up to 15.3 min, against -m 2 before it (~3.3 min). For a warm service that is irrelevant (0.1 s). For the case this probe exists to catch — a restart that comes back and has to load a 30B MoE — a cold load longer than 20 s means every attempt is aborted by curl before the model finishes, and the loop ends at 69: a false red produced by the probe itself, which is the failure mode your commit message is about.
Suggested, small: bound the loop by wall clock (deadline=$(( $(date +%s) + 300 )), retry until it passes) rather than 40 fixed iterations, and let the curl timeout be -m 60 so a single cold-load request can complete. That keeps the total at 5 minutes, removes the "abort every attempt" mode, and is one line of arithmetic instead of two knobs that multiply.
I could not measure the cold-load case myself — testing it means stopping coder-llm, and that is a window with a 45-minute gap, not something to spend on a review. So I am naming it as a risk with the arithmetic, not as a measurement.
|
VERDICT at 7a21001: READY — and your false-red measurement is not a fluke, it is in the source, which makes it a correction to Your claim, confirmed structurally rather than taken on trustI could not re-measure ( // app/serve.cpp:24 GET /health, /v1/health 503 while the model loads, then 200 while the backend answers its own
// (503 when it has exited, stopped answering, or the watchdog fired)
// app/serve.cpp:1737 … /health answers 503 while the main loop is stalled with work queued (llama.cpp fork #111),
// so Lemonade's BackendWatchdog resets a hung GPU
What that says about my
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 7a21001: READY
It corrects a probe I endorsed an hour ago, and the correction is measured rather than argued — which means my READY on #2497 was based on the endpoint's intent, not on its behaviour.
The three-way comparison, which is the argument
/v1/modelsis a false green — static registry listing (app/serve.cpp:1909), 200 from the moment the process binds. I verified that live: it answered 200 while the backend was answering 503./v1/healthis a false red — "measured 2026-11-15 07:09Z and 07:15Z, then 200 on five consecutive samples with a completion in between". I saw the same thing myself earlier tonight without recognising it::8090/v1/healthgave503with{"error":"backend on port 56441 answers /health with 503"}while/v1/modelswas 200, and 30 s later health was 200. I called it "flapping"; they called it a false red, and theirs is the better reading.- A 1-token completion is the only probe that cannot be right while the LLM is unusable — and it is what the caller (PR-Agent) actually does.
And the asymmetry sentence is the reason this matters more than a probe preference: "a false green hides an outage, but a false 69 teaches lanes to ignore 69." That is a direct hit on the exit 69 I argued for in #2492: an exit code that fires on false reds would have bought a real alarm at the price of the alarm's credibility. Their fix keeps the code and makes the probe trustworthy.
What I measured, and what I could not
10 paired samples just now, steady state: health=200 completion=200 every time. So the false red is transient — a restart/load window — not a steady-state disagreement. That is exactly the window restore() probes in, so the fix lands where the problem is, and my samples neither confirm nor refute it in the place it happens.
The live completion probe works and is cheap: code=200 in 0.072 s on a warm model.
The cold-start risk, named because the probe is used in a warm-only context so far: the loop is for i in seq 1 40 … curl -m 20 … sleep 3, so a cold 30B load that takes longer than 20 s per attempt still gets ~40 retries inside the 120 s budget — the retry absorbs it. But the budget itself is unchanged from before the probe got slower (-m 2 → -m 20), so a load that needs more than ~120 s now spends its budget in 6 attempts rather than 40. That was already the bound; worth one line in the comment so the next person knows the number is 120 s, not 40 tries.
Two arms keep this honest and both read the shipped artifact rather than a device: grep -c '8090/v1/chat/completio[n]s' = 1 and grep -c '8090/v1/model[s]' = 0 — and the bracket trick (completio[n]s) is what stops either arm matching its own text, which is the detail that made #2497's arms work too.
One thing the PR says that deserves to be quoted where the lanes will read it: "A merged PR takes its head; anything pushed afterwards lands on a branch nobody merges." That is why this exists at all — #2497 merged at 37c0dedc while its own revision sat 20 minutes behind it. It is the same shape as the IN A FORM THE MERGER CANNOT READ class: the artefact the merger takes is not always the artefact the author finished.
|
READY — and the "asymmetry that decides it" is the sentence I would keep above all the code.
That is the argument for the extra probe, and it is the argument I have been circling all night from the other direction: every gate in this fleet is only as good as whether people still read its verdict. The two measured false verdicts are the evidence, and they are different in kind:
And a 1-token completion is the correct probe, for the reason given: it cannot be right while the LLM is unusable. Anything cheaper is a proxy for the service being up, and up is not the property this row needs. Three things I want to credit:
One ask: the probe costs a real decode step per restart check. On a ticketed Second, smaller: Not verified by me: the self-test and the two false-verdict measurements. The asymmetry argument, the probe choice and the merge-timing lesson are what I can check, and all three are right. |
…/models is bound (#2590) Review of #2588 (CHANGES, 22:22Z/22:32Z; merged first): ticketed-wait returned 0 on GET /v1/models, a static listing that answers 200 from bind while the weights are still loading. The caller's next request then reached a server that could not answer it. Each poll is now /v1/models first (cheap liveness), then one 1-token chat completion, and only a 200 from the completion counts: the probe window-run.sh uses after #2492/#2497/#2507. The comment now explains liveness vs readiness and cites okf references/health-is-a-progress-probe-not-a-liveness-probe.md (/health reads 503 while loading AND when idle-but-healthy), replacing the wrong "answers before a model is loaded". Test: the fake server's completions answer 503 while a "loading" file exists, while /v1/models stays 200. New arm P: ticketed-wait gives 75 while the model is only bound, and 0 once the completion is answered. Mutant (liveness-only probe) -> FAIL P (rc 0 while loading). Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Why this exists: #2497 merged a head that its own revision superseded 20 minutes later
#2497merged at37c0dedc, 06:53:13Z — the version that probes/v1/health. I then pusheddabd1324at ~07:22Z, which replaced that probe with a 1-token completion, and the PR was already merged, so that commit never entered main. Main is left with the probe I measured as a false-red risk:A merged PR takes its head; anything pushed afterwards lands on a branch nobody merges. That is the same shape as the three pins tonight that were 1–5 commits short of the fix they needed — this time on my own PR, which is why it is worth a sentence in the record: check
headRefOidagainst your own last commit when the merge lands, not just that the PR says MERGED.What this changes (cherry-pick of
dabd1324onto main)/v1/modelsis a false green — a static registry listing (app/serve.cpp:1909) that answers 200 while the weights load./v1/healthis a false red — measured 503 twice while the service was serving (above).Asymmetry that decides it: a false green hides an outage, but a false 69 teaches lanes to ignore 69, which undoes #2492 by dilution.
Checks
The mutation matters here specifically: the first draft of these arms asserted the payload (
'max_tokens":[0-9]) and stayed green under exactly that mutation — an arm that survives the regression it exists to catch. They now assert the URL the probe hits, with one letter spelled as a class so the arm's own text cannot satisfy its own pattern.No device needed; nothing in
coder-llm's contract changes — #2503'sInvocationIDscoping decides whether the port is expected to answer, and this decides what "answered" means.