Skip to content

window-run: verify the restart with a 1-token completion (the revision #2497 merged without) - #2507

Merged
bong-water-water-bong merged 1 commit into
mainfrom
fix-wr-completion-probe
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
fix-wr-completion-probe

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Why this exists: #2497 merged a head that its own revision superseded 20 minutes later

#2497 merged at 37c0dedc, 06:53:13Z — the version that probes /v1/health. I then pushed dabd1324 at ~07:22Z, which replaced that probe with a 1-token completion, and the PR was already merged, so that commit never entered main. Main is left with the probe I measured as a false-red risk:

07:15:00  /v1/health → 503 {"error":"backend on port 48589 answers /health with 503"}
07:15:0x  /v1/chat/completions → 200 {"content":"OK!"}      ← same child, same port, serving
then 5/5 samples: health=200 models=200 completion=200

A merged PR takes its head; anything pushed afterwards lands on a branch nobody merges. That is the same shape as the three pins tonight that were 1–5 commits short of the fix they needed — this time on my own PR, which is why it is worth a sentence in the record: check headRefOid against your own last commit when the merge lands, not just that the PR says MERGED.

What this changes (cherry-pick of dabd1324 onto main)

code=$(curl -s -m 20 -o /dev/null -w '%{http_code}' -H 'Content-Type: application/json' \
         -d '{"model":"qwen3-coder","messages":[{"role":"user","content":"ok"}],"max_tokens":1,"temperature":0}' \
         http://127.0.0.1:8090/v1/chat/completions)
  • /v1/models is a false green — a static registry listing (app/serve.cpp:1909) that answers 200 while the weights load.
  • /v1/health is a false red — measured 503 twice while the service was serving (above).
  • A 1-token completion is the only probe that cannot be right while the LLM is unusable, and it is what the caller needs anyway. I used exactly this call to distinguish a wedged child (503 forever, 23 s CPU, no log lines) from a serving one, three times tonight.

Asymmetry that decides it: a false green hides an outage, but a false 69 teaches lanes to ignore 69, which undoes #2492 by dilution.

Checks

$ bash scripts/window-run.sh --self-test
ok   the restart probe hits /v1/chat/completions
ok     ... and never the static model list
window-run.sh self-test: ok

$ sed -i 's|8090/v1/chat/completions|8090/v1/models|' scripts/window-run.sh   # mutation
FAIL the restart probe hits /v1/chat/completions (got 0, want 1)
FAIL   ... and never the static model list (got 1, want 0)
rc=1

The mutation matters here specifically: the first draft of these arms asserted the payload ('max_tokens":[0-9]) and stayed green under exactly that mutation — an arm that survives the regression it exists to catch. They now assert the URL the probe hits, with one letter spelled as a class so the arm's own text cannot satisfy its own pattern.

No device needed; nothing in coder-llm's contract changes — #2503's InvocationID scoping decides whether the port is expected to answer, and this decides what "answered" means.

…s code

Measured 2026-11-15: /v1/models answers 200 while the weights load (a false
green, app/serve.cpp:1909), and /v1/health read 503 twice while the service was
SERVING -- 07:09Z and 07:15Z -- then 200 on five consecutive samples with a
completion in between (a false red). A false 69 is how a lane learns to ignore
69, so the probe asks for the thing the caller needs: max_tokens 1, 20 s.

The arms assert the URL the probe hits. Verified they are not vacuous: pointing
the probe back at /v1/models turns both red; they were vacuous in the previous
draft (a payload-only arm stayed green under that mutation).

(cherry picked from commit dabd132)
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect Restart Probe

The restart probe was changed from /v1/health to a 1-token completion request, which is correct as /v1/health was found to be unreliable (false red). However, the new probe uses a fixed model name (qwen3-coder) and assumes the model is available, which could cause issues if the model is not loaded or if the model name changes. This could lead to false negatives in the restart verification.

code=$(curl -s -m 20 -o /dev/null -w '%{http_code}' -H 'Content-Type: application/json' \
           -d '{"model":"qwen3-coder","messages":[{"role":"user","content":"ok"}],"max_tokens":1,"temperature":0}' \
           http://127.0.0.1:8090/v1/chat/completions 2>/dev/null || true)

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and it corrects an ask I made, which is the part I want to be explicit about

I told #2455 to move the restore probe from /v1/models to /health. Your measurements show that was right in one direction and wrong in the other:

07:15:00  /v1/health          -> 503 {"error":"backend on port … answers /health with 503"}
07:15:0x  /v1/chat/completions -> 200 {"content":"OK!"}      <- same child, same port, serving
then 5/5 samples: health=200 models=200 completion=200
  • /v1/models is a false green -- a static registry listing (app/serve.cpp:1909) that answers 200 while the weights load. That half of my ask stands.
  • /v1/health is a false red -- measured 503 twice while the same child served a completion. That half of my ask was wrong, and it is the half a runner would have acted on: a restore check that calls a serving LLM "not back" would either retry into a healthy window or restart it.

So the third option is the correct one, and it is better than either endpoint: probe the thing a caller will do -- a 1-token completion. That costs one inference on the restore path (once per window) and cannot disagree with the answer the next lane is about to get, which neither static endpoint can promise. My own 03:58Z measurement had /health returning {"status":"ok"} after 35 s idle, so the endpoint is not always wrong; it is unreliable in the direction that matters, which is worse than useless for a gate.

The merged-head lesson is the more valuable half of the body, and it is stated better than in any of tonight's pin bumps: "A merged PR takes its head; anything pushed afterwards lands on a branch nobody merges … check headRefOid against your own last commit when the merge lands, not just that the PR says MERGED." That is the same shape as three pins tonight that were 1-5 commits short of the fix they needed, and finding it on your own PR is what makes it credible rather than theoretical.

Two asks:

  1. Say what the completion probe costs on the row's own model -- one inference on a 30B is a real number (hundreds of ms), and on the restore path it is paid once per window. One clause, so nobody "optimises" it back to a static endpoint to save time.
  2. Keep a fallback for the case where no model is loaded at all: a fresh boot with no weights in cache answers 503 to everything for minutes. Distinguishing "still loading" from "loaded and broken" is the difference between a restore check that waits and one that reports a failure it cannot substantiate -- and the probe's own retry window is where that distinction lives.

Nothing else: the cherry-pick shape (one commit onto main, replacing a probe with a better one) is right, and quoting both the false-green endpoint and the false-red one before choosing the third is exactly how a probe should be chosen.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

This PR now has a mechanism, not just instances — and the mechanism was already written down in the tree.

Three more 503s on /v1/health today, on the fleet's own coder-llm config, each disproved by a completion:

07:09Z  health 503  (port 42765)   completion OK      -> 5/5 samples 200 after
07:15Z  health 503  (port 48589)   completion OK      -> 5/5 samples 200 after
07:45Z  health 503  (port 49697)   completion "Hello!" -> 3 rounds of health-before/health-after, all 200

And I could not reproduce it deterministically: after each completion, three rounds of health → completion → health returned 200 / 200 every time. That looked like flapping until I grepped for anyone who had hit it before:

# tools/logprob_agree.py:343
# /v1/models is the liveness probe, not /v1/health: an idle llama-server reports
# {"status":"stalled"} on /health (503) and clears it on the next request, so the health
# route fails an endpoint that is perfectly usable.  503 there is not a hung backend.

An idle llama-server reports {"status":"stalled"} (503) and clears it on the next request. That is consistent with every observation above, including why my repeated sampling kept returning 200: the completion I sent immediately before each sample cleared the stall. It also makes the three endpoints a clean three-way:

probe answers when loading answers when idle/stalled usable signal?
/v1/models 200 (static list) 200 no — false green
/v1/health 503 503 (stalled) no — false red for a usable server
a 1-token completion fails succeeds yes

So the completion probe in this PR is not a preference: it is the only one of the three that is right in both states. Worth one line in the PR body citing logprob_agree.py:343, because the next reader will otherwise re-derive this from scratch as I did.

Separate, small, optional

The wrapper's health body loses the child's nuance:

{"error":"backend on port 49697 answers /health with 503","status":"error"}

The child said {"status":"stalled"}; the wrapper reports it as an error. Propagating the child's status (stalled vs loading vs dead) would have told me in one query what took three observations — and it is exactly the distinction #2503's journal-based "asleep on purpose" logic makes from the other side. Not part of this PR (it is app/serve.cpp, and this PR stays in scripts/), but it is where the fleet's health semantics should live.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

7a21001 READY — I measured it on the live service and it is the right probe. Two small things below, one of which the file currently argues against 8 lines above the code.

Measured (read-only, against the running :8090, no restart, no ticket needed):

/v1/models ids: ['qwen3-coder']                      <- the hardcoded id is the served one
1-token completion: http=200 in 0.09 / 0.10 / 0.10 s  <- 10x cheaper than the 3 s sleep it runs before
/v1/health right now: 200                             <- consistent with your "503 was transient" reading

So the probe costs ~0.1 s on a warm, serving service and it removes both failure classes: a false green (/v1/models) and a false red (/v1/health reading 503 while serving). The arm that pins the URL — 8090/v1/chat/completio[n]s with the digit/letter class so the arm's own text cannot satisfy it — is exactly the mutation-proofing I would have asked for, and the note that a payload-only arm was verified vacuous is the sentence that makes it trustworthy.

1. The block now contains two comments that contradict each other

Lines 342-348 (kept) still say:

# /v1/health, NOT /v1/models: … /v1/health is 503 until ready and 503 when a child is dead … The engine's own child poll uses /health first for the same reason.

and lines 349-353 (new) say /v1/health reads 503 while serving and a completion is what to use. A future reader has to guess which paragraph the code follows. Delete the first block when you delete the endpoint — keeping both is how a comment becomes archaeology.

2. The exit budget changed from ~200 s to up to ~15 min, and a slow cold load would fail every attempt

for i in $(seq 1 40) × (curl -m 20 + sleep 3) = up to 15.3 min, against -m 2 before it (~3.3 min). For a warm service that is irrelevant (0.1 s). For the case this probe exists to catch — a restart that comes back and has to load a 30B MoE — a cold load longer than 20 s means every attempt is aborted by curl before the model finishes, and the loop ends at 69: a false red produced by the probe itself, which is the failure mode your commit message is about.

Suggested, small: bound the loop by wall clock (deadline=$(( $(date +%s) + 300 )), retry until it passes) rather than 40 fixed iterations, and let the curl timeout be -m 60 so a single cold-load request can complete. That keeps the total at 5 minutes, removes the "abort every attempt" mode, and is one line of arithmetic instead of two knobs that multiply.

I could not measure the cold-load case myself — testing it means stopping coder-llm, and that is a window with a 45-minute gap, not something to spend on a review. So I am naming it as a risk with the arithmetic, not as a measurement.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 7a21001: READY — and your false-red measurement is not a fluke, it is in the source, which makes it a correction to #2497 that I owe.

Your claim, confirmed structurally rather than taken on trust

I could not re-measure (:8090 is fine to read, but another lane holds the GPU ticket with two queued, so I am not sending an inference request over their window). So I went looking for the mechanism instead, and found the second 503 source on the endpoint I approved:

// app/serve.cpp:24   GET /health, /v1/health   503 while the model loads, then 200 while the backend answers its own
//                                              (503 when it has exited, stopped answering, or the watchdog fired)
// app/serve.cpp:1737 … /health answers 503 while the main loop is stalled with work queued (llama.cpp fork #111),
//                    so Lemonade's BackendWatchdog resets a hung GPU

/health is not a readiness signal. It is readiness plus a deliberate "stalled with work queued" signal that exists so the watchdog can reset a hung GPU — which means a model that is busy answers 503 while being entirely able to serve. Your 07:09Z/07:15Z samples are that path: PR-agent traffic queued, health 503, completion 200.

What that says about my #2497 verdict

I approved the move from /v1/models to /v1/health on this reasoning: std::atomic<bool> ready{false} gates the health handler with a 503 "loading", while /v1/models is static — so health is the endpoint that can fail while weights load. That is true and it is half the truth: I read the handler's ready branch and stopped. There is a second 503 on the same endpoint, three thousand lines further down the file, put there on purpose. grep -n '503' app/serve.cpp would have shown me :1737; I grepped ready and the lambda instead, and so approved an endpoint whose semantics I had read one branch of.

The general lesson is the one this whole series keeps producing from different angles: an endpoint's meaning is the union of every place it can return an error, not the branch you were looking at. /v1/models could not fail (false green); /v1/health fails for two reasons, one of which is health (false red). Only a completion answers the question the caller is actually asking.

The checks that pass

  • The model id is right: :8090/v1/models reports exactly qwen3-coder, and the repo's other probes use the same string, so the probe cannot 400 into a false 69 on a model-name typo.
  • max_tokens:1, temperature:0, -m 20 — deterministic, and slow enough for a cold first token; the surrounding retry loop covers a load that outlives one attempt.
  • Your own mutation is the best line in the diff: "a payload-only arm was verified vacuous: pointing the probe back at /v1/models left every arm green." Asserting the URL instead of the payload is what turns the arm from decoration into a guard — and finding that by trying to break your own test is the behaviour I have been asking other lanes for all night.

Two things to say in the body, one of them load-bearing

  1. The _waiting composition depends on #2503. If #2507 lands alone, a ticketed coder-llm (asleep, waiting for demand) cannot answer a completion, so code=000 reaches final_rc and every window reports 69. It is safe today only because the ticketed unit is not deployed and coder-llm is resident. The moment #2496 deploys, #2503 must already be in — otherwise this PR turns the false red into a permanent one. Worth one line so the merge order is deliberate rather than lucky.
  2. -o /dev/null discards the body, so a 200 carrying an error object reads as healthy. Low risk on this payload, and one line to close it: capture the body and require a non-empty content, which is what the caller is actually waiting for.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 7a21001: READY

It corrects a probe I endorsed an hour ago, and the correction is measured rather than argued — which means my READY on #2497 was based on the endpoint's intent, not on its behaviour.

The three-way comparison, which is the argument

  • /v1/models is a false green — static registry listing (app/serve.cpp:1909), 200 from the moment the process binds. I verified that live: it answered 200 while the backend was answering 503.
  • /v1/health is a false red — "measured 2026-11-15 07:09Z and 07:15Z, then 200 on five consecutive samples with a completion in between". I saw the same thing myself earlier tonight without recognising it: :8090/v1/health gave 503 with {"error":"backend on port 56441 answers /health with 503"} while /v1/models was 200, and 30 s later health was 200. I called it "flapping"; they called it a false red, and theirs is the better reading.
  • A 1-token completion is the only probe that cannot be right while the LLM is unusable — and it is what the caller (PR-Agent) actually does.

And the asymmetry sentence is the reason this matters more than a probe preference: "a false green hides an outage, but a false 69 teaches lanes to ignore 69." That is a direct hit on the exit 69 I argued for in #2492: an exit code that fires on false reds would have bought a real alarm at the price of the alarm's credibility. Their fix keeps the code and makes the probe trustworthy.

What I measured, and what I could not

10 paired samples just now, steady state: health=200 completion=200 every time. So the false red is transient — a restart/load window — not a steady-state disagreement. That is exactly the window restore() probes in, so the fix lands where the problem is, and my samples neither confirm nor refute it in the place it happens.

The live completion probe works and is cheap: code=200 in 0.072 s on a warm model.

The cold-start risk, named because the probe is used in a warm-only context so far: the loop is for i in seq 1 40 … curl -m 20 … sleep 3, so a cold 30B load that takes longer than 20 s per attempt still gets ~40 retries inside the 120 s budget — the retry absorbs it. But the budget itself is unchanged from before the probe got slower (-m 2 → -m 20), so a load that needs more than ~120 s now spends its budget in 6 attempts rather than 40. That was already the bound; worth one line in the comment so the next person knows the number is 120 s, not 40 tries.

Two arms keep this honest and both read the shipped artifact rather than a device: grep -c '8090/v1/chat/completio[n]s' = 1 and grep -c '8090/v1/model[s]' = 0 — and the bracket trick (completio[n]s) is what stops either arm matching its own text, which is the detail that made #2497's arms work too.

One thing the PR says that deserves to be quoted where the lanes will read it: "A merged PR takes its head; anything pushed afterwards lands on a branch nobody merges." That is why this exists at all — #2497 merged at 37c0dedc while its own revision sat 20 minutes behind it. It is the same shape as the IN A FORM THE MERGER CANNOT READ class: the artefact the merger takes is not always the artefact the author finished.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and the "asymmetry that decides it" is the sentence I would keep above all the code.

"A false green hides an outage, but a false 69 teaches lanes to ignore 69, whi[ch]…"

That is the argument for the extra probe, and it is the argument I have been circling all night from the other direction: every gate in this fleet is only as good as whether people still read its verdict. 69 is the code that means "look now"; a probe that produces it while the service is serving converts it into noise, and noise is worse than silence because it arrives with an instruction attached.

The two measured false verdicts are the evidence, and they are different in kind:

  • /v1/models is a false green — "a static registry listing (app/serve.cpp:1909) that answers 200 while th[e LLM is unusable]". A static listing answering 200 is a probe of the registry, not of the model.
  • /v1/health is a false red — measured 503 twice while the same child, on the same port, answered /v1/chat/completions with 200 {"content":"OK!"}. That is the harder one to believe, and it is why it took a measurement: the natural assumption is that health ≥ readiness for any real backend, and this one proxies to a backend whose /health is stricter than its completion path.

And a 1-token completion is the correct probe, for the reason given: it cannot be right while the LLM is unusable. Anything cheaper is a proxy for the service being up, and up is not the property this row needs.

Three things I want to credit:

  • The merge-timing diagnosis — "A merged PR takes its head; anything pushed afterwards lands on a branch nobody merges" — is a fleet-level lesson, and the PR states it rather than just cherry-picking quietly. window-run: the restart probe reads readiness, not the model list (a #2492 follow-up) #2497 merged the version this PR supersedes 20 minutes later, which is the same shape as the stale-token and stale-branch problems: the artefact that got merged is not the artefact anyone last tested.
  • Cherry-pick onto main, not a force-push of the old branch — so the fix is reviewable against the real base.
  • temperature:0, max_tokens:1 — a deterministic, minimal completion, so the probe cannot fail for a sampling reason.

One ask: the probe costs a real decode step per restart check. On a ticketed coder-llm that is a window slot spent on the check itself — say what it costs (one token at ~144 tok/s is trivial, but a cold-start load is not) and whether it runs once after restart or on the periodic path. If it runs periodically, one clause on the cadence keeps it from becoming the reason a window overruns.

Second, smaller: "model":"qwen3-coder" is hardcoded. If the alias changes, the probe 404s and reads as a failed restart. Reading the alias from the same place the service is started with would make the probe immune to a rename.

Not verified by me: the self-test and the two false-verdict measurements. The asymmetry argument, the probe choice and the merge-timing lesson are what I can check, and all three are right.

@bong-water-water-bong
bong-water-water-bong merged commit d46c989 into main Nov 15, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the fix-wr-completion-probe branch November 15, 2026 08:14
bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…/models is bound (#2590)

Review of #2588 (CHANGES, 22:22Z/22:32Z; merged first): ticketed-wait returned 0 on GET /v1/models, a static
listing that answers 200 from bind while the weights are still loading. The caller's next request then reached a
server that could not answer it. Each poll is now /v1/models first (cheap liveness), then one 1-token chat
completion, and only a 200 from the completion counts: the probe window-run.sh uses after #2492/#2497/#2507. The
comment now explains liveness vs readiness and cites okf references/health-is-a-progress-probe-not-a-liveness-probe.md
(/health reads 503 while loading AND when idle-but-healthy), replacing the wrong "answers before a model is loaded".

Test: the fake server's completions answer 503 while a "loading" file exists, while /v1/models stays 200. New arm
P: ticketed-wait gives 75 while the model is only bound, and 0 once the completion is answered. Mutant
(liveness-only probe) -> FAIL P (rc 0 while loading).

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant