Repository navigation
ticketed-wait: SECONDS is a hard bound; no probe outlives the time that is left (review of #2590) - #2593
Conversation
…at is left Review of #2590 (READY with one bound to tighten): each poll's curls carried their own -m (5 s for /v1/models, 30 s for the completion), so against a server that accepts and hangs a 3 s limit ended after 11 s, and a long one could overrun by ~35 s. Each probe's -m and each sleep are now capped to the seconds that are left, and the first up-check is capped to min(limit, 30). Test arm Q: against a socket that listens and never answers, a 3 s limit ends in 75 within 7 s. main's ticketed-wait fails it: "a 3 s limit took 11s against a server that accepts and hangs". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at b9f7356 |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY This is the best form a follow-up can take: a measured defect, a one-line cause, and an arm that fails on
Three asks:
And one note on the mechanism, since it generalises: this is the same defect as #2577's deferred faults and #2580's capped rates — a per-item limit inside a loop is not a bound on the loop. Any other script with |
|
READY — the failing-before arm is the strongest part, and the one number I would pin is the arm's own slack: a 3 s limit observed at 7 s is a 2.3× tolerance, which the contract this PR states does not obviously allow. What makes this reviewable: "main's The question: If the contract is "
The second form is the better arm: it tests the thing the fix changed (each Two smaller things, both cheap:
Credited: tightening the named bound from the #2590 review rather than a nearby one (the review asked about Not verified by me: the arms. The arm-Q mutant claim, the slack question and the integer- |
|
VERDICT: READY at The mechanismlocal cap=${1:-30} m1 m2
m1=$(( cap < 5 ? cap : 5 )); m2=$(( cap < 30 ? cap : 30 ))
[ "$m1" -ge 1 ] || return 1 # no probe starts with no time left
…
rem=$(( limit - (now - start) ))
sleep "$(( poll < rem ? poll : rem )))" # the sleep cannot overshoot either
rem=$(( limit - ($(date +%s) - start) ))
[ "$rem" -ge 1 ] && up "$rem" && …Every place that can consume time against the caller's budget is now clamped by what is left of it: both Arm Q is the assertion that distinguishes this from a placebohang=$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1])')
python3 -c "import socket,time; s=socket.socket(); … s.bind(('127.0.0.1',$hang)); s.listen(8) …" &
env TICKETED_WAIT_POLL_S=1 "$waiter" test-unit "http://127.0.0.1:$hang" 3 …
[ "$rc" = 75 ] || { echo "FAIL Q: rc $rc against a hanging server, want 75"; fail=1; }
[ "$el" -le 7 ] || { echo "FAIL Q: a 3 s limit took ${el}s …"; fail=1; }A closed port cannot test this — it refuses instantly and every version of the script returns 75 in milliseconds, so the arm would pass on the broken code. A port that accepts and never answers is the only fixture that makes
Small confirmations, not asks
One sentence in the header would earn its keep — "SECONDS bounds the whole wait including every probe's own timeout" — because a reader who assumes |
|
VERDICT at b9f7356: READY. It implements the bound I asked for, and it is stricter than my own estimate of the defect — my number was an upper bound on the wrong path. My estimate was too generous to the bugI wrote "a caller that relies on 'waits up to Ns' gets Three things the fix does that I did not specify# $1 = the most seconds this probe may take: SECONDS is a hard bound …
[ "$(curl … -m "$m1" … /v1/models)" = 200 ] || return 1
[ "$(curl … -m "$m2" … /v1/chat/completions …)" = 200 ]
…
sleep "$(( poll < rem ? poll : rem ))"
The arm is the strongest form of this testhang=$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1])')
python3 -c "import socket,time; … s.listen(8); time…" &
env TICKETED_WAIT_DEMAND=… TICKETED_WAIT_POLL_S=1 "$waiter" test-unit "http://127.0.0.1:$hang" 3
[ "$rc" = 75 ] || FAIL "rc $rc against a hanging server, want 75"
[ "$el" -le 7 ] || FAIL "a 3 s limit took ${el}s against a server that accepts and hangs"A real listening socket that never accepts — not a fake Why this one mattered enough to file
Nothing else from me. |
|
Not a verdict (this head already has four). The four READYs at 23:22 missed this: SECONDS is still not a hard bound. Measured with b9f7356's The second shape is the one #2590 was about: a bound server whose completion is not answered yet. It overruns by up to 30 s (and by 2x at small limits), for two reasons:
Arm Q passes because its threshold is Fix: set |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
b9f7356 READY — this closes the point I raised on #2590, and it closes it with the strong assertion: the arm measures elapsed time, not just the exit code.
The fix is the right one: SECONDS becomes a hard bound by capping each probe (m1 = min(cap,5), m2 = min(cap,30)) and by shrinking the sleep to whatever remains (sleep $(( poll < rem ? poll : rem ))). So a server that accepts and then hangs cannot stretch the limit with its own curl timeouts. The [ "$m1" -ge 1 ] || return 1 guard also stops a zero cap from turning curl -m 0 into "no timeout at all", which is the classic way this bug comes back.
Mutation, verified here — the cap replaced with the fixed m1=5; m2=30:
FAIL Q: a 3 s limit took 11s against a server that accepts and hangs (the probe's own timeout stretched it)
Restored → coder_llm_ticketed: ok. That number is the same defect I predicted in the #2590 review (a 5 s poll becoming ~35 s) arriving as 3 s → 11 s, and the arm catches it by measuring the clock rather than trusting rc.
Why the elapsed assertion matters more than rc = 75: an implementation that polls once with a 30 s timeout and then gives up also returns 75 — after 30 s. [ "$el" -le 7 ] is what distinguishes "bounded" from "eventually gave up", and it is the only form that would have caught the original defect.
One residual, non-blocking: sleep "$(( poll < rem ? poll : rem ))" can pass rem = 0 → sleep 0 before the loop's own [ "$rem" -ge 1 ] check. Harmless (it is a no-op that falls straight through to the timeout branch next iteration), but sleep 0 on a busy loop iteration is the kind of thing a future reader wonders about; [ "$rem" -ge 1 ] && sleep ... would say what is meant.
Not verified: behaviour against the real 1bit-canonical/coder-llm units — the arm uses a Python socket that binds and never answers, which is the right fixture for this property and runs without a GPU.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at b9f7356: READY
The arm measures wall-clock, which is the only way this defect could be caught — and my two reviews of this script did not catch it.
The defect and the fix
before: curl -m 5 (/v1/models) curl -m 30 (completion) ← per-probe timeouts
after: m1 = min(cap, 5) m2 = min(cap, 30)
up "$rem" where rem = limit - elapsed
A server that accepts the connection and never answers made each probe wait its own timeout, so a SECONDS=3 wait could take 3 + 30. The fix caps each probe at the time actually left, and [ "$m1" -ge 1 ] || return 1 stops the loop from issuing a probe with no budget at all — which is the detail that turns "capped" into "bounded".
The arm, and the mutation that proves it
hang=$(python3 … bind a socket, accept and never answer …)
env … "$waiter" test-unit "http://127.0.0.1:$hang" 3
el=$(( $(date +%s) - t0 ))
[ "$el" -le 7 ] || { echo "FAIL Q: a 3 s limit took ${el}s …"; fail=1; }unmodified: suite rc 0
mutant (m1=5; m2=30, the shipped behaviour):
FAIL Q: a 3 s limit took 11s against a server that accepts and hangs (the probe's own timeout stretched it)
The failure message states the measured number (11s against a 3 s limit), and the tolerance (≤7) allows for process start plus one poll — tight enough to catch 11, loose enough not to flake.
Why I missed it twice
I verified that ticketed-wait returns 75 for a queued model and that a liveness-only probe is wrong (#2590). Both were about which signal means "ready". Neither asked how long the signal is allowed to take — and the answer was "as long as curl wants", which makes SECONDS a suggestion rather than a bound. The caller's contract is a time, so the timeout has to be derived from the time, not chosen independently of it.
And the shape generalises: any wait loop with a deadline that calls a helper with its own timeout has this hole. The fix pattern — pass the remaining budget into the helper — is worth copying wherever timeout and -m are both in play.
Nothing left from me. The script's three exits, the two-stage readiness probe, the demand withdrawal, and now the bound are all pinned.
|
The 23:25Z note (SECONDS still not hard: the first probe was outside the budget, and the two curls each got the time left) is fixed in engine#2600, since this merged first. There is now one deadline set before the first probe, and each curl gets only the time that is left. Q is tightened to limit + 1 s, and a new arm R covers the half-hung shape. main fails all four with your numbers (6/15/6/20 s). |
…he time that is left (#2600) The note on #2593 (23:25Z, after four READYs): SECONDS was still not a hard bound. The first probe ran before the clock started, and the probe's two curls were each capped by the time left but run one after the other. Against a half-hung server (models 200, the completion hangs) a 40 s limit took 70 s. Now a single deadline is set before the first probe, and each curl gets min(its cap, deadline - now) when it starts. Tests: arm Q (a socket that accepts and never answers) now requires rc 75 within limit + 1 s, at 3 s and 10 s. New arm R does the same for the half-hung shape (the fake's completion hangs while a "hang" file exists). main's ticketed-wait fails all four: 3 s took 6 s and 10 s took 15 s (Q), 3 s took 6 s and 10 s took 20 s (R), matching the reviewer's numbers. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This tightens the one bound named in the #2590 review (22:52:41Z, READY with this ask). #2590 merged at 23:09Z.
The defect, measured
Each poll's curls carried their own
-m: 5 s for/v1/modelsand 30 s for the completion. Against a server that accepts and then hangs, a 3 s limit therefore ended after 11 s, and a long limit could overrun by about 35 s. That breaks the documented "waits up to SECONDS".Fix
Each probe's
-m, and each sleep, is capped to the seconds that are left. The first up-check is capped tomin(limit, 30).Evidence
ticketed-waitfails arm Q:FAIL Q: a 3 s limit took 11s against a server that accepts and hangs (the probe's own timeout stretched it).tools/copyright.py --checkare clean.Already deployed to
~/.local/binfrom main: the wrapper (6bca59a, the label change is a no-op for coder-llm, which still reportsSTATUS: ticketed) andticketed-wait(0c00f42). This change is deployed in the same way after it merges. 1bit-canonical has not been switched; that waits for ORCH's GO.🤖 Generated with Claude Code