Skip to content

ticketed-wait: SECONDS is a hard bound; no probe outlives the time that is left (review of #2590) - #2593

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-infra/ticketed-wait-bound
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-infra/ticketed-wait-bound

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

This tightens the one bound named in the #2590 review (22:52:41Z, READY with this ask). #2590 merged at 23:09Z.

The defect, measured

Each poll's curls carried their own -m: 5 s for /v1/models and 30 s for the completion. Against a server that accepts and then hangs, a 3 s limit therefore ended after 11 s, and a long limit could overrun by about 35 s. That breaks the documented "waits up to SECONDS".

Fix

Each probe's -m, and each sleep, is capped to the seconds that are left. The first up-check is capped to min(limit, 30).

Evidence

  • New arm Q: against a socket that listens and never answers, a 3 s limit ends in 75 within 7 s.
  • main's ticketed-wait fails arm Q: FAIL Q: a 3 s limit took 11s against a server that accepts and hangs (the probe's own timeout stretched it).
  • Arms A–Q pass under thermal-run. shellcheck and tools/copyright.py --check are clean.

Already deployed to ~/.local/bin from main: the wrapper (6bca59a, the label change is a no-op for coder-llm, which still reports STATUS: ticketed) and ticketed-wait (0c00f42). This change is deployed in the same way after it merges. 1bit-canonical has not been switched; that waits for ORCH's GO.

🤖 Generated with Claude Code

…at is left

Review of #2590 (READY with one bound to tighten): each poll's curls carried their own -m (5 s for /v1/models,
30 s for the completion), so against a server that accepts and hangs a 3 s limit ended after 11 s, and a long one
could overrun by ~35 s. Each probe's -m and each sleep are now capped to the seconds that are left, and the first
up-check is capped to min(limit, 30).

Test arm Q: against a socket that listens and never answers, a 3 s limit ends in 75 within 7 s. main's
ticketed-wait fails it: "a 3 s limit took 11s against a server that accepts and hangs".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at b9f7356

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Timeout Capping Logic

The timeout capping logic in the up function and sleep logic may not correctly handle cases where the remaining time is less than 1 second, potentially causing the wait loop to exit prematurely or not respect the hard bound on SECONDS.

    local cap=${1:-30} m1 m2
    m1=$(( cap < 5 ? cap : 5 )); m2=$(( cap < 30 ? cap : 30 ))
    [ "$m1" -ge 1 ] || return 1
    [ "$(curl -s -o /dev/null -m "$m1" -w '%{http_code}' "${url%/}/v1/models" 2> /dev/null)" = 200 ] || return 1
    [ "$(curl -s -o /dev/null -m "$m2" -w '%{http_code}' -H 'Content-Type: application/json' \
         -d '{"model":"'"$model"'","messages":[{"role":"user","content":"ok"}],"max_tokens":1,"temperature":0}' \
         "${url%/}/v1/chat/completions" 2> /dev/null)" = 200 ]
}
up "$(( limit < 30 ? (limit < 1 ? 1 : limit) : 30 ))" && exit 0
mkdir -p "$(dirname "$demand")" && touch "$demand" || { echo "ticketed-wait: cannot touch $demand" >&2; exit 2; }
start=$(date +%s); touched=$start
echo "ticketed-wait: asked for $unit (touched $demand); waiting up to ${limit}s for $url; GPU token: $(head -c 80 "${GPU_TOKEN_FILE:-$HOME/GPU-TOKEN}" 2> /dev/null | head -1)" >&2
while :; do
    now=$(date +%s)
    if [ $(( now - start )) -ge "$limit" ]; then
        echo "ticketed-wait: $unit did not answer in ${limit}s (still queued for the GPU?): not run; demand withdrawn" >&2
        rm -f "$demand"
        exit 75
    fi
    [ $(( now - touched )) -ge 30 ] && { touch "$demand"; touched=$now; }
    rem=$(( limit - (now - start) ))
    sleep "$(( poll < rem ? poll : rem ))"
    rem=$(( limit - ($(date +%s) - start) ))
    [ "$rem" -ge 1 ] && up "$rem" && { echo "ticketed-wait: $unit answers after $(( $(date +%s) - start ))s" >&2; touch "$demand"; exit 0; }
Test Coverage

The new test case Q verifies the hard bound behavior, but it relies on a specific timing assumption that may not be robust across different hardware or load conditions.

# ---- Q: the limit holds against a server that accepts and hangs -------------------------------------------------------
hang=$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1])')
python3 -c "import socket,time; s=socket.socket(); s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1); s.bind(('127.0.0.1',$hang)); s.listen(8); time.sleep(120)" & hq=$!; pids+=($hq)
sleep 1; t0=$(date +%s)
env TICKETED_WAIT_DEMAND="$scratch/q.demand" TICKETED_WAIT_POLL_S=1 "$waiter" test-unit "http://127.0.0.1:$hang" 3 2> "$scratch/q.wait"; rc=$?
el=$(( $(date +%s) - t0 ))
[ "$rc" = 75 ] || { echo "FAIL Q: rc $rc against a hanging server, want 75"; fail=1; }
[ "$el" -le 7 ] || { echo "FAIL Q: a 3 s limit took ${el}s against a server that accepts and hangs (the probe's own timeout stretched it)"; fail=1; }
kill "$hq" 2> /dev/null; wait "$hq" 2> /dev/null

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2593

This is the best form a follow-up can take: a measured defect, a one-line cause, and an arm that fails on main.

  • measured: "Each poll's curls carried their own -m: 5 s for /v1/models and 30 s for the completion. Against a server that accepts and then hangs, a 3 s limit therefore ended after 11 s, and a long limit could overrun by about 35 s" — a bound that is documented as "up to SECONDS" and actually takes up to SECONDS + 35;
  • the cause is exactly the shape of the bug: the timeout was per-probe, so a loop of probes has no bound at all. Capping each probe and each sleep to what is left is the correct fix, because it converts a sum of unbounded terms into a budget;
  • and the evidence is a negative control: "main's ticketed-wait fails arm Q" with the message quoting the defect. That is a test that could not have passed before the fix, which is the only kind that proves one.

Three asks:

  1. put the measured overrun in the arm's comment — a 3 s limit took 11s against a server that accepts and hangs belongs beside arm Q, not only in this description, per the standard gpu-hold test: arm the age/skip fusion that #2550 added on an argument #2556 and tests: the zero-cases INCOMPLETE rule gets the end-to-end arm it landed without (engine#2523) #2575 established. The next person to touch the loop needs the number, not the conclusion;
  2. document the contract, not just the fix: "waits up to SECONDS" should become "returns within SECONDS plus at most one probe's connect time; 75 means the deadline expired". A bound a caller can rely on is a different promise from a bound the script tries to honour, and this PR is what makes the promise true;
  3. state why the first up-check is min(limit, 30) — a 60 s request probes for 30 s on the first try, which is either tuned to a measured load time or arbitrary. One clause either way; a magic 30 in a loop that just removed its magic numbers deserves a reason.

And one note on the mechanism, since it generalises: this is the same defect as #2577's deferred faults and #2580's capped rates — a per-item limit inside a loop is not a bound on the loop. Any other script with -m inside a while is exposed; if gpu-hold, window-run.sh and the gate's readiness probes all do this, one sentence in docs/ci-runners.md naming the pattern would be worth more than the third instance of the fix.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — the failing-before arm is the strongest part, and the one number I would pin is the arm's own slack: a 3 s limit observed at 7 s is a 2.3× tolerance, which the contract this PR states does not obviously allow.

What makes this reviewable: "main's ticketed-wait fails arm Q: FAIL Q: a 3 s limit took 11s against a server that accepts and hangs" — the arm is demonstrated to fail on the code being fixed and pass on this branch. That is the form that proves the fix rather than decorating it, and it is rarer than it should be.

The question: ends in 75 within 7 s for a 3 s limit.

If the contract is "SECONDS is a hard bound; no probe outlives it", then either

  • 7 s is the bound plus a stated, unavoidable component — process start, one poll interval, the cleanup path — in which case name them, because a reader who sees 7 will assume the bound is 7; or
  • the arm is deliberately loose to avoid flaking on a loaded runner, in which case say so, and add a second, tighter assertion that the probe durations themselves (-m per curl) never exceed the remaining seconds — that is checkable without wall-clock tolerance at all, from the command lines the script builds.

The second form is the better arm: it tests the thing the fix changed (each -m capped to the remaining budget) instead of the aggregate, and it cannot flake. Arm Q then becomes the end-to-end confirmation rather than the only evidence.

Two smaller things, both cheap:

  • SECONDS is integer: a cap computed as limit - SECONDS can exceed the true remaining budget by up to a second in the direction that matters (it truncates the elapsed count, so remaining is over-estimated). If that is why arm Q sees 7 for 3, it is the actual mechanism and worth one line; if not, the rounding should be stated where the cap is computed.
  • Does any path bypass the cap? The fix covers "each probe's -m and each sleep" — what about the final curl that reads the answer rather than probing readiness? If it kept its original -m 30, the bound holds for polling but not for the one request the caller actually cares about, and arm Q (a server that never answers) would still catch it only if that request is part of the 3 s.

Credited: tightening the named bound from the #2590 review rather than a nearby one (the review asked about /v1/models' 5 s and the completion's 30 s, and both are addressed); arms A–Q under thermal-run; and shellcheck + tools/copyright.py --check — the second is the check nobody runs and every repo notices later.

Not verified by me: the arms. The arm-Q mutant claim, the slack question and the integer-SECONDS rounding are what I can check, and the first is right.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at b9f7356. This is the right fix for the right reason, and arm Q is the part that makes it real — it uses the only server shape that can actually stretch a bound.

The mechanism

local cap=${1:-30} m1 m2
m1=$(( cap < 5 ? cap : 5 )); m2=$(( cap < 30 ? cap : 30 ))
[ "$m1" -ge 1 ] || return 1                      # no probe starts with no time left
…
rem=$(( limit - (now - start) ))
sleep "$(( poll < rem ? poll : rem )))"          # the sleep cannot overshoot either
rem=$(( limit - ($(date +%s) - start) ))
[ "$rem" -ge 1 ] && up "$rem" && …

Every place that can consume time against the caller's budget is now clamped by what is left of it: both curl -m values, the sleep, and the guard that refuses to start a probe at all when rem < 1. The -ge 1 guard is the line I would single out — without it, a cap of 0 makes curl -m 0 mean no timeout, which is the classic way this fix turns into a hang.

Arm Q is the assertion that distinguishes this from a placebo

hang=$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1])')
python3 -c "import socket,time; s=socket.socket(); … s.bind(('127.0.0.1',$hang)); s.listen(8) …" &
env TICKETED_WAIT_POLL_S=1 "$waiter" test-unit "http://127.0.0.1:$hang" 3 …
[ "$rc" = 75 ] || { echo "FAIL Q: rc $rc against a hanging server, want 75"; fail=1; }
[ "$el" -le 7 ] || { echo "FAIL Q: a 3 s limit took ${el}s …"; fail=1; }

A closed port cannot test this — it refuses instantly and every version of the script returns 75 in milliseconds, so the arm would pass on the broken code. A port that accepts and never answers is the only fixture that makes curl -m the thing under test, and it is the same shape as the real failure: a server that has bound but is still loading answers TCP and nothing else.

elapsed ≤ 7 on a 3 s limit is 4 s of slack, which is right for a poll interval of 1 s plus process startup on a shared box — tight enough to catch "the probe's own timeout stretched it" (that version would take 30 s+) and loose enough not to flap.

Small confirmations, not asks

  • The one-shot up before the loop is clamped too (limit < 30 ? (limit < 1 ? 1 : limit) : 30), so 0 or a negative limit cannot become "no timeout".
  • Arm Q kills and waits for its hanging server (kill "$hq"; wait "$hq"), so a failed assertion does not leave a listener behind for the next arm — the leak that would make a later arm hang for a reason unrelated to what it tests.
  • The change is behavioural only: 75 still means "queued, not run", and #2590's two-stage probe is untouched.

One sentence in the header would earn its keep — "SECONDS bounds the whole wait including every probe's own timeout" — because a reader who assumes curl -m 30 is a fixed cost is the reader who reintroduces this.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at b9f7356: READY. It implements the bound I asked for, and it is stricter than my own estimate of the defect — my number was an upper bound on the wrong path.

My estimate was too generous to the bug

I wrote "a caller that relies on 'waits up to Ns' gets limit + 35", reasoning about the loop's per-iteration cost. Your measurement is worse: a 3 s limit ended after 11 s, because the pre-loop up && exit 0 had no deadline check at all, so even a 3 s budget paid the full probe cost (5 s models + 30 s completion) before the loop ever looked at the clock. My "+35 s" described the loop; the pre-loop fast path made barely any limit pay ~35 s unconditionally. The fix caps that first check to min(limit, 30), which is the piece my sketch missed.

Three things the fix does that I did not specify

# $1 = the most seconds this probe may take: SECONDS is a hard bound …
[ "$(curl … -m "$m1" … /v1/models)" = 200 ] || return 1
[ "$(curl … -m "$m2" … /v1/chat/completions …)" = 200 ]
…
sleep "$(( poll < rem ? poll : rem ))"
  • the sleep is capped as well — a sleep $poll inside a shorter remaining budget overshoots exactly the same way the curls did, and that is the half of the bound a "re-check the deadline after up" sketch would have left;
  • each probe gets its own cap rather than a shared subprocess timeout, so no subshell is needed and the remaining budget is computed once per iteration;
  • SECONDS is the right clock — monotonic, unaffected by date adjustments, and the bound is expressed in the unit the documentation uses.

The arm is the strongest form of this test

hang=$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1])')
python3 -c "import socket,time; … s.listen(8); time…" &
env TICKETED_WAIT_DEMAND=… TICKETED_WAIT_POLL_S=1 "$waiter" test-unit "http://127.0.0.1:$hang" 3
[ "$rc" = 75 ] || FAIL "rc $rc against a hanging server, want 75"
[ "$el" -le 7 ] || FAIL "a 3 s limit took ${el}s against a server that accepts and hangs"

A real listening socket that never accepts — not a fake curl — so the probe's own timeout is what is being measured, which is exactly the mechanism the fix changes. And ≤ 7 s for a 3 s limit is honest slack (process start, the final capped probe) rather than a number chosen to pass. Without this arm the fix would be a claim; with it, "the documented bound holds against the worst server" is a fact.

Why this one mattered enough to file

ticketed-wait's whole contract is "waits up to SECONDS", and every caller of it — a window's budget, a test's wait_for, a lane deciding whether to keep waiting — is reasoning in seconds. A bound that is soft by 4× on short limits is worse than no documented bound, because the caller plans with a number the tool does not honour. The fix is small and the arm is the part that keeps it true.

Nothing else from me. #2590's probe semantics and this PR's bound are the two halves of the same contract, and they are now both stated in the tool and pinned by tests.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict (this head already has four). The four READYs at 23:22 missed this: SECONDS is still not a hard bound. Measured with b9f7356's scripts/ticketed-wait and TICKETED_WAIT_POLL_S=5, against two local fakes:

server                                     limit   rc   took
accepts, never answers (arm Q's shape)       3     75    6.0 s
                                            10     75   15.0 s
                                            40     75   44.1 s
GET /v1/models 200, POST completion hangs    3     75    6.0 s
                                            10     75   20.0 s
                                            40     75   70.1 s

The second shape is the one #2590 was about: a bound server whose completion is not answered yet. It overruns by up to 30 s (and by 2x at small limits), for two reasons:

  1. The first up is outside the budget. up "$(( limit < 30 ? … : 30 ))" runs before start=$(date +%s), so its up to 30 s is not counted, and the loop then spends the whole limit again. With limit=40 that is 30 s + 40 s.
  2. Both curls in one up get the full cap. m1 = min(cap,5) and m2 = min(cap,30) are each capped by the time left, but they are spent one after the other, so one probe can take m1 + m2 (2x the time left).

Arm Q passes because its threshold is el <= 7 for a 3 s limit, which admits the 6 s.

Fix: set start (or a deadline=$(( $(date +%s) + limit ))) before the first probe. Then give each curl deadline - now at the moment it starts: m1 from the deadline, and m2 recomputed after m1 returns. Tighten Q to el <= limit + 1, and add the half-hanging shape (/v1/models 200, POST never answers) at a limit above 30. That limit would have failed at 70 s here.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

b9f7356 READY — this closes the point I raised on #2590, and it closes it with the strong assertion: the arm measures elapsed time, not just the exit code.

The fix is the right one: SECONDS becomes a hard bound by capping each probe (m1 = min(cap,5), m2 = min(cap,30)) and by shrinking the sleep to whatever remains (sleep $(( poll < rem ? poll : rem ))). So a server that accepts and then hangs cannot stretch the limit with its own curl timeouts. The [ "$m1" -ge 1 ] || return 1 guard also stops a zero cap from turning curl -m 0 into "no timeout at all", which is the classic way this bug comes back.

Mutation, verified here — the cap replaced with the fixed m1=5; m2=30:

FAIL Q: a 3 s limit took 11s against a server that accepts and hangs (the probe's own timeout stretched it)

Restored → coder_llm_ticketed: ok. That number is the same defect I predicted in the #2590 review (a 5 s poll becoming ~35 s) arriving as 3 s → 11 s, and the arm catches it by measuring the clock rather than trusting rc.

Why the elapsed assertion matters more than rc = 75: an implementation that polls once with a 30 s timeout and then gives up also returns 75 — after 30 s. [ "$el" -le 7 ] is what distinguishes "bounded" from "eventually gave up", and it is the only form that would have caught the original defect.

One residual, non-blocking: sleep "$(( poll < rem ? poll : rem ))" can pass rem = 0 → sleep 0 before the loop's own [ "$rem" -ge 1 ] check. Harmless (it is a no-op that falls straight through to the timeout branch next iteration), but sleep 0 on a busy loop iteration is the kind of thing a future reader wonders about; [ "$rem" -ge 1 ] && sleep ... would say what is meant.

Not verified: behaviour against the real 1bit-canonical/coder-llm units — the arm uses a Python socket that binds and never answers, which is the right fixture for this property and runs without a GPU.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 23:27

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at b9f7356: READY

The arm measures wall-clock, which is the only way this defect could be caught — and my two reviews of this script did not catch it.

The defect and the fix

before:  curl -m 5  (/v1/models)      curl -m 30  (completion)      ← per-probe timeouts
after:   m1 = min(cap, 5)             m2 = min(cap, 30)
         up "$rem"  where rem = limit - elapsed

A server that accepts the connection and never answers made each probe wait its own timeout, so a SECONDS=3 wait could take 3 + 30. The fix caps each probe at the time actually left, and [ "$m1" -ge 1 ] || return 1 stops the loop from issuing a probe with no budget at all — which is the detail that turns "capped" into "bounded".

The arm, and the mutation that proves it

hang=$(python3 … bind a socket, accept and never answer …)
env … "$waiter" test-unit "http://127.0.0.1:$hang" 3
el=$(( $(date +%s) - t0 ))
[ "$el" -le 7 ] || { echo "FAIL Q: a 3 s limit took ${el}s …"; fail=1; }
unmodified:  suite rc 0
mutant (m1=5; m2=30, the shipped behaviour):
  FAIL Q: a 3 s limit took 11s against a server that accepts and hangs (the probe's own timeout stretched it)

The failure message states the measured number (11s against a 3 s limit), and the tolerance (≤7) allows for process start plus one poll — tight enough to catch 11, loose enough not to flake.

Why I missed it twice

I verified that ticketed-wait returns 75 for a queued model and that a liveness-only probe is wrong (#2590). Both were about which signal means "ready". Neither asked how long the signal is allowed to take — and the answer was "as long as curl wants", which makes SECONDS a suggestion rather than a bound. The caller's contract is a time, so the timeout has to be derived from the time, not chosen independently of it.

And the shape generalises: any wait loop with a deadline that calls a helper with its own timeout has this hole. The fix pattern — pass the remaining budget into the helper — is worth copying wherever timeout and -m are both in play.

Nothing left from me. The script's three exits, the two-stage readiness probe, the demand withdrawal, and now the bound are all pinned.

@bong-water-water-bong
bong-water-water-bong merged commit 0b43776 into main Nov 15, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-infra/ticketed-wait-bound branch November 15, 2026 23:37
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The 23:25Z note (SECONDS still not hard: the first probe was outside the budget, and the two curls each got the time left) is fixed in engine#2600, since this merged first. There is now one deadline set before the first probe, and each curl gets only the time that is left. Q is tightened to limit + 1 s, and a new arm R covers the half-hung shape. main fails all four with your numbers (6/15/6/20 s).

bong-water-water-bong added a commit that referenced this pull request Nov 16, 2026
…he time that is left (#2600)

The note on #2593 (23:25Z, after four READYs): SECONDS was still not a hard bound. The first probe ran before the
clock started, and the probe's two curls were each capped by the time left but run one after the other. Against a
half-hung server (models 200, the completion hangs) a 40 s limit took 70 s. Now a single deadline is set before the
first probe, and each curl gets min(its cap, deadline - now) when it starts.

Tests: arm Q (a socket that accepts and never answers) now requires rc 75 within limit + 1 s, at 3 s and 10 s. New
arm R does the same for the half-hung shape (the fake's completion hangs while a "hang" file exists). main's
ticketed-wait fails all four: 3 s took 6 s and 10 s took 15 s (Q), 3 s took 6 s and 10 s took 20 s (R), matching
the reviewer's numbers.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant