You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test-hybrid_route: a derived backend port collides with the previous section's orphaned fake_backend (EADDRINUSE reported as 'did not become ready') #2537
test-hybrid_route fails when the same backend port is still held by the previous section's backend process, which stop() never signals.
Seen
engine#2523 run 43707970055 / job 132793096698 (step Test, 1 of 340 failed):
120/340 Test #120: hybrid_route .............................***Failed 22.86 sec
FAIL serve never reported a port; log=/tmp/tmp.KS4y0RZHI5/f.log
File "/usr/lib/python3.14/socketserver.py", line 478, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use
1bit serve: backend 0 did not become ready
FAIL serve starts with a provider that finishes after its deadline
Three facts that line up
The port is derived, not searched. Earlier rows in the same log show warm-up request … on port 38037 for two different serve invocations, and the bind that fails is again backend 0. tests/fake_backend.py binds whatever --port it is handed, so the value comes from the engine's per-serve allocation, and it repeats across serves in one test.
stop() kills the pid, not the tree.tests/hybrid_route.sh:72 is stop() { kill "$pid" 2>/dev/null; wait "$pid" 2>/dev/null; pid=; }. 1bit serve is reaped; tests/fake_backend.py is its child, so it is orphaned rather than signalled, and it keeps its listener.
The engine also reports the failure as "backend 0 did not become ready", which reads like a hang; the child's EADDRINUSE is what it should name.
What fixes it (and what does not)
Do not retry the bind. A retry hides the one case worth failing on: a serve whose backend could not start.
Test side, cheapest: start each serve in its own process group and kill the group (setsid, then kill -- -$pgid), and wait for the port to be free before the next start. Note that $! after setsid is the parent's pid, so the pid has to be written out by the child.
Engine side, correct: allocate the backend port per serve (bind and hold, then hand the bound port to the backend) instead of deriving it from the backend index, so two serves in one test cannot collide even with an incomplete teardown.
Acceptance
Either fix, plus: the backend-failure path reports the child's EADDRINUSE rather than "did not become ready", and an arm that kills a serve and immediately starts the next one on the same derived port. test-hybrid_route needs no GPU or model, so it runs in CI.
Claiming this (cl-1077, 0 open PRs). Plan: read how test-hybrid_route derives the backend port and tears down fake_backend between sections; make each section reap its child (wait on the pid) and/or bind an ephemeral port, then reproduce the collision before/after.
Measured on this box: stop() does not orphan the backend, and the backend port differs per serve — two of the three facts do not reproduce
I claimed this issue at 13:30Z and ran the harness against a real 1bit (~/wt/cl-npuperf-amdcol/build/1bit, engine 1c613e2), then measured the orphan directly rather than through the row.
Fact 2 (the orphan) does not hold here. Two runs, each: start serve, wait for the health answer so the backend has bound, record serve's children and their listening ports, kill $pid; wait $pid, then look again:
run 1: serve=2842483 kids=[2842487] held=[127.0.0.1:49881] after-kill=[none alive] ports-left=[none]
run 2: serve=2843703 kids=[2843706] held=[127.0.0.1:34225] after-kill=[none alive] ports-left=[none]
serve's teardown signals its backend.stop() killing only $pid is enough: the child is gone and its port is released.
Fact 1 (the port repeats across serves) did not reproduce either: 49881 and 34225 in consecutive serves of the same row, so the backend port is not a stable function of the backend index on this build (--port 0 for serve, backend port picked after).
And the arm I wrote for it passes on the old code, which is the test that settles it: I added stop() -> start with "no listener on the section's port" and then ran the same script with stop() reverted to the pid-only one-liner — both pass. An arm that passes before and after pins nothing.
What I therefore did not ship
The process-group setsid + kill-the-group + wait-for-port-free patch from the issue's "test side, cheapest" list. It compiles as a change and it passed the row, but on the measurement above it changes nothing, and shipping it would have retired this issue against a mechanism that is not the one biting. tests/hybrid_route.sh is unchanged.
What I think is actually left
One known leftover holder is already fixed: tests/smoke_serve.sh's impostor case detached a listener that outlived its row (measured: LISTEN 0 5 127.0.0.1:PORT survived a killed parent) — that is engine#2534, merged at 13:07Z. In a 340-row job the impostor ran before hybrid_route; whether it is the holder is unproven, but it is a proven leak of exactly the shape needed.
The message is still worth fixing, and it is engine-side: 1bit serve: backend 0 did not become ready is what a bind failure also reads as. Naming the child's EADDRINUSE (issue acceptance item 2) is a real improvement even when the collision is somebody else's process, because it tells the reader that the port was taken rather than that the backend hung.
PR #2541 @be56f92 (cl-1077). Agrees with the 13:17Z measurement: no orphan (my instrumented stop() found every child gone after the wait, 6 cycles). The cause left is free_port()'s release-then-bind window (the backend binds after its load; fake_backend sleeps 0.5 s first), which a parallel ctest can fill with another process on the same ephemeral port. The PR relaunches a backend that died of EADDRINUSE on a fresh port (max 2, nothing else retried) and names the cause; a new hybrid_route arm fails on main ("serve never reported a port", exactly the CI log) and passes with the fix (37/0).
Reopened: #2543 merged and closed this, but it fixed a different hazard than the one I described here. My diagnosis in the body above is wrong; this comment is the correction.
What I claimed — "the port is derived, not searched" and "stop() leaves an orphaned fake_backend.py holding it" — was inferred from one log tail. engine#2541 instrumented stop() to record serve's children before the kill and measured no orphan and no derived port. The real mechanism is the one app/serve.cpp already documents beside port_owner() (line 412):
// free_port() releases its port before the child binds it, so another local process could
// bind it first and answer the readiness probe; serve must not send it client requests.
Under ctest -j (340 rows) another test is handed that ephemeral port inside the free_port() → child-binds window, and fake_backend.py's 0.5 s sleep is what makes the window wide. So this stays open until the fix lands in serve — detect the child's EADDRINUSE, relaunch on a fresh port, and name it instead of reporting "backend 0 did not become ready" — which is #2541.
What merged (#2543) is still worth having and is not a fix for this: 1bit serve is now signalled as a process group in tests/hybrid_route.sh, so a child of serve cannot outlive its stop. That removes a hazard; it does not close the free_port() window, and a test can still collide after it.
Record for the closed issue: the fix is in main and I verified the hunk, so if this ever recurs the log line to search is named here.
app/serve.cpp:2795-2819 (engine#2541):
// engine#2537: a backend that died because its port was taken is relaunched on a fresh one, twice at most.for (intretry=1; !laya&&i<backends.size() &&retry <= kPortTakenRetries; ++retry) {
if (children[i]->wait_ready(…) ||g_stop) break;
if (children[i]->alive() || !*port_taken[i]) break; // only a DEAD child that printed EADDRINUSE"backend %zu could not bind port %d (address already in use: another process took it after serve picked it); relaunching on port %d (%d of %d)"
Three properties make that safe rather than a retry: only a dead backend with the flag set is retried (any other start failure still fails serve); the flag is cleared immediately before the relaunch so it cannot be consumed by the next backend's wait; and the retries are bounded with a final message that names the port (could not bind port … after N fresh ports; giving up) instead of "did not become ready".
Also in main: tests/hybrid_route.sh runs each serve under setsid and tears down the process group (#2543) — the hazard I originally mistook for this cause.
Not verified: a post-fix reproduction of the collision; it is a scheduling race I cannot schedule. What is verified is that the path exists, is entered only under the conditions that produced the failure, and reports the cause by name.
If it recurs: search the log for could not bind port … relaunching on port …. Its absence with a red hybrid_route row means a different cause, and this issue number is where to say so.
Activity
bong-water-water-bong commentedon Nov 15, 2026
Claiming this (cl-1077, 0 open PRs). Plan: read how test-hybrid_route derives the backend port and tears down fake_backend between sections; make each section reap its child (wait on the pid) and/or bind an ephemeral port, then reproduce the collision before/after.
bong-water-water-bong commentedon Nov 15, 2026
Measured on this box:
stop()does not orphan the backend, and the backend port differs per serve — two of the three facts do not reproduceI claimed this issue at 13:30Z and ran the harness against a real
1bit(~/wt/cl-npuperf-amdcol/build/1bit, engine1c613e2), then measured the orphan directly rather than through the row.Fact 2 (the orphan) does not hold here. Two runs, each: start serve, wait for the health answer so the backend has bound, record serve's children and their listening ports,
kill $pid; wait $pid, then look again:serve's teardown signals its backend.
stop()killing only$pidis enough: the child is gone and its port is released.Fact 1 (the port repeats across serves) did not reproduce either:
49881and34225in consecutive serves of the same row, so the backend port is not a stable function of the backend index on this build (--port 0for serve, backend port picked after).And the arm I wrote for it passes on the old code, which is the test that settles it: I added
stop() -> startwith "no listener on the section's port" and then ran the same script withstop()reverted to the pid-only one-liner — both pass. An arm that passes before and after pins nothing.What I therefore did not ship
The process-group
setsid+ kill-the-group + wait-for-port-free patch from the issue's "test side, cheapest" list. It compiles as a change and it passed the row, but on the measurement above it changes nothing, and shipping it would have retired this issue against a mechanism that is not the one biting.tests/hybrid_route.shis unchanged.What I think is actually left
tests/smoke_serve.sh's impostor case detached a listener that outlived its row (measured:LISTEN 0 5 127.0.0.1:PORTsurvived a killed parent) — that is engine#2534, merged at 13:07Z. In a 340-row job the impostor ran beforehybrid_route; whether it is the holder is unproven, but it is a proven leak of exactly the shape needed.1bit serve: backend 0 did not become readyis what a bind failure also reads as. Naming the child'sEADDRINUSE(issue acceptance item 2) is a real improvement even when the collision is somebody else's process, because it tells the reader that the port was taken rather than that the backend hung.Releasing the claim rather than leaving a PR that would close this issue without a mechanism.
bong-water-water-bong commentedon Nov 15, 2026
PR #2541 @be56f92 (cl-1077). Agrees with the 13:17Z measurement: no orphan (my instrumented stop() found every child gone after the wait, 6 cycles). The cause left is free_port()'s release-then-bind window (the backend binds after its load; fake_backend sleeps 0.5 s first), which a parallel ctest can fill with another process on the same ephemeral port. The PR relaunches a backend that died of EADDRINUSE on a fresh port (max 2, nothing else retried) and names the cause; a new hybrid_route arm fails on main ("serve never reported a port", exactly the CI log) and passes with the fix (37/0).
bong-water-water-bong commentedon Nov 15, 2026
Reopened: #2543 merged and closed this, but it fixed a different hazard than the one I described here. My diagnosis in the body above is wrong; this comment is the correction.
What I claimed — "the port is derived, not searched" and "
stop()leaves an orphanedfake_backend.pyholding it" — was inferred from one log tail. engine#2541 instrumentedstop()to record serve's children before the kill and measured no orphan and no derived port. The real mechanism is the oneapp/serve.cppalready documents besideport_owner()(line 412):Under
ctest -j(340 rows) another test is handed that ephemeral port inside thefree_port()→ child-binds window, andfake_backend.py's 0.5 s sleep is what makes the window wide. So this stays open until the fix lands inserve— detect the child'sEADDRINUSE, relaunch on a fresh port, and name it instead of reporting "backend 0 did not become ready" — which is #2541.What merged (#2543) is still worth having and is not a fix for this:
1bit serveis now signalled as a process group intests/hybrid_route.sh, so a child of serve cannot outlive its stop. That removes a hazard; it does not close thefree_port()window, and a test can still collide after it.tests: the #2537 comments say how serve picks a port without naming f…
bong-water-water-bong commentedon Nov 15, 2026
Record for the closed issue: the fix is in
mainand I verified the hunk, so if this ever recurs the log line to search is named here.app/serve.cpp:2795-2819(engine#2541):Three properties make that safe rather than a retry: only a dead backend with the flag set is retried (any other start failure still fails serve); the flag is cleared immediately before the relaunch so it cannot be consumed by the next backend's wait; and the retries are bounded with a final message that names the port (
could not bind port … after N fresh ports; giving up) instead of "did not become ready".Also in
main:tests/hybrid_route.shruns each serve undersetsidand tears down the process group (#2543) — the hazard I originally mistook for this cause.Not verified: a post-fix reproduction of the collision; it is a scheduling race I cannot schedule. What is verified is that the path exists, is entered only under the conditions that produced the failure, and reports the cause by name.
If it recurs: search the log for
could not bind port … relaunching on port …. Its absence with a redhybrid_routerow means a different cause, and this issue number is where to say so.