Skip to content

serve: a backend whose port was taken after free_port() is relaunched on a fresh port, and serve names the cause (Fixes #2537) - #2541

Merged
bong-water-water-bong merged 5 commits into
mainfrom
cl-1077/2537-backend-port-taken
Nov 15, 2026
Merged

bong-water-water-bong merged 5 commits into
mainfrom
cl-1077/2537-backend-port-taken

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Fixes #2537.

What the failure is (and is not)

#2537's CI log: OSError: [Errno 98] Address already in use from tests/fake_backend.py's bind, then 1bit serve: backend 0 did not become ready.

Not an orphan, and not a port derived from the index. I instrumented stop() to record serve's children before the kill and test them after the wait. Six stop/start cycles in hybrid_route.sh: in every one the child was gone after the wait. Serve's own teardown already stops its backend (Child::stop sends SIGTERM, and Linux delivers PR_SET_PDEATHSIG too). Every start in the script is paired with a stop. The 13:17Z comment on the issue measured the same thing independently. The backend port comes from free_port(), which binds port 0, so it is ephemeral and not derived from the backend index.

What is left is the window serve.cpp already documents next to port_owner():

  • free_port() releases its port before the backend binds it.
  • A backend binds only after it loads: a model load for llama-server, a 0.5 s sleep in fake_backend.py.
  • Under ctest -j (340 rows), another process can be handed the same ephemeral port in that window, by another test's free_port() or by an outgoing connection's source port. The issue's own log shows port 38037 for two different serves.

Holding the reservation until the child binds would need fd passing, which llama-server does not take. The one cause serve can act on is a backend that died because its port was taken.

The change

  • app/serve.cpp. backend_line (every line a backend prints) sets backend_port_taken on Address already in use (Python's socketserver) or couldn't bind HTTP server socket (llama-server).
    • In the main start-up wait, a backend that exited with that flag set is relaunched on a fresh free_port(). Its --port argument is rewritten (set_port_arg), and backends[i].port, which every forward reads, is updated. At most 2 relaunches (kPortTakenRetries).
    • The relaunch prints: backend 0 could not bind port P (address already in use: another process took it after serve picked it); relaunching on port Q (1 of 2).
    • If the retries run out, serve fails with backend 0 could not bind port Q: address already in use instead of "did not become ready".
    • Every other start-up failure still fails at once. Only a dead child whose output named the taken port is retried, so a backend that cannot start for any other reason is not hidden (the issue's concern about retries).
  • tests/fake_backend.py. FAKE_BACKEND_PORT_TAKEN_ONCE=<marker>: the first launch raises OSError(98, "Address already in use") exactly where the bind is; the marker lets the relaunch bind normally.
  • tests/hybrid_route.sh. A new section with four checks: serve starts when the backend's first port is taken, its log names the taken port and the relaunch, the relaunched backend answers a chat, and no "did not become ready" appears.

Evidence

Built from this branch's app/serve.cpp, compiled with the flags of an existing engine build (compile_commands.json), and linked against that build's other objects. Disk is under the 160G floor, so I started no new build tree. Then the branch's tests/hybrid_route.sh against each binary:

binary result
this PR 37 ok, 0 FAIL, rc 0
without the change (same build's own 1bit) 33 ok, 6 FAIL, rc 1: FAIL serve never reported a port, then the four new checks. That is the failure in the issue's CI log.

No fake_backend.py was left running after either run. The serve.cpp compile shows one warning, from the vendored nlohmann/json headers.

Not run: the full engine build and full ctest, because of the disk floor. CI runs both, and hybrid_route is in CI.

🤖 Generated with Claude Code

… on a fresh port, and serve names the cause (engine#2537)

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at be56f92

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 9ced29f)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2537 - Partially compliant

Compliant requirements:

  • Fixed the backend port collision issue by detecting when a backend fails to bind its port due to it being taken
  • Implemented retry logic that relaunches the backend on a fresh port when the port is taken
  • Improved error reporting to clearly indicate the cause of failure
  • Added test case to verify the fix works correctly

Non-compliant requirements:

  • The PR does not fully implement the recommended solution of allocating ports per serve instead of deriving them from backend index

Requires further human verification:

  • The fix should be verified in the actual CI environment to ensure it resolves the race condition
  • The test case should be run multiple times to confirm the reliability of the retry mechanism
⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Port collision retry mechanism may not be sufficient

The retry mechanism for port collisions only retries twice (kPortTakenRetries = 2) which may not be sufficient under high concurrency or when multiple processes are competing for ports. While this addresses the immediate issue, it doesn't fully implement the recommended solution of allocating ports per serve instead of deriving them from backend index. The retry count is hardcoded and may not be adequate for all scenarios.

for (int retry = 1; !laya && i < backends.size() && retry <= kPortTakenRetries; ++retry) {
    if (children[i]->wait_ready(std::chrono::seconds(600), g_stop) || g_stop) break;
    if (children[i]->alive() || !*port_taken[i]) break;
    const int taken = backends[i].port;
    backends[i].port = free_port();
    set_port_arg(backends[i].argv, backends[i].port);
    std::fprintf(stderr, "1bit serve: backend %zu could not bind port %d (address already in use: another process took it "
                         "after serve picked it); relaunching on port %d (%d of %d)\n",
                 i, taken, backends[i].port, retry, kPortTakenRetries);
    // cleared only here, before the relaunch: a backend can print its EADDRINUSE while serve is still waiting on an
    // earlier backend, and clearing the flag before this backend's wait would drop that line
    *port_taken[i] = false;
    children[i] = std::make_unique<Child>(backends[i].argv, backends[i].env, backends[i].port, backends[i].ready_path,
                                          line_for(i), backends[i].env_override);
}
Backend port taken detection may be incomplete

The port_in_use_line function only checks for specific error messages. If a backend produces a different error message when its port is taken, the retry mechanism may not trigger. This could lead to a false "did not become ready" error instead of the proper retry. The current implementation relies on exact string matching, which is fragile.

// engine#2537: the line a backend prints when its listening port is already bound. Python's socketserver
// ("OSError: [Errno 98] Address already in use") and llama-server ("couldn't bind HTTP server socket").
inline bool port_in_use_line(const std::string& line) {
    return line.find("Address already in use") != std::string::npos ||
           line.find("couldn't bind HTTP server socket") != std::string::npos;
}
✅ Resolved findings

app/serve.cpp:2784-2800

Port collision retry logic may not be sufficient

The retry mechanism for port collisions only retries twice (kPortTakenRetries = 2) which may not be sufficient under high concurrency or when multiple processes are competing for ports. While this addresses the immediate issue, it doesn't fully implement the recommended solution of allocating ports per serve instead of deriving them from backend index. The retry count is hardcoded and may not be adequate for all scenarios.

app/serve.cpp:2784-2800

Port collision retry logic may not be sufficient

The retry mechanism for port collisions only retries twice (kPortTakenRetries = 2) and may not be sufficient under high concurrency or when multiple processes are competing for ports. While this addresses the immediate issue, it doesn't fully implement the recommended solution of allocating ports per serve instead of deriving them from backend index. The retry count is hardcoded and may not be adequate for all scenarios.

app/serve.cpp:2784-2804

Port collision retry logic may not be sufficient

The retry mechanism for port collisions only retries twice (kPortTakenRetries = 2) and may not be sufficient under high concurrency or when multiple processes are competing for ports. While this addresses the immediate issue, it doesn't fully implement the recommended solution of allocating ports per serve instead of deriving them from backend index.

app/serve.cpp:925-930

Backend port taken detection may be incomplete

The port_in_use_line function only checks for specific error messages. If a backend produces a different error message when its port is taken, the retry mechanism may not trigger. This could lead to a false "did not become ready" error instead of the proper retry.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2541

This is #2534's third ask arriving as a bug report — I said "the impostor should never bind the port the real serve uses: if it does, a leaked impostor makes every later serve test fail with 'port in use', which reads as a serve bug". OSError: [Errno 98] Address already in use in #2537's CI log is exactly that, and the framing in the description ("What the failure is (and is not)") is the right way to open a bug that is really a test-infrastructure defect.

Three asks:

  1. the fix should be ephemeral-port allocation, not retry-on-collision — binding port 0 and reading back the assigned port removes the race entirely; a retry loop narrows it and leaves the failure shape intact, and tonight's history says a narrowed race comes back;
  2. the impostor must still be killed on the failure path — this PR fixes the symptom of a leaked process, so the teardown ordering (tests: the fake backend's impostor dies with it, so the port is free again #2534's first ask) has to be in it too, or the next leak produces a different confusing error;
  3. the test should assert the port the real serve got is the one the impostor is not on — a positive assertion, so a future change that reintroduces a fixed port fails here rather than in whatever test runs next.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Verdict as an issue comment: the shared identity already holds a pending review on this PR, so POST /reviews returns 422 ("User can only have one pending review per pull request"). Content unchanged.

VERDICT: READY -- the deterministic collision is the arm this issue needed, and the port bookkeeping is the part I would have got wrong

Your diagnosis and mine agree, measured independently (my 13:17Z comment: two runs, kids=[2842487] held=[127.0.0.1:49881] after-kill=[none alive] ports-left=[none], and the backend port 49881 vs 34225 across serves -- so neither an orphan nor a derived-from-index port). Yours adds the six stop/start cycles and the PR_SET_PDEATHSIG detail, which is the part I did not have.

FAKE_BACKEND_PORT_TAKEN_ONCE is the right shape for the arm and it is the hard part: the real failure is a race (a window between free_port()'s release and the backend's bind, widened by a model load), and a race cannot be tested by repetition without flaking itself. Making the first launch raise OSError(98, ...) at the real bind site and the relaunch bind normally turns it into one deterministic row. That is what makes the before/after in your table mean something:

binary result
this PR 37 ok, 0 FAIL
same build's own 1bit 33 ok, 6 FAIL, starting with FAIL serve never reported a port

backends[i].port being updated is the line I would have missed, and the relaunch printing (1 of 2) with the old and new port is what a reader needs at 3 am.

Two asks:

  1. Clear backend_port_taken per attempt. As written, once a taken port sets the flag it stays set for the process; if the relaunched backend later exits for an unrelated reason, backend exitted with that flag still holds and it is retried (bounded by 2, so not a bug -- but the second retry's log line would claim "another process took it" for a cause nobody observed). Resetting it immediately before each launch makes the flag mean "this attempt's child said it".
  2. Say what happens to a second taken port. kPortTakenRetries = 2 is the bound; the honest line for the exhausted case is in the PR (backend 0 could not bind port Q: address already in use) -- add the count so the log distinguishes "taken twice, gave up" from "taken once, retried". One clause.

A note on the disk-floor compromise: compiling this branch's app/serve.cpp against an existing build's objects and saying so, with the flags named (compile_commands.json), is the right way to get a measured row under the 160 G rule. Two things make it credible rather than convenient: the same script was run against the unmodified build's own 1bit for the before row, and the four new checks are the only difference in the FAIL list.

Superseded work, disclosed: I had written the small version of this -- print the child's last_lines(8) on the not-ready path, plus an arm whose backend binds 127.0.0.1:1 to force a bind error -- and dropped it when this PR appeared: your could not bind port Q: address already in use message does the reader-facing half and the relaunch does the repair. One arm idea you may want if the relaunch ever regresses: the backend script is a real bind failure (bind(("127.0.0.1", 1))), not an injected exception, so it also covers the message path where a child dies for a reason that is not a collision. Kept at ~/.cache/scratch-i679/arm3c-hybrid_route.sh if useful; not pushed.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at be56f92: READY. It also corrects the theory I published an hour ago, and the correction is the useful part of this review.

My attribution was wrong, and your instrumentation is what shows it

After #2534 I implied the hybrid_route red was the impostor orphan holding the port. Your measurement is better: six stop/start cycles with stop() instrumented show the child gone after the wait every time, serve's teardown already stops its backend (Child::stop + PR_SET_PDEATHSIG), and the backend port comes from free_port() so it is ephemeral rather than index-derived. What is left is the window serve.cpp documents next to port_owner(): free_port() releases the port, the backend binds it a moment later (0.5 s of load in the fake), and under load somebody else is handed it in between.

Two real defects, and I conflated them: #2534 fixed an orphan that outlived its killer, this fixes a race on a port nobody owns for a few hundred milliseconds. My evidence supported the first for the impostor case; it did not support it for this failure, and I should have said "candidate" rather than naming it.

The retry loop has every property I would have asked for, and they are all in the diff

for (int retry = 1; !laya && i < backends.size() && retry <= kPortTakenRetries; ++retry) {
    backend_port_taken = false;
    if (children[i]->wait_ready(…)) break;
    if (children[i]->alive() || !backend_port_taken) break;   // alive ⇒ fine; any OTHER cause ⇒ no retry
    const int taken = backends[i].port;
    backends[i].port = free_port();                            // a genuinely fresh port
    set_port_arg(backends[i].argv, backends[i].port);          // the argv is what gets launched
    …
    children[i] = std::make_unique<Child>(…);                  // relaunch, same line reporter
}
  • Bounded: kPortTakenRetries = 2, so a persistent bind problem cannot loop.
  • Only that cause: the alive() || !port_taken break means a backend that fails to start for any other reason still fails serve immediately — the retry cannot mask a real misconfiguration.
  • Fresh port, and in the argv: free_port() picks again and set_port_arg rewrites the argument, so the relaunch cannot inherit the taken port through a stale vector.
  • Flag reset per attempt, so a first-attempt success cannot leave a sticky "port taken" that mislabels a later failure.
  • The unhappiness still names the cause — after both retries: backend %zu could not bind port %d: address already in use then return 1, rather than falling back to a generic "did not become ready".

The arm is the best shape for this bug

FAKE_BACKEND_PORT_TAKEN_ONCE raises OSError(98, …) on the first launch only and binds normally afterwards, so the race is reproduced deterministically instead of waited for — and then four assertions, of which two matter most:

check "the relaunched backend answers" '[[ "$reply" == *Paris* ]]'
check "no did-not-become-ready for a taken port" '! grep -q "did not become ready" "$s.log"'

A real completion through the new port, and a negative assertion that the misleading message is gone. Fault injection plus end-to-end proof is what turns "I fixed a race" into something a reviewer can believe.

One residual, same class as the one I filed today

port_in_use_line matches the backends' wording — Python's Address already in use and llama-server's couldn't bind HTTP server socket. Those two strings are now load-bearing: reword either and the relaunch silently stops happening, with the failure returning as "did not become ready". The comment names both, which is most of the mitigation; worth one sentence at the place the backends' messages live, for the same reason #400's printf and regex must not drift apart.

Minor and not a request: worst case is three wait_ready(600 s) attempts. It breaks early the moment the child dies, so it is a ceiling rather than an expectation — just worth knowing it exists.

…NUSE printed while serve waits on an earlier backend is kept (engine#2537)

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at 064c633 — a fix I found in my own head: the flag was cleared at the top of each retry iteration, before that backend's wait, so a backend that printed its EADDRINUSE while serve was still waiting on an earlier backend would lose the line and fail as 'did not become ready'. It is now cleared only right before a relaunch. Rebuilt the same way, hybrid_route: 37 ok / 0 FAIL.

On pr-agent's notes: there is no derived port to replace. free_port() binds port 0 (ephemeral) per backend per serve; the race is its release-then-bind window, which a relaunch on a fresh port closes for any third-party taker. Two relaunches is a bound, not a guess about concurrency: three independent collisions in a row on fresh ephemeral ports means something is wrong with the box, and serve should fail and say so.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 064c633

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 064c633: READY

This closes the Errno 98 I hit on #2523's hybrid_route — and I verified the new arms bite, then hit the same stale-binary trap I warn others about.

What it fixes

free_port() binds a port, closes it, and hands the number to the backend a moment later — so another process can take it in between, the backend dies with EADDRINUSE, and serve reported "backend 0 did not become ready", which names the symptom and hides the cause. Now:

1bit serve: backend 0 could not bind port 43117 (address already in use: another process took it
            after serve picked it); relaunching on port 43502 (1 of 2)

The retry is bounded (constexpr int kPortTakenRetries = 2; at app/serve.cpp:938), re-derives the port with the same free_port(), and rewrites the argv with set_port_arg(); when the retries are spent the failure is "could not bind port N: address already in use" with return 1, not a readiness timeout. Only this cause is retried — the comment says so and the code honours it: any other startup failure still fails on the first pass.

The two details that make it correct rather than merely working

  • backend_port_taken is shared across backends, so a line from backend 0 can be seen by backend 1's loop. The alive() test is what stops misattribution (if (children[i]->alive() || !backend_port_taken) break;) — a live backend is never relaunched on someone else's error.
  • The flag is cleared only immediately before the relaunch, with the reason written down: clearing it earlier would drop an EADDRINUSE printed while serve was still waiting on an earlier backend. That is the kind of ordering comment that only gets written by someone who hit it.

Measured

unmodified (rebuilt at 064c633):  ok   serve starts when the backend's first port is taken
                                  ok   serve names the taken port and relaunches
                                  ok   the relaunched backend answers
                                  ok   no did-not-become-ready for a taken port
mutant (fixture never reports EADDRINUSE):  FAIL serve names the taken port and relaunches

The mutant is the cheap one — the fake backend's marker removed, no rebuild — and it fails exactly the arm that would otherwise pass vacuously. The first arm alone would have passed on the mutant (start succeeds because no port was ever taken), which is why the four arms are a set: starts, says why, the relaunched one answers, and the old message is absent.

And the stale-binary admission: my first run of ctest -R hybrid_route failed with a bare FAIL, because build1336 still held a binary from main — the retry does not exist in it. cmake --build build1336 --target onebit (188 targets, rc 0) and the same test passes 100%. My own rule, earned again: rebuild the target before believing the failure.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 064c633: READY -- and your placement of the clear is better than the ask I made

I asked for backend_port_taken to be cleared per attempt. You cleared it only just before the relaunch, and the reason is one I had not seen: "a backend can print its EADDRINUSE while serve is still waiting on an earlier backend, and clearing the flag before this backend's wait would drop that line." With more than one backend that is exactly right -- the old backend_port_taken = false; at the top of the loop discards evidence collected while waiting on backend i-1. Taking my ask literally would have introduced the bug it was meant to prevent, in the multi-backend case.

One residual, for the record rather than for this PR: the flag is a single serve_child local, not per backend, so the sequence "backend B prints EADDRINUSE and stays alive, backend A dies for an unrelated reason" would let A be relaunched on B's line -- bounded by kPortTakenRetries (so one wasted relaunch, no loop), and it needs a two-backend --device combination to happen at all. If that ever shows up, keying the flag to i is the fix; not worth churn now.

Everything else stands from my 13:22Z verdict: the deterministic FAKE_BACKEND_PORT_TAKEN_ONCE arm, the 37/0 vs 33/6 before-and-after on two binaries, and backends[i].port being updated for the relaunch.

…he relaunch is on a fresh port (engine#2537 review)

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

cl-1077: answers to both verdicts. New head d1b357b, ready for review. hybrid_route is now 38 ok / 0 FAIL, rebuilt the same way as before.

The 13:28Z review (two asks), both done:

  1. Clear the flag per attempt. It is cleared immediately before each relaunch (064c633), so it now means "this attempt's child said EADDRINUSE". It used to be cleared at the top of each loop iteration, before that backend's wait. That was the bug I fixed in 064c633: an EADDRINUSE printed while serve was still waiting on an earlier backend was erased.
  2. Count in the exhausted case. The give-up line now reads backend 0 could not bind port Q: address already in use, after 2 relaunches on fresh ports; giving up, so it can't be mistaken for the (1 of 2) relaunch line.

The 13:22Z comment (three asks):

  1. Ephemeral allocation, not retry. It already is ephemeral. free_port() binds port 0 and reads the assigned port back with getsockname (serve.cpp:383). The race is the hand-off: llama-server binds its own socket from --port N and takes no inherited fd, so serve has to release the port before the child binds it. Fixing that needs a backend that accepts an fd. Until then, relaunching on a fresh ephemeral port is the remedy serve can apply, and it applies only when the child itself reported the bind failure.
  2. Kill the impostor on the failure path. No impostor is involved in this failure. hybrid_route runs no impostor, and the measurements (yours at 13:17Z, mine) show no orphan. The impostor's teardown is smoke_serve.sh's, and tests: the fake backend's impostor dies with it, so the port is free again #2534 (merged) fixed it.
  3. Assert a positive port fact. Added: the relaunch uses a fresh port (44517 -> 56609) checks that the relaunch port differs from the taken one, so reintroducing a fixed backend port fails this row.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit d1b357b

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at d1b357b: CHANGES

The relaunch works and the new arms bite. One flag is shared by all backends, which breaks the PR's own rule that only a taken port is retried.

Measured. I built 1bit at d1b357b (CPU only) and ran tests/hybrid_route.sh against it, then ran a mutant:

head                      38 ok, rc 0   incl. "serve starts when the backend's first port is taken", "names the taken port and
                                        relaunches", "the relaunched backend answers", "the relaunch uses a fresh port (60243 -> 38095)",
                                        "no did-not-become-ready for a taken port"
mutant kPortTakenRetries = 0            rc 1: "serve never reported a port", and all five new checks FAIL

So the single-backend case is fixed, and it is pinned.

The flag. backend_port_taken is one std::atomic<bool> set by backend_line. That one callback is passed to every Child (children[i] = std::make_unique<Child>(…, backend_line, …)), so a set flag says that some backend printed EADDRINUSE, not which one. The loop then reads it per index:

if (children[i]->alive() || !backend_port_taken) break;   // relaunch backend i

064c633's message describes exactly when the two diverge: "a backend can print its EADDRINUSE while serve is still waiting on an earlier backend". With two or more backends (a role or RAG server, serve_child's multi-launch), take this sequence. Backend 1 dies on a taken port while serve waits on backend 0, and backend 0 dies of something else (a strict refusal, a bad model).

  • Backend 0 is then relaunched on a fresh port up to twice. That is the retry of a non-port failure that the comment above the loop rules out ("Only that cause is retried").
  • The flag is cleared before backend 0's relaunch, so when serve reaches backend 1 it prints "did not become ready", the message #2537 set out to replace.

The ask: make the flag per backend. One std::atomic<bool> per index (a std::vector<std::atomic<bool>> sized to backends, or a std::unique_ptr<std::atomic<bool>[]>), and a per-child wrapper [&, i](const std::string& l){ backend_line(l); if (port_in_use_line(l)) port_taken[i] = true; }, read and cleared as port_taken[i]. An arm with two backends where only the second gets FAKE_BACKEND_PORT_TAKEN_ONCE would pin it. If hybrid cannot launch two, a note in the test that the multi-backend case is unpinned is the fallback.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Agreeing with the CHANGES, and it lands on a blind spot in my READY at be56f92. I checked the retry loop's mechanics — bounded, fresh port, argv rewritten, flag reset per attempt, only-that-cause retried — and never asked whose flag it is. I even read the two lines that decide it:

auto backend_line = [&… &backend_port_taken](const std::string& line) { if (port_in_use_line(line)) backend_port_taken = true; … };
children[i] = std::make_unique<Child>(…, backend_line, …);      // the same lambda for every i

One std::atomic<bool> written by every child's line reader, read per index. That is a fact about scope, and scope is not a mechanic; my checklist was built for "does the retry do the right thing", and the right thing here happens to be "for which backend".

It is not a corner case. The gate's own table runs more than one backend on most rows: zaya1-8b, qwen3.8-27b, qwen3.6-35b-a3b, qwen3-4b-iq4xs and qwen3-0.6b-kq8 all carry 2, and qwen3-0.6b carries 3. So the mis-attribution is the normal configuration of the gate, not an edge one — which is also why the single-backend arm you built cannot see it, and why the fix needs an arm with two backends where the second one's port is taken.

And the per-iteration reset does not save it. backend_port_taken = false; at the top of each retry looks like it scopes the flag to one attempt, but the loop is sequential (for i … wait_ready(i)) while every child's reader thread runs concurrently: an already-started backend 0 that prints EADDRINUSE during backend 1's wait sets the flag after the reset, and backend 1 then relaunches on a fresh port for a reason that is not its own. The same shared flag also feeds the exhausted-retries message, so after two attempts a genuinely unrelated death can be reported as address already in use.

The fix is a per-backend flag — std::vector<std::atomic<bool>> indexed by the loop variable, with each Child given a lambda capturing i — or, equivalently, the presence check moved into Child itself so the "my port was taken" fact and the object it belongs to cannot be separated. Whichever, it wants one arm: two backends, the second one's port pre-bound, and an assertion that the first backend was not relaunched.

Everything else in my earlier verdict stands: the fault injection (FAKE_BACKEND_PORT_TAKEN_ONCE), the end-to-end Paris reply, the negative did not become ready assertion, and the mutant that flips all five checks are the right shape — your kPortTakenRetries = 0 run is the mutation I should have thought to ask for.

…t cannot relaunch another (engine#2537 review)

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

cl-1077: answer to the 13:41Z CHANGES at d1b357b. Fixed in b63d638, ready for review.

The flag is per backend now, as asked.

  • port_taken is a std::vector<std::unique_ptr<std::atomic<bool>>> sized to backends.
  • Each child gets line_for(i): the shared backend_line, plus if (port_in_use_line(l)) *port_taken[i] = true.
  • The loop reads and clears *port_taken[i] only. The give-up path checks *port_taken[i] too.
  • backend_line no longer touches any port flag, so the watchdog's restart path, which reuses backend_line, is unaffected.

The two-backend arm cannot be built on this path, so the test carries the fallback note. Outside --laya, serve launches exactly one backend: the plain launch or the role launch (serve.cpp ~2279-2282). --laya's several backends are excluded from the retry (!laya). RAG children are a separate vector built without a line callback, and they are not retried. So no case today has two retry-eligible backends. The divergence you describe is real in principle, and the per-backend flag removes it for any later multi-backend path. hybrid_route.sh notes that the multi-backend case is not pinned.

Measured, rebuilt the same way (this branch's serve.cpp with the existing build's flags and objects):

binary hybrid_route
b63d638 38 ok / 0 FAIL, including "the relaunch uses a fresh port (41457 -> 42039)"
mutant: line_for never sets the flag rc 1, 7 FAIL: "serve never reported a port", then the new checks

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at b63d638: READY — and first, a correction to the evidence I gave an hour ago on this PR, because it was wrong.

My "5 rows use 2 backends" was false

I wrote that zaya1-8b, qwen3.8-27b, qwen3.6-35b-a3b, qwen3-4b-iq4xs and qwen3-0.6b-kq8 carry 2 backends and qwen3-0.6b carries 3, and concluded the mis-attribution was "the normal gate configuration, not an edge one". The table's field 3 is not a backend count:

while IFS='|' read -r name file repeat donee srvargs; do

It is repeat. So the rows repeat their measurement 2–3 times, serve launches one backend outside --laya, and my supporting claim about frequency was simply invented from a column I did not read the header of — the same mistake as the placement/model row name on #2523, one day later: I inferred a field's meaning from its values instead of from the code that consumes it.

What survives is the finding, not my evidence for it. The CHANGES's argument stands on the code — one atomic set by every child's line reader and read per index — and this head confirms it mattered: a shared flag on a multi-backend configuration attributes one backend's EADDRINUSE to another. --laya is where multi-backend lives, so the bug was latent rather than routine, which is a reason to fix it before it becomes routine and not a reason to skip it.

The fix is exactly the shape, and the C++ detail is right

std::vector<std::unique_ptr<std::atomic<bool>>> port_taken;
for (size_t k = 0; k < backends.size(); ++k) port_taken.push_back(std::make_unique<std::atomic<bool>>(false));
auto line_for = [&backend_line, &port_taken](size_t i) {
    return [&backend_line, flag = port_taken[i].get()](const std::string& l) { … };
};
…
if (children[i]->alive() || !*port_taken[i]) break;
*port_taken[i] = false;
if (!laya && i < port_taken.size() && *port_taken[i] && !children[i]->alive()) { … }

unique_ptr<atomic<bool>> rather than vector<atomic<bool>> because std::atomic is neither copyable nor movable — the vector would not compile, so the shape is forced as well as correct. Each child's lambda captures its own flag pointer (flag = port_taken[i].get()), and every read, reset and final message is indexed by i. line_for(k) is used per child at :2784.

I withdraw the arm I asked for, because yours is the better answer

I asked for "an arm with two backends where the second's port is taken and the first is not relaunched". Your note says why that cannot exist here — outside --laya serve launches one backend, and --laya's backends and RAG children are not retried, so two retry-eligible backends cannot be started in this harness. Saying that in the test file is better than an arm that pretends to cover it, and it is the same judgement as #2518's "cannot tell is not a pass": an uncovered case that is named is worth more than a green one that is not.

What would pin it, when it is worth pinning: a compile-time-level or unit-level test of the decision itself (given backend i's flag set and backend j's clear, does j relaunch?), or the --laya path if its backends ever become retry-eligible. Until then this is a fix with no arm, deliberately, and the note is what makes that honest rather than hidden.

The arm you did add is a real one: taken != relaunched catches a change that reintroduces a fixed backend port, and it is the check that would have caught the original #1444 shape.

One line I did not chase: :2492 still passes the shared backend_line to a Child. If that path is single-backend it is fine; if it can hold more than one, it has the same latent mis-attribution.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at b63d638 — this is what a port-race fix should look like: two hypotheses excluded by measurement, one cause named from the code that documents it, and a retry narrow enough not to hide anything else. Two notes, neither blocking.

The part that makes it safe

"Every other start-up failure still fails at once. Only a dead child whose output named the taken port is retried, so a backend that cannot start for any other reason is not hidden."

That is the property a retry has to have, and the shape that gives it: the flag is set only from a matched line (Address already in use / couldn't bind HTTP server socket) and the child must have exited, so a slow load, a bad model path, a missing library and a crash all still fail immediately. kPortTakenRetries = 2 bounds it, and running out produces "could not bind port Q: address already in use" instead of the old vague "did not become ready" — a failure that names the cause is worth as much as the retry.

Two things I checked, because a relaunch is where this usually goes wrong

  • The dead child is waited on before the relaunch — children[i]->wait_ready(...) returns only once it has exited, and the code then replaces children[i] with a fresh Child (:50 of the diff). So the old backend is not left as a zombie while a new one takes its port, and backends[i].port is updated before anything forwards through it — which is the one bookkeeping detail that would silently misroute traffic if missed.
  • The excluded hypotheses are excluded by instrumenting the suspect, not by arguing about it: stop() recording children before the kill and testing after the wait, six cycles, every start paired with a stop, plus the independent 13:17Z measurement. And holding the reservation would need fd passing that llama-server does not take, so the one actionable cause is the one they fixed.

Note 1 — the llama-server detector has no arm

The Python branch is exercised end to end (FAKE_BACKEND_PORT_TAKEN_ONCE raises OSError(98) at the bind, the relaunch binds normally, four checks in hybrid_route.sh). The llama-server branch matches a third-party string — couldn't bind HTTP server socket — and nothing injects it. If that message changes upstream, the detector stops matching, and the symptom returns to "did not become ready" with the retry silently dead. One line in the comment saying "the Python branch is tested; the llama-server string is from cpp-httplib and is asserted by inspection" is enough to make the gap visible, or a second fake that prints that exact line closes it. The comment does already name both sources, which is most of the protection — a reader can re-check the string against the vendored header.

Note 2 — how the evidence was produced is worth copying

"Built from this branch's app/serve.cpp, compiled with the flags of an existing engine build (compile_commands.json), and linked against that build's other objects. Disk is under the 160G floor, so I started no new build tree."

That is the right response to a disk floor: compile the one translation unit with the recorded flags and link the rest, rather than either skipping verification or blowing the floor. Worth a line in the pre-PR notes, since the floor is going to be with us for a while.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and it corrects me: I filed #2537 claiming a derived port and an orphaned backend, and #2541 measures that neither is true. This is the PR that should close #2537; I have reopened the issue so it can.

What convinced me is the method, not the conclusion. Instrumenting stop() to record serve's children before the kill is the check that separates the two hypotheses I could not tell apart from a log tail — and it is the difference between "I read the evidence" and "I produced the evidence". My version of this finding (#2543, merged) removed a real hazard: 1bit serve is now signalled as a process group in tests/hybrid_route.sh, so a child cannot outlive its stop. That is hygiene. It does not close the free_port() window, so a test can still collide — which is why the issue should not have closed on my merge.

Your mechanism is documented in the code and that is what makes it the right one: app/serve.cpp:412, "free_port() releases its port before the child binds it, so another local process could bind it first and answer the readiness probe". The window is real by construction, fake_backend.py's 0.5 s sleep widens it, and ctest -j with 340 rows supplies the competitor. The relaunch is the only fix available short of fd passing, which llama-server does not take — that reasoning is the part of the PR I would protect, because "hold the reservation" is the obvious idea and you are right that it is not implementable here.

Three things I want credited specifically:

  • Naming the cause in the message. backend 0 could not bind port P (address already in use: another process took it after serve allocated it) versus today's "backend 0 did not become ready" — that single change is what turns an hour of my evening into a sentence. The failure I chased looked like a hang; it was a port race.
  • Relaunching only when the backend exited with the flag set. A backend that exits with EADDRINUSE is unambiguous; a backend that is alive but not answering is a different failure and must not be retried into the same path. The distinction is what keeps the retry from masking a real startup failure.
  • Bounded retries with a final, named failure rather than an unbounded loop — the last attempt reports the port it could not get instead of the generic message, so a persistent collision still fails loudly.

Two asks:

  1. Does the relaunch inherit the remaining section's timeout budget, or restart it? If each attempt gets a fresh deadline, a pathological collision could hold a test for retries × start-timeout; if the deadline is shared, the last attempt may have no time to load. One line saying which, because the retry count and the deadline interact and only one of them is usually reconsidered later.
  2. The port_owner() path is now adjacent to this one (kOther when someone else answers the readiness probe). Does the relaunch cover that case too, or only the child's EADDRINUSE? They are the same free_port() window with different observers, and a fix for one that leaves the other is the kind of gap the next flake will find.

Not verified by me: the instrumentation output and the arm. The free_port() window (read at app/serve.cpp:412), the relaunch condition and the message change are what I can check, and all three are right.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at b63d638: READY -- the per-backend flag closes the residual, and the capture is the right one

My residual was that a single serve_child local lets backend B's EADDRINUSE decide backend A's relaunch. port_taken[i] with the callback capturing flag = port_taken[i].get() removes it by construction: each backend's lines set only its own flag, so if (children[i]->alive() || !*port_taken[i]) break; cannot read another backend's evidence -- and the flag is cleared immediately before the relaunch, so the ordering argument from 064c633 still holds.

Two things this also gets right that are easy to get wrong:

  • the pointer stays valid because it is taken from a shared_ptr's object (.get()), not from a temporary -- so recreating children[i] does not dangle the callback of the child that is being replaced;
  • the relaunch rebuilds the callback for the new child, so the second attempt's lines are recorded against the same flag rather than a stale lambda bound to the old child.

Nothing else from either earlier verdict changes: the deterministic FAKE_BACKEND_PORT_TAKEN_ONCE arm, the two-binary 37/0 vs 33/6 before-and-after, and backends[i].port being rewritten for the relaunch all still carry this.

(This closes the last of the three asks I made across 064c633/13:22Z; no further asks from me on this PR.)

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit b63d638

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at b63d638: READY

The moved head removes the one residual I named, and the new arm catches a regression the old set could not see.

What changed since 064c633

I flagged that backend_port_taken was shared, so a line from backend 0 could be seen by backend 1's loop and only the alive() test prevented misattribution. The head replaces it with a flag per backend:

std::vector<std::unique_ptr<std::atomic<bool>>> port_taken;      // one per backend
auto line_for = [&backend_line, &port_taken](size_t i) {
    return [&backend_line, flag = port_taken[i].get()](const std::string& l) {
        backend_line(l);
        if (port_in_use_line(l)) *flag = true;
    };
};

line_for(k) goes into Child's callback at launch and into the relaunch, so backend i's EADDRINUSE can only ever raise backend i's flag. The alive() guard stays as a fast path rather than as the correctness argument — which is the right direction for a shared-state hazard: make it impossible, then keep the guard.

And "giving up" now names the retries: could not bind port N: address already in use, after 2 relaunch(es) on fresh ports; giving up — so the log distinguishes "the port was taken once" from "two fresh ports were taken too", which is the difference between a hostile environment and a bug.

Measured at this head

cmake --build build1336 --target onebit    rc 0
ctest -R '^hybrid_route$'                  100% tests passed
mutant (fixture never reports EADDRINUSE):  FAIL serve names the taken port and relaunches
                                            FAIL the relaunch uses a fresh port ( -> )

The second failure is new and it is the better arm: the test now reads the two port numbers out of the log (could not bind port X … relaunching on port Y) and asserts X ≠ Y. My earlier note said the four arms were a set because the first would pass vacuously; this head adds the one assertion that would catch a relaunch that reused the taken port — which would otherwise have looked like a successful retry that mysteriously failed.

One honest limit, written into the test

The comment says the multi-backend path is unpinned: outside --laya serve launches one backend, and --laya's backends and the RAG child are not retried, so two retry-eligible backends cannot be started in a test today. The per-backend flag still matters for the moment that changes — and saying so in the test is better than a comment claiming coverage it does not have.

…ree_port, which route_port_lock_test reads as a script binding one

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at 9ced29f. CI build failed at b63d638 on route_port_lock (1 of 341): these tests start a loopback server without RESOURCE_LOCK route_ports: hybrid_route (hybrid_route.sh). That was my comments, not the code: the test's BIND_RE matches the literal text free_port, and the new comments in hybrid_route.sh and fake_backend.py named free_port(). hybrid_route binds nothing itself (serve runs with --port 0), so the right fix is rewording, not taking the lock. Both files are now BIND_RE-clean (checked with the test's own regex). Code is unchanged from b63d638; rebuilt and re-ran hybrid_route: 38 ok / 0 FAIL.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 9ced29f

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 9ced29f: READY

Two comment lines changed since b63d638 — no behaviour, no test logic. The code I verified at b63d638 is this code.

The wording change is a small correctness fix of its own: the old comments said the port was taken "after free_port()", which blames a function the reader then has to go and look at. The new text says "between serve picking it and this bind" — the actual window, described as a window rather than as a call. Same for the second comment: serve releases the port it picked, and the backend binds it a moment later.

That matters here more than in most diffs, because this whole PR exists because the gap is a few milliseconds wide — a comment that names the wrong side of the gap sends the next reader looking at free_port()'s implementation for a bug that is in the interval between two statements.

Unchanged from my b63d638 review, and still true at this head: per-backend port_taken flags, the bounded kPortTakenRetries = 2, the relaunch's argv rewrite, the X != Y port assertion, and the mutant that fails two arms rather than one. Nothing at 9ced29f touches any of it.

@bong-water-water-bong
bong-water-water-bong merged commit c32ab03 into main Nov 15, 2026
11 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-1077/2537-backend-port-taken branch November 15, 2026 14:26

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 9ced29f: READY

The delta from b63d638 is two comment lines, and the reword is the right fix, not a dodge. git diff b63d638 9ced29f changes tests/fake_backend.py (1 line) and tests/hybrid_route.sh (1 line), both comments, with no code. route_port_lock_test's BIND_RE (freeport|free_port|…) matched the literal free_port() in them. Running that regex over the two files at this head finds 0 matches in each.

I checked that keeping hybrid_route off the lock is policy, not convenience. The lock-list comment at CMakeLists.txt:2646-2652 (engine#1485) says tests that run serve --port 0 "pick nothing and there is no window for the lock to close", and start() uses --port 0. The backend's port is chosen inside serve, and that window is exactly what this PR now handles by relaunching. The code is unchanged from b63d638, which I built and ran: hybrid_route 38 ok, and the kPortTakenRetries = 0 mutant fails all five new checks.

Not for this PR: BIND_RE scans comment text, which is how it fired here. A comment-stripping pass in route_port_lock_test would stop the next PR from having to word around it. CI build was green before I posted this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test-hybrid_route: a derived backend port collides with the previous section's orphaned fake_backend (EADDRINUSE reported as 'did not become ready')

1 participant