You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
tests: 19 of 24 scripts that pick a port with bind(0) still lose the release-then-bind race; make serve --port 0 report the port it bound #1485
Follow-up to #1444 (closed by #1446). The five scripts there now retry a collision they already named. The rest of the inventory, audited rather than quoted (I repeated the number from the closing comment and it was wrong: parallel_args.sh and metrics_serve.sh already carry the detection), is 24 scripts that pick a front port with the same bind(0)-and-release one-liner:
The 11 without detection fail as a generic "serve did not answer /health", which reads as a defect in whatever the test checks.
The two shapes, and why I would take the second
One shared shell helper with the retry loop, sourced by the 19 (17 shell + 1 python + recipes_route/smoke_serve with several picks each). Cheapest to write, keeps the race and therefore keeps retrying.
serve --port 0 + print the port it bound. Removes the race instead of retrying around it: nothing is picked and released, so there is no window. app/serve.cpp already gets the bound port from srv.bind_to_port() — it returns the port, and the two "ready on" lines print o.port instead. With --port 0 today the server binds an ephemeral port and then prints port 0, so the capability is one line away but unusable.
Preference: (2) as the mechanism plus one small shared parser for the ready line, then migrate the 19 in batches. That kills the flake class instead of padding it, and it also fixes the backend ports, which serve allocates with free_port() for the same reason.
Acceptance
serve -m X --port 0 binds an ephemeral port and its ready line names the bound port (both serve paths: the laya-models one at app/serve.cpp:2026 and the plain one at :2735), and bind_to_port's negative return is still reported as cannot listen on host:port.
one script migrated as the reference, showing the parse, so the 18 after it are mechanical;
no script loses a test because of a port it did not choose.
Claimed by lane pi-i669 — it is the direct continuation of #1444, and I would rather land the engine capability plus one reference script than open a 19-file refactor in one go. Say if you want the shared-helper shape instead and I will not build (2).
Claiming, and the first PR is open: #1490 at e4558f9e60 — serve --port 0 now reports the port the kernel bound (both bind sites) and tests/greedy_shortcut.sh is migrated as the reference, so the remaining 18 are mechanical.
Two things worth having on this issue rather than only in the PR:
bind_to_port() returns bool, not the port — my first version of the patch assumed otherwise and would have printed 1 as the port. The port comes from bind_to_any_port(), which shares the bind_internal(host, 0, …) getsockname path. Recorded because the next person to touch this will assume the same thing.
Remaining work after #1490 lands: the 8 detect-and-exit scripts (drop their detection and their pick, use --port 0), then the 11 with no detection, in batches of a few. smoke_serve.sh needs care — one of its three picks is deliberate (it holds a port to check that serve refuses it), so it keeps that one.
Every test script that runs 1bit serve now takes its port from serve's own ready line. The last one is in #1588 (npu_repack_path — its pick fed a flag nothing consumed; there is no curl in that file at all), so this issue is down to its two permanent exceptions, and I would close it once #1588 is reviewed.
Migrated across the batches: the five from #1446 (retry, before --port 0 existed) plus ten more from --port 0 + serve_port_wait — greedy_shortcut, smoke_serve (two of three picks), parallel_args, hrx_runtime_route, metrics_serve, watchdog_serve, pm_route, prism_route, recipes_route, serve_e2e, anchor_rerun, mem_admit_route, laya_gguf_route, laya_models_e2e, laya_route_e2e, npu_repack_path.
Permanently out, and now written into the lock list's comment so nobody reopens the sweep:hrx_cpu_weight_parity.sh and kvshare_fd_handoff.py both run llama-server, not 1bit serve — their port is not ours to choose, and kvshare_fd_handoff.py's free_port() feeds --port to that binary.
One correction to this issue's own title for the record: it says "19 of 24 scripts with no collision check". Audited, the split was 5 retry / 8 detect-and-exit / 11 with no detection (13 picks), and parallel_args and metrics_serve already carried the detection the title implied they lacked. The audit table in the first comment is the number to trust.
Status audit at origin/main, per script rather than from the batch descriptions — the inventory in the issue body can now be retired, and I think this can close.
Every failure in the tables above was "the test picks a front port with bind(0), releases it, and serve binds later". That shape is gone from every script
that starts 1bit serve. What still contains a bind(("127.0.0.1", 0)) is exactly nine scripts, and each is one of three deliberate cases:
script picks serve_port_wait llama-server refs
hrx_cpu_weight_parity 1 0 3 llama-server owns the port
pm_laya_route 1 0 1 llama-server + model-only test
pm_route 1 2 8 backends on fixed ports; reads serve's port back; holds the lock
smoke_serve 1 4 4 reads serve's port back (4 waits); holds the lock
device_route 1 0 3 retry (#1446); holds the lock
hrx_cwsr_route 1 0 2 retry (#1446); holds the lock
hybrid_route 1 0 4 retry (#1446); holds the lock
hybrid_kvshare_route 1 0 2 retry (#1446); holds the lock
hybrid_e2e 1 0 8 retry (#1446); model-only
Read as a rule: no 1bit serve front port is picked-and-released anywhere in the tree. Either the port belongs to llama-server (not ours to choose),
or the script reads back the port serve reported, or it is one of the five that explicitly retry a collision and hold RESOURCE_LOCK route_ports.
The scripts that left the tables entirely, by batch: anchor_rerun, serve_e2e, mem_admit_route, laya_gguf_route, laya_models_e2e, laya_route_e2e, npu_repack_path (batches 3-6), plus hrx_runtime_route, metrics_serve, parallel_args, prism_route, recipes_route, watchdog_serve, greedy_shortcut in the earlier sweeps — and smoke_serve moved from "no detection at all" to a script that reads the bound port and keeps
its lock.
Which means the ask in the title is delivered: serve --port 0 reports the port it bound, and every caller that needed it now reads it back. The
residual picks are llama-server's ports and the retry set #1446 accepted.
Two things worth keeping from the exercise, both of which the batches got right: the census (tests/route_port_lock_test.py) descends with the lock list in the
same commit as each migration (15 → 12 → 10 → 9), and the one script that fell outside every guard — anchor_rerun, unconfigured, so not in the census at all
— was migrated before being wired in, so it arrived without a window.
Correcting one word in my own audit above, because #1602 found the case that refutes it. I wrote that "no 1bit serve front port is picked-and-released
anywhere in the tree". The table under it was scoped to tests/*.sh, and that part is right — but the sentence was not, and engine #1602 is the
counter-example:
a pick-and-release one hop from the script the census reads, invisible to a .sh-text sweep because the bind is in the Python driver. #1602 fixes it the right
way — adds the test to _onebit_route_tests so it takes RESOURCE_LOCK route_ports — and leaves the floor at 9, because found counts scripts that match BIND_RE and this one's script does not.
So the accurate closing statement is: no 1bit serve front port is picked-and-released in any tests/*.sh, and the one instance in a driver is being fixed
now. If the issue stays open for the driver layer, the next step is the same audit one level down — grep -l free_port tests/*.py and check each caller's
lock, which is exactly how #1602 was found.
The driver-layer audit finished, and it found the second caller — so the remaining work in this issue is now exactly one line, not a class to be explored:
grep -l free_port tests/*.py
kvshare_fd_handoff.py <- two callers
route_port_lock_test.py <- the census itself (the match is its own regex, not a bind)
callers of kvshare_fd_handoff.py
tests/hybrid_kvshare_e2e.sh being fixed in #1602
tests/kvshare_orphan_test.sh imports the driver at :34, starts an HRX llama-server (its own comment at :17) -> no lock today
There are no other Python pickers in tests/, so the "one hop from the script the census reads" class has exactly two members and #1602 covers one of them. kvshare_orphan_test wants the same one-line addition to _onebit_route_tests — the precedent is hrx_cpu_weight_parity, which stays in that list because it
starts llama-server itself: that port is not ours to choose, but it is ours to serialise.
So, with the correction above, the accurate state of this issue is: no 1bit serve front port is picked-and-released in any tests/*.sh; the two driver-mediated
pickers are hybrid_kvshare_e2e (PR #1602) and kvshare_orphan_test (one line, same fix). That is the whole remainder.
Retracting the "two members" line from my driver audit two comments up — it is one member. The second caller of kvshare_fd_handoff.py is not a port picker:
tests/kvshare_orphan_test.sh:34 import kvshare_fd_handoff as k
tests/kvshare_orphan_test.sh:35 subprocess.Popen([sys.executable, "-c", "import time; time.sleep(120)"], preexec_fn=k.die_with_parent, …)
It imports the driver for die_with_parent (a process-group helper) and starts a sleeping Python child — no server, no port, and free_port() is only called
from main() (:112), which that test never runs. So:
With that, the accurate remainder of this issue is nothing: no 1bit serve front port is picked-and-released in any tests/*.sh, and the single
driver-mediated picker is being fixed now. My earlier "anywhere in the tree" sentence was still wrong and #1602 is still the counter-example — it is just the only one.
Census re-measured on main 8ff2814e — the table in this issue is stale by about twelve scripts, and the remaining work is two PRs plus the one this comment's author just opened.
hrx_runtime_route parallel_args (they inspect the fake backend's record and never curl the front port), npu_repack_path ("nothing here calls the server")
--port 0 with a local read-back
1
greedy_shortcut — and its sed is end-anchored, so it matches only serve's plain ready line
So shape 2 won, and it is nearly finished — serve --port 0 already reports the bound port (app/serve.cpp:2004 binds bind_to_any_port when the flag is 0), and no shared retry loop was ever needed.
The two scripts no open PR covers are in #1663: hrx_cpu_weight_parity.sh (drops free_port()) and greedy_shortcut.sh (drops its local end-anchored sed for the shared helper — under a laya flag that parser returns empty and the script fails as "serve never reported a port", which is the #1523/#1525 finding). It also adds tests/serve_port_test.sh, which pins both read-backs (the serve line's three shapes, an IPv6 host, and llama-server's own srv http_bind: listening on line) — script-based, so it runs in CI with no device, and restoring the end-anchored pattern fails two of its checks.
Suggested close condition: this issue is done when #1607 and #1637 merge and #1663 is in; nothing else in the inventory still picks a port and releases it.
Closing: every test script that runs a server now takes the port the server reports
Both halves are merged, so the pick-and-release race this issue was filed about no longer exists in the test suite.
1bit serve — --port 0 (llama.cpp#230 gave the ready line the bound port) plus tests/serve_port.sh's serve_port_wait, which reads it back and fails in a tenth of a second when the process is gone:
family
what happened
the five that already detected the collision (#1446)
first given a bounded re-pick, then migrated to --port 0 and the retry deleted — a retry was the right stopgap only while the race existed
llama.cpp#264's sibling work (#1663) read that server's own listening on http://… line, so hrx_cpu_weight_parity and kvshare_fd_handoff.py take their port the same way
Still picking a port, deliberately, and each says why:smoke_serve keeps one squatter (it checks that serve refuses a
held port — the case --port 0 cannot cover); print_backend/template_pin/lemonade_hrx use fixed 43xxx ports because
the number is part of what the row asserts, and --print-backend does not bind at all; bench_selftest binds through tools/bench.py.
The census moved with every batch._onebit_route_tests now names exactly the tests that still choose their port up
front (6 bare, 3 configuration-dependent), and the census floor went 15 → 5 as scripts stopped matching the pattern. Every
migration PR carried its lock-list change in the same commit — that is the lesson #1537's CI failure taught, and after it
the census never went red again.
Three traps found by doing it rather than reading it, worth having in the record for whoever migrates a family next:
A parse anchored on $ matched only one of three ready-line shapes (tests/serve_port.sh: parse all three ready-line shapes, not just the unsuffixed one (fixes review finding on #1525) #1531): serve prints three, two with trailing
text. My first helper matched the unsuffixed one, which is exactly the shape my three callers happened to use — the tests
passed and the next (laya) caller would have failed. Taking the URL token whole and the port after its last colon
fixed it, and the arm was added.
The end-to-end run the owner asked for (one run once all issues are solved) is not mine to take — #1485 is about test
harnesses, and the suite it fixes is exercised by CI on every PR. Closing on that basis.
Activity
bong-water-water-bong commentedon Nov 8, 2026
Claiming, and the first PR is open: #1490 at
e4558f9e60—serve --port 0now reports the port the kernel bound (both bind sites) andtests/greedy_shortcut.shis migrated as the reference, so the remaining 18 are mechanical.Two things worth having on this issue rather than only in the PR:
recipes_routeandsmoke_servepick three each).parallel_argsandmetrics_servealready carried the detection.bind_to_port()returnsbool, not the port — my first version of the patch assumed otherwise and would have printed1as the port. The port comes frombind_to_any_port(), which shares thebind_internal(host, 0, …)getsockname path. Recorded because the next person to touch this will assume the same thing.Remaining work after #1490 lands: the 8 detect-and-exit scripts (drop their detection and their pick, use
--port 0), then the 11 with no detection, in batches of a few.smoke_serve.shneeds care — one of its three picks is deliberate (it holds a port to check that serve refuses it), so it keeps that one.serve: --port 0 reports the port the kernel gave it, so a test can st…
tests: three more scripts take a port serve reports instead of guessi…
tests: five more scripts take a port serve reports instead of guessin…
tests: serve_e2e and anchor_rerun take the port serve reports; the ce…
tests: mem_admit_route and laya_gguf_route take the port serve report…
tests: laya_models_e2e and laya_route_e2e take the port serve reports…
bong-water-water-bong commentedon Nov 8, 2026
Every test script that runs
1bit servenow takes its port from serve's own ready line. The last one is in #1588 (npu_repack_path— its pick fed a flag nothing consumed; there is nocurlin that file at all), so this issue is down to its two permanent exceptions, and I would close it once #1588 is reviewed.Migrated across the batches: the five from #1446 (retry, before
--port 0existed) plus ten more from--port 0+serve_port_wait—greedy_shortcut,smoke_serve(two of three picks),parallel_args,hrx_runtime_route,metrics_serve,watchdog_serve,pm_route,prism_route,recipes_route,serve_e2e,anchor_rerun,mem_admit_route,laya_gguf_route,laya_models_e2e,laya_route_e2e,npu_repack_path.Permanently out, and now written into the lock list's comment so nobody reopens the sweep:
hrx_cpu_weight_parity.shandkvshare_fd_handoff.pyboth runllama-server, not1bit serve— their port is not ours to choose, andkvshare_fd_handoff.py'sfree_port()feeds--portto that binary.One correction to this issue's own title for the record: it says "19 of 24 scripts with no collision check". Audited, the split was 5 retry / 8 detect-and-exit / 11 with no detection (13 picks), and
parallel_argsandmetrics_servealready carried the detection the title implied they lacked. The audit table in the first comment is the number to trust.tests: npu_repack_path takes a port nobody uses; census moves for the…
bong-water-water-bong commentedon Nov 9, 2026
Status audit at
origin/main, per script rather than from the batch descriptions — the inventory in the issue body can now be retired, and I think this can close.Every failure in the tables above was "the test picks a front port with
bind(0), releases it, andservebinds later". That shape is gone from every scriptthat starts
1bit serve. What still contains abind(("127.0.0.1", 0))is exactly nine scripts, and each is one of three deliberate cases:Read as a rule: no
1bit servefront port is picked-and-released anywhere in the tree. Either the port belongs tollama-server(not ours to choose),or the script reads back the port
servereported, or it is one of the five that explicitly retry a collision and holdRESOURCE_LOCK route_ports.The scripts that left the tables entirely, by batch:
anchor_rerun,serve_e2e,mem_admit_route,laya_gguf_route,laya_models_e2e,laya_route_e2e,npu_repack_path(batches 3-6), plushrx_runtime_route,metrics_serve,parallel_args,prism_route,recipes_route,watchdog_serve,greedy_shortcutin the earlier sweeps — andsmoke_servemoved from "no detection at all" to a script that reads the bound port and keepsits lock.
Which means the ask in the title is delivered:
serve --port 0reports the port it bound, and every caller that needed it now reads it back. Theresidual picks are llama-server's ports and the retry set #1446 accepted.
Two things worth keeping from the exercise, both of which the batches got right: the census (
tests/route_port_lock_test.py) descends with the lock list in thesame commit as each migration (15 → 12 → 10 → 9), and the one script that fell outside every guard —
anchor_rerun, unconfigured, so not in the census at all— was migrated before being wired in, so it arrived without a window.
bong-water-water-bong commentedon Nov 9, 2026
Correcting one word in my own audit above, because #1602 found the case that refutes it. I wrote that "no
1bit servefront port is picked-and-releasedanywhere in the tree". The table under it was scoped to
tests/*.sh, and that part is right — but the sentence was not, and engine #1602 is thecounter-example:
a pick-and-release one hop from the script the census reads, invisible to a
.sh-text sweep because the bind is in the Python driver. #1602 fixes it the rightway — adds the test to
_onebit_route_testsso it takesRESOURCE_LOCK route_ports— and leaves the floor at 9, becausefoundcounts scripts that matchBIND_REand this one's script does not.So the accurate closing statement is: no
1bit servefront port is picked-and-released in anytests/*.sh, and the one instance in a driver is being fixednow. If the issue stays open for the driver layer, the next step is the same audit one level down —
grep -l free_port tests/*.pyand check each caller'slock, which is exactly how #1602 was found.
bong-water-water-bong commentedon Nov 9, 2026
The driver-layer audit finished, and it found the second caller — so the remaining work in this issue is now exactly one line, not a class to be explored:
There are no other Python pickers in
tests/, so the "one hop from the script the census reads" class has exactly two members and #1602 covers one of them.kvshare_orphan_testwants the same one-line addition to_onebit_route_tests— the precedent ishrx_cpu_weight_parity, which stays in that list because itstarts
llama-serveritself: that port is not ours to choose, but it is ours to serialise.So, with the correction above, the accurate state of this issue is: no
1bit servefront port is picked-and-released in anytests/*.sh; the two driver-mediatedpickers are
hybrid_kvshare_e2e(PR #1602) andkvshare_orphan_test(one line, same fix). That is the whole remainder.bong-water-water-bong commentedon Nov 9, 2026
Retracting the "two members" line from my driver audit two comments up — it is one member. The second caller of
kvshare_fd_handoff.pyis not a port picker:It imports the driver for
die_with_parent(a process-group helper) and starts a sleeping Python child — no server, no port, andfree_port()is only calledfrom
main()(:112), which that test never runs. So:hybrid_kvshare_e2e, fixed by ci: hybrid_kvshare_e2e picks its port in its driver, so it needs the route-port lock (engine#1485) #1602 (READY);kvshare_orphan_testcorrectly holds no route lock, and I have withdrawn my issue tests: kvshare_orphan_test picks its port in its driver and holds no route lock (same window, one hop from the census) #1603 and the line I suggested adding to ci: hybrid_kvshare_e2e picks its port in its driver, so it needs the route-port lock (engine#1485) #1602.With that, the accurate remainder of this issue is nothing: no
1bit servefront port is picked-and-released in anytests/*.sh, and the singledriver-mediated picker is being fixed now. My earlier "anywhere in the tree" sentence was still wrong and #1602 is still the counter-example — it is just the
only one.
ci: hybrid_kvshare_e2e picks its port in its driver, so it belongs in…
bong-water-water-bong commentedon Nov 9, 2026
Census re-measured on main
8ff2814e— the table in this issue is stale by about twelve scripts, and the remaining work is two PRs plus the one this comment's author just opened.--port 0andserve_port_wait)metrics_serve pm_route prism_route recipes_route watchdog_serve anchor_rerun laya_gguf_route laya_models_e2e laya_route_e2e mem_admit_route serve_e2e smoke_serve--port 0, nothing to read backhrx_runtime_route parallel_args(they inspect the fake backend's record and never curl the front port),npu_repack_path("nothing here calls the server")--port 0with a local read-backgreedy_shortcut— and itssedis end-anchored, so it matches only serve's plain ready linedevice_route hrx_cwsr_route hybrid_e2e hybrid_kvshare_route hybrid_route pm_laya_route(#1607),kvshare_fd_handoff.py(#1637),hrx_cpu_weight_paritySo shape 2 won, and it is nearly finished —
serve --port 0already reports the bound port (app/serve.cpp:2004bindsbind_to_any_portwhen the flag is 0), and no shared retry loop was ever needed.The two scripts no open PR covers are in #1663:
hrx_cpu_weight_parity.sh(dropsfree_port()) andgreedy_shortcut.sh(drops its local end-anchoredsedfor the shared helper — under a laya flag that parser returns empty and the script fails as "serve never reported a port", which is the #1523/#1525 finding). It also addstests/serve_port_test.sh, which pins both read-backs (the serve line's three shapes, an IPv6 host, and llama-server's ownsrv http_bind: listening online) — script-based, so it runs in CI with no device, and restoring the end-anchored pattern fails two of its checks.Suggested close condition: this issue is done when #1607 and #1637 merge and #1663 is in; nothing else in the inventory still picks a port and releases it.
tests: the handoff driver's provider dies with it too, and its fork i…
tests: read back the port llama-server bound, and migrate the last tw…
tests: read back the port llama-server bound, and migrate the last tw…
bong-water-water-bong commentedon Nov 9, 2026
Closing: every test script that runs a server now takes the port the server reports
Both halves are merged, so the pick-and-release race this issue was filed about no longer exists in the test suite.
1bit serve—--port 0(llama.cpp#230 gave the ready line the bound port) plustests/serve_port.sh'sserve_port_wait, which reads it back and fails in a tenth of a second when the process is gone:--port 0and the retry deleted — a retry was the right stopgap only while the race existed--port 0+serve_port_waitgreedy_shortcut,smoke_serve(2 of 3 picks),parallel_args,hrx_runtime_route,metrics_serve,watchdog_serve,pm_route,prism_route,recipes_route,serve_e2e,anchor_rerun,mem_admit_route,laya_gguf_route,laya_models_e2e,laya_route_e2e,npu_repack_pathllama-serverscriptslistening on http://…line, sohrx_cpu_weight_parityandkvshare_fd_handoff.pytake their port the same wayStill picking a port, deliberately, and each says why:
smoke_servekeeps one squatter (it checks that serve refuses aheld port — the case
--port 0cannot cover);print_backend/template_pin/lemonade_hrxuse fixed43xxxports becausethe number is part of what the row asserts, and
--print-backenddoes not bind at all;bench_selftestbinds throughtools/bench.py.The census moved with every batch.
_onebit_route_testsnow names exactly the tests that still choose their port upfront (6 bare, 3 configuration-dependent), and the census floor went 15 → 5 as scripts stopped matching the pattern. Every
migration PR carried its lock-list change in the same commit — that is the lesson #1537's CI failure taught, and after it
the census never went red again.
Three traps found by doing it rather than reading it, worth having in the record for whoever migrates a family next:
unmatched name is a failure now, which is how the device-half darkness below was found at all.
$matched only one of three ready-line shapes (tests/serve_port.sh: parse all three ready-line shapes, not just the unsuffixed one (fixes review finding on #1525) #1531):serveprints three, two with trailingtext. My first helper matched the unsuffixed one, which is exactly the shape my three callers happened to use — the tests
passed and the next (laya) caller would have failed. Taking the URL token whole and the port after its last colon
fixed it, and the arm was added.
serve_port_waitreturns 1 while theprocess is still alive, so the failure branch was precisely the case that left a live
1bit serveholding the device.That is the fleet's own orphan shape; the trap now goes in before the wait.
The end-to-end run the owner asked for (one run once all issues are solved) is not mine to take — #1485 is about test
harnesses, and the suite it fixes is exercised by CI on every PR. Closing on that basis.
tests: FLM 1.0.12's two device-free server faults — client hangup and…
serve: the in-process embedding route announces the port it bound
tests: register the two #1779 serve tests where CI can see them (skip…