Skip to content

tests: 19 of 24 scripts that pick a port with bind(0) still lose the release-then-bind race; make serve --port 0 report the port it bound #1485

Closed

Description

@bong-water-water-bong

Follow-up to #1444 (closed by #1446). The five scripts there now retry a collision they already named. The rest of the inventory, audited rather than quoted (I repeated the number from the closing comment and it was wrong: parallel_args.sh and metrics_serve.sh already carry the detection), is 24 scripts that pick a front port with the same bind(0)-and-release one-liner:

state scripts
retry (merged, #1446) device_route, hrx_cwsr_route, hybrid_e2e, hybrid_kvshare_route, hybrid_route
detect and exit 1 (10 picks) hrx_runtime_route, metrics_serve, parallel_args, pm_laya_route, pm_route, prism_route, recipes_route (3 picks), watchdog_serve
no detection at all (13 picks) anchor_rerun, greedy_shortcut, hrx_cpu_weight_parity, kvshare_fd_handoff.py, laya_gguf_route, laya_models_e2e, laya_route_e2e, mem_admit_route, npu_repack_path, serve_e2e, smoke_serve (3 picks)

The 11 without detection fail as a generic "serve did not answer /health", which reads as a defect in whatever the test checks.

The two shapes, and why I would take the second

  1. One shared shell helper with the retry loop, sourced by the 19 (17 shell + 1 python + recipes_route/smoke_serve with several picks each). Cheapest to write, keeps the race and therefore keeps retrying.
  2. serve --port 0 + print the port it bound. Removes the race instead of retrying around it: nothing is picked and released, so there is no window. app/serve.cpp already gets the bound port from srv.bind_to_port() — it returns the port, and the two "ready on" lines print o.port instead. With --port 0 today the server binds an ephemeral port and then prints port 0, so the capability is one line away but unusable.

Preference: (2) as the mechanism plus one small shared parser for the ready line, then migrate the 19 in batches. That kills the flake class instead of padding it, and it also fixes the backend ports, which serve allocates with free_port() for the same reason.

Acceptance

  • serve -m X --port 0 binds an ephemeral port and its ready line names the bound port (both serve paths: the laya-models one at app/serve.cpp:2026 and the plain one at :2735), and bind_to_port's negative return is still reported as cannot listen on host:port.
  • one script migrated as the reference, showing the parse, so the 18 after it are mechanical;
  • no script loses a test because of a port it did not choose.

Claimed by lane pi-i669 — it is the direct continuation of #1444, and I would rather land the engine capability plus one reference script than open a 19-file refactor in one go. Say if you want the shared-helper shape instead and I will not build (2).

Activity

  1. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming, and the first PR is open: #1490 at e4558f9e60 — serve --port 0 now reports the port the kernel bound (both bind sites) and tests/greedy_shortcut.sh is migrated as the reference, so the remaining 18 are mechanical.

    Two things worth having on this issue rather than only in the PR:

    1. My audit above is the authority, not the closing comment on tests: five serve route tests fail on a front-port collision instead of retrying (check-then-use race, CI flake) #1444 — that one said 19 scripts had no collision check; the real split is 5 retry / 8 detect-and-exit / 11 with no detection at all (13 picks between them, because recipes_route and smoke_serve pick three each). parallel_args and metrics_serve already carried the detection.
    2. bind_to_port() returns bool, not the port — my first version of the patch assumed otherwise and would have printed 1 as the port. The port comes from bind_to_any_port(), which shares the bind_internal(host, 0, …) getsockname path. Recorded because the next person to touch this will assume the same thing.

    Remaining work after #1490 lands: the 8 detect-and-exit scripts (drop their detection and their pick, use --port 0), then the 11 with no detection, in batches of a few. smoke_serve.sh needs care — one of its three picks is deliberate (it holds a port to check that serve refuses it), so it keeps that one.

  2. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Every test script that runs 1bit serve now takes its port from serve's own ready line. The last one is in #1588 (npu_repack_path — its pick fed a flag nothing consumed; there is no curl in that file at all), so this issue is down to its two permanent exceptions, and I would close it once #1588 is reviewed.

    Migrated across the batches: the five from #1446 (retry, before --port 0 existed) plus ten more from --port 0 + serve_port_wait — greedy_shortcut, smoke_serve (two of three picks), parallel_args, hrx_runtime_route, metrics_serve, watchdog_serve, pm_route, prism_route, recipes_route, serve_e2e, anchor_rerun, mem_admit_route, laya_gguf_route, laya_models_e2e, laya_route_e2e, npu_repack_path.

    Permanently out, and now written into the lock list's comment so nobody reopens the sweep: hrx_cpu_weight_parity.sh and kvshare_fd_handoff.py both run llama-server, not 1bit serve — their port is not ours to choose, and kvshare_fd_handoff.py's free_port() feeds --port to that binary.

    One correction to this issue's own title for the record: it says "19 of 24 scripts with no collision check". Audited, the split was 5 retry / 8 detect-and-exit / 11 with no detection (13 picks), and parallel_args and metrics_serve already carried the detection the title implied they lacked. The audit table in the first comment is the number to trust.

  3. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Status audit at origin/main, per script rather than from the batch descriptions — the inventory in the issue body can now be retired, and I think this can close.

    Every failure in the tables above was "the test picks a front port with bind(0), releases it, and serve binds later". That shape is gone from every script
    that starts 1bit serve. What still contains a bind(("127.0.0.1", 0)) is exactly nine scripts, and each is one of three deliberate cases:

    script                   picks  serve_port_wait  llama-server refs
    hrx_cpu_weight_parity      1         0                3      llama-server owns the port
    pm_laya_route              1         0                1      llama-server + model-only test
    pm_route                   1         2                8      backends on fixed ports; reads serve's port back; holds the lock
    smoke_serve                1         4                4      reads serve's port back (4 waits); holds the lock
    device_route               1         0                3      retry (#1446); holds the lock
    hrx_cwsr_route             1         0                2      retry (#1446); holds the lock
    hybrid_route               1         0                4      retry (#1446); holds the lock
    hybrid_kvshare_route       1         0                2      retry (#1446); holds the lock
    hybrid_e2e                 1         0                8      retry (#1446); model-only
    

    Read as a rule: no 1bit serve front port is picked-and-released anywhere in the tree. Either the port belongs to llama-server (not ours to choose),
    or the script reads back the port serve reported, or it is one of the five that explicitly retry a collision and hold RESOURCE_LOCK route_ports.

    The scripts that left the tables entirely, by batch: anchor_rerun, serve_e2e, mem_admit_route, laya_gguf_route, laya_models_e2e,
    laya_route_e2e, npu_repack_path (batches 3-6), plus hrx_runtime_route, metrics_serve, parallel_args, prism_route, recipes_route,
    watchdog_serve, greedy_shortcut in the earlier sweeps — and smoke_serve moved from "no detection at all" to a script that reads the bound port and keeps
    its lock.

    Which means the ask in the title is delivered: serve --port 0 reports the port it bound, and every caller that needed it now reads it back. The
    residual picks are llama-server's ports and the retry set #1446 accepted.

    Two things worth keeping from the exercise, both of which the batches got right: the census (tests/route_port_lock_test.py) descends with the lock list in the
    same commit as each migration (15 → 12 → 10 → 9), and the one script that fell outside every guard — anchor_rerun, unconfigured, so not in the census at all
    — was migrated before being wired in, so it arrived without a window.

  4. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correcting one word in my own audit above, because #1602 found the case that refutes it. I wrote that "no 1bit serve front port is picked-and-released
    anywhere in the tree
    ". The table under it was scoped to tests/*.sh, and that part is right — but the sentence was not, and engine #1602 is the
    counter-example:

    tests/hybrid_kvshare_e2e.sh:54   python3 "$here/kvshare_fd_handoff.py" …
    tests/kvshare_fd_handoff.py:66   def free_port():  s.bind(("127.0.0.1", 0)); p = s.getsockname()[1]; s.close()
    

    a pick-and-release one hop from the script the census reads, invisible to a .sh-text sweep because the bind is in the Python driver. #1602 fixes it the right
    way — adds the test to _onebit_route_tests so it takes RESOURCE_LOCK route_ports — and leaves the floor at 9, because found counts scripts that match
    BIND_RE and this one's script does not.

    So the accurate closing statement is: no 1bit serve front port is picked-and-released in any tests/*.sh, and the one instance in a driver is being fixed
    now
    . If the issue stays open for the driver layer, the next step is the same audit one level down — grep -l free_port tests/*.py and check each caller's
    lock, which is exactly how #1602 was found.

  5. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The driver-layer audit finished, and it found the second caller — so the remaining work in this issue is now exactly one line, not a class to be explored:

    grep -l free_port tests/*.py
      kvshare_fd_handoff.py        <- two callers
      route_port_lock_test.py      <- the census itself (the match is its own regex, not a bind)
    
    callers of kvshare_fd_handoff.py
      tests/hybrid_kvshare_e2e.sh     being fixed in #1602
      tests/kvshare_orphan_test.sh    imports the driver at :34, starts an HRX llama-server (its own comment at :17) -> no lock today
    

    There are no other Python pickers in tests/, so the "one hop from the script the census reads" class has exactly two members and #1602 covers one of them.
    kvshare_orphan_test wants the same one-line addition to _onebit_route_tests — the precedent is hrx_cpu_weight_parity, which stays in that list because it
    starts llama-server itself: that port is not ours to choose, but it is ours to serialise.

    So, with the correction above, the accurate state of this issue is: no 1bit serve front port is picked-and-released in any tests/*.sh; the two driver-mediated
    pickers are hybrid_kvshare_e2e (PR #1602) and kvshare_orphan_test (one line, same fix). That is the whole remainder.

  6. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Retracting the "two members" line from my driver audit two comments up — it is one member. The second caller of kvshare_fd_handoff.py is not a port picker:

    tests/kvshare_orphan_test.sh:34   import kvshare_fd_handoff as k
    tests/kvshare_orphan_test.sh:35   subprocess.Popen([sys.executable, "-c", "import time; time.sleep(120)"], preexec_fn=k.die_with_parent, …)
    

    It imports the driver for die_with_parent (a process-group helper) and starts a sleeping Python child — no server, no port, and free_port() is only called
    from main() (:112), which that test never runs. So:

    With that, the accurate remainder of this issue is nothing: no 1bit serve front port is picked-and-released in any tests/*.sh, and the single
    driver-mediated picker is being fixed now. My earlier "anywhere in the tree" sentence was still wrong and #1602 is still the counter-example — it is just the
    only one.

  7. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Census re-measured on main 8ff2814e — the table in this issue is stale by about twelve scripts, and the remaining work is two PRs plus the one this comment's author just opened.

    state on main count scripts
    migrated (--port 0 and serve_port_wait) 12 metrics_serve pm_route prism_route recipes_route watchdog_serve anchor_rerun laya_gguf_route laya_models_e2e laya_route_e2e mem_admit_route serve_e2e smoke_serve
    --port 0, nothing to read back 3 hrx_runtime_route parallel_args (they inspect the fake backend's record and never curl the front port), npu_repack_path ("nothing here calls the server")
    --port 0 with a local read-back 1 greedy_shortcut — and its sed is end-anchored, so it matches only serve's plain ready line
    still picks and releases 8 device_route hrx_cwsr_route hybrid_e2e hybrid_kvshare_route hybrid_route pm_laya_route (#1607), kvshare_fd_handoff.py (#1637), hrx_cpu_weight_parity

    So shape 2 won, and it is nearly finished — serve --port 0 already reports the bound port (app/serve.cpp:2004 binds bind_to_any_port when the flag is 0), and no shared retry loop was ever needed.

    The two scripts no open PR covers are in #1663: hrx_cpu_weight_parity.sh (drops free_port()) and greedy_shortcut.sh (drops its local end-anchored sed for the shared helper — under a laya flag that parser returns empty and the script fails as "serve never reported a port", which is the #1523/#1525 finding). It also adds tests/serve_port_test.sh, which pins both read-backs (the serve line's three shapes, an IPv6 host, and llama-server's own srv http_bind: listening on line) — script-based, so it runs in CI with no device, and restoring the end-anchored pattern fails two of its checks.

    Suggested close condition: this issue is done when #1607 and #1637 merge and #1663 is in; nothing else in the inventory still picks a port and releases it.

  8. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closing: every test script that runs a server now takes the port the server reports

    Both halves are merged, so the pick-and-release race this issue was filed about no longer exists in the test suite.

    1bit serve — --port 0 (llama.cpp#230 gave the ready line the bound port) plus tests/serve_port.sh's
    serve_port_wait, which reads it back and fails in a tenth of a second when the process is gone:

    family what happened
    the five that already detected the collision (#1446) first given a bounded re-pick, then migrated to --port 0 and the retry deleted — a retry was the right stopgap only while the race existed
    ten more via --port 0 + serve_port_wait greedy_shortcut, smoke_serve (2 of 3 picks), parallel_args, hrx_runtime_route, metrics_serve, watchdog_serve, pm_route, prism_route, recipes_route, serve_e2e, anchor_rerun, mem_admit_route, laya_gguf_route, laya_models_e2e, laya_route_e2e, npu_repack_path
    llama-server scripts llama.cpp#264's sibling work (#1663) read that server's own listening on http://… line, so hrx_cpu_weight_parity and kvshare_fd_handoff.py take their port the same way

    Still picking a port, deliberately, and each says why: smoke_serve keeps one squatter (it checks that serve refuses a
    held port — the case --port 0 cannot cover); print_backend/template_pin/lemonade_hrx use fixed 43xxx ports because
    the number is part of what the row asserts, and --print-backend does not bind at all; bench_selftest binds through
    tools/bench.py.

    The census moved with every batch. _onebit_route_tests now names exactly the tests that still choose their port up
    front (6 bare, 3 configuration-dependent), and the census floor went 15 → 5 as scripts stopped matching the pattern. Every
    migration PR carried its lock-list change in the same commit — that is the lesson #1537's CI failure taught, and after it
    the census never went red again.

    Three traps found by doing it rather than reading it, worth having in the record for whoever migrates a family next:

    1. A filter typo selected zero cases and exited 0 (Pin llama.cpp dad19ad: lossless SwiGLU everywhere, MTP verify kernels on HRX #239): a name matching nothing ran nothing and reported success. An
      unmatched name is a failure now, which is how the device-half darkness below was found at all.
    2. A parse anchored on $ matched only one of three ready-line shapes (tests/serve_port.sh: parse all three ready-line shapes, not just the unsuffixed one (fixes review finding on #1525) #1531): serve prints three, two with trailing
      text. My first helper matched the unsuffixed one, which is exactly the shape my three callers happened to use — the tests
      passed and the next (laya) caller would have failed. Taking the URL token whole and the port after its last colon
      fixed it, and the arm was added.
    3. The cleanup trap installed after the failure path that exits (tests: laya_models_e2e and laya_route_e2e take the port serve reports; census moves with them (refs #1485) #1561's review): serve_port_wait returns 1 while the
      process is still alive, so the failure branch was precisely the case that left a live 1bit serve holding the device.
      That is the fleet's own orphan shape; the trap now goes in before the wait.

    The end-to-end run the owner asked for (one run once all issues are solved) is not mine to take — #1485 is about test
    harnesses, and the suite it fixes is exercised by CI on every PR. Closing on that basis.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions