Repository navigation
pm_route: a dead port is bound-and-held, not freeport()ed (the arm that failed on main at 50b68e76) - #2355
Conversation
main CI at 50b68e7 failed 'pm_route' on the arm "an unreachable executor: the PM says so and still answers". The cause is the arm's setup, not the feature: `dead_port=$(freeport)` binds port 0, reads the number and CLOSES the socket, so between that close and the PM's connect the kernel is free to hand the same port to anything else on the box -- and on a busy self-hosted runner something does, which makes the "unreachable" executor reachable and the assertion fail. Caught it in the act: a port from freeport() is re-bindable immediately after it returns. `deadport()` holds a socket that is bound and never `listen()`ed, for the life of the test. That gives both properties the arm needs and freeport() only gave the first: connect(127.0.0.1, port) -> ECONNREFUSED (a bound socket with no backlog refuses) bind(127.0.0.1, port) -> EADDRINUSE (nobody else can take it and 'revive' the executor) Measured: 0 of 20 trials could take the dead port (freeport() could be taken every time), and the whole `ctest -R '^pm_route$'` row passes with the change. Not a behaviour change: only the port the *test* declares dead, and the socket is killed in the existing cleanup trap with the other helpers.
PR Reviewer Guide 🔍(Review updated until commit ef3af6a)Here are some key observations to aid the review process:
|
|
VERDICT: READY This is the right fix for the right reason, and the diagnosis is the valuable part: The replacement is exactly "dead" in both senses: Two details I would tighten, neither blocking:
The Not verified by me: a run of |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: CHANGES -- the new helper is exactly right, the racy helper it replaces is still in the file, and the holder is not killed on the way out
The diagnosis and the replacement are both correct, and this is the sharpest form of the fix: "bound, never listen()ed, and held open for the life of the test... Here connect() gets ECONNREFUSED (a bound socket with no backlog refuses) and bind() gets EADDRINUSE, so the arm tests what it says." That is the whole subtlety in one sentence -- a listen()ed socket would accept and the arm would pass for the wrong reason, and a closed socket is a race. Refusing and un-bindable is the only state that makes "unreachable executor" deterministic.
Two things stop it from being the fix rather than one instance of it:
-
freeport()is still defined and still called five more times intests/pm_route.sh. The PR's own comment explains whyfreeport()is unsound -- "the kernel is free to hand that port to anything else on the box" -- so leaving it in place is leaving the trap armed for the next person who needs a port. Either those five sites get the same treatment, or one sentence says what kind they are (a port the test willbind()itself immediately is not the same hazard), so the next reader does not have to re-derive which uses are safe. -
The holder is never killed.
pids+=($!)collects the pid, but nothing in the difftraps orkills it, and the python sleeps for a fixed 600 s. So a test that fails, or exits early, leaves a python process holding a port for up to ten minutes -- one per run, and on a busy runner that is the same pollution the fix exists to prevent, just slower. Atrap 'kill ${pids[@]} 2>/dev/null' EXIT(plus INT/TERM) makes the hold bounded by the test, and thensleep 600is a safety net rather than the lifetime.
Nothing else to raise: waiting on the port file with sleep 0.02 up to 200 times is the right way to avoid a fixed sleep, and s.getsockname()[1] after bind(..., 0) is the only correct way to learn the port.
|
VERDICT: READY at The defect is in the arm's setup, and the body says so first
That sentence is the whole review method: a failing test is a claim about either the feature or the harness, and the first job is to find out which. Here the arm wants a port nothing is listening on, and it got one by: freeport() { python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1])'; }which binds to an ephemeral port, reads the number, and closes the socket. Between the close and the feature's own connect, the port is free for anyone — and on a box running a dozen parallel lanes, "anyone" is not hypothetical. So the arm is a time-of-check-to-time-of-use race whose window is exactly the gap the function creates. The fix is the right shape: hold the thing you are testing aboutBound-and-held means the port is occupied by the test deliberately, so "unreachable" is a property under the test's control rather than a hope about scheduling. That is the same discipline as #2232's Two asks
Why a 17-line harness fix earns a verdictBecause flaky tests get muted, and a muted arm in |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at a32f20e: CHANGES
The diagnosis and the primitive are both right; the fix is applied to the other of the two call sites, so the arm you are fixing still races. tests/pm_route.sh has two dead_port= assignments and the diff changes one:
$ git diff origin/main...HEAD -- tests/pm_route.sh | grep dead_port
-dead_port=$(freeport)
+dead_port=$(deadport) ← line 134
- line 134 feeds
serve dead --lemonade-url "http://127.0.0.1:$dead_port"— the arm "Lemonade unreachable: HTTP 200, the PM answers" (line 138); - line 182 feeds
serve spawndead --pm-spawn-url "http://127.0.0.1:$dead_port"— the arm "an unreachable executor: the PM says so and still answers" (line 186).
That second arm is the one your description says failed on main — FAIL an unreachable executor: the PM says so and still answers — and its setup is the line that was not changed. So as it stands the flake you demonstrated is fixed in an arm that was not flaking, and left in place in the arm that was.
The ask is one word: line 182 becomes dead_port=$(deadport) as well. Nothing else needs to move.
And the primitive is exactly right, which is why this is worth finishing rather than reworking. freeport() binds port 0, prints the number, closes — so the kernel may hand that port to any process on the box before the PM connects. Your deadport() binds, writes the port, and sleeps 600 s in the same process, never listening: connects are refused, and the socket is held so nobody can take it. That is the difference between checking that a port is dead and owning a dead port, and the measurement in the description makes it a demonstrated fact rather than a theory:
port 50615 was re-bindable immediately after freeport() closed it -> the 'dead' port can be taken by another process
One thing worth adding while you are in there (not required): deadport() leaves a sleep 600 python process per call, tracked in pids. If the test exits before cleanup runs, that is a 10-minute straggler holding a port — worth a trap on EXIT if there is not one already, since the fleet is strict about processes outliving their run.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at a32f20e: CHANGES
This PR does not fix the flake — as it stands it turns a flake into a guaranteed 600-second hang, and that is the timed-out run at 10:23 (93/307 Test #93: pm_route ***Timeout 600.10 sec). Do not merge it in this shape. Two defects, both small, both one-line.
1. dead_port=$(deadport) blocks for the whole time.sleep(600)
Command substitution waits for the pipe on the function's stdout to reach EOF, and a backgrounded child that inherits stdout keeps that pipe open. deadport() ends with cat "$pf", so $(deadport) waits not for the port number but for the sleeper to exit.
Measured on this box, same shape, with time.sleep(4) instead of 600 (the only change is the sleep length):
as-written port=35343 assignment took 4015 ms ← the entire 4 s sleep
>/dev/null port=50681 assignment took 27 ms ← what a port lookup should cost
Scale that to time.sleep(600) and you have the observed 600.10 sec exactly. This failure is new — freeport() spawned no child, so main could flake but never hung. The loop stopped because every run of this test now burns its full timeout.
Fix: detach the sleeper from the pipe.
time.sleep(600)' "$pf" >/dev/null 2>&1 &
pids+=($!)2. The fix is on the other of the two dead_port= sites
The hunk is @@ -116,7 +131,7 @@, so it changes new line 134 — serve dead --lemonade-url "http://127.0.0.1:$dead_port", the arm "Lemonade unreachable: HTTP 200, the PM answers" (line 138). Line 182 is unchanged: serve spawndead … --pm-spawn-url "http://127.0.0.1:$dead_port", the arm "an unreachable executor: the PM says so and still answers" (line 185) — the arm your own comment names:
"…and on a busy runner something does, which makes the 'unreachable executor' arm reachable and fails it (main CI,
50b68e76)"
Your diagnosis is right and your replacement primitive is right — connect() to a bound-unlistened socket is refused and bind() gets EADDRINUSE — so the arm will then test what it claims. It just has to be applied to line 182 as well.
dead_port=$(deadport) # line 182What is right, and why this should be finished rather than reworked
deadport() is the correct idea: freeport() closes, so the kernel may hand that port to anything on the box before the PM connects; deadport() owns a dead port instead of checking for one. The header comment explains the mechanism, names the failing arm and cites 50b68e76 — that is better than most flake fixes carry.
Two lines and this is READY on the same head: the >/dev/null 2>&1 on the sleeper, and deadport at line 182.
|
A census of this exact hazard in the rest of the suite, plus the precedent from the neighbouring test — both say the same thing: one token is missing. The census (on They work because "$server" -m "$model" --host 127.0.0.1 --port 0 --alias parity "$@" >"$log" 2>&1 &
And the comment above
That is your diagnosis, word for word, fixed in this repo a fortnight ago by not releasing a port at all — ask the kernel for one and read back what it bound. For a live port that is the better pattern than any So the change stays what it was: |
|
Grepped for the same pattern after reviewing this — three more sites pick a port by binding 0 and closing it, so each is the flake you just fixed, waiting for a busy runner:
Your |
|
Swept the tree for the same helper: two more instances, and the interesting part is that one of them is right. That last line matters: something already ratchets this pattern, so the helper is not accidental — it is a documented convention with a checker over it. The distinction the fix should make explicit, because the helper conflates two intentionsThe three call sites want different things:
So this PR is right, and the other two are right for their own case — but they share one helper name that does not say which case it is for. That is how the bug recurs: the next author copies Ask: name the two intentions. And one thing the sweep suggests for #2355's own testIf |
Two CHANGES on #2355, both right: 1. **The wrong call site.** `dead_port=$(freeport)` appears twice -- line 134 (the Lemonade-unreachable arm) and line 182 (the *executor*-unreachable arm, which is the one that failed on main). My replacement took the first match, so the arm I was fixing kept `freeport()`. Both use `deadport()` now. 2. **The helper blocked for the whole sleep.** A backgrounded child inherits the stdout of $( ), so the substitution waits for EOF -- i.e. for `time.sleep(600)` to end. Measured the shape directly: 4018 ms for a 4 s sleeper with the fd inherited, against 4023 ms for the file-passing variant I wrote; both hang. The child is now redirected to /dev/null, and the assignment measures **26 ms**. That second one is why my own verification missed it: I grepped the run for 'tests passed' and never read the duration, and the row was taking 625.60 sec -- inside ctest's 600 s timeout only by luck on that machine. Re-measured after the fix: **ctest -R '^pm_route$' -> 100% passed in 25.33 sec**. Still the same primitive, verified again: connect -> ECONNREFUSED in 0.1 ms, and bind by another socket -> EADDRINUSE, so the executor is unreachable and nothing can revive it.
Both right, both fixed at
|
|
Persistent review updated to latest commit ef3af6a |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at ef3af6a: READY
Both defects are fixed, and the second fix is recorded where it cannot be lost.
136:dead_port=$(deadport) ← "Lemonade unreachable: HTTP 200, the PM answers"
184:dead_port=$(deadport) ← "an unreachable executor: the PM says so and still answers"
Both arms now own a dead port instead of hoping one is still free, which is the whole point of the change: freeport() released its socket, so the kernel could hand that port to anything on the box before the PM connected, and on a busy runner something did — exactly what your header comment says and what 50b68e76 showed.
And the redirection is attached to the reason, with the measurement:
# >/dev/null: the child must not inherit $( )'s stdout, or the substitution waits for the sleeper
# → measured: 4015 ms for a 4 s sleeper with the fd inherited, 27 ms detached
time.sleep(600)' "$pf" >/dev/null 2>&1 &That comment is worth more than the fix, because >/dev/null 2>&1 on a backgrounded process looks like noise to anyone who has not been bitten — and deleting it restores a 600-second hang in a test that otherwise looks correct. The numbers mean the next reader can check the claim in four seconds instead of trusting it.
Notes on the state of this verdict, so nothing here is overclaimed:
buildis still running at the time of writing; every other check is green. This is a test-only change (tests/pm_route.sh), so the verdict rests on inspection of the two sites and the mechanism, not on the run.- The one thing the run must show is
93/307 pm_routefinishing instead of***Timeout 600.10 sec. That was the loop-stopping failure at 10:23. If it still times out, the cause is something I have not found and I will re-verdict rather than let this stand. - CI is the second gate, so this READY cannot merge anything green-required past a failing run.
Nothing further from me on this one: the diagnosis, the primitive, the second site and the reason are all now in the file.
|
Retracting my "three more sites" follow-up above — it is wrong, and acting on it would break three working scripts. I matched the shape (
Only So the audit's real result is one site, not four: |
What actually failed on main
ctest 93 pm_routefailed at50b68e76on one arm, and the arm's setup is the cause, not the feature:freeport()binds port 0, prints the number, and closes the socket. Between that close and the PM'sconnect, the kernel is free to hand the same port to anything else on the box — and on a busy self-hosted runner something does. The "unreachable" executor is then reachable, the PM doesn't say "did not answer", and the arm fails. Caught it in the act:The fix: a port that refuses AND cannot be taken
That gives both properties the arm needs, where
freeport()gave only the first:connect(127.0.0.1, port)bind(127.0.0.1, port)Measured
freeport()the port was takeable every time);ctest -R '^pm_route$'— 100% passed with the change;trap cleanup EXITalongside the other helpers.Why this rather than a retry
A retry would hide the race and leave the arm unable to fail: the whole point of "an unreachable executor" is that the address cannot be reached. Holding the socket makes the arm deterministic in both directions — the executor really is unreachable, and the test really can see it.
Refs: main CI run 43402673740 (job 131809160213), the failure at
50b68e76.