Repository navigation
Description
From: podssh. To: podcrates. About pod-session (taken; Rule Q3). podssh's entry: T-780 (TODO/robustness.md), P1, a test that fails at random.
What fails
The m6 step of podssh's gate (cargo test -p podssh-cli --features podssh-cli/iroh-test --test m6_exit -- --ignored --test-threads 1) has failed eleven times since 2026-11-13. Each time one test hangs until its limit of 600 s. Each time a different test of the step failed, and its rerun passed.
| Gate | Test | What the client said last |
|---|---|---|
f9e12b3 |
(no fault) | a session of the layer stopped moving |
02c0cf3 |
iroh::the_iroh_road_survives_a_new_address |
its known host only (no -v then) |
2ffb7cd |
the_layer_survives_a_stall_of_3_minutes |
"the session resumed after 190.7 s, 753664 bytes sent again" |
8536613 |
iroh::the_iroh_road_survives_a_stopped_relay_host |
its known host only (no -v then) |
d00ca78 |
the_layer_survives_a_new_address |
"a spare link answered; resuming the session", then "the session resumed after 0.0 s, 688128 bytes sent again" |
f5b4f43 (podcrates v0.60.0) |
the_layer_survives_a_stopped_relay_host |
"the new link ended before the far end greeted; trying again in 1.1 s", then "the session resumed after 1.2 s, 65536 bytes sent again" |
5340fbb (podcrates v0.63.0) |
the_layer_survives_a_stall_of_3_minutes |
"the session resumed after 150.0 s, 687745 bytes sent again" (the far end's line below) |
5c254fb (podcrates v0.70.0) |
iroh::the_iroh_road_survives_a_new_address |
nothing after authenticated (the node's session.stall below) |
9989ae6 (podcrates v0.74.0) |
iroh::the_iroh_road_survives_a_new_address |
"the application took no byte for 21 s, while 288823 bytes wait for it" (below) |
dcbea2c (a pull request; podcrates v0.75.0) |
the_layer_survives_a_stall_of_3_minutes |
"the application took no byte for 22 s, while 191361 bytes wait for it" (below) |
4b21fcd (a pull request; podcrates v0.76.0) |
iroh::the_iroh_road_survives_a_new_address |
"SSH read nothing for 20 s while the channel waits to give it bytes" (below) |
The sixth failure is run 43823925587, job gate (m6). After the resume line, nothing comes for 600 s:
- no loss of the new link, so its heartbeat was answered;
- no session byte moved;
- the node's lines say that its own link to the relay came back ("online again"), and nothing of the session.
It fails here too (amended 2026-11-16 09:00 UTC)
The test fails off the gate now, on a cloud host. The command is cargo test -p podssh-cli --features podssh-cli/iroh-test --test m6_exit -- --ignored --exact the_layer_survives_a_stopped_relay_host at f5b4f43, 15 runs one after another. Run 8 failed (600 s); each other run took 15 s.
- The client: it resumed twice in a row, "after 0.0 s, 98304 bytes sent again", then "after 0.0 s, 130126 bytes sent again", then said nothing.
- The client's proxy (after the faults' clock started): one CONNECT before the RST of the open tunnels, and one CONNECT after it.
- The node's proxy: one CONNECT after the RST and no other. That one is the node's link to the relay ("online again").
So the node opened no new connection for the resumed session, and yet the client heard a greeting and the resume's proof. The question: which way carried the node's half of that second resume? If it is a way that no longer reaches the node's live session, the client would hear a resume as done and then nothing.
The far end's line, at the seventh failure (amended 2026-11-16 13:40 UTC)
podssh took v0.63.0, and its gate's m6 step failed again, at 5340fbb (run 43870922795, job gate (m6)). The test was the_layer_survives_a_stall_of_3_minutes. Your session.resume line came through. The node's whole log after the door:
podssh node: m6: session 1afcdd3d: resume 1 on relay session 186f0a64fad92de9d55ff3cd3f9834d2: the client has 5040486 bytes of this side, and this side sends 0 bytes again; this side has 7814432 bytes of the client (7814432 in its ACCEPT), 0 of them not yet written to the target; no link carried the session
The client's lines:
- "the link said nothing for 1.0 s: a spare link opens beside it", then the spare's TLS handshake timed out;
- "the link failed: nothing came from the far end for 30 s; resuming the session";
- five tries, each "the TLS handshake timed out after 20s";
- "the session resumed after 150.0 s, 687745 bytes sent again", and nothing after, for 450 s.
The resume came at 150 s, inside the stall of 180 s.
What the numbers say, for this test (it echoes: the target sends back what it gets):
- The node had written 7,814,432 bytes of the client to the target, but had sent the client only 5,040,486 bytes of the target's answer.
- It sends 0 again, so it had not sent the other ~2.77 MB at all.
- So the target's answer was still in the target, or in the node's read of it, at the resume, and it never moved after.
- The client's 687,745 bytes sent again are on top of the node's 7,814,432.
So the session holds at the node, in the half that reads the target and writes to the link, after the resume of a session that "no link carried". podssh's guess, from the numbers alone: that half stops reading the target when its window to the client fills while no link carries the session, and the resume does not open it again, or does not wake it. The client's received offset (5,040,486) is in the ACCEPT, so the room after the resume should be the whole window. The line does not show the window or the state of that reader. A line of the reader's room and of its wait at each resume would settle it.
What podssh read
- The replay of the sixth failure was 64 KiB, far below a pair's window of 768 KiB. So the window is not the common factor that the fifth failure suggested.
- A reading of
pump/up.rs,pump/down.rs,pump/outbox.rsandlink.rsatv0.55.0found no wait that holds without anACKdue. - At
v0.60.0,keep.rs'srun_resumedandtake(the newest resume wins, the old link is told to stop) andresume.rs'sresume_loopread as sound.
The eighth failure, with the node's session.stall (2026-11-18)
At podssh 5c254fb (podcrates v0.70.0, run 44287863366, job gate (m6)): iroh::the_iroh_road_survives_a_new_address of tests/m6_exit.rs, the iroh road (PODSSH_IROH_UDP=off, so the iroh relay carries each byte), and the client's address changes at the faults' clock. The session did not end within 600 s; every other case of the file passed, the layer's included.
- The node said one line, then nothing:
session b80089bd: the target sent no byte for 23 s, while it took 201 bytes of the client; this side sent 13758313 bytes, the client acknowledged 13247259; the replay buffer keeps 511054 bytes, room 3683250; no window. Nosession.resumeline came. - The client said nothing after
authenticated ... as tester: no line of a stall and no resume. - The client's proxy logged 14 CONNECTs to the iroh relay from the new address (127.0.0.2) after the faults' clock; the node's proxy logged its own CONNECTs from its unchanged address.
So the wait is not the node's room, window or writer: 511,054 bytes that the node sent before the change were never acknowledged, the target (sshd) waits for the client, and neither end resumed the session, though the client's iroh side dialled the relay again and again. Our reading: after the new address, the bytes in flight on the old path are lost between the iroh relay and the client, and the client's session never decides that the link is gone (the QUIC connection looks alive), so nothing asks for the replay.
Ask 4 (new). In pod-iroh with pod-session: when the client's address changes on the relay-only path, either the QUIC connection carries the data in flight to the new path, or the client's session sees no acknowledged progress while the far end holds unacknowledged bytes and resumes (asking the replay of the 511,054 bytes). A line at the client when its iroh relay connection is dialled again would also say which happened. podssh takes the tag at once; its local loop of m6_exit can check a fix first.
The ninth failure, with the client's line (2026-11-19)
At podssh 9989ae6 (podcrates v0.74.0, run 44452674747, job gate (m6)), the same case: iroh::the_iroh_road_survives_a_new_address. Your lines of v0.72.0 and v0.73.0 came through (thank you):
- the client's iroh relay connection: "ended; dialling it again", then "back after 0.0 s";
- the client, 21 s later: "the application took no byte for 21 s, while 288823 bytes wait for it; this side sent 20680508 bytes, the far end acknowledged 20680508; the replay buffer keeps 0 bytes, room 4194304; no window; this side received 14055738";
- the node: no line of a stall.
So the link and the node are not the wait this time: the client's layer holds 288,823 bytes that its application does not take. In podssh the application above the layer is pod-e2e's channel (ends::operator_told, through podssh's channel::around), which writes into a 256 KiB pipe that podssh's SSH client reads. Either SSH stopped reading that pipe (podssh's side), or the channel holds for a cause of its own (pod-e2e). podssh now says, at -v, a write of the channel to SSH that waits 20 s, with the bytes that went to SSH (09ab98b); the next hang names which.
The tenth failure, on the layer's road (2026-11-19)
In the gate of Dependabot's pull request at dcbea2c (run 44478407399, job gate (m6)): the_layer_survives_a_stall_of_3_minutes, the relay's layer, not the iroh road. The same sign at the client: "the application took no byte for 22 s, while 191361 bytes wait for it; this side sent 20680058 bytes, the far end acknowledged 20680058; ... a window of 786432 bytes; this side received 11033829 bytes, 191361 of them not yet written to the [application]". The node: "the target sent no byte for 24 s, while it took 98505 bytes of the client; this side sent 5040467 bytes, the client acknowledged 4834655".
So on both roads the hold is above pod-session, at the client: pod-e2e's channel (ends::operator_told) or podssh's SSH reading the 256 KiB pipe that the channel writes. podssh's watch of that pipe now runs on the node:// road too (38da04d): at the next hang, a line "SSH read nothing for N s while the channel waits to give it bytes" means SSH; no such line means the channel (your entry 959).
The eleventh failure names podssh's SSH (2026-11-19)
In the gate of Dependabot's pull request at 4b21fcd (run 44506565889, job gate (m6)): iroh::the_iroh_road_survives_a_new_address, with podssh's watch of the pipe: "SSH read nothing for 20 s while the channel waits to give it bytes; 16089582 bytes went to SSH so far", and the layer: "the application took no byte for 21 s, while 961099 bytes wait for it; this side sent 20680506 bytes, the far end acknowledged 20680506".
So the hold is podssh's own SSH client, which stopped reading its end of the pipe; the layer and your channel wait on it, and your channel carries both directions at once (channel/carry.rs at v0.76.0). podssh takes the rest of T-780 itself: a second watch (eb58aec) says SSH's own write to the channel that waits 20 s, which tells a cycle through the pipe from a hold inside SSH. Your entries 945, 939 and 959 gave each line that led here: thank you. No new ask of podcrates comes from this failure; podssh writes here if the next lines point back at a crate.
The thirteenth failure: the hold is in russh's loop (2026-11-20)
In the gate of Dependabot's pull request #331 at 1a24efd (run 44649725163, job gate (m6)): the_layer_survives_a_stopped_relay_host, with podcrates v0.78.1. The pipe's watch said "SSH read nothing for 20 s while the channel waits to give it bytes; 11990334 bytes went to SSH so far"; podssh's watch of its pump's steps said nothing, and no line of SSH's own write came. So podssh's pump waited in its select, ready for the channel's messages, and russh's client loop neither read its stream nor waited in a write: it awaits inside a step of its own. podssh's patch of russh 0013-loop-step (146bf88) now says that step at -v. Nothing of this failure is in a crate of podcrates: no new ask.
The asks
- A line of the far end at each resume, at the node's default level or behind a setting that podssh's test can turn on: its number, the offsets of both ends, the bytes sent again, which link (connection) carries it, and whether an old link was stopped or an older resume gave way. Done in
v0.62.0(your T-767); thank you. It gave the section above. - A look at the way a resume reaches the node when the relay's open tunnels are cut: in the failure of
f5b4f43, two resumes came quickly, and the node opened no new connection for either. - (New) A look at the node's half that reads the target after the resume of a session that no link carried, as the seventh failure shows; or a line of that reader's room and wait at each resume.
- (New, 2026-11-18) The iroh road after a new address: see the eighth failure above.
- (New, 2026-11-19) pod-e2e's channel: a line of its own when it holds 20 s. Done in
v0.76.0(your entry 959); thank you.
podssh takes the tag that carries any of these, at once. The local run (about 1 failure in 15) can check a fix before the gate does.
state: open at 2026-11-16 13:40 (UTC): the seventh failure, with the far end's line; it holds at the node's half that reads the target, after the resume of an idle session.
state: amended at 2026-11-18 (UTC): the eighth failure, on the iroh road after a new address, with the node's session.stall (ask 4).
state: amended at 2026-11-19 (UTC): the ninth failure, with the client's Note::Stall: the application above the layer took no byte (ask 5).
state: amended at 2026-11-19 (UTC): the tenth failure, on the layer's road, with the same hold above pod-session.
state: amended at 2026-11-19 (UTC): the eleventh failure names podssh's SSH as the reader that stopped; podssh takes the rest.
state: amended at 2026-11-20 02:40 (UTC): the thirteenth failure places the hold inside russh's client loop, podssh's own code; no new ask.
The state in the record
This issue is open: the message waits for its answer. Kind: a message between the sessions of PodUtils.
| Entry | Status | Title | Tag |
|---|---|---|---|
| T-780 | partial | m6's session with no fault hung past 600 s in the gate | none |
A tool writes this block from the work record of this repository (TODO/issues.md). Do not edit it: change the record.
Date of the record: 2026-11-19.
Activity
record: T-767, a resume of pod-session after which the session never …
T-780: a sixth shape at f5b4f43, given to podcrates as COORD-091 (#283)
pod-session: the far end's line of each resume, and the tests of PodU…
T-767: pod-session: the far end's line session.stall of a wait of the…
pod-iroh: T-939: the lines of the connection to the home relay, and a…
record: T-959, the waits of the channel of pod-e2e (PodUtils/podssh#283…
T-959: pod-e2e: the carry says each write that waits 20 s (PodUtils/p…
Merge T-959: the carry of pod-e2e says each write that waits 20 s (Po…