Let a deployment run its own rendezvous, and give the lab three NATs - #58
Merged
Merged
Conversation
Drop's Rust tests cover almost everything in-process, and nothing that needs a topology. Loopback has no NAT and no round-trip time, so hole punching, fallback and the acknowledgement window are all unexercised — and the endpoint-drop bug fixed on the transport branch was invisible to every loopback test for exactly that reason. The plan proposes `netlab/`: real binaries in Linux network namespaces, asserting on arrival, carrier, fallback and throughput. It reimplements no part of the protocol, which is decisions.md entry 11's rule applied to a second non-Rust consumer. Two findings from probing the machine shape it, and both contradict the brief it was written from: Root is not needed. This process has no capabilities at all and the full lab still works, because an unprivileged user namespace grants CAP_NET_ADMIN inside itself — verified with veth, netem and an iptables NAT rule. So the entry condition is "can a namespace be obtained", not "is CAP_NET_ADMIN held", and it has three answers rather than two. The direct path cannot run hermetically as the code stands. It reaches the public internet in three independent places, and one cannot be routed around: `publishable` refuses every address a lab may use, including the documentation ranges, because entry 14 strips private addresses from published records on purpose. Entry 14 states the consequence itself — two peers on one LAN can no longer find each other through the DHT — and a netns lab is a LAN. So the relay topologies come first; they need no source change and carry the throughput claim. The direct topologies wait on a decision about production surface, written up as open question 1 rather than settled inside a test directory. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Which path a transfer took is reported as prose on stderr — "Path relay
(encrypted; the relay cannot read it)", "Falling back to the relay." A
harness matching on those is matching on sentences written to be reworded,
and the first sympathetic rewrite breaks it silently.
Add one stable line beside the prose, never instead of it:
drop-status: path=relay fallback=rendezvous
`path` names the carrier that actually moved bytes; `fallback` says why it
was not the other one, distinguishing a direct path that could not be set up
from a receiver that looked and correctly found the sender absent. Only the
receiver can tell those apart, so only it reports `no-record`.
Behind `--status` and `DROP_STATUS`, off by default, because the prose is
the product and an unasked-for key=value line is noise. The variable only
turns it on, so a harness exports it once and spawns many.
Not a `--json` mode. A JSON object invites a reporting schema to grow inside
it and then the schema is a compatibility surface; what anything needs to
know is which carrier and why.
The end-to-end test spawns the real binary rather than calling into the
library like the rest of transfer.rs. An in-process assertion would pin the
string while leaving the flag parsing, the option plumbing and the choice of
stream unchecked — which is precisely what a lab spawning `drop` depends on.
A second test asserts the line is absent by default, since the first alone
would pass just as well if it were unconditional. 156 tests, up from 153.
Also corrects commands.md, which still said the direct path was unreachable
from the binary. It has been reachable since --transport landed; what a
local run actually fails to cover is the topology, not the code path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds netlab/: the real drop and api binaries, running in Linux network namespaces, across a router and two segments. Loopback has no router and no second segment, so nothing in the Rust suite had moved a Drop transfer across one. It needs no privilege. An unprivileged user namespace grants CAP_NET_ADMIN inside itself, so the pytest session re-executes into `unshare -Urnm` and carries on; a kernel that refuses skips every test with a message naming what was tried. Root is case one of three, not the requirement. No part of the Drop protocol is implemented here. The lab starts binaries and inspects a checksum, an exit code, and the machine-readable line `drop --status` prints — matching that rather than the prose, which is written to be reworded. The phase's stated gate was that the UDP-blocked test should fail when the iptables rule is removed. It cannot, and finding out why is the most useful thing this produced: with no route to the internet the direct path cannot be set up whether or not UDP is forwarded, so the topology cannot attribute the fallback to the block. That is the plan's own "passes while proving less than it looks like" risk, met on the first topology. The test now claims what it shows — the fallback fires, completes across a routed network, and is reported — and the README says what it does not. Two negative controls that do discriminate replace the gate: with the router not forwarding, the ends cannot reach each other at all; with no relay running, a relayed transfer fails. So the router is shown to be carrying the transfer and the relay to be relaying it. Two things cost real time and are written down so they are not rediscovered. pytest replaces fds 1 and 2 with capture buffers before any hook runs, and a process that execs inherits them — so the re-executed session wrote its whole output into a buffer nothing survived to read, and the run passed silently in zero seconds. And inside a user namespace this process is uid 0 without owning anything, so `ip netns` needs a tmpfs over its directory. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phases 2 and 3 of the network lab plan: throughput against the `window / RTT` claim in protocol.md, and delivery across a lossy path. Both were the next unblocked phases, and both needed a correction before they measured anything. The round trip that binds the sender's window is four traversals, not two. protocol.md line 86 has the receiver acknowledge bytes and the relay merely forward them on, so the loop is sender to relay to receiver for the chunk and back again for its acknowledgement. Delaying by half, as the phase was written, would have doubled the ceiling and let every measured rate sit comfortably under it while checking nothing. Three more things would each have produced a passing lane that proved less than it looked like: - `measure_rtt` read 15-20% high, because resolving an unresolved neighbour costs its own round trip across the same delayed link and ping averages that first reply in. The lane divides by that number. - A single transfer's wall clock is mostly handshake at these round trips - 2.7s, 5.5s and 9.5s of setup - and setup grows with RTT, so dividing bytes by seconds would have reported the handshake as throughput and still looked inversely proportional to RTT, which is the very shape being checked. Rates are a slope across two payload sizes, which cancels it. - Debug binaries move 6 MiB/s against 600 MiB/s optimised, below every ceiling under test, so the window could never have been the binding constraint. The lab builds --release. For loss, a hung transfer was surfacing as `LabError`, the exception meaning the lab itself broke - while a transfer that never finishes and never errors is exactly the bug the phase hunts. It is now a `Transfer.timed_out` outcome, the same conflation Phase 1 fixed on a different path. And at zero RTT a 1% drop rate completes as fast as no loss at all, so timing cannot show the impairment exists; the qdisc is read back and asserted instead, with observed loss recorded rather than gated, since a run that happens to lose nothing must not fail a build. Every measured rate lands at 41-53% of its ceiling, consistently. The window is enforced and does scale with the round trip; what costs the other half is not established, so it is recorded as unexplained and the ceiling assertion is one-sided to keep the gap from becoming the thing under test. The 400-800ms ratio is the noisy point and its margin is written down, so a future failure there is read as drift rather than as a discovery. The README now says where pytest comes from. It claimed `pytest netlab/` was the whole invocation, and the virtualenv it needs is gitignored. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cy7f61694MXmv7LYGK1i7n
Two halves of one problem. The direct path's rendezvous — the relay that makes an endpoint reachable and the DHT that carries its address — was compiled in, so a network with egress filtering could not use the direct path at all, and the lab could not test it hermetically. DROP_RENDEZVOUS_RELAY and DROP_RENDEZVOUS_BOOTSTRAP point both somewhere else. Environment variables rather than flags: rendezvous is a property of the network a machine is on, set once and never changed between two transfers, and a flag would charge every reader of --help for something almost nobody sets. A malformed value is an error and never a silent default — an operator who meant to keep rendezvous inside their network and quietly got the public DHT has lost exactly what they configured, invisibly. Recorded as decisions.md entry 15, with what an operator takes on in security.md. netlab/rendezvous/ runs an iroh relay and a three-node mainline testnet in one process. Deliberately outside the workspace: iroh-relay's server feature pulls in hyper, rustls and an ACME client, and none of that belongs in a binary shipped for four targets. Three findings changed the lab rather than the code: - Phase 4's assertions tested nothing. Carrier::Direct is reported as soon as rendezvous and connect succeed, and nothing consults whether a hole was punched — exactly as direct.rs's own doc says. A punch is now measured by counting bytes on the rendezvous link, and the NAT's mapping behaviour by comparing source ports seen by two destinations, rather than inferred from an iptables rule. - The lab died where it should have skipped. user_namespaces_available() read two sysctls that both say yes on Ubuntu 24.04+, where the refusal actually comes from apparmor_restrict_unprivileged_userns; conftest acts on that with execvp, so a wrong yes left no process alive to report anything. The probe now attempts a namespace instead of predicting one. - Outbound UDP is not blocked here, only port 53 is. Two DHT routers answered, and the reply carried this host's public address — which turns the objection to reaching the public DHT from an argument into a measurement. Suite run 2026-09-10: 7 passed, 1 failed. The failure is test_a_full_cone_nat_is_punched_through, which reproduces three runs of three while plain LAN and symmetric NAT both pass, so the QUIC path and rendezvous work and only the punch does not. Landed as a known failure and recorded as an open Phase 4 item rather than held back; see the plan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The hosted relay is closed. `DEFAULT_SERVER` is deleted rather than repointed, because a default naming a host that no longer answers is worse than no default at all: it turns "you have not configured a relay" into a TLS failure against somebody else's DNS, at a hostname this project no longer controls. Entry 8 is the reason it cannot simply be corrected in place. It argued for a stable API hostname precisely because the value is compiled in — so every binary already installed keeps reaching for it until its owner installs a new one. What entry 8 wanted, it can no longer have; the honest response is to stop shipping a guess. Recorded as decisions.md entry 16, which supersedes entry 8 rather than rewriting it. What this changes for a person: - `--server` and `DROP_SERVER` have no default. Unset, and empty, both mean no relay. - `--transport relay` without one is an error, raised before the payload is read, because being told the relay is missing is worth nothing after a wait to compress a directory. - `auto` without one is peer-to-peer that says so rather than falling back to nowhere. `may_fall_back` now takes the configured relay so it can tell *no relay configured* from *`--transport p2p` forbids falling back* — they were one branch before, and a person who never asked for `p2p` should not be told their transport forbids a fallback they did not choose. It also costs the one thing the relay was still for: a browser cannot speak QUIC and can only meet a CLI at a relay, so browser transfers now need an operator to run one. Stated in `--help` rather than discovered. The help's OPTIONS block is regrouped while it is being edited. It listed `-h` and `-V`, which work alone, beside `-s` and `-t`, which do not, with no way to tell them apart — so `drop -s` answered "unknown command `-s`" about an option that exists. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`drop`, `drop send` and `drop recv`, typed on their own in a terminal, open an interface: a chooser, a file browser, a code field, a destination picker, and a few checkboxes. Everything else is unchanged, which is the point — the flags are the program-facing surface and every harness that drives them keeps working untouched. The activation rule is deliberately conservative, and stricter than "no arguments on a terminal": - All three streams must be terminals, **stdout included**, though the interface never writes there. The transfer code goes to stdout so it survives a pipe, so a redirected stdout is somebody collecting that code and they should get it rather than a screen. - `DROP_STATUS` counts even with a bare command line. A harness exports it once and every `drop` it spawns inherits it. - **Any** flag means the command, not a curated list of flags that imply non-interactive use — that list would be wrong the first time somebody added a flag. Terminal lifecycle lands before any screen is drawn, because the worst failure here is not a bad screen but a shell that no longer echoes. `send` answers a signal with `std::process::exit(130)` and no unwinding, so the spool file can be deleted by hand; no destructor runs on that path, so `restore()` lives over an atomic and the handler calls it — before deleting the spool file, since that one cannot be recovered from by hand. `recv` had no termination handler at all and now has one. Ctrl-C also had to become an ordinary key: in raw mode the driver stops turning it into SIGINT. Three things the first person to use it found, each a design error rather than a discoverability one: - `enter` opened a folder while `s` selected one, so the gesture everybody makes first did the wrong thing. `enter` now picks whatever is highlighted, `→` opens, and choosing the folder you are standing in is the first row of the listing rather than a key to know about. - That was only safe once `esc` meant back. It meant "give up entirely", so one mistaken keystroke was unrecoverable. The flow is now a loop whose next screen is whichever answer is still missing. - The code field took any non-empty string and let `TransferCode::parse` reject it two screens later, from a program that had already exited. It is checked where it is typed, by the same parser. The sender's announcement says who the code is for and that `drop recv` on its own now works. ratatui and crossterm, both pure Rust, which is the bar `flate2` is already held to for four prebuilt targets. Release binary 26,988,848 → 27,482,824 bytes, +1.8%. Progress still prints below the closed interface; putting it on screen, and closing both sides on cancel, is phase 3 of the plan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Stop shipping a relay that is not there
Ask a person what they want instead of making them know
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two halves of one problem: the direct path's rendezvous was compiled in, so a network with egress filtering could not use the direct path at all, and the lab could not test it hermetically.
The feature
DROP_RENDEZVOUS_RELAYandDROP_RENDEZVOUS_BOOTSTRAPpoint the iroh relay and the DHT bootstrap somewhere else. Environment variables rather than flags: rendezvous is a property of the network a machine is on, set once and never changed between two transfers.A malformed value is an error, never a silent default. An operator who meant to keep rendezvous inside their network and quietly got the public DHT has lost exactly what they configured, invisibly.
Recorded as
decisions.mdentry 15, with what an operator takes on insecurity.md.The lab
netlab/rendezvous/runs an iroh relay and a three-node mainline testnet in one process, deliberately outside the workspace —iroh-relay's server feature pulls in hyper, rustls and an ACME client, none of which belongs in a binary shipped for four targets.Three findings changed the lab rather than the code:
Carrier::Directis reported as soon as rendezvous and connect succeed; nothing consults whether a hole was punched. A punch is now measured by counting bytes on the rendezvous link, and NAT mapping behaviour by comparing source ports seen by two destinations.user_namespaces_available()read two sysctls that both say yes on Ubuntu 24.04+, where the refusal comes fromapparmor_restrict_unprivileged_userns.conftestacts on that withexecvp, so a wrong yes left no process alive to report anything. The probe now attempts a namespace instead of predicting one.Known failure
First run 2026-09-10: 7 passed, 1 failed.
test_a_full_cone_nat_is_punched_throughreproduces three runs of three.It is not the direct path in general — topology 1 completes with no
apiprocess anywhere, and topology 3 completes over the rendezvous relay. The NAT measures as endpoint-independent (37848to two destinations) and nothing fell back to a relay. The sender reports a QUICauthentication failed; the receiver times out.Landed as a known failure with the evidence in the plan. Next step is packet capture, not another assertion.
Verification
171 Rust tests,
fmt,clippy --workspace --all-targets,check-secrets.sh— all pass.🤖 Generated with Claude Code