diff --git a/README.md b/README.md index f07914f..5812ebb 100644 --- a/README.md +++ b/README.md @@ -41,11 +41,13 @@ in [docs/apply.md](docs/apply.md). | `keel network revert` | Put back the interfaces file a pending network change replaced; what the revert timer runs. `--boot` restores the file only, for the boot unit | | `keel network wireguard key` | Print this node's WireGuard public key, making the key pair first when there is none (root, live system only); the private key is never printed (decision 0020, docs/spec.md) | | `keel network wireguard suggest-address` | Print a random unique local IPv6 address with its /64, for the first node of an overlay | -| `keel mesh create` | On the first node: make the WireGuard mesh, a random unique local /64 with this node at `::1`, converged and confirmed by itself after the route check (decision 0048, [docs/mesh.md](docs/mesh.md)) | +| `keel mesh create` | On the first node: make the WireGuard mesh, a random unique local /64 with this node at `::1`, converged and confirmed by itself after the route check (decision 0048, [docs/mesh.md](docs/mesh.md)). With `--adopt`, on one node of a mesh built by hand: give the mesh its identity | | `keel mesh invite` | Print the `keel mesh join - <<< keel1:` line a new node runs to join this node's WireGuard mesh, valid one hour and once; reserve the new node's overlay address and start the invite's root helper, `keel-mesh-invite@`, and its unprivileged listener, `keel-mesh-listen@` ([docs/mesh.md](docs/mesh.md)) | | `keel mesh join` | Join the mesh with that token: the request to the inviter's port, pinned and authenticated, this node's spec written and applied, and both windows confirmed over the overlay; prints `keel mesh accept keel1a:...` for the inviter when its port cannot be reached. With `--dry-run`, print the change and apply nothing ([docs/mesh.md](docs/mesh.md)) | | `keel mesh accept` | On the inviter, the line a join printed when it could not reach the port ([docs/mesh.md](docs/mesh.md)) | -| `keel mesh status` | This node's peers, their last handshakes, and the pending invites; never a secret ([docs/mesh.md](docs/mesh.md)) | +| `keel mesh sync` | Add the members this node's peers know and it does not, on admission evidence a member it trusts signed, over the overlay, confirmed by a WireGuard handshake; run at boot and every 15 minutes by `keel-mesh-sync.timer`. With `--adopt ADDRESS`, take that member's mesh identity first: the repair of a split ([docs/mesh.md](docs/mesh.md)) | +| `keel mesh remove` | Remove a peer from this node's spec, under the window for `keel network confirm`, and keep a signed tombstone so no `keel mesh sync` adds it again ([docs/mesh.md](docs/mesh.md)) | +| `keel mesh status` | This node's mesh identity, its peers, their last handshakes, and the pending invites; never a secret ([docs/mesh.md](docs/mesh.md)) | | `keel notify` | What monit runs when a check of the `monitor` section fails, lasts or recovers: sends one message, saying what happened and what to do, to every channel the section declares (email, Telegram, ntfy, a webhook). It reads the channels from `/etc/keel/monitor.json`, which `apply --system` writes as root, never from the spec, reads the tokens from their secret files and prints no URL (decision 0021, docs/apply.md) | | `keel inspect` | Write a spec from the running machine, or from an offline root, and report every field with its source or why it was not inferred; secrets are never read (brief sections 5.2 and 7) | | `keel diff` | Report drift between the spec and the running machine, or an offline root, field by field, through the same collector `inspect` uses; secrets are never compared and their files need not exist, nothing is written (brief section 5.2) | @@ -143,16 +145,16 @@ Defined in one place, `keel/exits.py`, and reproduced here. | 12 | `ASSEMBLE_FAILED` | The rootfs is not empty or cannot be created, or `tar` or `zstd` failed while extracting or packing | | 13 | `INSPECT_INCOMPLETE` | `inspect` wrote the spec, but a required field (hostname, fqdn, interfaces, alerts, updates) could not be inferred; or `diff` found no drift, but a declared field could not be observed. The report says which and why | | 14 | `DRIFT_FOUND` | `diff` found at least one declared field whose observed value differs. Drift wins over unobserved fields, so a report with both exits 14 | -| 15 | `APPLY_NEEDS_ROOT` | The system phase (`--system`, `--system-only`) on the live system was run by a user other than root; nothing was written, not even the conf. Also `keel network wireguard key` making a key, and `keel mesh create`, `invite`, `join`, `accept` and `serve`, on the live system | +| 15 | `APPLY_NEEDS_ROOT` | The system phase (`--system`, `--system-only`) on the live system was run by a user other than root; nothing was written, not even the conf. Also `keel network wireguard key` making a key, and `keel mesh create`, `invite`, `join`, `accept`, `serve`, `sync`, `members` and `remove`, on the live system | | 16 | `APPLY_FAILED` | The system phase could not make at least one change; the output names it. The conf was written, where the run writes one, and every other change was made, so the run can be repeated | | 17 | `CHANNEL_INVALID` | A channel pointer, or the record of the one this instance follows, does not parse or fails validation. Also a pointer signed in the future, one claiming more than 30 days of life, and a revision's archived pointer naming another revision | | 18 | `CHANNEL_UNVERIFIED` | A channel pointer is not signed by a key that may move a channel, or no keyring was given to check it against. A key that is revoked or expired is refused here: gpgv exits 0 for both and still prints `VALIDSIG`, so `GOODSIG` is what is required | | 19 | `CHANNEL_EXPIRED` | A channel pointer is past its expiry. A mirror that is stale, broken or hostile holds an appliance on an old release by not updating, so this is an error and never a warning | | 20 | `CHANNEL_ROLLBACK` | A channel pointer names an earlier revision than the one this instance is on; `--allow-rollback`, or naming the release and revision, is how going back is asked for | -| 21 | `NETWORK_NOT_CONFIRMED` | `keel network confirm` refused: no change is waiting, or it was not run from a new session over the new configuration, or a console (docs/apply.md). Also `keel mesh create`, `join` and `accept` whose overlay change was not confirmed: it reverts by itself when its window ends | +| 21 | `NETWORK_NOT_CONFIRMED` | `keel network confirm` refused: no change is waiting, or it was not run from a new session over the new configuration, or a console (docs/apply.md). Also `keel mesh create`, `join`, `accept` and `sync` whose overlay change was not confirmed: it reverts by itself when its window ends | | 22 | `NOTIFY_FAILED` | `keel notify` reached no channel: `/etc/keel/monitor.json` is missing or refused, declares none, or every one failed; the message then went to syslog (user.crit) and root's mailbox. One channel that takes the message is success; the others' failures are printed | | 23 | `MESH_TOKEN_INVALID` | `keel mesh join` could not use the token: not a `keel1:` token, mistyped or truncated (its checksum), of a later format, inconsistent, or expired; or `keel mesh accept` could not read its `keel1a:` line. The message never holds the token or its secret | -| 24 | `MESH_REFUSED` | `keel mesh` refused: no overlay to invite into, no key yet, no endpoint, no free address in the prefix, another invite pending on the port; this node is in another mesh, at another address of it, or the change would make a spec validation refuses; a network change already waits in its window; the inviter refused the join (used, expired, bad HMAC), or the certificate is not the pinned one; no route to the inviter, or neither node can reach the other | +| 24 | `MESH_REFUSED` | `keel mesh` refused: no overlay to invite into, no key yet, no endpoint, no free address in the prefix, another invite pending on the port; this node is in another mesh, at another address of it, or the change would make a spec validation refuses; a network change already waits in its window; the inviter refused the join (used, expired, bad HMAC), or the certificate is not the pinned one; no route to the inviter, or neither node can reach the other; an invite whose peers hold another mesh identity, or none; a sync from members of another identity | Two rules that callers depend on: diff --git a/debian/changelog b/debian/changelog index c98b502..315abf2 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,68 @@ +keel (0.18.0) unstable; urgency=medium + + * keel mesh makes a full mesh before etcd (handbook decision 0048, + "Until etcd exists"). A test on three Keel Web nodes across two + sites found a star: a node that joined through one member was never + learned by the others. The join's answer already named the members + the inviter knows; the new node now takes them as peers in its + first change. The inviter, once a join is confirmed, announces the + new node to each of its other peers over the overlay, and each adds + it under the window of decision 0018, confirmed by a WireGuard + handshake from the new node. A member offline then, and a node that + joined through the fallback, pull the members with the new + keel mesh sync, which keel-mesh-sync.timer runs at boot and every + 15 minutes. A member that never answered is tried again after an + hour, and announcements waiting are applied in one change, under + one window. + * Only an admitted join adds a peer. Each node has an Ed25519 signing + key made on the machine; an inviter signs evidence of each + admission (the mesh, the invite, the node's keys, address and + endpoint, the time), sent in the join's authenticated answer and + with the node's entry in every roster. A member takes a roster's + entry only on evidence of an invite signed by a key it already + trusts: its own, its inviter's, a member admitted the same way, or + a trust root that keel mesh create --adopt or keel mesh sync + --adopt made of the spec's peers, the operator's act for a mesh + built by hand, whose signing key is bound only from a roster the + node fetched from the root itself. docs/mesh.md says what this + changes in 0048's threat model; rotating a signing key is follow-up + work. + * keel mesh remove removes a node from the mesh before etcd: a + tombstone signed with the node's key, the peer out of its spec + under the window, confirmed by a handshake from a peer it keeps, + and its roster sent to the others, each of which drops the peer + through one window when the signer may remove it (its admitter, a + trust root, or the node itself). Tombstones are kept for good, + 1024 at most and 64 per signer, every roster carries them all, and + no sync adds a removed key again. + * The members' channel is split as an invite is: keel-mesh-members, + the root helper (started by keel-mesh-members.path once the node is + in a mesh), starts the listener keel-mesh-members-listen with a + dynamic user, no capability and a sandbox, on TCP 51821 of the + overlay address only, bound to the overlay's interface, so + WireGuard authenticates who speaks, behind the invite listener's + limits: a deadline per request, 4 connections and 2 per source, + size caps, and sources refused too often. keel's firewall accepts + that port on the overlay's interface only. + * The mesh's identity no longer splits. Only a join or the operator's + --adopt sets it: keel mesh invite on a node whose peers hold another + identity, or with peers and none of its own, is refused; keel mesh + create --adopt gives a mesh built by hand its identity, keel mesh + sync --adopt ADDRESS takes a member's, which repairs a split. keel + mesh status shows the identity. + * keel mesh invite and join find this node's endpoint on a DHCP or + SLAAC host: the uplink's global, stable IPv6 address (never a + unique local, deprecated or privacy one, nor one of wg*), and a + public IPv4 one, in the order of keel-core's console banner. They + say which, and --endpoint still overrides. With only a privacy or + an RFC 1918 address, invite is refused and asks for --endpoint. + * keel mesh join, create and accept no longer print apply's "reverts + in 120 s unless keel network confirm" while they confirm their own + change: they say "confirming over the overlay…" and the result, and + show apply's lines only when the change is not confirmed. + + -- Marcos Mendez Sat, 03 Oct 2026 23:30:00 +0000 + keel (0.17.0) unstable; urgency=medium * The second part of handbook decision 0048, joining the mesh with one diff --git a/debian/keel.keel-mesh-members.path b/debian/keel.keel-mesh-members.path new file mode 100644 index 0000000..9a9b7ae --- /dev/null +++ b/debian/keel.keel-mesh-members.path @@ -0,0 +1,15 @@ +# Start the members' channel once this node is in a mesh (decision 0048). +# +# The mesh's identity is written by keel mesh create, join, sync and +# create --adopt; from then on the node answers its peers over the +# overlay: its roster for their keel mesh sync, and the announcement of +# a node another member admitted (docs/mesh.md, "Members learn of each +# other"). +[Unit] +Description=Start keel's mesh members' channel once this node is in a mesh + +[Path] +PathExists=/var/lib/keel/mesh/identity + +[Install] +WantedBy=multi-user.target diff --git a/debian/keel.keel-mesh-members.service b/debian/keel.keel-mesh-members.service new file mode 100644 index 0000000..a0b5659 --- /dev/null +++ b/debian/keel.keel-mesh-members.service @@ -0,0 +1,25 @@ +# The members' channel of keel mesh (decision 0048): the root helper. +# It holds the spec, the trust store and this node's signing key, and +# starts the unprivileged listener as the transient unit +# keel-mesh-members-listen (a dynamic user, no capability, the sandbox +# of keel.mesh.bridge.LISTENER_PROPERTIES), which listens on TCP 51821 +# on this node's overlay address, bound to the overlay's interface, and +# forwards each request over a 0600 unix socket in its 0700 runtime +# directory, the peer checked against the unit's MainPID. The helper +# answers a member's keel mesh sync with this node's roster, and adds +# the members an announcement names with admission evidence it trusts, +# under the window of decision 0018, confirmed by a WireGuard handshake +# (docs/mesh.md). Started by keel-mesh-members.path once the node is in +# a mesh. +[Unit] +Description=keel mesh: the members' channel on the overlay +After=network-online.target +Wants=network-online.target +# a node in no mesh has no members to answer, also when the package's +# installation starts the unit +ConditionPathExists=/var/lib/keel/mesh/identity + +[Service] +ExecStart=/usr/bin/keel mesh members +Restart=on-failure +RestartSec=10 diff --git a/debian/keel.keel-mesh-sync.service b/debian/keel.keel-mesh-sync.service new file mode 100644 index 0000000..7b9601c --- /dev/null +++ b/debian/keel.keel-mesh-sync.service @@ -0,0 +1,13 @@ +# keel mesh sync (decision 0048): add the members this node's peers +# know and it does not, so a member that was offline when a new node +# was announced, or a node that joined through the fallback, ends up a +# peer of every other one. Run by keel-mesh-sync.timer, at boot and +# every 15 minutes; a node in no mesh has nothing to sync. +[Unit] +Description=keel mesh: learn the members this node's peers know +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +ExecStart=/usr/bin/keel mesh sync diff --git a/debian/keel.keel-mesh-sync.timer b/debian/keel.keel-mesh-sync.timer new file mode 100644 index 0000000..59d8586 --- /dev/null +++ b/debian/keel.keel-mesh-sync.timer @@ -0,0 +1,11 @@ +# keel mesh sync at boot and every 15 minutes (decision 0048) +[Unit] +Description=keel mesh: learn the members this node's peers know, regularly + +[Timer] +OnBootSec=2min +OnUnitActiveSec=15min +RandomizedDelaySec=1min + +[Install] +WantedBy=timers.target diff --git a/debian/rules b/debian/rules index a9848ec..68691c1 100755 --- a/debian/rules +++ b/debian/rules @@ -8,3 +8,5 @@ export PYTHONDONTWRITEBYTECODE=1 override_dh_installsystemd: dh_installsystemd --name=keel-network-revert --no-start dh_installsystemd --name=keel-firewall --no-start + dh_installsystemd --name=keel-mesh-members + dh_installsystemd --name=keel-mesh-sync diff --git a/docs/apply.md b/docs/apply.md index 50679f8..e15064d 100644 --- a/docs/apply.md +++ b/docs/apply.md @@ -1118,6 +1118,10 @@ drop, and these rules: - every `mesh` port of an enabled overlay, on the WireGuard interface only, and none (the step says which) while the spec declares no overlay; +- with an overlay, TCP 51821 on the WireGuard interface only: the + members' channel of keel mesh, which `keel mesh sync` and the + announcement of a new node use between members ([docs/mesh.md](mesh.md), + "Members learn of each other"); - with an overlay, the TCP ports of the pending mesh invites: a named set, `mesh_invites`, of ports with a timeout, and one rule that accepts TCP to a port in it. `keel mesh invite` adds its port for the @@ -1383,6 +1387,7 @@ used survived, so: | SSH, over the uplink | the session started after the change, came from another machine, and arrived at an address another interface of this machine holds now. confirm says the overlay itself was not tested, and at which address a peer would test it | | A console, a process attached from a container's host | always, as for the uplink | | The mesh session of a join (decision 0048), for the overlay change that join made and no other | on the inviter, a WireGuard handshake from the new peer's key since the join (`wg show latest-handshakes`) and the join's confirmation request signed with the invite's HMAC key, both checked by the root side; on the new node, the inviter's signed answer over the tunnel ([docs/mesh.md](mesh.md)) | +| A member added by `keel mesh sync` or by an inviter's announcement (decision 0048, "Until etcd exists"), for the overlay change that added it and no other | a WireGuard handshake from the key of a member that change added, since the change (`wg show latest-handshakes`), which only that member, reaching this node over the new configuration, can complete; after the route check ([docs/mesh.md](mesh.md)) | | `keel mesh create`, for the change that created a mesh with no peer and no other | always, after the route check: an overlay with no peer routes its own private prefix only, and no peer exists to confirm it (decision 0048, second round, point 1) | | Anything else, a session older than the change, a session from this machine to itself, or one arriving at an address the overlay does not declare | refused (exit 21) | diff --git a/docs/mesh.md b/docs/mesh.md index ed01f0b..726ddaf 100644 --- a/docs/mesh.md +++ b/docs/mesh.md @@ -22,11 +22,16 @@ operator writes by hand, `address` and a peer, converged by the same There is no `mesh` section in the spec, and a mesh built by joins and one typed by hand are the same spec. -**What exists in this version:** `create`, `invite` and its listener, -`join`, the fallback `accept`, and `status`. Not yet: etcd (it forms at -the third node in cloud advanced, Phase 5), the announcement of a new -node to the inviter's other peers, `keel mesh remove`, `keel mesh -invites` and `invite --cancel`, and the boot unit that removes invites +**What exists in this version:** `create` (and `create --adopt`), +`invite` and its listener, `join`, the fallback `accept`, `status`, and +a full mesh before etcd: the announcement of a new node to the +inviter's other peers, `keel mesh sync`, and the members' channel they +use ("Members learn of each other"), only with admission evidence +("Admission evidence and trust"), and `keel mesh remove` before etcd, +whose tombstone makes the other members drop the node too. Not yet: +etcd (it forms at the third node in cloud advanced, Phase 5), the +rotation of a node's signing key, `keel mesh invites` and `invite +--cancel`, and the boot unit that removes invites which expired while the machine was down. confconsole's "Invite a node" and "Join a mesh" screens call these commands, in confconsole. @@ -34,6 +39,7 @@ and "Join a mesh" screens call these commands, in confconsole. ``` keel mesh create [--network-window SECONDS] +keel mesh create --adopt ``` Root, on the first node, whose spec declares no overlay address. It @@ -58,8 +64,21 @@ created the mesh: this node is fd2a:9c41:7e03::1/64 on wg0, WireGuard on UDP 518 keel mesh invite prints the line that joins the next node ``` -apply's own lines, and the confirmation's, go to standard error; -standard output holds the summary alone. +The confirmation's lines go to standard error, after `confirming the +new overlay…`; standard output holds the summary alone. apply's own +lines, among them its "reverts in 120 s unless `keel network +confirm`", are held, and shown only when the change does not come up +or is not confirmed: `create` confirms it itself. + +`--adopt` is for a mesh built by hand before `keel mesh` existed, whose +nodes hold no identity yet ("The mesh's identity"): on one node of it, +it keeps that node's identity, else takes the one its peers hold, else +makes one, and prints it; it is refused when the peers hold another +identity than this node's, or several. It also makes the peers the +spec lists this node's trust roots ("Admission evidence and trust"): +the operator's act is what vouches for them. Each other node then runs +`keel mesh sync --adopt` with this node's overlay address. Nothing is +applied. ## keel mesh invite @@ -76,11 +95,16 @@ It: here; 2. takes the endpoints the new node will reach this one at: the `--endpoint` addresses (one per family, for a port forward say), or - else the static addresses `network.interfaces` declares, IPv6 first; + else the static addresses `network.interfaces` declares, or else, on + a DHCP or SLAAC host, the ones found on the uplink ("The endpoint"), + IPv6 first, saying which; 3. makes a key pair and a self signed certificate for this invite only (`openssl req -x509`, P-256), whose SHA-256 fingerprint the token carries for the new node to pin; -4. makes the mesh's identity if this node has none (below); +4. checks the mesh's identity with its peers over the overlay: refused + when they hold another one than this node's, it takes theirs when + it has none, and makes one only for a node with no peer ("The mesh's + identity"); 5. under the mesh's lock, removes the expired invites, reserves a free address (below) and writes the pending invite; another invite pending on the same TCP port is refused (each listener holds its port), and @@ -99,6 +123,31 @@ the number of WireGuard's UDP port, so a provider's firewall needs one number for both. Under `--root` no listener is started, and the line can be checked with `keel mesh join --dry-run` only. +### The endpoint + +A host whose uplink DHCP or SLAAC configures declares no static +address, so `invite` and `join` find their endpoint in what `ip` +shows (keel.mesh.endpoint): + +- on the uplink, the interfaces the default routes leave through (any + interface without a default route), never a WireGuard one (`wg*`); +- a global IPv6 address that is not unique local (`fc00::/7`), not + `deprecated`, and not a SLAAC privacy address (`temporary`), a static + one before a SLAAC or DHCPv6 one (`dynamic`): the order of keel-core's + console banner (`keel_banner_pick_ipv6`, a shell function of the + image, so the rule is kept here in Python and the two agree); +- beside it, or alone, a public IPv4 address. + +Each address chosen is said on standard error (`endpoint +2804:…:a035, found on eth0 (--endpoint overrides it)`). A privacy +address changes within a day and an RFC 1918 or carrier NAT address +(`100.64.0.0/10`) is reached from its own network only, so neither is +ever taken. A host that holds nothing better has no endpoint found: +`invite` is refused, says why (`2001:db8::7 is a SLAAC privacy address, +the only global IPv6 one eth0 holds…`) and asks for `--endpoint`, a +stable address or a port forward's; `join` sends no endpoint, as a node +behind NAT does, and says why. + ### The address A random address of the inviter's overlay prefix, drawn with Python's @@ -134,21 +183,26 @@ It: another is in another mesh, and is refused), and makes its key pair if it has none; 4. finds its own endpoint: `--endpoint`, else a static address - `network.interfaces` declares, else a global address of an interface - other than the overlay's; IPv6 first; none when it has none; + `network.interfaces` declares, else the one found on the uplink ("The + endpoint"); IPv6 first; none when it has none; 5. sends the join request to the inviter's HTTPS port, at its IPv6 endpoint and then its IPv4 one, comparing the SHA-256 of the certificate the inviter presents with the token's fingerprint before a byte of the request is sent; 6. with the signed answer, which must carry its own nonce, the invite - id, the inviter's key and address, writes its own spec (its address - and the inviter as a peer) and applies it under the window, with + id, the inviter's key and address, writes its own spec (its address, + the inviter as a peer, and the other members the answer names, as + `keel mesh sync` takes them) and applies it under the window, with `--skip-uplink`; -7. opens the mesh session to the inviter's overlay address, retrying - every two seconds while the tunnel comes up: the inviter confirms its - window on receiving it, and this node confirms its own on the - inviter's signed answer. No `keel network confirm` is typed on - either node. +7. says `confirming over the overlay…` and opens the mesh session to + the inviter's overlay address, retrying every two seconds while the + tunnel comes up: the inviter confirms its window on receiving it, and + this node confirms its own on the inviter's signed answer. No `keel + network confirm` is typed on either node, and apply's "reverts in + 120 s unless `keel network confirm`" is shown only when the + confirmation fails; +8. the inviter then announces this node to its other peers ("Members + learn of each other"). At the end, on standard output: @@ -158,11 +212,14 @@ peer: nb9/izIukqWXM7gnBpe7hki4jKZZChWOW1wEfONn82E= at fd2a:9c41:7e03::1, through the mesh is up with 2 nodes; etcd starts when a third node joins, because 2 etcd members cannot lose one and keep a majority ``` -apply's lines, the confirmation's, and every refusal go to standard -error. When the inviter knows other peers, the last line says how many -nodes the mesh has, and that this node reaches the inviter alone until -they learn of it, which this keel does not do yet (0048, "Until etcd -exists"). +The confirmation's lines and every refusal go to standard error. When +the inviter knows other peers, the last line says how many nodes the +mesh has, that this node has them all as peers, and that the inviter +announces it to them: + +``` +the mesh has 3 nodes: this node has them all as peers, and the inviter announces this node to the 1 other(s) over the overlay (one offline now learns of it at its next keel mesh sync); etcd is not part of this keel +``` A join that fails after the request was accepted (the tunnel never answers within the window, say) reverts on both sides by itself, and @@ -183,6 +240,9 @@ keel mesh accept keel1a: to run on the inviter, then waits for the inviter over the overlay as in step 7. Too close to the expiry (under 30 seconds) it refuses instead. +The line carries no answer, so no member: once confirmed, `join` pulls +them from the inviter (`learning the other members from the +inviter…`, a `keel mesh sync --from` the inviter). ### --dry-run @@ -242,13 +302,305 @@ A line for another invite, or one changed in the copy, is refused before anything is spent; a second use of the token finds no pending invite. +## Members learn of each other + +Until etcd exists, members learn about each other through the inviter +(0048, "The third node and after"), so that the mesh is a full one and +not a star around the node that invited: + +1. the join's answer names the other members the inviter knows, each + with its admission evidence, and the new node takes those it can + verify as peers in its first change; +2. the inviter, once the join is confirmed (by `serve` or `accept`), + announces the new node to each of its other peers over the overlay, + at once; each adds it and applies, as below; +3. a member offline then, and a node that joined through the fallback, + pull the members with `keel mesh sync`, which + `keel-mesh-sync.timer` runs two minutes after boot and every 15 + minutes. + +### Admission evidence and trust + +0048 lets only an admitted join add a peer. The overlay authenticates +who speaks (below); it does not make what a member says about other +nodes true, and a member that could push any peer to the others could +add a node no member admitted, whose own handshake would then confirm +the change. So every member a roster names is taken only with evidence +of its admission: + +- **a signing key per node.** Each node has an Ed25519 key pair, made on + the machine with `openssl genpkey` the first time it joins, invites, + adopts or serves its members (`/var/lib/keel/mesh/node.key`, 0600, + never printed). Its public half travels in the join request, in the + `keel1a:` line, in the join's and the confirmation's answers, and in + each roster; +- **evidence.** An inviter that admits a node signs, with its key, the + mesh's identity, the invite id, the new node's WireGuard and signing + keys, its overlay address and endpoint, and the time. It keeps that + evidence, sends it to the new node in the answer, which the invite's + HMAC authenticates, and joins it to the node's entry in every roster + it gives; +- **trust.** A node trusts its own key; the inviter it joined through, + whose signing key it learned from that authenticated answer; and the + trust roots its operator named (below). It takes a roster's entry only + when its evidence names an invite, the entry's key and address, is for + this mesh, and was signed by a key it trusts; the node it takes is + then trusted in turn, so evidence chains from member to member (A + admitted B, B admitted C: a node that trusts A takes both). An entry + without valid evidence is left out, whoever sends it. Evidence always + names an invite: no member vouches for a node it did not admit; +- **trust roots.** A mesh built by hand before `keel mesh` has no + evidence. `keel mesh create --adopt` on one node and `keel mesh sync + --adopt` on the others, explicit acts of the operator, make the peers + each node's spec lists its trust roots. A root's signing key is + learned from the root itself, in a roster this node fetched from the + root's own overlay address (`keel mesh sync`), which WireGuard + authenticates as that root's; never from an announcement, whose + source only the listener reports. A root vouches for nobody: a node + is another's root only if that node's operator listed it; +- **tombstones.** `keel mesh remove` records a removal, signed with the + remover's key. A node takes a tombstone only when its signer may + remove that node: **the member that admitted it** (as the evidence + the node keeps names), **a trust root**, or **the node itself**, + leaving. Any other member, trusted or not, removes a node from its own + spec only. A node that takes a tombstone drops the peer from its spec, + every peer a pull or the pending announcements remove in one change, + under one 0018 window (below), and never takes the key again; +- **how many, and for how long.** Tombstones are kept for good: a + removed key never comes back, and a tombstone that expired could let + an old roster bring it back to a member that was offline. A node keeps + at most 1024 of them, at most 64 signed by one key (so one member + cannot fill the store), and each roster carries all it keeps. A node + whose store is full refuses `keel mesh remove`, and takes no further + tombstone. + +The state is `/var/lib/keel/mesh/trust.json`: each trusted member's +signing key, whether it is a root, and its evidence; and the +tombstones. A damaged file is refused, never guessed at: `keel mesh sync +--adopt` makes the peers roots again. + +**What this changes in 0048's threat model.** Before, a member could +make every other member add any peer it named, as long as it spoke over +the overlay. Now a peer is added only on evidence that a member the node +already trusts admitted it, and that evidence names the invite and the +member that signed it, so every added peer can be traced to the member +that admitted it. + +**Any trusted member can vouch for a new node**, as 0048 says any +member can invite: the evidence it signs is enough for every member +that trusts it, directly or through the chain. So a compromised member +(its root, or its signing key) can admit any node it likes into the +whole mesh, signed in its own name, and every member will add that node +as a peer. It can also remove the nodes it admitted, and any node if it +is a root. What it can no longer do is add a node under another +member's name, add one to a member that does not trust it, remove a +node it did not admit (unless it is a root), or bring back a key whose +removal was taken. The way out of a compromised member is to remove it +(by its admitter or a root): its tombstone makes it trusted no more, +and evidence it signed afterwards is left out; what it admitted before +stays, and each such node is removed the same way. A handbook PR asks +that 0048 be amended to say so. + +**A signing key is not rotated yet.** A node keeps the key it made; a +new key signed by the old one is follow-up work, recorded in the same +handbook PR. A node whose key may be compromised is removed and joins +again with a new invite, which makes it a new key pair. + +### The members' channel + +As an invite is (keel#75), the channel is two processes: + +- **the root helper**, `keel mesh members` in + `keel-mesh-members.service` (started by `keel-mesh-members.path` once + the node holds a mesh identity), which alone holds the spec, the trust + store and the signing key. It starts the listener as the transient + unit `keel-mesh-members-listen`, with `DynamicUser=yes`, no + capability and the sandbox of the invite's listener (below, "The + listener"), connects to the unix socket the listener binds in its + RuntimeDirectory (`/run/keel/mesh-members`, 0700, the socket 0600), + checks with `SO_PEERCRED` that the other end is the unit's MainPID and + not root, and sends it the overlay's interface, address and port, + which hold no key; +- **the listener**, `keel mesh members-listen`, which refuses to run + with any capability, listens on TCP 51821 on this node's overlay + address, applies the limits below, and forwards what passed to the + root side, one JSON message per line; the root side treats each as + untrusted. + +The listener's socket is bound to the overlay's interface +(`SO_BINDTODEVICE`), as is every client's: a request arrives, and an +answer comes back, only through WireGuard. WireGuard accepts a packet +on that interface only when it was decrypted with the key of the peer +whose `allowed_ips` hold its source address, so the address a request +comes from names the member that sent it, and the answer to one sent to +a member's address can only come from that member: 0048's "the overlay +authenticates the announcement: it comes from a peer's address, which +only that peer's key can use". That authenticates who speaks; what it +vouches for needs evidence (above). There is no TLS (WireGuard +encrypts). + +| Request | Body | Answer | +| --- | --- | --- | +| `GET /v1/members` | none | 200, this node's roster | +| `POST /v1/announce` | the sender's roster, the new node in it | 202, queued and applied after the answer | + +A roster is the node's mesh identity (hex, or null), its WireGuard and +signing keys, its overlay address, every peer its spec declares +(`public_key`, `endpoint` or null, `address`, and the evidence it keeps +for it, or null), at most 256, and all the tombstones it keeps (at most +1024). The +root side refuses (403, logged) a request from an address of no peer's +`allowed_ips`, and an announcement whose roster names another key than +the sender's. + +Against a member, or anything on the overlay, that sends too much: + +- at most 4 connections are read at once, and 2 from one source; +- reading a request has a deadline of 15 seconds, and each read a + timeout of 5; +- the request line and headers are capped at 8 KiB, the body at 512 KiB; +- a source refused 5 times within a minute is not answered until the + minute has passed; +- announcements wait in a queue that keeps the latest one per sender + and at most 64 senders (a 65th is refused with 503: its member's next + sync brings it); one worker applies everything waiting in one change, + under one window, not one window per announcement, once 10 seconds + passed with no new one: so the answer has reached its sender before + the overlay goes down and up (a connection bound to the old interface + would not survive it), and announcements close together share the + window. + +When the overlay goes down and up, as each apply of it does, the +interface comes back as another one and the socket is bound again. +keel's firewall, where it is enabled, accepts TCP 51821 on the overlay's +interface alone (docs/apply.md). + +### What a member takes, and how it confirms + +From the rosters, the members whose evidence it verifies (above) and +that it does not know: never its own key or a key it has, never an +address off its overlay prefix, its own address, or one inside a peer's +`allowed_ips`; nothing it declares is replaced. Each becomes a peer with +the endpoint its evidence names, routed its address as a `/128`, in this +node's own spec (0013), all in one change applied with `apply +--system-only --skip-uplink` under 0018's window. Then it says +`confirming over the overlay…`, sends each new member a packet every +two seconds, which makes WireGuard start a handshake, and confirms its +window on the first handshake from a new member's key since the change +(`wg show latest-handshakes`), which only that member, reaching +this node over the new configuration, can complete; the route check of +apply.md runs first, as for every overlay change. A change that no new +member answers within the window reverts by itself, the spec is put +back as it was (so it says what the machine runs, and the members are +new again to the next sync), and those members are not tried again for +an hour (`unreached.json`), so a node that is off does not cost the +mesh an overlay change at every timer. One sync runs at a time +(`sync.lock`); a sync that finds a network change waiting in its window +leaves the members to the next one. + +## keel mesh sync + +``` +keel mesh sync [--from ADDRESS]... [--network-window SECONDS] +keel mesh sync --adopt ADDRESS +``` + +Root. Asks the roster of every peer (or of the `--from` ones, overlay +addresses of peers), at once, and adds the members they know, as +above. A peer that does not answer is said and skipped; one that +answers as another key or address is left out. A node in no mesh, or +with no peer, has nothing to sync. + +``` +member 9vm/AzCVQQsWt1cU2eyjom4cABkNuodiHL8nDjKohEE= added at fd2a:9c41:7e03::7, through [2001:db8:3::30]:51820 +confirming over the overlay… +confirmed from a WireGuard handshake from 9vm/AzCVQQsWt1cU2eyjom4cABkNuodiHL8nDjKohEE= (14:02:11 UTC), a member its peers named +``` + +`--adopt ADDRESS` repairs a split identity (below): it takes the +identity of the member at that overlay address in place of this +node's, then syncs. It is refused while an invite of this node is +pending, since its token carries the old identity. + +## The mesh's identity + +Every node of a mesh holds the same identity (`/var/lib/keel/mesh/ +identity`), which each token carries and etcd will take as its cluster +token. `create` makes it, `join` keeps the token's, and nothing else +sets it but the operator's `--adopt`: + +- a node with no identity takes none from a roster or an announcement: + its sync is refused and names `--adopt`; +- a member of another identity gives nothing; the node says which + identities differ and how to repair it; +- `invite` on a node with peers checks their identities first: another + identity than its own refuses the invite, naming `keel mesh sync + --adopt`; with no identity of its own it refuses, naming `keel mesh + create --adopt`, rather than make a second identity for a mesh. Peers + that do not answer are warned about and decide nothing. + +**A mesh built by hand**, or one whose nodes ran keel 0.17 (which kept +no evidence): `keel mesh create --adopt` on one of its nodes (web-1, +say), then, on each other node, `keel mesh sync --adopt `. Each act makes the node's spec peers its trust +roots, whose signing keys are bound as each root's members' channel +answers that node's sync. A root vouches for nobody, so a peer that a +node's spec lacks and that no member admitted with an invite is added +by hand, as the mesh was built: write it into that node's spec, then +run `keel mesh sync --adopt` there again, which makes it a root too. +Nodes invited from then on carry evidence and spread by themselves. + +**A split** is what keel 0.17 could leave: an invite on a node of a +hand-built mesh that held no identity made one of its own, while the +others held another. On the odd node (web-2 in the test that found it, +holding `cfc81bf2…` while web-1 and web-3 hold `9fe71b35…`): + +``` +keel mesh sync --adopt +``` + +takes web-1's identity, says `this node now holds the mesh identity +9fe71b35… of
(was cfc81bf2…), and trusts the peers its spec +lists as roots`, and syncs. web-3 joined web-1 with keel 0.17, which +signed no evidence, so the full repair is: `keel mesh create --adopt` +on web-1; on web-2, web-3 written into its spec as a peer (its key, its +endpoint, its overlay address as a /128), then `keel mesh sync --adopt +`; on web-3, web-2 written into its spec likewise, then `keel +mesh sync --adopt `. `keel mesh status` shows each node's +identity. + +## keel mesh remove + +``` +keel mesh remove KEY|ADDRESS [--network-window SECONDS] +``` + +Root. "Before etcd, the command removes the peer on the member it ran on +and sends the same announcement as a join, so the others drop it too" +(0048): + +1. it records a tombstone for the peer of that WireGuard key or overlay + address, signed with this node's key; +2. it removes the peer from this node's spec and applies the change + under 0018's window, confirmed as a sync's is: by a WireGuard + handshake from a peer it keeps since the change, or, with no peer + left, by the route check alone, as `keel mesh create` confirms a mesh + with no peer; not confirmed, the spec is put back and the tombstone + kept, and the next sync drops the peer again; +3. it sends its roster, the tombstone in it, to each peer it keeps. + +Each member takes the tombstone if this node may remove that peer (it +admitted it, it is a root, or it is the peer itself), and drops the peer +through one window of its own; one offline then does at its next sync. +etcd's `member remove` is not here yet. + ## keel mesh status ``` keel mesh status ``` -This node's overlay, its public key, each peer of the spec with its +This node's overlay, the mesh's identity, its public key, each peer of the spec with its `allowed_ips`, its endpoint (as WireGuard last saw it) and its last handshake, and the pending invites: id, reserved address, port, expiry, and whether it was used and waits for its confirmation. `wg show` is @@ -266,13 +618,21 @@ the body. | Request | Fields | Over | | --- | --- | --- | -| `POST /v1/join` | `invite_id`, `public_key`, `endpoint` (`[address]:port`, or null), `address` (the reserved one, with its length), `nonce` (16 random bytes, hex), `time` (seconds since the epoch) | the uplink | +| `POST /v1/join` | `invite_id`, `public_key`, `sign_key` (the new node's Ed25519 key), `endpoint` (`[address]:port`, or null), `address` (the reserved one, with its length), `nonce` (16 random bytes, hex), `time` (seconds since the epoch) | the uplink | | `POST /v1/confirm` | `invite_id`, `public_key`, `nonce`, `time` | the overlay | | Answer (200) | Fields | | --- | --- | -| to a join | `invite_id`, `nonce` (the request's), `public_key` and `address` (the inviter's), `peers` (the other members the inviter knows: `public_key`, `endpoint` or null, `address`), `etcd` (`none` in this keel), `window` (the seconds the inviter's change waits) | -| to a confirmation | `invite_id`, `nonce`, `confirmed` (whether the inviter kept its change), `detail` | +| to a join | `invite_id`, `nonce` (the request's), `public_key` and `address` (the inviter's), `sign_key` (the inviter's signing key), `admission` (the evidence of this admission, signed with it), `peers` (the other members the inviter knows: `public_key`, `endpoint` or null, `address`, and `admission`, the evidence it keeps, or null), `etcd` (`none` in this keel), `window` (the seconds the inviter's change waits) | +| to a confirmation | `invite_id`, `nonce`, `confirmed` (whether the inviter kept its change), `detail`, `sign_key` (the inviter's: what the fallback's node learns it by) | + +The evidence (`admission`) is `mesh_id` (hex), `invite_id` (empty for a +trust root's), `public_key`, `sign_key`, `address`, `endpoint` or null, +`time`, `by` (the signer's signing key) and `signature` (Ed25519, in +base64, over `keel mesh admission 1\n` and those fields, but the +signature, one per line). A tombstone is `mesh_id`, `public_key`, +`time`, `by` and `signature`, over `keel mesh removal 1\n` and those +fields. An answer is signed the same way, with the method `ANSWER` and the request's path, so the new node knows it came from the node that made @@ -280,9 +640,9 @@ the token as well as from the pinned certificate. A refusal is a JSON `{"error": reason}`, unsigned: it gives nothing to act on. The fallback's `keel1a:` line carries the join request's fields in -binary, base64url without padding like the token, about 150 characters: +binary, base64url without padding like the token, about 200 characters: flags (1 byte: an IPv6 or an IPv4 endpoint follows, at most one), the -new node's key (32), the endpoint and its UDP port (16 or 4, and 2), +new node's key (32) and signing key (32), the endpoint and its UDP port (16 or 4, and 2), the reserved address and its length (16, 1), the invite id (8), the time (4), an HMAC-SHA256 with the invite's key (32), and a checksum (the first 4 bytes of the SHA-256 of `keel1a:` and the rest), so a @@ -407,7 +767,10 @@ is taken (two lost SYNs still connect), an answer 180, the mesh session is retried every 2 seconds for the whole window, the request's time may be five minutes from the inviter's clock, and the listener's deadlines are per request. The integration test runs on clean links and again on -250 ms ±25 ms with 2% loss on every link (`tc netem`). +250 ms ±25 ms with 2% loss on every link (`tc netem`). The members' channel gives a connection 10 seconds and an +answer 30, a member's handshake is waited for the whole window with a +packet sent every 2 seconds, and the announcement and the sync ask +every member at once rather than one after another. ## The firewall port @@ -487,6 +850,10 @@ backup set. | --- | --- | | `invites/.json` | one pending invite: the reserved address, the expiry, the HTTPS port, the invite's certificate and TLS key, and the HMAC key | | `identity` | the mesh's identity, 16 bytes in hex | +| `node.key` | this node's Ed25519 signing key, PEM, never printed | +| `trust.json` | the members this node trusts (signing key, root or not, admission evidence), and the tombstones | +| `unreached.json` | the members a sync added that never answered, and when: each is left out for an hour | +| `sync.lock` | held while a sync or an announcement adds members | The invite file holds an HMAC key derived from the secret, never the secret, so it cannot be turned back into the token. An invite is @@ -495,10 +862,12 @@ request is told it was already used. The listener's end, or accept's, removes the file. The mesh identity is made once and kept: a file that does not hold one -is an error, never replaced, since every other node of the mesh knows -the first. The node that joins keeps the token's, and one that keeps -another is refused. A mesh built by hand before `keel mesh` existed gets -it from its first invite. +is an error, never replaced by an invite, since every other node of the +mesh knows the first. The node that joins keeps the token's, and one +that keeps another is refused. A mesh built by hand before `keel mesh` +existed gets it from `keel mesh create --adopt` on one node, and `keel +mesh sync --adopt` replaces it on a node that holds another than its +members' ("The mesh's identity"). **The spec** is written by `create`, the root helper, `join` and `accept` through keel's writer for its own state (keel.network.marker's @@ -527,11 +896,11 @@ listener's lines go to its own unit's journal, under the same rule. | --- | --- | | 0 | the line, the change or the status was printed; the join or the accept confirmed | | 2, 3 | the spec cannot be read or is invalid | -| 15 | `create`, `invite`, `join`, `accept`, `serve` on the live system, not as root | +| 15 | `create`, `invite`, `join`, `accept`, `serve`, `sync`, `members`, `remove` on the live system, not as root | | 16 | `wg` cannot read or make the key, `openssl` cannot make the certificate, the mesh identity is damaged, the invite's listener cannot start or bind, or this node's change did not come up | -| 21 | the change was applied and not confirmed (the other side did not answer within the window, or refused, or the route check did): it reverts by itself | +| 21 | the change was applied and not confirmed (the other side did not answer within the window, or refused, or the route check did; for `sync`, no new member completed a handshake): it reverts by itself | | 23 | the token, or `accept`'s line, is mistyped, truncated, of a later format, inconsistent, or expired | -| 24 | refused: no overlay to invite into, no key yet, no endpoint, no free address, another invite on the port; a node in another mesh or at another address of it, or a change the spec would refuse; a network change waiting; no route to the inviter, or neither node can reach the other; the inviter refused the join (used, expired, bad HMAC), its certificate is not the pinned one, or its answer is not the answer to the request | +| 24 | refused: no overlay to invite into, no key yet, no endpoint, no free address, another invite on the port; a node in another mesh or at another address of it, or a change the spec would refuse; a network change waiting; an invite whose peers hold another mesh identity, or none while this node has none; members of another identity, an address that is not a peer's, or an identity repair while an invite is pending; no route to the inviter, or neither node can reach the other; the inviter refused the join (used, expired, bad HMAC), its certificate is not the pinned one, or its answer is not the answer to the request | ## Tests @@ -564,6 +933,17 @@ At these seams, each written down before its tests: real `keel.network.confirm` deciding (`tests/test_mesh_joining.py`, `tests/test_mesh_inviting.py`, `tests/test_mesh_create_status.py`, `tests/test_network_mesh_confirm.py`); +- the endpoint found on the uplink (`tests/test_mesh_endpoint.py`), the + roster and the peers taken from it (`tests/test_mesh_members.py`), the + signing key and the evidence with the real openssl + (`tests/test_mesh_signing.py`, `tests/test_mesh_trust.py`: chains, + forgeries, keys not trusted, roots, tombstones), the listener's front + and limits with real sockets on the loopback + (`tests/test_mesh_memberlink.py`), the root helper and the listener in + one process over their unix socket (`tests/test_mesh_memberd.py`), and + sync, the announcement, the identity's adoption and repair, and + remove against a real Node (`tests/test_mesh_sync.py`, + `tests/test_mesh_adopt.py`, `tests/test_mesh_remove.py`); - the CLI through `keel.cli.main` (`tests/test_mesh_cli.py`, `tests/test_mesh_cli_live.py`); - end to end, `tests/test_mesh_netns.py` runs `tests/mesh_netns.py` as @@ -571,7 +951,14 @@ At these seams, each written down before its tests: namespaces joined to it by veths: real `wg-quick`, real TLS, the real listener, join, accept and confirmation logic. One node joins through the listener and one through the fallback, both ping the inviter over - the overlay with no `keel network confirm` typed, the spent invite is + the overlay with no `keel network confirm` typed; the inviter + announces the second to the first, which adds it, and the second + pulls the first from the inviter (`keel mesh sync`), so the two are + each other's confirmed peers and ping each other over the overlay + (routed between their namespaces through the inviter's, as the + internet would): a full mesh, on evidence the inviter signed, the + members' channels each a root helper and a listener without + capabilities; the spent invite is refused, the inviter's journal holds no secret, and the listener, a process of its own under `setpriv` with every capability dropped, reports a CapEff of 0. apply there writes keel's rendered file and diff --git a/keel/__init__.py b/keel/__init__.py index 1c3e915..3368618 100644 --- a/keel/__init__.py +++ b/keel/__init__.py @@ -6,4 +6,4 @@ clients of it. """ -__version__ = "0.17.0" +__version__ = "0.18.0" diff --git a/keel/manifest/firewall.py b/keel/manifest/firewall.py index c2422bb..f95d22e 100644 --- a/keel/manifest/firewall.py +++ b/keel/manifest/firewall.py @@ -58,6 +58,7 @@ # the TCP ports of the pending mesh invites (decision 0048): keel mesh # invite adds its port with a timeout, so it closes by itself INVITES_SET = "mesh_invites" +MEMBERS_PORT = 51821 INVITES_SET_TEXT = ( f"\tset {INVITES_SET} {{\n\t\ttype inet_service\n" "\t\tflags timeout\n\t}\n") @@ -112,6 +113,9 @@ def render(resolved: Resolved, states: dict, wireguard: dict | None, sets = "" if iface: rules += _rules(f'iifname "{iface}" ', opened_mesh) + # keel mesh sync and the announcement of a new node, between + # members, on the overlay alone (keel.mesh.members) + rules.append(f'iifname "{iface}" tcp dport {MEMBERS_PORT} accept') rules.append(f"tcp dport @{INVITES_SET} accept") sets = INVITES_SET_TEXT if bridges: diff --git a/keel/mesh/acceptline.py b/keel/mesh/acceptline.py index f242bfa..621fc7f 100644 --- a/keel/mesh/acceptline.py +++ b/keel/mesh/acceptline.py @@ -7,6 +7,8 @@ flags 1 bit 0: an IPv6 endpoint follows, bit 1: an IPv4 one public key 32 the new node's WireGuard key + signing key 32 the new node's Ed25519 key, which its admission + names (keel.mesh.trust) endpoint 16|4 when a flag is set, and its UDP port (2) address 16 the address the invite reserved prefix length 1 @@ -41,7 +43,7 @@ IPV4_ENDPOINT = 0x02 LABEL = b"keel mesh accept\n" MAC_BYTES = 32 -HEAD = struct.Struct(f">B{KEY_BYTES}s") +HEAD = struct.Struct(f">B{KEY_BYTES}s{KEY_BYTES}s") PORT = struct.Struct(">H") BODY = struct.Struct(f">16sB8sI{MAC_BYTES}s") @@ -60,6 +62,7 @@ class Accept: address: str invite_id: str time: int + sign_key: str @dataclass(frozen=True) @@ -99,7 +102,8 @@ def encode(accept: Accept, key: bytes) -> str: flags = IPV6_ENDPOINT if address.version == 6 else IPV4_ENDPOINT tail = address.packed + PORT.pack(int(port)) own = ipaddress.IPv6Interface(accept.address) - covered = HEAD.pack(flags, key_bytes(accept.public_key)) + tail + covered = HEAD.pack(flags, key_bytes(accept.public_key), + key_bytes(accept.sign_key)) + tail covered += BODY.pack(own.ip.packed, own.network.prefixlen, bytes.fromhex(accept.invite_id), accept.time, b"")[:-MAC_BYTES] @@ -137,7 +141,7 @@ def parse(text: str) -> Sealed: def unpack(payload: bytes) -> Sealed: - flags, public = HEAD.unpack_from(payload) + flags, public, signer = HEAD.unpack_from(payload) if flags not in (0, IPV6_ENDPOINT, IPV4_ENDPOINT): raise ValueError(f"flags {flags:#04x}: one endpoint at most") offset, shown = HEAD.size, None @@ -155,13 +159,16 @@ def unpack(payload: bytes) -> Sealed: accept=Accept(public_key=base64.b64encode(public).decode(), endpoint=shown, address=f"{ipaddress.IPv6Address(address)}/{length}", - invite_id=invite.hex(), time=time), + invite_id=invite.hex(), time=time, + sign_key=base64.b64encode(signer).decode()), covered=payload[:-MAC_BYTES], mac=sealed) def check(accept: Accept) -> None: if key_bytes(accept.public_key) is None: raise AcceptError("the new node's key is not a WireGuard key") + if key_bytes(accept.sign_key) is None: + raise AcceptError("the new node's signing key is not a key") try: endpoint(accept.endpoint) except ProtocolError as e: diff --git a/keel/mesh/admit.py b/keel/mesh/admit.py index 009b03b..9092d4d 100644 --- a/keel/mesh/admit.py +++ b/keel/mesh/admit.py @@ -33,12 +33,13 @@ import sys import time from collections.abc import Callable -from dataclasses import dataclass +from dataclasses import dataclass, replace from datetime import datetime, timedelta -from keel.mesh import invites, protocol +from keel.mesh import identity, invites, protocol, signing, trust from keel.mesh.node import Node, NodeError from keel.mesh.protocol import ProtocolError +from keel.mesh.signing import SigningError from keel.network import marker, session from keel.network.wireguard import same_key @@ -238,11 +239,15 @@ def admitted(self, request: protocol.JoinRequest, peer: str) -> Response: if request.endpoint: entry["endpoint"] = request.endpoint try: + evidence = self.evidence(request.public_key, request.sign_key, + address, request.endpoint) change = self.node.admit(entry) + for line in change.lines: + self.log(f"{self.name}: {line}") problem = None if change.made else ( f"this node's change did not come up (apply exited" f" {change.code})") - except NodeError as e: + except (NodeError, ValueError, SigningError) as e: problem = str(e) if problem: self.confirmed = False @@ -256,12 +261,21 @@ def admitted(self, request: protocol.JoinRequest, peer: str) -> Response: answer = protocol.JoinAnswer( invite_id=self.invite.invite_id, nonce=request.nonce, public_key=self.public_key, address=self.address, - peers=self.node.peers(request.public_key), etcd="none", - window=window) + peers=evidenced(self.root, self.node.peers(request.public_key)), + etcd="none", window=window, sign_key=evidence.by, + admission=evidence) self.log(f"{self.name}: peer applied; waiting {window} s for the" " new node over the overlay") return self.signed(protocol.JOIN, protocol.dumps(answer)) + def evidence(self, key: str, sign_key: str, address: str, + endpoint: str | None) -> protocol.Admission: + """The new node's admission, signed with this node's key and + kept in its trust store (keel.mesh.trust); raises ValueError or + SigningError""" + return admitted(self.root, self.invite.invite_id, key, sign_key, + address, endpoint, self.clock()) + def confirm(self, signature: str | None, body: bytes | None) -> Response: if self.joined is None: @@ -269,6 +283,12 @@ def confirm(self, signature: str | None, request = self.checked(protocol.CONFIRM, signature, body) if request.public_key != self.joined.public_key: raise Refusal(403, "the confirmation is not the new node's") + try: + # the answer names it: the fallback's node learns it so + signer = signing.public(self.root) + except SigningError as e: + raise Refusal(500, f"this node cannot name its signing key" + f" ({e})") from None seen = self.handshake() if seen is None: raise Refusal(409, f"no WireGuard handshake from" @@ -285,7 +305,7 @@ def confirm(self, signature: str | None, self.confirmed = confirmed answer = protocol.ConfirmAnswer( invite_id=self.invite.invite_id, nonce=request.nonce, - confirmed=confirmed, detail="; ".join(lines)) + confirmed=confirmed, detail="; ".join(lines), sign_key=signer) found = self.signed(protocol.CONFIRM, protocol.dumps(answer)) return Response(found.status, found.body, found.signature, True) @@ -309,6 +329,35 @@ def signed(self, path: str, body: bytes) -> Response: self.invite.hmac_key, protocol.ANSWER, path, body)) +def admitted(root: str, invite_id: str, key: str, sign_key: str, + address: str, endpoint: str | None, + now: datetime) -> protocol.Admission: + """Evidence that this node admits `key` by the invite, signed and + kept; raises ValueError (no identity, a damaged store) or + SigningError""" + mesh_id = identity.read(root) + if mesh_id is None: + raise ValueError("this node holds no mesh identity to admit into") + signing.ensure(root) + found = trust.admit(root, mesh_id, invite_id, key, sign_key, address, + endpoint, now) + store = trust.load(root) + trust.recorded(store, found) + trust.save(root, store) + return found + + +def evidenced(root: str, peers: tuple[protocol.Peer, ...]) -> tuple[ + protocol.Peer, ...]: + """`peers` with the evidence this node keeps for each, if any""" + try: + store = trust.load(root) + except ValueError: + store = trust.Store() + return tuple(replace(one, admission=store.evidence(one.public_key)) + for one in peers) + + def stopped(admitter: Admitter) -> str: """How the invite ended, in one line for the log""" if admitter.cancelled: diff --git a/keel/mesh/adopt.py b/keel/mesh/adopt.py new file mode 100644 index 0000000..bb90cc8 --- /dev/null +++ b/keel/mesh/adopt.py @@ -0,0 +1,182 @@ +# Copyright (c) 2026 KeelLinux maintainers +"""The mesh's identity and trust roots, set by the operator (0048) + +Only a join (keel.mesh.joining) or an explicit act of the operator sets +a node's mesh identity; no roster does (keel.mesh.sync). The acts are: + +- `keel mesh create --adopt` (`adopt_mesh`), on one node of a mesh built + by hand before `keel mesh` existed: its own identity, else the one + its peers hold, else a new one; +- `keel mesh sync --adopt ADDRESS` (`adopt_from`), on each other node, + and to repair a split: the identity of the member at ADDRESS in place + of this node's. + +Each also makes the peers the spec lists this node's trust roots +(keel.mesh.trust): a mesh built by hand has no admission evidence, and +the operator's act is what vouches for the nodes it lists. And `keel +mesh invite` never makes a second identity for a mesh +(`invite_identity`). +""" + +from collections.abc import Callable + +from keel import exits +from keel.mesh import identity, invites, signing, trust +from keel.mesh.node import NodeError +from keel.mesh.signing import SigningError +from keel.mesh.sync import Syncer, asked, pull, short + + +class Refused(ValueError): + """An invite that would split the mesh's identity, and why""" + + +def rooted(syncer: Syncer) -> trust.Store: + """This node's signing key made, and the peers of its spec its trust + roots; raises SigningError, ValueError (a damaged store)""" + signing.ensure(syncer.root) + store = trust.load(syncer.root) + trust.make_roots(store, (one.public_key + for one in syncer.node.peers(""))) + trust.save(syncer.root, store) + return store + + +def adopt_from(syncer: Syncer, host: str) -> int: + """`keel mesh sync --adopt ADDRESS`: the identity of the member at + `host` in place of this node's, the spec's peers made trust roots, + then a sync""" + public, problem = syncer.node.public_key() + if public is None: + syncer.err(problem) + return exits.MESH_REFUSED + peer = next((one for one in syncer.node.peers(public) + if one.address == host), None) + if peer is None: + syncer.err(f"{host}: not a peer of this node; the identity is taken" + " from a member this node knows") + return exits.MESH_REFUSED + waiting = invites.pending(syncer.root, syncer.clock()) + if waiting: + syncer.err(f"{len(waiting)} pending invite(s) carry this node's" + " identity in their tokens: wait until they are used or" + " expire, then adopt") + return exits.MESH_REFUSED + found = asked(syncer, (peer,)) + if not found: + return exits.MESH_REFUSED + if found[0].identity is None: + syncer.err(f"member {host} holds no mesh identity: keel mesh create" + " --adopt there first") + return exits.MESH_REFUSED + try: + rooted(syncer) + except (SigningError, ValueError) as e: + syncer.err(str(e)) + return exits.MESH_REFUSED + before = identity.replace(syncer.root, found[0].identity) + syncer.err(f"this node now holds the mesh identity" + f" {short(found[0].identity)} of {host}" + f" (was {short(before) if before else 'none'}), and trusts" + " the peers its spec lists as roots") + return pull(syncer) + + +def peer_identities(syncer: Syncer) -> tuple[dict[bytes, str], int, list]: + """(each identity this node's peers hold, with the address of one of + them; how many peers it has; the rosters that answered)""" + public, problem = syncer.node.public_key() + if public is None: + raise ValueError(problem) + known = syncer.node.peers(public) + found = asked(syncer, known) + named: dict[bytes, str] = {} + for one in found: + if one.identity is not None: + named.setdefault(one.identity, one.address) + return named, len(known), found + + +def adopt_mesh(syncer: Syncer, out: Callable[[str], None]) -> int: + """`keel mesh create --adopt`: one identity for a mesh built by hand + + On the node chosen: its own identity, else the one its peers hold, + else a new one; refused when its peers hold another than its own, + or disagree. The spec's peers become its trust roots, and those + that answered with the identity are bound to their signing keys. + """ + try: + overlay = syncer.node.overlay() + if not overlay.get("address"): + raise ValueError("this node has no overlay: keel mesh create" + " makes a new mesh, without --adopt") + own = identity.read(syncer.root) + holders, _, rosters = peer_identities(syncer) + except (ValueError, NodeError) as e: + syncer.err(str(e)) + return exits.MESH_REFUSED + named = set(holders) + if own is not None and named - {own}: + syncer.err(f"this node holds {short(own)} and its peers" + f" {', '.join(short(one) for one in sorted(named))}: on" + " this node, keel mesh sync --adopt with the address of" + " a member of the identity to keep") + return exits.MESH_REFUSED + if own is None and len(named) > 1: + syncer.err("the peers name different mesh identities: keep one" + " with keel mesh sync --adopt on the others") + return exits.MESH_REFUSED + try: + store = rooted(syncer) + except (SigningError, ValueError) as e: + syncer.err(str(e)) + return exits.MESH_REFUSED + if own is None: + own = named.pop() if named else identity.ensure(syncer.root) + identity.adopt(syncer.root, own) + # fetched by this node from each root's own address (`asked`) + for one in rosters: + if one.identity == own: + trust.bind_root(store, one.public_key, one.sign_key) + trust.save(syncer.root, store) + out(f"the mesh's identity is {own.hex()}, held by this node, which" + f" trusts the {len(syncer.node.peers(''))} peer(s) its spec lists" + " as roots") + out("on each other node: keel mesh sync --adopt with this node's" + " overlay address") + return exits.OK + + +def invite_identity(syncer: Syncer) -> bytes: + """The identity an invite carries, never a second one for a mesh + + A node with no peer makes one, as before. One with peers checks what + they hold: another than its own refuses, with the repair; with none + of its own it refuses too, since only a join or the operator's + --adopt sets it. Peers that do not answer are warned about, and + decide nothing. Raises Refused, or ValueError for an identity file + damaged or a key that cannot be read. + """ + own = identity.read(syncer.root) + holders, count, rosters = peer_identities(syncer) + named = set(holders) + if count == 0: + return identity.ensure(syncer.root) + if own is None: + raise Refused( + f"this node has {count} peer(s) and no mesh identity: an invite" + " here would make a second identity for the mesh. Run keel" + " mesh create --adopt on one node of the mesh, and keel mesh" + " sync --adopt with that node's address on the others, this" + " one included") + if named - {own}: + other = sorted(named - {own})[0] + raise Refused( + f"this node's peers are in mesh {short(other)} and this node" + f" in {short(own)}: an invite here would add a node to neither." + f" keel mesh sync --adopt {holders[other]} takes their" + " identity, then invite again") + if len(rosters) < count: + syncer.err(f"Warning: {count - len(rosters)} of this node's {count}" + " peer(s) did not say which mesh they are in") + return own diff --git a/keel/mesh/bridge.py b/keel/mesh/bridge.py index c589def..963b600 100644 --- a/keel/mesh/bridge.py +++ b/keel/mesh/bridge.py @@ -200,6 +200,33 @@ def answered(admitter: Admitter, data: bytes) -> dict: "signature": response.signature, "final": response.final} +def connected(path: str, started, sleep: Callable[[float], None], + trusted: Callable[[int, int], None], + skipped: Callable[[int, int], None]) -> socket.socket: + """The socket at `path`, its peer checked with SO_PEERCRED against + `started.trusted(pid, uid)`; untrusted ones are skipped until the + listener shows or CONNECT_WAIT passes. Raises BridgeError""" + deadline = time.monotonic() + CONNECT_WAIT + while time.monotonic() < deadline: + conn = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + try: + conn.connect(path) + except OSError: + conn.close() + sleep(RETRY) + continue + pid, uid, _ = PEERCRED.unpack(conn.getsockopt( + socket.SOL_SOCKET, socket.SO_PEERCRED, PEERCRED.size)) + if started.trusted(pid, uid): + trusted(pid, uid) + return conn + conn.close() + skipped(pid, uid) + sleep(RETRY) + raise BridgeError(f"the listener did not connect within" + f" {CONNECT_WAIT} s") + + @dataclass class Bridge: """The root helper of one invite @@ -229,29 +256,12 @@ def run(self) -> None: def connected(self, started) -> socket.socket: """The listener's socket, its peer checked; untrusted ones are skipped until the listener shows or CONNECT_WAIT passes""" - deadline = time.monotonic() + CONNECT_WAIT - while time.monotonic() < deadline: - conn = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - try: - conn.connect(self.path) - except OSError: - conn.close() - self.sleep(RETRY) - continue - pid, uid, _ = PEERCRED.unpack(conn.getsockopt( - socket.SOL_SOCKET, socket.SO_PEERCRED, PEERCRED.size)) - if started.trusted(pid, uid): - self.log(f"invite {self.params.invite_id}: the listener" - f" (pid {pid}, uid {uid}) serves TCP port" - f" {self.params.port}") - return conn - conn.close() - self.log(f"invite {self.params.invite_id}: skipped process" - f" {pid} (uid {uid}) at {self.path}: it is not the" - " listener") - self.sleep(RETRY) - raise BridgeError(f"the listener did not connect within" - f" {CONNECT_WAIT} s") + name = f"invite {self.params.invite_id}" + return connected(self.path, started, self.sleep, lambda pid, uid: ( + self.log(f"{name}: the listener (pid {pid}, uid {uid}) serves" + f" TCP port {self.params.port}")), lambda pid, uid: ( + self.log(f"{name}: skipped process {pid} (uid {uid}) at" + f" {self.path}: it is not the listener"))) def exchange(self, conn: socket.socket) -> None: with pem_fds(self.certificate, self.tls_key) as fds: diff --git a/keel/mesh/channel.py b/keel/mesh/channel.py index b64f533..7db9880 100644 --- a/keel/mesh/channel.py +++ b/keel/mesh/channel.py @@ -159,11 +159,11 @@ def exchange(tls: ssl.SSLSocket, host: str, port: int, path: str, try: connection.request(protocol.METHOD, path, body, headers) response = connection.getresponse() - data = response.read(protocol.MAX_BODY + 1) + data = response.read(protocol.MAX_ANSWER + 1) except (OSError, http.client.HTTPException) as e: raise ChannelError(f"{shown(host, port)} did not answer: {e}") \ from None - if len(data) > protocol.MAX_BODY: + if len(data) > protocol.MAX_ANSWER: raise ChannelError(f"the answer from {shown(host, port)} is longer" " than any the protocol has") return response.status, response.getheader(protocol.SIGNATURE), data diff --git a/keel/mesh/commands.py b/keel/mesh/commands.py index f4c935a..95e151a 100644 --- a/keel/mesh/commands.py +++ b/keel/mesh/commands.py @@ -18,6 +18,8 @@ import secrets import signal import sys +import threading +from collections.abc import Callable from dataclasses import replace from datetime import datetime, timezone @@ -27,24 +29,24 @@ from keel.commands import error, manifest_facts, read_spec from keel.inspect import ROOT_DEFAULT from keel.mesh import ( + adopt, allocate, bridge, certificate, create, + endpoint, identity, invites, inviting, join, joining, + memberd, ports, + remove, status, + sync, ) -from keel.mesh.node import ( - Node, - NodeError, - global_addresses, - static_addresses, -) +from keel.mesh.node import Node, NodeError, static_addresses from keel.mesh.token import ( LIFETIME, MAX_TEXT, @@ -78,13 +80,17 @@ def mesh_invite(args) -> int: if public is None: return code try: - hosts = endpoints(args.endpoint, doc) + hosts = invite_endpoints(args.endpoint, doc, root == ROOT_DEFAULT, + err) except ValueError as e: error(str(e)) return exits.MESH_REFUSED try: + mesh_id = invite_identity(root, args.spec) tls_key, cert = certificate.make() - mesh_id = identity.ensure(root) + except adopt.Refused as e: + error(str(e)) + return exits.MESH_REFUSED except (certificate.CertificateError, ValueError) as e: error(str(e)) return exits.APPLY_FAILED @@ -109,6 +115,16 @@ def mesh_invite(args) -> int: return exits.OK +def invite_identity(root: str, path: str) -> bytes: + """The mesh identity the invite carries: on the live system, never a + second one for the mesh its peers are in (keel.mesh.sync); raises + sync.Refused, or ValueError""" + if root != ROOT_DEFAULT: + return identity.ensure(root) + return adopt.invite_identity(sync.Syncer( + Node(root, path), utcnow, err)) + + def listening(made: invites.Pending, path: str, root: str, now: datetime) -> tuple[str, int]: """Start the invite's listener and open its port; what was done @@ -212,16 +228,26 @@ def public_key(root: str, overlay: dict) -> tuple[str | None, int]: return public, exits.OK -def endpoints(declared: list[str] | None, doc: dict) -> tuple[str, ...]: +def invite_endpoints(declared: list[str] | None, doc: dict, live_system: bool, + say: Callable[[str], None]) -> tuple[str, ...]: """Where the new node reaches this one: IPv6 first, one per family The --endpoint addresses when given (a port forward, say), else the - static addresses network.interfaces declares. Raises ValueError. + static addresses network.interfaces declares, else, on the live + system, the uplink's (keel.mesh.endpoint), each said with `say`. + Raises ValueError. """ if declared: hosts = [ipaddress.ip_address(one) for one in declared] else: hosts = list(static_addresses(doc)) + why: tuple[str, ...] = () + if not hosts and live_system: + found = endpoint.detect(live.output) + for line in found.said: + say(line) + why = found.refused + hosts = [ipaddress.ip_address(one) for one in found.addresses] found = {} for host in hosts: if host.version in found and declared: @@ -230,9 +256,11 @@ def endpoints(declared: list[str] | None, doc: dict) -> tuple[str, ...]: found.setdefault(host.version, host) if not found: raise ValueError( - "no endpoint: network.interfaces declares no static address" - " the new node could reach this one at; give it with" - " --endpoint ADDRESS") + "no endpoint: network.interfaces declares no static address," + " and none was found on the uplink, that the new node could" + f" reach this one at{': ' if why else ''}{'; '.join(why)}." + " Give it with --endpoint ADDRESS (a stable address, or a port" + " forward's)") return tuple(str(found[version]) for version in (6, 4) if version in found) @@ -271,18 +299,19 @@ def mesh_join(args) -> int: node = live_node(args) try: doc = node.document() - endpoint = joining.own_endpoint( - args.endpoint, doc, - global_addresses(live.output(GLOBAL_ADDRESSES) or ""), - wireguard.interface(overlay_of(doc) or {})) + found = None + if not args.endpoint and not list(static_addresses(doc)): + found = endpoint.detect(live.output) + for line in found.said: + err(line) + for line in found.refused: + err(f"no endpoint is sent, as from a node behind NAT:" + f" {line}; --endpoint gives one") + own = joining.own_endpoint(args.endpoint, doc, found) except (NodeError, ValueError) as e: error(str(e)) return exits.MESH_REFUSED - return joining.run(joining.Joiner(node, token, utcnow, out, err), - endpoint) - - -GLOBAL_ADDRESSES = ("ip", "-o", "address", "show", "scope", "global") + return joining.run(joining.Joiner(node, token, utcnow, out, err), own) def join_dry_run(args) -> int: @@ -352,13 +381,75 @@ def as_root(args, verb: str) -> int: def mesh_create(args) -> int: - """Make the mesh on its first node""" + """Make the mesh on its first node; with --adopt, give a mesh built + by hand its identity""" code = as_root(args, "keel mesh create") if code != exits.OK: return code + if args.adopt: + return adopt.adopt_mesh(syncer(args), out) return create.create(live_node(args), out, err) +def syncer(args) -> sync.Syncer: + return sync.Syncer(live_node(args), utcnow, err) + + +def overlay_address(text: str) -> str: + """An overlay address as the spec writes it; ValueError""" + try: + return str(ipaddress.IPv6Address(text)) + except ValueError: + raise ValueError(f"{text}: not an IPv6 address; a member is named" + " by its overlay address") from None + + +def mesh_sync(args) -> int: + """Pull the members this node's peers know; with --adopt, take a + member's identity first""" + code = as_root(args, "keel mesh sync") + if code != exits.OK: + return code + try: + hosts = tuple(overlay_address(one) for one in args.source or ()) + adopted = overlay_address(args.adopt) if args.adopt else None + except ValueError as e: + error(str(e)) + return exits.MESH_REFUSED + if adopted: + return adopt.adopt_from(syncer(args), adopted) + return sync.pull(syncer(args), hosts) + + +def mesh_members(args) -> int: + """What keel-mesh-members.service runs: the members' channel's root + helper, which starts its unprivileged listener""" + code = as_root(args, "keel mesh members") + if code != exits.OK: + return code + stop = threading.Event() + + def stopped(signum, frame): + stop.set() + # out of the bridge's read, through serve's cleanup + raise SystemExit(0) + signal.signal(signal.SIGTERM, stopped) + return memberd.serve(syncer(args), stop) + + +def mesh_members_listen(args) -> int: + """What keel-mesh-members-listen runs: the unprivileged listener""" + return memberd.listen(args.socket, utcnow, err) + + +def mesh_remove(args) -> int: + """A peer out of this node's spec, and its tombstone""" + code = as_root(args, "keel mesh remove") + if code != exits.OK: + return code + return remove.remove(syncer(args), args.member, out) + + def inviter(args) -> inviting.Inviter: return inviting.Inviter(live_node(args), utcnow, out, err) diff --git a/keel/mesh/create.py b/keel/mesh/create.py index a3dfc33..ecce555 100644 --- a/keel/mesh/create.py +++ b/keel/mesh/create.py @@ -45,12 +45,18 @@ def create(node: Node, out: Callable[[str], None], err(str(e)) return exits.MESH_REFUSED if change.made is None: + change.shown(err) err(f"the overlay did not come up under the window (apply exited" f" {change.code}); the spec declares it, and the next apply" " --system brings it up") return exits.APPLY_FAILED + # apply's lines are held: they say the change reverts unless + # confirmed, and create confirms it itself; shown when it cannot + err("confirming the new overlay…") kept, lines = node.confirm( session.Origin(session.SELF, "keel mesh create"), change.made) + if not kept: + change.shown(err) for line in lines: err(line) if not kept: diff --git a/keel/mesh/endpoint.py b/keel/mesh/endpoint.py new file mode 100644 index 0000000..49f8bfb --- /dev/null +++ b/keel/mesh/endpoint.py @@ -0,0 +1,144 @@ +# Copyright (c) 2026 KeelLinux maintainers +"""The address another node reaches this one at, found on the uplink + +`keel mesh invite` puts this node's endpoint in the token, and `keel +mesh join` sends its own in the request. `--endpoint` gives it, and a +static address network.interfaces declares is taken next; a host whose +uplink is configured by DHCP or SLAAC declares none, so it is found +here, from what `ip` says the machine holds: + +- on the uplink, the interfaces the default routes leave through (any + interface when there is no default route), never a WireGuard one + (`wg*`), whose addresses are the overlay's; +- a global IPv6 address that is not a unique local one (fc00::/7), not + deprecated and not a SLAAC privacy address (`temporary`), a static + one before a SLAAC or DHCPv6 one (`dynamic`): the order of keel-core's + console banner (keel_banner_pick_ipv6) and confconsole; +- beside it, or instead of it, a public IPv4 address. + +A privacy address changes within a day and an RFC 1918 or carrier NAT +one is reached from its own network only, so neither is ever taken: a +host that holds nothing better has no endpoint found, and the reason +says so and names `--endpoint` (`invite` refuses; `join` sends none, as +a node behind NAT does). +""" + +import ipaddress +from collections.abc import Callable +from dataclasses import dataclass + +HELD = ("ip", "-o", "address", "show", "scope", "global") +ROUTES = (("ip", "-6", "route", "show", "default"), + ("ip", "-4", "route", "show", "default")) +OVERLAY_PREFIX = "wg" +ULA = ipaddress.ip_network("fc00::/7") +# RFC 1918, and the shared space of RFC 6598 a carrier's NAT gives +PRIVATE4 = tuple(ipaddress.ip_network(one) for one in ( + "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "100.64.0.0/10")) +STATIC, DYNAMIC = 0, 1 + + +@dataclass(frozen=True) +class Held: + """An address an interface holds, and the flags `ip` gave it""" + + iface: str + address: ipaddress.IPv4Address | ipaddress.IPv6Address + flags: frozenset[str] + + +@dataclass(frozen=True) +class Choice: + """The addresses chosen, IPv6 first, what to tell the operator, and, + when none was, why""" + + addresses: tuple[str, ...] + said: tuple[str, ...] = () + refused: tuple[str, ...] = () + + +def held(text: str) -> list[Held]: + """`ip -o address show scope global` output, in its order""" + found = [] + for line in text.splitlines(): + fields = line.replace("\\", " ").split() + if len(fields) < 4 or fields[2] not in ("inet", "inet6"): + continue + try: + address = ipaddress.ip_interface(fields[3]).ip + except ValueError: + continue + found.append(Held(fields[1].split("@")[0], address, + frozenset(fields[4:]))) + return found + + +def uplinks(text: str) -> tuple[str, ...]: + """`ip route show default` output: the interfaces, in order""" + found = [] + for line in text.splitlines(): + fields = line.split() + if "dev" in fields and fields.index("dev") + 1 < len(fields): + found.append(fields[fields.index("dev") + 1]) + return tuple(dict.fromkeys(found)) + + +def candidates(found: list[Held], links: tuple[str, ...]) -> list[Held]: + return [one for one in found + if not one.iface.startswith(OVERLAY_PREFIX) + and (not links or one.iface in links) + and not (one.address.is_loopback or one.address.is_link_local + or one.address.is_multicast)] + + +def rank(one: Held) -> int: + return DYNAMIC if "dynamic" in one.flags else STATIC + + +def pick(found: list[Held], links: tuple[str, ...]) -> Choice: + """The endpoints among `found`, on the interfaces `links`""" + usable = candidates(found, links) + six = [one for one in usable if one.address.version == 6 + and one.address not in ULA and "deprecated" not in one.flags] + stable = sorted((one for one in six if "temporary" not in one.flags), + key=rank) + privacy = [one for one in six if "temporary" in one.flags] + four = [one for one in usable if one.address.version == 4] + public = [one for one in four + if not any(one.address in net for net in PRIVATE4)] + chosen = stable[:1] + public[:1] + if chosen: + said = [described(one) for one in chosen] + if privacy and not stable: + said.append(f"the SLAAC privacy address {privacy[0].address} was" + " left out: it changes within a day") + return Choice(tuple(str(one.address) for one in chosen), + tuple(said)) + refused = [] + if privacy: + refused.append( + f"{privacy[0].address} is a SLAAC privacy address, the only" + f" global IPv6 one {privacy[0].iface} holds: it changes within" + " a day, and the other node's peer entry would then point" + " nowhere") + if four: + refused.append( + f"{four[0].address} is a private address (RFC 1918 or carrier" + f" NAT), the only IPv4 one {four[0].iface} holds: a node outside" + " its network reaches it only through a port forward") + if not refused: + refused.append("no global address on the uplink" + f" ({', '.join(links) or 'no default route'}) that" + " another node could reach") + return Choice((), (), tuple(refused)) + + +def described(one: Held) -> str: + return (f"endpoint {one.address}, found on {one.iface}" + " (--endpoint overrides it)") + + +def detect(output: Callable[[tuple[str, ...]], str | None]) -> Choice: + """The endpoints of this machine, as `ip` shows it now""" + routes = "".join(output(argv) or "" for argv in ROUTES) + return pick(held(output(HELD) or ""), uplinks(routes)) diff --git a/keel/mesh/identity.py b/keel/mesh/identity.py index 77a9d55..af020be 100644 --- a/keel/mesh/identity.py +++ b/keel/mesh/identity.py @@ -5,8 +5,11 @@ the nodes a join adds know which mesh they are in, and etcd takes them as its cluster token when it forms at the third node (decision 0048). The node that creates the mesh makes them; a mesh built by hand before -`keel mesh` existed gets them from its first invite. A joining node -keeps the identity its token carries. +`keel mesh` existed gets them from `keel mesh create --adopt` on one of +its nodes, and the others from it through `keel mesh sync`. A joining +node keeps the identity its token carries, and a node that holds +another than its members' takes theirs with `keel mesh sync --adopt` +(keel.mesh.sync). """ import secrets @@ -48,6 +51,24 @@ def adopt(root: str, found: bytes) -> None: " names another one than the token's") +def replace(root: str, found: bytes) -> bytes | None: + """Take `found` in place of the identity this node keeps, which is + returned (None when it kept none) + + The repair of a split (keel mesh sync --adopt): a node that made an + identity of its own in a mesh whose members keep another takes + theirs. A damaged file is replaced too: the members' identity is + the one to keep. + """ + with locked(root): + try: + before = read(root) + except ValueError: + before = None + write_private(root, IDENTITY, found.hex() + "\n") + return before + + def read(root: str) -> bytes | None: """The identity, None when not made yet; ValueError when damaged""" try: diff --git a/keel/mesh/inviting.py b/keel/mesh/inviting.py index 59700c8..642496a 100644 --- a/keel/mesh/inviting.py +++ b/keel/mesh/inviting.py @@ -9,7 +9,9 @@ (`serve` here), which starts the unprivileged listener on the invite's TCP port, on every address, and answers what it forwards through keel.mesh.bridge; when it ends, however it ends, it stops the listener -and removes the invite's file and its port from keel's firewall. +and removes the invite's file and its port from keel's firewall. A join +confirmed is then announced to this node's other peers over the overlay +(keel.mesh.sync), so the mesh stays a full one. `keel mesh accept keel1a:...` is the fallback's other half: the new node could not reach the port, applied its own side, and printed the line. @@ -28,11 +30,12 @@ from datetime import datetime, timedelta from keel import exits -from keel.mesh import acceptline, bridge, invites, ports -from keel.mesh.admit import Admitter, Joined, stopped +from keel.mesh import acceptline, bridge, invites, ports, sync +from keel.mesh.admit import Admitter, Joined, admitted, stopped from keel.mesh.bridge import Bridge, BridgeError, socket_path from keel.mesh.listener import Params from keel.mesh.node import Node, NodeError +from keel.mesh.signing import SigningError from keel.network import live, wgkeys, wireguard from keel.system import DEFAULT_WINDOW @@ -79,11 +82,20 @@ class Inviter: bridged: Callable[[Bridge], None] = Bridge.run output: Callable[[tuple[str, ...]], str | None] = field( default=live.output) + # tells the other members of a confirmed join (keel.mesh.sync) + announce: Callable[[str], None] | None = None @property def root(self) -> str: return self.node.root + def announced(self, key: str) -> None: + """The new node `key` announced to this node's other peers""" + if self.announce is not None: + self.announce(key) + return + sync.announce(sync.Syncer(self.node, self.clock, self.err), key) + def own(self) -> tuple[str | None, str]: """(public key, overlay address with its length); key None and the reason when it cannot be read""" @@ -156,6 +168,8 @@ def serve_invite(inviter: Inviter, invite_id: str) -> int: finally: inviter.close(found) inviter.err(stopped(admitter)) + if admitter.confirmed: + inviter.announced(admitter.joined.public_key) return code @@ -214,11 +228,14 @@ def accepted(inviter: Inviter, spent: invites.Pending, return exits.MESH_REFUSED since = int(inviter.clock().timestamp()) try: + admitted(inviter.root, spent.invite_id, line.public_key, + line.sign_key, joiner, line.endpoint, inviter.clock()) change = inviter.node.admit(peer) - except NodeError as e: + except (NodeError, ValueError, SigningError) as e: inviter.err(f"{e}; the invite is spent") return exits.MESH_REFUSED if change.made is None: + change.shown(inviter.err) inviter.err(f"this node's change did not come up (apply exited" f" {change.code}); the invite is spent") return exits.APPLY_FAILED @@ -238,12 +255,16 @@ def accepted(inviter: Inviter, spent: invites.Pending, spent.invite_id, spent.expires, own, spent.https_port, own, (line.public_key, joiner, until))) except (BridgeError, OSError, ssl.SSLError) as e: + change.shown(inviter.err) inviter.err(f"cannot serve [{own}]:{spent.https_port}: {e}; this" " node's change reverts by itself") return exits.NETWORK_NOT_CONFIRMED if not admitter.confirmed: + # apply's lines, held while this node confirmed itself + change.shown(inviter.err) inviter.err(stopped(admitter)) return exits.NETWORK_NOT_CONFIRMED + inviter.announced(line.public_key) inviter.out(f"accepted: {line.public_key} is a peer of this node at" f" {joiner}") return exits.OK diff --git a/keel/mesh/joining.py b/keel/mesh/joining.py index 0cddf73..516857b 100644 --- a/keel/mesh/joining.py +++ b/keel/mesh/joining.py @@ -10,7 +10,8 @@ certificate the token names: its key, its endpoint, the reserved address, a nonce and the time, under an HMAC with the invite's key. 4. With the signed answer, it writes its own spec (its address, the - inviter as a peer) and applies it under the window. + inviter and the other members it names as peers) and applies it + under the window. 5. It opens the mesh session over the overlay, to the inviter's overlay address: the inviter confirms its own window on receiving it, and this node confirms its own on the inviter's signed answer. @@ -18,7 +19,13 @@ When the HTTPS port gives no TCP answer, this node applies its own side under a window lengthened to the invite's remaining time (up to 15 minutes), prints `keel mesh accept keel1a:...` for the inviter, and -waits for the inviter over the overlay, as in step 5. +waits for the inviter over the overlay, as in step 5; then it pulls +the other members from the inviter (keel mesh sync), which the line +could not carry. + +The answer of a join carries the other members the inviter knows: they +are peers of this node from its first change, and the inviter announces +this node to them (keel.mesh.sync). """ import ipaddress @@ -29,12 +36,23 @@ from datetime import datetime from keel import exits -from keel.mesh import acceptline, channel, identity, join, protocol +from keel.mesh import ( + acceptline, + channel, + identity, + join, + members, + protocol, + signing, + sync, + trust, +) from keel.mesh.acceptline import Accept from keel.mesh.channel import ChannelError, Forged, Refused, Unreachable -from keel.mesh.node import Node, NodeError, static_addresses +from keel.mesh.endpoint import Choice +from keel.mesh.node import Change, Node, NodeError, static_addresses from keel.mesh.token import Token, hmac_key, shown -from keel.network import live, marker, session, wgkeys, wireguard +from keel.network import live, session, wgkeys, wireguard from keel.system.ovstate import overlay_of # how often the mesh session is tried while the tunnel comes up @@ -61,11 +79,20 @@ class Joiner: post: Callable[..., channel.Reply] = channel.post sleep: Callable[[float], None] = time.sleep route_dev: Callable[[str], str | None] = field(default=live.route_dev) + # pulls the members from the inviter at its overlay address + learn: Callable[[str], int] | None = None @property def key(self) -> bytes: return hmac_key(self.token.secret) + def learned(self, inviter: str) -> int: + """keel mesh sync from the inviter alone (keel.mesh.sync)""" + if self.learn is not None: + return self.learn(inviter) + return sync.pull(sync.Syncer(self.node, self.clock, self.err), + (inviter,)) + def refused(self, message: str, code: int = exits.MESH_REFUSED) -> int: self.err(message) return code @@ -107,12 +134,16 @@ def run(joiner: Joiner, endpoint: str | None) -> int: public, problem = own_key(joiner.node.root, overlay) if public is None: return joiner.refused(problem, exits.APPLY_FAILED) + try: + signer = signing.ensure(joiner.node.root) + except signing.SigningError as e: + return joiner.refused(str(e), exits.APPLY_FAILED) own = f"{endpoint_text(endpoint)}:{wireguard.port(overlay)}" \ if endpoint else None request = protocol.JoinRequest( invite_id=token.invite_id, public_key=public, endpoint=own, address=token.assigned, nonce=protocol.new_nonce(), - time=protocol.seconds(joiner.clock())) + time=protocol.seconds(joiner.clock()), sign_key=signer) return requested(joiner, after, request, reach) @@ -141,14 +172,56 @@ def requested(joiner: Joiner, after: dict, request: protocol.JoinRequest, not wireguard.same_key(answer.public_key, token.public_key): return joiner.refused("the inviter's answer is not the answer to" " this request") + if not admitted_by(answer, request, token): + return joiner.refused("the inviter's answer carries no valid" + " evidence of this node's admission") # kept only once the inviter has admitted this node identity.adopt(joiner.node.root, token.mesh_id) + # the other members the inviter knows, peers of this node too, those + # whose admission it can verify: the inviter announces this node to + # them (keel.mesh.sync) + try: + taken = trusting(joiner, answer.sign_key, request.sign_key, + answer.peers) + except ValueError as e: + return joiner.refused(str(e)) + after, _ = members.with_members(after, taken, request.public_key) change = applied(joiner, after) if change is None: return exits.APPLY_FAILED return confirmed(joiner, change, request.public_key, answer.peers) +def admitted_by(answer: protocol.JoinAnswer, request: protocol.JoinRequest, + token: Token) -> bool: + """Whether the answer's evidence is this node's admission, signed by + the inviter's signing key the answer names""" + found = answer.admission + return (found.by == answer.sign_key + and wireguard.same_key(found.public_key, request.public_key) + and found.sign_key == request.sign_key + and found.address == str(ipaddress.IPv6Interface( + token.assigned).ip) + and found.invite_id == token.invite_id + and found.mesh_id == token.mesh_id.hex() + and signing.verified(found.by, found.message(), + found.signature)) + + +def trusting(joiner: Joiner, inviter_signer: str, own_signer: str, + peers: tuple[protocol.Peer, ...]) -> tuple[protocol.Peer, ...]: + """The inviter trusted as this node's root, its signing key as the + invite's HMAC authenticated it; the members of `peers` whose evidence + chains to it. Raises ValueError for a damaged trust store""" + root = joiner.node.root + store = trust.load(root) + trust.make_roots(store, (joiner.token.public_key,)) + trust.bind_root(store, joiner.token.public_key, inviter_signer) + taken = trust.accepted(store, own_signer, joiner.token.mesh_id, peers) + trust.save(root, store) + return taken + + def fallback(joiner: Joiner, after: dict, request: protocol.JoinRequest) -> int: """No TCP answer: apply this side, print the line for the inviter""" @@ -171,37 +244,60 @@ def fallback(joiner: Joiner, after: dict, line = acceptline.encode(Accept( public_key=request.public_key, endpoint=request.endpoint, address=token.assigned, invite_id=token.invite_id, - time=protocol.seconds(joiner.clock())), joiner.key) + time=protocol.seconds(joiner.clock()), sign_key=request.sign_key), + joiner.key) joiner.out(f"keel mesh accept {line}") joiner.err(f"the inviter's TCP port {token.https_port} gave no answer:" " run the line above on the inviter within" f" {window // 60} minutes. This node waits for it over the" " overlay, and its change reverts by itself if it never" " comes") - return confirmed(joiner, change, request.public_key, ()) + code = confirmed(joiner, change, request.public_key, ()) + if code == exits.OK: + # the accept line carries no answer, so no member: pulled now + joiner.err("learning the other members from the inviter…") + joiner.learned(str(ipaddress.IPv6Interface(token.address).ip)) + return code def applied(joiner: Joiner, after: dict, - window: int | None = None) -> marker.Pending | None: + window: int | None = None) -> Change | None: + """This node's change, applied and waiting in its window; None when + it did not come up""" try: change = joiner.node.change(after, window) except NodeError as e: joiner.err(str(e)) return None if change.made is None: + change.shown(joiner.err) joiner.err(f"this node's overlay did not come up under the window" f" (apply exited {change.code}); the token is spent and" " the inviter's change reverts by itself: ask it for a" " new invite") - return change.made + return None + return change + + +def unconfirmed(joiner: Joiner, change: Change, message: str) -> int: + """apply's own lines, its revert warning among them, then why""" + change.shown(joiner.err) + return joiner.refused(message, exits.NETWORK_NOT_CONFIRMED) -def confirmed(joiner: Joiner, made: marker.Pending, public: str, +def confirmed(joiner: Joiner, change: Change, public: str, peers: tuple[protocol.Peer, ...]) -> int: - """The mesh session over the overlay, then this node's confirmation""" + """The mesh session over the overlay, then this node's confirmation + + apply's lines are held (keel.mesh.node.Change): this node confirms + its change itself, so "unless `keel network confirm`" is shown only + when it could not. + """ token = joiner.token + made = change.made inviter = str(ipaddress.IPv6Interface(token.address).ip) deadline = joiner.clock().timestamp() + made.window + joiner.err("confirming over the overlay…") while True: request = protocol.ConfirmRequest( invite_id=token.invite_id, public_key=public, @@ -214,35 +310,40 @@ def confirmed(joiner: Joiner, made: marker.Pending, public: str, connect_timeout=RETRY) break except (Refused, Forged) as e: - return joiner.refused(f"the mesh session was refused: {e}; this" - " node's change reverts by itself", - exits.NETWORK_NOT_CONFIRMED) + return unconfirmed(joiner, change, f"the mesh session was" + f" refused: {e}; this node's change reverts" + " by itself") except ChannelError: if joiner.clock().timestamp() + RETRY >= deadline: - return joiner.refused( - f"the inviter did not answer over the overlay at" - f" [{inviter}]:{token.https_port} within the window;" - " this node's change reverts by itself", - exits.NETWORK_NOT_CONFIRMED) + return unconfirmed( + joiner, change, f"the inviter did not answer over the" + f" overlay at [{inviter}]:{token.https_port} within the" + " window; this node's change reverts by itself") joiner.sleep(RETRY) try: answer = protocol.confirm_answer(reply.body) except protocol.ProtocolError as e: - return joiner.refused(f"the inviter's answer is malformed: {e}", - exits.NETWORK_NOT_CONFIRMED) + return unconfirmed(joiner, change, + f"the inviter's answer is malformed: {e}") if (answer.invite_id, answer.nonce) != (token.invite_id, request.nonce): - return joiner.refused("the inviter's answer is not the answer to" - " this mesh session; this node's change" - " reverts by itself", - exits.NETWORK_NOT_CONFIRMED) + return unconfirmed(joiner, change, "the inviter's answer is not the" + " answer to this mesh session; this node's change" + " reverts by itself") if not answer.confirmed: - return joiner.refused( - f"the inviter did not keep its change ({answer.detail}); this" - " node's change reverts by itself", exits.NETWORK_NOT_CONFIRMED) + return unconfirmed( + joiner, change, f"the inviter did not keep its change" + f" ({answer.detail}); this node's change reverts by itself") + try: + # the fallback's node learns the inviter's signing key here + trusting(joiner, answer.sign_key, public_signer(joiner), ()) + except (ValueError, signing.SigningError) as e: + return unconfirmed(joiner, change, str(e)) origin = session.Origin( session.MESH, f"the inviter's answer over the overlay (invite" f" {token.invite_id})", None, reply.local, reply.peer) kept, lines = joiner.node.confirm(origin, made) + if not kept: + change.shown(joiner.err) for line in lines: joiner.err(line) if not kept: @@ -251,6 +352,10 @@ def confirmed(joiner: Joiner, made: marker.Pending, public: str, return exits.OK +def public_signer(joiner: Joiner) -> str: + return signing.public(joiner.node.root) + + def summary(joiner: Joiner, peers: tuple[protocol.Peer, ...]) -> None: """What `join` prints at the end, and confconsole shows""" token = joiner.token @@ -263,10 +368,11 @@ def summary(joiner: Joiner, peers: tuple[protocol.Peer, ...]) -> None: if not peers: joiner.out(TWO_NODES) return - joiner.out(f"the mesh has {len(peers) + 2} nodes; this node reaches the" - f" inviter, and the {len(peers)} other(s) once they learn of" - " it, which this keel does not do yet; etcd is not part of" - " this keel") + joiner.out(f"the mesh has {len(peers) + 2} nodes: this node has them all" + f" as peers, and the inviter announces this node to the" + f" {len(peers)} other(s) over the overlay (one offline now" + " learns of it at its next keel mesh sync); etcd is not part" + " of this keel") def own_key(root: str, overlay: dict) -> tuple[str | None, str]: @@ -287,17 +393,15 @@ def endpoint_text(address: str) -> str: def own_endpoint(declared: str | None, doc: dict, - held: list[tuple[str, str]], iface: str) -> str | None: + found: Choice | None) -> str | None: """Where the inviter reaches this node: `--endpoint`, else a static - address network.interfaces declares, else a global address `held` - by an interface other than the overlay's; IPv6 first, None without - one another node could reach""" + address network.interfaces declares, else the one `found` on the + uplink (keel.mesh.endpoint); IPv6 first, None without one another + node could reach""" if declared: return str(ipaddress.ip_address(declared)) - found = list(static_addresses(doc)) + [ - ipaddress.ip_address(address) for name, address in held - if name != iface] - usable = sorted((one for one in found if not ( - one.is_loopback or one.is_link_local or one.is_multicast - or one.is_unspecified)), key=lambda one: one.version, reverse=True) - return str(usable[0]) if usable else None + hosts = sorted(static_addresses(doc), key=lambda one: one.version, + reverse=True) + if hosts: + return str(hosts[0]) + return found.addresses[0] if found and found.addresses else None diff --git a/keel/mesh/memberd.py b/keel/mesh/memberd.py new file mode 100644 index 0000000..b87b7b9 --- /dev/null +++ b/keel/mesh/memberd.py @@ -0,0 +1,334 @@ +# Copyright (c) 2026 KeelLinux maintainers +"""keel-mesh-members: the members' channel, split as an invite's is + +As keel#75 splits an invite (keel.mesh.bridge), the members' channel is +two processes: + +- **the root helper**, `keel mesh members` in keel-mesh-members.service, + which alone holds the spec, the trust store and this node's signing + key. It starts the listener as its own unit, keel-mesh-members-listen, + with a dynamic user, no capability and the sandbox of + keel.mesh.bridge.LISTENER_PROPERTIES, connects to the unix socket the + listener binds in its RuntimeDirectory (/run/keel/mesh-members, 0700, + the socket 0600), checks with SO_PEERCRED that the other end is the + unit's MainPID and not root, and sends the overlay's interface, + address and port, which hold no key; +- **the listener**, `keel mesh members-listen SOCKET`, which refuses to + run with any capability, binds TCP 51821 through the overlay's + interface, applies keel.mesh.memberlink's limits, and forwards what + passed to the root side, as one JSON message per line. + +The root side (`Members`) treats what the listener forwards as +untrusted: the source must be in a spec peer's allowed_ips, an +announcement must parse whole and name its sender's key. It answers the +roster at once and queues an announcement (`Pending`): one per sender, +the latest replacing an earlier one, at most MAX_PENDING senders. One +worker applies everything pending in one change, under one window +(keel.mesh.sync.announced), after verifying each entry's evidence +(keel.mesh.trust). +""" + +import base64 +import binascii +import json +import os +import socket +import sys +import threading +import time +from collections.abc import Callable +from datetime import datetime + +from keel import exits +from keel.mesh import bridge, memberlink, members, signing, sync +from keel.mesh.bridge import BridgeError, Unit, capabilities, line, send +from keel.mesh.memberlink import ANNOUNCE, GET, LIST, POST, Answer, refused +from keel.mesh.members import PORT, Roster +from keel.mesh.protocol import ProtocolError +from keel.mesh.signing import SigningError +from keel.network.wireguard import same_key + +UNIT = "keel-mesh-members-listen" +RUNTIME = "keel/mesh-members" +SOCKET = "bridge.sock" +MAX_PENDING = 64 +WORKER_POLL = 0.2 +# how long the worker waits after the last announcement before it +# applies: the answer must reach the sender before the overlay goes +# down and up, which the sockets bound to the old interface would not +# survive, and announcements close together share one window +SETTLE = 10.0 +WAIT_OVERLAY = 2.0 + + +class Pending: + """The announcements waiting: the latest per sender, MAX_PENDING + senders at most, given out once none came for `settle` seconds""" + + def __init__(self, limit: int = MAX_PENDING, settle: float = SETTLE, + clock: Callable[[], float] = time.monotonic): + self.limit = limit + self.settle = settle + self.clock = clock + self.ready = threading.Condition() + self.waiting: dict[str, Roster] = {} + self.last = 0.0 + + def put(self, source: str, found: Roster) -> bool: + with self.ready: + if source not in self.waiting and \ + len(self.waiting) >= self.limit: + return False + self.waiting[source] = found + self.last = self.clock() + self.ready.notify() + return True + + def take(self, timeout: float) -> list[tuple[str, Roster]]: + """Everything waiting, at once, once it settled; empty after + `timeout` otherwise""" + with self.ready: + if not self.waiting: + self.ready.wait(timeout) + quiet = self.last + self.settle - self.clock() + if not self.waiting or quiet > 0: + if self.waiting: + self.ready.wait(min(quiet, timeout)) + return [] + found = list(self.waiting.items()) + self.waiting.clear() + return found + + +class Members: + """The root side's answers, as a function of the request + + `roster` gives this node's roster now, or raises ValueError; + `member_of` the key of the peer whose allowed_ips hold a source + address, None for any other. + """ + + def __init__(self, roster: Callable[[], Roster], + member_of: Callable[[str], str | None], pending: Pending, + log: Callable[[str], None]): + self.roster = roster + self.member_of = member_of + self.pending = pending + self.log = log + + def handle(self, method: str, path: str, body: bytes | None, + source: str) -> Answer: + key = self.member_of(source) + if key is None: + self.log(f"refused {method} {path[:64]!r} from {source}: not a" + " peer of this node") + return refused(403, "not a member of this node's mesh") + if (method, path) == (GET, LIST): + try: + return Answer(200, members.dumps(self.roster())) + except ValueError as e: + self.log(f"cannot answer {source}: {e}") + return refused(503, "this node cannot give its roster now") + if (method, path) != (POST, ANNOUNCE) or body is None: + return refused(404, "no such request") + try: + found = members.loads(body) + except ProtocolError as e: + return refused(400, f"malformed roster: {e}") + if not same_key(found.public_key, key): + self.log(f"refused an announcement from {source}: it names" + f" {found.public_key}, and {source} is {key}'s") + return refused(403, "the roster is not the sender's") + if not self.pending.put(source, found): + return refused(503, "too many announcements wait here: the" + " next keel mesh sync brings yours") + return Answer(memberlink.ACCEPTED, b'{"queued": true}') + + +def answered(handler: Members, data: bytes) -> dict: + """The root side's answer to one message; raises BridgeError""" + try: + found = json.loads(data.decode()) + if found.get("op") != "member": + raise ValueError(f"unknown operation {found.get('op')!r}") + body = found.get("body") + answer = handler.handle( + str(found["method"]), str(found["path"]), + None if body is None else base64.b64decode(body, validate=True), + bridge.address(found["source"])) + except (ValueError, KeyError, TypeError, AttributeError, + binascii.Error) as e: + raise BridgeError(f"the listener sent what it should not: {e}") \ + from None + return {"status": answer.status, + "body": base64.b64encode(answer.body).decode()} + + +class Remote: + """The listener's backend: the root side, over the bridge; one + question at a time. A root side gone stops the listener.""" + + def __init__(self, sock: socket.socket, stop: threading.Event): + self.sock = sock + self.stop = stop + self.lock = threading.Lock() + + def handle(self, method: str, path: str, body: bytes | None, + source: str) -> Answer: + with self.lock: + try: + send(self.sock, {"op": "member", "method": method, + "path": path, "source": source, + "body": None if body is None else + base64.b64encode(body).decode()}) + found = line(self.sock) + except (OSError, BridgeError): + found = None + if found is None: + self.stop.set() + return refused(503, "the root side is gone") + data = json.loads(found.decode()) + return Answer(int(data["status"]), base64.b64decode(data["body"])) + + +def socket_path(root: str) -> str: + return os.path.join(root, "run", RUNTIME, SOCKET) + + +def listener_command(path: str) -> tuple[str, ...]: + return (sys.executable, "-m", "keel", "mesh", "members-listen", path) + + +def start_unit(path: str, run: Callable[[tuple[str, ...]], str | None], + output: Callable[[tuple[str, ...]], str | None]) -> Unit: + """Start keel-mesh-members-listen; raises BridgeError""" + run(("systemctl", "stop", UNIT)) + problem = run(("systemd-run", f"--unit={UNIT}", "--collect", "--quiet", + f"--property=RuntimeDirectory={RUNTIME}", + "--property=RuntimeDirectoryMode=0700", + *(f"--property={one}" + for one in bridge.LISTENER_PROPERTIES), + "--description=keel: the unprivileged listener of the" + " mesh members' channel", *listener_command(path))) + if problem: + raise BridgeError(f"the listener could not start: {problem}") + return Unit(UNIT, run, output) + + +def listen(path: str, clock: Callable[[], datetime], + log: Callable[[str], None], status: str = bridge.STATUS, + poll: float = memberlink.POLL, + stop: threading.Event | None = None) -> int: + """`keel mesh members-listen`: the unprivileged listener's life""" + with open(status) as fob: + found = capabilities(fob.read()) + if found != 0: + log(f"the members' listener runs without capabilities, and this" + f" process has {found:#x}: refused") + return exits.APPLY_FAILED + sock = bridge.helper(path, os.getuid()) + if sock is None: + log(f"no root helper came to {path} within {bridge.CONNECT_WAIT} s") + return exits.APPLY_FAILED + with sock: + try: + params = json.loads((line(sock) or b"").decode()) + iface, address = str(params["iface"]), str(params["address"]) + port = int(params["port"]) + except (ValueError, KeyError, TypeError, BridgeError) as e: + log(f"the root side sent no parameters the listener can use" + f" ({e or 'nothing'}): refused") + return exits.APPLY_FAILED + stop = stop or threading.Event() + log(f"the members' channel on [{address}]:{port} through {iface}," + " without capabilities") + memberlink.serve(memberlink.Front(Remote(sock, stop), clock, log), + iface, address, stop, port, poll) + return exits.OK + + +def message(conn: socket.socket, stop: threading.Event, + poll: float = memberlink.POLL) -> bytes | None: + """The listener's next message; None once it is gone or `stop` is + set, which is looked at between messages""" + conn.settimeout(poll) + while not stop.is_set(): + try: + first = conn.recv(1) + except TimeoutError: + continue + if not first: + return None + conn.settimeout(None) + rest = b"" if first == b"\n" else line(conn) + return None if rest is None else first + rest + return None + + +def overlay_of(where: Callable[[], tuple[str, str] | None], + stop: threading.Event) -> tuple[str, str] | None: + """The overlay's interface and address, waited for""" + while not stop.is_set(): + found = where() + if found is not None: + return found + stop.wait(WAIT_OVERLAY) + return None + + +def work(syncer: sync.Syncer, pending: Pending, + stop: threading.Event) -> None: + """The worker: everything pending, in one change at a time""" + while not stop.is_set(): + waiting = pending.take(WORKER_POLL) + if waiting: + sync.announced(syncer, waiting) + + +def serve(syncer: sync.Syncer, stop: threading.Event, + start: Callable[[str], object] | None = None, + where: Callable[[], tuple[str, str] | None] | None = None, + port: int = PORT, path: str | None = None, + sleep: Callable[[float], None] = time.sleep, + settle: float = SETTLE) -> int: + """`keel mesh members`: the root helper, until `stop` or the listener + is gone""" + try: + signing.ensure(syncer.root) + except SigningError as e: + syncer.err(str(e)) + return exits.APPLY_FAILED + found = overlay_of(where or (lambda: sync.where(syncer.node)), stop) + if found is None: + return exits.OK + path = path or socket_path(syncer.root) + pending = Pending(settle=settle) + worker = threading.Thread(target=work, args=(syncer, pending, stop)) + worker.start() + handler = Members(lambda: sync.offered(syncer), + lambda source: sync.member_of(syncer.node, source), + pending, syncer.err) + code = exits.OK + started = None + try: + started = (start or (lambda where_to: start_unit( + where_to, syncer.node.run, syncer.node.output)))(path) + with bridge.connected(path, started, sleep, lambda pid, uid: ( + syncer.err(f"the members' listener (pid {pid}, uid {uid})" + f" serves [{found[1]}]:{port}")), + lambda pid, uid: syncer.err( + f"skipped process {pid} (uid {uid}) at {path}: it is" + " not the members' listener")) as conn: + send(conn, {"iface": found[0], "address": found[1], + "port": port}) + while (data := message(conn, stop)) is not None: + send(conn, answered(handler, data)) + except (BridgeError, OSError) as e: + syncer.err(f"the members' channel stopped: {e}") + code = exits.APPLY_FAILED + finally: + if started is not None: + started.stop() + stop.set() + worker.join() + return code diff --git a/keel/mesh/memberlink.py b/keel/mesh/memberlink.py new file mode 100644 index 0000000..e391458 --- /dev/null +++ b/keel/mesh/memberlink.py @@ -0,0 +1,324 @@ +# Copyright (c) 2026 KeelLinux maintainers +"""The members' channel: HTTP on the overlay, WireGuard its proof + +What `keel mesh sync` pulls and what the inviter announces +(keel.mesh.members) goes between two members' overlay addresses, on +TCP 51821, and both ends bind their socket to the overlay's interface +(SO_BINDTODEVICE): the server takes only what arrived through +WireGuard, and the client's request leaves, and its answer comes back, +only through it. WireGuard accepts a packet on that interface only when +it was decrypted with the key of the peer whose allowed_ips hold its +source address. So the root side knows which member asks by the source +address, and refuses any other; the client knows the answer came from +the member it asked. What a roster says about other members is taken +only with admission evidence (keel.mesh.trust): the channel +authenticates who speaks, not what it vouches for. + +The server is the unprivileged listener's (keel.mesh.memberd), and is +bounded as the invite's listener is (keel.mesh.listener): at most 4 +connections at once and one per source, each request read under a +deadline of 15 seconds and each read a timeout of 5, the request line +and headers capped at 8 KiB and the body at 64 KiB, and a source +refused 5 times within a minute not answered until the minute has +passed. What passed goes to the root side (`Front.backend`). When the +overlay goes down and up, as each apply of it does, the interface +comes back as another one and the socket is bound again. +""" + +import http.client +import json +import socket +import threading +from collections.abc import Callable +from dataclasses import dataclass +from datetime import datetime +from http.server import BaseHTTPRequestHandler + +from keel.mesh import members +from keel.mesh.channel import plain, reason, shown +from keel.mesh.listener import ( + DRAIN, + MAX_HEADERS, + READ_TIMEOUT, + Capped, + Limiter, + shut, +) +from keel.mesh.members import ANNOUNCE, LIST, MAX_BODY, PORT, Roster +from keel.mesh.protocol import CONTENT_TYPE, ProtocolError + +# two lost SYNs on a poor link still connect (keel.mesh.channel) +CONNECT_TIMEOUT = 10 +ANSWER_TIMEOUT = 30 +# what a probe waits: it only makes WireGuard start a handshake +TOUCH_TIMEOUT = 2 +POLL = 1.0 +GET, POST = "GET", "POST" +ACCEPTED = 202 +SLOTS = 4 +PER_SOURCE = 2 +REQUEST_DEADLINE = 15 + + +class LinkError(Exception): + """The exchange with a member did not complete, and why""" + + +def connected(host: str, iface: str, port: int, + timeout: float) -> socket.socket: + """A TCP connection to `host` through `iface` alone""" + raw = socket.socket(socket.AF_INET6, socket.SOCK_STREAM) + try: + raw.setsockopt(socket.SOL_SOCKET, socket.SO_BINDTODEVICE, + iface.encode()) + raw.settimeout(timeout) + raw.connect((host, port)) + except OSError as e: + raw.close() + raise LinkError(f"{shown(host, port)} through {iface}:" + f" {e.strerror or e}") from None + return raw + + +def exchange(host: str, iface: str, method: str, path: str, + body: bytes = b"", port: int = PORT, + connect_timeout: float = CONNECT_TIMEOUT, + answer_timeout: float = ANSWER_TIMEOUT) -> tuple[int, bytes]: + """(status, body) of one request; raises LinkError""" + raw = connected(host, iface, port, connect_timeout) + raw.settimeout(answer_timeout) + connection = http.client.HTTPConnection(host, port) + connection.sock = raw + try: + connection.request(method, path, body or None, + {"Content-Type": CONTENT_TYPE, + "Connection": "close"}) + response = connection.getresponse() + data = response.read(MAX_BODY + 1) + except (OSError, http.client.HTTPException) as e: + raise LinkError(f"{shown(host, port)} did not answer: {e}") \ + from None + finally: + connection.close() + if len(data) > MAX_BODY: + raise LinkError(f"the answer from {shown(host, port)} is longer than" + " any roster") + return response.status, data + + +def fetch(host: str, iface: str, port: int = PORT) -> Roster: + """The roster of the member at `host`; raises LinkError""" + status, data = exchange(host, iface, GET, LIST, port=port) + if status != 200: + raise LinkError(f"{shown(host, port)} refused: {reason(data, status)}") + try: + return members.loads(data) + except ProtocolError as e: + raise LinkError(f"the roster from {shown(host, port)} is" + f" malformed: {e}") from None + + +def tell(host: str, iface: str, roster: Roster, port: int = PORT) -> None: + """Announce `roster` to the member at `host`; raises LinkError""" + status, data = exchange(host, iface, POST, ANNOUNCE, + members.dumps(roster), port=port) + if status != ACCEPTED: + raise LinkError(f"{shown(host, port)} refused: {reason(data, status)}") + + +def touch(host: str, iface: str, port: int = PORT) -> None: + """Send `host` a packet through `iface`, so that WireGuard starts a + handshake with its peer; whatever answers, or nothing""" + try: + connected(host, iface, port, TOUCH_TIMEOUT).close() + except LinkError: + pass + + +@dataclass(frozen=True) +class Answer: + status: int + body: bytes + + +def refused(status: int, why: str) -> Answer: + return Answer(status, json.dumps({"error": why}).encode()) + + +class Front: + """The listener's checks before the root side, per source + + `backend` is the root side (keel.mesh.memberd.Members in one + process, memberd.Remote across the bridge): handle(method, path, + body, source) -> Answer. + """ + + def __init__(self, backend, clock: Callable[[], datetime], + log: Callable[[str], None]): + self.backend = backend + self.log = log + self.lock = threading.Lock() + self.limiter = Limiter(clock) + + def blocked(self, source: str) -> bool: + with self.lock: + return self.limiter.blocked(source) + + def handle(self, method: str, path: str, body: bytes | None, + source: str) -> Answer: + if (method, path) not in ((GET, LIST), (POST, ANNOUNCE)): + found = refused(404, "no such request") + elif body is None: + found = refused(413, "longer than any roster") + else: + found = self.backend.handle(method, path, body, source) + if found.status >= 400: + with self.lock: + self.limiter.refused(source) + self.log(f"refused {method} {path[:64]!r} from {source}:" + f" {found.status}") + return found + + +@dataclass +class Connection: + """What a handler needs of the connection it serves""" + + front: Front + read: threading.Timer + + +class Handler(BaseHTTPRequestHandler): + server_version = "keel-mesh-members" + sys_version = "" + + def setup(self): + super().setup() + self.rfile = Capped(self.rfile, MAX_HEADERS) + + def answer(self): + try: + length = int(self.headers.get("Content-Length") or 0) + except ValueError: + length = -1 + self.rfile.budget = MAX_BODY + body = self.rfile.read(length) if 0 <= length <= MAX_BODY else None + # read whole: the deadline is for reading + self.server.read.cancel() + found = self.server.front.handle(self.command, self.path, body, + plain(self.client_address[0])) + self.send_response(found.status) + self.send_header("Content-Type", CONTENT_TYPE) + self.send_header("Content-Length", str(len(found.body))) + self.end_headers() + self.wfile.write(found.body) + + do_GET = do_POST = do_PUT = answer + + def log_message(self, format, *args): # noqa: A002 + """The front logs what it refuses, and no header""" + + +def serve_one(front: Front, raw: socket.socket, source: str, + release: Callable[[], None]) -> None: + read = threading.Timer(REQUEST_DEADLINE, shut, ([raw],)) + read.start() + try: + raw.settimeout(READ_TIMEOUT) + Handler(raw, (source, 0), Connection(front, read)) + except (OSError, ValueError): + pass + finally: + read.cancel() + raw.close() + release() + + +def bound(iface: str, address: str, port: int) -> socket.socket: + """The server's socket on `address`, through `iface` alone; raises + OSError while either is missing""" + server = socket.socket(socket.AF_INET6, socket.SOCK_STREAM) + try: + server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + server.setsockopt(socket.SOL_SOCKET, socket.SO_BINDTODEVICE, + iface.encode()) + server.bind((address, port)) + server.listen(16) + except OSError: + server.close() + raise + return server + + +def index(iface: str) -> int | None: + try: + return socket.if_nametoindex(iface) + except OSError: + return None + + +class Slots: + """At most SLOTS connections at once, PER_SOURCE from one source""" + + def __init__(self): + self.free = threading.BoundedSemaphore(SLOTS) + self.guard = threading.Lock() + self.active: dict[str, int] = {} + + def take(self, source: str) -> Callable[[], None] | None: + with self.guard: + if self.active.get(source, 0) >= PER_SOURCE or \ + not self.free.acquire(blocking=False): + return None + self.active[source] = self.active.get(source, 0) + 1 + + def release(): + with self.guard: + self.active[source] -= 1 + self.free.release() + return release + + +def serve(front: Front, iface: str, address: str, stop: threading.Event, + port: int = PORT, poll: float = POLL) -> None: + """Answer members on `address` through `iface` until `stop`, the + socket bound again whenever the interface comes back as another""" + server, at = None, None + slots = Slots() + threads: list[threading.Thread] = [] + while not stop.is_set(): + if server is not None and index(iface) != at: + server.close() + server = None + if server is None: + at = index(iface) + try: + if at is None: + raise OSError(f"no interface {iface}") + server = bound(iface, address, port) + server.settimeout(poll) + except OSError: + server = None + stop.wait(poll) + continue + try: + raw, found = server.accept() + except TimeoutError: + continue + except OSError: + server.close() + server = None + continue + source = plain(found[0]) + release = None if front.blocked(source) else slots.take(source) + if release is None: + raw.close() + continue + thread = threading.Thread(target=serve_one, args=( + front, raw, source, release), daemon=True) + thread.start() + threads = [one for one in threads if one.is_alive()] + [thread] + if server is not None: + server.close() + for thread in threads: + thread.join(DRAIN) diff --git a/keel/mesh/members.py b/keel/mesh/members.py new file mode 100644 index 0000000..6e673ff --- /dev/null +++ b/keel/mesh/members.py @@ -0,0 +1,139 @@ +# Copyright (c) 2026 KeelLinux maintainers +"""What the members of a mesh tell each other (decision 0048) + +"Until etcd exists, members learn about each other through the +inviter": a member's roster is its mesh identity, its own WireGuard and +signing keys and overlay address, every peer its spec declares (key, +endpoint, address) with the evidence of its admission it keeps, and the +tombstones of the keys removed (keel.mesh.trust). It goes over the +overlay alone, between two members' overlay addresses (keel.mesh.sync): + + GET /v1/members -> Roster, what `keel mesh sync` pulls + POST /v1/announce Roster, what the inviter pushes once a join is + confirmed: its roster, the new node in it + +WireGuard binds a peer's overlay address to that peer's key, so a +connection that arrives on the overlay interface from an address of a +peer's allowed_ips came from that peer, and the answer to one sent to a +peer's address can only come from it: the overlay authenticates who +speaks. What it says of other members is taken only with evidence a +trusted member signed (keel.mesh.trust.accepted). + +`with_members` is the change the members taken make to a spec: those +it does not know become peers, each routed its address as a /128, and +nothing it knows is ever replaced. Everything here is pure. +""" + +import ipaddress +import json +from dataclasses import asdict, dataclass + +from keel.manifest.firewall import MEMBERS_PORT +from keel.mesh import protocol +from keel.mesh.protocol import Peer, ProtocolError, Removal +from keel.mesh.token import MESH_ID_BYTES +from keel.network.wireguard import allowed, same_key + +# TCP, on the overlay address alone: never the invite's port, which +# faces the network on every address. keel's firewall accepts it on the +# overlay's interface (keel.manifest.firewall, which keeps the number) +PORT = MEMBERS_PORT +LIST = "/v1/members" +ANNOUNCE = "/v1/announce" +MAX_MEMBERS = 256 +# every tombstone the node keeps (keel.mesh.trust.MAX_REMOVED) +MAX_REMOVED = 1024 +# a member with its evidence is about 600 bytes, a tombstone 300 +MAX_BODY = 524288 + + +@dataclass(frozen=True) +class Roster: + """A member's identity (None before it has one), WireGuard key, + signing key, overlay address, the peers its spec declares with + their evidence, and the tombstones it keeps""" + + identity: bytes | None + public_key: str + sign_key: str + address: str + members: tuple[Peer, ...] + removed: tuple[Removal, ...] = () + + +def dumps(roster: Roster) -> bytes: + return json.dumps({ + "identity": roster.identity.hex() if roster.identity else None, + "public_key": roster.public_key, "sign_key": roster.sign_key, + "address": roster.address, + "members": [asdict(one) for one in roster.members], + "removed": [asdict(one) for one in roster.removed]}, + sort_keys=True).encode() + + +def loads(body: bytes) -> Roster: + data = protocol.loaded(body) + found = protocol.field(data, "members", list) + gone = protocol.field(data, "removed", list) + if len(found) > MAX_MEMBERS or len(gone) > MAX_REMOVED: + raise ProtocolError(f"more than {MAX_MEMBERS} members or" + f" {MAX_REMOVED} tombstones") + return Roster(identity=mesh_identity(data.get("identity")), + public_key=protocol.key(data, "public_key"), + sign_key=protocol.key(data, "sign_key"), + address=protocol.overlay(data, "address", False), + members=tuple(protocol.peer(one) for one in found), + removed=tuple(protocol.removal(one) for one in gone)) + + +def mesh_identity(value: object) -> bytes | None: + if value is None: + return None + try: + found = bytes.fromhex(value) if isinstance(value, str) else b"" + except ValueError: + found = b"" + if len(found) != MESH_ID_BYTES: + raise ProtocolError("identity is not a mesh identity") + return found + + +def with_members(doc: dict, found: tuple[Peer, ...], + own_key: str) -> tuple[dict, tuple[Peer, ...]]: + """`doc` with the members of `found` it can take as peers, as a new + document, and those members; `doc` itself when there are none + + A member is left out when its key is this node's or a peer's + already, or its address is off this node's overlay prefix, this + node's own, or inside a peer's allowed_ips: a roster never replaces + or shadows what the spec declares. + """ + network = doc.get("network") or {} + overlay = dict((network.get("overlay") or {}).get("wireguard") or {}) + if not overlay.get("address"): + return doc, () + own = ipaddress.IPv6Interface(str(overlay["address"])) + peers = list(overlay.get("peers") or []) + keys = [own_key] + [str(one.get("public_key")) for one in peers] + taken = [ipaddress.ip_network(f"{own.ip}/128")] + [ + ipaddress.ip_network(net) for one in peers for net in allowed(one)] + added = [] + for member in found: + address = ipaddress.IPv6Address(member.address) + if any(same_key(member.public_key, key) for key in keys) or \ + address not in own.network or \ + any(address in net for net in taken): + continue + entry = {"public_key": member.public_key} + if member.endpoint: + entry["endpoint"] = member.endpoint + entry["allowed_ips"] = [f"{address}/128"] + peers.append(entry) + keys.append(member.public_key) + taken.append(ipaddress.ip_network(f"{address}/128")) + added.append(member) + if not added: + return doc, () + return {**doc, "network": {**network, "overlay": { + **(network.get("overlay") or {}), + "wireguard": {**overlay, "peers": peers}}}}, tuple(added) diff --git a/keel/mesh/node.py b/keel/mesh/node.py index 777cbe2..e896cc0 100644 --- a/keel/mesh/node.py +++ b/keel/mesh/node.py @@ -17,9 +17,9 @@ """ import contextlib +import io import ipaddress import os -import sys from collections.abc import Callable from dataclasses import dataclass, field @@ -29,7 +29,7 @@ from keel.commands import apply_system, manifest_facts from keel.mesh.protocol import Peer from keel.network import confirm as netconfirm -from keel.network import live, marker, session, wireguard +from keel.network import live, marker, session, wgkeys, wireguard from keel.network.marker import write_private from keel.network.wireguard import allowed, same_key from keel.system import DEFAULT_WINDOW @@ -42,22 +42,29 @@ class NodeError(Exception): @dataclass(frozen=True) class Change: - """apply's exit code, and the overlay change waiting, if it made one""" + """apply's exit code, the overlay change waiting, if it made one, and + the lines apply printed + + The lines are held, not printed: they say the change "reverts in + 120 s unless `keel network confirm`", which is not what the operator + has to do when keel mesh confirms it itself. A flow shows them when + the change does not come up or is not confirmed (`shown`). + """ code: int made: marker.Pending | None + lines: tuple[str, ...] = () + def shown(self, say: Callable[[str], None]) -> None: + for line in self.lines: + say(line) -def live_apply(doc: dict, root: str, window: int) -> int: - """apply --system-only --skip-uplink, its lines on standard error - Standard output stays the command's own: the line to paste, or the - summary confconsole shows. - """ - with contextlib.redirect_stdout(sys.stderr): - return apply_system(doc, root, False, "keel mesh", - defer_certificate=True, network_window=window, - skip_uplink=True) +def live_apply(doc: dict, root: str, window: int) -> int: + """apply --system-only --skip-uplink""" + return apply_system(doc, root, False, "keel mesh", + defer_certificate=True, network_window=window, + skip_uplink=True) @dataclass @@ -106,12 +113,16 @@ def write(self, doc: dict) -> None: def change(self, doc: dict, window: int | None = None) -> Change: """Write `doc` and apply it; the change waiting, if any""" self.write(doc) - code = self.apply(doc, self.root, window or self.window) + said = io.StringIO() + # standard output stays the command's own: the line to paste, + # or the summary confconsole shows + with contextlib.redirect_stdout(said): + code = self.apply(doc, self.root, window or self.window) made = marker.read(self.root) if marker.exists(self.root) else None if made is not None and (made.kind != marker.OVERLAY or made.changed_at is None): made = None - return Change(code, made) + return Change(code, made, tuple(said.getvalue().splitlines())) def waiting(self) -> bool: return marker.exists(self.root) @@ -134,6 +145,15 @@ def peers(self, but: str) -> tuple[Peer, ...]: hosts[0].split("/")[0])) return tuple(found) + def public_key(self) -> tuple[str | None, str]: + """This node's public key, as `wg pubkey` gives it from its key + file; None, and why, when it cannot""" + overlay = self.overlay() + path = os.path.join(self.root, + wireguard.key_path(overlay).lstrip("/")) + public, problem = wgkeys.public(path) + return public, str(problem or "") + def handshake(self, key: str) -> int | None: """When `key` last completed a WireGuard handshake with this node, in seconds since the epoch, as `wg show IFACE latest-handshakes` @@ -163,17 +183,6 @@ def static_addresses(doc: dict): yield ipaddress.ip_interface(str(block["address"])).ip -def global_addresses(text: str) -> list[tuple[str, str]]: - """`ip -o address show scope global` output: (interface, address)""" - found = [] - for line in text.splitlines(): - fields = line.split() - if len(fields) > 3 and fields[2] in ("inet", "inet6"): - found.append((fields[1].split("@")[0], - fields[3].split("/")[0])) - return found - - def with_peer(doc: dict, peer: dict) -> dict: """`doc` with the peer added to its overlay, as a new document @@ -192,6 +201,17 @@ def with_peer(doc: dict, peer: dict) -> dict: **overlay, "wireguard": wireguard}}} +def without_peer(doc: dict, key: str) -> dict: + """`doc` without the peer of key `key`, as a new document""" + network = dict(doc.get("network") or {}) + overlay = dict(network.get("overlay") or {}) + wireguard = dict(overlay.get("wireguard") or {}) + wireguard["peers"] = [one for one in wireguard.get("peers") or [] + if not same_key(str(one.get("public_key")), key)] + return {**doc, "network": {**network, "overlay": { + **overlay, "wireguard": wireguard}}} + + def with_overlay(doc: dict, fields: dict) -> dict: """`doc` with fields of its overlay set, as a new document""" network = dict(doc.get("network") or {}) diff --git a/keel/mesh/parser.py b/keel/mesh/parser.py index 03d4b6d..4c7c474 100644 --- a/keel/mesh/parser.py +++ b/keel/mesh/parser.py @@ -100,9 +100,67 @@ def actions(mesh_actions, options: Options) -> None: " this node at ::1, confirmed by itself (root)", ) options.common(create_parser) + create_parser.add_argument( + "--adopt", action="store_true", + help="on one node of a mesh built by hand: give the mesh its" + " identity (this node's, else its peers', else a new one), which" + " the others take with keel mesh sync", + ) window(options, create_parser) options.root(create_parser, "create the mesh in") create_parser.set_defaults(handler=commands.mesh_create) + sync_parser = mesh_actions.add_parser( + "sync", + help="add the members this node's peers know and it does not, over" + " the overlay, confirmed by a WireGuard handshake (root); run at" + " boot and by keel-mesh-sync.timer", + ) + options.common(sync_parser) + sync_parser.add_argument( + "--from", dest="source", action="append", default=None, + metavar="ADDRESS", + help="ask the member at this overlay address alone; may be" + " repeated (default: every peer)", + ) + sync_parser.add_argument( + "--adopt", default=None, metavar="ADDRESS", + help="first take the mesh identity of the member at this overlay" + " address in place of this node's: the repair of a split", + ) + window(options, sync_parser) + options.root(sync_parser, "sync the mesh of") + sync_parser.set_defaults(handler=commands.mesh_sync) + members_parser = mesh_actions.add_parser( + "members", + help="the members' channel on the overlay, which keel-mesh-members" + " runs: rosters for keel mesh sync, announcements of new nodes;" + " not run by hand", + ) + options.common(members_parser) + window(options, members_parser) + options.root(members_parser, "serve the mesh of") + members_parser.set_defaults(handler=commands.mesh_members) + members_listen = mesh_actions.add_parser( + "members-listen", + help="the unprivileged listener keel mesh members starts as the" + " unit keel-mesh-members-listen; not run by hand", + ) + members_listen.add_argument("socket", metavar="SOCKET", + help="the root helper's unix socket") + members_listen.set_defaults(handler=commands.mesh_members_listen) + remove_parser = mesh_actions.add_parser( + "remove", + help="remove a peer from this node's spec, applied under the window" + " for keel network confirm, and keep its tombstone so no keel mesh" + " sync adds it again (root)", + ) + options.common(remove_parser) + remove_parser.add_argument( + "member", metavar="KEY|ADDRESS", + help="the peer's WireGuard public key or its overlay address") + window(options, remove_parser) + options.root(remove_parser, "remove the peer in") + remove_parser.set_defaults(handler=commands.mesh_remove) accept_parser = mesh_actions.add_parser( "accept", help="on the inviter, the keel mesh accept line a join printed when" diff --git a/keel/mesh/protocol.py b/keel/mesh/protocol.py index 3598b0f..748495f 100644 --- a/keel/mesh/protocol.py +++ b/keel/mesh/protocol.py @@ -34,13 +34,20 @@ ANSWER = "ANSWER" SIGNATURE = "X-Keel-Mesh-Signature" CONTENT_TYPE = "application/json" -# a join request is about 300 bytes, an answer grows with the peers +# a join request is about 350 bytes: what the listener reads at most MAX_BODY = 16384 +# an answer grows with the peers and their evidence, about 600 bytes each +MAX_ANSWER = 262144 # how far a request's time may be from the inviter's clock SKEW = timedelta(minutes=5) NONCE_BYTES = 16 LABEL = b"keel mesh 1\n" ID_RE = re.compile(r"^[0-9a-f]{16}$") +MESH_ID_RE = re.compile(r"^[0-9a-f]{32}$") +# an Ed25519 signature, 64 bytes in base64 +ED25519_RE = re.compile(r"^[A-Za-z0-9+/]{86}==$") +ADMISSION_LABEL = b"keel mesh admission 1\n" +REMOVAL_LABEL = b"keel mesh removal 1\n" NONCE_RE = re.compile(rf"^[0-9a-f]{{{NONCE_BYTES * 2}}}$") SIGNATURE_RE = re.compile(r"^[0-9a-f]{64}$") @@ -49,13 +56,59 @@ class ProtocolError(ValueError): """A message that is not one of the protocol's, and why""" +@dataclass(frozen=True) +class Admission: + """Evidence that a member admitted a node (keel.mesh.trust) + + Signed with the admitting member's signing key (`by`) over the mesh + identity, the invite id (which keel.mesh.trust requires: no member + vouches for a node it did not admit), the node's WireGuard key and + signing key, its overlay address and endpoint, and the time. + """ + + mesh_id: str + invite_id: str + public_key: str + sign_key: str + address: str + endpoint: str | None + time: int + by: str + signature: str + + def message(self) -> bytes: + return ADMISSION_LABEL + "\n".join(( + self.mesh_id, self.invite_id, self.public_key, self.sign_key, + self.address, self.endpoint or "", str(self.time), + self.by)).encode() + + +@dataclass(frozen=True) +class Removal: + """A tombstone: a member removed this node's key, signed with its + signing key (`by`), so no sync adds it again""" + + mesh_id: str + public_key: str + time: int + by: str + signature: str + + def message(self) -> bytes: + return REMOVAL_LABEL + "\n".join(( + self.mesh_id, self.public_key, str(self.time), + self.by)).encode() + + @dataclass(frozen=True) class Peer: - """A member as the answer names it: key, endpoint or None, address""" + """A member as the answer names it: key, endpoint or None, address, + and the evidence of its admission, None when there is none""" public_key: str endpoint: str | None address: str + admission: Admission | None = None @dataclass(frozen=True) @@ -69,6 +122,8 @@ class JoinRequest: address: str nonce: str time: int + # the new node's signing key, which its admission names + sign_key: str @dataclass(frozen=True) @@ -84,6 +139,9 @@ class JoinAnswer: peers: tuple[Peer, ...] etcd: str window: int + # the inviter's signing key, and the new node's admission it signed + sign_key: str + admission: Admission @dataclass(frozen=True) @@ -102,6 +160,8 @@ class ConfirmAnswer: nonce: str confirmed: bool detail: str + # the inviter's signing key: what the fallback's node learns it by + sign_key: str def sign(key: bytes, method: str, path: str, body: bytes) -> str: @@ -209,7 +269,7 @@ def join_request(body: bytes) -> JoinRequest: endpoint=endpoint(data.get("endpoint")), address=overlay(data, "address", True), nonce=matching(data, "nonce", NONCE_RE), - time=number(data, "time")) + time=number(data, "time"), sign_key=key(data, "sign_key")) def join_answer(body: bytes) -> JoinAnswer: @@ -225,15 +285,44 @@ def join_answer(body: bytes) -> JoinAnswer: public_key=key(data, "public_key"), address=overlay(data, "address", True), peers=tuple(peer(one) for one in peers), - etcd=etcd, window=number(data, "window")) + etcd=etcd, window=number(data, "window"), + sign_key=key(data, "sign_key"), + admission=admission(data.get("admission"))) def peer(data: object) -> Peer: if not isinstance(data, dict): raise ProtocolError("a peer is not a JSON object") + found = data.get("admission") return Peer(public_key=key(data, "public_key"), endpoint=endpoint(data.get("endpoint")), - address=overlay(data, "address", False)) + address=overlay(data, "address", False), + admission=None if found is None else admission(found)) + + +def admission(data: object) -> Admission: + if not isinstance(data, dict): + raise ProtocolError("an admission is not a JSON object") + invite = field(data, "invite_id", str) + if invite and not ID_RE.match(invite): + raise ProtocolError("invite_id is malformed") + return Admission( + mesh_id=matching(data, "mesh_id", MESH_ID_RE), invite_id=invite, + public_key=key(data, "public_key"), sign_key=key(data, "sign_key"), + address=overlay(data, "address", False), + endpoint=endpoint(data.get("endpoint")), time=number(data, "time"), + by=key(data, "by"), signature=matching(data, "signature", + ED25519_RE)) + + +def removal(data: object) -> Removal: + if not isinstance(data, dict): + raise ProtocolError("a removal is not a JSON object") + return Removal( + mesh_id=matching(data, "mesh_id", MESH_ID_RE), + public_key=key(data, "public_key"), time=number(data, "time"), + by=key(data, "by"), signature=matching(data, "signature", + ED25519_RE)) def confirm_request(body: bytes) -> ConfirmRequest: @@ -251,4 +340,4 @@ def confirm_answer(body: bytes) -> ConfirmAnswer: invite_id=matching(data, "invite_id", ID_RE), nonce=matching(data, "nonce", NONCE_RE), confirmed=field(data, "confirmed", bool), - detail=field(data, "detail", str)) + detail=field(data, "detail", str), sign_key=key(data, "sign_key")) diff --git a/keel/mesh/remove.py b/keel/mesh/remove.py new file mode 100644 index 0000000..09e175a --- /dev/null +++ b/keel/mesh/remove.py @@ -0,0 +1,81 @@ +# Copyright (c) 2026 KeelLinux maintainers +"""keel mesh remove: a node out of the mesh, before etcd (0048) + +"Before etcd, the command removes the peer on the member it ran on and +sends the same announcement as a join, so the others drop it too." On +the member it runs on: a tombstone, signed with this node's key +(keel.mesh.trust), and the peer out of this node's spec (0013), applied +under 0018's window and confirmed as keel mesh sync confirms a change +(keel.mesh.sync): by a handshake from a peer it keeps, or, with no peer +left, by the route check alone. Then this node's roster, the tombstone +in it, goes to each of its other peers. Each takes the tombstone only +when the key that signed it may remove that node: the member that +admitted it, a trust root, or the node itself (trust.may_remove); it +then drops the peer through one window of its own, and no sync adds the +key again. A member offline then learns it at its next sync. etcd's +`member remove` is not here yet. +""" + +import ipaddress +from collections.abc import Callable + +from keel import exits +from keel.mesh import identity, signing, sync, trust +from keel.mesh.node import NodeError, without_peer +from keel.mesh.signing import SigningError +from keel.mesh.sync import Syncer +from keel.network.wireguard import same_key + + +def named(syncer: Syncer, which: str) -> tuple[str, str] | None: + """(key, overlay address) of the peer `which` names: its key, or an + address its allowed_ips hold""" + try: + address = ipaddress.ip_address(which) + except ValueError: + address = None + for one in syncer.node.peers(""): + if same_key(one.public_key, which) or ( + address is not None + and ipaddress.ip_address(one.address) == address): + return one.public_key, one.address + return None + + +def remove(syncer: Syncer, which: str, out: Callable[[str], None]) -> int: + try: + found = named(syncer, which) + own = identity.read(syncer.root) + if found is None: + raise ValueError(f"{which}: no peer of this node has that key" + " or overlay address") + if own is None: + raise ValueError("this node holds no mesh identity to sign the" + " removal in") + if syncer.node.waiting(): + raise ValueError("a network change waits for its confirmation" + " on this node: confirm or revert it, then" + " remove") + signing.ensure(syncer.root) + store = trust.load(syncer.root) + if not trust.room_for(store, signing.public(syncer.root)): + raise ValueError(f"this node keeps {len(store.removed)}" + " tombstones, as many as it may") + doc = syncer.node.document() + trust.record_removal(store, trust.removal(syncer.root, own, + found[0], syncer.clock())) + trust.save(syncer.root, store) + except (ValueError, NodeError, SigningError) as e: + syncer.err(str(e)) + return exits.MESH_REFUSED + code = sync.changed_and_confirmed(syncer, doc, + without_peer(doc, found[0]), (), + (found[0],)) + if code != exits.OK: + syncer.err("the tombstone is kept: keel mesh sync drops the peer" + " again once the change can be confirmed") + return code + sync.announce(syncer, found[0], "the removal") + out(f"removed {found[0]} at {found[1]}: the other members drop it on" + " its tombstone, and no keel mesh sync adds it again") + return exits.OK diff --git a/keel/mesh/signing.py b/keel/mesh/signing.py new file mode 100644 index 0000000..6de8dc3 --- /dev/null +++ b/keel/mesh/signing.py @@ -0,0 +1,112 @@ +# Copyright (c) 2026 KeelLinux maintainers +"""This node's signing key: Ed25519, long-term, made on the machine + +What a member signs is admission evidence: that it admitted a node, by +which invite, with which keys, at which address (keel.mesh.trust). The +key pair is made by `openssl genpkey` (openssl is a dependency of keel) +on the machine itself, never in an image, and kept in +/var/lib/keel/mesh/node.key, 0600; it is never printed. Its public half, +32 bytes in base64 as WireGuard writes its keys, travels in the join's +authenticated answer and in each roster. + +openssl signs and verifies Ed25519 only from files (`pkeyutl -rawin`), +so the message, the signature and a public key are written to a +private temporary directory for the call. +""" + +import base64 +import os +import subprocess +import tempfile + +from keel.mesh import DIR +from keel.mesh.invites import locked +from keel.network.marker import path, write_private +from keel.network.wireguard import key_bytes + +KEY = f"{DIR}/node.key" +OPENSSL = "openssl" +TIMEOUT = 30 +# the DER of an Ed25519 SubjectPublicKeyInfo, before the 32 key bytes +SPKI_PREFIX = bytes.fromhex("302a300506032b6570032100") +SIGNATURE_BYTES = 64 + + +class SigningError(Exception): + """openssl could not make, read or use this node's signing key""" + + +def openssl(*argv: str) -> subprocess.CompletedProcess: + try: + return subprocess.run([OPENSSL, *argv], capture_output=True, + check=False, timeout=TIMEOUT) + except (OSError, subprocess.TimeoutExpired) as e: + raise SigningError(f"openssl could not be run: {e}") from None + + +def ensure(root: str) -> str: + """This node's public signing key, the pair made under the mesh's + lock when there is none; raises SigningError""" + with locked(root): + if not os.path.exists(path(root, KEY)): + made = openssl("genpkey", "-algorithm", "ed25519") + if made.returncode != 0 or b"PRIVATE KEY" not in made.stdout: + raise SigningError(f"openssl genpkey exited" + f" {made.returncode}: " + f"{made.stderr.decode()[-200:]}") + write_private(root, KEY, made.stdout.decode()) + return public(root) + + +def public(root: str) -> str: + """This node's public signing key; raises SigningError without one""" + if not os.path.exists(path(root, KEY)): + raise SigningError(f"no signing key at /{KEY}: this node holds" + " none yet") + found = openssl("pkey", "-in", path(root, KEY), "-pubout", + "-outform", "DER") + if found.returncode != 0 or not found.stdout.startswith(SPKI_PREFIX): + raise SigningError(f"/{KEY} does not hold an Ed25519 key") + return base64.b64encode(found.stdout[len(SPKI_PREFIX):]).decode() + + +def sign(root: str, message: bytes) -> str: + """The signature of `message` with this node's key, in base64""" + with tempfile.TemporaryDirectory(prefix="keel-sign-") as scratch: + inside = os.path.join(scratch, "message") + out = os.path.join(scratch, "signature") + with open(inside, "wb") as fob: + fob.write(message) + done = openssl("pkeyutl", "-sign", "-inkey", path(root, KEY), + "-rawin", "-in", inside, "-out", out) + if done.returncode != 0: + raise SigningError(f"openssl could not sign with /{KEY}:" + f" {done.stderr.decode()[-200:]}") + with open(out, "rb") as fob: + return base64.b64encode(fob.read()).decode() + + +def verified(key: str, message: bytes, signature: str) -> bool: + """Whether `signature` is `key`'s over `message`; never raises for a + malformed key or signature, which are no signature""" + raw = key_bytes(key) + try: + sig = base64.b64decode(signature, validate=True) + except (ValueError, TypeError): + return False + if raw is None or len(sig) != SIGNATURE_BYTES: + return False + with tempfile.TemporaryDirectory(prefix="keel-verify-") as scratch: + files = {} + for name, data in (("key", SPKI_PREFIX + raw), ("message", message), + ("signature", sig)): + files[name] = os.path.join(scratch, name) + with open(files[name], "wb") as fob: + fob.write(data) + try: + done = openssl("pkeyutl", "-verify", "-pubin", "-keyform", + "DER", "-inkey", files["key"], "-rawin", "-in", + files["message"], "-sigfile", files["signature"]) + except SigningError: + return False + return done.returncode == 0 diff --git a/keel/mesh/status.py b/keel/mesh/status.py index bc4848c..35541be 100644 --- a/keel/mesh/status.py +++ b/keel/mesh/status.py @@ -1,18 +1,18 @@ # Copyright (c) 2026 KeelLinux maintainers """keel mesh status: the peers, their handshakes, the pending invites -Read from the spec, from `wg show` (its public-key, endpoints and -latest-handshakes views, never `dump` or `private-key`, which hold the -private key) and from the invite files, of which only the id, the -reserved address, the port and the expiry are shown. Nothing secret is -printed. +Read from the spec, from the mesh's identity, from `wg show` (its +public-key, endpoints and latest-handshakes views, never `dump` or +`private-key`, which hold the private key) and from the invite files, +of which only the id, the reserved address, the port and the expiry are +shown. Nothing secret is printed. """ import ipaddress from collections.abc import Callable from datetime import datetime -from keel.mesh import invites +from keel.mesh import identity, invites from keel.mesh.token import shown from keel.network import wireguard from keel.network.wireguard import allowed, same_key @@ -39,6 +39,20 @@ def handshake(value: str | None, now: datetime) -> str: return f"handshake {max(ago, 0)} s ago" +def mesh_identity(root: str) -> str: + """Which mesh this node is in: its identity, as every member holds + it (keel.mesh.identity)""" + try: + found = identity.read(root) + except ValueError as e: + return f"mesh identity: {e}" + if found is None: + return ("mesh identity: none yet (a mesh built by hand gets one with" + " keel mesh create --adopt on one node, keel mesh sync on" + " the others)") + return f"mesh identity: {found.hex()}" + + def lines(overlay: dict | None, root: str, now: datetime, output: Reader | None) -> list[str]: """What status prints; `output` None off the live system""" @@ -47,7 +61,7 @@ def lines(overlay: dict | None, root: str, now: datetime, " mesh join joins one"] iface = wireguard.interface(overlay) found = [f"this node: {overlay['address']} on {iface}, WireGuard on" - f" UDP {wireguard.port(overlay)}"] + f" UDP {wireguard.port(overlay)}", mesh_identity(root)] if output is None: found.append("not the live system: no handshake read") endpoints = handshakes = {} diff --git a/keel/mesh/sync.py b/keel/mesh/sync.py new file mode 100644 index 0000000..278fc15 --- /dev/null +++ b/keel/mesh/sync.py @@ -0,0 +1,504 @@ +# Copyright (c) 2026 KeelLinux maintainers +"""A full mesh before etcd: keel mesh sync and the announcement (0048) + +"Until etcd exists, members learn about each other through the +inviter." Three things make every member a peer of every other one: + +- the join's answer carries the members the inviter knows, with their + admission evidence, and the new node takes those it can verify as + peers in the same change (keel.mesh.joining); +- the inviter, once the join is confirmed, announces the new node to + each of its other peers over the overlay (`announce`); +- a member that was offline then, or one that joined by the fallback, + pulls the roster of its peers with `keel mesh sync` (`pull`), at boot + and from a timer (keel-mesh-sync.timer). + +What a member is told goes over the members' channel +(keel.mesh.memberlink), which WireGuard authenticates: only a peer's key +can use its overlay address. That says who speaks, not whom it may +vouch for: a roster's entry is taken only with admission evidence that +chains to a key this node trusts (keel.mesh.trust), and never for a key +with a tombstone, so one member cannot make the others take a peer no +member admitted. What it takes is keel.mesh.members.with_members: +members it does not know, never one that would replace or shadow a peer +of its spec. It writes them into its own spec (0013) and applies the +change under 0018's window, all of them in one change, and the change +is confirmed by a WireGuard handshake from a key it added since the +change was made, which only that member, reaching this node over the +new configuration, can complete; this node sends each one a packet to +start it. The route check of apply.md runs first, as for any overlay +change. A change no new member answers within the window is left to +revert, the spec is put back as it was, and those members are not +tried again for an hour (`UNREACHED`), so an offline node does not +cost the mesh an overlay change at every timer. + +The mesh's identity travels with each roster, and a roster of another +mesh gives nothing. A node with no identity takes none from a roster: +only a join or an operator's `--adopt` sets it (keel.mesh.adopt). +""" + +import fcntl +import ipaddress +import json +import os +import time +from collections.abc import Callable, Iterator +from concurrent.futures import ThreadPoolExecutor +from contextlib import contextmanager +from dataclasses import dataclass +from datetime import datetime, timedelta, timezone + +from keel import exits +from keel.mesh import ( + DIR, + FILE_MODE, + identity, + invites, + memberlink, + members, + signing, + trust, +) +from keel.mesh.memberlink import LinkError +from keel.mesh.members import Roster +from keel.mesh.node import Node, NodeError, without_peer +from keel.mesh.protocol import Peer +from keel.mesh.signing import SigningError +from keel.network import session, wireguard +from keel.network.marker import path, write_private +from keel.network.wireguard import allowed, same_key + +# how often a handshake is looked for, and each new member sent a packet +RETRY = 2.0 +# how long a member that never answered is left out +BACKOFF = timedelta(hours=1) +UNREACHED = f"{DIR}/unreached.json" +SYNC_LOCK = f"{DIR}/sync.lock" +PARALLEL = 8 + + +@dataclass +class Syncer: + """This node as a member; the members' channel as fields""" + + node: Node + clock: Callable[[], datetime] + err: Callable[[str], None] + fetch: Callable[[str, str], Roster] = memberlink.fetch + tell: Callable[[str, str, Roster], None] = memberlink.tell + touch: Callable[[str, str], None] = memberlink.touch + sleep: Callable[[float], None] = time.sleep + + @property + def root(self) -> str: + return self.node.root + + def iface(self) -> str: + return wireguard.interface(self.node.overlay()) + + +@contextmanager +def sync_lock(root: str, wait: bool) -> Iterator[bool]: + """Whether this process holds the lock of the members' changes: one + sync at a time. Not the mesh's own lock, which an invite takes.""" + invites.ensure(root) + fd = os.open(path(root, SYNC_LOCK), os.O_RDWR | os.O_CREAT, FILE_MODE) + try: + try: + fcntl.flock(fd, fcntl.LOCK_EX | (0 if wait else fcntl.LOCK_NB)) + except BlockingIOError: + yield False + return + yield True + finally: + os.close(fd) + + +def short(found: bytes) -> str: + return f"{found.hex()[:8]}…" + + +def roster(syncer: Syncer) -> Roster: + """This node's roster: its peers with the evidence it keeps, and its + tombstones; raises NodeError when a key cannot be read, ValueError + for a damaged store""" + public, problem = syncer.node.public_key() + if public is None: + raise NodeError(problem) + try: + signer = signing.public(syncer.root) + except SigningError as e: + raise NodeError(str(e)) from None + store = trust.load(syncer.root) + overlay = syncer.node.overlay() + return Roster( + identity.read(syncer.root), public, signer, + str(ipaddress.IPv6Interface(str(overlay["address"])).ip), + tuple(Peer(one.public_key, one.endpoint, one.address, + store.evidence(one.public_key)) + for one in syncer.node.peers(public)), + tuple(store.removed.values())) + + +def offered(syncer: Syncer) -> Roster: + """What the members' service answers; ValueError when it cannot""" + try: + return roster(syncer) + except NodeError as e: + raise ValueError(str(e)) from None + + +def member_of(node: Node, source: str) -> str | None: + """The key of the peer whose allowed_ips hold `source`""" + try: + address = ipaddress.ip_address(source) + peers = node.overlay().get("peers") or [] + except (ValueError, NodeError): + return None + for peer in peers: + if any(address in ipaddress.ip_network(net) for net in allowed(peer)): + return str(peer.get("public_key")) + return None + + +def where(node: Node) -> tuple[str, str] | None: + """The overlay's interface and this node's address on it""" + try: + overlay = node.overlay() + except NodeError: + return None + if not overlay.get("address"): + return None + return (wireguard.interface(overlay), + str(ipaddress.IPv6Interface(str(overlay["address"])).ip)) + + +def asked(syncer: Syncer, peers: tuple[Peer, ...]) -> list[Roster]: + """The rosters of `peers`, asked at once; those that do not answer, + or answer as another member, are said and left out""" + iface = syncer.iface() + + def one(peer: Peer) -> Roster | str: + try: + found = syncer.fetch(peer.address, iface) + except LinkError as e: + return f"member {peer.address} did not answer: {e}" + if not same_key(found.public_key, peer.public_key) or \ + found.address != peer.address: + return (f"member {peer.address} answered as another member" + f" ({found.public_key} at {found.address}); its roster" + " is left out") + return found + if not peers: + return [] + with ThreadPoolExecutor(min(PARALLEL, len(peers))) as pool: + answers = list(pool.map(one, peers)) + found = [] + for answer in answers: + if isinstance(answer, str): + syncer.err(answer) + else: + found.append(answer) + return found + + +def pull(syncer: Syncer, hosts: tuple[str, ...] = ()) -> int: + """`keel mesh sync`: the members this node's peers know, added""" + try: + overlay = syncer.node.overlay() + except NodeError as e: + syncer.err(str(e)) + return exits.MESH_REFUSED + if not overlay.get("address"): + syncer.err("this node is in no mesh: nothing to sync") + return exits.OK + public, problem = syncer.node.public_key() + if public is None: + syncer.err(problem) + return exits.MESH_REFUSED + known = syncer.node.peers(public) + if not known: + syncer.err("this node has no peer yet: nothing to sync") + return exits.OK + chosen = known + if hosts: + chosen = tuple(peer for peer in known if peer.address in hosts) + strangers = set(hosts) - {peer.address for peer in chosen} + if strangers: + syncer.err(f"{', '.join(sorted(strangers))}: not a peer of this" + " node; keel mesh sync asks its peers only") + return exits.MESH_REFUSED + with sync_lock(syncer.root, False) as held: + if not held: + syncer.err("another keel mesh sync is adding members; this one" + " leaves it to finish") + return exits.OK + rosters = asked(syncer, chosen) + if not rosters: + syncer.err("no member answered over the overlay; nothing was" + " changed, and the next keel mesh sync asks again") + return exits.OK + return absorb(syncer, rosters, public, fetched=True) + + +def mismatch(own: bytes, other: Roster) -> str: + return (f"member {other.address} is in mesh" + f" {short(other.identity) if other.identity else 'none'}," + f" this node in {short(own)}: nothing is taken from it. If this" + f" node is the odd one, keel mesh sync --adopt {other.address}" + " takes that member's identity in place of its own") + + +@dataclass +class Taken: + """What the rosters gave: the members whose evidence verified, the + keys a tombstone removed, and the exit code so far""" + + members: tuple[Peer, ...] + removed: tuple[str, ...] + code: int + + +def verified(syncer: Syncer, rosters: list[Roster], own: bytes, + store: trust.Store, fetched: bool) -> Taken: + """The members of `rosters` whose evidence chains to a key this node + trusts, and the tombstones it takes (keel.mesh.trust). A trust root's + signing key is bound only from a roster `fetched` by this node from + the root's own address (`pull`), never from an announcement""" + signer = signing.public(syncer.root) + code, entries, gone = exits.OK, [], [] + for one in rosters: + if one.identity != own: + syncer.err(mismatch(own, one)) + code = exits.MESH_REFUSED + continue + if fetched and trust.bind_root(store, one.public_key, one.sign_key): + syncer.err(f"trust root {one.address} signs with {one.sign_key}") + entries += one.members + gone += one.removed + found = trust.accepted(store, signer, own, entries) + removed = trust.removals(store, signer, own, gone) + return Taken(tuple(one for one in found + if not store.gone(one.public_key)), removed, code) + + +def absorb(syncer: Syncer, rosters: list[Roster], public: str, + fetched: bool = False) -> int: + """The members `rosters` name, with evidence, that this node does not + know, and the peers their tombstones remove: written, applied in one + change, and confirmed by a handshake""" + try: + own = identity.read(syncer.root) + if own is None: + raise ValueError("this node holds no mesh identity: only a" + " join, or keel mesh create --adopt or keel" + " mesh sync --adopt, sets it; nothing is" + " taken") + store = trust.load(syncer.root) + taken = verified(syncer, rosters, own, store, fetched) + trust.save(syncer.root, store) + doc = syncer.node.document() + except (ValueError, NodeError, SigningError) as e: + syncer.err(str(e)) + return exits.MESH_REFUSED + now = syncer.clock() + resting = unreached(syncer.root, now) + candidates = [] + for member in taken.members: + if member.public_key in resting: + syncer.err(f"{member.public_key} is tried again after" + f" {resting[member.public_key]:%H:%M} UTC: it" + " did not answer the last time") + continue + candidates.append(member) + after, added = members.with_members(doc, tuple(candidates), public) + dropped = tuple(one.public_key for one in syncer.node.peers(public) + if store.gone(one.public_key)) + for key in dropped: + after = without_peer(after, key) + if not added and not dropped: + syncer.err("no member this node does not know: it is a peer of" + " every admitted member its peers know") + return taken.code + if syncer.node.waiting(): + syncer.err("a network change waits for its confirmation on this" + " node: the members are added, or removed, by the next" + " keel mesh sync, once it is confirmed or reverted") + return exits.MESH_REFUSED + found = changed_and_confirmed(syncer, doc, after, added, dropped) + return found if found != exits.OK else taken.code + + +def announced(syncer: Syncer, waiting: list[tuple[str, Roster]]) -> int: + """What the members' service applies: every announcement pending, + in one change, under one window""" + for source, found in waiting: + syncer.err(f"an announcement from {source}: {len(found.members)}" + " member(s)") + public, problem = syncer.node.public_key() + if public is None: + syncer.err(problem) + return exits.MESH_REFUSED + with sync_lock(syncer.root, True): + return absorb(syncer, [found for _, found in waiting], public, + fetched=False) + + +def put_back(syncer: Syncer, doc: dict) -> None: + """The spec as it was before the members were added: the revert puts + the overlay back, and the spec says what the machine runs, so the + next sync finds them new again""" + try: + syncer.node.write(doc) + except NodeError as e: + syncer.err(str(e)) + + +def changed_and_confirmed(syncer: Syncer, doc: dict, after: dict, + added: tuple[Peer, ...], + dropped: tuple[str, ...]) -> int: + """`after` written and applied under the window, and confirmed by a + WireGuard handshake since the change: from a member it added, else, + when it only removes, from any peer it keeps; with no peer left, by + the route check alone, as `keel mesh create` confirms a mesh with no + peer. Not confirmed, the spec is put back as it was""" + since = int(syncer.clock().timestamp()) + try: + change = syncer.node.change(after) + except NodeError as e: + syncer.err(str(e)) + return exits.MESH_REFUSED + if change.made is None: + change.shown(syncer.err) + put_back(syncer, doc) + syncer.err(f"this node's overlay did not come up (apply exited" + f" {change.code}); the spec is put back as it was") + return exits.APPLY_FAILED + for member in added: + syncer.err(f"member {member.public_key} added at {member.address}," + f" through {member.endpoint or 'no endpoint (it reaches' + ' this node)'}") + for key in dropped: + syncer.err(f"member {key} removed: a tombstone signed by a key that" + " may remove it names it") + proof = added or syncer.node.peers("") + syncer.err("confirming over the overlay…") + if proof: + seen = handshake(syncer, proof, since, since + change.made.window) + if seen is None: + return unconfirmed(syncer, change, doc, added) + member, at = seen + origin = session.Origin( + session.MESH, f"a WireGuard handshake from {member.public_key}" + f" ({at:%H:%M:%S} UTC), a member of the changed overlay", None, + where(syncer.node)[1], member.address) + else: + origin = session.Origin(session.SELF, "keel mesh, which removed" + " the last peer,") + kept, lines = syncer.node.confirm(origin, change.made) + if not kept: + change.shown(syncer.err) + put_back(syncer, doc) + for line in lines: + syncer.err(line) + if not kept: + return exits.NETWORK_NOT_CONFIRMED + forget(syncer.root, added) + return exits.OK + + +def unconfirmed(syncer: Syncer, change, doc: dict, + added: tuple[Peer, ...]) -> int: + change.shown(syncer.err) + put_back(syncer, doc) + remember(syncer.root, added, syncer.clock()) + syncer.err("no member completed a WireGuard handshake within the" + " window: the change reverts by itself, the spec is put" + " back as it was, and keel mesh sync tries again (a new" + " member after an hour)") + return exits.NETWORK_NOT_CONFIRMED + + +def handshake(syncer: Syncer, added: tuple[Peer, ...], since: int, + deadline: int) -> tuple[Peer, datetime] | None: + """The first member of `added` to complete a WireGuard handshake + since the change, each sent a packet every RETRY seconds""" + iface = syncer.iface() + while True: + for member in added: + at = syncer.node.handshake(member.public_key) + if at is not None and at >= since: + return member, datetime.fromtimestamp(at, timezone.utc) + if syncer.clock().timestamp() + RETRY >= deadline: + return None + for member in added: + syncer.touch(member.address, iface) + syncer.sleep(RETRY) + + +def unreached(root: str, now: datetime) -> dict[str, datetime]: + """The members that did not answer within the last BACKOFF, and when + each may be tried again""" + try: + with open(path(root, UNREACHED)) as fob: + data = json.load(fob) + except (OSError, ValueError): + return {} + found = {} + for key, at in (data.items() if isinstance(data, dict) else ()): + if isinstance(at, int): + again = datetime.fromtimestamp(at, timezone.utc) + BACKOFF + if again > now: + found[key] = again + return found + + +def write_unreached(root: str, data: dict[str, int]) -> None: + invites.ensure(root) + write_private(root, UNREACHED, json.dumps(data, sort_keys=True) + "\n") + + +def remember(root: str, added: tuple[Peer, ...], now: datetime) -> None: + data = {key: int((again - BACKOFF).timestamp()) + for key, again in unreached(root, now).items()} + data.update({one.public_key: int(now.timestamp()) for one in added}) + write_unreached(root, data) + + +def forget(root: str, added: tuple[Peer, ...]) -> None: + now = datetime.now(timezone.utc) + data = {key: int((again - BACKOFF).timestamp()) + for key, again in unreached(root, now).items() + if key not in {one.public_key for one in added}} + write_unreached(root, data) + + +def announce(syncer: Syncer, key: str, what: str = "the new node") -> None: + """This node's roster, to each of its peers but `key`: once a join is + confirmed, it names the new node with its evidence; once a node is + removed, it carries the tombstone""" + try: + mine = roster(syncer) + except (NodeError, ValueError) as e: + syncer.err(f"{what} is not announced: {e}; the other members" + " learn of it at their next keel mesh sync") + return + iface = syncer.iface() + others = [one for one in mine.members + if not same_key(one.public_key, key)] + + def one(peer: Peer) -> str | None: + try: + syncer.tell(peer.address, iface, mine) + except LinkError as e: + return f"{peer.address} did not take it ({e})" + return None + if not others: + return + with ThreadPoolExecutor(min(PARALLEL, len(others))) as pool: + missed = [found for found in pool.map(one, others) if found] + syncer.err(f"announced {key} to {len(others) - len(missed)} of the" + f" {len(others)} other member(s) over the overlay") + if missed: + syncer.err(f"{'; '.join(missed)}: it learns of {what} at its next" + " keel mesh sync") diff --git a/keel/mesh/trust.py b/keel/mesh/trust.py new file mode 100644 index 0000000..4f06758 --- /dev/null +++ b/keel/mesh/trust.py @@ -0,0 +1,264 @@ +# Copyright (c) 2026 KeelLinux maintainers +"""Who may vouch for a member: admission evidence and the trust store + +0048 lets only an admitted join add a peer. A node that a member +admits gets a long-term signing key (keel.mesh.signing), and the member +signs evidence of the admission (protocol.Admission): the mesh's +identity, the invite, the node's WireGuard and signing keys, its +overlay address and endpoint, and the time. The new node learns the +inviter's signing key and its own admission in the join's answer, +which the invite's HMAC authenticates. + +A roster entry is taken (keel.mesh.sync) only when its admission was +signed by a key this node already trusts: its own, the inviter it +joined through, a member admitted the same way, or a trust root. The +signer of an entry it takes is trusted in turn, so evidence chains from +member to member; an entry without valid evidence is left out. A mesh +built by hand before `keel mesh` has no evidence: `keel mesh create +--adopt` and `keel mesh sync --adopt`, explicit acts of the operator, +make the peers the spec lists trust roots. A root's signing key is +learned from the root itself, in a roster this node fetched from the +root's own overlay address, which WireGuard authenticates; never from +an announcement. Evidence always names an invite: no member, root or +not, vouches for a node it did not admit. + +A removed key leaves a tombstone (protocol.Removal), signed by the +remover's key. A node takes a tombstone only from a key that may remove +that node: the admitting member's, a trust root's, or the node's own; +it then drops the key from its spec (keel.mesh.sync) and never takes it +again. Tombstones are kept for good, at most MAX_REMOVED of them and +MAX_REMOVED_PER_SIGNER from one signer, and every roster carries them +all. + +State, under /var/lib/keel/mesh/trust.json, 0600, as the invites are. +""" + +import json +from collections.abc import Iterable +from dataclasses import asdict, dataclass, field +from datetime import datetime + +from keel.mesh import DIR, invites, protocol, signing +from keel.mesh.protocol import Admission, Peer, ProtocolError, Removal +from keel.network.marker import path, write_private +from keel.network.wireguard import same_key + +TRUST = f"{DIR}/trust.json" +# tombstones are kept for good (a removed key never comes back), and +# every roster carries them all: the cap bounds both, per signer too +MAX_REMOVED = 1024 +MAX_REMOVED_PER_SIGNER = 64 + + +@dataclass +class Member: + """A member this node trusts: its signing key once known, whether + the operator made it a root, and the evidence it was admitted by""" + + sign_key: str | None + root: bool = False + admission: Admission | None = None + + +@dataclass +class Store: + members: dict[str, Member] = field(default_factory=dict) + removed: dict[str, Removal] = field(default_factory=dict) + + def find(self, key: str) -> str | None: + """The store's spelling of `key`, None when it has none""" + return next((one for one in self.members if same_key(one, key)), + None) + + def gone(self, key: str) -> bool: + return any(same_key(one, key) for one in self.removed) + + def signers(self, own: str) -> set[str]: + return {own} | {one.sign_key for one in self.members.values() + if one.sign_key} + + def evidence(self, key: str) -> Admission | None: + found = self.find(key) + return None if found is None else self.members[found].admission + + +def load(root: str) -> Store: + """The store; empty when there is none, ValueError when damaged""" + try: + with open(path(root, TRUST)) as fob: + data = json.load(fob) + members = { + str(key): Member( + one["sign_key"], bool(one["root"]), + None if one["admission"] is None + else protocol.admission(one["admission"])) + for key, one in data["members"].items()} + removed = {str(key): protocol.removal(one) + for key, one in data["removed"].items()} + except FileNotFoundError: + return Store() + except (ValueError, KeyError, TypeError, AttributeError, + ProtocolError): + raise ValueError(f"/{TRUST} is damaged: keel does not guess whom" + " this node trusts; keel mesh sync --adopt makes" + " its peers trust roots again") from None + return Store(members, removed) + + +def save(root: str, store: Store) -> None: + invites.ensure(root) + data = {"members": {key: {"sign_key": one.sign_key, "root": one.root, + "admission": None if one.admission is None + else asdict(one.admission)} + for key, one in store.members.items()}, + "removed": {key: asdict(one) + for key, one in store.removed.items()}} + write_private(root, TRUST, json.dumps(data, sort_keys=True) + "\n") + + +def admit(root: str, mesh_id: bytes, invite_id: str, public_key: str, + sign_key: str, address: str, endpoint: str | None, + now: datetime) -> Admission: + """Evidence, signed with this node's key, that it admitted the node; + raises signing.SigningError""" + draft = Admission(mesh_id.hex(), invite_id, public_key, sign_key, + address, endpoint, int(now.timestamp()), + signing.public(root), "") + return Admission(**{**asdict(draft), + "signature": signing.sign(root, draft.message())}) + + +def removal(root: str, mesh_id: bytes, public_key: str, + now: datetime) -> Removal: + """A tombstone for `public_key`, signed with this node's key""" + draft = Removal(mesh_id.hex(), public_key, int(now.timestamp()), + signing.public(root), "") + return Removal(**{**asdict(draft), + "signature": signing.sign(root, draft.message())}) + + +def matches(entry: Peer, mesh_id: bytes) -> bool: + found = entry.admission + return (same_key(found.public_key, entry.public_key) + and found.address == entry.address + and found.mesh_id == mesh_id.hex()) + + +def accepted(store: Store, own: str, mesh_id: bytes, + entries: Iterable[Peer]) -> tuple[Peer, ...]: + """The entries whose evidence chains to a key this node trusts, as + their admissions name them; each is recorded in `store`, and its + signing key trusted in turn. An entry without evidence, with + evidence of no invite, for another key, address or mesh, by a key + not trusted, or whose key has a tombstone, is left out.""" + waiting = [one for one in entries + if one.admission is not None and one.admission.invite_id + and matches(one, mesh_id) and not store.gone(one.public_key)] + trusted = store.signers(own) + taken: list[Peer] = [] + progress = True + while progress: + progress = False + for entry in list(waiting): + found = entry.admission + if found.by not in trusted: + continue + waiting.remove(entry) + if not signing.verified(found.by, found.message(), + found.signature): + continue + taken.append(Peer(found.public_key, found.endpoint, + found.address, found)) + if recorded(store, found): + trusted.add(found.sign_key) + progress = True + return tuple(taken) + + +def recorded(store: Store, found: Admission) -> bool: + """`found` recorded for its node; whether its signing key is the one + the store holds for the node: a node keeps the first key it was + known by, and evidence naming another one does not make it trusted""" + known = store.find(found.public_key) + if known is None: + store.members[found.public_key] = Member(found.sign_key, False, + found) + return True + member = store.members[known] + if member.sign_key is None: + member.sign_key = found.sign_key + if member.sign_key != found.sign_key: + return False + if member.admission is None: + member.admission = found + return True + + +def may_remove(store: Store, own: str, found: Removal) -> bool: + """Whether `found` was signed by a key that may remove its node: this + node's own, a trust root's, the node's own (it leaves), or the key + of the member that admitted it, as the evidence this node keeps + says. Any other member, trusted or not, removes it only from its + own spec.""" + roots = {one.sign_key for one in store.members.values() + if one.root and one.sign_key} + if found.by == own or found.by in roots: + return True + known = store.find(found.public_key) + if known is None: + return False + member = store.members[known] + return found.by == member.sign_key or ( + member.admission is not None and found.by == member.admission.by) + + +def room_for(store: Store, signer: str) -> bool: + """Whether the store takes another tombstone signed by `signer`""" + return len(store.removed) < MAX_REMOVED and sum( + 1 for one in store.removed.values() + if one.by == signer) < MAX_REMOVED_PER_SIGNER + + +def removals(store: Store, own: str, mesh_id: bytes, + found: Iterable[Removal]) -> tuple[str, ...]: + """The tombstones signed by a key that may remove their node + (`may_remove`), within the caps (`room_for`), recorded: the key is + trusted no more, and never taken again. The keys newly removed""" + taken = [] + for one in found: + if store.gone(one.public_key) or one.mesh_id != mesh_id.hex() or \ + not may_remove(store, own, one) or \ + not room_for(store, one.by) or \ + not signing.verified(one.by, one.message(), one.signature): + continue + record_removal(store, one) + taken.append(one.public_key) + return tuple(taken) + + +def record_removal(store: Store, found: Removal) -> None: + store.removed[found.public_key] = found + key = store.find(found.public_key) + if key is not None: + del store.members[key] + + +def make_roots(store: Store, keys: Iterable[str]) -> None: + """The operator's act: these members are trusted as roots""" + for key in keys: + known = store.find(key) + if known is None: + store.members[key] = Member(None, True) + else: + store.members[known].root = True + + +def bind_root(store: Store, key: str, sign_key: str) -> bool: + """A root's signing key, as the root itself gave it over the overlay; + whether it was bound now (a root bound before keeps its key)""" + known = store.find(key) + if known is None or not store.members[known].root or \ + store.members[known].sign_key is not None: + return False + store.members[known].sign_key = sign_key + return True diff --git a/keel/network/confirm.py b/keel/network/confirm.py index 1802e66..6ed1e95 100644 --- a/keel/network/confirm.py +++ b/keel/network/confirm.py @@ -10,7 +10,8 @@ needs no such proof. Decision 0048 adds two sources, for an overlay change keel mesh made and -no other: a join's authenticated session over the overlay (session.MESH), +no other: a join's authenticated session over the overlay, or a +WireGuard handshake from a member keel mesh sync added (session.MESH), and `keel mesh create` for a mesh with no peer yet (session.SELF). Each names the change it made, the marker it read back after its apply, and confirms only that one, after the same route check. @@ -309,8 +310,8 @@ def overlay_lines(pending: marker.Pending, origin: session.Origin, probes: Probes) -> list[str]: """Which of the two paths this confirmation tested""" if origin.kind == session.SELF: - return ["the overlay has no peer yet, so nothing can cross it:" - " keel mesh create confirmed it once the routes to the" + return [f"the overlay has no peer, so nothing can cross it:" + f" {origin.detail} confirmed it once the routes to the" " gateways and to the operator's session were found to" " still leave through the uplink"] if origin.kind == session.MESH: diff --git a/pyproject.toml b/pyproject.toml index c81bd1d..36b543f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "keel" -version = "0.17.0" +version = "0.18.0" authors = [{name = "KeelLinux maintainers"}] description = "Declarative management of one appliance instance" readme = "README.md" diff --git a/tests/mesh_helpers.py b/tests/mesh_helpers.py index c97fa45..b7289f5 100644 --- a/tests/mesh_helpers.py +++ b/tests/mesh_helpers.py @@ -12,7 +12,7 @@ from dataclasses import dataclass, field from datetime import datetime, timedelta, timezone -from keel.mesh import certificate, invites, protocol +from keel.mesh import certificate, identity, invites, protocol from keel.mesh.node import Change from keel.mesh.token import hmac_key, invite_id from keel.network import marker @@ -23,6 +23,13 @@ JOINER = "FHKH10gOWeK2bXHZPg8y+oPTprv556bwrmmRkbyEPgg=" INVITER = "nb9/izIukqWXM7gnBpe7hki4jKZZChWOW1wEfONn82E=" OTHER = "9vm/AzCVQQsWt1cU2eyjom4cABkNuodiHL8nDjKohEE=" +# a signing key's spelling, where no signature is checked +SIGNER = "AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8=" +MESH = bytes(range(16)) +EVIDENCE = protocol.Admission( + mesh_id=MESH.hex(), invite_id=invite_id(SECRET), public_key=JOINER, + sign_key=SIGNER, address="fd00:6b65:1::3", endpoint=None, time=0, + by=SIGNER, signature="A" * 86 + "==") NOW = datetime(2026, 10, 3, 12, 0, tzinfo=timezone.utc) ASSIGNED = "fd00:6b65:1::3/64" OWN = "fd00:6b65:1::1/64" @@ -47,7 +54,11 @@ def reserved(root: str, port: int = 51820, expires: datetime | None = None, invite_id=invite_id(secret), address=ASSIGNED, expires=expires or NOW + timedelta(hours=1), https_port=port, certificate=cert, hmac_key=hmac_key(secret), tls_key=key) - return invites.reserve(root, NOW, lambda others: made) + found = invites.reserve(root, NOW, lambda others: made) + # the inviter is in a mesh: it signs the new node's admission + if identity.read(root) is None: + identity.adopt(root, MESH) + return found @dataclass @@ -97,7 +108,8 @@ def __call__(self) -> datetime: def join_body(**changed) -> bytes: data = {"invite_id": INVITE, "public_key": JOINER, "endpoint": "[2001:db8:2::20]:51820", "address": ASSIGNED, - "nonce": protocol.new_nonce(), "time": protocol.seconds(NOW)} + "nonce": protocol.new_nonce(), "time": protocol.seconds(NOW), + "sign_key": SIGNER} data.update(changed) return json.dumps(data).encode() diff --git a/tests/mesh_netns.py b/tests/mesh_netns.py index c21eea0..c1cda4f 100644 --- a/tests/mesh_netns.py +++ b/tests/mesh_netns.py @@ -5,8 +5,10 @@ (tests/wgtools.py says how); never run by hand on a machine. This namespace is the inviter A. Two children, B and C, are network namespaces of their own (`unshare -n`), each joined to A by a veth on -its own /64. Then, with the real wg-quick, the real TLS listener and -client, the real flows and the real confirmation logic of 0018: +its own /64, with a default route through A, which forwards between +them as the internet would. Then, with the real wg-quick, the real TLS +listener and client, the real flows and the real confirmation logic of +0018: 1. A is in the mesh: its overlay up, and `keel mesh invite` through keel.cli prints the line; @@ -17,10 +19,17 @@ TCP answer, applies its side and prints `keel mesh accept`, which A runs (keel.mesh.inviting.accept); both windows are confirmed over the overlay, and C pings A; + A then announces C to B over the overlay, B adds C and confirms it + by C's WireGuard handshake, and C, whose accept line carried no + member, pulls B from A (keel mesh sync): the mesh is a full one, and + B and C ping each other over it; 4. the second token used again, from a fresh scratch root on C: the port gives no answer, so the join falls back again, and A refuses its line: the invite is spent. +A and B run the members' channel (keel.mesh.sync.serve) as +keel-mesh-members does, on their overlay address through wg0. + Two things are not keel's own here. apply: a live apply arms systemd timers on the host, which no namespace isolates, so apply writes keel's rendered wg-quick file under the node's scratch root, runs `wg-quick up` @@ -54,7 +63,7 @@ from datetime import datetime, timezone from keel import cli -from keel.mesh import inviting, joining +from keel.mesh import inviting, joining, memberd, sync from keel.mesh.node import Node from keel.mesh.token import parse from keel.network import marker, wireguard @@ -128,28 +137,73 @@ def node(root: str, path: str) -> Node: return Node(root, path, apply=netns_apply, run=Recorder()) -def ping(address: str) -> bool: - """One answer of five is enough: the link may lose packets""" - return subprocess.run(["ping", "-6", "-c", "5", "-i", "0.3", "-W", "3", - address], capture_output=True, - check=False).returncode == 0 +def ping(address: str, rounds: int = 3) -> bool: + """One answer of five is enough: the link may lose packets. Up to + three rounds: a tunnel the last change brought down and up needs a + new handshake, and on a poor link a lost initiation is sent again + only after WireGuard's 5 s rekey timeout""" + return any(subprocess.run(["ping", "-6", "-c", "5", "-i", "0.3", "-W", + "3", address], capture_output=True, + check=False).returncode == 0 + for _ in range(rounds)) + +def joiner(token_text: str, endpoint: str, after: str) -> None: + """In B's or C's namespace: join, then ping A over the overlay -def joiner(token_text: str, endpoint: str) -> None: - """In B's or C's namespace: join, then ping A over the overlay""" + `after` "serve": stay, as a member, with the members' channel up + (keel-mesh-members), answering the driver's `check` lines on + standard input; an overlay address: ping that member too, the one + this node learned of after its join. + """ root, path = scratch("version: 1\n") out, err = [], [] def said(line): out.append(line) print(line, flush=True) - found = joining.Joiner(node(root, path), parse(token_text, utcnow()), + here = node(root, path) + found = joining.Joiner(here, parse(token_text, utcnow()), utcnow, said, err.append) code = joining.run(found, endpoint) last = marker.last(root) - print("RESULT " + json.dumps({ - "code": code, "out": out, "err": err, "ping": ping(OVERLAY_A), - "outcome": last.outcome if last else None}), flush=True) + report = {"code": code, "out": out, "err": err, "ping": ping(OVERLAY_A), + "outcome": last.outcome if last else None, + "address": here.overlay().get("address"), + "key": here.public_key()[0]} + if after not in ("serve", "-"): + report["ping_other"] = ping(after) + report["peers"] = [one.get("public_key") + for one in here.overlay().get("peers") or []] + print("RESULT " + json.dumps(report), flush=True) + if after == "serve": + member(here, root) + + +def member(here: Node, root: str) -> None: + """keel-mesh-members, until the driver says `quit`; `check ADDRESS + KEY` waits for the member KEY to be this node's confirmed peer, and + pings it""" + log: list[str] = [] + service, stop = members_service(here, log.append) + for line in sys.stdin: + words = line.split() + if words[:1] != ["check"]: + break + deadline = time.monotonic() + 300 + while time.monotonic() < deadline: + keys = [one.get("public_key") + for one in here.overlay().get("peers") or []] + if words[2] in keys and not marker.exists(root) and \ + marker.last(root).outcome == marker.CONFIRMED: + break + time.sleep(1) + last = marker.last(root) + print("CHECK " + json.dumps({ + "peers": keys, "outcome": last.outcome if last else None, + "ping": ping(words[1]), "log": log}), flush=True) + stop.set() + service.join(30) CHILDREN: list[subprocess.Popen] = [] @@ -170,12 +224,13 @@ def said(line): class Spawned: """`keel mesh listen` as its own process, with no capability""" - def __init__(self, invite_id: str, path: str, seconds: int): + def __init__(self, invite_id: str, path: str, seconds: int, + action: str = "listen"): with open(LISTENER_LOG, "a") as log: self.process = subprocess.Popen( ["setpriv", "--no-new-privs", "--bounding-set", "-all", "--inh-caps", "-all", "--ambient-caps", "-all", - sys.executable, "-m", "keel", "mesh", "listen", path], + sys.executable, "-m", "keel", "mesh", action, path], stdout=log, stderr=log, cwd=LISTENER_HOME, env={**os.environ, "PYTHONPATH": LISTENER_HOME}) @@ -190,6 +245,23 @@ def stop(self) -> None: self.process.wait(10) +def members_listener(path: str) -> Spawned: + """`keel mesh members-listen`, the members' channel's unprivileged + listener, as keel-mesh-members-listen runs it""" + return Spawned("members", path, 0, "members-listen") + + +def members_service(here: Node, log) -> tuple[threading.Thread, + threading.Event]: + """keel-mesh-members: the root helper, and its listener""" + stop = threading.Event() + service = threading.Thread(target=memberd.serve, args=( + sync.Syncer(here, utcnow, log), stop), kwargs={ + "start": members_listener}) + service.start() + return service, stop + + def shaped(device: str, pid: int | None = None) -> None: if NETEM: delay, jitter, loss = NETEM @@ -227,7 +299,10 @@ def link(name: str) -> int: for argv in (("ip", "link", "set", "lo", "up"), ("ip", "-6", "address", "add", f"{net}::2/64", "dev", "uplink", "nodad"), - ("ip", "link", "set", "uplink", "up")): + ("ip", "link", "set", "uplink", "up"), + # the other children through A, as the internet would + ("ip", "-6", "route", "add", "default", "via", f"{net}::1", + "dev", "uplink")): sh(*argv, pid=child.pid) shaped(f"to{name}") shaped("uplink", child.pid) @@ -245,19 +320,24 @@ def invite(root: str, path: str, name: str, port: int) -> str: return out.getvalue().split()[-1] -def run_joiner(pid: int, token_text: str, name: str) -> subprocess.Popen: +def run_joiner(pid: int, token_text: str, name: str, + after: str = "-") -> subprocess.Popen: return subprocess.Popen( ["nsenter", "-t", str(pid), "-n", sys.executable, __file__, "join", - token_text, f"{LINKS[name]}::2"], stdout=subprocess.PIPE, - stderr=subprocess.PIPE, text=True) + token_text, f"{LINKS[name]}::2", after], stdin=subprocess.PIPE, + stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True) -def result(process: subprocess.Popen) -> dict: - out, err = process.communicate(timeout=300) - for line in out.splitlines(): - if line.startswith("RESULT "): - return json.loads(line[len("RESULT "):]) - raise RuntimeError(f"no result: {out}\n{err}") +def result(process: subprocess.Popen, prefix: str = "RESULT ") -> dict: + """The next line of `process` that starts with `prefix`, read as it + comes: a member that stays prints more later""" + seen = [] + for line in process.stdout: + if line.startswith(prefix): + return json.loads(line[len(prefix):]) + seen.append(line) + raise RuntimeError(f"no {prefix.strip()}: {''.join(seen)[-3000:]}\n" + f"{process.stderr.read()[-3000:]}") def accept_line(process: subprocess.Popen) -> str: @@ -270,6 +350,9 @@ def accept_line(process: subprocess.Popen) -> str: def inviter() -> None: sh("ip", "link", "set", "lo", "up") + # A routes between B and C: their endpoints reach each other + with open("/proc/sys/net/ipv6/conf/all/forwarding", "w") as fob: + fob.write("1\n") pids = {name: link(name) for name in LINKS} root, path = scratch("version: 1\nnetwork:\n overlay:\n wireguard:\n" f" address: {OVERLAY_A}/64\n") @@ -283,7 +366,11 @@ def inviter() -> None: journal = [] report = {} - # 1-2. the root helper and its unprivileged listener, and B joins + # A's members' channel, as keel-mesh-members runs it + service, stop = members_service(a, journal.append) + + # 1-2. the root helper and its unprivileged listener, and B joins; + # B stays, a member with its own members' channel first = invite(root, path, "B", 51900) a_side = inviting.Inviter(a, utcnow, journal.append, journal.append, run=Recorder(), output=lambda argv: None, @@ -293,23 +380,34 @@ def inviter() -> None: inviting.serve_invite(a_side, parse(first, utcnow()).invite_id))) thread.start() listening(f"{LINKS['B']}::1", 51900) - report["b"] = result(run_joiner(pids["B"], first, "B")) + b_process = run_joiner(pids["B"], first, "B", "serve") + report["b"] = result(b_process) thread.join(120) report["serve"] = codes report["a_after_b"] = marker.last(root).outcome + b_address = report["b"]["address"].split("/")[0] - # 3. nothing listens on the second invite's port: the fallback + # 3. nothing listens on the second invite's port: the fallback. A + # announces C to B once C is confirmed, and C, whose accept line + # carried no member, pulls B from A (keel mesh sync) second = invite(root, path, "C", 51901) accepting = inviting.Inviter(a, utcnow, journal.append, journal.append, run=Recorder(), output=lambda argv: None, start=Spawned) - process = run_joiner(pids["C"], second, "C") + process = run_joiner(pids["C"], second, "C", b_address) report["accept"] = inviting.accept(accepting, accept_line(process)) - rest, err = process.communicate(timeout=300) - report["c"] = next( - (json.loads(one[len("RESULT "):]) for one in rest.splitlines() - if one.startswith("RESULT ")), {"err": err}) + report["c"] = result(process) + process.wait(60) report["a_after_c"] = marker.last(root).outcome + # B learned of C through A's announcement, and confirmed it + b_process.stdin.write(f"check {report['c']['address'].split('/')[0]}" + f" {report['c']['key']}\n") + b_process.stdin.flush() + report["b_after_c"] = result(b_process, "CHECK ") + b_process.stdin.close() + b_process.wait(60) + stop.set() + service.join(30) # 4. the second token again: A refuses the line, the join is stopped process = run_joiner(pids["C"], second, "C") @@ -334,7 +432,7 @@ def inviter() -> None: if __name__ == "__main__": if sys.argv[1:2] == ["join"]: - joiner(sys.argv[2], sys.argv[3]) + joiner(sys.argv[2], sys.argv[3], sys.argv[4]) else: try: inviter() diff --git a/tests/mesh_sync_helpers.py b/tests/mesh_sync_helpers.py new file mode 100644 index 0000000..79fc76c --- /dev/null +++ b/tests/mesh_sync_helpers.py @@ -0,0 +1,140 @@ +# Copyright (c) 2026 KeelLinux maintainers +"""What the tests of sync, adopt, remove and the members' service share + +This node is JOINER at fd00:6b65:1::3, a real Node on a scratch root +(tests/mesh_wire.py: apply arms the window's marker, the real +keel.network.confirm decides), with a real signing key; INVITER at ::1 +is its one peer and its trust root, whose signing key lives in a scratch +root of its own (`INVITER_ROOT`), so its evidence is really signed. The +members' channel is `Network`: rosters by address, what was told and +touched, the latest handshakes. +""" + +import atexit +import shutil +import tempfile +import unittest +from datetime import timedelta +from unittest import mock + +import yaml +from mesh_helpers import INVITER, JOINER, MESH, NOW, OTHER, Clock +from mesh_wire import node + +from keel.mesh import identity, signing, sync, trust +from keel.mesh.memberlink import LinkError +from keel.mesh.members import Roster +from keel.mesh.protocol import Peer + +FOURTH = "AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8=" +SPLIT = bytes(16) +SPEC = f"""\ +version: 1 +network: + overlay: + wireguard: + address: fd00:6b65:1::3/64 + peers: + - public_key: {INVITER} + endpoint: '[2001:db8:1::10]:51820' + allowed_ips: [fd00:6b65:1::1/128] +""" +INVITER_ROOT = tempfile.mkdtemp(prefix="keel-inviter-") +atexit.register(shutil.rmtree, INVITER_ROOT, True) +INVITER_SIGNER = signing.ensure(INVITER_ROOT) +# OTHER's signing key: any, nothing it signs is checked here +OTHER_SIGNER = FOURTH + + +def evidence(key: str = OTHER, address: str = "fd00:6b65:1::7", + endpoint: str | None = "[2001:db8:3::30]:51820", + root: str = INVITER_ROOT, mesh: bytes = MESH, + sign_key: str = OTHER_SIGNER): + """The admission of `key`, signed by the inviter (or `root`)""" + return trust.admit(root, mesh, "0123456789abcdef", key, sign_key, + address, endpoint, NOW) + + +def admitted(key: str = OTHER, **changed) -> Peer: + found = evidence(key, **changed) + return Peer(found.public_key, found.endpoint, found.address, found) + + +def inviter_roster(*entries: Peer, identity_of: bytes | None = MESH, + removed=()) -> Roster: + return Roster(identity_of, INVITER, INVITER_SIGNER, "fd00:6b65:1::1", + (Peer(JOINER, None, "fd00:6b65:1::3"),) + entries, + tuple(removed)) + + +class Network: + """The members as this node reaches them over the overlay""" + + def __init__(self, rosters=None, handshakes=None): + self.rosters = dict(rosters or {}) + self.told, self.touched = [], [] + self.down: set[str] = set() + # key -> seconds since the epoch of its last handshake + self.handshakes = dict(handshakes or {}) + + def fetch(self, host, iface): + if host in self.down or host not in self.rosters: + raise LinkError(f"[{host}]:51821 through {iface}: timed out") + return self.rosters[host] + + def tell(self, host, iface, roster): + if host in self.down: + raise LinkError(f"[{host}]:51821 through {iface}: timed out") + self.told.append((host, roster)) + + def touch(self, host, iface): + self.touched.append(host) + + def output(self, argv): + if argv[-1] == "latest-handshakes": + return "".join(f"{key}\t{at}\n" + for key, at in self.handshakes.items()) + return None + + +class Case(unittest.TestCase): + """This node in MESH, trusting INVITER as its root""" + + def setUp(self): + self.root = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.root) + self.node = node(self.root, SPEC) + self.net = Network({"fd00:6b65:1::1": inviter_roster(admitted())}, + handshakes={OTHER: int(NOW.timestamp())}) + self.node.output = self.net.output + self.clock = Clock() + self.err, self.slept = [], [] + self.syncer = sync.Syncer( + self.node, self.clock, self.err.append, fetch=self.net.fetch, + tell=self.net.tell, touch=self.net.touch, sleep=self.later) + keys = mock.patch("keel.mesh.node.wgkeys.public", + return_value=(JOINER, None)) + keys.start() + self.addCleanup(keys.stop) + identity.adopt(self.root, MESH) + self.signer = signing.ensure(self.root) + self.trusting(INVITER, INVITER_SIGNER) + + def trusting(self, key: str, sign_key: str | None) -> None: + store = trust.load(self.root) + trust.make_roots(store, (key,)) + if sign_key: + trust.bind_root(store, key, sign_key) + trust.save(self.root, store) + + def later(self, seconds): + self.slept.append(seconds) + self.clock.now += timedelta(seconds=seconds) + + def peers(self): + with open(self.node.path) as fob: + doc = yaml.safe_load(fob) + return doc["network"]["overlay"]["wireguard"]["peers"] + + def said(self): + return "\n".join(self.err) diff --git a/tests/test_manifest_firewall.py b/tests/test_manifest_firewall.py index d13a4e3..3cd7949 100644 --- a/tests/test_manifest_firewall.py +++ b/tests/test_manifest_firewall.py @@ -108,6 +108,15 @@ def test_a_named_set_for_the_pending_invites_ports(self): without = self.render(CORE_OFF) self.assertNotIn("mesh_invites", without.text) + def test_the_members_port_on_the_overlay_alone(self): + """keel mesh sync and the announcement of a new node, between + members over the overlay (decision 0048, keel.mesh.members)""" + found = self.render(ADVANCED, WG) + self.assertIn('\t\tiifname "wg0" tcp dport 51821 accept\n', + found.text) + self.assertNftAccepts(found.text) + self.assertNotIn("51821", self.render(CORE_OFF).text) + def test_the_overlay_s_defaults(self): found = self.render(ADVANCED, {"address": "fd00:1::1/64"}) self.assertIn('iifname "wg0"', found.text) diff --git a/tests/test_mesh_acceptline.py b/tests/test_mesh_acceptline.py index 11bf56f..e8ade10 100644 --- a/tests/test_mesh_acceptline.py +++ b/tests/test_mesh_acceptline.py @@ -1,7 +1,7 @@ # Copyright (c) 2026 KeelLinux maintainers """The keel1a: line the fallback prints for the inviter (decision 0048) -The new node's key, its endpoint, the reserved address, the invite id +The new node's keys, its endpoint, the reserved address, the invite id and the time, with an HMAC keyed with the invite's HMAC key and a checksum, so a mistyped paste and a line for another invite are told apart and neither is accepted. @@ -16,7 +16,8 @@ JOINER = "FHKH10gOWeK2bXHZPg8y+oPTprv556bwrmmRkbyEPgg=" LINE = Accept(public_key=JOINER, endpoint="[2001:db8:2::20]:51820", address="fd00:6b65:1::3/64", invite_id="0123456789abcdef", - time=1_790_000_000) + time=1_790_000_000, + sign_key="AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8=") class TestRoundTrip(unittest.TestCase): @@ -27,7 +28,7 @@ def test_each_endpoint(self): made = Accept(**{**LINE.__dict__, "endpoint": endpoint}) text = acceptline.encode(made, KEY) self.assertTrue(text.startswith("keel1a:")) - self.assertLess(len(text), 200) + self.assertLess(len(text), 250) sealed = acceptline.parse(text) self.assertEqual(sealed.accept, made) self.assertTrue(sealed.authentic(KEY)) @@ -63,13 +64,14 @@ def test_a_layout_its_flags_do_not_match(self): (bytes([4]) + bytes(100), "flags"), (bytes([3]) + bytes(100), "flags"), (bytes([1]) + bytes(10), "malformed"), - (bytes([0]) + bytes(32) + bytes(61) + b"x", "bytes where")): + (bytes([0]) + bytes(64) + bytes(61) + b"x", "bytes where")): with self.subTest(payload=payload[:2]): self.refused(acceptline.wrap(payload), words) def test_values_no_inviter_could_use(self): for changed, words in ( ({"public_key": "A" * 43 + "B"}, "not a WireGuard key"), + ({"sign_key": "x"}, "signing key is not a key"), ({"endpoint": "[::1]:51820"}, "cannot be reached"), ({"address": "2001:db8::3/64"}, "not an overlay address")): with self.subTest(changed=changed): diff --git a/tests/test_mesh_admit.py b/tests/test_mesh_admit.py index 80093b3..cac0b63 100644 --- a/tests/test_mesh_admit.py +++ b/tests/test_mesh_admit.py @@ -22,9 +22,11 @@ JOINER, KEY, MADE, + MESH, NOW, OTHER, OWN, + SIGNER, Clock, FakeNode, confirm_body, @@ -33,7 +35,7 @@ signature, ) -from keel.mesh import admit, invites, protocol +from keel.mesh import admit, identity, invites, protocol, signing, trust from keel.mesh.admit import Admitter, stopped from keel.mesh.node import NodeError from keel.mesh.protocol import Peer @@ -92,6 +94,17 @@ def test_a_join_admits_the_new_node_and_answers_signed(self): answer.address, answer.etcd, answer.window), (INVITE, INVITER, OWN, "none", 120)) self.assertEqual(answer.peers, self.node.known) + # the evidence of the admission, signed with this node's key + evidence = answer.admission + self.assertEqual(answer.sign_key, signing.public(self.root)) + self.assertEqual((evidence.by, evidence.public_key, + evidence.sign_key, evidence.address, + evidence.invite_id, evidence.mesh_id), + (answer.sign_key, JOINER, SIGNER, OVERLAY_NEW, + INVITE, MESH.hex())) + self.assertTrue(signing.verified(evidence.by, evidence.message(), + evidence.signature)) + self.assertEqual(trust.load(self.root).evidence(JOINER), evidence) self.assertEqual(self.node.admitted, [{ "public_key": JOINER, "allowed_ips": [f"{OVERLAY_NEW}/128"], "endpoint": "[2001:db8:2::20]:51820"}]) @@ -138,6 +151,43 @@ def refuse(peer, window=None): self.assertIn("cannot be written: disk full; the invite is spent", self.logged[-1]) + def test_the_peers_carry_the_evidence_this_node_keeps(self): + self.node.known = (Peer(OTHER, None, "fd00:6b65:1::7"),) + signing.ensure(self.root) + store = trust.Store() + kept = trust.admit(self.root, MESH, "1123456789abcdef", OTHER, SIGNER, + "fd00:6b65:1::7", None, NOW) + trust.recorded(store, kept) + trust.save(self.root, store) + answer = protocol.join_answer(self.join().body) + self.assertEqual(answer.peers[0].admission, kept) + + def test_a_damaged_trust_store_spends_the_invite(self): + os.makedirs(os.path.dirname(join(self.root, trust.TRUST)), + exist_ok=True) + with open(join(self.root, trust.TRUST), "w") as fob: + fob.write("x") + self.assertEqual(self.join().status, 500) + self.assertIn("damaged", self.logged[-1]) + self.assertEqual(admit.evidenced(self.root, ( + Peer(OTHER, None, "fd00:6b65:1::7"),))[0].admission, None) + + def test_no_identity_or_no_signing_key_spends_the_invite(self): + os.remove(join(self.root, identity.IDENTITY)) + self.assertEqual(self.join().status, 500) + self.assertIn("no mesh identity", self.logged[-1]) + self.assertEqual(self.node.admitted, []) + other = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, other) + listener = Admitter(other, reserved(other), INVITER, OWN, self.node, + self.clock, self.logged.append) + with mock.patch("keel.mesh.admit.signing.ensure", + side_effect=signing.SigningError("no openssl")): + found = listener.forward(protocol.JOIN, signature( + protocol.JOIN, body := join_body()), body, "::", UPLINK) + self.assertEqual(found.status, 500) + self.assertIn("no openssl", self.logged[-1]) + def test_a_change_waiting_leaves_the_invite_pending(self): self.node.is_waiting = True found = self.join() @@ -203,6 +253,14 @@ def test_no_secret_in_the_log(self): class TestConfirm(Case): + def test_a_signing_key_that_cannot_be_named(self): + self.join() + with mock.patch("keel.mesh.admit.signing.public", + side_effect=signing.SigningError("gone")): + found = self.confirm() + self.assertEqual(found.status, 500) + self.assertEqual(self.node.origins, []) + def test_before_any_join(self): found = self.confirm() self.assertEqual(found.status, 409) diff --git a/tests/test_mesh_adopt.py b/tests/test_mesh_adopt.py new file mode 100644 index 0000000..a2aaf59 --- /dev/null +++ b/tests/test_mesh_adopt.py @@ -0,0 +1,214 @@ +# Copyright (c) 2026 KeelLinux maintainers +"""keel.mesh.adopt: the identity and the trust roots, set by the operator + +`keel mesh sync --adopt` repairs a split and gives a node of a mesh +built by hand its identity and roots; `keel mesh create --adopt` gives +such a mesh one identity; `keel mesh invite` never makes a second. +""" + +import os +from unittest import mock + +from mesh_helpers import INVITER, MESH, reserved +from mesh_sync_helpers import ( + FOURTH, + INVITER_SIGNER, + SPLIT, + Case, + admitted, + inviter_roster, +) + +from keel import exits +from keel.mesh import adopt, identity, signing, trust +from keel.mesh.members import Roster + + +class TestAdoptFrom(Case): + """keel mesh sync --adopt ADDRESS""" + + def test_the_split_repaired(self): + """web-2 made its own identity: it takes web-1's, trusts its + peers as roots, then syncs""" + identity.replace(self.root, SPLIT) + os.remove(os.path.join(self.root, trust.TRUST)) + self.assertEqual(adopt.adopt_from(self.syncer, "fd00:6b65:1::1"), + exits.OK, self.err) + self.assertEqual(identity.read(self.root), MESH) + self.assertIn("this node now holds the mesh identity 00010203", + self.said()) + self.assertIn("(was 00000000", self.said()) + found = trust.load(self.root).members[INVITER] + self.assertTrue(found.root) + self.assertEqual(found.sign_key, INVITER_SIGNER) + self.assertEqual(len(self.peers()), 2) + + def test_a_node_with_no_identity_yet(self): + os.remove(os.path.join(self.root, identity.IDENTITY)) + self.assertEqual(adopt.adopt_from(self.syncer, "fd00:6b65:1::1"), + exits.OK, self.err) + self.assertIn("(was none)", self.said()) + + def test_refused(self): + self.assertEqual(adopt.adopt_from(self.syncer, "fd00:6b65:1::9"), + exits.MESH_REFUSED) + self.assertIn("not a peer of this node", self.err[-1]) + self.net.down.add("fd00:6b65:1::1") + self.assertEqual(adopt.adopt_from(self.syncer, "fd00:6b65:1::1"), + exits.MESH_REFUSED) + self.assertIn("did not answer", self.err[-1]) + self.net.down.clear() + self.net.rosters["fd00:6b65:1::1"] = inviter_roster( + identity_of=None) + self.assertEqual(adopt.adopt_from(self.syncer, "fd00:6b65:1::1"), + exits.MESH_REFUSED) + self.assertIn("holds no mesh identity", self.err[-1]) + self.assertEqual(identity.read(self.root), MESH) + + def test_no_repair_while_an_invite_carries_the_old_identity(self): + reserved(self.root) + identity.replace(self.root, SPLIT) + self.assertEqual(adopt.adopt_from(self.syncer, "fd00:6b65:1::1"), + exits.MESH_REFUSED) + self.assertIn("pending invite", self.err[-1]) + self.assertEqual(identity.read(self.root), SPLIT) + + def test_no_key_or_no_signing_key(self): + with mock.patch("keel.mesh.node.wgkeys.public", + return_value=(None, "no key")): + self.assertEqual(adopt.adopt_from(self.syncer, + "fd00:6b65:1::1"), + exits.MESH_REFUSED) + self.assertEqual(self.err[-1], "no key") + with mock.patch("keel.mesh.adopt.signing.ensure", + side_effect=signing.SigningError("no openssl")): + self.assertEqual(adopt.adopt_from(self.syncer, + "fd00:6b65:1::1"), + exits.MESH_REFUSED) + self.assertEqual(self.err[-1], "no openssl") + + +class TestIdentityFile(Case): + def test_a_damaged_identity_is_replaced_by_the_members(self): + with open(os.path.join(self.root, identity.IDENTITY), "w") as fob: + fob.write("x\n") + self.assertIsNone(identity.replace(self.root, MESH)) + self.assertEqual(identity.read(self.root), MESH) + + +class TestAdoptMesh(Case): + """keel mesh create --adopt: one identity for a mesh built by hand""" + + def setUp(self): + super().setUp() + os.remove(os.path.join(self.root, identity.IDENTITY)) + os.remove(os.path.join(self.root, trust.TRUST)) + self.out = [] + self.net.rosters["fd00:6b65:1::1"] = inviter_roster( + identity_of=None) + + def test_a_new_identity_when_no_member_has_one(self): + self.assertEqual(adopt.adopt_mesh(self.syncer, self.out.append), + exits.OK, self.err) + made = identity.read(self.root) + self.assertIsNotNone(made) + self.assertIn(made.hex(), self.out[0]) + self.assertIn("keel mesh sync --adopt", self.out[1]) + found = trust.load(self.root).members[INVITER] + # a root, its signing key bound once it answers with the identity + self.assertTrue(found.root) + self.assertIsNone(found.sign_key) + + def test_the_identity_a_member_has_and_its_root_bound(self): + self.net.rosters["fd00:6b65:1::1"] = inviter_roster(admitted()) + self.assertEqual(adopt.adopt_mesh(self.syncer, self.out.append), + exits.OK) + self.assertEqual(identity.read(self.root), MESH) + found = trust.load(self.root).members[INVITER] + self.assertEqual(found.sign_key, INVITER_SIGNER) + # bound, and vouched for by no one + self.assertIsNone(found.admission) + + def test_its_own_kept(self): + identity.adopt(self.root, SPLIT) + self.assertEqual(adopt.adopt_mesh(self.syncer, self.out.append), + exits.OK) + self.assertEqual(identity.read(self.root), SPLIT) + + def test_refused_on_a_split(self): + identity.adopt(self.root, SPLIT) + self.net.rosters["fd00:6b65:1::1"] = inviter_roster() + self.assertEqual(adopt.adopt_mesh(self.syncer, self.out.append), + exits.MESH_REFUSED) + self.assertIn("keel mesh sync --adopt", self.err[-1]) + + def test_refused_when_members_disagree(self): + with open(self.node.path, "a") as fob: + fob.write(f" - public_key: {FOURTH}\n" + " allowed_ips: [fd00:6b65:1::9/128]\n") + self.net.rosters["fd00:6b65:1::1"] = inviter_roster() + self.net.rosters["fd00:6b65:1::9"] = Roster( + SPLIT, FOURTH, INVITER_SIGNER, "fd00:6b65:1::9", ()) + self.assertEqual(adopt.adopt_mesh(self.syncer, self.out.append), + exits.MESH_REFUSED) + self.assertIsNone(identity.read(self.root)) + + def test_no_overlay(self): + with open(self.node.path, "w") as fob: + fob.write("version: 1\n") + self.assertEqual(adopt.adopt_mesh(self.syncer, self.out.append), + exits.MESH_REFUSED) + self.assertIn("keel mesh create makes", self.err[-1]) + + def test_a_damaged_trust_store(self): + os.makedirs(os.path.dirname(os.path.join(self.root, trust.TRUST)), + exist_ok=True) + with open(os.path.join(self.root, trust.TRUST), "w") as fob: + fob.write("x") + self.assertEqual(adopt.adopt_mesh(self.syncer, self.out.append), + exits.MESH_REFUSED) + self.assertIn("damaged", self.err[-1]) + + +class TestInviteIdentity(Case): + """invite never makes a second identity for a mesh""" + + def setUp(self): + super().setUp() + self.net.rosters["fd00:6b65:1::1"] = inviter_roster() + + def test_its_own_when_the_members_agree(self): + self.assertEqual(adopt.invite_identity(self.syncer), MESH) + + def test_a_node_with_no_peer_makes_one(self): + os.remove(os.path.join(self.root, identity.IDENTITY)) + with open(self.node.path, "w") as fob: + fob.write("version: 1\nnetwork:\n overlay:\n wireguard:\n" + " address: fd00:6b65:1::3/64\n") + self.assertEqual(len(adopt.invite_identity(self.syncer)), 16) + + def test_refused_with_peers_and_no_identity_even_when_they_have_one( + self): + os.remove(os.path.join(self.root, identity.IDENTITY)) + with self.assertRaises(adopt.Refused) as raised: + adopt.invite_identity(self.syncer) + self.assertIn("keel mesh create --adopt", str(raised.exception)) + self.assertIsNone(identity.read(self.root)) + + def test_refused_on_a_split(self): + identity.replace(self.root, SPLIT) + with self.assertRaises(adopt.Refused) as raised: + adopt.invite_identity(self.syncer) + self.assertIn("keel mesh sync --adopt fd00:6b65:1::1", + str(raised.exception)) + + def test_members_that_cannot_be_asked_are_warned_about(self): + self.net.down.add("fd00:6b65:1::1") + self.assertEqual(adopt.invite_identity(self.syncer), MESH) + self.assertIn("Warning", self.err[-1]) + + def test_no_key(self): + with mock.patch("keel.mesh.node.wgkeys.public", + return_value=(None, "no key")), \ + self.assertRaises(ValueError): + adopt.invite_identity(self.syncer) diff --git a/tests/test_mesh_channel.py b/tests/test_mesh_channel.py index f62b21a..8dfe304 100644 --- a/tests/test_mesh_channel.py +++ b/tests/test_mesh_channel.py @@ -251,7 +251,7 @@ def test_an_answer_not_signed_with_the_key(self): class TooLong(Listener): def handle(self, *args): - return Response(200, b"x" * (protocol.MAX_BODY + 1), None) + return Response(200, b"x" * (protocol.MAX_ANSWER + 1), None) class TestTooLongAnswer(Served): diff --git a/tests/test_mesh_cli.py b/tests/test_mesh_cli.py index e4ebd33..00f16e7 100644 --- a/tests/test_mesh_cli.py +++ b/tests/test_mesh_cli.py @@ -26,8 +26,9 @@ from helpers import spec # noqa: F401 from keel import cli, exits -from keel.mesh import invites +from keel.mesh import adopt, commands, invites from keel.mesh import token as mesh_token +from keel.mesh.endpoint import Choice from keel.mesh.identity import IDENTITY OVERLAY = """\ @@ -306,6 +307,18 @@ def test_openssl_missing_or_failing(self): self.assertIn(words, err) self.assertEqual(self.invite_files(), []) + def test_an_invite_that_would_split_the_identity(self): + self.write_spec(OVERLAY) + self.key_file() + with self.public(), mock.patch( + "keel.mesh.commands.invite_identity", + side_effect=adopt.Refused("this node's peers are in mesh" + " 9fe71b35…")): + code, out, err = self.invite() + self.assertEqual((code, out), (exits.MESH_REFUSED, "")) + self.assertIn("9fe71b35", err) + self.assertEqual(self.invite_files(), []) + def test_a_damaged_identity_is_not_replaced(self): self.write_spec(OVERLAY) self.key_file() @@ -328,6 +341,48 @@ def test_the_live_system_needs_root(self): self.assertIn("must run as root", err) +class TestInviteEndpoints(unittest.TestCase): + """--endpoint, else the static addresses, else found on the uplink""" + + STATIC = {"network": {"interfaces": {"eth0": {"ipv6": { + "method": "static", "address": "2001:db8:1::10/64"}}}}} + + def test_static_addresses_before_the_uplink(self): + said = [] + with mock.patch("keel.mesh.commands.endpoint.detect") as detect: + self.assertEqual(commands.invite_endpoints( + None, self.STATIC, True, said.append), ("2001:db8:1::10",)) + self.assertEqual(commands.invite_endpoints( + ["192.0.2.1"], {}, True, said.append), ("192.0.2.1",)) + detect.assert_not_called() + self.assertEqual(said, []) + + def test_found_on_the_uplink_of_the_live_system(self): + said = [] + found = Choice(("2001:db8:7::9",), + ("endpoint 2001:db8:7::9, found on eth0",)) + with mock.patch("keel.mesh.commands.endpoint.detect", + return_value=found) as detect: + self.assertEqual(commands.invite_endpoints( + None, {}, True, said.append), ("2001:db8:7::9",)) + with self.assertRaises(ValueError): + commands.invite_endpoints(None, {}, False, said.append) + self.assertEqual(detect.call_count, 1) + self.assertEqual(said, ["endpoint 2001:db8:7::9, found on eth0"]) + + def test_only_a_privacy_or_private_address_is_refused(self): + said = [] + found = Choice((), (), ("2001:db8::7 is a SLAAC privacy address",)) + with mock.patch("keel.mesh.commands.endpoint.detect", + return_value=found), \ + self.assertRaises(ValueError) as raised: + commands.invite_endpoints(None, {}, True, said.append) + self.assertIn("--endpoint", str(raised.exception)) + self.assertIn("none was found on the uplink", str(raised.exception)) + self.assertIn("SLAAC privacy address", str(raised.exception)) + self.assertEqual(said, []) + + class TestJoinDryRun(Case): def token(self, **overrides) -> str: values = dict( diff --git a/tests/test_mesh_cli_live.py b/tests/test_mesh_cli_live.py index 544090c..7d8f178 100644 --- a/tests/test_mesh_cli_live.py +++ b/tests/test_mesh_cli_live.py @@ -21,7 +21,8 @@ from test_mesh_cli import run_cli from keel import exits -from keel.mesh import commands, invites, inviting +from keel.mesh import commands, identity, invites, inviting +from keel.mesh.endpoint import Choice from keel.mesh.token import encode from keel.network import session @@ -60,7 +61,8 @@ class TestNeedsRoot(Case): def test_each_command_that_changes_the_machine(self): for argv in (("create",), ("accept", "keel1a:x"), ("serve", "0123456789abcdef"), - ("join", "keel1:x")): + ("join", "keel1:x"), ("sync",), ("members",), + ("create", "--adopt"), ("remove", "fd00::7")): with self.subTest(argv=argv), \ mock.patch("os.geteuid", return_value=1000): code, out, err = run_cli("mesh", *argv) @@ -119,6 +121,34 @@ def test_a_token_given_as_an_argument_is_warned_about(self): _, _, err = self.mesh("join", "-", stdin=line) self.assertNotIn("Warning", err) + def test_join_finds_its_endpoint_on_the_uplink(self): + with open(self.spec, "w") as fob: + fob.write("version: 1\n") + found = Choice(("2001:db8:7::9",), ("endpoint 2001:db8:7::9, found" + " on eth0",)) + with mock.patch("keel.mesh.commands.joining.run", + return_value=exits.OK) as joined, \ + mock.patch("keel.mesh.commands.endpoint.detect", + return_value=found): + code, _, err = self.mesh("join", "-", stdin=self.line()) + self.assertEqual(code, exits.OK) + self.assertEqual(joined.call_args.args[1], "2001:db8:7::9") + self.assertIn("endpoint 2001:db8:7::9, found on eth0", err) + + def test_join_without_an_endpoint_says_why(self): + with open(self.spec, "w") as fob: + fob.write("version: 1\n") + found = Choice((), (), ("10.0.0.5 is a private address",)) + with mock.patch("keel.mesh.commands.joining.run", + return_value=exits.OK) as joined, \ + mock.patch("keel.mesh.commands.endpoint.detect", + return_value=found): + code, _, err = self.mesh("join", "-", stdin=self.line()) + self.assertEqual(code, exits.OK) + self.assertIsNone(joined.call_args.args[1]) + self.assertIn("no endpoint is sent, as from a node behind NAT:" + " 10.0.0.5 is a private address", err) + def test_listen_needs_no_root(self): with mock.patch("os.geteuid", return_value=61234), \ mock.patch("keel.mesh.commands.bridge.listen", @@ -157,6 +187,85 @@ def test_serve(self): self.assertEqual(signal.getsignal(signal.SIGTERM), inviting.terminated) + def test_sync(self): + with mock.patch("keel.mesh.commands.sync.pull", + return_value=exits.OK) as pulled: + self.assertEqual(self.mesh("sync")[0], exits.OK) + self.assertEqual(self.mesh("sync", "--from", "fd00:6b65:1::01", + "--from", "fd00:6b65:1::7")[0], + exits.OK) + syncer, hosts = pulled.call_args_list[0].args + self.assertEqual((syncer.node.root, hosts), (self.root, ())) + self.assertEqual(pulled.call_args_list[1].args[1], + ("fd00:6b65:1::1", "fd00:6b65:1::7")) + + def test_sync_adopt(self): + with mock.patch("keel.mesh.commands.adopt.adopt_from", + return_value=exits.OK) as adopted: + self.assertEqual(self.mesh("sync", "--adopt", + "fd00:6b65:1::1")[0], exits.OK) + self.assertEqual(adopted.call_args.args[1], "fd00:6b65:1::1") + + def test_sync_with_an_address_that_is_not_an_overlay_one(self): + for argv in (("--from", "x"), ("--adopt", "192.0.2.1")): + with self.subTest(argv=argv): + code, _, err = self.mesh("sync", *argv) + self.assertEqual(code, exits.MESH_REFUSED) + self.assertIn("not an IPv6 address", err) + + def test_members(self): + """the root helper, until SIGTERM sets its stop and ends it + through its cleanup""" + before = signal.getsignal(signal.SIGTERM) + self.addCleanup(signal.signal, signal.SIGTERM, before) + + def serve(syncer, stop): + with self.assertRaises(SystemExit): + signal.getsignal(signal.SIGTERM)(signal.SIGTERM, None) + self.assertTrue(stop.is_set()) + return exits.OK + with mock.patch("keel.mesh.commands.memberd.serve", + side_effect=serve) as served: + self.assertEqual(self.mesh("members")[0], exits.OK) + self.assertEqual(served.call_args.args[0].node.root, self.root) + + def test_members_listen_needs_no_root(self): + with mock.patch("os.geteuid", return_value=61234), \ + mock.patch("keel.mesh.commands.memberd.listen", + return_value=exits.OK) as listened: + code, _, _ = run_cli("mesh", "members-listen", + "/run/keel/mesh-members/bridge.sock") + self.assertEqual(code, exits.OK) + self.assertEqual(listened.call_args.args[0], + "/run/keel/mesh-members/bridge.sock") + + def test_remove(self): + with mock.patch("keel.mesh.commands.remove.remove", + return_value=exits.OK) as removed: + self.assertEqual(self.mesh("remove", "fd00::7")[0], exits.OK) + syncer, which, _ = removed.call_args.args + self.assertEqual((syncer.node.root, which), (self.root, "fd00::7")) + + def test_create_adopt(self): + with mock.patch("keel.mesh.commands.adopt.adopt_mesh", + return_value=exits.OK) as adopted, \ + mock.patch("keel.mesh.commands.create.create") as made: + self.assertEqual(self.mesh("create", "--adopt")[0], exits.OK) + made.assert_not_called() + self.assertEqual(adopted.call_args.args[0].node.root, self.root) + + def test_the_identity_of_an_invite(self): + with mock.patch("keel.mesh.commands.adopt.invite_identity", + return_value=bytes(16)) as asked: + made = commands.invite_identity(self.root, self.spec) + self.assertEqual(made, identity.read(self.root)) + asked.assert_not_called() + with mock.patch("keel.mesh.commands.ROOT_DEFAULT", self.root): + self.assertEqual(commands.invite_identity(self.root, + self.spec), + bytes(16)) + self.assertEqual(asked.call_args.args[0].node.path, self.spec) + def test_status(self): with open(self.spec, "a") as fob: fob.write(" overlay:\n wireguard:\n" diff --git a/tests/test_mesh_create_status.py b/tests/test_mesh_create_status.py index 5c9c13f..cfa7f51 100644 --- a/tests/test_mesh_create_status.py +++ b/tests/test_mesh_create_status.py @@ -25,13 +25,19 @@ from keel.mesh.node import ( Node, NodeError, - global_addresses, live_apply, with_overlay, ) from keel.network import marker +def saying(doc, root, window): + """apply as the live one prints it""" + print(f"keel mesh: bring the overlay wg0 up; it reverts in {window} s" + " unless `keel network confirm` is run from a new session") + return Armed()(doc, root, window) + + class TestCreate(unittest.TestCase): def setUp(self): self.root = tempfile.mkdtemp() @@ -63,9 +69,19 @@ def test_a_random_prefix_this_node_at_one_confirmed_by_itself(self): def test_the_operator_session_routed_into_it_reverts_it(self): self.node.probes = lambda: probes("wg0") + self.node.apply = saying self.assertEqual(self.create(), exits.NETWORK_NOT_CONFIRMED) self.assertTrue(marker.exists(self.root)) - self.assertIn("leaves through wg0", self.err[0]) + self.assertEqual(self.err[0], "confirming the new overlay…") + self.assertIn("unless `keel network confirm`", self.err[1]) + self.assertIn("leaves through wg0", self.err[2]) + + def test_apply_s_revert_warning_is_not_shown_when_it_confirms(self): + self.node.apply = saying + self.assertEqual(self.create(), exits.OK) + self.assertNotIn("unless", "\n".join(self.err)) + self.assertEqual(self.err[:2], ["confirming the new overlay…", + "confirmed from keel mesh create"]) def test_a_node_already_in_a_mesh(self): with open(self.node.path, "w") as fob: @@ -86,9 +102,13 @@ def test_a_damaged_identity(self): self.assertIn("does not hold a mesh identity", self.err[0]) def test_the_overlay_does_not_come_up(self): - self.node.apply = Armed(code=16, arms=False) + def failing(doc, root, window): + print("wg-quick up failed") + return 16 + self.node.apply = failing self.assertEqual(self.create(), exits.APPLY_FAILED) - self.assertIn("apply exited 16", self.err[0]) + self.assertEqual(self.err[0], "wg-quick up failed") + self.assertIn("apply exited 16", self.err[1]) class TestStatus(unittest.TestCase): @@ -111,7 +131,10 @@ def wg(self, argv): def test_the_peers_their_handshakes_and_the_invites(self): reserved(self.root) + identity.adopt(self.root, bytes(range(16))) found = status.lines(self.overlay, self.root, NOW, self.wg) + self.assertEqual(found.pop(1), "mesh identity:" + " 000102030405060708090a0b0c0d0e0f") self.assertEqual(found[:4], [ "this node: fd00:6b65:1::1/64 on wg0, WireGuard on UDP 51820", "public key: nb9/izIukqWXM7gnBpe7hki4jKZZChWOW1wEfONn82E=", @@ -139,12 +162,22 @@ def test_no_secret_is_asked_for(self): def test_an_interface_that_is_down(self): found = status.lines(self.overlay, self.root, NOW, lambda argv: None) self.assertIn("public key: unknown (is the interface up?)", found) - self.assertIn("no handshake known", found[3]) + self.assertIn("no handshake known", found[4]) def test_off_the_live_system(self): found = status.lines(self.overlay, self.root, NOW, None) - self.assertEqual(found[1], "not the live system: no handshake read") - self.assertTrue(found[3].endswith("[2001:db8:2::20]:51820")) + self.assertEqual(found[2], "not the live system: no handshake read") + self.assertTrue(found[4].endswith("[2001:db8:2::20]:51820")) + + def test_no_identity_yet_or_a_damaged_one(self): + found = status.lines(self.overlay, self.root, NOW, None) + self.assertIn("none yet", found[1]) + self.assertIn("keel mesh create --adopt", found[1]) + os.makedirs(os.path.join(self.root, "var/lib/keel/mesh")) + with open(os.path.join(self.root, identity.IDENTITY), "w") as fob: + fob.write("x\n") + found = status.lines(self.overlay, self.root, NOW, None) + self.assertIn("does not hold a mesh identity", found[1]) def test_no_mesh(self): for overlay in (None, {}): @@ -228,18 +261,29 @@ def test_the_peers_the_answer_carries(self): self.assertEqual(found.peers(OTHER)[0].address, "fd00:6b65:1::3") self.assertEqual(found.peers(JOINER), ()) - def test_the_live_apply_writes_its_lines_to_standard_error(self): + def test_the_live_apply(self): with mock.patch("keel.mesh.node.apply_system", - side_effect=lambda *args, **kwargs: print("line") - or 0) as applied, \ - mock.patch("sys.stderr") as err, \ - mock.patch("sys.stdout") as out: + return_value=0) as applied: self.assertEqual(live_apply({"version": 1}, "/", 150), 0) applied.assert_called_once_with( {"version": 1}, "/", False, "keel mesh", defer_certificate=True, network_window=150, skip_uplink=True) - self.assertTrue(err.write.called) + + def test_apply_s_lines_are_held_by_the_change_not_printed(self): + def saying(doc, root, window): + print("it reverts in 120 s unless `keel network confirm`") + return Armed()(doc, root, window) + found = node(self.root, INVITER_SPEC, apply=saying) + with mock.patch("sys.stdout") as out: + change = found.admit({"public_key": JOINER, + "allowed_ips": ["fd00:6b65:1::3/128"]}) self.assertFalse(out.write.called) + self.assertEqual(change.lines, + ("it reverts in 120 s unless `keel network" + " confirm`",)) + said = [] + change.shown(said.append) + self.assertEqual(said, list(change.lines)) def test_a_spec_that_cannot_be_written(self): found = node(self.root) @@ -276,13 +320,6 @@ def wg(argv): found.output = lambda argv: None self.assertIsNone(found.handshake(JOINER)) - def test_global_addresses(self): - text = ("2: eth0 inet6 2001:db8::5/64 scope global \\\n" - "3: wg0@NONE inet6 fd00::1/64 scope global\n" - "garbage\n") - self.assertEqual(global_addresses(text), [ - ("eth0", "2001:db8::5"), ("wg0", "fd00::1")]) - def test_defaults_are_live(self): found = Node("/", "/etc/keel/instance.yaml") self.assertIs(found.apply, live_apply) diff --git a/tests/test_mesh_endpoint.py b/tests/test_mesh_endpoint.py new file mode 100644 index 0000000..93319b0 --- /dev/null +++ b/tests/test_mesh_endpoint.py @@ -0,0 +1,134 @@ +# Copyright (c) 2026 KeelLinux maintainers +"""keel.mesh.endpoint: the address another node reaches this one at + +Found on the uplink when the spec declares no static address (a DHCP or +SLAAC host): its global, stable IPv6 address, else a public IPv4 one; +never a unique local address or one of the overlay's. +""" + +import unittest + +from keel.mesh import endpoint + +# `ip -o address show scope global` on a SLAAC and DHCP host +HELD = """\ +2: eth0 inet 10.88.5.212/24 metric 100 brd 10.88.5.255 scope global dynamic eth0\\ valid_lft 81578sec preferred_lft 81578sec +2: eth0 inet6 2804:710:d0:5:be24:11ff:fece:a035/64 scope global dynamic mngtmpaddr noprefixroute \\ valid_lft 2591782sec preferred_lft 604582sec +3: lxcbr0 inet 10.0.3.1/24 brd 10.0.3.255 scope global lxcbr0\\ valid_lft forever preferred_lft forever +3: lxcbr0 inet6 fc42:5009:ba4b:5ab0::1/64 scope global \\ valid_lft forever preferred_lft forever +4: wg0 inet6 fd00:6b65:1::1/64 scope global \\ valid_lft forever preferred_lft forever +""" # noqa: E501 +ROUTES6 = ("default nhid 61 via fe80::be24:11ff:fe71:707d dev eth0 proto ra" + " metric 100 expires 1581sec pref medium\n") +ROUTES4 = "default via 10.88.5.254 dev eth0 proto dhcp src 10.88.5.212\n" + + +def line(iface: str, address: str, flags: str = "") -> str: + family = "inet6" if ":" in address else "inet" + return (f"2: {iface} {family} {address} scope global {flags}" + " \\ valid_lft 100sec preferred_lft 100sec\n") + + +def reader(held: str, routes6: str = ROUTES6, routes4: str = ROUTES4): + def output(argv): + if argv[:3] == ("ip", "-o", "address"): + return held + if argv[:2] == ("ip", "-6"): + return routes6 + if argv[:2] == ("ip", "-4"): + return routes4 + raise AssertionError(argv) + return output + + +class TestHeld(unittest.TestCase): + def test_parsed_with_their_flags(self): + found = endpoint.held(HELD) + self.assertEqual(found[1].iface, "eth0") + self.assertEqual(str(found[1].address), + "2804:710:d0:5:be24:11ff:fece:a035") + self.assertIn("mngtmpaddr", found[1].flags) + self.assertEqual([one.iface for one in found], + ["eth0", "eth0", "lxcbr0", "lxcbr0", "wg0"]) + + def test_what_is_not_an_address_line_is_skipped(self): + self.assertEqual(endpoint.held("garbage\n1: lo inet6 x/1\n"), []) + + def test_uplinks_from_the_default_routes(self): + self.assertEqual(endpoint.uplinks(ROUTES6 + ROUTES4), ("eth0",)) + self.assertEqual(endpoint.uplinks("default via 192.0.2.1\n"), ()) + + +class TestPick(unittest.TestCase): + def pick(self, held, uplinks=("eth0",)): + return endpoint.pick(endpoint.held(held), uplinks) + + def test_the_uplink_s_stable_ipv6_and_not_rfc1918(self): + found = self.pick(HELD) + self.assertEqual(found.addresses, + ("2804:710:d0:5:be24:11ff:fece:a035",)) + self.assertEqual(found.refused, ()) + self.assertIn("eth0", found.said[0]) + + def test_a_static_address_before_a_slaac_one(self): + found = self.pick(line("eth0", "2001:db8::a/64", "dynamic mngtmpaddr") + + line("eth0", "2001:db8::5/64")) + self.assertEqual(found.addresses, ("2001:db8::5",)) + + def test_a_public_ipv4_beside_the_ipv6(self): + found = self.pick(line("eth0", "2001:db8::5/64") + + line("eth0", "198.51.100.7/24", "dynamic")) + self.assertEqual(found.addresses, ("2001:db8::5", "198.51.100.7")) + + def test_no_ula_no_wg_and_nothing_off_the_uplink(self): + found = self.pick(line("eth0", "fd12::5/64") + + line("wg0", "2001:db8:7::1/64") + + line("eth1", "2001:db8:8::1/64")) + self.assertEqual(found.addresses, ()) + self.assertIn("no global address", found.refused[0]) + + def test_without_a_default_route_any_interface_but_wg(self): + found = self.pick(line("wg0", "2001:db8:7::1/64") + + line("eth1", "2001:db8:8::1/64"), uplinks=()) + self.assertEqual(found.addresses, ("2001:db8:8::1",)) + + def test_deprecated_and_temporary_are_not_stable(self): + found = self.pick(line("eth0", "2001:db8::d/64", "deprecated dynamic") + + line("eth0", "2001:db8::7/64", + "temporary dynamic") + + line("eth0", "203.0.113.9/24")) + self.assertEqual(found.addresses, ("203.0.113.9",)) + self.assertEqual(found.refused, ()) + self.assertIn("privacy", found.said[-1]) + + def test_only_a_privacy_address_is_refused(self): + found = self.pick(line("eth0", "2001:db8::7/64", "temporary dynamic")) + self.assertEqual(found.addresses, ()) + self.assertIn("privacy address", found.refused[0]) + + def test_only_rfc1918_or_carrier_nat_is_refused(self): + for address in ("10.0.0.5/24", "100.64.1.5/10"): + found = self.pick(line("eth0", address, "dynamic")) + self.assertEqual(found.addresses, ()) + self.assertIn("RFC 1918", found.refused[0]) + + def test_a_privacy_ipv6_and_rfc1918_are_both_refused(self): + found = self.pick(line("eth0", "2001:db8::7/64", "temporary") + + line("eth0", "192.168.1.5/24")) + self.assertEqual(found.addresses, ()) + self.assertEqual(len(found.refused), 2) + + +class TestDetect(unittest.TestCase): + def test_through_ip(self): + found = endpoint.detect(reader(HELD)) + self.assertEqual(found.addresses, + ("2804:710:d0:5:be24:11ff:fece:a035",)) + + def test_ip_gives_no_answer(self): + found = endpoint.detect(lambda argv: None) + self.assertEqual(found.addresses, ()) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_mesh_inviting.py b/tests/test_mesh_inviting.py index add4aba..88cadfe 100644 --- a/tests/test_mesh_inviting.py +++ b/tests/test_mesh_inviting.py @@ -22,6 +22,7 @@ JOINER, KEY, NOW, + SIGNER, Clock, confirm_body, join_body, @@ -46,6 +47,13 @@ UPLINK = "2001:db8:2::20" +def saying(doc, root, window): + """apply as the live one prints it""" + print(f"keel mesh: bring the overlay wg0 up; it reverts in {window} s" + " unless `keel network confirm` is run from a new session") + return Armed()(doc, root, window) + + class Played: """Bridge.run, without a listener: the requests it would forward""" @@ -85,6 +93,8 @@ def setUp(self): self.inviter = inviting.Inviter( self.node, Clock(), self.out.append, self.err.append, run=self.run, output=lambda argv: "table inet keel {}") + self.announced = [] + self.inviter.announce = self.announced.append keys = mock.patch("keel.mesh.inviting.wgkeys.public", return_value=(INVITER, None)) self.public = keys.start() @@ -128,6 +138,16 @@ def test_a_join_and_its_confirmation(self): "mesh_invites", "{ 51820 }"), self.run.calls) self.assertEqual(self.err[-1], f"invite {INVITE}: joined and" " confirmed") + self.assertEqual(self.announced, [JOINER]) + + def test_apply_s_lines_go_to_the_journal(self): + self.node.apply = saying + self.inviter.bridged = Played(join_request(), confirm_request()) + self.assertEqual(inviting.serve_invite(self.inviter, INVITE), + exits.OK) + self.assertIn(f"invite {INVITE}: keel mesh: bring the overlay wg0" + " up; it reverts in 120 s unless `keel network" + " confirm` is run from a new session", self.err) def test_sigterm_ends_through_the_cleanup(self): with self.assertRaises(SystemExit): @@ -153,6 +173,7 @@ def test_a_port_already_taken(self): self.assertIn("Address in use", self.err[1]) self.assertFalse(os.path.exists(self.file())) self.assertIn("expired unused", self.err[-1]) + self.assertEqual(self.announced, []) def test_another_invite_on_the_port_keeps_it_open(self): reserved(self.root, secret=bytes(32)) @@ -165,7 +186,7 @@ class TestAccept(Case): def line(self, key=KEY, **changed): values = dict(public_key=JOINER, endpoint="[2001:db8:2::20]:51820", address=ASSIGNED, invite_id=INVITE, - time=protocol.seconds(NOW)) + time=protocol.seconds(NOW), sign_key=SIGNER) values.update(changed) return acceptline.encode(Accept(**values), key) @@ -188,6 +209,7 @@ def test_the_peer_applied_and_the_mesh_session_on_the_overlay(self): self.run.calls) self.assertEqual(self.out, [f"accepted: {JOINER} is a peer of this" f" node at {JOINER_OVERLAY}"]) + self.assertEqual(self.announced, [JOINER]) self.assertFalse(os.path.exists(self.file())) self.assertFalse(marker.exists(self.root)) @@ -234,8 +256,12 @@ def test_an_invite_used_already(self): self.assertIn("already used", self.err[0]) def test_the_change_does_not_come_up(self): - self.node.apply = Armed(code=16, arms=False) + def failing(doc, root, window): + print("wg-quick up failed") + return 16 + self.node.apply = failing self.assertEqual(self.accept(), exits.APPLY_FAILED) + self.assertEqual(self.err[-2], "wg-quick up failed") self.assertIn("apply exited 16", self.err[-1]) self.assertFalse(os.path.exists(self.file())) @@ -246,10 +272,18 @@ def test_a_spec_that_cannot_take_the_peer(self): self.assertEqual(self.err[-1], "full; the invite is spent") def test_the_new_node_never_comes(self): + self.node.apply = saying self.assertEqual(self.accept(played=Played()), exits.NETWORK_NOT_CONFIRMED) self.assertIn("not confirmed", self.err[-1]) + self.assertIn("unless `keel network confirm`", self.err[-2]) self.assertTrue(marker.exists(self.root)) + self.assertEqual(self.announced, []) + + def test_apply_s_revert_warning_is_not_shown_when_it_confirms(self): + self.node.apply = saying + self.assertEqual(self.accept(), exits.OK) + self.assertNotIn("unless", "\n".join(self.err)) def test_the_overlay_address_cannot_be_bound(self): found = self.accept(played=Played(fail=OSError(99, "not here"))) @@ -302,6 +336,13 @@ def test_the_listener_starts_as_its_unit_unless_told(self): unit.assert_called_once_with(INVITE, "path", 30, self.run, self.inviter.output) + def test_the_announcement_goes_through_keel_mesh_sync(self): + self.inviter.announce = None + with mock.patch("keel.mesh.inviting.sync.announce") as announce: + self.inviter.announced(JOINER) + syncer, key = announce.call_args.args + self.assertEqual((syncer.node, key), (self.node, JOINER)) + def test_own(self): self.assertEqual(self.inviter.own(), (INVITER, "fd00:6b65:1::1/64")) self.public.return_value = (None, "unreadable") diff --git a/tests/test_mesh_joining.py b/tests/test_mesh_joining.py index efb7e82..59b6528 100644 --- a/tests/test_mesh_joining.py +++ b/tests/test_mesh_joining.py @@ -12,6 +12,7 @@ import shutil import tempfile import unittest +from dataclasses import replace from datetime import timedelta from unittest import mock @@ -20,8 +21,10 @@ ASSIGNED, INVITER, JOINER, + MESH, NOW, OTHER, + SIGNER, Clock, reserved, ) @@ -37,8 +40,17 @@ ) from keel import exits -from keel.mesh import acceptline, identity, invites, joining, protocol +from keel.mesh import ( + acceptline, + identity, + invites, + joining, + protocol, + signing, + trust, +) from keel.mesh.admit import Admitter, Joined +from keel.mesh.endpoint import Choice from keel.mesh.listener import Listener, Params from keel.mesh.node import NodeError from keel.mesh.token import hmac_key @@ -47,6 +59,15 @@ JOINER_ENDPOINT = "2001:db8:2::20" +class Saying(Armed): + """apply, and the lines it prints, as the live one does""" + + def __call__(self, doc, root, window): + print(f"keel mesh: bring the overlay wg0 up; it reverts in {window}" + " s unless `keel network confirm` is run from a new session") + return super().__call__(doc, root, window) + + def spec_of(found): with open(found.path) as fob: return yaml.safe_load(fob) @@ -71,7 +92,7 @@ def setUp(self): self.invite.expires, "::", 51820, "fd00:6b65:1::1"), self.admitter, self.clock, self.logged.append) self.wire = Wire(self.listener, token(self.invite).fingerprint) - self.out, self.err, self.slept = [], [], [] + self.out, self.err, self.slept, self.learned = [], [], [], [] keys = mock.patch.multiple( "keel.mesh.joining.wgkeys", generate=mock.DEFAULT, public=mock.DEFAULT) @@ -86,7 +107,8 @@ def joining(self, endpoint=JOINER_ENDPOINT, route="eth0", **changed): self.joiner, token(self.invite, **changed), self.clock, self.out.append, self.err.append, post=self.wire, sleep=self.slept.append, - route_dev=lambda host: route) + route_dev=lambda host: route, + learn=lambda host: self.learned.append(host) or exits.OK) return joining.run(self.found, endpoint) @@ -117,18 +139,123 @@ def test_both_specs_and_both_windows(self): self.assertEqual([path for _, path in self.wire.calls], [protocol.JOIN, protocol.CONFIRM]) self.assertEqual(self.wire.calls[1][0], "fd00:6b65:1::1") + # the answer carried the members: nothing to pull + self.assertEqual(self.learned, []) def test_a_node_behind_nat_sends_no_endpoint(self): self.assertEqual(self.joining(endpoint=None), exits.OK, self.err) theirs = spec_of(self.inviter)["network"]["overlay"]["wireguard"] self.assertNotIn("endpoint", theirs["peers"][0]) - def test_a_mesh_of_three_says_what_this_keel_does_not_do(self): + def other_member(self, evidenced=True): + """OTHER, a peer of the inviter, with evidence the inviter signed""" with open(self.inviter.path, "a") as fob: fob.write(f" peers:\n - public_key: {OTHER}\n" + " endpoint: '[2001:db8:3::30]:51820'\n" " allowed_ips: [fd00:6b65:1::7/128]\n") + if evidenced: + signing.ensure(self.inviter_root) + store = trust.load(self.inviter_root) + trust.recorded(store, trust.admit( + self.inviter_root, MESH, "1123456789abcdef", OTHER, SIGNER, + "fd00:6b65:1::7", "[2001:db8:3::30]:51820", NOW)) + trust.save(self.inviter_root, store) + + def test_a_member_without_evidence_is_not_taken(self): + self.other_member(evidenced=False) + self.assertEqual(self.joining(), exits.OK, self.err) + mine = spec_of(self.joiner)["network"]["overlay"]["wireguard"] + self.assertEqual(len(mine["peers"]), 1) + + def test_the_inviter_is_this_node_s_trust_root(self): + self.assertEqual(self.joining(), exits.OK, self.err) + store = trust.load(self.joiner_root) + self.assertTrue(store.members[INVITER].root) + self.assertEqual(store.members[INVITER].sign_key, + signing.public(self.inviter_root)) + # and the inviter keeps this node's evidence + evidence = trust.load(self.inviter_root).evidence(JOINER) + self.assertEqual(evidence.sign_key, signing.public(self.joiner_root)) + self.assertEqual(evidence.invite_id, self.invite.invite_id) + + def test_evidence_that_is_not_this_node_s_admission(self): + real = self.wire + + def forged(host, port, path, body, *args, **kwargs): + reply = real(host, port, path, body, *args, **kwargs) + if path != protocol.JOIN: + return reply + data = protocol.join_answer(reply.body) + other = protocol.dumps(replace(data, admission=replace( + data.admission, address="fd00:6b65:1::9"))) + return reply.__class__(other, reply.local, reply.peer) + self.wire = forged + self.assertEqual(self.joining(), exits.MESH_REFUSED) + self.assertIn("no valid evidence of this node's admission", + self.err[-1]) + + def test_a_damaged_trust_store(self): + os.makedirs(os.path.join(self.joiner_root, "var/lib/keel/mesh"), + exist_ok=True) + with open(os.path.join(self.joiner_root, trust.TRUST), "w") as fob: + fob.write("x") + self.assertEqual(self.joining(), exits.MESH_REFUSED) + self.assertIn("damaged", self.err[-1]) + + def test_no_signing_key_can_be_made(self): + with mock.patch("keel.mesh.joining.signing.ensure", + side_effect=signing.SigningError("no openssl")): + self.assertEqual(self.joining(), exits.APPLY_FAILED) + self.assertEqual(self.err[-1], "no openssl") + + def test_the_other_members_the_inviter_knows_are_peers_too(self): + self.other_member() self.assertEqual(self.joining(), exits.OK, self.err) + mine = spec_of(self.joiner)["network"]["overlay"]["wireguard"] + self.assertEqual(mine["peers"][1], { + "public_key": OTHER, "endpoint": "[2001:db8:3::30]:51820", + "allowed_ips": ["fd00:6b65:1::7/128"]}) + self.assertEqual(len(mine["peers"]), 2) self.assertIn("the mesh has 3 nodes", self.out[-1]) + self.assertIn("the inviter announces this node to the 1 other", + self.out[-1]) + + def test_a_member_this_node_cannot_take_is_left_out(self): + """its own key, or the inviter's again, even with evidence: never + written, the join goes on (the rest: test_mesh_members.py)""" + answer = (protocol.Peer(JOINER, None, "fd00:6b65:1::8"), + protocol.Peer(INVITER, None, "fd00:6b65:1::9")) + signing.ensure(self.inviter_root) + store = trust.load(self.inviter_root) + for one in answer: + trust.recorded(store, trust.admit( + self.inviter_root, MESH, "1123456789abcdef", + one.public_key, SIGNER, + one.address, None, NOW)) + trust.save(self.inviter_root, store) + with mock.patch.object(self.inviter, "peers", return_value=answer): + self.assertEqual(self.joining(), exits.OK, self.err) + mine = spec_of(self.joiner)["network"]["overlay"]["wireguard"] + self.assertEqual([one["public_key"] for one in mine["peers"]], + [INVITER]) + + def test_apply_s_revert_warning_only_when_the_confirmation_fails(self): + """the join confirms itself: `unless keel network confirm` is not + what the operator has to do""" + self.joiner.apply = Saying() + self.assertEqual(self.joining(), exits.OK, self.err) + said = "\n".join(self.err) + self.assertNotIn("unless `keel network confirm`", said) + first = self.err.index("confirming over the overlay…") + self.assertTrue(self.err[first + 1].startswith("confirmed from")) + + def test_apply_s_revert_warning_when_the_confirmation_fails(self): + self.joiner.apply = Saying() + self.joiner.probes = lambda: probes("wg0") + self.assertEqual(self.joining(), exits.NETWORK_NOT_CONFIRMED) + said = "\n".join(self.err) + self.assertIn("confirming over the overlay…", said) + self.assertIn("reverts in 120 s unless `keel network confirm`", said) def test_the_mesh_session_is_retried_while_the_tunnel_comes_up(self): calls = [] @@ -326,6 +453,7 @@ def setUp(self): def accepted(self): """What accept leaves the inviter in: the peer applied, the listener waiting for the mesh session""" + signing.ensure(self.inviter_root) self.inviter.admit({"public_key": JOINER, "allowed_ips": [f"{JOINER_OVERLAY}/128"]}) until = NOW + timedelta(seconds=120) @@ -349,6 +477,41 @@ def test_the_line_for_the_inviter_and_the_wait(self): [(doc, window)] = self.joiner.apply.documents self.assertEqual(window, 900) self.assertEqual(self.out[-1], joining.TWO_NODES) + # no answer carried the inviter's members: they are pulled + self.assertEqual(self.learned, ["fd00:6b65:1::1"]) + self.assertIn("learning the other members from the inviter…", + self.err) + + def test_the_members_are_pulled_with_keel_mesh_sync(self): + found = joining.Joiner(self.joiner, token(self.invite), self.clock, + self.out.append, self.err.append) + with mock.patch("keel.mesh.joining.sync.pull", + return_value=exits.OK) as pulled: + self.assertEqual(found.learned("fd00:6b65:1::1"), exits.OK) + syncer, hosts = pulled.call_args.args + self.assertEqual((syncer.node, hosts), + (self.joiner, ("fd00:6b65:1::1",))) + + def test_the_inviter_s_signing_key_is_learned_from_its_answer(self): + self.accepted() + self.assertEqual(self.joining(), exits.OK, self.err) + found = trust.load(self.joiner_root).members[INVITER] + self.assertEqual(found.sign_key, signing.public(self.inviter_root)) + + def test_a_trust_store_that_cannot_take_the_inviter(self): + self.accepted() + os.makedirs(os.path.join(self.joiner_root, "var/lib/keel/mesh"), + exist_ok=True) + with open(os.path.join(self.joiner_root, trust.TRUST), "w") as fob: + fob.write("x") + self.assertEqual(self.joining(), exits.NETWORK_NOT_CONFIRMED) + self.assertIn("damaged", self.err[-1]) + self.assertEqual(self.learned, []) + + def test_nothing_is_pulled_when_the_join_is_not_confirmed(self): + """the inviter never ran accept: the mesh session is refused""" + self.assertEqual(self.joining(), exits.NETWORK_NOT_CONFIRMED) + self.assertEqual(self.learned, []) def test_the_window_is_the_time_left_to_the_expiry(self): self.accepted() @@ -375,20 +538,19 @@ def test_this_side_does_not_come_up(self): class TestOwnEndpoint(unittest.TestCase): def test_declared_first(self): - self.assertEqual(joining.own_endpoint("2001:db8::5", {}, [], "wg0"), + self.assertEqual(joining.own_endpoint("2001:db8::5", {}, None), "2001:db8::5") - def test_static_then_global_ipv6_first(self): + def test_static_then_found_on_the_uplink(self): doc = {"network": {"interfaces": {"eth0": { "ipv4": {"method": "static", "address": "192.0.2.5/24"}}}}} - held = [("eth0", "198.51.100.1"), ("wg0", "fd00::1"), - ("eth1", "2001:db8::9"), ("lo", "::1")] - self.assertEqual(joining.own_endpoint(None, doc, held, "wg0"), - "2001:db8::9") - self.assertEqual(joining.own_endpoint(None, doc, [], "wg0"), + found = Choice(("2001:db8::9", "198.51.100.1")) + self.assertEqual(joining.own_endpoint(None, doc, found), "192.0.2.5") - self.assertIsNone(joining.own_endpoint(None, {}, [("lo", "::1")], - "wg0")) + self.assertEqual(joining.own_endpoint(None, {}, found), + "2001:db8::9") + self.assertIsNone(joining.own_endpoint(None, {}, Choice(()))) + self.assertIsNone(joining.own_endpoint(None, {}, None)) def test_endpoint_text(self): self.assertEqual(joining.endpoint_text("192.0.2.1"), "192.0.2.1") diff --git a/tests/test_mesh_listener.py b/tests/test_mesh_listener.py index 780953b..4090509 100644 --- a/tests/test_mesh_listener.py +++ b/tests/test_mesh_listener.py @@ -16,9 +16,11 @@ from unittest import mock from mesh_helpers import ( + EVIDENCE, INVITE, JOINER, NOW, + SIGNER, Clock, confirm_body, join_body, @@ -35,7 +37,8 @@ ANSWER = protocol.dumps(protocol.JoinAnswer( invite_id=INVITE, nonce="00" * 16, public_key="nb9/izIukqWXM7gnBpe7hki4jKZZChWOW1wEfONn82E=", - address="fd00:6b65:1::1/64", peers=(), etcd="none", window=120)) + address="fd00:6b65:1::1/64", peers=(), etcd="none", window=120, + sign_key=SIGNER, admission=EVIDENCE)) def params(**changed) -> Params: @@ -95,7 +98,8 @@ def test_a_good_join_is_forwarded_and_its_answer_learnt(self): def test_the_confirmation_over_the_overlay_only(self): self.join() self.backend.body = protocol.dumps(protocol.ConfirmAnswer( - invite_id=INVITE, nonce="00" * 16, confirmed=True, detail="")) + invite_id=INVITE, nonce="00" * 16, confirmed=True, detail="", + sign_key=SIGNER)) for local, peer in ((UPLINK, JOINER_OVERLAY), (OVERLAY, UPLINK)): self.assertEqual(self.confirm(local, peer).status, 403) self.assertEqual(self.confirm().status, 200) diff --git a/tests/test_mesh_memberd.py b/tests/test_mesh_memberd.py new file mode 100644 index 0000000..50b3e8b --- /dev/null +++ b/tests/test_mesh_memberd.py @@ -0,0 +1,338 @@ +# Copyright (c) 2026 KeelLinux maintainers +"""keel.mesh.memberd: the members' channel, root helper and listener + +The root side's answers as a function, the queue of announcements, the +bridge's messages, the listener's unit, and the whole path in one +process: the root helper and the unprivileged listener over their unix +socket, a member's pull and announcement over TCP on the loopback, the +announcement applied by the worker. +""" + +import base64 +import json +import os +import shutil +import socket +import tempfile +import threading +import unittest +from datetime import datetime, timezone +from unittest import mock + +from mesh_helpers import INVITER, JOINER, OTHER +from mesh_sync_helpers import Case, admitted, inviter_roster +from test_network_window import Recorder + +from keel import exits +from keel.mesh import memberd, memberlink, members, signing +from keel.mesh.bridge import BridgeError +from keel.mesh.memberd import Members, Pending + + +def roster(key=INVITER): + return members.Roster(None, key, key, "fd00:6b65:1::1", ()) + + +class TestPending(unittest.TestCase): + def test_given_once_it_settled(self): + now = [100.0] + found = Pending(settle=10, clock=lambda: now[0]) + found.put("a", roster()) + self.assertEqual(found.take(0.01), []) + now[0] += 5 + found.put("b", roster()) + now[0] += 9 + self.assertEqual(found.take(0.01), []) + now[0] += 1 + self.assertEqual([one for one, _ in found.take(0.01)], ["a", "b"]) + + def test_the_latest_per_sender_bounded(self): + found = Pending(limit=2, settle=0) + self.assertTrue(found.put("a", roster(INVITER))) + self.assertTrue(found.put("a", roster(JOINER))) + self.assertTrue(found.put("b", roster())) + self.assertFalse(found.put("c", roster())) + self.assertTrue(found.put("b", roster())) + taken = found.take(0) + self.assertEqual([(source, one.public_key) for source, one in taken], + [("a", JOINER), ("b", INVITER)]) + self.assertEqual(found.take(0.01), []) + + +class TestMembers(unittest.TestCase): + def setUp(self): + self.logged = [] + self.pending = Pending(limit=1, settle=0) + self.roster = roster() + self.handler = Members(self.give, self.member_of, self.pending, + self.logged.append) + + def give(self): + if self.roster is None: + raise ValueError("no key") + return self.roster + + @staticmethod + def member_of(source): + return {"fd00::1": INVITER, "fd00::3": JOINER}.get(source) + + def body(self, key=INVITER): + return members.dumps(roster(key)) + + def test_answers(self): + for args, status in ( + (("GET", "/v1/members", b"", "fd00::9"), 403), + (("GET", "/v1/members", b"", "fd00::1"), 200), + (("PUT", "/v1/members", b"", "fd00::1"), 404), + (("POST", "/v1/announce", None, "fd00::1"), 404), + (("POST", "/v1/announce", b"{}", "fd00::1"), 400), + (("POST", "/v1/announce", self.body(JOINER), "fd00::1"), + 403), + (("POST", "/v1/announce", self.body(), "fd00::1"), 202), + (("POST", "/v1/announce", self.body(JOINER), "fd00::3"), + 503)): + with self.subTest(args=args[:2] + args[3:]): + self.assertEqual(self.handler.handle(*args).status, status) + self.roster = None + self.assertEqual(self.handler.handle("GET", "/v1/members", b"", + "fd00::1").status, 503) + self.assertIn("not a peer of this node", self.logged[0]) + + def test_the_bridge_s_messages(self): + found = memberd.answered(self.handler, json.dumps({ + "op": "member", "method": "GET", "path": "/v1/members", + "body": base64.b64encode(b"").decode(), + "source": "fd00::1"}).encode()) + self.assertEqual(found["status"], 200) + found = memberd.answered(self.handler, json.dumps({ + "op": "member", "method": "POST", "path": "/v1/announce", + "body": None, "source": "fd00::1"}).encode()) + self.assertEqual(found["status"], 404) + for data in (b"x", b"{}", json.dumps({"op": "member"}).encode(), + json.dumps({"op": "member", "method": "GET", + "path": "/", "body": "!", + "source": "fd00::1"}).encode(), + json.dumps({"op": "member", "method": "GET", + "path": "/", "body": None, + "source": "x"}).encode()): + with self.subTest(data=data), self.assertRaises(BridgeError): + memberd.answered(self.handler, data) + + +class TestRemote(unittest.TestCase): + def test_a_question_and_the_root_side_gone(self): + ours, theirs = socket.socketpair() + self.addCleanup(ours.close) + stop = threading.Event() + remote = memberd.Remote(ours, stop) + + def root(): + memberd.line(theirs) + memberd.send(theirs, {"status": 200, "body": base64.b64encode( + b"ok").decode()}) + memberd.line(theirs) + theirs.close() + thread = threading.Thread(target=root) + thread.start() + found = remote.handle("GET", "/v1/members", b"", "fd00::1") + self.assertEqual((found.status, found.body), (200, b"ok")) + found = remote.handle("POST", "/v1/announce", None, "fd00::1") + thread.join(5) + self.assertEqual(found.status, 503) + self.assertTrue(stop.is_set()) + ours.close() + self.assertEqual(remote.handle("GET", "/v1/members", b"", + "fd00::1").status, 503) + + +class TestMessage(unittest.TestCase): + def test_a_message_the_end_and_a_stop(self): + ours, theirs = socket.socketpair() + self.addCleanup(ours.close) + stop = threading.Event() + theirs.sendall(b'{"a": 1}\n\n{"b"') + self.assertEqual(memberd.message(ours, stop, 0.01), b'{"a": 1}\n') + self.assertEqual(memberd.message(ours, stop, 0.01), b"\n") + theirs.close() + # a message cut short, then the end + self.assertIsNone(memberd.message(ours, stop, 0.01)) + self.assertIsNone(memberd.message(ours, stop, 0.01)) + ours2, theirs2 = socket.socketpair() + self.addCleanup(ours2.close) + self.addCleanup(theirs2.close) + threading.Timer(0.05, stop.set).start() + self.assertIsNone(memberd.message(ours2, stop, 0.01)) + + +class TestUnit(unittest.TestCase): + def test_the_listener_runs_as_a_sandboxed_transient_unit(self): + run = Recorder() + found = memberd.start_unit("/run/keel/mesh-members/bridge.sock", run, + lambda argv: "42\n") + self.assertEqual(run.calls[0], ("systemctl", "stop", + "keel-mesh-members-listen")) + argv = run.calls[1] + self.assertIn("--property=DynamicUser=yes", argv) + self.assertIn("--property=CapabilityBoundingSet=", argv) + self.assertIn("--property=RuntimeDirectory=keel/mesh-members", argv) + self.assertIn("--property=RuntimeDirectoryMode=0700", argv) + self.assertEqual(argv[-2:], ("members-listen", + "/run/keel/mesh-members/bridge.sock")) + self.assertTrue(found.trusted(42, 61000)) + self.assertFalse(found.trusted(42, 0)) + + def test_a_unit_that_cannot_start(self): + with self.assertRaises(BridgeError): + memberd.start_unit("/x", lambda argv: "failed" + if argv[0] == "systemd-run" else None, + lambda argv: None) + + +def status_file(text: str) -> str: + fd, found = tempfile.mkstemp() + with os.fdopen(fd, "w") as fob: + fob.write(text) + return found + + +class TestListen(unittest.TestCase): + def setUp(self): + self.logged = [] + self.clean = status_file("CapEff:\t0000000000000000\n") + self.addCleanup(os.remove, self.clean) + + def listen(self, status, **kwargs): + return memberd.listen("/nonexistent/x.sock", + lambda: datetime.now(timezone.utc), + self.logged.append, status=status, **kwargs) + + def test_a_listener_with_capabilities_refuses_to_run(self): + caps = status_file("CapEff:\t0000000000003000\n") + self.addCleanup(os.remove, caps) + self.assertEqual(self.listen(caps), exits.APPLY_FAILED) + self.assertIn("without capabilities", self.logged[0]) + + def test_no_root_helper(self): + with mock.patch("keel.mesh.memberd.bridge.helper", + return_value=None): + self.assertEqual(self.listen(self.clean), exits.APPLY_FAILED) + self.assertIn("no root helper", self.logged[0]) + + def test_parameters_it_cannot_use(self): + for sent in (b"", b"x\n", b'{"iface": "lo"}\n'): + ours, theirs = socket.socketpair() + theirs.sendall(sent) + theirs.close() + with self.subTest(sent=sent), \ + mock.patch("keel.mesh.memberd.bridge.helper", + return_value=ours): + self.assertEqual(self.listen(self.clean), + exits.APPLY_FAILED) + self.assertIn("no parameters", self.logged[-1]) + + +class Started: + """The listener's unit, as a thread of this process""" + + def __init__(self, path, status, stop): + self.stop_listener = stop + self.thread = threading.Thread(target=memberd.listen, args=( + path, lambda: datetime.now(timezone.utc), lambda line: None, + status, 0.05, stop)) + self.thread.start() + self.stopped = False + + def trusted(self, pid, uid): + return pid == os.getpid() + + def stop(self): + self.stopped = True + + +def free_port() -> int: + with socket.socket(socket.AF_INET6) as found: + found.bind(("::1", 0)) + return found.getsockname()[1] + + +class TestTheWholePath(Case): + def setUp(self): + super().setUp() + self.run_dir = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.run_dir) + self.status = status_file("CapEff:\t0000000000000000\n") + self.addCleanup(os.remove, self.status) + + def test_a_pull_and_an_announcement_through_both_processes(self): + port = free_port() + stop, listener_stop = threading.Event(), threading.Event() + started = [] + + def start(path): + started.append(Started(path, self.status, listener_stop)) + return started[0] + codes = [] + member = mock.patch("keel.mesh.memberd.sync.member_of", + side_effect=lambda node, source: + INVITER if source == "::1" else None) + member.start() + self.addCleanup(member.stop) + root = threading.Thread(target=lambda: codes.append(memberd.serve( + self.syncer, stop, start=start, where=lambda: ("lo", "::1"), + port=port, path=os.path.join(self.run_dir, "bridge.sock"), + settle=0.1))) + root.start() + for _ in range(200): + try: + found = memberlink.fetch("::1", "lo", port) + break + except memberlink.LinkError: + stop.wait(0.05) + self.assertEqual(found.public_key, JOINER) + self.assertEqual(found.sign_key, self.signer) + memberlink.tell("::1", "lo", inviter_roster(admitted()), port) + for _ in range(200): + if len(self.peers()) == 2: + break + stop.wait(0.05) + self.assertEqual(self.peers()[1]["public_key"], OTHER) + # the root helper's stop ends it between messages + stop.set() + root.join(10) + listener_stop.set() + started[0].thread.join(10) + self.assertEqual(codes, [exits.OK]) + self.assertTrue(started[0].stopped) + self.assertIn("an announcement from ::1: 2 member(s)", self.err) + + def test_no_signing_key_no_overlay_no_listener(self): + with mock.patch("keel.mesh.memberd.signing.ensure", + side_effect=signing.SigningError("no openssl")): + self.assertEqual(memberd.serve(self.syncer, threading.Event()), + exits.APPLY_FAILED) + stop = threading.Event() + threading.Timer(0.1, stop.set).start() + with open(self.node.path, "w") as fob: + fob.write("version: 1\n") + with mock.patch("keel.mesh.memberd.WAIT_OVERLAY", 0.02): + self.assertEqual(memberd.serve(self.syncer, stop), exits.OK) + + def fail(path): + raise BridgeError("the listener could not start: no systemd") + self.assertEqual(memberd.serve(self.syncer, threading.Event(), + start=fail, + where=lambda: ("lo", "::1")), + exits.APPLY_FAILED) + self.assertIn("no systemd", self.err[-1]) + + def test_the_default_listener_is_its_unit(self): + with mock.patch("keel.mesh.memberd.start_unit", + side_effect=BridgeError("x")) as started: + memberd.serve(self.syncer, threading.Event(), + where=lambda: ("lo", "::1")) + self.assertEqual(started.call_args.args[0], + memberd.socket_path(self.root)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_mesh_memberlink.py b/tests/test_mesh_memberlink.py new file mode 100644 index 0000000..6d638c6 --- /dev/null +++ b/tests/test_mesh_memberlink.py @@ -0,0 +1,307 @@ +# Copyright (c) 2026 KeelLinux maintainers +"""keel.mesh.memberlink: the members' channel, with real sockets + +On the loopback, which stands for the overlay's interface: both ends +bound to `lo` as they are to wg0. The listener's front applies its +limits (a deadline per request, slots, one connection per source, size +caps, sources refused too often) and hands what passed to a backend +that stands for the root side (keel.mesh.memberd.Members). +""" + +import socket +import threading +import unittest +from datetime import datetime, timezone +from unittest import mock + +from mesh_helpers import INVITER, OTHER, SIGNER + +from keel.mesh import memberlink, members +from keel.mesh.memberlink import Answer, Front, LinkError, refused +from keel.mesh.protocol import Peer + +ROSTER = members.Roster(bytes(range(16)), INVITER, SIGNER, "fd00:6b65:1::1", + (Peer(OTHER, None, "fd00:6b65:1::7"),)) + + +def free_port() -> int: + with socket.socket(socket.AF_INET6) as found: + found.bind(("::1", 0)) + return found.getsockname()[1] + + +class Backend: + """The root side, as the front sees it: the roster, announcements + taken, or a refusal when told""" + + def __init__(self): + self.calls = [] + self.refuse = False + + def handle(self, method, path, body, source): + self.calls.append((method, path, source)) + if self.refuse: + return refused(403, "not a member of this node's mesh") + if method == "GET": + return Answer(200, members.dumps(ROSTER)) + return Answer(memberlink.ACCEPTED, b'{"queued": true}') + + +class Served(unittest.TestCase): + """The listener's server on [::1]:port through lo, in a thread""" + + def setUp(self): + self.port = free_port() + self.logged = [] + self.backend = Backend() + self.front = Front(self.backend, + lambda: datetime.now(timezone.utc), + self.logged.append) + self.stop = threading.Event() + self.thread = threading.Thread(target=memberlink.serve, args=( + self.front, "lo", "::1", self.stop, self.port, 0.05)) + self.thread.start() + self.addCleanup(self.thread.join, 10) + self.addCleanup(self.stop.set) + for _ in range(100): + try: + socket.create_connection(("::1", self.port), 1).close() + break + except OSError: + self.stop.wait(0.02) + + def raw(self, request: bytes) -> bytes: + """What the server answers, b"" for a connection it closed""" + data = b"" + with socket.create_connection(("::1", self.port), 5) as raw: + try: + raw.sendall(request) + while chunk := raw.recv(4096): + data += chunk + except ConnectionResetError: + pass + return data + + +class TestChannel(Served): + def test_a_member_pulls_the_roster(self): + self.assertEqual(memberlink.fetch("::1", "lo", self.port), ROSTER) + self.assertEqual(self.backend.calls, [("GET", "/v1/members", + "::1")]) + + def test_a_member_announces(self): + memberlink.tell("::1", "lo", ROSTER, self.port) + self.assertEqual(self.backend.calls[0][:2], ("POST", "/v1/announce")) + + def test_refused_by_the_root_side(self): + self.backend.refuse = True + with self.assertRaises(LinkError) as raised: + memberlink.fetch("::1", "lo", self.port) + self.assertIn("not a member", str(raised.exception)) + with self.assertRaises(LinkError): + memberlink.tell("::1", "lo", ROSTER, self.port) + self.assertIn("refused GET '/v1/members' from ::1: 403", + self.logged) + + def test_what_never_reaches_the_root_side(self): + for request, words in ( + (b"GET /nope HTTP/1.1\r\nHost: x\r\nConnection: close" + b"\r\n\r\n", b"no such request"), + (f"POST /v1/announce HTTP/1.1\r\nHost: x\r\n" + f"Content-Length: {members.MAX_BODY + 1}\r\n" + "Connection: close\r\n\r\n".encode(), + b"longer than any roster"), + (b"POST /v1/announce HTTP/1.1\r\nHost: x\r\n" + b"Content-Length: x\r\nConnection: close\r\n\r\n", + b"longer than any roster")): + with self.subTest(request=request[:20]): + self.assertIn(words, self.raw(request)) + self.assertEqual(self.backend.calls, []) + + def test_a_source_refused_too_often_is_not_answered(self): + self.backend.refuse = True + for _ in range(5): + with self.assertRaises(LinkError): + memberlink.fetch("::1", "lo", self.port) + self.assertEqual(self.raw(b"GET /v1/members HTTP/1.1\r\n\r\n"), + b"") + self.assertEqual(len(self.backend.calls), 5) + + def test_headers_beyond_the_cap_are_not_answered(self): + data = self.raw(b"GET /v1/members HTTP/1.1\r\nX: " + + b"a" * 9000 + b"\r\n\r\n") + self.assertEqual(data, b"") + self.assertEqual(memberlink.fetch("::1", "lo", self.port), ROSTER) + + def test_connections_per_source_are_capped(self): + for _ in range(memberlink.PER_SOURCE): + held = socket.create_connection(("::1", self.port), 5) + self.addCleanup(held.close) + held.sendall(b"GET /v1/members HTTP/1.1\r\n") + self.stop.wait(0.2) + self.assertEqual(self.raw(b"GET /v1/members HTTP/1.1\r\n\r\n"), + b"") + + def test_a_probe_raises_nothing(self): + memberlink.touch("::1", "lo", self.port) + memberlink.touch("::1", "lo", free_port()) + + +class TestDeadline(unittest.TestCase): + def test_a_request_read_too_slowly_is_dropped(self): + with mock.patch("keel.mesh.memberlink.REQUEST_DEADLINE", 0.3): + test = TestChannel("test_a_member_pulls_the_roster") + test.setUp() + try: + self.assertEqual(test.raw(b"GET /v1/members HTTP/1.1\r\n"), + b"") + finally: + test.doCleanups() + + +class TestSlots(unittest.TestCase): + def test_a_slot_per_connection_one_per_source(self): + slots = memberlink.Slots() + taken = [slots.take(f"fd00::{n}") for n in range(memberlink.SLOTS)] + self.assertTrue(all(taken)) + self.assertIsNone(slots.take("fd00::99")) + taken[0]() + self.assertIsNotNone(slots.take("fd00::1")) + self.assertIsNone(slots.take("fd00::0")) + other = memberlink.Slots() + for _ in range(memberlink.PER_SOURCE): + self.assertIsNotNone(other.take("fd00::1")) + self.assertIsNone(other.take("fd00::1")) + + +class Raw(threading.Thread): + """A server that answers one connection with `reply`, as given""" + + def __init__(self, reply: bytes): + super().__init__() + self.reply = reply + self.server = socket.socket(socket.AF_INET6) + self.server.bind(("::1", 0)) + self.server.listen(1) + self.port = self.server.getsockname()[1] + self.start() + + def run(self): + with self.server: + raw, _ = self.server.accept() + with raw: + raw.recv(65536) + raw.sendall(self.reply) + + +def http(body: bytes, status: str = "200 OK") -> bytes: + return (f"HTTP/1.1 {status}\r\nContent-Length: {len(body)}\r\n" + "Connection: close\r\n\r\n").encode() + body + + +class TestClient(unittest.TestCase): + def test_nothing_listens(self): + with self.assertRaises(LinkError) as raised: + memberlink.fetch("::1", "lo", free_port()) + self.assertIn("through lo", str(raised.exception)) + + def test_no_such_interface(self): + with self.assertRaises(LinkError): + memberlink.fetch("::1", "wg-none", free_port()) + + def test_a_malformed_roster(self): + server = Raw(http(b"{}")) + with self.assertRaises(LinkError) as raised: + memberlink.fetch("::1", "lo", server.port) + self.assertIn("malformed", str(raised.exception)) + server.join(5) + + def test_an_answer_longer_than_any_roster(self): + server = Raw(http(b"x" * (members.MAX_BODY + 1))) + with self.assertRaises(LinkError) as raised: + memberlink.fetch("::1", "lo", server.port) + self.assertIn("longer than any roster", str(raised.exception)) + server.join(5) + + def test_not_an_http_answer(self): + server = Raw(b"garbage\r\n\r\n") + with self.assertRaises(LinkError) as raised: + memberlink.fetch("::1", "lo", server.port) + self.assertIn("did not answer", str(raised.exception)) + server.join(5) + + def test_an_announcement_not_taken(self): + server = Raw(http(b'{"error": "busy"}', "409 Conflict")) + with self.assertRaises(LinkError) as raised: + memberlink.tell("::1", "lo", ROSTER, server.port) + self.assertIn("refused: busy", str(raised.exception)) + server.join(5) + + +class TestIndex(unittest.TestCase): + def test_an_interface_s_index_or_none(self): + self.assertEqual(memberlink.index("lo"), socket.if_nametoindex("lo")) + self.assertIsNone(memberlink.index("wg-none")) + + +class TestServeLoop(unittest.TestCase): + """The socket follows the overlay: bound once it exists, again when + the interface comes back as another""" + + def front(self): + return Front(Backend(), lambda: datetime.now(timezone.utc), + lambda line: None) + + def test_bound_when_the_interface_appears_and_again_when_it_changes( + self): + stop = threading.Event() + indexes = iter([None, 1, 1, 2, 2]) + + def index(iface): + value = next(indexes, 2) + if value == 2 and not stop.is_set(): + threading.Timer(0.3, stop.set).start() + return value + with mock.patch("keel.mesh.memberlink.index", side_effect=index): + memberlink.serve(self.front(), "lo", "::1", stop, free_port(), + 0.02) + self.assertTrue(stop.is_set()) + + def test_an_address_that_cannot_be_bound_yet(self): + stop = threading.Event() + threading.Timer(0.2, stop.set).start() + memberlink.serve(self.front(), "lo", "fd00:6b65:9::1", stop, + free_port(), 0.01) + self.assertTrue(stop.is_set()) + + def test_a_socket_that_fails_is_bound_again(self): + stop = threading.Event() + real = memberlink.bound + made = [] + + class Failing: + def __init__(self, inner): + self.inner = inner + + def settimeout(self, value): + self.inner.settimeout(value) + + def accept(self): + raise OSError(9, "Bad file descriptor") + + def close(self): + self.inner.close() + + def bound(iface, address, number): + made.append(1) + if len(made) > 1: + stop.set() + return Failing(real(iface, address, number)) + with mock.patch("keel.mesh.memberlink.bound", side_effect=bound): + memberlink.serve(self.front(), "lo", "::1", stop, free_port(), + 0.01) + self.assertEqual(len(made), 2) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_mesh_members.py b/tests/test_mesh_members.py new file mode 100644 index 0000000..81161ab --- /dev/null +++ b/tests/test_mesh_members.py @@ -0,0 +1,125 @@ +# Copyright (c) 2026 KeelLinux maintainers +"""keel.mesh.members: the roster members send each other, and the peers +a node takes from it, pure""" + +import json +import unittest + +from mesh_helpers import INVITER, JOINER, OTHER + +from keel.mesh import members +from keel.mesh.protocol import Admission, Peer, ProtocolError, Removal + +DOC = {"version": 1, "network": {"overlay": {"wireguard": { + "address": "fd00:6b65:1::3/64", + "peers": [{"public_key": INVITER, "endpoint": "[2001:db8:1::10]:51821", + "allowed_ips": ["fd00:6b65:1::1/128"]}]}}}} + + +FOURTH = "AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8=" +EVIDENCE = Admission( + mesh_id=bytes(range(16)).hex(), invite_id="", public_key=OTHER, + sign_key=FOURTH, address="fd00:6b65:1::7", + endpoint="[2001:db8:3::30]:51820", time=5, by=FOURTH, + signature="A" * 86 + "==") +GONE = Removal(bytes(range(16)).hex(), JOINER, 5, FOURTH, "A" * 86 + "==") + + +def roster(**changed) -> members.Roster: + values = dict(identity=bytes(range(16)), public_key=INVITER, + sign_key=FOURTH, address="fd00:6b65:1::1", + members=(Peer(OTHER, "[2001:db8:3::30]:51820", + "fd00:6b65:1::7", EVIDENCE), + Peer(JOINER, None, "fd00:6b65:1::3")), + removed=(GONE,)) + values.update(changed) + return members.Roster(**values) + + +class TestRoster(unittest.TestCase): + def test_round_trip(self): + for one in (roster(), roster(identity=None, members=(), + removed=())): + with self.subTest(one=one): + self.assertEqual(members.loads(members.dumps(one)), one) + + def test_the_wire_form(self): + data = json.loads(members.dumps(roster())) + self.assertEqual(data["identity"], bytes(range(16)).hex()) + self.assertEqual(data["members"][0]["address"], "fd00:6b65:1::7") + + def test_malformed(self): + good = json.loads(members.dumps(roster())) + for name, value in (("identity", "zz"), ("identity", "00"), + ("identity", 5), ("public_key", "x"), + ("sign_key", "x"), ("removed", [{}]), + ("removed", {}), + ("address", "fd00::1/64"), ("members", {}), + ("members", [{"public_key": OTHER}]), + ("members", [5])): + with self.subTest(name=name, value=value), \ + self.assertRaises(ProtocolError): + members.loads(json.dumps({**good, name: value}).encode()) + with self.assertRaises(ProtocolError): + members.loads(b"[]") + with self.assertRaises(ProtocolError): + members.loads(b"{" * 100) + + def test_too_many_members_or_tombstones(self): + many = tuple(Peer(OTHER, None, f"fd00:6b65:1::{n:x}") + for n in range(members.MAX_MEMBERS + 1)) + with self.assertRaises(ProtocolError): + members.loads(members.dumps(roster(members=many))) + with self.assertRaises(ProtocolError): + members.loads(members.dumps(roster( + removed=(GONE,) * (members.MAX_REMOVED + 1)))) + + +class TestWithMembers(unittest.TestCase): + def test_new_members_become_peers(self): + after, added = members.with_members( + DOC, (Peer(OTHER, "[2001:db8:3::30]:51820", "fd00:6b65:1::7"), + Peer(FOURTH, None, "fd00:6b65:1::9")), JOINER) + self.assertEqual([one.public_key for one in added], [OTHER, FOURTH]) + peers = after["network"]["overlay"]["wireguard"]["peers"] + self.assertEqual(peers[1:], [ + {"public_key": OTHER, "endpoint": "[2001:db8:3::30]:51820", + "allowed_ips": ["fd00:6b65:1::7/128"]}, + {"public_key": FOURTH, "allowed_ips": ["fd00:6b65:1::9/128"]}]) + self.assertEqual(len(DOC["network"]["overlay"]["wireguard"]["peers"]), + 1, "the document read is not changed") + + def test_what_is_never_taken(self): + refused = (Peer(JOINER, None, "fd00:6b65:1::8"), # its own key + Peer(INVITER, None, "fd00:6b65:1::9"), # a known key + Peer(OTHER, None, "fd00:6b65:1::1"), # a taken address + Peer(OTHER, None, "fd00:6b65:1::3"), # its own address + Peer(OTHER, None, "fd00:9999::1")) # another prefix + after, added = members.with_members(DOC, refused, JOINER) + self.assertEqual(added, ()) + self.assertIs(after, DOC) + + def test_one_key_or_one_address_once(self): + _, added = members.with_members( + DOC, (Peer(OTHER, None, "fd00:6b65:1::7"), + Peer(OTHER, None, "fd00:6b65:1::8"), + Peer(FOURTH, None, "fd00:6b65:1::7")), JOINER) + self.assertEqual(added, (Peer(OTHER, None, "fd00:6b65:1::7"),)) + + def test_a_peer_routed_a_whole_prefix_takes_it(self): + doc = {"network": {"overlay": {"wireguard": { + "address": "fd00:6b65:1::3/64", "peers": [ + {"public_key": INVITER, + "allowed_ips": ["fd00:6b65:1::/112"]}]}}}} + _, added = members.with_members( + doc, (Peer(OTHER, None, "fd00:6b65:1::7"),), JOINER) + self.assertEqual(added, ()) + + def test_no_overlay_takes_nothing(self): + after, added = members.with_members( + {"version": 1}, (Peer(OTHER, None, "fd00:6b65:1::7"),), JOINER) + self.assertEqual((after, added), ({"version": 1}, ())) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_mesh_netns.py b/tests/test_mesh_netns.py index d6709d4..01e0c2c 100644 --- a/tests/test_mesh_netns.py +++ b/tests/test_mesh_netns.py @@ -7,7 +7,9 @@ wg and wg-quick, which need the wireguard kernel module. Three nodes joined by veths: B joins A through A's listener, C through the fallback and `keel mesh accept`, both ping A over the overlay with no `keel -network confirm` typed, the spent invite is refused, no secret reaches +network confirm` typed, A announces C to B and C pulls B from A, so B +and C are peers and ping each other (a full mesh), the spent invite is +refused, no secret reaches A's journal, and the listener that faced the network held no capability. In CI a missing tool or namespace fails instead of skipping. @@ -99,14 +101,32 @@ def test_c_joins_through_the_fallback(self): self.assertEqual(self.found["peers"], [None, 25]) self.assertEqual(self.found["invites_left"], []) + def test_the_mesh_is_full(self): + """C joined through A alone: A announced it to B, and C pulled B + from A; B and C are each other's peers, confirmed, over the + overlay (decision 0048, "Until etcd exists")""" + b, c, checked = (self.found["b"], self.found["c"], + self.found["b_after_c"]) + self.assertIn(c["key"], checked["peers"], checked) + self.assertEqual(checked["outcome"], "confirmed", checked) + self.assertTrue(checked["ping"], checked) + self.assertIn(b["key"], c["peers"], c) + self.assertTrue(c["ping_other"], c) + self.assertIn(f"announced {c['key']} to 1 of the 1 other member(s)", + "\n".join(self.found["journal"])) + self.assertIn("an announcement from", "\n".join(checked["log"])) + self.assertIn("learning the other members from the inviter…", + c["err"]) + def test_a_spent_invite_is_refused(self): code, said = self.found["again"] self.assertEqual(code, exits.MESH_REFUSED) self.assertIn("no pending invite", said[0]) def test_the_listener_held_no_capability(self): + # the two invites' listeners, and A's members' listener self.assertEqual(self.found["capabilities"], - ["0000000000000000"] * 2) + ["0000000000000000"] * 3) self.assertEqual(self.found["netem"], self.netem.split()) def test_no_secret_in_the_inviter_s_journal(self): diff --git a/tests/test_mesh_protocol.py b/tests/test_mesh_protocol.py index 3aa0141..fb0e8d3 100644 --- a/tests/test_mesh_protocol.py +++ b/tests/test_mesh_protocol.py @@ -12,6 +12,7 @@ from keel.mesh import protocol from keel.mesh.protocol import ( + Admission, ConfirmAnswer, ConfirmRequest, JoinAnswer, @@ -25,16 +26,23 @@ INVITER = "nb9/izIukqWXM7gnBpe7hki4jKZZChWOW1wEfONn82E=" NOW = datetime(2026, 10, 3, 12, 0, tzinfo=timezone.utc) NONCE = "00112233445566778899aabbccddeeff" +SIGNER = "AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8=" REQUEST = JoinRequest( invite_id="0123456789abcdef", public_key=JOINER, endpoint="[2001:db8:2::20]:51820", address="fd00:6b65:1::3/64", - nonce=NONCE, time=int(NOW.timestamp())) + nonce=NONCE, time=int(NOW.timestamp()), sign_key=SIGNER) +EVIDENCE = Admission( + mesh_id="00" * 16, invite_id="0123456789abcdef", public_key=JOINER, + sign_key=SIGNER, address="fd00:6b65:1::3", endpoint=None, time=5, + by=SIGNER, signature="A" * 86 + "==") ANSWER = JoinAnswer( invite_id="0123456789abcdef", nonce=NONCE, public_key=INVITER, address="fd00:6b65:1::1/64", peers=(Peer(public_key=JOINER, endpoint=None, - address="fd00:6b65:1::7"),), - etcd="none", window=120) + address="fd00:6b65:1::7"), + Peer(JOINER, "[2001:db8::7]:51820", "fd00:6b65:1::3", + EVIDENCE),), + etcd="none", window=120, sign_key=SIGNER, admission=EVIDENCE) def body(message, **changed): @@ -70,13 +78,50 @@ def test_the_answer_is_signed_apart_from_the_request(self): self.assertNotEqual(request, answer) +class TestEvidence(unittest.TestCase): + def test_what_is_signed(self): + self.assertEqual(EVIDENCE.message(), ( + b"keel mesh admission 1\n" + b"00" * 16 + b"\n0123456789abcdef" + + f"\n{JOINER}\n{SIGNER}\nfd00:6b65:1::3\n\n5\n{SIGNER}" + .encode())) + gone = protocol.Removal("00" * 16, JOINER, 5, SIGNER, "x") + self.assertEqual(gone.message(), ( + b"keel mesh removal 1\n" + b"00" * 16 + + f"\n{JOINER}\n5\n{SIGNER}".encode())) + + def test_round_trips(self): + data = json.loads(protocol.dumps(EVIDENCE)) + self.assertEqual(protocol.admission(data), EVIDENCE) + self.assertEqual(protocol.admission({**data, "invite_id": ""}), + Admission(**{**data, "invite_id": ""})) + gone = protocol.Removal("00" * 16, JOINER, 5, SIGNER, + "A" * 86 + "==") + self.assertEqual(protocol.removal(json.loads(protocol.dumps(gone))), + gone) + + def test_malformed(self): + data = json.loads(protocol.dumps(EVIDENCE)) + for changed in ({"invite_id": "x"}, {"mesh_id": "00"}, + {"signature": "A" * 88}, {"by": "x"}, + {"sign_key": 1}, {"address": "fd00::3/64"}): + with self.subTest(changed=changed), \ + self.assertRaises(ProtocolError): + protocol.admission({**data, **changed}) + for bad in ([], None, "x"): + with self.assertRaises(ProtocolError): + protocol.admission(bad) + with self.assertRaises(ProtocolError): + protocol.removal(bad) + + class TestMessages(unittest.TestCase): def test_round_trips(self): confirm = ConfirmRequest(invite_id="0123456789abcdef", public_key=JOINER, nonce=NONCE, time=int(NOW.timestamp())) confirmed = ConfirmAnswer(invite_id="0123456789abcdef", nonce=NONCE, - confirmed=True, detail="confirmed") + confirmed=True, detail="confirmed", + sign_key=SIGNER) for message, load in ( (REQUEST, protocol.join_request), (REQUEST.__class__(**{**REQUEST.__dict__, "endpoint": None}), diff --git a/tests/test_mesh_remove.py b/tests/test_mesh_remove.py new file mode 100644 index 0000000..b038a9b --- /dev/null +++ b/tests/test_mesh_remove.py @@ -0,0 +1,87 @@ +# Copyright (c) 2026 KeelLinux maintainers +"""keel.mesh.remove: a peer out of the spec, and a tombstone that keeps +every sync from adding it again""" + +import os + +from mesh_helpers import INVITER, MESH, NOW, OTHER +from mesh_sync_helpers import FOURTH, Case +from mesh_wire import Armed + +from keel import exits +from keel.mesh import identity, remove, signing, sync, trust +from keel.network import marker + + +class TestRemove(Case): + def setUp(self): + super().setUp() + self.out = [] + # OTHER is a peer, admitted by the inviter + self.assertEqual(sync.pull(self.syncer), exits.OK, self.err) + marker.clear(self.root) + self.node.apply = Armed() + self.err.clear() + # the inviter, kept, answers over the changed overlay + self.net.handshakes[INVITER] = int(NOW.timestamp()) + + def test_removed_confirmed_announced_and_never_synced_back(self): + self.assertEqual(remove.remove(self.syncer, OTHER, self.out.append), + exits.OK, self.err) + self.assertEqual([one["public_key"] for one in self.peers()], + [INVITER]) + self.assertIn("the other members drop it", self.out[0]) + self.assertEqual(marker.last(self.root).outcome, marker.CONFIRMED) + self.assertIn(f"a WireGuard handshake from {INVITER}", self.said()) + store = trust.load(self.root) + gone = store.removed[OTHER] + self.assertEqual(gone.by, self.signer) + self.assertTrue(signing.verified(gone.by, gone.message(), + gone.signature)) + self.assertIsNone(store.find(OTHER)) + # the inviter is told: the roster it gets carries the tombstone + [(host, told)] = self.net.told + self.assertEqual((host, told.removed), ("fd00:6b65:1::1", (gone,))) + # the inviter still names OTHER, with its evidence: not taken + self.assertEqual(sync.pull(self.syncer), exits.OK) + self.assertEqual(len(self.peers()), 1) + + def test_by_its_overlay_address(self): + self.assertEqual(remove.remove(self.syncer, "fd00:6b65:1::7", + self.out.append), exits.OK) + self.assertEqual(len(self.peers()), 1) + + def test_refused(self): + for which in ("fd00:6b65:1::9", "not a peer"): + self.assertEqual(remove.remove(self.syncer, which, + self.out.append), + exits.MESH_REFUSED) + self.assertIn("no peer of this node", self.err[-1]) + marker.save(self.root, "") + marker.write(self.root, marker.Pending("eth0", "x", 120)) + self.assertEqual(remove.remove(self.syncer, OTHER, self.out.append), + exits.MESH_REFUSED) + self.assertIn("a network change waits", self.err[-1]) + marker.clear(self.root) + store = trust.load(self.root) + full = trust.removal(self.root, MESH, FOURTH, NOW) + store.removed = {str(n): full + for n in range(trust.MAX_REMOVED_PER_SIGNER)} + trust.save(self.root, store) + self.assertEqual(remove.remove(self.syncer, OTHER, self.out.append), + exits.MESH_REFUSED) + self.assertIn("as many as it may", self.err[-1]) + os.remove(os.path.join(self.root, identity.IDENTITY)) + self.assertEqual(remove.remove(self.syncer, OTHER, self.out.append), + exits.MESH_REFUSED) + self.assertIn("no mesh identity", self.err[-1]) + self.assertEqual(len(self.peers()), 2) + + def test_not_confirmed_the_tombstone_is_kept(self): + self.net.handshakes = {} + self.assertEqual(remove.remove(self.syncer, OTHER, self.out.append), + exits.NETWORK_NOT_CONFIRMED) + self.assertIn("the tombstone is kept", self.err[-1]) + self.assertEqual(len(self.peers()), 2) + self.assertTrue(trust.load(self.root).gone(OTHER)) + self.assertEqual(self.net.told, []) diff --git a/tests/test_mesh_signing.py b/tests/test_mesh_signing.py new file mode 100644 index 0000000..9f790f8 --- /dev/null +++ b/tests/test_mesh_signing.py @@ -0,0 +1,80 @@ +# Copyright (c) 2026 KeelLinux maintainers +"""keel.mesh.signing: this node's Ed25519 key, with the real openssl""" + +import base64 +import os +import shutil +import stat +import subprocess +import tempfile +import unittest +from unittest import mock + +from keel.mesh import signing + + +class Case(unittest.TestCase): + def setUp(self): + self.root = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.root) + + +class TestKey(Case): + def test_made_once_root_s_alone(self): + public = signing.ensure(self.root) + self.assertEqual(len(base64.b64decode(public)), 32) + self.assertEqual(signing.ensure(self.root), public) + mode = os.stat(os.path.join(self.root, signing.KEY)).st_mode + self.assertEqual(stat.S_IMODE(mode), 0o600) + + def test_no_key_yet(self): + with self.assertRaises(signing.SigningError): + signing.public(self.root) + + def test_a_file_that_is_no_key(self): + os.makedirs(os.path.join(self.root, "var/lib/keel/mesh")) + with open(os.path.join(self.root, signing.KEY), "w") as fob: + fob.write("nope\n") + with self.assertRaises(signing.SigningError): + signing.public(self.root) + with self.assertRaises(signing.SigningError): + signing.sign(self.root, b"x") + + def test_openssl_failing_or_missing(self): + failed = subprocess.CompletedProcess([], 1, b"", b"no") + with mock.patch("keel.mesh.signing.subprocess.run", + return_value=failed), \ + self.assertRaises(signing.SigningError): + signing.ensure(self.root) + with mock.patch("keel.mesh.signing.subprocess.run", + side_effect=OSError("not found")), \ + self.assertRaises(signing.SigningError): + signing.ensure(self.root) + self.assertFalse(os.path.exists(os.path.join(self.root, + signing.KEY))) + + +class TestSignatures(Case): + def test_signed_and_verified(self): + public = signing.ensure(self.root) + signature = signing.sign(self.root, b"message") + self.assertTrue(signing.verified(public, b"message", signature)) + self.assertFalse(signing.verified(public, b"messagf", signature)) + + def test_another_key_or_garbage_is_no_signature(self): + public = signing.ensure(self.root) + signature = signing.sign(self.root, b"message") + other = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, other) + self.assertFalse(signing.verified(signing.ensure(other), b"message", + signature)) + for key, sig in ((public, "!!"), (public, "AAAA"), ("x", signature)): + self.assertFalse(signing.verified(key, b"message", sig)) + with mock.patch("keel.mesh.signing.subprocess.run", + side_effect=OSError("not found")): + self.assertFalse(signing.verified(public, b"message", + signature)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_mesh_sync.py b/tests/test_mesh_sync.py new file mode 100644 index 0000000..b0f4c87 --- /dev/null +++ b/tests/test_mesh_sync.py @@ -0,0 +1,471 @@ +# Copyright (c) 2026 KeelLinux maintainers +"""keel.mesh.sync: a full mesh before etcd (decision 0048, amended) + +A real Node on a scratch root and real Ed25519 evidence +(tests/mesh_sync_helpers.py), the members' channel replaced by +functions: what a pull adds and how it is confirmed, that nothing +without evidence a trusted key signed is ever taken, tombstones, the +trust roots bound and vouched for, the announcement, the backoff, the +announcements applied in one window, and a node without an identity +taking none. +""" + +import json +import os +import shutil +import tempfile +import unittest +from dataclasses import replace +from datetime import timedelta +from unittest import mock + +from mesh_helpers import INVITER, JOINER, MESH, NOW, OTHER +from mesh_sync_helpers import ( + FOURTH, + INVITER_ROOT, + INVITER_SIGNER, + SPLIT, + Case, + admitted, + evidence, + inviter_roster, +) +from mesh_wire import Armed, probes + +from keel import exits +from keel.mesh import identity, signing, sync, trust +from keel.mesh.members import Roster +from keel.mesh.protocol import Peer +from keel.network import marker + + +class TestPull(Case): + def test_a_member_the_inviter_knows_is_added_and_confirmed(self): + self.assertEqual(sync.pull(self.syncer), exits.OK, self.err) + self.assertEqual(self.peers()[1], { + "public_key": OTHER, "endpoint": "[2001:db8:3::30]:51820", + "allowed_ips": ["fd00:6b65:1::7/128"]}) + self.assertFalse(marker.exists(self.root)) + self.assertEqual(marker.last(self.root).outcome, marker.CONFIRMED) + self.assertIn("confirming over the overlay…", self.err) + self.assertIn(f"the overlay was tested: a WireGuard handshake from" + f" {OTHER}", self.said()) + + def test_the_handshake_is_waited_for_and_asked_for(self): + self.net.handshakes = {} + + def later(seconds): + self.later(seconds) + if len(self.slept) == 2: + self.net.handshakes[OTHER] = int(self.clock().timestamp()) + self.syncer.sleep = later + self.assertEqual(sync.pull(self.syncer), exits.OK, self.err) + self.assertEqual(self.net.touched, ["fd00:6b65:1::7"] * 2) + + def test_a_handshake_from_before_the_change_is_no_proof(self): + self.net.handshakes = {OTHER: int(NOW.timestamp()) - 1} + self.node.apply = SayingArmed() + self.assertEqual(sync.pull(self.syncer), exits.NETWORK_NOT_CONFIRMED) + self.assertTrue(marker.exists(self.root)) + self.assertIn("unless `keel network confirm`", self.said()) + self.assertIn("no member completed a WireGuard handshake", + self.err[-1]) + # within the window, retried every RETRY seconds + self.assertLessEqual(sum(self.slept), 120) + + def test_a_member_that_never_answered_waits_an_hour(self): + self.net.handshakes = {} + sync.pull(self.syncer) + # the spec is put back as the revert puts the overlay back + self.assertEqual(len(self.peers()), 1) + marker.clear(self.root) + self.node.apply = Armed() + self.clock.now = NOW + timedelta(minutes=30) + self.assertEqual(sync.pull(self.syncer), exits.OK) + self.assertEqual(len(self.peers()), 1) + self.assertIn(f"{OTHER} is tried again after", self.said()) + self.clock.now = NOW + timedelta(minutes=70) + self.net.handshakes = {OTHER: int(self.clock().timestamp())} + self.assertEqual(sync.pull(self.syncer), exits.OK, self.err) + self.assertEqual(len(self.peers()), 2) + with open(os.path.join(self.root, sync.UNREACHED)) as fob: + self.assertEqual(json.load(fob), {}) + + def test_the_route_check_refuses(self): + self.node.probes = lambda: probes("wg0") + self.assertEqual(sync.pull(self.syncer), exits.NETWORK_NOT_CONFIRMED) + self.assertIn("leaves through wg0", self.said()) + self.assertEqual(len(self.peers()), 1) + + def test_nothing_new(self): + self.net.rosters["fd00:6b65:1::1"] = inviter_roster() + self.assertEqual(sync.pull(self.syncer), exits.OK) + self.assertIn("no member this node does not know", self.err[-1]) + self.assertEqual(self.node.apply.documents, []) + + def test_no_member_answers(self): + self.net.down.add("fd00:6b65:1::1") + self.assertEqual(sync.pull(self.syncer), exits.OK) + self.assertIn("did not answer", self.err[0]) + self.assertIn("no member answered", self.err[-1]) + + def test_a_member_that_answers_as_another(self): + self.net.rosters["fd00:6b65:1::1"] = Roster( + MESH, FOURTH, INVITER_SIGNER, "fd00:6b65:1::1", ()) + self.assertEqual(sync.pull(self.syncer), exits.OK) + self.assertIn("answered as another member", self.err[0]) + + def test_from_one_member_only(self): + self.assertEqual(sync.pull(self.syncer, ("fd00:6b65:1::1",)), + exits.OK) + self.assertEqual(sync.pull(self.syncer, ("fd00:6b65:1::9",)), + exits.MESH_REFUSED) + self.assertIn("not a peer of this node", self.err[-1]) + + def test_no_mesh_and_no_peer(self): + with open(self.node.path, "w") as fob: + fob.write("version: 1\n") + self.assertEqual(sync.pull(self.syncer), exits.OK) + self.assertIn("in no mesh", self.err[-1]) + with open(self.node.path, "w") as fob: + fob.write("version: 1\nnetwork:\n overlay:\n wireguard:\n" + " address: fd00:6b65:1::3/64\n") + self.assertEqual(sync.pull(self.syncer), exits.OK) + self.assertIn("no peer yet", self.err[-1]) + + def test_a_change_waiting_defers(self): + marker.save(self.root, "") + marker.write(self.root, marker.Pending("eth0", "x", 120)) + self.assertEqual(sync.pull(self.syncer), exits.MESH_REFUSED) + self.assertIn("a network change waits", self.err[-1]) + + def test_another_sync_running(self): + with sync.sync_lock(self.root, False) as held: + self.assertTrue(held) + self.assertEqual(sync.pull(self.syncer), exits.OK) + self.assertIn("another keel mesh sync", self.err[-1]) + + def test_the_change_does_not_come_up(self): + def failing(doc, root, window): + print("wg-quick up failed") + return 16 + self.node.apply = failing + self.assertEqual(sync.pull(self.syncer), exits.APPLY_FAILED) + self.assertEqual(self.err[-2], "wg-quick up failed") + self.assertIn("put back as it was", self.err[-1]) + self.assertEqual(len(self.peers()), 1) + + def test_a_spec_that_cannot_be_put_back_is_said(self): + self.node.apply = Armed(code=16, arms=False) + real = self.node.write + calls = [] + + def write(doc): + calls.append(doc) + if len(calls) > 1: + raise sync.NodeError("read-only file system") + real(doc) + with mock.patch.object(self.node, "write", side_effect=write): + self.assertEqual(sync.pull(self.syncer), exits.APPLY_FAILED) + self.assertIn("read-only file system", self.said()) + + def test_a_spec_that_cannot_take_them(self): + with mock.patch.object(self.node, "write", + side_effect=sync.NodeError("full")): + self.assertEqual(sync.pull(self.syncer), exits.MESH_REFUSED) + self.assertEqual(self.err[-1], "full") + + def test_no_key_to_name_this_node(self): + with mock.patch("keel.mesh.node.wgkeys.public", + return_value=(None, "no key")): + self.assertEqual(sync.pull(self.syncer), exits.MESH_REFUSED) + self.assertEqual(self.err[-1], "no key") + + +class TestEvidence(Case): + """0048: only an admitted join adds a peer""" + + def taken(self, *entries, **changed) -> list: + self.net.rosters["fd00:6b65:1::1"] = inviter_roster(*entries, + **changed) + sync.pull(self.syncer) + return [one["public_key"] for one in self.peers()[1:]] + + def test_an_entry_without_evidence_is_not_taken(self): + self.assertEqual(self.taken(Peer(OTHER, None, "fd00:6b65:1::7")), []) + self.assertIn("no member this node does not know", self.err[-1]) + + def test_forged_or_mismatched_evidence_is_not_taken(self): + found = admitted() + for entry in ( + replace(found, admission=replace(found.admission, + address="fd00:6b65:1::8")), + replace(found, address="fd00:6b65:1::8"), + replace(found, admission=replace(found.admission, + signature="A" * 86 + "==")), + admitted(mesh=SPLIT)): + with self.subTest(entry=entry): + self.assertEqual(self.taken(entry), []) + + def test_evidence_by_a_key_this_node_does_not_trust(self): + stranger = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, stranger) + signing.ensure(stranger) + self.assertEqual(self.taken(admitted(root=stranger)), []) + + def test_a_chain_of_evidence(self): + """OTHER, admitted by the inviter, admitted FOURTH""" + other_root = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, other_root) + of_other = admitted(sign_key=signing.ensure(other_root)) + of_fourth = admitted(FOURTH, address="fd00:6b65:1::9", + endpoint=None, root=other_root) + self.net.handshakes[FOURTH] = int(NOW.timestamp()) + self.assertEqual(self.taken(of_fourth, of_other), [OTHER, FOURTH]) + + def test_a_tombstone_keeps_a_member_out_here_and_is_kept(self): + gone = trust.removal(INVITER_ROOT, MESH, OTHER, NOW) + self.assertEqual(self.taken(admitted(), removed=(gone,)), []) + self.assertTrue(trust.load(self.root).gone(OTHER)) + + def unbound_root(self): + store = trust.Store() + trust.make_roots(store, (INVITER,)) + trust.save(self.root, store) + + def test_a_trust_root_is_bound_from_its_own_roster_alone(self): + self.unbound_root() + self.assertEqual(self.taken(admitted()), [OTHER]) + found = trust.load(self.root).members[INVITER] + self.assertEqual(found.sign_key, INVITER_SIGNER) + # and vouched for by no one + self.assertIsNone(found.admission) + self.assertIn("trust root fd00:6b65:1::1 signs with", self.said()) + + def test_an_announcement_binds_no_root(self): + """its source is the listener's word, not a roster this node + fetched from the root's own address""" + self.unbound_root() + sync.announced(self.syncer, [("fd00:6b65:1::1", + inviter_roster(admitted()))]) + self.assertIsNone(trust.load(self.root).members[INVITER].sign_key) + self.assertEqual(len(self.peers()), 1) + + def test_a_damaged_trust_store(self): + with open(os.path.join(self.root, trust.TRUST), "w") as fob: + fob.write("x") + self.assertEqual(sync.pull(self.syncer), exits.MESH_REFUSED) + self.assertIn("damaged", self.err[-1]) + + +class TestTombstones(Case): + """0048: when a node is removed, the others drop it too""" + + def setUp(self): + super().setUp() + self.assertEqual(sync.pull(self.syncer), exits.OK, self.err) + marker.clear(self.root) + self.node.apply = Armed() + self.err.clear() + # the inviter answers over the changed overlay + self.net.handshakes[INVITER] = int(NOW.timestamp()) + + def test_the_admitter_s_tombstone_drops_the_peer_in_one_window(self): + gone = trust.removal(INVITER_ROOT, MESH, OTHER, NOW) + self.net.rosters["fd00:6b65:1::1"] = inviter_roster(removed=(gone,)) + self.assertEqual(sync.pull(self.syncer), exits.OK, self.err) + self.assertEqual([one["public_key"] for one in self.peers()], + [INVITER]) + self.assertEqual(len(self.node.apply.documents), 1) + self.assertIn(f"member {OTHER} removed", self.said()) + self.assertIn(f"a WireGuard handshake from {INVITER}", self.said()) + self.assertTrue(trust.load(self.root).gone(OTHER)) + + def test_a_tombstone_from_a_member_that_may_not_remove(self): + stranger = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, stranger) + signing.ensure(stranger) + gone = trust.removal(stranger, MESH, OTHER, NOW) + self.net.rosters["fd00:6b65:1::1"] = inviter_roster(removed=(gone,)) + sync.pull(self.syncer) + self.assertEqual(len(self.peers()), 2) + self.assertFalse(trust.load(self.root).gone(OTHER)) + + def test_the_last_peer_removed_is_confirmed_by_the_route_check(self): + store = trust.load(self.root) + trust.record_removal(store, trust.removal(self.root, MESH, OTHER, + NOW)) + trust.record_removal(store, trust.removal(self.root, MESH, INVITER, + NOW)) + trust.save(self.root, store) + self.net.rosters["fd00:6b65:1::1"] = inviter_roster() + self.assertEqual(sync.pull(self.syncer), exits.OK, self.err) + self.assertEqual(self.peers(), []) + self.assertIn("removed the last peer", self.said()) + + def test_peers_it_keeps_that_never_answer(self): + self.net.handshakes = {} + gone = trust.removal(INVITER_ROOT, MESH, OTHER, NOW) + self.net.rosters["fd00:6b65:1::1"] = inviter_roster(removed=(gone,)) + self.assertEqual(sync.pull(self.syncer), exits.NETWORK_NOT_CONFIRMED) + # put back; the tombstone stays, and the next sync drops it again + self.assertEqual(len(self.peers()), 2) + self.assertTrue(trust.load(self.root).gone(OTHER)) + + +class TestIdentity(Case): + def test_a_node_without_one_takes_none_from_a_roster(self): + os.remove(os.path.join(self.root, identity.IDENTITY)) + self.assertEqual(sync.pull(self.syncer), exits.MESH_REFUSED) + self.assertIsNone(identity.read(self.root)) + self.assertIn("only a join, or keel mesh create --adopt", + self.err[-1]) + self.assertEqual(len(self.peers()), 1) + + def test_a_member_of_another_identity_gives_nothing(self): + identity.replace(self.root, SPLIT) + self.net.rosters["fd00:6b65:1::1"] = inviter_roster( + admitted(mesh=SPLIT), identity_of=MESH) + self.assertEqual(sync.pull(self.syncer), exits.MESH_REFUSED) + self.assertEqual(len(self.peers()), 1) + self.assertIn("keel mesh sync --adopt fd00:6b65:1::1", + self.said()) + self.net.rosters["fd00:6b65:1::1"] = inviter_roster(identity_of=None) + sync.pull(self.syncer) + self.assertIn("is in mesh none", self.said()) + + def test_a_damaged_identity(self): + with open(os.path.join(self.root, identity.IDENTITY), "w") as fob: + fob.write("x\n") + self.assertEqual(sync.pull(self.syncer), exits.MESH_REFUSED) + self.assertIn("does not hold a mesh identity", self.err[-1]) + + +class TestAnnounced(Case): + """What the members' service applies: every announcement pending, + in one change""" + + def test_all_pending_in_one_window(self): + self.net.handshakes[FOURTH] = int(NOW.timestamp()) + code = sync.announced(self.syncer, [ + ("fd00:6b65:1::1", inviter_roster(admitted())), + ("fd00:6b65:1::1", inviter_roster(admitted( + FOURTH, address="fd00:6b65:1::9", endpoint=None)))]) + self.assertEqual(code, exits.OK, self.err) + self.assertEqual(len(self.node.apply.documents), 1) + self.assertEqual(len(self.peers()), 3) + self.assertIn("an announcement from fd00:6b65:1::1: 2 member(s)", + self.err) + + def test_no_key(self): + with mock.patch("keel.mesh.node.wgkeys.public", + return_value=(None, "no key")): + self.assertEqual(sync.announced(self.syncer, [ + ("fd00:6b65:1::1", inviter_roster())]), exits.MESH_REFUSED) + self.assertEqual(self.err[-1], "no key") + + +class TestAnnounce(Case): + def setUp(self): + super().setUp() + with open(self.node.path, "a") as fob: + fob.write(f" - public_key: {OTHER}\n" + " allowed_ips: [fd00:6b65:1::7/128]\n" + f" - public_key: {FOURTH}\n" + " endpoint: '[2001:db8:4::40]:51820'\n" + " allowed_ips: [fd00:6b65:1::9/128]\n") + # FOURTH, admitted by this node + self.fourth = evidence(FOURTH, "fd00:6b65:1::9", + "[2001:db8:4::40]:51820", root=self.root) + store = trust.load(self.root) + trust.recorded(store, self.fourth) + trust.save(self.root, store) + + def test_every_other_member_is_told(self): + self.net.down.add("fd00:6b65:1::7") + sync.announce(self.syncer, FOURTH) + [(host, roster)] = self.net.told + self.assertEqual(host, "fd00:6b65:1::1") + self.assertEqual(roster.public_key, JOINER) + self.assertEqual(roster.identity, MESH) + self.assertIn(Peer(FOURTH, "[2001:db8:4::40]:51820", + "fd00:6b65:1::9", self.fourth), roster.members) + self.assertEqual(roster.sign_key, self.signer) + self.assertIn("announced", self.err[-2]) + self.assertIn("fd00:6b65:1::7 did not take it", self.err[-1]) + + def test_all_told(self): + sync.announce(self.syncer, FOURTH) + self.assertEqual(len(self.net.told), 2) + self.assertEqual(self.err, [f"announced {FOURTH} to 2 of the 2 other" + " member(s) over the overlay"]) + + def test_a_node_whose_key_cannot_be_read_announces_nothing(self): + with mock.patch("keel.mesh.node.wgkeys.public", + return_value=(None, "no key")): + sync.announce(self.syncer, FOURTH) + self.assertEqual(self.net.told, []) + self.assertIn("no key", self.err[-1]) + + +class TestEdges(Case): + def test_member_of(self): + self.assertEqual(sync.member_of(self.node, "fd00:6b65:1::1"), + INVITER) + self.assertIsNone(sync.member_of(self.node, "fd00:6b65:1::2")) + self.assertIsNone(sync.member_of(self.node, "not an address")) + with open(self.node.path, "w") as fob: + fob.write("version: [") + self.assertIsNone(sync.member_of(self.node, "fd00:6b65:1::1")) + self.assertEqual(sync.pull(self.syncer), exits.MESH_REFUSED) + + def test_a_backoff_file_keel_did_not_write(self): + for text in ("[]", '{"x": "y"}', "nope"): + with open(os.path.join(self.root, sync.UNREACHED), "w") as fob: + fob.write(text) + self.assertEqual(sync.unreached(self.root, NOW), {}) + + def test_no_other_member_to_announce_to(self): + sync.announce(self.syncer, INVITER) + self.assertEqual((self.net.told, self.err), ([], [])) + + def test_a_roster_this_node_cannot_give(self): + with mock.patch("keel.mesh.node.wgkeys.public", + return_value=(None, "no key")): + with self.assertRaises(sync.NodeError): + sync.roster(self.syncer) + with self.assertRaises(ValueError): + sync.offered(self.syncer) + os.remove(os.path.join(self.root, signing.KEY)) + with self.assertRaises(ValueError): + sync.offered(self.syncer) + + def test_where(self): + self.assertEqual(sync.where(self.node), ("wg0", "fd00:6b65:1::3")) + with open(self.node.path, "w") as fob: + fob.write("version: 1\n") + self.assertIsNone(sync.where(self.node)) + with open(self.node.path, "w") as fob: + fob.write("version: [") + self.assertIsNone(sync.where(self.node)) + + +class SayingArmed(Armed): + def __call__(self, doc, root, window): + print(f"keel mesh: bring the overlay wg0 up; it reverts in {window}" + " s unless `keel network confirm` is run from a new session") + return super().__call__(doc, root, window) + + +class TestLock(unittest.TestCase): + def test_waits_or_not(self): + root = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, root) + with sync.sync_lock(root, True) as held: + self.assertTrue(held) + with sync.sync_lock(root, False) as again: + self.assertFalse(again) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_mesh_trust.py b/tests/test_mesh_trust.py new file mode 100644 index 0000000..2587a90 --- /dev/null +++ b/tests/test_mesh_trust.py @@ -0,0 +1,241 @@ +# Copyright (c) 2026 KeelLinux maintainers +"""keel.mesh.trust: admission evidence, chains of it, roots, tombstones + +With real Ed25519 keys (openssl) in scratch roots: A admits B, B +admits C; a node that trusts A takes B, and C through B, and nothing +whose evidence is forged, for another key, mesh or address, by a key it +does not trust, or for a key with a tombstone. +""" + +import json +import os +import shutil +import tempfile +import unittest +from dataclasses import replace +from datetime import datetime, timezone + +from keel.mesh import signing, trust +from keel.mesh.protocol import Peer + +NOW = datetime(2026, 10, 3, 12, 0, tzinfo=timezone.utc) +MESH = bytes(range(16)) +WG_A = "nb9/izIukqWXM7gnBpe7hki4jKZZChWOW1wEfONn82E=" +WG_B = "FHKH10gOWeK2bXHZPg8y+oPTprv556bwrmmRkbyEPgg=" +WG_C = "9vm/AzCVQQsWt1cU2eyjom4cABkNuodiHL8nDjKohEE=" +WG_D = "AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8=" + + +def node() -> str: + root = tempfile.mkdtemp() + signing.ensure(root) + return root + + +class Case(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.a, cls.b, cls.c, cls.me = node(), node(), node(), node() + cls.sign = {name: signing.public(getattr(cls, name)) + for name in ("a", "b", "c", "me")} + cls.of_b = trust.admit(cls.a, MESH, "0123456789abcdef", WG_B, + cls.sign["b"], "fd00::b", + "[2001:db8::b]:51820", NOW) + cls.of_c = trust.admit(cls.b, MESH, "1123456789abcdef", WG_C, + cls.sign["c"], "fd00::c", None, NOW) + + @classmethod + def tearDownClass(cls): + for root in (cls.a, cls.b, cls.c, cls.me): + shutil.rmtree(root) + + def store_trusting_a(self) -> trust.Store: + return trust.Store({WG_A: trust.Member(self.sign["a"], True)}) + + def entry(self, found, **changed) -> Peer: + values = dict(public_key=found.public_key, endpoint=found.endpoint, + address=found.address, admission=found) + values.update(changed) + return Peer(**values) + + +class TestEvidence(Case): + def test_signed_by_the_admitting_member(self): + self.assertEqual(self.of_b.by, self.sign["a"]) + self.assertTrue(signing.verified(self.of_b.by, self.of_b.message(), + self.of_b.signature)) + + def test_a_chain_is_followed_whatever_the_order(self): + store = self.store_trusting_a() + found = trust.accepted(store, self.sign["me"], MESH, ( + self.entry(self.of_c), self.entry(self.of_b))) + self.assertEqual({one.public_key for one in found}, {WG_B, WG_C}) + self.assertEqual(store.members[WG_C].sign_key, self.sign["c"]) + self.assertEqual(store.evidence(WG_B), self.of_b) + # the first evidence is kept + later = trust.admit(self.a, MESH, "2123456789abcdef", WG_B, + self.sign["b"], "fd00::b", None, NOW) + trust.accepted(store, self.sign["me"], MESH, (self.entry(later),)) + self.assertEqual(store.evidence(WG_B), self.of_b) + self.assertIsNone(trust.Store().evidence(WG_B)) + + def test_what_is_left_out(self): + forged = replace(self.of_b, address="fd00::99") + resigned = replace(self.of_b, signature=self.of_c.signature) + cases = ( + self.entry(self.of_b, admission=None), + self.entry(self.of_b, public_key=WG_D), + self.entry(self.of_b, address="fd00::99"), + self.entry(forged), + self.entry(resigned), + self.entry(replace(self.of_b, mesh_id=bytes(16).hex())), + self.entry(self.of_c)) # by B, whom nothing trusts here + for one in cases: + with self.subTest(one=one): + store = self.store_trusting_a() + self.assertEqual(trust.accepted(store, self.sign["me"], + MESH, (one,)), ()) + self.assertIsNone(store.find(WG_B)) + + def test_a_tombstone_keeps_a_key_out(self): + store = self.store_trusting_a() + gone = trust.removal(self.a, MESH, WG_B, NOW) + self.assertEqual(trust.removals(store, self.sign["me"], MESH, + (gone,)), (WG_B,)) + self.assertEqual(trust.accepted(store, self.sign["me"], MESH, + (self.entry(self.of_b),)), ()) + + def test_evidence_without_an_invite_is_no_admission(self): + store = self.store_trusting_a() + vouched = trust.admit(self.a, MESH, "", WG_B, self.sign["b"], + "fd00::b", None, NOW) + self.assertEqual(trust.accepted(store, self.sign["me"], MESH, + (self.entry(vouched),)), ()) + + def chain(self) -> trust.Store: + """A root, B admitted by A, C admitted by B, all known""" + store = self.store_trusting_a() + trust.accepted(store, self.sign["me"], MESH, ( + self.entry(self.of_b), self.entry(self.of_c))) + return store + + def test_who_may_remove_a_node(self): + """its admitter, a root, the node itself, or this node; no other + member, trusted or not""" + for remover, key, taken in ( + ("b", WG_C, True), # B admitted C + ("a", WG_C, True), # A is a root + ("c", WG_C, True), # C leaves + ("me", WG_C, True), # this node + ("c", WG_B, False), # C did not admit B + ("b", WG_D, False)): # nobody knows D's admitter + with self.subTest(remover=remover, key=key): + store = self.chain() + found = trust.removals(store, self.sign["me"], MESH, ( + trust.removal(getattr(self, remover), MESH, key, NOW),)) + self.assertEqual(found, (key,) if taken else ()) + store = self.store_trusting_a() + # a root removes a node this node never knew + self.assertEqual(trust.removals(store, self.sign["me"], MESH, ( + trust.removal(self.a, MESH, WG_D, NOW),)), (WG_D,)) + + def test_tombstones_that_are_not_taken(self): + store = self.store_trusting_a() + good = trust.removal(self.a, MESH, WG_B, NOW) + for one in (replace(good, signature=self.of_b.signature), + replace(good, mesh_id=bytes(16).hex())): + self.assertEqual(trust.removals(store, self.sign["me"], MESH, + (one,)), ()) + self.assertEqual(trust.removals(store, self.sign["me"], MESH, + (good, good)), (WG_B,)) + + def test_the_caps(self): + good = trust.removal(self.a, MESH, WG_C, NOW) + full = self.store_trusting_a() + full.removed = {str(n): replace(good, by=f"x{n}") + for n in range(trust.MAX_REMOVED)} + self.assertEqual(trust.removals(full, self.sign["me"], MESH, + (good,)), ()) + per_signer = self.store_trusting_a() + per_signer.removed = { + str(n): good for n in range(trust.MAX_REMOVED_PER_SIGNER)} + self.assertFalse(trust.room_for(per_signer, self.sign["a"])) + self.assertTrue(trust.room_for(per_signer, self.sign["b"])) + self.assertEqual(trust.removals(per_signer, self.sign["me"], MESH, + (good,)), ()) + + def test_a_removed_member_is_trusted_no_more(self): + store = self.store_trusting_a() + store.members[WG_B] = trust.Member(self.sign["b"]) + trust.removals(store, self.sign["me"], MESH, + (trust.removal(self.a, MESH, WG_B, NOW),)) + self.assertNotIn(self.sign["b"], store.signers(self.sign["me"])) + + +class TestKnownMembers(Case): + def test_a_root_without_a_key_takes_the_one_its_evidence_names(self): + store = self.store_trusting_a() + trust.make_roots(store, (WG_B,)) + found = trust.accepted(store, self.sign["me"], MESH, ( + self.entry(self.of_c), self.entry(self.of_b))) + self.assertEqual(len(found), 2) + self.assertEqual(store.members[WG_B].sign_key, self.sign["b"]) + self.assertEqual(store.members[WG_B].admission, self.of_b) + + def test_evidence_naming_another_key_trusts_it_not(self): + store = self.store_trusting_a() + store.members[WG_B] = trust.Member(self.sign["me"]) + found = trust.accepted(store, self.sign["me"], MESH, ( + self.entry(self.of_c), self.entry(self.of_b))) + # B is evidenced, but C, signed by the key B was not known by, + # is not taken + self.assertEqual([one.public_key for one in found], [WG_B]) + self.assertEqual(store.members[WG_B].sign_key, self.sign["me"]) + self.assertIsNone(store.members[WG_B].admission) + + +class TestRoots(Case): + def test_a_root_is_bound_to_the_key_it_gives_once(self): + store = trust.Store() + trust.make_roots(store, (WG_A,)) + self.assertTrue(trust.bind_root(store, WG_A, self.sign["a"])) + self.assertFalse(trust.bind_root(store, WG_A, self.sign["b"])) + self.assertEqual(store.members[WG_A].sign_key, self.sign["a"]) + self.assertFalse(trust.bind_root(store, WG_B, self.sign["b"])) + + def test_a_known_member_made_a_root(self): + store = trust.Store({WG_B: trust.Member(self.sign["b"])}) + trust.make_roots(store, (WG_B,)) + self.assertTrue(store.members[WG_B].root) + self.assertFalse(trust.bind_root(store, WG_B, self.sign["c"])) + + +class TestStore(Case): + def test_round_trip(self): + root = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, root) + self.assertEqual(trust.load(root), trust.Store()) + store = self.store_trusting_a() + trust.accepted(store, self.sign["me"], MESH, (self.entry(self.of_b),)) + trust.removals(store, self.sign["me"], MESH, + (trust.removal(self.a, MESH, WG_C, NOW),)) + trust.save(root, store) + self.assertEqual(trust.load(root), store) + self.assertEqual(os.stat(os.path.join(root, trust.TRUST)).st_mode + & 0o777, 0o600) + + def test_damaged(self): + root = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, root) + os.makedirs(os.path.join(root, "var/lib/keel/mesh")) + for text in ("x", "[]", json.dumps({"members": {"k": {}}, + "removed": {}})): + with open(os.path.join(root, trust.TRUST), "w") as fob: + fob.write(text) + with self.assertRaises(ValueError) as raised: + trust.load(root) + self.assertIn("damaged", str(raised.exception)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_network_mesh_confirm.py b/tests/test_network_mesh_confirm.py index 5c98d29..a24687e 100644 --- a/tests/test_network_mesh_confirm.py +++ b/tests/test_network_mesh_confirm.py @@ -65,7 +65,7 @@ def test_the_join_session_over_the_overlay_confirms(self): def test_create_confirms_a_mesh_with_no_peer_itself(self): (confirmed, lines), _ = self.confirm(own(), clients=("2001:db8:9::5",)) self.assertTrue(confirmed, lines) - self.assertIn("no peer yet", lines[1]) + self.assertIn("has no peer, so", lines[1]) self.assertIn("still leave through the uplink", lines[1]) def test_only_the_change_keel_mesh_made(self): @@ -121,7 +121,7 @@ class TestLines(unittest.TestCase): def test_overlay_lines_of_each_origin(self): self.assertEqual(netconfirm.overlay_lines( pending(), own(), probes()), [ - "the overlay has no peer yet, so nothing can cross it:" + "the overlay has no peer, so nothing can cross it:" " keel mesh create confirmed it once the routes to the" " gateways and to the operator's session were found to" " still leave through the uplink"])