feat: keel mesh makes a full mesh, keeps one identity, finds its endpoint

README.md

@@ -41,11 +41,13 @@ in [docs/apply.md](docs/apply.md).
 | `keel network revert` | Put back the interfaces file a pending network change replaced; what the revert timer runs. `--boot` restores the file only, for the boot unit |
 | `keel network wireguard key` | Print this node's WireGuard public key, making the key pair first when there is none (root, live system only); the private key is never printed (decision 0020, docs/spec.md) |
 | `keel network wireguard suggest-address` | Print a random unique local IPv6 address with its /64, for the first node of an overlay |
-| `keel mesh create` | On the first node: make the WireGuard mesh, a random unique local /64 with this node at `::1`, converged and confirmed by itself after the route check (decision 0048, [docs/mesh.md](docs/mesh.md)) |
+| `keel mesh create` | On the first node: make the WireGuard mesh, a random unique local /64 with this node at `::1`, converged and confirmed by itself after the route check (decision 0048, [docs/mesh.md](docs/mesh.md)). With `--adopt`, on one node of a mesh built by hand: give the mesh its identity |
 | `keel mesh invite` | Print the `keel mesh join - <<< keel1:<token>` line a new node runs to join this node's WireGuard mesh, valid one hour and once; reserve the new node's overlay address and start the invite's root helper, `keel-mesh-invite@<id>`, and its unprivileged listener, `keel-mesh-listen@<id>` ([docs/mesh.md](docs/mesh.md)) |
 | `keel mesh join` | Join the mesh with that token: the request to the inviter's port, pinned and authenticated, this node's spec written and applied, and both windows confirmed over the overlay; prints `keel mesh accept keel1a:...` for the inviter when its port cannot be reached. With `--dry-run`, print the change and apply nothing ([docs/mesh.md](docs/mesh.md)) |
 | `keel mesh accept` | On the inviter, the line a join printed when it could not reach the port ([docs/mesh.md](docs/mesh.md)) |
-| `keel mesh status` | This node's peers, their last handshakes, and the pending invites; never a secret ([docs/mesh.md](docs/mesh.md)) |
+| `keel mesh sync` | Add the members this node's peers know and it does not, on admission evidence a member it trusts signed, over the overlay, confirmed by a WireGuard handshake; run at boot and every 15 minutes by `keel-mesh-sync.timer`. With `--adopt ADDRESS`, take that member's mesh identity first: the repair of a split ([docs/mesh.md](docs/mesh.md)) |
+| `keel mesh remove` | Remove a peer from this node's spec, under the window for `keel network confirm`, and keep a signed tombstone so no `keel mesh sync` adds it again ([docs/mesh.md](docs/mesh.md)) |
+| `keel mesh status` | This node's mesh identity, its peers, their last handshakes, and the pending invites; never a secret ([docs/mesh.md](docs/mesh.md)) |
 | `keel notify` | What monit runs when a check of the `monitor` section fails, lasts or recovers: sends one message, saying what happened and what to do, to every channel the section declares (email, Telegram, ntfy, a webhook). It reads the channels from `/etc/keel/monitor.json`, which `apply --system` writes as root, never from the spec, reads the tokens from their secret files and prints no URL (decision 0021, docs/apply.md) |
 | `keel inspect` | Write a spec from the running machine, or from an offline root, and report every field with its source or why it was not inferred; secrets are never read (brief sections 5.2 and 7) |
 | `keel diff` | Report drift between the spec and the running machine, or an offline root, field by field, through the same collector `inspect` uses; secrets are never compared and their files need not exist, nothing is written (brief section 5.2) |
@@ -143,16 +145,16 @@ Defined in one place, `keel/exits.py`, and reproduced here.
 | 12 | `ASSEMBLE_FAILED` | The rootfs is not empty or cannot be created, or `tar` or `zstd` failed while extracting or packing |
 | 13 | `INSPECT_INCOMPLETE` | `inspect` wrote the spec, but a required field (hostname, fqdn, interfaces, alerts, updates) could not be inferred; or `diff` found no drift, but a declared field could not be observed. The report says which and why |
 | 14 | `DRIFT_FOUND` | `diff` found at least one declared field whose observed value differs. Drift wins over unobserved fields, so a report with both exits 14 |
-| 15 | `APPLY_NEEDS_ROOT` | The system phase (`--system`, `--system-only`) on the live system was run by a user other than root; nothing was written, not even the conf. Also `keel network wireguard key` making a key, and `keel mesh create`, `invite`, `join`, `accept` and `serve`, on the live system |
+| 15 | `APPLY_NEEDS_ROOT` | The system phase (`--system`, `--system-only`) on the live system was run by a user other than root; nothing was written, not even the conf. Also `keel network wireguard key` making a key, and `keel mesh create`, `invite`, `join`, `accept`, `serve`, `sync`, `members` and `remove`, on the live system |
 | 16 | `APPLY_FAILED` | The system phase could not make at least one change; the output names it. The conf was written, where the run writes one, and every other change was made, so the run can be repeated |
 | 17 | `CHANNEL_INVALID` | A channel pointer, or the record of the one this instance follows, does not parse or fails validation. Also a pointer signed in the future, one claiming more than 30 days of life, and a revision's archived pointer naming another revision |
 | 18 | `CHANNEL_UNVERIFIED` | A channel pointer is not signed by a key that may move a channel, or no keyring was given to check it against. A key that is revoked or expired is refused here: gpgv exits 0 for both and still prints `VALIDSIG`, so `GOODSIG` is what is required |
 | 19 | `CHANNEL_EXPIRED` | A channel pointer is past its expiry. A mirror that is stale, broken or hostile holds an appliance on an old release by not updating, so this is an error and never a warning |
 | 20 | `CHANNEL_ROLLBACK` | A channel pointer names an earlier revision than the one this instance is on; `--allow-rollback`, or naming the release and revision, is how going back is asked for |
-| 21 | `NETWORK_NOT_CONFIRMED` | `keel network confirm` refused: no change is waiting, or it was not run from a new session over the new configuration, or a console (docs/apply.md). Also `keel mesh create`, `join` and `accept` whose overlay change was not confirmed: it reverts by itself when its window ends |
+| 21 | `NETWORK_NOT_CONFIRMED` | `keel network confirm` refused: no change is waiting, or it was not run from a new session over the new configuration, or a console (docs/apply.md). Also `keel mesh create`, `join`, `accept` and `sync` whose overlay change was not confirmed: it reverts by itself when its window ends |
 | 22 | `NOTIFY_FAILED` | `keel notify` reached no channel: `/etc/keel/monitor.json` is missing or refused, declares none, or every one failed; the message then went to syslog (user.crit) and root's mailbox. One channel that takes the message is success; the others' failures are printed |
 | 23 | `MESH_TOKEN_INVALID` | `keel mesh join` could not use the token: not a `keel1:` token, mistyped or truncated (its checksum), of a later format, inconsistent, or expired; or `keel mesh accept` could not read its `keel1a:` line. The message never holds the token or its secret |
-| 24 | `MESH_REFUSED` | `keel mesh` refused: no overlay to invite into, no key yet, no endpoint, no free address in the prefix, another invite pending on the port; this node is in another mesh, at another address of it, or the change would make a spec validation refuses; a network change already waits in its window; the inviter refused the join (used, expired, bad HMAC), or the certificate is not the pinned one; no route to the inviter, or neither node can reach the other |
+| 24 | `MESH_REFUSED` | `keel mesh` refused: no overlay to invite into, no key yet, no endpoint, no free address in the prefix, another invite pending on the port; this node is in another mesh, at another address of it, or the change would make a spec validation refuses; a network change already waits in its window; the inviter refused the join (used, expired, bad HMAC), or the certificate is not the pinned one; no route to the inviter, or neither node can reach the other; an invite whose peers hold another mesh identity, or none; a sync from members of another identity |
 
 Two rules that callers depend on:
 

debian/changelog

@@ -1,3 +1,68 @@
+keel (0.18.0) unstable; urgency=medium
+
+  * keel mesh makes a full mesh before etcd (handbook decision 0048,
+    "Until etcd exists"). A test on three Keel Web nodes across two
+    sites found a star: a node that joined through one member was never
+    learned by the others. The join's answer already named the members
+    the inviter knows; the new node now takes them as peers in its
+    first change. The inviter, once a join is confirmed, announces the
+    new node to each of its other peers over the overlay, and each adds
+    it under the window of decision 0018, confirmed by a WireGuard
+    handshake from the new node. A member offline then, and a node that
+    joined through the fallback, pull the members with the new
+    keel mesh sync, which keel-mesh-sync.timer runs at boot and every
+    15 minutes. A member that never answered is tried again after an
+    hour, and announcements waiting are applied in one change, under
+    one window.
+  * Only an admitted join adds a peer. Each node has an Ed25519 signing
+    key made on the machine; an inviter signs evidence of each
+    admission (the mesh, the invite, the node's keys, address and
+    endpoint, the time), sent in the join's authenticated answer and
+    with the node's entry in every roster. A member takes a roster's
+    entry only on evidence of an invite signed by a key it already
+    trusts: its own, its inviter's, a member admitted the same way, or
+    a trust root that keel mesh create --adopt or keel mesh sync
+    --adopt made of the spec's peers, the operator's act for a mesh
+    built by hand, whose signing key is bound only from a roster the
+    node fetched from the root itself. docs/mesh.md says what this
+    changes in 0048's threat model; rotating a signing key is follow-up
+    work.
+  * keel mesh remove removes a node from the mesh before etcd: a
+    tombstone signed with the node's key, the peer out of its spec
+    under the window, confirmed by a handshake from a peer it keeps,
+    and its roster sent to the others, each of which drops the peer
+    through one window when the signer may remove it (its admitter, a
+    trust root, or the node itself). Tombstones are kept for good,
+    1024 at most and 64 per signer, every roster carries them all, and
+    no sync adds a removed key again.
+  * The members' channel is split as an invite is: keel-mesh-members,
+    the root helper (started by keel-mesh-members.path once the node is
+    in a mesh), starts the listener keel-mesh-members-listen with a
+    dynamic user, no capability and a sandbox, on TCP 51821 of the
+    overlay address only, bound to the overlay's interface, so
+    WireGuard authenticates who speaks, behind the invite listener's
+    limits: a deadline per request, 4 connections and 2 per source,
+    size caps, and sources refused too often. keel's firewall accepts
+    that port on the overlay's interface only.
+  * The mesh's identity no longer splits. Only a join or the operator's
+    --adopt sets it: keel mesh invite on a node whose peers hold another
+    identity, or with peers and none of its own, is refused; keel mesh
+    create --adopt gives a mesh built by hand its identity, keel mesh
+    sync --adopt ADDRESS takes a member's, which repairs a split. keel
+    mesh status shows the identity.
+  * keel mesh invite and join find this node's endpoint on a DHCP or
+    SLAAC host: the uplink's global, stable IPv6 address (never a
+    unique local, deprecated or privacy one, nor one of wg*), and a
+    public IPv4 one, in the order of keel-core's console banner. They
+    say which, and --endpoint still overrides. With only a privacy or
+    an RFC 1918 address, invite is refused and asks for --endpoint.
+  * keel mesh join, create and accept no longer print apply's "reverts
+    in 120 s unless keel network confirm" while they confirm their own
+    change: they say "confirming over the overlay…" and the result, and
+    show apply's lines only when the change is not confirmed.
+
+ -- Marcos Mendez <mendez.foto@gmail.com>  Sat, 03 Oct 2026 23:30:00 +0000
+
 keel (0.17.0) unstable; urgency=medium
 
   * The second part of handbook decision 0048, joining the mesh with one

debian/keel.keel-mesh-members.path

@@ -0,0 +1,15 @@
+# Start the members' channel once this node is in a mesh (decision 0048).
+#
+# The mesh's identity is written by keel mesh create, join, sync and
+# create --adopt; from then on the node answers its peers over the
+# overlay: its roster for their keel mesh sync, and the announcement of
+# a node another member admitted (docs/mesh.md, "Members learn of each
+# other").
+[Unit]
+Description=Start keel's mesh members' channel once this node is in a mesh
+
+[Path]
+PathExists=/var/lib/keel/mesh/identity
+
+[Install]
+WantedBy=multi-user.target

debian/keel.keel-mesh-members.service

@@ -0,0 +1,25 @@
+# The members' channel of keel mesh (decision 0048): the root helper.
+# It holds the spec, the trust store and this node's signing key, and
+# starts the unprivileged listener as the transient unit
+# keel-mesh-members-listen (a dynamic user, no capability, the sandbox
+# of keel.mesh.bridge.LISTENER_PROPERTIES), which listens on TCP 51821
+# on this node's overlay address, bound to the overlay's interface, and
+# forwards each request over a 0600 unix socket in its 0700 runtime
+# directory, the peer checked against the unit's MainPID. The helper
+# answers a member's keel mesh sync with this node's roster, and adds
+# the members an announcement names with admission evidence it trusts,
+# under the window of decision 0018, confirmed by a WireGuard handshake
+# (docs/mesh.md). Started by keel-mesh-members.path once the node is in
+# a mesh.
+[Unit]
+Description=keel mesh: the members' channel on the overlay
+After=network-online.target
+Wants=network-online.target
+# a node in no mesh has no members to answer, also when the package's
+# installation starts the unit
+ConditionPathExists=/var/lib/keel/mesh/identity
+
+[Service]
+ExecStart=/usr/bin/keel mesh members
+Restart=on-failure
+RestartSec=10

debian/keel.keel-mesh-sync.service

@@ -0,0 +1,13 @@
+# keel mesh sync (decision 0048): add the members this node's peers
+# know and it does not, so a member that was offline when a new node
+# was announced, or a node that joined through the fallback, ends up a
+# peer of every other one. Run by keel-mesh-sync.timer, at boot and
+# every 15 minutes; a node in no mesh has nothing to sync.
+[Unit]
+Description=keel mesh: learn the members this node's peers know
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/keel mesh sync

debian/keel.keel-mesh-sync.timer

@@ -0,0 +1,11 @@
+# keel mesh sync at boot and every 15 minutes (decision 0048)
+[Unit]
+Description=keel mesh: learn the members this node's peers know, regularly
+
+[Timer]
+OnBootSec=2min
+OnUnitActiveSec=15min
+RandomizedDelaySec=1min
+
+[Install]
+WantedBy=timers.target

debian/rules

@@ -8,3 +8,5 @@ export PYTHONDONTWRITEBYTECODE=1
 override_dh_installsystemd:
 	dh_installsystemd --name=keel-network-revert --no-start
 	dh_installsystemd --name=keel-firewall --no-start
+	dh_installsystemd --name=keel-mesh-members
+	dh_installsystemd --name=keel-mesh-sync

docs/apply.md

@@ -1118,6 +1118,10 @@ drop, and these rules:
 - every `mesh` port of an enabled overlay, on the WireGuard interface
   only, and none (the step says which) while the spec declares no
   overlay;
+- with an overlay, TCP 51821 on the WireGuard interface only: the
+  members' channel of keel mesh, which `keel mesh sync` and the
+  announcement of a new node use between members ([docs/mesh.md](mesh.md),
+  "Members learn of each other");
 - with an overlay, the TCP ports of the pending mesh invites: a named
   set, `mesh_invites`, of ports with a timeout, and one rule that
   accepts TCP to a port in it. `keel mesh invite` adds its port for the
@@ -1383,6 +1387,7 @@ used survived, so:
 | SSH, over the uplink | the session started after the change, came from another machine, and arrived at an address another interface of this machine holds now. confirm says the overlay itself was not tested, and at which address a peer would test it |
 | A console, a process attached from a container's host | always, as for the uplink |
 | The mesh session of a join (decision 0048), for the overlay change that join made and no other | on the inviter, a WireGuard handshake from the new peer's key since the join (`wg show <if> latest-handshakes`) and the join's confirmation request signed with the invite's HMAC key, both checked by the root side; on the new node, the inviter's signed answer over the tunnel ([docs/mesh.md](mesh.md)) |
+| A member added by `keel mesh sync` or by an inviter's announcement (decision 0048, "Until etcd exists"), for the overlay change that added it and no other | a WireGuard handshake from the key of a member that change added, since the change (`wg show <if> latest-handshakes`), which only that member, reaching this node over the new configuration, can complete; after the route check ([docs/mesh.md](mesh.md)) |
 | `keel mesh create`, for the change that created a mesh with no peer and no other | always, after the route check: an overlay with no peer routes its own private prefix only, and no peer exists to confirm it (decision 0048, second round, point 1) |
 | Anything else, a session older than the change, a session from this machine to itself, or one arriving at an address the overlay does not declare | refused (exit 21) |