feat: Phase A, the API service, membership and WireGuard key exchange

.github/workflows/packages.yml

@@ -0,0 +1,47 @@
+# The three packages of debian/ (handbook decision 0039): built, source
+# and binary, with dpkg-buildpackage and linted with lintian, any error or
+# warning failing, on trixie, the release they are for. Then
+# tests/package-smoke.sh installs python3-keel-cloud and keel-cloud-api in
+# the same container, whose systemd runs, and checks the service: its own
+# user, [::]:8443, TLS only. The check is "build / trixie".
+#
+# keel-overlay-cloud is built and linted here but not installed: it
+# depends on keel, which is in the Keel repository and not in Debian. The
+# whole path, two nodes and the service, is the real test of README.md.
+#
+# Nothing is published here: publishing is the maintainer's attended
+# release (Keel-Linux/apt). The job runs in a Debian trixie system
+# container on the self-hosted runner through lxc-trixie.yml, as every
+# Keel package does; no application containers.
+name: packages
+
+on:
+  pull_request:
+  push:
+    branches: [master]
+
+permissions:
+  contents: read
+
+jobs:
+  build:
+    uses: keel-linux/.github/.github/workflows/lxc-trixie.yml@main
+    with:
+      timeout: 20
+      systemd: true
+      artifact-dir: dist
+      artifact-name: keel-cloud-debs
+      run: |
+        apt-get update -qq
+        apt-get install -y -qq --no-install-recommends \
+          ca-certificates curl build-essential debhelper dh-python \
+          pybuild-plugin-pyproject python3-all python3-setuptools lintian
+        work="$(mktemp -d)"
+        cp -a . "$work/src"
+        rm -rf "$work/src/.git"
+        (cd "$work/src" && dpkg-buildpackage -us -uc)
+        lintian --fail-on error,warning --info --display-info \
+          "$work"/keel-cloud_*.changes
+        mkdir -p dist
+        cp "$work"/*.deb "$work"/*.dsc "$work"/*.tar.* "$work"/*.changes dist/
+        bash tests/package-smoke.sh dist

.github/workflows/tests.yml

@@ -0,0 +1,35 @@
+# The tests of the keel_cloud package, with the coverage gate of 95
+# percent (pyproject.toml, fail_under; raise only). The check is
+# "coverage / trixie".
+#
+# Everything comes from Debian 13: the service's aiohttp, pytest and its
+# aiohttp and coverage plugins, and openssl for the tests' certificate.
+# Nothing is installed from PyPI. A test that cannot run fails rather than
+# skips (pyproject.toml, filterwarnings).
+#
+# The job runs in a Debian trixie system container, an unprivileged LXC
+# container on the self-hosted runner, through the reusable workflow
+# lxc-trixie.yml of Keel-Linux/.github. Keel runs no application
+# containers in CI. A pull request from a fork never reaches the
+# self-hosted runner.
+name: tests
+
+on:
+  pull_request:
+  push:
+    branches: [master]
+
+permissions:
+  contents: read
+
+jobs:
+  coverage:
+    uses: keel-linux/.github/.github/workflows/lxc-trixie.yml@main
+    with:
+      timeout: 15
+      run: |
+        apt-get update -qq
+        apt-get install -y -qq --no-install-recommends \
+          ca-certificates openssl python3 python3-aiohttp python3-yaml \
+          python3-pytest python3-pytest-aiohttp python3-pytest-cov
+        python3 -m pytest -q --cov --cov-report=term-missing

.gitignore

@@ -0,0 +1,18 @@
+__pycache__/
+*.pyc
+.coverage
+.pytest_cache/
+build/
+*.egg-info/
+.pybuild/
+debian/.debhelper/
+debian/debhelper-build-stamp
+debian/files
+debian/*.substvars
+debian/*.debhelper
+debian/*.debhelper.log
+debian/tmp/
+debian/python3-keel-cloud/
+debian/keel-cloud-api/
+debian/keel-overlay-cloud/
+dist/

README.md

@@ -8,23 +8,266 @@ every feature a node in Keel Cloud has (handbook decisions 0020 and 0046).
 Any operator can run their own instance from the same packages.
 
 The design is handbook decision 0046, "Keel Cloud, version 1 scope". This
-repository starts with its Phase A: the API service, and membership and
-WireGuard key exchange for the nodes of a set. DNS (Phase B) and the
-registry view (Phase C) come later.
+repository holds its **Phase A**: the API service, and membership and
+WireGuard key exchange for the nodes of a set. DNS with a health check
+(Phase B) and the registry view (Phase C) come later and are not here.
 
 ## Defaults pending the maintainer's decision
 
 0046 is a proposal. Its recommended answers are the working defaults
 here, each kept in one place so that changing one later is cheap:
 
-| Open question in 0046 | Working default |
-| --- | --- |
-| 1. The language | Python, with Debian 13 packages only; nothing is installed from PyPI and nothing goes into the system Python with pip |
-| 2. The scope of an API key | two scopes: an account key, which manages the account, and a narrower enrollment key, which nodes hold |
-| 3. Operator confirmation of new peers | required by default; automatic admission is a per-set choice |
-| 4. The entry secret's lifetime | one reusable entry secret per set, until the operator rotates it |
-| 6. How a node learns of changes | outbound long polling over HTTPS; Keel Cloud never connects to a node |
-| Transport | IPv6 first, TLS only |
+| Open question in 0046 | Working default | Where it lives |
+| --- | --- | --- |
+| 1. The language | Python, with Debian 13 packages only (`python3-aiohttp`, `python3-yaml`); nothing is installed from PyPI and nothing goes into the system Python with pip | `debian/control` |
+| 2. The scope of an API key | two scopes: an **account key**, which manages the account, and a narrower **enrollment key**, which nodes hold and which can only register nodes and read their sets | `keel_cloud/keys.py`, the `principal(...)` scopes in `keel_cloud/api/app.py` |
+| 3. Operator confirmation of new peers | **required by default**; automatic admission is a per-set choice (`keel cloud auto-admit on`) | `_admitted` in `keel_cloud/node/pins.py` |
+| 4. The entry secret's lifetime | **one reusable entry secret per set**, until the operator rotates it; rotation does not affect peers already pinned | `keel_cloud/proof.py` |
+| 6. How a node learns of changes | **outbound long polling** over HTTPS, at most 55 seconds a request, and a retry every 60 seconds when the service is unreachable; Keel Cloud never connects to a node | `MAX_WAIT` in `keel_cloud/api/app.py`, `WAIT` and `RETRY` in `keel_cloud/node/agent.py` |
+| Transport | **IPv6 first, TLS only**: the service listens on `[::]:8443` with TLS and has no plain HTTP listener | `conf/api.conf`, `keel_cloud/api/server.py` |
+
+Questions 5, 7 and 8 of 0046 concern Phase B or are already answered (the
+repository exists).
+
+## The packages
+
+One Debian source package (0039), three binary packages:
+
+| Package | Installed on | What it is |
+| --- | --- | --- |
+| `python3-keel-cloud` | both sides | the Python package `keel_cloud`: the node record, the entry secret proof, the key formats, the service and the agent |
+| `keel-cloud-api` | the Keel Cloud machine | `keel-cloud-api`, the service, as the systemd unit `keel-cloud-api.service` running as the system user `keel-cloud`; `keel-cloud`, its command line; `/etc/keel-cloud/api.conf` |
+| `keel-overlay-cloud` | a Keel node | `keel-cloud-node`, the node agent, also run as `keel cloud` (keel 0.16.0 or later), and `keel-cloud-node.service`, installed **disabled** |
+
+## The API, version 1
+
+Every request but the health check carries an API key as a bearer token.
+Errors are JSON, `{"error": "..."}`.
+
+| Method and path | Key | What it does |
+| --- | --- | --- |
+| `GET /v1/health` | none | `{"status": "ok", "version": ...}` |
+| `POST /v1/keys` | account | make a key, an enrollment key by default; it is returned once |
+| `GET /v1/sets` | account | the account's sets |
+| `PATCH /v1/sets/{set}` | account | `{"auto_admit": true, "proof": ...}` with the automatic admission proof, or `{"auto_admit": false}` |
+| `POST /v1/sets/{set}/peers` | enrollment or account | register or update a node: `{"record": ..., "proof": ...}`; the set is made on the first registration |
+| `GET /v1/sets/{set}/peers` | enrollment or account | the set's nodes, each with its record, its record proof, its status (`pending` or `confirmed`) and its confirmation; the set's automatic admission proof; `?since=REVISION&wait=SECONDS` is the long poll |
+| `POST /v1/sets/{set}/peers/confirm` | account | `{"public_key": ..., "confirmation": ...}`: the operator admits a pending node |
+
+The service checks the shape of the proofs and relays them; it cannot
+check or make their value. Accounts and keys are made on the instance
+with the command line, which runs as `keel-cloud` even when started as
+root:
+
+```
+keel-cloud account create acme          # prints the account key, once
+keel-cloud key create acme              # prints an enrollment key, once
+keel-cloud key revoke acme <id>
+keel-cloud peer list acme shop
+keel-cloud peer remove acme shop <public key>
+keel-cloud set auto-admit acme shop off
+```
+
+Confirming a node and turning automatic admission on need the set's entry
+secret, so they are done from a node of the set (`keel cloud confirm`,
+`keel cloud auto-admit on`), never from the instance.
+
+A node record is public information only: the WireGuard public key, the
+overlay addresses, the endpoint, the set, and the appliance, role and site
+labels (`keel_cloud/record.py`).
+
+## The trust model
+
+What the service stores (`keel_cloud/api/store.py`, SQLite): accounts, the
+**SHA-256 of each API key** and never the key, sets, and per node its
+public record and its proof. It never receives an entry secret, a node's
+private key or application data, so it cannot store them.
+
+**The entry secret proof.** Each set has an entry secret, 32 random bytes
+made by the set's first node (`keel cloud entry-secret`) and given by the
+operator to every node that joins. A node proves it holds the secret with
+an HMAC-SHA256, keyed with it, over its whole record; the service stores
+and relays the proof
+and cannot make one, for its own key or for a changed endpoint or address.
+Every node checks every record of its set against the secret before it
+admits it (`keel_cloud/node/pins.py`).
+
+**Two keys, two scopes.** The enrollment key a node holds can register
+and read; it cannot confirm a peer, change a set or make a key, so a key
+read off a node does not admit anything.
+
+**The operator confirms new peers, verifiably.** A new node is held until
+the operator runs `keel cloud confirm KEY --account-key-file FILE` on a
+node of the set. That node checks the new node's record proof, makes a
+**confirmation**, an HMAC with the entry secret over the new node's set,
+key and overlay addresses, and sends it with the account key. Every node
+checks the confirmation before it admits the peer, so the service cannot
+confirm a node by itself, and a confirmation cannot be moved to another
+key or other addresses. Automatic admission works the same way: it is
+turned on from a node (`keel cloud auto-admit on`) with a proof every node
+checks, so the service cannot turn it on either.
+
+**Peers are pinned per set, on each node.** Once admitted, a peer's key
+and overlay addresses are kept in `/var/lib/keel-cloud-node/pins.json`
+(root, 0600). Keel Cloud can then bring a newer endpoint for a pinned key,
+with a valid proof, and nothing else: it cannot replace the key, change
+its addresses, roll its record back, date it ahead, or remove it. A new
+key for a known node is a new peer, held like any other. A peer's
+addresses must lie inside this node's overlay prefixes, so a peer can
+never claim a route to anything else, and a peer the operator declared
+by hand in the spec is left as it is. `keel cloud forget KEY` removes a
+peer from the node for good: Keel Cloud cannot bring that key back. The
+service keeps matching rules on its side: a registered key keeps its
+addresses, and two nodes of a set never share one.
+
+**What a compromised Keel Cloud can and cannot do.** It can only propose
+and relay: it can withhold updates, show the operator records that are
+not real, or stop answering. Any record, confirmation or automatic
+admission it made or changed fails its proof on every node and is not
+admitted. It cannot add a peer to a set, cannot confirm one, cannot
+change a pinned peer's key or addresses, cannot read traffic between
+nodes (WireGuard), and cannot reach into a node: the agent only calls
+out, follows no redirect, and reads answers of bounded size. Admitting a
+node takes both the entry secret, which only nodes hold, and the account
+key, which only the operator holds; a leaked entry secret with a node's
+enrollment key can register a node but not admit it.
+
+**Known limits of Phase A.** Nodes of an account share enrollment keys,
+so a holder of one can register nodes in any set of the account (they
+stay held) or occupy a key or address that a real node then cannot
+register under; per-node keys and quotas are for a later phase. The
+service runs SQLite on the event loop, which is fine at Phase A's scale
+of a few writes a minute.
+
+**The node writes only its own spec.** The agent writes the admitted
+peers into `network.overlay.wireguard.peers` of this node's
+`/etc/keel/instance.yaml`, after `keel spec validate` accepts the new file,
+and keeps every peer the operator declared by hand. It never writes
+WireGuard's configuration: `keel spec apply --system` converges the spec
+and brings the overlay up under the confirmation window of decision 0018,
+and `keel network confirm` keeps it.
+
+## A node in a set
+
+The spec's `cloud` section (keel 0.16.0, docs/spec.md "cloud" in keel):
+
+```yaml
+cloud:
+  endpoint: https://cloud.example.org:8443
+  api_key:
+    file: /etc/keel/secrets/cloud_api_key          # the enrollment key, 0600
+  entry_secret:
+    file: /etc/keel/secrets/cloud_entry_secret     # the set's secret, 0600
+  set: shop
+  ca_file: /etc/keel/cloud-ca.pem                  # only for a self-signed instance
+```
+
+The node also declares its own side of the overlay,
+`network.overlay.wireguard` with its `address`, as for any Keel overlay.
+Then:
+
+```
+keel cloud entry-secret            # first node only: makes and prints the secret
+keel cloud enroll                  # register this node
+keel cloud sync                    # held until the operator confirms
+keel cloud confirm KEY --account-key-file FILE    # the operator, once per new node
+keel cloud sync                    # the confirmed peers go into the spec
+keel spec apply --system           # converge the overlay, under its window
+keel network confirm               # keep it
+keel cloud status                  # the set, and what this node admitted
+keel cloud forget KEY              # remove a peer here, for good
+```
+
+The account key file is needed only while confirming; keep it off the
+nodes otherwise.
+
+`systemctl enable --now keel-cloud-node` does the enroll and sync for good,
+with long polling; the apply and the confirmation stay the operator's.
+
+## TLS
+
+The service has no plain HTTP listener. At installation the package makes
+a **self-signed certificate for development** in `/etc/keel-cloud/tls/`
+(an ECDSA P-256 key, `root:keel-cloud` 0640), for the machine's name; to
+make one that also names the addresses nodes use:
+
+```
+/usr/libexec/keel-cloud-api/make-dev-certificate cloud.example.org 2001:db8::10
+systemctl restart keel-cloud-api
+```
+
+Nodes then trust it through `cloud.ca_file`.
+
+**A certificate from ACME, later.** For an instance on the Internet,
+`certificate` and `private_key` in `/etc/keel-cloud/api.conf` point at a
+certificate an ACME client keeps current, and a renewal hook restarts
+`keel-cloud-api`; nodes then leave `cloud.ca_file` out. 0046 has Keel
+Cloud behind Keel Web, with certificates issued by DNS-01 through its own
+Keel DNS, so port 80 is never needed; that arrives with Phase B and the
+high availability work, and is not built here.
+
+## Tests
+
+```
+apt install python3-aiohttp python3-yaml python3-pytest \
+    python3-pytest-aiohttp python3-pytest-cov openssl
+python3 -m pytest --cov
+```
+
+The gate is 95 percent of lines and branches (`pyproject.toml`); the
+suite runs the agent against the real service over TLS on `[::1]`, with
+only the machine (keel and ip) faked. CI runs only through the reusable
+workflow `lxc-trixie.yml` of Keel-Linux/.github, in a Debian trixie system
+container on the self-hosted runner:
+
+| Check | Workflow | What it runs |
+| --- | --- | --- |
+| `coverage / trixie` | `.github/workflows/tests.yml` | the suite and the coverage gate |
+| `build / trixie` | `.github/workflows/packages.yml` | `dpkg-buildpackage`, `lintian` failing on any warning, and `tests/package-smoke.sh`: the service installed, running as `keel-cloud` on `[::]:8443`, TLS only, keys stored as hashes |
+
+## The real test
+
+On 2026-10-02, on the test VM, three LXC containers made from the Keel
+Core image (step 8): `kc-api` with `keel-cloud-api`, and `kc-a` and `kc-b`
+with keel 0.16.0 and `keel-overlay-cloud`, each node given only the
+enrollment key, the set's entry secret and the instance's certificate.
+
+- The service ran as `keel-cloud`, listening on `[::]:8443`; plain HTTP
+  got no answer; the database held no key in plain text.
+- Both nodes enrolled and were held as pending. A node's enrollment key
+  was refused (403) when it tried to confirm.
+- A compromised service was simulated by editing its database: it marked
+  both nodes confirmed with a made-up confirmation and turned automatic
+  admission on with a made-up proof. Both nodes still held each other.
+- The operator confirmed each node from the other with `keel cloud
+  confirm` and the account key, removed afterwards; each node's spec then
+  gained the other as its only peer, with its endpoint, its `/128` and a
+  keepalive, and pinned it.
+- `keel spec apply --system` brought `wg0` up on each node under the
+  window, `keel network confirm` kept it and enabled `wg-quick@wg0`, both
+  nodes had a handshake, and each pinged the other over the overlay;
+  `keel diff` showed every overlay field `same`.
+- The compromised service then moved one node's endpoint and added a
+  bogus peer marked confirmed. The other node refused both, since neither
+  proof verified, left its spec unchanged and kept the overlay up; the
+  real node's next registration was accepted again.
+- `keel-cloud-node.service` ran on a node under its hardening; when the
+  other node registered a new endpoint, the long poll brought it into the
+  spec within seconds, on an outbound connection only.
+
+The containers were destroyed afterwards.
+
+## Not in Phase A
+
+- DNS with a health check (Phase B) and the registry view (Phase C).
+- An overlay manifest for `cloud` and its installation screen (0046: state
+  `ask` in every mode), and carrying the entry secret through the first
+  boot screens; today the operator writes the two secret files.
+- The shared secrets of 0041 over the tunnel, and a third node with a
+  wrong entry secret refused, which are part of 0046's criterion for
+  Phase A on built images; the unit tests cover the refusal.
+- High availability: Keel Cloud on three sites with etcd (0046), instead
+  of SQLite on one machine.
 
 ## License
 

conf/api.conf

@@ -0,0 +1,12 @@
+# Keel Cloud API service, read by keel-cloud-api and keel-cloud.
+# TLS only: there is no plain HTTP listener. The certificate the package
+# made at installation is self-signed, for development; see README.md,
+# "TLS", for a certificate from ACME.
+[api]
+listen = ::
+port = 8443
+certificate = /etc/keel-cloud/tls/cert.pem
+private_key = /etc/keel-cloud/tls/key.pem
+database = /var/lib/keel-cloud/cloud.db
+# How often a long poll looks for a change, in seconds.
+poll_interval = 1