From be66caf13e1723a8c66837a4908ce99a2c40960b Mon Sep 17 00:00:00 2001 From: Ralf Anton Beier Date: Thu, 27 Aug 2026 06:47:08 +0200 Subject: [PATCH] Concept: trust-root custody and succession for v1.0 The decisions were being made incrementally across three issues and a dozen comments, which is not a form anybody can review -- including the person writing them. This states the problem, the options, the trade-offs and the questions we cannot answer ourselves, in one place, and backs it with typed artifacts rather than prose alone. Nothing here is decided. The document's own position is only about ORDER: build succession first (REQ-SUCCESSION-001), because it is what makes every other custody choice reversible. Every argument in the earlier discussion collapsed into "the first choice is forever" and pushed toward spending more to avoid being stuck -- which is a constraint we can simply remove, cheaply, with no hardware and no upstream dependency. The signing seam (REQ-SIGNERSEAM-001) lands alongside it: `wsc-dsse` already exposes `trait DsseSigner` taking arbitrary PAE bytes, and varve bypasses it by hardcoding `Ed25519DsseSigner::from_bytes`. Both v1.0 paths need that seam, so neither can be evaluated without it. DD-027 records the ordering as `proposed`, not accepted -- it goes to persona review before it goes to a decision. --- artifacts/requirements.yaml | 120 ++++++++++++++++ docs/root-custody-concept.md | 269 +++++++++++++++++++++++++++++++++++ 2 files changed, 389 insertions(+) create mode 100644 docs/root-custody-concept.md diff --git a/artifacts/requirements.yaml b/artifacts/requirements.yaml index 31a16bd0..7d2b2bae 100644 --- a/artifacts/requirements.yaml +++ b/artifacts/requirements.yaml @@ -3453,6 +3453,55 @@ artifacts: no index at all. tags: [architecture, security] + - id: DD-027 + type: design-decision + title: Succession first — make the custody choice reversible before making it + status: proposed + release: v1.0.0 + description: > + Concept under review; see `docs/root-custody-concept.md` for the full + argument, the options and the questions we cannot answer ourselves. + . + The `pulseengine` realm root exists only as a write-only CI secret: it + cannot be backed up, moved, recovered, rotated or revoked. Fixing that at + v1.0 means deciding four things at once — whether a long-lived secret + exists in CI at all, where it lives, how a root is succeeded, and how it + is backed up. + . + This decision proposes the ORDER rather than the answers. Succession + (REQ-SUCCESSION-001) is built first, because it is what makes every other + choice reversible: with it, the hardware, the validation level and even + the algorithm stop being permanent, and the v1.0 custody decision can be + made on its merits instead of on fear of being stuck. The signing seam + (REQ-SIGNERSEAM-001) is built alongside it, because a hardware key and a + keyless identity both need it and neither can be evaluated without it. + . + Only after both does the keys-versus-keyless question get decided. DD-026 + rejected keyless partly because the upstream air-gapped verifier was a + stub that failed open; that premise has since expired, so the question is + reopened rather than reversed — the structural half of DD-026 (a keyless + consumer still pins a long-lived bundle key) still stands, and what + changes is that relocating the long-lived key to something used rarely is + most of the value. + links: + - type: satisfies + target: REQ-CEREMONY-001 + - type: traces-to + target: REQ-SUCCESSION-001 + - type: traces-to + target: REQ-SIGNERSEAM-001 + - type: traces-to + target: DD-026 + fields: + rationale: > + Every custody argument we made collapsed into "the first choice is + forever", and each time it pushed toward spending more to avoid being + stuck. Succession dissolves that: it is cheap, it is entirely ours to + build, it needs no hardware and no upstream decision, and it converts + an irreversible bet into a revisable one. Deciding hardware or keyless + before it is deciding under a constraint we can simply remove. + tags: [architecture, security, process] + - id: REQ-INGEST-001 type: requirement title: varve ingests what upstream actually publishes, and records which mechanism vouched @@ -3596,6 +3645,77 @@ artifacts: rather than assumed — a release publishing a cosign-signed SHA256SUMS.txt uses rung 1 and does not depend on it. + - id: REQ-SIGNERSEAM-001 + type: requirement + title: Signing goes through a seam, so the key can live somewhere other than a file + status: draft + release: v0.31.0 + description: > + `--key ` is varve's entire key input. There is no PKCS#11 provider, + no KMS backend, no hardware path — at some moment the raw 128 characters + exist in a file or on a pipe. That is what `docs root-ceremony` designs + around, and it is why the realm root currently lives in CI as a secret + that cannot be backed up, moved or recovered (DD-027). + . + The seam already exists upstream and varve does not use it: + `wsc-dsse` exposes `trait DsseSigner { fn sign(&self, pae: &[u8]); fn + key_id(&self); }` and `DsseEnvelope::sign(payload, type, &dyn + DsseSigner)`, taking arbitrary PAE bytes. `verify.rs` bypasses it by + hardcoding `Ed25519DsseSigner::from_bytes`, so key material is raw bytes + all the way down. + . + Clauses: (1) `sign_layer_manifest` and every other signing entry point + shall take a `&dyn DsseSigner` rather than key bytes. (2) The existing + file-backed behaviour shall be preserved exactly, as a `DsseSigner` + implementation — the current signing tests are the oracle, and a + behaviour change means the refactor is wrong. (3) A hardware-backed + implementation shall be possible with no change to varve's signing call + sites and no upstream dependency. (4) The refactor shall not widen what + reads the secret: exactly the implementations that need key material get + it, and `varve verify`, `install`, `status` and every export continue to + need only the public half. + . + This is the gate for both v1.0 custody paths — a hardware-held key and a + keyless identity both need it — so it lands before either is chosen. + links: + - type: traces-to + target: DD-027 + + - id: REQ-SUCCESSION-001 + type: requirement + title: A realm root can be succeeded without stranding the consumers pinned to it + status: draft + release: v1.0.0 + description: > + varve has no key rotation. There is one root per realm and no mechanism + to succeed it: nothing signs "this new root replaces the old one", and no + consumer would check such a statement if it were produced. The documented + compromise plan is manual — publish a new realm and reach every consumer + through the channel that bootstrapped them. + . + That is honest but it has two costs. It does not scale, and a legitimate + rotation is indistinguishable from an attacker telling consumers to + re-pin. It also makes every other custody decision permanent, which + inflates them: "buy the validated device because validation cannot be + retrofitted" is only decisive if the root is forever. + . + Clauses: (1) A succession statement shall carry the new root's public key + and be signed by BOTH the old root and the new one — the old proving + authority, the new proving possession, so a mistyped key cannot enthrone + a root nobody holds. (2) It shall carry a monotonic counter, and an older + statement shall be refused, or a retired root can be replayed. (3) + Verification shall be offline and clockless, like the anti-rollback + counter it mirrors — a consumer must not need a network or a correct + clock to learn its realm has a new root. (4) Applying a succession shall + be an explicit, logged action that rewrites the pinned `trust-root`, + never a silent update: a consumer's trust anchor changing without their + knowledge is the attack this mechanism could otherwise become. (5) The + root shall support a THRESHOLD of keys, and the threshold shall be + recorded at mint time — with a single root key, whoever holds it can + redirect the realm durably, and a threshold cannot be added afterwards. + (6) A realm that has never succeeded shall behave exactly as today, so + the mechanism costs nothing until it is used. + - id: REQ-LAYERADAPT-001 type: requirement title: A realm's manifest is translated into assembler inputs by varve, exactly or not at all diff --git a/docs/root-custody-concept.md b/docs/root-custody-concept.md new file mode 100644 index 00000000..09fba08a --- /dev/null +++ b/docs/root-custody-concept.md @@ -0,0 +1,269 @@ +# Trust-root custody and succession — a concept for v1.0 + +**Status: draft, under review. Nothing here is decided.** + +This document exists because the decisions were being made incrementally across +three issues and a dozen comments, which is not a form anybody can review. It +states the problem, the options, the trade-offs, and the questions we cannot +answer ourselves, in one place. + +## Tracked artifacts + +This document is the prose behind typed artifacts in `artifacts/requirements.yaml`; +the artifacts are authoritative and this is the argument for them. + +| artifact | status | what it holds | +|---|---|---| +| **DD-027** | `proposed` | this concept — the ORDER of the decisions, not the answers | +| **REQ-SUCCESSION-001** | `draft`, v1.0.0 | §3 — a root can be succeeded without stranding consumers | +| **REQ-SIGNERSEAM-001** | `draft`, v0.31.0 | §8 step 1 — signing through `DsseSigner`, the gate for both paths | +| REQ-CEREMONY-001 | `approved`, v1.0.0 | the ceremony DD-027 satisfies | +| DD-026 | `accepted` | keyless rejected; §5 records which premise expired | +| DD-005 | `accepted` | offline anti-rollback, the counter §3 mirrors | + +Verify the trace with `rivet validate --explain DD-027`. + +Related issues: [#110] (the problem), [#112] (custody design), [sigil#268] +(upstream asks), and `varve docs root-ceremony` (what we tell users to do). + +--- + +## 1. The problem, concretely + +A varve **realm** is defined by one root public key. Every layer a consumer +accepts is accepted because it verifies against that key, which the consumer +pins by hand as 64 hex characters. There is no authority above it. + +The `pulseengine` realm's root was generated on 2026-08-07 straight into CI. Its +commit message records the whole custody model in one sentence: + +> Secret half exists only as the `VARVE_ROLLING_KEY` repo secret. + +Verified: never committed, no copy on any machine we control, and GitHub +repository secrets are write-only — nobody can read one back, by design. + +So the key that signs every layer in the realm: + +* **cannot be backed up** — there is nothing to back up from; +* **cannot be moved** — which is why the layers repository can hold the realm's + *contents* but not sign them; +* **cannot be recovered** — if that secret or that repository is lost, the realm + can never be signed again, and every pinned consumer is frozen on the layer + they already installed; +* **cannot be rotated or revoked** — varve has neither mechanism, so a leak is + as terminal as a loss. + +`varve docs root-ceremony` prescribes paper backup in two locations, split +custody, an access log and an annual read test. varve's own root has none of +them. The document was honest that the root is "provisional"; it never said +"unrecoverable", which is what provisional turned out to mean. + +**A key with a single write-only copy is not in custody.** + +## 2. What must be decided at v1.0 + +v1.0 is already the moment varve mints a real root and opens the qualified +channel (REQ-CEREMONY-001). Because minting a root is the only moment the +format, the algorithm and the custody model can change cheaply, four decisions +land together: + +1. **Does a long-lived secret exist in CI at all?** (keys vs keyless) +2. **Where does the long-lived key live?** (file, HSM, or a Sigstore bundle key) +3. **How is a root succeeded** without stranding consumers? +4. **How many backups, of what, held by whom?** + +Decision 3 is the one that makes the others reversible, so it is treated first. + +## 3. The invariant: succession makes every other choice reversible + +Today a root is forever. That single fact inflates every other decision — it is +why "buy the FIPS device because validation cannot be retrofitted" sounded +compelling, and why choosing an algorithm feels irreversible. + +The solved version of this problem is **TUF root rotation**: + +> A new root is signed by a threshold of BOTH the old and the new root keys. A +> client holding only the old root verifies the chain forward to the new one, +> with no out-of-band contact. + +For varve concretely: + +* a **succession statement** carrying the new root's public key, signed by the + old root *and* by the new root — the latter proving possession, so a typo + cannot enthrone a key nobody holds; +* `varve` verifies it against the currently pinned root and updates + `varve-realms.toml` as an explicit, logged action — never silently; +* it carries a **counter**, and an older statement is refused, or an attacker + replays a retired root; +* **thresholds are what make it safe.** With one root key, whoever holds it can + redirect the realm permanently. That is already true today for signing layers, + but succession makes redirection *durable*, so a threshold (say 2-of-3, each + key in its own device) is what stops one compromised holder carrying the realm + away. + +**Thresholds are a ceremony input.** The number of root keys is decided when the +root is minted; it cannot be added afterwards. + +If succession exists, then: the hardware choice is not permanent, the FIPS +choice is not permanent, and the algorithm choice is *nearly* not permanent +(succession can carry a new algorithm if the verifier supports both). + +## 4. Path A — a long-lived key, held properly + +Keep an ed25519 root. Move it out of CI and into hardware, under a real +ceremony. + +**What it fixes:** the key becomes backupable, restorable, and split across +custodians. #110 stops being true. + +**What it does not fix:** a long-lived secret still exists, and CI still needs +*something* to sign with. Either the HSM is reachable from a self-hosted runner +(a network path to the signing key, which is the thing an air-gapped ceremony +exists to avoid), or deposits become a manual step someone performs. + +**That tension is the honest weakness of Path A**, and it is not addressed by +better hardware. It is addressed by deciding that layer signing is a deliberate +act rather than a CI side-effect — which is what `docs root-ceremony` already +says ("use it as rarely as possible"), and which today's daily-scan-then-propose +workflow is already shaped for: the scan proposes, a human merges, and only the +merge dispatches a deposit. + +**Implementation:** entirely ours. `wsc-dsse` already exposes +`pub trait DsseSigner { fn sign(&self, pae: &[u8]) -> …; fn key_id(&self) -> … }` +and `DsseEnvelope::sign(payload, type, &dyn DsseSigner)`. varve does not use it +as a seam — `verify.rs` hardcodes `Ed25519DsseSigner::from_bytes`. Refactoring +to the trait unblocks a hardware signer with **no upstream dependency**. + +## 5. Path B — keyless + +No long-lived secret in CI. Each deposit gets an ephemeral Sigstore identity via +OIDC; verification uses Fulcio roots and a Rekor key from an offline trust +bundle. + +**What it fixes:** the thing we actually fear. A compromised CI cannot sign +anything after the compromise ends, because there is no durable key to steal. + +**What it does not fix:** the trust bundle is itself signed with a long-lived +key that every consumer must pin. In `wsc`'s own words: + +> The bundle is signed with a long-lived offline key. Devices verify the +> signature against a pre-provisioned public key before using. + +So keyless **relocates** the long-lived secret rather than abolishing it. But +relocation is close to the whole point: a bundle-signing key is used rarely, +changes rarely, and can be held under exactly the ceremony this document +describes. **The question is not "keys or no keys", it is which key has to be +online.** + +**Status of the earlier rejection.** DD-025 proposed keyless; DD-026 rejected +it, partly because wsc 0.10.0's air-gapped verifier was a stub that failed open. +**That premise expired.** Verified against 0.11.0 source: the verifier performs +certificate-chain validation to a bundle Fulcio root, leaf validity at Rekor's +`integrated_time`, mandatory Rekor SET verification, ECDSA-P256 over the digest, +revocation and identity checks, and Rekor body binding — all offline — and it +documents the two things it deliberately skips (the Rekor Merkle inclusion +proof, because the verifier computes the wrong shard root for Rekor v2 and +failing closed would reject legitimate signatures; and SCT/CT, for want of a +provisioned CT key). + +**Implementation:** needs sigil. Signing over an arbitrary digest (sigil#256), +a cosign-bundle adapter (#260), the trust-bundle rotation story, and a +non-vacuous air-gapped test (#258). We offered to build the code; two of those +are design decisions that are sigil's to make. + +## 6. Custody and backup model (applies to either path) + +The framing that makes this tractable: **with no revocation, loss and compromise +are both terminal.** More copies reduce loss risk and raise compromise risk. +Secret sharing breaks the trade-off — one share is useless, so copies of shares +are cheap. + +With an HSM the split is not of the key but of the **wrap key**, which changes +the arithmetic usefully: + +| artifact | how many | why | +|---|---|---| +| HSM devices | 2 | primary + restore target; the second is what makes the read test real | +| wrap-key shares | 3-of-5, paper, tamper-evident, separate custodians and locations | one share is worthless; three must fail together to lose it | +| wrapped key backup | ≥3 copies, ≥2 media types, ≥1 offsite | inert without the wrap key, so redundancy is cheap here | +| public half | stored separately, and published | the restore check needs something to compare against | + +3-of-5 rather than 4-of-7 because **the threshold should be chosen for the worst +realistic day**, not the median one — 4-of-7 fails closed as soon as two people +are unreachable. + +**The read test is the step everyone skips and the one that matters.** Annually: +restore onto the second device, sign a throwaway payload, and confirm +`varve pubkey` prints the published root character for character. A safe full of +unreadable media, discovered in the year you need it, is the failure this +prevents. + +**Durability comes from the wrapped blob plus the split wrap key, not from the +second device.** The second device buys availability and a non-destructive +restore test. + +## 7. Hardware + +| option | ed25519 | backupable | verdict | +|---|---|---|---| +| YubiKey (PIV) | 5.7+ only | **no** — non-exportable | recreates #110 in hardware | +| TPM 2.0 | no (P-256) | no | plus `wsc` cannot persist/reload TPM keys (sigil#268) | +| Nitrokey HSM 2 | **no, and no plans** | yes (DKEK n-of-m) | would force an algorithm change | +| **YubiHSM 2** | **yes** | **yes** (wrap key, M-of-N) | fits | + +Prices (yubico.com/de, incl. VAT): YubiHSM 2 **€773.50**; YubiHSM 2 FIPS 140-3 +**€1130.50**. Two devices: **€1547** or **€2261**. + +**FIPS is not required for us** — we are an EU project and CRA is our regime, +not US federal procurement. It buys a certificate number an assessor can check, +which matters to a project whose whole thesis is checkable evidence. But given +succession (§3), it is **not a permanent choice**, so the cheaper device now and +a rotation later is defensible. + +Prior art worth copying rather than inventing: Oxide Computer's +[`offline-keystore`](https://github.com/oxidecomputer/offline-keystore) — Rust, +YubiHSM2, wrap-key splitting, built for exactly this ceremony. + +## 8. Sequencing + +Hardware is **not** the first purchase, because varve cannot drive it yet: +`--key ` is the entire key input. + +1. **`DsseSigner` seam in varve** — behaviour-preserving refactor; existing + signing tests are the oracle. Unblocks *both* paths. +2. **Root succession** — design, requirement, implementation. Makes every later + choice reversible, so it should precede the choices. +3. **Decide keys vs keyless**, with sigil's answers in hand. +4. **Then buy hardware**, and rehearse a full ceremony with a key we intend to + throw away before minting the real one. + +## 9. What we are NOT proposing + +* Not extracting the current secret from CI. It is technically possible with + repository write access and would expose the root permanently, with no + revocation to recover. The inability to read it back is a property worth + keeping. +* Not minting a new key for the current realm outside a ceremony. Consumers pin + the old root; without succession, a new key strands them. +* Not adopting TUF wholesale. Only its root-rotation semantics. +* Not blocking layer publication on any of this. Deposits continue from + `pulseengine/varve` meanwhile. + +## 10. Questions we cannot answer ourselves + +1. **Is CI-signed layer publication acceptable at all** for a qualified channel, + or must qualified layers be signed by a deliberate human act? This determines + whether Path A's weakness (§4) is fatal. +2. **How many root keys**, and what threshold? A ceremony input; cannot be + changed later. +3. **Is the succession statement's trust model sound**, or does it hand an + attacker holding one root key a durable realm takeover that layer-signing + alone does not? +4. **Does an assessor actually credit** a FIPS-validated module here, or is a + documented ceremony with split custody sufficient evidence? +5. **Is 3-of-5 right** for an organisation this size, or does it fail closed + too often to be run honestly? + +[#110]: https://github.com/pulseengine/varve/issues/110 +[#112]: https://github.com/pulseengine/varve/issues/112 +[sigil#268]: https://github.com/pulseengine/sigil/issues/268