Root custody design for REQ-CEREMONY-001: hardware, succession, and how many backups

[avrabe]

Design input for REQ-CEREMONY-001, prompted by #110 (the current root has one write-only copy and cannot be moved or recovered). Three questions, answered in the order they constrain each other: what hardware, how a root is succeeded without stranding consumers, and how many backups.

1. Hardware — the tension is non-exportable vs. backupable

The obvious hardware answers make #110 worse, not better:

option	holds a root?	backupable?	verdict
YubiKey (PIV)	yes, non-exportable	no	recreates #110 in hardware — a single copy nobody can duplicate
TPM 2.0	P-256 only, board-bound	no (not without duplicable-key policy)	plus wsc cannot persist/reload TPM keys today (sigil#268), and varve roots are ed25519
YubiHSM 2	yes, ed25519	yes, under a wrap key	fits

YubiHSM 2 is the one that fits, and the ed25519 support is why. varve trust roots are ed25519, pinned as 64 hex characters in every consumer's config. A TPM-held root would have to be P-256 — a breaking change to the trust-root format on top of everything else v1.0 already changes. YubiHSM 2 signs ed25519 natively, so the root format survives.

The decisive feature is not the signing, it is the wrap key: objects are exported under wrap for backup, and the wrap key itself can be split M-of-N across custodians, with a configurable threshold (Yubico's own worked example is 4-of-7 for an AD root CA).

That resolves the tension. The private key never leaves hardware in the clear, and it can be restored to a second device — which is precisely what the current root cannot do.

Prior art worth copying rather than reinventing: Oxide Computer's offline-keystore — a Rust replacement for yubihsm-setup implementing wrap-key creation and splitting for their own key ceremonies, built on the yubihsm crate. A company solving this exact problem in the open, in our language.

Buy two devices, not one: a backup you have never restored is a hypothesis, and the second device is what turns the annual read test into an actual restore.

2. Succession — "switch or exchange while keeping the status"

Today varve has no rotation and no revocation. docs root-ceremony says the compromise plan is manual: publish a new realm and reach every consumer through the channel that bootstrapped them. That is honest but it does not scale, and it means a rotation is indistinguishable from an attacker telling consumers to re-pin.

The solved version of this problem is TUF root rotation, and the mechanism is worth adopting even if we adopt none of the rest of TUF:

A new root is signed by a threshold of BOTH the old and the new root keys. A client holding only the old root can verify the chain forward to the new one, with no out-of-band contact.

Concretely for varve:

• A root-succession statement: the new root's public key, signed by the old root and by the new root (the latter proving possession, so a typo cannot enthrone a key nobody holds).
• varve verifies that statement against the currently pinned root, and updates trust-root in varve-realms.toml — as an explicit, logged action, not silently.
• Thresholds are what make this safe. With a single root key, whoever holds it can redirect the realm permanently. That is already true today (they can sign any layer), but succession makes the redirection durable, so a threshold — say 2-of-3 root keys, each in its own HSM — is what stops one compromised device from carrying the realm away.
• Anti-rollback applies to succession too: a succession statement carries a counter, and an older one is refused, or an attacker replays a retired root.

This is new requirement territory, not a clause of the existing one. It should be written up before the ceremony, because the number of root keys is a ceremony input — you cannot add a threshold afterwards.

3. How many backups — the answer differs by artifact

The framing that makes this tractable: with no revocation, loss and compromise are both terminal. More copies reduce loss risk and raise compromise risk. M-of-N secret sharing is what breaks the trade-off — a single share is useless, so copies of shares are cheap, while reconstruction still needs a quorum.

With a YubiHSM the split is not of the key but of the wrap key, which changes the arithmetic in a useful way:

artifact	how many	why
HSM devices	2	primary + restore target. The second is what makes the read test real.
wrap key shares	3-of-5, paper, tamper-evident, five separate custodians and locations	a single share is worthless, so no location is a compromise; three must fail simultaneously to lose it
wrapped key backup (the encrypted blob)	≥3 copies, ≥2 media types, ≥1 offsite	it is inert without the wrap key, so copies are nearly free — this is where redundancy is cheap, so buy it here
the public half	stored separately, and published	the restore check has to compare against something

3-of-5 rather than 4-of-7 because five custodians is a plausible number of people for this organisation and 4-of-7 fails closed too easily when someone is unreachable. The threshold should be chosen for the worst realistic day, not the median one.

The read test is the part people skip and the part that matters. Annually: restore the wrapped backup onto the second HSM, sign a throwaway payload, and confirm varve pubkey prints the published trust root character for character. A safe full of unreadable media discovered in the year you need it is the failure mode this prevents, and varve's own docs already prescribe the test — for a key that has never had one.

What this changes upstream

If the root lives in a YubiHSM, varve needs to sign through it, and varve today reads keys from a file only — --key <FILE> is the entire key input. That is REQ-CEREMONY-001 work: a signer seam (wsc already has HardwareSigner), plus a yubihsm backend. Raised on sigil#268, where we offered to build the hardware pieces rather than request them.

Caveats

• Device models, firmware and FIPS variants differ in algorithm support; confirm ed25519 on the exact SKU before the ceremony, not during it.
• None of this is verified against hardware we own. It is a design to be validated by buying two devices and running a rehearsal ceremony with a throwaway key first — the first ceremony should mint a key we intend to discard, so the procedure is debugged before it holds anything real.

Related: #110 (no backup today), REQ-CEREMONY-001 (v1.0.0), sigil#268 (hardware signer support).

Sources: YubiHSM 2 backup and restore, YubiHSM 2 product page, oxidecomputer/offline-keystore

[avrabe]

FIPS or non-FIPS — recommendation: FIPS, with one thing to confirm first

The algorithm worry turns out not to apply. The FIPS approved-mode disabled list is:

ecdsa-sha1, eck256, rsa-pkcs1-decrypt, rsa-pkcs1-sha1, rsa-pss-sha1

ed25519 is not on it. (Note eck256 is secp256k1, the Koblitz curve — not NIST P-256.) So approved mode does not appear to cost us the algorithm varve roots are built on, which was the one thing that would have ruled FIPS out outright.

Why FIPS, for this project specifically

The reason is not compliance — we are an EU project and CRA is the regime that matters to us, not a US federal procurement requirement. The reason is that varve's entire thesis is evidence an assessor can check rather than assurances they must accept. "The realm root is held in a FIPS 140-3 Level 3 validated module" is a checkable claim with a certificate number behind it. "It is in an HSM" is not. Given that everything else in this project is built to be verifiable by a third party, holding the root to a weaker standard than we hold our layers to would be inconsistent.

Two practical arguments in the same direction:

• Validation cannot be added later. The device is the device. If a customer or an assessor asks the question in two years, the answer is fixed at purchase.
• FIPS hardware is a superset of optionality. A FIPS device can be operated without entering approved mode; a non-FIPS device can never make the claim. Paying the delta buys a door that stays open.

The price difference is real but small measured against a key intended to outlive several releases — and against the cost of the ceremony itself, which is mostly people's time.

The one thing to confirm before ordering

ed25519 in approved mode, on the exact SKU and firmware. The disabled list above is strong evidence but it is not the same as a positive statement that ed25519 is available and usable for signing in approved mode. Confirm against the FIPS Security Policy for the validated firmware (the module's algorithm table on the NIST CMVP certificate is authoritative), or ask Yubico directly, before the order.

The failure mode if we get this wrong is not small: FIPS mode is a one-way door on the device, and if ed25519 turns out unavailable in approved mode we are choosing between abandoning the FIPS claim and changing varve's trust-root format to P-256 — which is a breaking change for every consumer who has pinned 64 hex characters by hand.

What does not change either way

Everything in the parent issue: two devices, wrap key split 3-of-5 on paper, ≥3 copies of the wrapped blob across ≥2 media with ≥1 offsite, an annual restore-and-varve pubkey read test, and a first ceremony that mints a key we intend to throw away.

Sources: FIPS mode / disabled algorithms, YubiHSM 2 FIPS product page, FIPS 140-3 validation announcement

[avrabe]

Revising the FIPS recommendation downward — and the reason is in this issue already

Real prices (yubico.com/de, incl. VAT):

	per device	× 2
YubiHSM 2 FIPS 140-3 v2.4	€1130.50	€2261
YubiHSM 2 v2.4	€773.50	€1547

I called that delta "real but small". €714 on top of a €1547 floor is not small for a project with no customers yet, and I should not have waved at it.

Nitrokey HSM 2 is ruled out, for the record, so nobody re-explores it: it has the same n-of-m DKEK backup model at a fraction of the price, but no ed25519 support and no plans to add it. Using it means moving varve roots to ECDSA — the breaking change this issue exists to avoid.

The argument I made for FIPS defeats itself

I argued FIPS partly on: "validation cannot be added later — the device is the device." That is only decisive if the root is permanent. And the middle section of this very issue argues we must build root succession (TUF-style: new root signed by a threshold of both old and new keys, so consumers follow the chain without out-of-band contact).

If succession works, the hardware choice stops being permanent. Buy non-FIPS now; rotate to a FIPS-held root the day someone actually requires it — that is a supported operation rather than a catastrophe, which is the entire point of building succession.

The version of my earlier argument that survives is narrower: if we never build succession, the first hardware choice is forever, so buy the expensive one. Turning that around: paying the FIPS premium today is paying €714 to avoid building succession — and we need succession regardless, because a root that cannot be rotated is exactly the hole #110 describes.

Do not buy anything yet

The blocker is software and it is free:

varve cannot use an HSM at all today. --key <FILE> is the entire key input — no PKCS#11, no KMS, no signer seam. Until that exists (REQ-CEREMONY-001, plus the HardwareSigner work raised on sigil#268), an HSM is a paperweight in a drawer. Buying hardware before the software that can drive it is how you end up with an untested ceremony and depreciating hardware.

Order of operations:

1. Build the signer seam in varve and the yubihsm backend. Free, and it is the real gate.
2. Build root succession (new requirement per the parent issue). Free, and it is what makes step 3 reversible.
3. Then buy. Two non-FIPS devices, €1547, unless a customer or assessor is asking for FIPS by then — in which case buy FIPS and skip a rotation.

One clarification on the second device

Durability does not come from the second HSM. It comes from the wrapped backup blob plus the split wrap key: if the device dies, the key survives and restores onto a replacement. The second device buys availability (sign again the same day) and makes the restore test non-destructive.

So if budget is the binding constraint, one device plus a genuinely tested wrapped backup is defensible — with eyes open that a device failure means days of downtime while a replacement ships, and that varve has no revocation, so downtime is strictly better than the alternatives.

What is not defensible is one device and an untested backup. That is the hypothesis-not-a-backup failure, and it is the one that ends realms.

[avrabe]

Who implements what — and the hardware path needs nothing from sigil

Verified against wsc-dsse 0.11.0 source rather than assumed. The seam is already there and already open:

pub trait DsseSigner {
    fn sign(&self, pae: &[u8]) -> Result<Vec<u8>, DsseError>;
    fn key_id(&self) -> Option<String> { None }
}

pub fn sign(payload: &[u8], payload_type: &str, signer: &dyn DsseSigner) -> Result<Self, DsseError>

Two methods, a trait object, and sign takes arbitrary PAE bytes — no assumption that the payload is a WASM module. varve does not currently use this as a seam: verify.rs hardcodes Ed25519DsseSigner::from_bytes(secret_key, …), so the key is raw bytes all the way down.

This splits the v1.0 decision into two paths with very different costs.

Path A — HSM-held ed25519 root: entirely ours, startable today

Nothing in sigil needs to change. varve implements DsseSigner against a YubiHSM and passes it to the same DsseEnvelope::sign it already calls.

1. Refactor sign_layer_manifest to take &dyn DsseSigner instead of &[u8] key material. Pure refactor, no behaviour change, and it is the whole gate — everything else plugs into it.
2. FileDsseSigner — today's --key <FILE> behaviour, moved behind the trait. Existing tests keep passing or the refactor is wrong.
3. YubiHsmDsseSigner — the yubihsm crate (the same one Oxide's offline-keystore uses), ed25519, key handle by config.
4. Root succession — varve's trust model end to end: the signed succession statement, its counter, verification against the pinned root, and the varve-realms.toml update. sigil has no concept of realms or layers; this could not live anywhere else.
5. Upgrade wsc 0.10.0 → 0.11.0. varve is a minor behind, and 0.11.0 is where the air-gapped verifier stopped being a stub.

Steps 1, 2 and 4 need no hardware to develop and no upstream cooperation. Step 3 can be written against the device's simulator/connector and only needs hardware to validate.

Path B — keyless: needs sigil, and needs decisions before code

The keyless path does not go through DsseSigner at all — it goes through KeylessSignature, Fulcio certs and Rekor entries. That is where sigil#268's asks bite: signing over an arbitrary digest (#256), the cosign-bundle adapter (#260), the trust-bundle rotation story, and the vacuous air-gapped test (#258). We offered to build those, but two of them are design decisions that are sigil's to make, and writing code before they are made would waste the work.

What this changes about sequencing

I said earlier "do not buy hardware yet, the blocker is software." That is still right, but the blocker is smaller than I implied and it is ours:

• The refactor at step 1 is the real gate, it is a day of work, and it unblocks both paths — a DsseSigner seam is required for keyless too.
• Path A has no external dependency. If v1.0 lands on "one long-lived key, held properly", we can build the whole thing without waiting on anyone.
• Path B is gated on sigil decisions, not sigil effort — we volunteered the effort.

That asymmetry is worth weighing in the v1.0 decision itself. Keyless is attractive for taking the long-lived secret out of CI, but it puts the critical path through another repository's design calls. Path A is slower on the security thesis and entirely under our control.

Proposed split

work	owner	blocked on
DsseSigner seam in varve	varve	nothing — start now
FileDsseSigner	varve	the seam
YubiHsmDsseSigner	varve	the seam; hardware only to validate
Root succession	varve	nothing — design in this issue
wsc 0.10 → 0.11 upgrade	varve	nothing
#258 vacuous air-gapped test	varve contributes to sigil	their go-ahead
TPM persistent keys	varve contributes to sigil	their go-ahead
#256 arbitrary-digest keyless signing	sigil	their API decision
Trust-bundle rotation story	sigil	their design decision

Only the bottom two are genuinely someone else's to decide.