Signed snapshots: cryptographic verification for shared L3 cache

[clemlesne]

Business need

When s3_bucket is configured, L2 disk snapshots are uploaded to S3 and shared across hosts (L3 cache). Today, a snapshot downloaded from L3 undergoes only a structural qemu-img check — there is no authenticity verification. An attacker with write access to the S3 bucket (compromised credentials, misconfigured ACL, MITM on a custom s3_endpoint_url) can replace a snapshot with a backdoored one. The cache key is deterministic and publicly computable from the package list, making targeted replacement trivial.

This is the same gap that exists in Firecracker's snapshot model, where their docs explicitly state "snapshot files are trusted" and push signing responsibility onto integrators.

Benefits

• Supply-chain integrity — guarantees that snapshots loaded into VMs were produced by a trusted builder, not injected by an attacker.
• Shared cache confidence — enables multi-host deployments to share snapshots over S3 without trusting the storage layer blindly.
• Auditability — signed payloads include timestamp and signer identity, enabling forensic tracing.
• Industry-first for microVM snapshot managers — neither Firecracker nor QEMU provide built-in snapshot signing today.

Explored alternatives

Approach	Pros	Cons	Verdict
HMAC-SHA256 (symmetric)	Simple, fast, no infra	Shared secret across all hosts — one compromise leaks key; no non-repudiation	Rejected
Ed25519 (asymmetric)	Private key only on builders; fast (sign ~15µs, verify ~40µs); works offline; portable across S3-compatible backends	Requires distributing public key(s)	Selected
Sigstore keyless	Zero key management; Rekor transparency log for audit	Requires network + OIDC infra; not all deployments have this; adds ~15 transitive deps	Future option
AWS S3 native checksums	Free; auto-verified on transfer	Only protects transit, not authenticity — attacker with s3:PutObject can upload valid checksums	Complement, not replacement
dm-verity	Kernel-level block integrity; proven (Android Verified Boot)	Only protects the read-only EROFS base, not the mutable L2/L3 overlay	Complementary feature (separate issue)

Proposed implementation (overview)

Three layers

1. S3 SHA-256 checksums (free, immediate) — add ChecksumAlgorithm="SHA256" to the existing s3.upload_file() call. Protects against accidental corruption and unencrypted-endpoint MITM.

2. Ed25519 snapshot signing (core feature) — on upload, compute SHA-256(compressed_snapshot), sign a canonical payload {version, cache_key, sha256, timestamp} with Ed25519 private key, upload the signature as a .sig sidecar object alongside the snapshot. On download, verify signature against configured public key(s) before decompressing. cryptography library is already a transitive dependency (via joserfc).

3. Sigstore keyless (optional, future) — for CI/CD environments with OIDC (GitHub Actions, GCP), offer sigstore-python as an alternative signer backend. Adds transparency log integration. Not a prerequisite for the core feature.

New settings

snapshot_signing_private_key: str | None = None    # Ed25519 PEM (builders only)
snapshot_signing_public_keys: list[str] = []       # Multiple keys for rotation
snapshot_signing_enforce: bool = False              # True = reject unsigned snapshots

Rollout

• Phase 1: Sign on upload, warn-only on unsigned download (enforce=False). Old unsigned snapshots still work.
• Phase 2: After cache rotation (TTL=14 days), enable enforce=True.

Files impacted

• settings.py — new signing settings
• disk_snapshot_manager.py — sign on _upload_to_s3, verify on _download_from_s3
• New snapshot_signing.py — Ed25519 sign/verify + payload serialization (~100 lines)
• constants.py — SNAPSHOT_SIGNATURE_VERSION = 1

References

• Firecracker snapshot security model — same gap, signing left to integrators
• Python cryptography Ed25519 docs — signing API
• Sigstore — Signing Blobs — cosign sign-blob for arbitrary files
• sigstore-python — Python client for keyless signing
• AWS S3 Additional Checksums — native SHA-256 on upload/download
• dm-verity kernel docs — complementary base image integrity
• QEMU Security — no built-in snapshot signing
• OpenSSF: Scaling Supply Chain Security with Sigstore