doc/plans: Update OCI sealing spec (kernel sigs, flattened layers) by cgwalters · Pull Request #224 · composefs/composefs-rs

cgwalters · 2026-02-05T19:05:16Z

Two big goals:

Support for kernel-native fsverity signatures to be associated with a digest
Generalize the "flattened vs per-layer digest"; any layer can now have either.

allisonkarlitskaya

I started reviewing this before I noticed how large it was. A few comments from the first part...

allisonkarlitskaya · 2026-02-09T06:57:36Z

+### 1. Algorithm string format

-The sealing workflow in composefs-rs begins with `create_filesystem()` building the filesystem from OCI layers. Layer tar streams are imported via `import_layer()`, converting them to composefs split streams. Files 64 bytes or smaller are stored inline in the split stream, while larger files are stored in the object store with fsverity digests. Layers are processed in order, applying overlayfs semantics including whiteout handling (`.wh.` files). Hardlinks are tracked properly across layers to maintain filesystem semantics.
+The spec defines `${DIGEST}-${BLOCKSIZEBITS}` identifiers (e.g. `sha512-12`). Need to implement parsing and mapping to kernel constants (`FS_VERITY_HASH_ALG_SHA512`, 4096-byte blocks, no salt). This is a prerequisite for everything else.


I'm a proponent of the alg-blkbits approach, but maybe we also want a prefix like fsverity- to make sure people understand that it's not straight-up sha512 or not even a straight-up merkel tree. If we did do an fsverity prefix we could use the algorithm number instead, like fsverity-2-12. Just a bit of bikeshedding.

allisonkarlitskaya · 2026-02-09T07:03:34Z

+### 3. Persist manifest and config as regular files

-Two-level naming allows access by fsverity digest (verified) or by ref name (unverified). The `ensure_stream()` method provides idempotent stream creation with SHA256-based deduplication. Streams can reference other streams via digest maps stored in split stream headers, enabling the layer→config relationship tracking.
+The manifest is currently not persisted at all (fetched, parsed, discarded in `skopeo.rs`). The config is stored as splitstream via `write_config()`. Both need to be stored as regular files so `FS_IOC_ENABLE_VERITY` can be called on them.


We definitely don't need to write a file to the disk in order to calculate its fs-verity digest. What's the advantage of doing this? Will you continue to also store the splitstream? If not, GC is going to get a lot more complicated and also slower...

We definitely don't need to write a file to the disk in order to calculate its fs-verity digest. What's the advantage of doing this?

What I want to do is support "read + mount a container image with strict IPE enabled" and to do that we ideally have all of the metadata covered by fsverity as well.

That said, I think most policies are mainly concerned with denying execute-unsigned, not read-unsigned (as that gets obviously hard).

But it felt appealing to me to be able to say that the manifest and config are also just signed-fsverity files.

Or to say it differently that in theory one could omit e.g. cosign covering an image - just a composefs-signature artifact is enough as well to verify a complete image.

Will you continue to also store the splitstream? If not, GC is going to get a lot more complicated and also slower...

Yes, see https://github.com/containers/composefs-rs/pull/216/changes#diff-202658a04902f3f6a4578fa63cf9f6111413c5acfbcccd556631a9a2135dd3ffR163

That's all we need to do right?

Yes, I mean that should work just fine. I'm not sure what the value of IPE for "just data" here is though, since the kernel won't block that, and since we already have (and check) the canonical identifier (ie: sha256 content hash). fsverity (and dm-verity) are merkle trees because they're about protecting sparse access to large files without having to hash the entire thing at the start, which doesn't really apply to a JSON document...

Yes, I mean that should work just fine. I'm not sure what the value of IPE for "just data"

The way I was thinking about this more is from the "image integrity" angle and less about IPE specifically. It relates to the topic you brought up below.

If we have a kernel-fsverity signature that directly covers the manifest then "for free" we get verification when we read the manifest that it is valid.

And the manifest + config define things like the layer ordering. So unless a runtime validated some other signature (such as cosign) on startup it would still be possible to swap image layers around.

I guess backing up a bit though, even with this though someone could e.g. replace the logical tag for a floating quay.io/someorg/somecontainer with some other image on disk.

This relates to something pwithnall did for ostree in including bindings in the (signed) commit. In theory we could add an OCI extension that included the (or multiple) image name in the manifest as an annotation, and a runtime could validate then when it goes to run an image with that name that the app matches. I think DDIs basically have some of this because the os-release field can contain a name.
(And of course arguably...we could add container image tags into /usr/lib/os-release too...)

Anyways though, yes, kernel-fsverity signature on the manifest/config is not required under all threat models, but since it seems easy to do and (AFAIK) there's no downside, I'd like to do it.

I guess though backing up...there's really two cases.

The system configuration pins an image by explicit (manifest|config) digest

The there's a floating tag

For sure with 1) I don't think we need the kernel-fsverity signature for the manifest, config, it is indeed just "find image named by manifest|config digest", verify sha digest of object, then mount merged erofs with kernel-fsverity signature in use.

Of course, this whole thing is predicated a bit on "how do I verify the config which specifies that quay.io/foo/bar@sha256:... - for bootc LBIs that's very clear, it's covered by the UKI -> composefs-for-root. Other cases might use something like signed confexts (which I'd like to support being OCI+composefs too).

For floating tags, yeah we can't really defend against image swapping "offline" without strengthening what gets signed.

That said, for use cases like Kubernetes, there is an option for kubelet to re-ping the registry for images on startup for a reason related to this a bit - validating that a user can pull an image even it happens to be on disk.

In the end I just come back to this:

since it seems easy to do and (AFAIK) there's no downside, I'd like to do it.

allisonkarlitskaya · 2026-02-09T09:50:23Z

+
+### Signatures
+
+#### Linux kernel fsverity signatures (recommended)


This is a very significant departure from the way the trust model works now and how the object store works in general. From what I understand, each file can only have a single fs-verity signature on it, but we store objects by their content hash, which means if we had two objects enter the object store from differently-signed containers, we'd be in trouble, no?

Is this for every object or just the erofs image?

I'm also not sure that kernel-level fsverity signatures provide very strong protection (at the level that would be provided by signatures on disk images, for example) because they are on a file-by-file basis, assuming that is the intent here. If you ignore the userspace stuff, you could probably still use your ability to freely mix-and-match various individually-signed files into a system configuration that let's you do "bad things"...

From what I understand, each file can only have a single fs-verity signature on it, but we store objects by their content hash, which means if we had two objects enter the object store from differently-signed containers, we'd be in trouble, no?

This is a good point - however it's interesting as I think it's more of an implementation concern and not a spec concern.

Is this for every object or just the erofs image?

Yes exactly: we don't require fsverity signatures on individual objects (that form part of a split layer tarball). See this issue in a nutshell the goal is that having the fsverity signature on the EROFS blob + detecting overlay require-verity should be sufficient for chain-of-trust.

But that said, what would happen if e.g. two distinct images shared a layer? I think "first one wins" is sufficient for the rootful case. The Linux kernel fsverity signature mechanism only has one keyring which applies to everything, and we can't do anything different. In a future world where there's e.g. per-user keyrings or so...it would just preclude sharing the EROFS metadata blob between trust domains (root and rootless e.g.) right? We could still share the underlying layer objects via hardlinks.

If you ignore the userspace stuff, you could probably still use your ability to freely mix-and-match various individually-signed files into a system configuration that let's you do "bad things"...

Again not individual signed files, but only complete layers. I don't think think this is any different from e.g. dm-verity + IPE. I wouldn't dismiss this concern entirely, but it seems like it'd be quite difficult offhand in practice to craft such a chain.

Now that said one obvious thing even with layers is that this doesn't protect against e.g. rollback attacks. I think that type of thing needs to be out of scope of this spec.

I was thinking about this a bit more, and I think there is still an argument that we should support inline digests in the manifest (or config). It would naturally mean that we can reliably chain from "trust in manifest" to "trust in mounted root", which was again the original goal.

For cases where "trust in manifest" is implicitly handled (e.g. kubernetes API server tells us to run an image with this particular digest) it would Just Work as long as fsverity is supported by the underlying filesystem.

This support would give us an generic "out" for rootless/unprivileged use. (Though honestly in the medium term it'd be clearly nice to enhance the linux kernel fsverity to at least do something like "allow per-user keyrings for files owned by that user" or so)

But that said, what would happen if e.g. two distinct images shared a layer? I think "first one wins" is sufficient for the rootful case. The Linux kernel fsverity signature mechanism only has one keyring which applies to everything, and we can't do anything different. In a future world where there's e.g. per-user keyrings or so...it would just preclude sharing the EROFS metadata blob between trust domains (root and rootless e.g.) right? We could still share the underlying layer objects via hardlinks.

could you please elaborate on this point? For the runtime side, why would it be a problem as long as there is at least one accepted signature?

I agree, "one accepted signature" is fine - we can't do anything different today because the Linux kernel's fsverity mechanism only allows the same.

cgwalters · 2026-02-12T13:46:27Z

@@ -0,0 +1,74 @@
+# Standardized EROFS Metadata Serialization


When talking about creating a specification for the sealing, of course this heavily depends on a spec for the EROFS layout, which pulls back in all the debate in composefs/composefs#198

Now... #225 is starting to look at what it'd take to have us support being bit-for-bit compatible with the previous composefs-c (1.0) format.

In an ideal world perhaps we teach mkfs.erofs how to generate this too? This also relates a bit to uapi-group/specifications#207

hi @cgwalters, do you mean the erofs metadata arrangement or the sealing format?
For the sealing format, I'm fine to get any help as long as anyone has interest to port this to erofs-utils, it can be used to improve the interaction between composefs tools and erofs-utils.

As for the erofs metadata itself, I don't think erofs-utils should strictly align with mkcomposefs (just because erofs-utils itself already have different arrangement for different cases, but erofs-utils is always designed reproduciblely), also I think it sounds unnecessary since erofs metadata layout is flexible enough (yet composefs can definitely define a strict on-disk layout for all related stuffs.)

But my own TODO list is already overloaded, so I don't help on some practical development on this.

but erofs-utils is always designed reproduciblely)

Only within a specific binary version, right? You don't guarantee that a future mkfs.erofs wouldn't generate a different metadata layout, correct?

but erofs-utils is always designed reproduciblely)

Only within a specific binary version, right? You don't guarantee that a future mkfs.erofs wouldn't generate a different metadata layout, correct?

Yes, of course. but erofs-utils layout won't be frequently changed in the foreseen future I think.

Right, so a very important thing that we're trying to do with composefs+OCI is not change the wire format for OCI - we're not shipping EROFS on the wire, only generating it reproducibly on the client and server.

Hence we must:

Lock in the bit-for-bit file format basically forever

Have solid tooling to generate it (and ideally that tooling is easily accessible from multiple programming languages)

just one note: currently tar format also doesn't strictly the header/file order for example and various tools generate various tars, so I guess reproduciblely within a specific binary version is also fine as long as users can reproduce it with the tools/command line and the exact version.

I respect your choices but I don't see a manifest erofs metadata on the wire sounds inappropriate from whatever point of view (of course you could lock in the bit-for-bit file format forever.)

just one note: currently tar format also doesn't strictly the header/file order for example and various tools generate various tars, so I guess reproduciblely within a specific binary version is also fine as long as users can reproduce it with the tools/command line and the exact version.

Yes, this is a reason why the doc text here is proposing only a canonical mapping from composefs dumpfile ➡️ EROFS.

A composefs dumpfile has less representational flexibility (e.g. paths always start with /, xattrs are always sorted and serialized in band) I will actually try to ensure we have a "canonical dumpfile" format which should help.

I respect your choices but I don't see a manifest erofs metadata on the wire sounds inappropriate from whatever point of view (of course you could lock in the bit-for-bit file format forever.)

Hmmm. It is an interesting design choice; now that we've moved the signatures to a separate OCI artifact, indeed we could try to create a design where the EROFS-metadata is actually stored on the registry too.

However...it introduces the same "representational ambiguity" in that we'd be shipping both tar and metadata-EROFS and have to answer the question of: what happens when they disagree? This problem is one argument why zstd:chunked is still always validating against the diffid for security reasons.

But OTOH with composefs here we aren't actually trying to optimize incremental fetches, so we're always parsing the whole tarball anyways and could actually do something like:

fetch erofs-meta from detached composefs OCI signature

fetch layers

For each layer, parse the tarball and generate a canonical in-memory metadata representation (like a composefs dumpfile) and then compare it with the erofs-meta: if they differ, it's a fatal error

Hummm.....yes, I think such an approach would allow us to entirely punt on the problem domain of standardizing an EROFS layout, but at the cost of duplicating all of the metadata.

Is that the right call? I am...unsure.

I don't have the answer: I could find some technical ways, yet when it comes to OCI world, it really su*ks...

Right now the "sealed UKI" mode is relying on the bit-for-bit EROFS reproducibility, which argues for standardizing it.

But if we went all in on this external fsverity signature mechanism, then I think we could (per discussion) also convert basically all use cases for sealed UKIs to use it as well. (After some design work)

alexlarsson · 2026-02-19T09:25:16Z

+
+#### Linux kernel fsverity signatures (recommended)
+
+The primary signature mechanism is Linux kernel [fsverity built-in signature verification](https://docs.kernel.org/filesystems/fsverity.html#built-in-signature-verification). The kernel's `FS_IOC_ENABLE_VERITY` ioctl accepts a PKCS#7 signature that is verified against the `.fs-verity` keyring. This provides a clear chain of trust: the same component that controls data access (the kernel) also validates the signature. The kernel additionally integrates with the [IPE](https://docs.kernel.org/admin-guide/LSM/ipe.html) (Integrity Policy Enforcement) subsystem.


I would like to point out that I talked with the kernel fsverity maintainer at some point (@ebiggers) , and he told me that he considers the built in signatures for fs-verity to be lame, and shouldn't be used. And indeed, they are not enabled in the default fedora kernel config for example.

Is that a temporary thing that will eventually be fixed, or should we move on to dm-verity?

Honestly, I don't know.

The dm-verity built-in signatures have the same problems. Before using either one, please read https://docs.kernel.org/filesystems/fsverity.html#built-in-signature-verification

The status quo today is that for IPE only the kernel builtin signatures count - IOW if we were to do user space signatures, then we'd need to argue for some mechanism for a trusted user space process to have privileges to set that flag with the IPE maintainers, right?

Though really I think this sub-thread is best moved to composefs/composefs#360

Yeah I think the Fedora 34 (and openSUSE Leap 15.6) kernel configuration is incoherent; they set CONFIG_DM_VERITY_VERIFY_ROOTHASH_SIG=y but # CONFIG_FS_VERITY_BUILTIN_SIGNATURES is not set. Whereas for e.g. Debian 13, it sets both.

I understand that there's disagreement among some kernel maintainers as to whether in-kernel signatures should be used or not. But IPE and IMA also reuse the in-kernel signature infrastructure, and this proposal is explicitly targeting support for IPE.

cgwalters · 2026-03-05T14:39:58Z

@giuseppe and I had a realtime chat on this again and one interesting thing here if if we went to "metadata EROFS on registry" - we could get "incremental pulls" (estargz/zstd-chunked style) I think just including a tiny little bit of additional metadata: for each non-inline file, we include struct Offset { erofs_inode: u64, layer: u64 } - serialized in whatever format (compressed json would be fine) as an additional artifact.

Then incremental fetches are easy - we load the erofs and know we know from the fsverity digests there which objects we have and which we don't, and then we can use the layer offset to find the object (which would need the gzip-individual-files of estargz or zstd incremental or maybe in the future registry compression - then it's even simpler).

A compelling thing about such a model (over estargz and zstd-chunked) is that effectively the EROFS acts as a replacement for the TOC - it's a metadata format that is natively supported by the Linux kernel and has various userspace parsers.

cgwalters · 2026-03-05T19:50:16Z

🆕 OK I did another big rework of this, we now clearly support 2 modes, "erofs-alongside" and "canonical".

I am leaning towards pushing hard on the erofs-alongside. The thing is...the incremental fetch support would clearly be a really nice thing to have overall, and any such thing is going to need a separate metadata format, and it might as well be EROFS.

If in the future we define a "canonical" representation...maybe we can change to just enforce that canonical representation.

Per the sealing PR #224 (comment) Debian's kernel has support for fsverity sigs, but Fedora derivatives don't. In order to gain more diverse coverage here in general, but also in prep for that, add support for building on top of the bootcrew Debian image that especially can then use `bcvk ephemeral` for testing. Assisted-by: OpenCode (Claude claude-opus-4-6) Signed-off-by: Colin Walters <walters@verbum.org>

cgwalters · 2026-03-09T12:55:16Z

consider how we add incremental fetching more precisely, leave space for it
Truly ensure we can drop tar-split (canonical tar)
Talk to registry maintainers: will doubling the number of layers for popular images be a problem, and extra HTTP GET per layer
go over signature scheme

Alternatives

zstd-erofs where the erofs is a skippable frame

alexlarsson · 2026-03-09T14:28:12Z

So, I read this, and I generally like it. However, I think for the "secureboot" usecase it is maybe lacking a bit on details. As I understand it, the intended usecase is to ship the PKCS signatures and apply on download, and then load the required public key into the keychain from the initrd (Or I guess, maybe build it into the kernel image if that is possible?).

However, we also need to validate on boot that the rootfs is actuallty signed. Is the intent for the inird to do this directly? Or do we expect further work on IPE to do that?

Also, I like the ability of supporting the "digest-only verification", as it seems useful in some cases. However, for that to work with signatures it has to do some pre-processing when going from oci later to erofs image. The docs vaguely mentions this ("without /boot among other things"), but this needs to be much more explicit. I wonder if we could just have the manifest explicitly list which files should not be in the erofs image.

alexlarsson · 2026-03-09T14:35:52Z

However, we also need to validate on boot that the rootfs is actuallty signed. Is the intent for the inird to do this directly? Or do we expect further work on IPE to do that?

I guess if the initrd does the "check that rootfs is signed" then this could also have fallback for the case where CONFIG_FS_VERITY_BUILTIN_SIGNATURES is not enabled, by just verifying the signature (from a file next to the erofs) in userspace, based on a userspace copy of the public key in the initrd. In this case we wouldn't be able to do the IPE stuff, but we can at least have a trusted boot chain.

cgwalters · 2026-03-09T18:18:26Z

As I understand it, the intended usecase is to ship the PKCS signatures and apply on download,

Yes.

and then load the required public key into the keychain from the initrd (Or I guess, maybe build it into the kernel image if that is possible?).

Yes (with UKI, "builtin to kernel" is the same as "initrd" - I know you know this, but worth emphasizing).

this could also have fallback for the case where CONFIG_FS_VERITY_BUILTIN_SIGNATURES is not enabled,

Yes...I guess we should support that for now, but I would like to argue we at least strongly encourage it to be turned on.

* Add OCI artifact manifest and Referrers API support We're working on composefs/composefs-rs#224 and the new idea is we create a composefs OCI artifact. While working on this I realized we didn't have good enough support for OCI artifacts here. There's a few things, but especially the image index can have artifactType for example. There's also an API to find referrers, mirroring the distribution API. `fsck` now accepts unknown config media types (verifying the blob digest) rather than rejecting them, per the OCI image spec requirement that implementations MUST NOT error on unknown config mediaTypes. Assisted-by: OpenCode (Claude claude-opus-4-6) Signed-off-by: Colin Walters <walters@verbum.org> * Add ocidir CLI example with selftest for OCI artifacts While we have examples here, it feels really natural to put together a little CLI that wires us up to oci-distribution so we have a pure-Rust little demo of fetching from a registry and storing to ocidir. Since we have this, add a `selftest` command which can be used in CI to act as an integration test. Assisted-by: OpenCode (Claude claude-opus-4-6) Signed-off-by: Colin Walters <walters@verbum.org> * fsck: Verify config blob digests uniformly for all media types Previously, fsck_one_manifest only verified SHA-256 digests for config blobs with unknown media types, while known types (ImageConfig, EmptyJSON) only got size + JSON deserialization checks. This meant a targeted same-size modification to a config blob could go undetected for known types, which is inconsistent with how layer blobs are always digest-verified. Now all config blobs get their digest verified first, then known types additionally get structure validation via deserialization. This also extracts the duplicate digest verification logic into a shared verify_blob_digest() helper used by both config and layer verification. Also fix .unwrap() in write_config to use ? instead, since the function already returns Result. Assisted-by: OpenCode (Claude Opus 4) Signed-off-by: Colin Walters <walters@verbum.org> * fsck: Skip already-validated config blobs, use proper error type Two minor cleanups to verify_blob_digest: Use ? directly on Hasher::finish() instead of map_err to Error::Other, since From<ErrorStack> already maps to the more specific CryptographicError variant. Also skip config blob verification when the digest is already in the validated set, matching the existing layer deduplication logic. This avoids redundant I/O for the common case of artifact manifests sharing the empty config blob. Assisted-by: OpenCode (Claude Opus 4) Signed-off-by: Colin Walters <walters@verbum.org> --------- Signed-off-by: Colin Walters <walters@verbum.org>

cgwalters · 2026-06-26T15:09:45Z

OK, I completely rewrote this again. (Kind of tempting to make a new PR)

I think since we now have the v1 support merged - let's just go all in on that as our "standard erofs". I completely dropped support for the separate OCI artifact with EROFS.

We now support a metadata-only artifact as well as inlining the data into the manifest. I think the latter will be easy enough for people to adopt (e.g. once we ship the varlink support, having podman build --composefs-metadata=inline and --composefs-sign will be quite viable.

mike-sul · 2026-07-09T12:32:24Z

I'm designing a flow that uses composefs + a content-addressed object store (CAS) as the on-disk representation of container image rootfs for IoT/edge devices.

One requirement is image "sealing": at publish time, compute the fs-verity digest of the composefs/EROFS metadata image generated from the image’s fully merged rootfs, and bind that digest to OCI image metadata.

IIUC, for inline mode this PR moves the merged-rootfs digest from the earlier config-label model to a top-level annotation on the image manifest, for example:

composefs.merged.erofs.v1.fsverity-sha256-12=<fsverity-digest-of-the-erofs-metadata-image>

Is that correct?

Also, is this annotation key shape stable enough for new tooling to build against, or would you recommend waiting for the PR to settle/merge?

cgwalters · 2026-07-09T13:36:18Z

I'm designing a flow that uses composefs + a content-addressed object store (CAS) as the on-disk representation of container image rootfs for IoT/edge devices.

Is your project in Rust? WDYT about the goal of using cfsctl via varlink as a primary API? #322

Also, for the bootable host, have you looked at https://github.com/bootc-dev/bootc/ ? Would bootc-dev/bootc#2289 help?

One requirement is image "sealing": at publish time, compute the fs-verity digest of the composefs/EROFS metadata image generated from the image’s fully merged rootfs, and bind that digest to OCI image metadata.

Right now for bootc (and is supported here) we handle this via a kernel commandline in a signed UKI, see https://github.com/bootc-dev/bootc/blob/main/docs/src/experimental-composefs.md#how-sealed-images-work - were you interested in doing this host root via fsverity signatures?

IIUC, for inline mode this PR moves the merged-rootfs digest from the earlier config-label model to a top-level annotation on the image manifest, for example:

Right.

Also, is this annotation key shape stable enough for new tooling to build against, or would you recommend waiting for the PR to settle/merge?

Well, #226 has a lot of pending work that tries to implement the spec.

Were you planning a new/different implementation?

I will try to get more consensus on this spec soon across stakeholders, but I personally think what we have now is viable.

I've updated this PR.

giuseppe · 2026-07-10T07:58:54Z

+//!
+//! Unlike zstd:chunked, there are no trailing skippable frames (no embedded JSON TOC, no tar-split data). The offset map in the composefs metadata artifact serves as the TOC; the EROFS itself is generated locally by the client.
+//!
+//! Unlike zstd:chunked, there is no sub-file content-defined chunking. Composefs deduplicates at the whole-file level (by fsverity digest), so rolling-checksum chunk boundaries provide no dedup benefit. This simplifies the format and the offset map.


zstd:chunked uses sub-file content chunking to save on network traffic.

giuseppe · 2026-07-10T07:59:54Z

+//!
+//! ### Layer Format: composefs-chunked
+//!
+//! A composefs-chunked layer is a valid `tar+zstd` blob that any OCI client can pull and decompress normally. The difference is in how the zstd compression is structured internally: large files are compressed as independent zstd frames, making them individually addressable via byte offset.


do we really need to define a new format instead of building on top of what already exists?

What is missing in zstd:chunked to achieve this except the fs-verity digest?

giuseppe · 2026-07-10T08:06:40Z

+//!
+//! Files ≤ 64 bytes are already inline in the EROFS metadata (`INLINE_CONTENT_MAX`) and are never fetched from the tar layer during an incremental pull, regardless of framing.
+//!
+//! Unlike zstd:chunked, there are no trailing skippable frames (no embedded JSON TOC, no tar-split data). The offset map in the composefs metadata artifact serves as the TOC; the EROFS itself is generated locally by the client.


I prefer this to be self-contained and do not require an external artifact for each layer (doubling the number of objects on the registry compared to what we have today).

So what can we do to fix zstd:chunked to address this use case as well?

giuseppe

left some comments, I think it is better to fix/reuse zstd:chunked so it works with this use case as well, since it is already implemented and supported in containers tools. Having yet another format seems unnecessary and more difficult to maintain.

Personally, I am not a big fan of requiring artifacts, I'd prefer layers to be self contained (even if the additional information is stored in the compression layer).

We can change zstd:chunked to generate always the same stream so we can drop the tarsplit part. The missing part there was the canonical representation, that now it is possible to achieve.

Define how composefs attaches fsverity digests (and optional kernel PKCS#7 signatures) to OCI manifests, configs, and merged EROFS trees, covering both inline-annotation and separate-artifact modes. This supersedes the older doc/plans/oci-sealing-spec.md, moving the content into rustdoc so it renders and stays next to the code it describes. Signed-off-by: Colin Walters <walters@verbum.org>

cgwalters · 2026-07-10T12:14:51Z

left some comments, I think it is better to fix/reuse zstd:chunked so it works with this use case as well, since it is already implemented and supported in containers tools. Having yet another format seems unnecessary and more difficult to maintain.

Thanks for the review! In retrospect, I think I made a big mistake in trying to pivot this proposal to the "erofs as OCI artifact" - that's what led into the whole side realization that maybe we could do something zstd:chunked like using the fact that we now had an EROFS etc.

To recap, the rationale for EROFS-as-artifact was to avoid trying to standardize exactly how we generate the EROFS layout. But we now have that - we have two independent implementations of the composefs V1 EROFS (including one here) and we are now standardizing on that.

That means we can go back to focusing on something much simpler: how we ship the digest and fsverity signatures and that's it.

I think most of your comments here now apply to #356 - but yes...from my PoV it is not important to have a new chunking implementation at all, and trying to scope that in here was again a big mistake!

Personally, I am not a big fan of requiring artifacts, I'd prefer layers to be self contained (even if the additional information is stored in the compression layer).

My intuition here is that the dividing line here is: Including the digest inline in the image is conceptually straightforward - there's implementation details (like fsverity digest caching) to consider, but at some point I think podman build etc should just start including the digest.

The signature is harder because:

There are good operational reasons for wanting to split up the build process from a signing service
It's also useful to be able to add this data to images that were built before the spec existed (this applies to the digest too of course)

giuseppe

LGTM

mike-sul · 2026-07-10T12:56:50Z

I will try to get more consensus on this spec soon across stakeholders, but I personally think what we have now is viable.

I've updated this PR.

@cgwalters Thanks, that helps.

Is your project in Rust? WDYT about the goal of using cfsctl via varlink as a primary API? #322

Also, for the bootable host, have you looked at https://github.com/bootc-dev/bootc/ ? Would bootc-dev/bootc#2289 help?

The project is being written in Go and integrates with the "dockerd->containerd" path as a remote snapshotter. It targets regular application container images on IoT/edge devices, not bootable/system container images that bootc is already focused on.

On cfsctl/varlink: I’d use it today if it covered what I need, but it does not yet. The cfsctl/API is pre-stable, and sealing does not appear to be exposed end to end: Pull and ComputeId exist, but annotating the manifest and publishing the sealed manifest/index does not.

Were you planning a new/different implementation?

So for the MVP, I’ve been building on existing composefs userspace tooling such as mkcomposefs --from-file and the dump format, while keeping the semantics aligned with composefs-rs. Once the varlink API stabilizes, I’d prefer to replace custom code with upstream functionality for the parts it cleanly covers. The less composefs-specific code I own, the better. If it helps, I’m happy to write up the concrete gaps against our use cases.

I guess, there is one commitment regardless: reproducibility across implementations is central to this spec, so I’ll use Oci.ComputeId as a conformance check: the same inputs and EROFS format version should produce the same merged image ID. I’ll report anything that diverges.

cgwalters · 2026-07-10T16:13:39Z

On cfsctl/varlink: I’d use it today if it covered what I need, but it does not yet. The cfsctl/API is pre-stable, and sealing does not appear to be exposed end to end: Pull and ComputeId exist, but annotating the manifest and publishing the sealed manifest/index does not.

Look for more work to appear in #226 soon!

So for the MVP, I’ve been building on existing composefs userspace tooling such as mkcomposefs --from-file and the dump format, while keeping the semantics aligned with composefs-rs.

OK. There are some nontrivial details here to consider though, such as the EROFS digest depending also on the underlying file paths. We do have integration test coverage that there's compatibility with the original C mkcomposefs (and there's a brand new Rust implementation in this repo now) but it is certainly within possibility that there's something missed here.

What would probably be good is basically if your tooling is FOSS - post it. Or if it's not, at least please do try to evaluate compatibility vs what should hopefully soon land in this repo.

Once the varlink API stabilizes, I’d prefer to replace custom code with upstream functionality for the parts it cleanly covers. The less composefs-specific code I own, the better. If it helps, I’m happy to write up the concrete gaps against our use cases.

Yes please do!

mike-sul · 2026-07-13T16:48:06Z

@cgwalters Thanks. I’ll follow the work in #226 closely and evaluate compatibility with it as the implementation lands.

What would probably be good is basically if your tooling is FOSS - post it. Or if it's not, at least please do try to evaluate compatibility vs what should hopefully soon land in this repo.

The intent is for the tooling to be FOSS, but I still need to initiate and complete our internal open-source approval process before I can publish the repository.

I checked the convention used by composefs-rs: the relative backing-object path is derived from the file’s fsverity digest. I’ll use the same convention and compare the result with mkcomposefs, the rust implementation, and Oci.ComputeId and report any mismatch.

Once the varlink API stabilizes, I’d prefer to replace custom code with upstream functionality for the parts it cleanly covers. The less composefs-specific code I own, the better. If it helps, I’m happy to write up the concrete gaps against our use cases.

Yes please do!

Is issue #322 the best place to report these gaps?

mike-sul · 2026-07-14T11:31:20Z

@cgwalters This is the key gap I've discovered so far.

cgwalters mentioned this pull request Feb 5, 2026

integration with IPE tracker composefs/composefs#360

Open

cgwalters commented Feb 6, 2026

View reviewed changes

Comment thread doc/plans/oci-sealing-spec.md Outdated

cgwalters mentioned this pull request Feb 6, 2026

Initial reimplementation of composefs-c #225

Merged

cgwalters force-pushed the spec-enhancements branch from a404b59 to 2a23a13 Compare February 7, 2026 02:25

cgwalters mentioned this pull request Feb 8, 2026

WIP: Rough draft for updated generic OCI sealing #226

Draft

allisonkarlitskaya reviewed Feb 9, 2026

View reviewed changes

cgwalters force-pushed the spec-enhancements branch 2 times, most recently from fa8852b to b470f73 Compare February 12, 2026 02:08

cgwalters marked this pull request as ready for review February 12, 2026 13:39

cgwalters commented Feb 12, 2026

View reviewed changes

alexlarsson reviewed Feb 19, 2026

View reviewed changes

cgwalters mentioned this pull request Mar 5, 2026

ci: Add support for Debian+bcvk #245

Merged

cgwalters force-pushed the spec-enhancements branch from b470f73 to 5c76591 Compare March 5, 2026 19:37

cgwalters force-pushed the spec-enhancements branch from 5c76591 to c885dc3 Compare March 5, 2026 20:39

cgwalters requested review from alexlarsson and giuseppe March 6, 2026 19:37

cgwalters enabled auto-merge (rebase) March 9, 2026 12:44

cgwalters marked this pull request as draft March 9, 2026 12:52

auto-merge was automatically disabled March 9, 2026 12:52
Pull request was converted to draft

This was referenced Mar 11, 2026

oci: Add fast path for oci: transport using ocidir crate #233

Merged

composefs end state "v1" goal podman-container-tools/container-libs#119

Open

cgwalters force-pushed the spec-enhancements branch from 31ab76d to 8ffdf00 Compare April 13, 2026 15:26

alexlarsson mentioned this pull request May 6, 2026

Proposal for support for runtime composefs validation podman-container-tools/podman#28658

Draft

cgwalters force-pushed the spec-enhancements branch from 8ffdf00 to 143d805 Compare June 26, 2026 15:07

cgwalters force-pushed the spec-enhancements branch 2 times, most recently from b64d92c to b70793a Compare June 26, 2026 19:13

cgwalters mentioned this pull request Jul 1, 2026

Add support for OCI artifacts “attached” to an image via subject podman-container-tools/container-libs#221

Open

cgwalters added this to the 2.1 milestone Jul 9, 2026

cgwalters closed this Jul 9, 2026

auto-merge was automatically disabled July 9, 2026 12:13
Pull request was closed

cgwalters reopened this Jul 9, 2026

cgwalters force-pushed the spec-enhancements branch from b70793a to 85ff9b7 Compare July 9, 2026 13:31

cgwalters enabled auto-merge July 9, 2026 15:01

giuseppe reviewed Jul 10, 2026

View reviewed changes

cgwalters force-pushed the spec-enhancements branch from 85ff9b7 to d490828 Compare July 10, 2026 12:02

cgwalters force-pushed the spec-enhancements branch from d490828 to 573cdeb Compare July 10, 2026 12:03

cgwalters mentioned this pull request Jul 10, 2026

DRAFT: Add design doc for canonical tar #356

Draft

giuseppe approved these changes Jul 10, 2026

View reviewed changes

cgwalters added this pull request to the merge queue Jul 10, 2026

Merged via the queue into composefs:main with commit a29622c Jul 10, 2026
18 of 19 checks passed


		### Signatures

		#### Linux kernel fsverity signatures (recommended)


		#### Linux kernel fsverity signatures (recommended)

		The primary signature mechanism is Linux kernel [fsverity built-in signature verification](https://docs.kernel.org/filesystems/fsverity.html#built-in-signature-verification). The kernel's `FS_IOC_ENABLE_VERITY` ioctl accepts a PKCS#7 signature that is verified against the `.fs-verity` keyring. This provides a clear chain of trust: the same component that controls data access (the kernel) also validates the signature. The kernel additionally integrates with the [IPE](https://docs.kernel.org/admin-guide/LSM/ipe.html) (Integrity Policy Enforcement) subsystem.

Uh oh!

Conversation

cgwalters commented Feb 5, 2026

Uh oh!

Uh oh!

allisonkarlitskaya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hsiangkao Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hsiangkao Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgwalters commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters commented Mar 5, 2026

Uh oh!

cgwalters commented Mar 9, 2026

Alternatives

Uh oh!

alexlarsson commented Mar 9, 2026

Uh oh!

alexlarsson commented Mar 9, 2026

Uh oh!

cgwalters commented Mar 9, 2026

hsiangkao Feb 12, 2026 •

edited

Loading

hsiangkao Feb 12, 2026 •

edited

Loading

cgwalters commented Mar 5, 2026 •

edited

Loading