Skip to content
This repository was archived by the owner on Nov 8, 2026. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 16 additions & 10 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ This CLAUDE.md is the **authoritative source** for development guidelines. It is

## Critical Design Constraint: VM-Only Snapshotter

**This is a VM-only EROFS snapshotter** for containerd, designed exclusively for use with qemubox (and similar VM runtimes). It does **NOT** mount filesystems on the host - it returns raw file paths that VM runtimes pass to guests as virtio-blk devices.
**This is a VM-only EROFS snapshotter** for containerd, designed exclusively for use with spinbox (and similar VM runtimes). It does **NOT** mount filesystems on the host - it returns raw file paths that VM runtimes pass to guests as virtio-blk devices.

### What This Means

Expand All @@ -39,7 +39,7 @@ This CLAUDE.md is the **authoritative source** for development guidelines. It is
| Mounts filesystems on host | Returns file paths only |
| Returns mounted directory paths | Returns `.erofs` and `.img` file paths |
| Host kernel handles overlay | Guest VM handles overlay |
| Works with any runtime | Works only with VM runtimes (qemubox) |
| Works with any runtime | Works only with VM runtimes (spinbox) |

### Key Design Decisions

Expand All @@ -55,8 +55,8 @@ The snapshotter returns file paths, not mounted filesystems:
VMDK descriptors are **always** generated for multi-layer images. There is NO threshold configuration.

**DO NOT add `--fs-merge-threshold` or similar flags** - they were removed because:
- This snapshotter is exclusively for qemubox
- qemubox always needs VMDK for efficient multi-layer handling
- This snapshotter is exclusively for spinbox
- spinbox always needs VMDK for efficient multi-layer handling
- A threshold adds complexity without benefit

#### 3. No Overlay Returns [CRITICAL]
Expand Down Expand Up @@ -301,17 +301,23 @@ silently commit an empty layer).

### Commit Quiesce Contract [CRITICAL]

- Committing a container requires it to be **stopped** (not paused): the
guest's changes live inside `rwlayer.img` and are only consistent once the
VM released it.
- **Cold commit** (default) requires the container **stopped**: the guest's
changes live inside `rwlayer.img` and are only consistent once the VM
released it.
- Enforcement: `mountutils.MountExt4` takes flock + a whole-file **OFD lock**
(the lock family QEMU uses) before mounting and holds it until unmount;
a running VM yields `errdefs.ErrFailedPrecondition`.
- Guest layout contract: the VM runtime creates `upper/` and `work/` at the
ext4 root; commit/diff read `{ext4root}/upper`. Writes outside `upper/`
are invisible to commit.
- Hot commit (fsfreeze/sync cooperation) requires a qemubox-side protocol
that does not exist yet; until then it fails loudly by design.
- **Hot commit** (container paused + frozen) **is supported**: a runtime that
has paused the VM and frozen its filesystems (FIFREEZE) sets the
`containerd.io/snapshot/erofs.quiesced` label on Commit/Compare. The
snapshotter then mounts `rwlayer.img` read-only (`norecovery`) **without** the
OFD lock gate (a paused QEMU still holds its own image lock, indistinguishable
from a running one). The runtime owns the freeze contract — setting the label
without a real freeze risks a torn read. Without the label, committing a held
image still fails loudly by design.

See `internal/snapshotter/doc.go` ("Committing a Container") for details.

Expand Down Expand Up @@ -525,7 +531,7 @@ For full testing:

- Tests requiring host mounting should use `skipIfVMOnly(t)`
- Mount paths must be real file paths, not mounted directories
- The consumer (qemubox) detects `merged.vmdk` in the same directory as `fsmeta.erofs`
- The consumer (spinbox) detects `merged.vmdk` in the same directory as `fsmeta.erofs`
- Platform-specific code needs parity between `*_linux.go` and `*_other.go`

---
Expand Down
12 changes: 7 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ A containerd snapshotter that converts OCI image layers to EROFS and returns raw

| Aspect | containerd EROFS | erofs |
|--------|------------------|-------------|
| Target runtime | runc, crun (host containers) | qemubox (VM containers) |
| Target runtime | runc, crun (host containers) | spinbox (VM containers) |
| Layer source | Requires pre-converted EROFS | Converts tar→EROFS on pull |
| Multi-layer | Host overlayfs stacking | VMDK descriptor (single block device) |
| Returns | Mounted paths | Raw file paths |
Expand Down Expand Up @@ -136,8 +136,9 @@ When running a container, the snapshotter returns raw file paths with mount opti
```go
// View (read-only) with VMDK - single fsmeta mount with device= options
// VM runtime detects merged.vmdk in same directory and uses it for QEMU
// Multi-device EROFS uses format/erofs (the VM-only signal; see CLAUDE.md)
[]mount.Mount{{
Type: "erofs",
Type: "format/erofs",
Source: "/var/lib/erofs/snapshots/123/fsmeta.erofs",
Options: []string{"ro", "loop", "device=/path/to/layer1.erofs", "device=/path/to/layer2.erofs"},
}}
Expand All @@ -157,13 +158,14 @@ When running a container, the snapshotter returns raw file paths with mount opti
}

// Active (with writable layer) - EROFS lower + ext4 upper
// fsmeta with device= options is multi-device, so format/erofs
[]mount.Mount{
{Type: "erofs", Source: "/path/to/fsmeta.erofs", Options: []string{"ro", "loop", "device=..."}},
{Type: "ext4", Source: "/path/to/rwlayer.img", Options: []string{"rw", "loop"}},
{Type: "format/erofs", Source: "/path/to/fsmeta.erofs", Options: []string{"ro", "loop", "device=..."}},
{Type: "ext4", Source: "/path/to/rwlayer.img", Options: []string{"rw", "loop"}},
}
```

The VM runtime (qemubox) passes these as virtio-blk devices. The guest VM mounts them and creates an overlay.
The VM runtime (spinbox) passes these as virtio-blk devices. The guest VM mounts them and creates an overlay.

**Fallback behavior:** When fsmeta/VMDK generation fails (e.g., `mkfs.erofs` lacks `--aufs` support, or layers have incompatible block sizes from `--tar=i` mode), the snapshotter returns individual EROFS mounts. The consumer must handle stacking these layers.

Expand Down
2 changes: 1 addition & 1 deletion internal/mountutils/erofs_linux.go
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ type Ext4Opt func(*ext4MountConfig)
// consistent and will not change while it is read. QEMU keeps its own image
// lock held even when paused, so the default lock gate cannot tell a frozen
// VM from a running one and would reject the commit; this option is how a
// cooperating runtime (qemubox) signals "it is safe to read now". Using it
// cooperating runtime (spinbox) signals "it is safe to read now". Using it
// without an actual freeze risks a torn read of a live filesystem.
func WithoutImageLock() Ext4Opt {
return func(c *ext4MountConfig) { c.skipLock = true }
Expand Down
2 changes: 1 addition & 1 deletion internal/snapshotter/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
## Quick Overview

**What this package does**:
Implements the containerd snapshotter interface for VM-only EROFS containers. Returns file paths (not mounted directories) that VM runtimes like qemubox pass to guests as virtio-blk devices.
Implements the containerd snapshotter interface for VM-only EROFS containers. Returns file paths (not mounted directories) that VM runtimes like spinbox pass to guests as virtio-blk devices.

**Key responsibilities**:
- Snapshot lifecycle management (Prepare, View, Commit, Remove)
Expand Down
6 changes: 3 additions & 3 deletions internal/snapshotter/doc.go
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
// Package snapshotter implements the spin-erofs containerd snapshotter.
//
// This is a VM-only EROFS snapshotter for containerd, designed exclusively
// for use with qemubox and similar VM runtimes. It does NOT mount filesystems
// for use with spinbox and similar VM runtimes. It does NOT mount filesystems
// on the host - it returns raw file paths that VM runtimes pass to guests as
// virtio-blk devices.
//
Expand Down Expand Up @@ -56,7 +56,7 @@
// - The host mounts the ext4 read-only; journal recovery is permitted so a
// crashed guest still yields a consistent view.
//
// GUEST LAYOUT CONTRACT: the VM runtime (qemubox) must build its overlay
// GUEST LAYOUT CONTRACT: the VM runtime (spinbox) must build its overlay
// with upper/ and work/ at the root of the ext4 - the same layout
// mountBlockRwLayer prepares for extract snapshots. The differ and Commit
// read {ext4root}/upper as the layer content; anything written outside
Expand Down Expand Up @@ -84,7 +84,7 @@
// The "format/erofs" type signals VM-only mounts. Containerd's standard
// mount manager will reject it with "unsupported mount type" rather than
// the cryptic EINVAL that occurs when trying to mount EROFS with file
// paths in device= options. VM runtimes like qemubox handle this correctly.
// paths in device= options. VM runtimes like spinbox handle this correctly.
//
// # Fsmeta/VMDK Generation
//
Expand Down
2 changes: 1 addition & 1 deletion internal/snapshotter/mounts.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ func (s *snapshotter) waitForFsmeta(parentID string, timeout time.Duration) bool
// VM-only mount requiring special handling. Containerd's standard mount manager
// will reject "format/erofs" with a clear "unsupported mount type" error, rather
// than the cryptic EINVAL that occurs when it tries to mount EROFS with file paths
// in device= options. VM runtimes (like qemubox) and the custom mountutils.MountAll()
// in device= options. VM runtimes (like spinbox) and the custom mountutils.MountAll()
// understand this type and handle it correctly.
func (s *snapshotter) mountFsMeta(snap storage.Snapshot) (mount.Mount, bool) {
if len(snap.ParentIDs) == 0 {
Expand Down
2 changes: 1 addition & 1 deletion internal/snapshotter/snapshotter.go
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ func isMounted(target string) bool {
// avoiding TOCTOU race conditions that would occur with filesystem markers.
const extractLabel = "containerd.io/snapshot/erofs.extract"

// quiescedLabel is set by a cooperating VM runtime (qemubox) on a Commit call
// quiescedLabel is set by a cooperating VM runtime (spinbox) on a Commit call
// to assert that the container's writable layer is already quiesced - the VM
// is paused and its filesystems are frozen (FIFREEZE) - so commit may read the
// rwlayer image directly without taking the exclusive image lock.
Expand Down
2 changes: 1 addition & 1 deletion internal/snapshotter/snapshotter_linux_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ package snapshotter
// and shared helper functions used by other test files.
//
// NOTE: Most tests in this file are SKIPPED because the EROFS snapshotter
// is designed exclusively for VM runtimes (like qemubox). The snapshotter
// is designed exclusively for VM runtimes (like spinbox). The snapshotter
// returns raw file paths (EROFS blobs, ext4 images) that are passed to VMs
// as virtio-blk devices, NOT mounted on the host.
//
Expand Down
Loading