Skip to content

[WIP] Boot-disk repartition for in-field EVE-kvm ↔ EVE-k conversion - #6063

Draft
eriknordmark wants to merge 16 commits into
lf-edge:masterfrom
eriknordmark:kvm-to-k-resize
Draft

[WIP] Boot-disk repartition for in-field EVE-kvm ↔ EVE-k conversion#6063
eriknordmark wants to merge 16 commits into
lf-edge:masterfrom
eriknordmark:kvm-to-k-resize

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Boot-disk repartition for in-field EVE-kvm ↔ EVE-k conversion

Long-lived devices installed with EVE's old small boot-disk layout (IMGA/IMGB
as small as 300 MB, a small ESP) cannot hold the EVE-k (Kubernetes) rootfs and
its container/k3s tooling. Upgrading replaces rootfs contents but never
repartitions, so the small geometry persists. This branch repartitions the boot
disk in place to the larger EVE-k geometry at the moment the controller pushes a
cross-flavor base-OS update — growing ESP/IMGA/IMGB, and first shrinking the
/persist filesystem when that's the only way to free the space — so an
in-field device can be converted to EVE-k without re-imaging and without losing
/persist data or device identity.

A read-only pre-flight classifies the disk into proceed / grow / shrink /
insufficient. grow happens online; shrink backs up the device identity to
the CONFIG partition and reboots into an offline shrink+grow under storage-init;
insufficient declines the conversion cleanly and the device stays on its
current flavor. The design is idempotent and fail-safe: no failure mode bricks
the device, and even a power loss mid-shrink recovers to a bootable, remotely
manageable state.

See docs/STORAGE-RESIZING.md for the full design.

What's here

  • pkg/storage-resizer — standalone Go binary doing all the disk work
    (check/backup/shrink/grow/restore/cleanup) via diskfs/partitionresizer, kept
    out of the pillar module.
  • storage-init — runs the offline shrink/grow and post-mount
    restore/cleanup, re-syncing /config so PCR14 stays stable.
  • pkg/pillar/diskconvert — pillar orchestration that execs the resizer and
    selects the next step; unit-tested behind a fake Runner.
  • baseosmgr / nodeagent / zedagent — drive the conversion at the cross-flavor
    seam, reboot into the offline shrink when needed, and report
    ZDEVICE_STATE_CONVERTING with a sub-state.
  • volumemgr / upgradeconverter — recognize blobs already on /persist across
    the kvm→k namespace change so pre-staged content isn't re-downloaded; gate the
    cross-flavor block on the actual volume set.

PR dependencies

Draft — in-flight items before merge

  • pkg/pillar/vendor hand-carries the eve-api ZDEVICE_STATE_CONVERTING /
    ZDeviceSubState change; switch to make bump-eve-api once
    info: add ZDEVICE_STATE_CONVERTING state and ZDeviceSubState eve-api#149 merges.
  • Full end-to-end test plan still needs to be executed — the on-device
    conversion (grow and shrink+grow paths, and the power-loss recovery cases) has
    not yet been run through to completion on this branch.

How to test and validate this PR

Draft — the full on-device test plan has not yet been executed (see the
in-flight items above). Intended validation:

  • Unit: pkg/pillar/diskconvert orchestration tests (fake Runner) and the
    pkg/storage-resizer check/shrink/grow logic.
  • E2E under eden + swtpm on a device installed with the old small boot-disk
    geometry: push a cross-flavor (EVE-kvm→EVE-k) base-OS update and confirm
    each pre-flight class behaves correctly — grow (online), shrink
    (offline shrink+grow under storage-init, identity backed up to CONFIG and
    restored), and insufficient (conversion declined cleanly, device stays on
    its current flavor). Confirm /persist data and device identity survive,
    and that a power loss mid-shrink recovers to a bootable, remotely
    manageable state.

Changelog notes

Adds in-field boot-disk repartitioning so a device installed with EVE's old
small boot-disk layout can be converted between EVE-kvm and EVE-k on a
cross-flavor base-OS update, without re-imaging and without losing /persist
data or device identity.

PR Backports

  • 16.0-stable: No.
  • 14.5-stable: No.
  • 13.4-stable: No.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

🤖 Generated with Claude Code

@eriknordmark eriknordmark changed the title Boot-disk repartition for in-field EVE-kvm ↔ EVE-k conversion [WIP] Boot-disk repartition for in-field EVE-kvm ↔ EVE-k conversion Jun 19, 2026
@eriknordmark
eriknordmark force-pushed the kvm-to-k-resize branch 2 times, most recently from 6858d0a to 90fa14a Compare June 20, 2026 18:37
@github-actions
github-actions Bot requested a review from jsfakian June 20, 2026 21:59
@eriknordmark
eriknordmark force-pushed the kvm-to-k-resize branch 2 times, most recently from 053e426 to 36977ab Compare June 22, 2026 12:59
@eriknordmark
eriknordmark force-pushed the kvm-to-k-resize branch 7 times, most recently from 42fd8e5 to 58e45a6 Compare July 9, 2026 22:04
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 9, 2026
@codecov

codecov Bot commented Jul 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 28.86435% with 451 lines in your changes missing coverage. Please review.
✅ Project coverage is 23.05%. Comparing base (74ffe8e) to head (31a5c88).
⚠️ Report is 38 commits behind head on master.

Files with missing lines Patch % Lines
pkg/pillar/cmd/baseosmgr/convert.go 20.90% 140 Missing ⚠️
pkg/pillar/vault/handler_zfs.go 2.22% 88 Missing ⚠️
pkg/pillar/cmd/baseosmgr/baseosmgr.go 0.00% 49 Missing ⚠️
pkg/pillar/diskconvert/convert.go 46.66% 48 Missing ⚠️
...illar/cmd/upgradeconverter/containerd_namespace.go 71.68% 20 Missing and 12 partials ⚠️
pkg/pillar/cmd/nodeagent/handlebaseos.go 0.00% 18 Missing ⚠️
pkg/pillar/cmd/nodeagent/nodeagent.go 0.00% 18 Missing ⚠️
pkg/pillar/cmd/baseosmgr/handlebaseos.go 0.00% 17 Missing ⚠️
pkg/pillar/cmd/zedagent/reportinfo.go 0.00% 13 Missing ⚠️
pkg/pillar/zfs/zfs.go 0.00% 13 Missing ⚠️
... and 3 more
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6063      +/-   ##
==========================================
+ Coverage   22.64%   23.05%   +0.41%     
==========================================
  Files         507      520      +13     
  Lines       93195    95539    +2344     
==========================================
+ Hits        21103    22030     +927     
- Misses      70367    71586    +1219     
- Partials     1725     1923     +198     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 10, 2026
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 11, 2026
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 11, 2026
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 14, 2026
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 14, 2026
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 15, 2026
Rebuild on the rebuilt resize-allprs (restructured PR stack) plus fork#7
rebased onto the new lf-edge#6063. README updated for the stress variant.
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 15, 2026
Correct the README: fork#7 (resize-watchdog-stress) is a do-not-merge
stress harness layered on top only in this branch (build base lf-edge#6063), not
a conversion-feature PR. Move it out of the feature-chain table.
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 15, 2026
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 15, 2026
Move fork#7 (resize-watchdog-stress) out of the 'Included (conversion
feature chain)' table into its own 'not part of the chain' section: it is
a stress/soak-only, do-not-merge piece layered on this branch only, using
lf-edge#6063 merely as a build base.
@eriknordmark
eriknordmark force-pushed the kvm-to-k-resize branch 3 times, most recently from dae8740 to 2a29030 Compare July 17, 2026 01:20
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 17, 2026
eriknordmark and others added 15 commits July 20, 2026 15:06
After zedagent restarts pubVolumeConfig, volumemgr has reconciled its
on-disk recovery (populateExistingVolumesFormat*) with the controller-
intended set into VolumeStatus. Signal restart on pubVolumeStatus at the
tail of handleVolumeRestart so subscribers can use Restarted() as a gate
on the published set being in a known state, rather than Synchronized()
which only reflects the socket-level handshake.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The cross-flavor BaseOs upgrade between EVE-k and non-EVE-k was
unconditionally rejected. The rationale is that /persist/vault/volumes
layout differs across flavors, so existing volumes can break across the
switch. On a device with no volumes there is nothing to disturb, so the
block is overly strict.

Subscribe to VolumeConfig (from zedagent) and VolumeStatus (from
volumemgr) in baseosmgr and only error out on the flavor mismatch when
either set is non-empty. Wait at startup for both publications to signal
restart so the first BaseOsConfig after boot evaluates against the true
volume set; Synchronized() is not sufficient because it only confirms
the socket handshake.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Pillar's containerd client uses namespace eve-user-apps on EVE-kvm and
k8s.io on EVE-k (pkg/pillar/containerd/containerd.go init()). The user
containerd's content store at /persist/vault/containerd survives the
kvm→k upgrade, but the bolt metadata DB records blob ownership per
namespace, so pillar on EVE-k sees the store as empty under k8s.io
and re-downloads every blob that EVE-kvm had previously fetched.
Each blob's on-disk file gets overwritten with bit-identical content
on a fresh content.WriteBlob; pre-staged ContentTrees burn full
registry bandwidth on every upgrade.

Add a post-vault converter that runs once on the first EVE-k boot to
copy blob metadata records from eve-user-apps to k8s.io directly in
the bolt DB at /persist/vault/containerd/io.containerd.metadata.v1.bolt.
Metadata-only, no blob copies. The post-vault phase fits cleanly: the
vault is decrypted but the user containerd has not yet been started by
the kube container, so the bolt DB is idle and safe to mutate via
bbolt (already vendored). A sentinel file at /persist/vault/
containerd/.eve-namespace-port-done short-circuits subsequent boots.
No-op on EVE-kvm and on fresh-install EVE-k that has no prior
eve-user-apps records.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
populateInitBlobStatus() reconstructs BlobStatus from the user
containerd's content store on volumemgr startup so that blobs already
on /persist (e.g. those pulled before an EVE-kvm → EVE-k upgrade) are
recognized and don't trigger a re-download. On EVE-k it filters by the
EVEDownloadedLabel because the user containerd is shared with k3s and
we must not publish BlobStatus for k3s/longhorn/etc. pod images.

Pillar's CAS writes EVEDownloadedLabel only on manifest-shaped blobs
(and on the named image entry), not on the individual layer blobs the
manifest references. The manifest carries containerd.io/gc.ref.content.*
labels that point at every layer — the same set containerd's GC uses
to keep layers alive while the manifest is reachable. Filtering by the
label per-blob therefore rejects every layer with "Ignoring the blob
... not downloaded by eve", forcing volumemgr to re-fetch each layer
even though they're physically present on /persist.

Walk those labels transitively from each directly-labeled blob to
recover the full set pillar considers its own. Foreign manifests
(k3s pod images) don't carry the label so they don't seed the walk —
no false positives for shared blobs.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
A device whose /persist is ZFS and that is converted in-field from
EVE-kvm to EVE-k carries over a vault stored as a filesystem dataset,
but EVE-k expects the vault as a zvol formatted ext4 alongside a
separate etcd-storage zvol. The carried-over filesystem vault therefore
never reached a mounted state on the first EVE-k boot, so attestation
escrow could not complete and the device reboot-looped back to kvm.

On the first EVE-k boot the unlock path now detects that the vault is
still a filesystem dataset, mounts it so the device boots, and migrates
it in place to the zvol layout: it stages a new zvol, copies the vault
contents into it (containerd content store and metadata, downloader,
verifier, and configs all live inside the vault, so blobs and content
trees come across), then destroys the old dataset and renames the new
zvol into place. An empty etcd-storage zvol is created since etcd starts
fresh on EVE-k. The staging zvol survives until the rename and remount
succeed, so an interrupted attempt is reconstructed on the next boot.

The migration helper is parameterized on encryption: the TPM path
creates the new zvols encrypted with the staged unlock key, the no-TPM
SetupDefaultVault path creates them unencrypted (matching how no-TPM ZFS
vaults are created), so both paths end up with the same zvol layout.

This covers a ZFS device carrying content trees and blobs but no app
volumes; app-volume conversion is handled separately.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Vendored eve-api proto plus the pillar types.DeviceState/DeviceSubState mirror (four sub-states: unspecified, rebooting-to-resize, installing, rebooting-to-target). Hand-carried in vendor until lf-edge/eve-api#149 merges (then make bump-eve-api).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
Standalone Go binary doing the disk work (check/backup/shrink/grow/
restore/cleanup) via diskfs/partitionresizer, plus a run-watchdog
subcommand that holds/pets /dev/watchdog across a long offline op. The
--no-pet/--attempt escalating fault-injection mode is implemented but
only used by the separate stress-test PR; the product pets. The grow
also creates the reserved second EFI System partition (ESP-B) so a
converted device reaches the same layout as a fresh EVE-k install.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Execs the storage-resizer and selects the next conversion step; unit-tested behind a fake Runner.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
On a cross-flavor (kvm<->k) base-OS update, run the read-only pre-flight, proceed/grow/shrink/decline accordingly, reboot into the offline shrink when needed, and report ZDEVICE_STATE_CONVERTING with a sub-state.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
Handle the baseosmgr-requested reboot into the early-boot offline shrink/grow during a kvm<->k conversion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
Report ZDEVICE_STATE_CONVERTING and the conversion sub-state to the controller.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
Run the offline shrink/grow and post-mount restore/cleanup under storage-init (re-syncing /config so PCR14 stays stable), feed (pet) the hardware watchdog across it, and raise the resize reboot budget. Wire the eve-storage-resizer image into pillar's and storage-init's Dockerfiles.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
Bump the vendored partitionresizer to the build carrying the
declarative Apply API (grow relocations plus create-by-offset for the
reserved ESP-B) and pin the storage-init and pillar rootfs images to
the matching eve-storage-resizer build, so the shipped binary and the
vendored library stay in agreement. Kept as an isolated commit so the
dev replace can be stripped and re-pinned to the released version in
one step once partitionresizer merges upstream.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
When a watchdog reset interrupts the offline resize2fs shrink of /persist,
the next boot's e2fsck reconnects the orphaned subtrees it repairs into
/persist/lost+found. That rescued data is no longer owned by any agent
(newlogd, the image CAS GC, ...), so nothing ever reclaims it, yet it keeps
consuming P3 space and counts against the low-disk threshold.

Report the directory's usage so the space is attributable to the controller
and a nonzero value flags that a filesystem repair occurred, and add it to
the boot-time low-disk cleanup list. It is placed last, since it may hold the
only rescued copy of the data, so an emergency low-disk boot reclaims it only
after the known-disposable directories did not free enough.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
@eriknordmark eriknordmark added the main-quest The fate of the project rests on this PR. Prioritise review to advance the storyline! label Jul 20, 2026
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 20, 2026
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 20, 2026
Rebuild on current lf-edge/eve master. Component heads refreshed to the
restacked branches (lf-edge#6036, fork#6, lf-edge#6063, fork#4); the whole stack is
layered on the same rebased bases so the merges are clean.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 20, 2026
Rebuild on current lf-edge/eve master. Component heads refreshed to the
restacked branches (lf-edge#6036, fork#6, lf-edge#6063, fork#4); the whole stack is
layered on the same rebased bases so the merges are clean.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The declarative Apply/create-path work partitionresizer was carrying on
an eriknordmark fork has merged upstream (diskfs/partitionresizer lf-edge#23),
so drop the fork replace and pin the released pseudo-version
v1.0.1-0.20260723112941-bb074d80a25a. go-diskfs advances transitively to
the revision that release pins. The storage-resizer already targets the
Apply API, so no source change is needed; the vendored tree loses the
removed Run entrypoint and keeps the chaos delay backend.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

main-quest The fate of the project rests on this PR. Prioritise review to advance the storyline!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant