[WIP] Boot-disk repartition for in-field EVE-kvm ↔ EVE-k conversion - #6063
Draft
eriknordmark wants to merge 16 commits into
Draft
[WIP] Boot-disk repartition for in-field EVE-kvm ↔ EVE-k conversion#6063eriknordmark wants to merge 16 commits into
eriknordmark wants to merge 16 commits into
Conversation
github-actions
Bot
requested review from
OhmSpectator,
europaul and
milan-zededa
June 19, 2026 07:14
eriknordmark
force-pushed
the
kvm-to-k-resize
branch
2 times, most recently
from
June 20, 2026 18:37
6858d0a to
90fa14a
Compare
eriknordmark
force-pushed
the
kvm-to-k-resize
branch
2 times, most recently
from
June 22, 2026 12:59
053e426 to
36977ab
Compare
eriknordmark
force-pushed
the
kvm-to-k-resize
branch
from
July 2, 2026 17:15
643a8f4 to
5dbc563
Compare
eriknordmark
force-pushed
the
kvm-to-k-resize
branch
7 times, most recently
from
July 9, 2026 22:04
42fd8e5 to
58e45a6
Compare
eriknordmark
added a commit
to eriknordmark/eve
that referenced
this pull request
Jul 9, 2026
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #6063 +/- ##
==========================================
+ Coverage 22.64% 23.05% +0.41%
==========================================
Files 507 520 +13
Lines 93195 95539 +2344
==========================================
+ Hits 21103 22030 +927
- Misses 70367 71586 +1219
- Partials 1725 1923 +198 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
eriknordmark
added a commit
to eriknordmark/eve
that referenced
this pull request
Jul 10, 2026
eriknordmark
added a commit
to eriknordmark/eve
that referenced
this pull request
Jul 11, 2026
eriknordmark
added a commit
to eriknordmark/eve
that referenced
this pull request
Jul 11, 2026
eriknordmark
added a commit
to eriknordmark/eve
that referenced
this pull request
Jul 14, 2026
eriknordmark
added a commit
to eriknordmark/eve
that referenced
this pull request
Jul 14, 2026
eriknordmark
force-pushed
the
kvm-to-k-resize
branch
from
July 14, 2026 23:21
55c10f6 to
5f6ec99
Compare
eriknordmark
added a commit
to eriknordmark/eve
that referenced
this pull request
Jul 15, 2026
Rebuild on the rebuilt resize-allprs (restructured PR stack) plus fork#7 rebased onto the new lf-edge#6063. README updated for the stress variant.
eriknordmark
added a commit
to eriknordmark/eve
that referenced
this pull request
Jul 15, 2026
Correct the README: fork#7 (resize-watchdog-stress) is a do-not-merge stress harness layered on top only in this branch (build base lf-edge#6063), not a conversion-feature PR. Move it out of the feature-chain table.
eriknordmark
force-pushed
the
kvm-to-k-resize
branch
from
July 15, 2026 05:22
44feaa6 to
95f64b6
Compare
eriknordmark
added a commit
to eriknordmark/eve
that referenced
this pull request
Jul 15, 2026
eriknordmark
added a commit
to eriknordmark/eve
that referenced
this pull request
Jul 15, 2026
Move fork#7 (resize-watchdog-stress) out of the 'Included (conversion feature chain)' table into its own 'not part of the chain' section: it is a stress/soak-only, do-not-merge piece layered on this branch only, using lf-edge#6063 merely as a build base.
eriknordmark
force-pushed
the
kvm-to-k-resize
branch
3 times, most recently
from
July 17, 2026 01:20
dae8740 to
2a29030
Compare
eriknordmark
added a commit
to eriknordmark/eve
that referenced
this pull request
Jul 17, 2026
After zedagent restarts pubVolumeConfig, volumemgr has reconciled its on-disk recovery (populateExistingVolumesFormat*) with the controller- intended set into VolumeStatus. Signal restart on pubVolumeStatus at the tail of handleVolumeRestart so subscribers can use Restarted() as a gate on the published set being in a known state, rather than Synchronized() which only reflects the socket-level handshake. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The cross-flavor BaseOs upgrade between EVE-k and non-EVE-k was unconditionally rejected. The rationale is that /persist/vault/volumes layout differs across flavors, so existing volumes can break across the switch. On a device with no volumes there is nothing to disturb, so the block is overly strict. Subscribe to VolumeConfig (from zedagent) and VolumeStatus (from volumemgr) in baseosmgr and only error out on the flavor mismatch when either set is non-empty. Wait at startup for both publications to signal restart so the first BaseOsConfig after boot evaluates against the true volume set; Synchronized() is not sufficient because it only confirms the socket handshake. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Pillar's containerd client uses namespace eve-user-apps on EVE-kvm and k8s.io on EVE-k (pkg/pillar/containerd/containerd.go init()). The user containerd's content store at /persist/vault/containerd survives the kvm→k upgrade, but the bolt metadata DB records blob ownership per namespace, so pillar on EVE-k sees the store as empty under k8s.io and re-downloads every blob that EVE-kvm had previously fetched. Each blob's on-disk file gets overwritten with bit-identical content on a fresh content.WriteBlob; pre-staged ContentTrees burn full registry bandwidth on every upgrade. Add a post-vault converter that runs once on the first EVE-k boot to copy blob metadata records from eve-user-apps to k8s.io directly in the bolt DB at /persist/vault/containerd/io.containerd.metadata.v1.bolt. Metadata-only, no blob copies. The post-vault phase fits cleanly: the vault is decrypted but the user containerd has not yet been started by the kube container, so the bolt DB is idle and safe to mutate via bbolt (already vendored). A sentinel file at /persist/vault/ containerd/.eve-namespace-port-done short-circuits subsequent boots. No-op on EVE-kvm and on fresh-install EVE-k that has no prior eve-user-apps records. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
populateInitBlobStatus() reconstructs BlobStatus from the user containerd's content store on volumemgr startup so that blobs already on /persist (e.g. those pulled before an EVE-kvm → EVE-k upgrade) are recognized and don't trigger a re-download. On EVE-k it filters by the EVEDownloadedLabel because the user containerd is shared with k3s and we must not publish BlobStatus for k3s/longhorn/etc. pod images. Pillar's CAS writes EVEDownloadedLabel only on manifest-shaped blobs (and on the named image entry), not on the individual layer blobs the manifest references. The manifest carries containerd.io/gc.ref.content.* labels that point at every layer — the same set containerd's GC uses to keep layers alive while the manifest is reachable. Filtering by the label per-blob therefore rejects every layer with "Ignoring the blob ... not downloaded by eve", forcing volumemgr to re-fetch each layer even though they're physically present on /persist. Walk those labels transitively from each directly-labeled blob to recover the full set pillar considers its own. Foreign manifests (k3s pod images) don't carry the label so they don't seed the walk — no false positives for shared blobs. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
A device whose /persist is ZFS and that is converted in-field from EVE-kvm to EVE-k carries over a vault stored as a filesystem dataset, but EVE-k expects the vault as a zvol formatted ext4 alongside a separate etcd-storage zvol. The carried-over filesystem vault therefore never reached a mounted state on the first EVE-k boot, so attestation escrow could not complete and the device reboot-looped back to kvm. On the first EVE-k boot the unlock path now detects that the vault is still a filesystem dataset, mounts it so the device boots, and migrates it in place to the zvol layout: it stages a new zvol, copies the vault contents into it (containerd content store and metadata, downloader, verifier, and configs all live inside the vault, so blobs and content trees come across), then destroys the old dataset and renames the new zvol into place. An empty etcd-storage zvol is created since etcd starts fresh on EVE-k. The staging zvol survives until the rename and remount succeed, so an interrupted attempt is reconstructed on the next boot. The migration helper is parameterized on encryption: the TPM path creates the new zvols encrypted with the staged unlock key, the no-TPM SetupDefaultVault path creates them unencrypted (matching how no-TPM ZFS vaults are created), so both paths end up with the same zvol layout. This covers a ZFS device carrying content trees and blobs but no app volumes; app-volume conversion is handled separately. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Vendored eve-api proto plus the pillar types.DeviceState/DeviceSubState mirror (four sub-states: unspecified, rebooting-to-resize, installing, rebooting-to-target). Hand-carried in vendor until lf-edge/eve-api#149 merges (then make bump-eve-api). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Standalone Go binary doing the disk work (check/backup/shrink/grow/ restore/cleanup) via diskfs/partitionresizer, plus a run-watchdog subcommand that holds/pets /dev/watchdog across a long offline op. The --no-pet/--attempt escalating fault-injection mode is implemented but only used by the separate stress-test PR; the product pets. The grow also creates the reserved second EFI System partition (ESP-B) so a converted device reaches the same layout as a fresh EVE-k install. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Execs the storage-resizer and selects the next conversion step; unit-tested behind a fake Runner. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
On a cross-flavor (kvm<->k) base-OS update, run the read-only pre-flight, proceed/grow/shrink/decline accordingly, reboot into the offline shrink when needed, and report ZDEVICE_STATE_CONVERTING with a sub-state. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Handle the baseosmgr-requested reboot into the early-boot offline shrink/grow during a kvm<->k conversion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Report ZDEVICE_STATE_CONVERTING and the conversion sub-state to the controller. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Run the offline shrink/grow and post-mount restore/cleanup under storage-init (re-syncing /config so PCR14 stays stable), feed (pet) the hardware watchdog across it, and raise the resize reboot budget. Wire the eve-storage-resizer image into pillar's and storage-init's Dockerfiles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Bump the vendored partitionresizer to the build carrying the declarative Apply API (grow relocations plus create-by-offset for the reserved ESP-B) and pin the storage-init and pillar rootfs images to the matching eve-storage-resizer build, so the shipped binary and the vendored library stay in agreement. Kept as an isolated commit so the dev replace can be stripped and re-pinned to the released version in one step once partitionresizer merges upstream. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
When a watchdog reset interrupts the offline resize2fs shrink of /persist, the next boot's e2fsck reconnects the orphaned subtrees it repairs into /persist/lost+found. That rescued data is no longer owned by any agent (newlogd, the image CAS GC, ...), so nothing ever reclaims it, yet it keeps consuming P3 space and counts against the low-disk threshold. Report the directory's usage so the space is attributable to the controller and a nonzero value flags that a filesystem repair occurred, and add it to the boot-time low-disk cleanup list. It is placed last, since it may hold the only rescued copy of the data, so an emergency low-disk boot reclaims it only after the known-disposable directories did not free enough. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
eriknordmark
force-pushed
the
kvm-to-k-resize
branch
from
July 20, 2026 22:07
2a29030 to
31a5c88
Compare
eriknordmark
added a commit
to eriknordmark/eve
that referenced
this pull request
Jul 20, 2026
eriknordmark
added a commit
to eriknordmark/eve
that referenced
this pull request
Jul 20, 2026
Rebuild on current lf-edge/eve master. Component heads refreshed to the restacked branches (lf-edge#6036, fork#6, lf-edge#6063, fork#4); the whole stack is layered on the same rebased bases so the merges are clean. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
eriknordmark
added a commit
to eriknordmark/eve
that referenced
this pull request
Jul 20, 2026
Rebuild on current lf-edge/eve master. Component heads refreshed to the restacked branches (lf-edge#6036, fork#6, lf-edge#6063, fork#4); the whole stack is layered on the same rebased bases so the merges are clean. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The declarative Apply/create-path work partitionresizer was carrying on an eriknordmark fork has merged upstream (diskfs/partitionresizer lf-edge#23), so drop the fork replace and pin the released pseudo-version v1.0.1-0.20260723112941-bb074d80a25a. go-diskfs advances transitively to the revision that release pins. The storage-resizer already targets the Apply API, so no source change is needed; the vendored tree loses the removed Run entrypoint and keeps the chaos delay backend. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Boot-disk repartition for in-field EVE-kvm ↔ EVE-k conversion
Long-lived devices installed with EVE's old small boot-disk layout (IMGA/IMGB
as small as 300 MB, a small ESP) cannot hold the EVE-k (Kubernetes) rootfs and
its container/k3s tooling. Upgrading replaces rootfs contents but never
repartitions, so the small geometry persists. This branch repartitions the boot
disk in place to the larger EVE-k geometry at the moment the controller pushes a
cross-flavor base-OS update — growing ESP/IMGA/IMGB, and first shrinking the
/persistfilesystem when that's the only way to free the space — so anin-field device can be converted to EVE-k without re-imaging and without losing
/persistdata or device identity.A read-only pre-flight classifies the disk into
proceed/grow/shrink/insufficient.growhappens online;shrinkbacks up the device identity tothe CONFIG partition and reboots into an offline shrink+grow under storage-init;
insufficientdeclines the conversion cleanly and the device stays on itscurrent flavor. The design is idempotent and fail-safe: no failure mode bricks
the device, and even a power loss mid-shrink recovers to a bootable, remotely
manageable state.
See
docs/STORAGE-RESIZING.mdfor the full design.What's here
pkg/storage-resizer— standalone Go binary doing all the disk work(check/backup/shrink/grow/restore/cleanup) via
diskfs/partitionresizer, keptout of the pillar module.
storage-init— runs the offline shrink/grow and post-mountrestore/cleanup, re-syncing
/configso PCR14 stays stable.pkg/pillar/diskconvert— pillar orchestration that execs the resizer andselects the next step; unit-tested behind a fake Runner.
seam, reboot into the offline shrink when needed, and report
ZDEVICE_STATE_CONVERTINGwith a sub-state./persistacrossthe kvm→k namespace change so pre-staged content isn't re-downloaded; gate the
cross-flavor block on the actual volume set.
PR dependencies
volumemgr: persist deferred content-tree deletes) — the conversionrelies on the deferred content-tree delete to preserve CAS blobs across the
conversion reboot, so an app/base image isn't re-downloaded after the kvm→k
switch.
ZDEVICE_STATE_CONVERTING/ZDeviceSubState(hand-carried until it merges; see below).
Draft — in-flight items before merge
pkg/pillar/vendorhand-carries the eve-apiZDEVICE_STATE_CONVERTING/ZDeviceSubStatechange; switch tomake bump-eve-apionceinfo: add ZDEVICE_STATE_CONVERTING state and ZDeviceSubState eve-api#149 merges.
conversion (grow and shrink+grow paths, and the power-loss recovery cases) has
not yet been run through to completion on this branch.
How to test and validate this PR
Draft — the full on-device test plan has not yet been executed (see the
in-flight items above). Intended validation:
pkg/pillar/diskconvertorchestration tests (fake Runner) and thepkg/storage-resizercheck/shrink/grow logic.geometry: push a cross-flavor (EVE-kvm→EVE-k) base-OS update and confirm
each pre-flight class behaves correctly —
grow(online),shrink(offline shrink+grow under storage-init, identity backed up to CONFIG and
restored), and
insufficient(conversion declined cleanly, device stays onits current flavor). Confirm
/persistdata and device identity survive,and that a power loss mid-shrink recovers to a bootable, remotely
manageable state.
Changelog notes
Adds in-field boot-disk repartitioning so a device installed with EVE's old
small boot-disk layout can be converted between EVE-kvm and EVE-k on a
cross-flavor base-OS update, without re-imaging and without losing /persist
data or device identity.
PR Backports
Checklist
And the last but not least:
check them.
🤖 Generated with Claude Code