Skip to content

[WIP] tests/update_eve_image: kvm↔k cross-HV conversion + resize test suite - #1209

Draft
eriknordmark wants to merge 10 commits into
lf-edge:masterfrom
eriknordmark:kvm-to-k-eden-tests
Draft

[WIP] tests/update_eve_image: kvm↔k cross-HV conversion + resize test suite#1209
eriknordmark wants to merge 10 commits into
lf-edge:masterfrom
eriknordmark:kvm-to-k-eden-tests

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

End-to-end eden tests for the EVE-kvm ↔ EVE-k cross-flavor BaseOs upgrade
and the in-field boot-disk repartition (small → large GPT geometry). They
verify that OCI blobs are reused across the flavor switch (no re-download),
that a deployed app and its volume survive the conversion and a
controller-initiated reboot, that resize variants actually grow the
partitions, and that the TPM-sealed vault stays locally unlocked across the
geometry change.

All scenarios live under tests/update_eve_image/testdata/, each a
self-contained escript with helper scripts and READMEs:

  • update_eve_image_cross_hv.txt — base cross-HV upgrade; IMGB reaches
    active on the alternate flavor.
  • update_eve_image_cross_hv_with_contenttree.txt — pre-stages a
    ContentTree, asserts zero bytes re-downloaded after the switch.
  • update_eve_image_cross_hv_with_app_recreate.txt — app + volume survive
    the switch and a reboot with no re-download.
  • update_eve_image_kvm_to_k{,_grow,_resize,_refused,_volmig}.txt
    repartition variants asserting SMALL→LARGE partition growth,
    BaseOsStatus.Converting device state, blob reuse, local vault unlock
    (VaultStatus.UnlockMethod), and (volmig) live app-volume migration to a
    Longhorn PVC. The volmig leg covers both a VM app's disk volume and a
    container app's data volume; verify reads the container marker from the
    migrated block device, because EVE-k (kubevirt) does not auto-mount a
    container's data volume at its MountDir — tracked as HV=k: container app data volumes not mounted at MountDir when the app has no cloud-init user-data eve#6145.
  • update_eve_image_kvm_to_k_geom.txt — geometry-only matrix, parameterized
    by starting release, asserting the boot disk ends at the full EVE-k target
    2+2+10+10 (ESP-A 2G, reserved ESP-B 2G at GPT Make it possible to use pre-built EVE artifacts (instead of building local ones) #7, IMGA/IMGB 10G).
  • update_eve_image_kvm_to_k_persist_wipe_restore.txt — persist-wipe restore:
    offline identity + WiFi-cred decrypt after a /persist loss.
  • update_eve_image_kvm_to_k_backup_corrupt_restore.txt — backup-corruption
    restore: per-type validity gate + .bak fallback.

Disk-topology setup for the repartition / two-disk / ZFS cases is
prep-kvm-to-k-topology.sh (ext4-shrink, ext4-grow, twodisk-zfs,
twodisk-ext4, ext4-toofull, zfs-grow/zfs-notail).

The whole matrix can be driven in one command with run-kvm-to-k-tests.sh: it
reclaims the single-tenant eden slot before each leg, brings the topology up via
prep-kvm-to-k-topology.sh, runs the matching escript, and prints a PASS/FAIL
summary. RESIZE_EVE_VER=<tag> selects the image under test (required),
BRINGUP_EVE_VER the small-layout start release (default 12.1.0), and
ONLY=/SKIP= narrow to a subset — e.g. ONLY=twodisk-zfs,zfs-grow.

Supersedes #1196 and #1203, folding both into one branch. References
lf-edge/eve#6145.

@eriknordmark

Copy link
Copy Markdown
Contributor Author

kvm↔k conversion + resize matrix — full run 7/7 PASS

Ran the complete tests/update_eve_image/testdata/run-kvm-to-k-tests.sh matrix end-to-end on host eden (eden + swtpm, eve.tpm=true).

  • Branch: kvm-to-k-eden-tests @ 9eb853a — the test-suite fix this run validates is folded into 9a3fab4 ("tests/update_eve_image: kvm->k conversion test")
  • EVE image: lfedge/eve:0.0.0-resize-allprs-6e714932-{kvm,k} (local build), small-start bringup release 12.1.0
  • Date: 2026-07-24
===================== SUMMARY =====================
  image: lfedge/eve:0.0.0-resize-allprs-6e714932-{kvm,k}   bringup: 12.1.0
  ext4-shrink    PASS
  ext4-grow      PASS
  twodisk-ext4   PASS
  twodisk-zfs    PASS
  zfs-grow       PASS
  ext4-toofull   PASS
  zfs-notail     PASS
  ----
  PASS=7  FAIL=0  SKIP=0

Fix validated by this run

An earlier run of the same matrix had ext4-shrink FAIL and twodisk-zfs FAIL(prep) from a cross-leg state leak: the single-disk-ZFS legs write eve_install_zfs_with_raid_level into grub.cfg, which eden only rewrites when the certs dir lacks root-certificate.pem. The token therefore persisted into later non-ZFS legs (and across whole runs, since zfs-notail runs last), so those legs booted /persist as ZFS — mis-topologying the ext4 legs (shrink failed "persist is ZFS"; grow false-passed on the wrong fs) and colliding twodisk-zfs's zpool create … persist /dev/sdb against the accidental boot-disk pool.

9a3fab4 fixes it: every non-ZFS-boot leg wipes certs + adam/redis volumes before setup so a token-free grub.cfg is regenerated (P3 defaults to ext4), then asserts /run/eve.persist_type (after SSH is up) so a mis-topology fails loudly instead of false-passing. This 7/7 run is against that fix.

eriknordmark and others added 10 commits July 28, 2026 14:24
End-to-end eden tests for the EVE-kvm<->EVE-k cross-flavor BaseOs upgrade on
released images: a bare upgrade with no volumes (both directions), a pre-staged
ContentTree surviving the switch, and an app plus its volume surviving a
delete/redeploy across kvm->k and a controller-initiated device reboot with no
blob re-download. They run on released images with no local build, so they are
registered in the update_eve_image manifest.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
One parameterized escript for the in-field boot-disk repartition (SMALL->LARGE
GPT geometry) driven through the controller. It asserts the boot disk reaches the
full EVE-k 2+2+10+10 layout including the reserved ESP-B, that the conversion is
observed via device state, that the TPM-sealed vault stays locally unlocked across
the geometry change, and that a deferred-delete app's cached blobs are reused on
redeploy. EXPECT_DECISION selects the shrink vs grow variant and DISK_TOPOLOGY the
single/two-disk/zfs data-preservation invariant; the shrink path can pre-fill
/persist so the offline resize relocates real blocks. prep-kvm-to-k-topology.sh
brings eden up in each disk topology, run-kvm-to-k-tests.sh drives the matrix, and
README_kvm_to_k.md documents the image build, bringup, and knobs.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Covers the declined conversion. When the boot disk cannot free the space the
EVE-k layout needs -- an ext4 /persist too full to shrink, or a ZFS persist
(shrinking ZFS is unsupported) -- storage-resizer's check returns insufficient,
the kvm->k update is refused with BaseOsStatus.Error, the geometry is left
unchanged, and the device stays reachable and manageable. REFUSE_REASON selects
the too-full vs zfs case.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Proves an app instance and its data volume survive the kvm->k conversion without
recreate -- the qcow2 volume migrates to a Longhorn PVC with written data intact
-- for both a VM app and a container app. It starts on the LARGE geometry so the
conversion takes the no-shrink proceed path, and asserts the reserved ESP-B is
present, a precondition the large-start image must already satisfy.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Geometry-only conversion test, parameterized by the starting release. From every
historical boot-disk geometry a fielded device may carry, the conversion must
reach the full EVE-k target: ESP-A 2G, a reserved ESP-B 2G at partition 7, and
IMGA/IMGB 10G. It deploys no app and checks no vault, so it needs neither a TPM
nor a /config settle.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fault case for the conversion's identity fail-safe: force a full /persist loss
and verify the identity and connectivity-critical files are restored from the
/config backup during early boot with no controller reachable, including the
offline decrypt of a controller-encrypted credential from the restored ecdh cert.
It does not run a real shrink or grow.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sibling fault case to the persist-wipe restore test: partially corrupt the
backed-up critical files by truncating each, then verify the device still recovers
with no controller reachable via restore's per-type validity gate plus pillar's
.bak fallback.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Native EVE-k first boot (no conversion) with an app volume already present in the
config. Exercises lf-edge/eve #6121: volumemgr defers EVE-k volume creation until
cluster storage (the longhorn StorageClass and CDI control plane) is ready, so an
app requested before the cluster is up converges to RUNNING rather than parking in
CREATING_VOLUME. Also asserts the boot disk carries the reserved ESP-B.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The single-tenant guard refused to bring up a topology whenever any
qemu-system process with a -drive argument was running, so an unrelated VM
from another framework (evetest, a manual QEMU) was enough to make the
script exit 1 and claim an eden instance was live. Match eden's own QEMU
control-socket name instead -- eden always passes "<context>-qmp.sock"
regardless of eden.root/EDEN_HOME -- so a leaked eden QEMU is still caught
while foreign VMs, whose socket is a plain "qmp.sock", are ignored. The
refusal message now says which QEMU it means.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two restore escripts carry descriptive names, but their internals still
used the old fault-id naming: /tmp/f9-* and /tmp/f11-* scratch files, the
/tmp/f9cfg and /tmp/f9chk|f11chk mount points, an "f9-test" marker in
resize-failed.json, the injected WiFi SSID eden-f9 with password
eden-f9-psk, and prose calling the sibling test "F9". That kept dead
identifiers in circulation and invited them back into new material.

Rename them all onto a restore-* / eden-restore scheme. The injected SSID
and PSK values change with it; every producer and consumer of each token
moved together, so the tests remain self-consistent. F9/F11 stay valid only
as item numbers in the e2e test plan, alongside the other Fx entries.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@eriknordmark
eriknordmark force-pushed the kvm-to-k-eden-tests branch from 29346e1 to a370788 Compare July 28, 2026 21:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant