Skip to content

snapshot: align NVX manifest with upstream SaveVM/RestoreVM #1210

Open

Description

Objective

Track the NVX-side snapshot manifest alignment with microsoft/openvmm#4553 (SaveVM/RestoreVM), following the coordination reply.

We are willing to break compatibility with existing NVX snapshots and require recapture. Preserving NVX's current assignment of tag 8 is therefore not a requirement. The exact replacement tag assignments still need upstream confirmation; this issue does not record them as agreed.

Current baseline and proposed mapping

At the analyzed OpenVMM pin 0ac254109ed32fc5fa2b1a6fc526a10593b1f986, NVX uses manifest version 6, magic OPENVMM_SNAPSHOT_V6\0, and tag 8 for state_size_bytes: u64. microsoft/openvmm#4553 currently uses tag 8 for snapshot_id: Guid. Those fields have incompatible wire types.

The reply proposes this shared mapping:

Tag Field Status
8 snapshot_id Accept upstream's existing assignment.
18 state_size_bytes Requested for NVX; awaiting confirmation.
19 compatibility_bundle_digest Proposed upstream reservation.
20 memory_artifacts Proposed upstream reservation.

NVX's tags 9 and 10 are permanently retired; tags 11-17 already hold its other manifest fields. Do not reuse retired tags or independently assign a tag that upstream has chosen for another field. If upstream instead lands snapshot_id at 18, reconcile the mapping before implementation rather than also assigning 18 to state_size_bytes.

Work

  • Confirm the final shared tag map with the authors of ttrpc: add SaveVM and RestoreVM APIs openvmm#4553, including the reservations for planned fields. This issue does not require inventing semantics for fields that upstream has not implemented yet.
  • Make the source changes in the nested OpenVMM repository: relocate state_size_bytes as agreed, integrate the upstream snapshot ID, and update all affected full-VM and microVM manifest writers, readers, validation, fixtures, and tests.
  • Bump the manifest version and magic together for the incompatible layout change (v6 to v7 if v6 remains the current format). Preserve the size validation and other existing NVX snapshot invariants.
  • Reject unsupported old formats with an actionable [E_SNAPSHOT_VERSION] / recapture diagnostic. In particular, recognize an old v6 manifest before attempting to decode its integer tag 8 as a GUID, using a version-first/minimal-header decode or an equivalent explicit format-dispatch mechanism.
  • Decide and document the missing-ID policy in coordination with upstream. An optional GUID can handle a missing field in pre-ID upstream manifests, but does not solve NVX v6's tag-8 wire-type mismatch. Supporting restore of old NVX v6 snapshots is not required.
  • Update snapshot format/compatibility documentation and describe the intentional recapture requirement.
  • Once the OpenVMM changes are validated and available upstream, promote the resulting revision into NVX and update the OpenVMM upstream roadmap. Keep the gitlink promotion and roadmap update in separate commits under the repository's promotion workflow.

Acceptance criteria

  • The finalized manifest has unique, coordinated tags, and retired tags 9-10 remain unused.
  • New-format manifests round-trip, and affected save/restore paths retain NVX's state-size checks and other snapshot validation.
  • A serialized old v6 manifest containing integer tag 8 gets the explicit unsupported-version/recapture diagnostic, not a generic GUID/protobuf decode error.
  • Tests cover a manifest with no snapshot ID according to the agreed policy, and a malformed or missing ID in the new format is handled explicitly.
  • Existing v6 snapshots are documented as requiring recapture; no automatic conversion or backwards-compatible restore is promised.
  • Relevant OpenVMM package-scoped checks and affected NVX validation pass, with any unavailable platform coverage reported rather than assumed.

Source references

Scope is the manifest migration and diagnostics discussed in the reply, not unifying the full-VM RPC and guest-coordinated microVM capture protocols.

Activity

  1. ppenna commented on Nov 9, 2026

    @ppenna
    ContributorAuthor

    Plan update for this issue, as the NVX pull request #1417 implements it with microsoft/openvmm#4994.

    What changed since the issue was filed

    • Upstream moved snapshot_id to field 18 instead of taking field 8 (reply), and ttrpc: add SaveVM and RestoreVM APIs openvmm#4553's head still has it there, with a comment that reserves field 19 for compatibility_bundle_digest and field 20 for memory_artifacts. That is the case for which this issue asked to reconcile the mapping first rather than move state_size_bytes: it keeps field 8, and no field moves.
    • NVX's v4.0 chain reorganization (openvmm: pin microVM v4.0, reorganized onto upstream main #1238) restarted the manifest at version 2, with the magic OPENVMM_SNAPSHOT_V2\0, and numbered its fields 8-15 without gaps. Version 6, which this issue analyzed, is now an earlier format.

    The shared field map is now:

    Fields Owner
    1-7 upstream's manifest
    8-15 NVX: state_size_bytes, machine_contract, format_magic, saved_state_schema_version, saved_state_root_type, snapshot_tier, restore_policy, consumed_config_sections
    16-17 free
    18 upstream's snapshot_id (microsoft/openvmm#4553)
    19, 20 reserved upstream for compatibility_bundle_digest and memory_artifacts

    What #1417 does

    • No field moves and the manifest stays version 2, so no snapshot needs to be recaptured.
    • The diagnostic that this issue asked for was still missing, for a reason that the renumbering created. Restore decoded the whole manifest before it checked the version and magic, and earlier formats give some fields other types, so a version 6 snapshot that NVX release v0.1.0-dev.304f8e1686b4 captured failed to restore on KVM, MSHV, and WHP with failed to decode snapshot manifest: decoding failed: expected a varint. Restore now decodes the version alone, then the magic alone, and refuses any other with [E_SNAPSHOT_VERSION] snapshot manifest version 6 is not supported (expected 2); recapture the snapshot before it decodes the rest. The first manifests of version 2, whose fields 9 and 10 held artifact digests, are refused by their missing magic.
    • The manifest's declaration and the Guide's rule for extending it reserve fields 18-20.
    • Fields 9 and 10, which this issue wanted retired, keep the meaning that the v4.0 reorganization gave them: renumbering again would invalidate every current snapshot, and the version-then-magic check refuses the manifests that used them otherwise before it decodes them.
    • Existing snapshots of earlier releases need recapture, as doc/run.md documents; the pull request adds no conversion or backward-compatible restore.

    What remains here, once microsoft/openvmm#4553 merges and the chain rebases onto it:

    • Adopt upstream's snapshot_id (field 18) in the chain's SnapshotManifest, with upstream's policy for a missing ID (a zero GUID means none), and tests of a manifest without an ID and with a malformed one.
    • In the same pin promotion, let nvx-rs's reader of a template's manifest (nvx-rs/src/host/template/manifest.rs), which refuses fields that it does not know, accept field 18.
  2. ppenna commented on Nov 10, 2026

    @ppenna
    ContributorAuthor

    Status update: microsoft/openvmm#4553 has merged, but the chain has not been rebased onto it yet, so the two tasks left here wait for the next rebase of the chain.

    Upstream

    microsoft/openvmm#4553 merged into upstream main on 2026-11-10 as 64828d2b5, with the field map above:

    • SnapshotManifest gains snapshot_id: guid::Guid in field 18, with ZeroCopyEncoding, so a save writes it as 16 bytes, and a comment reserves field 19 for compatibility_bundle_digest and field 20 for memory_artifacts.
    • Missing-ID policy: a manifest without field 18 decodes with the zero GUID, which means that it has no ID. A field 18 whose length is not 16 bytes fails the manifest's decode (invalid byte size for type).
    • Each save records a new random ID, which the REPL logs and SaveVM returns. When the caller expects an ID, validate_expectations refuses a manifest with another ID or with none; RestoreVM expects one only when expected_snapshot_id is given and not empty.
    • write_snapshot now refuses a destination that exists, writes state.bin, links memory.bin, writes manifest.bin last, and removes what it created when a save fails.

    The chain

    As of 2026-11-10, feature/microvm/dev, at 58d3c3b1d, NVX's pin on dev, still sits on the fork base ba105dd7f (2026-11-06), and upstream main, at ecb13803a, is 22 commits past it, 64828d2b5 among them. The chain's SnapshotManifest has fields 1-15, with state_size_bytes still in field 8, and its comment keeps fields 18-20 free, so upstream's field fits without moving another.

    What remains here, for the next rebase of the chain onto an upstream main that includes 64828d2b5, as the roadmap plans it:

    • Adopt upstream's snapshot_id (field 18) in the chain's SnapshotManifest, with upstream's missing-ID policy, and tests of a manifest without an ID and with a malformed one. The rebase also reconciles upstream's write_snapshot with the chain's publication, which PR11's positions 031-033 and later commits change, and settles the ID that each of the chain's manifest writers records: the full-VM save in openvmm/openvmm_entry/src/vm_controller.rs, to which upstream's change gives a random ID, and the microVM capture in openvmm/openvmm_entry/src/vm_controller/microvm.rs, which upstream does not have.
    • In NVX's pin of that rebase, let nvx-rs's reader of a template's manifest (nvx-rs/src/host/template/manifest.rs), which refuses fields that it does not know, accept field 18 as 16 bytes and refuse any other length, as OpenVMM's decode does.

    No pull request until then; this issue stays open for those two tasks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area: snapshotInitial title-keyword triage suggestion

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions