diff --git a/Guide/src/reference/devices/virtio/virtio-fs.md b/Guide/src/reference/devices/virtio/virtio-fs.md index 576ee12e2bb..ca47e71c382 100644 --- a/Guide/src/reference/devices/virtio/virtio-fs.md +++ b/Guide/src/reference/devices/virtio/virtio-fs.md @@ -353,10 +353,18 @@ access, such as a handle open for writing on a file that a guest caller owns. OpenVMM releases that predate `--mount-owner` reject the device state of a `caller` attachment rather than restore it as `vmm`. -A snapshot captured without `--mount` records the slot as dormant. It may -restore without an attachment, or bind one new `--mount` attachment to the -slot, but not an aggregate. Because execution resumes after the cold-boot -mount hook, the guest must mount the newly attached backend explicitly: +A snapshot captured without `--mount` or `--mount-aggregate` records the slot +as dormant. It may restore without an attachment, or bind one new `--mount` +attachment to the slot. A platform snapshot may instead bind a new aggregate, +with `--mount-aggregate` and its `--mount-child` options, policy paths, and +`--mount-owner` mode, which OpenVMM validates as at a cold boot, before any +vCPU runs; snapshots of other tiers, and untiered snapshots, cannot. Either +attachment keeps the slot's device, whose stable ID, tag, MMIO range, +interrupt, features, and queues the snapshot fixes, and the captured kernel +command line, which has no `virtfs_*` tokens, so OpenVMM tells the guest +nothing about the new attachment. Because execution resumes after the +cold-boot mount hook, the guest must learn the attachment from its host and +mount the newly attached backend explicitly: ```bash mkdir -p /mnt/share diff --git a/Guide/src/reference/openvmm/management/cli.md b/Guide/src/reference/openvmm/management/cli.md index 95aa187ec0d..11bda46d1e3 100644 --- a/Guide/src/reference/openvmm/management/cli.md +++ b/Guide/src/reference/openvmm/management/cli.md @@ -352,9 +352,14 @@ describes the source definitions. same guest target and the same `--mount-child` options, in the same order, with the same canonical host paths, kinds, modes, and policy paths. A snapshot captured without a filesystem may remain dormant or bind one new - `--mount` to the slot. The resumed - guest must then explicitly run `mount -t virtiofs microvm ` - because its cold-boot mount hook has already completed. + `--mount` to the slot; a platform snapshot may instead bind a new + `--mount-aggregate` with its `--mount-child`, `--mount-deny`, + `--mount-allow`, `--mount-write`, and `--mount-owner` options, which + OpenVMM validates as it would at a cold boot. Either keeps the slot's device + and the captured command line, which has no `virtfs_*` token, so the resumed + guest must learn the guest target from its host and explicitly run + `mount -t virtiofs microvm `, because its cold-boot mount hook + has already completed. See [virtio-fs](../../devices/virtio/virtio-fs.md). * `--snapshot-destination `: Publish a microVM snapshot when the guest writes to PMIO port `0x605`. The destination must not exist and its parent @@ -447,8 +452,9 @@ describes the source definitions. options in the same order. It must reproduce the manifest's exact canonical host paths, guest target, and `ro`/`rw` modes while also supplying live roots with the same saved identities. A snapshot advertising the dormant - slot may instead accept one new `--mount`; snapshots without that - capability reject additive attachment. + slot may instead accept one new `--mount`, and a platform snapshot also a + new `--mount-aggregate`; snapshots without that capability reject additive + attachment. When the snapshot contains virtio-net, restore also requires `--network-profile portable`; the snapshot's profile and canonical egress diff --git a/Guide/src/user_guide/openvmm/snapshots.md b/Guide/src/user_guide/openvmm/snapshots.md index ef5678a2736..9c1e6359378 100644 --- a/Guide/src/user_guide/openvmm/snapshots.md +++ b/Guide/src/user_guide/openvmm/snapshots.md @@ -432,7 +432,9 @@ host paths, kinds, target, and modes, the same `--mount-deny`, `--mount-allow`, and `--mount-write` paths, and the same `--mount-owner` mode. OpenVMM independently validates the roots and every saved object identity before starting a vCPU. A dormant-slot snapshot may -restore without an attachment or bind one new `--mount` to the slot. +restore without an attachment or bind one new `--mount` to the slot, and a +platform snapshot may instead bind a new `--mount-aggregate`, which OpenVMM +validates as at a cold boot. For a new attachment, the resumed guest explicitly mounts tag `microvm`; the cold-boot mount hook does not run again. diff --git a/openvmm/openvmm_entry/src/cli_args/microvm.rs b/openvmm/openvmm_entry/src/cli_args/microvm.rs index 182ec4216f9..3900b794681 100644 --- a/openvmm/openvmm_entry/src/cli_args/microvm.rs +++ b/openvmm/openvmm_entry/src/cli_args/microvm.rs @@ -483,7 +483,11 @@ pub struct MicrovmCli { /// enter it, and it lists one directory or regular file per /// `--mount-child`, which has its own access mode. The guest mounts the /// `microvm` tag at GUEST_TARGET with `virtfs_aggregate=1` on its command - /// line, and is expected to bind-mount each child where it is needed. + /// line, and is expected to bind-mount each child where it is needed. A + /// platform snapshot's dormant slot may bind an aggregate on restore, + /// validated as at a cold boot; the resumed guest keeps its captured + /// command line, so it must learn of the aggregate from its host and + /// mount the `microvm` tag explicitly. #[clap( long = "mount-aggregate", value_name = "GUEST_TARGET", @@ -503,7 +507,8 @@ pub struct MicrovmCli { /// A file child's mode applies to the whole file, which no /// `--mount-deny`, `--mount-allow`, or `--mount-write` path may name. /// Host paths must not overlap: a file must not lie inside a directory - /// child. A restore requires the same children, in the same order. + /// child. A snapshot that attached an aggregate requires the same + /// children, in the same order, on restore. #[clap( long = "mount-child", value_name = "NAME,HOST_PATH[,ro|rw][,file]", diff --git a/openvmm/openvmm_entry/src/microvm/filesystem.rs b/openvmm/openvmm_entry/src/microvm/filesystem.rs index a089f45e134..ec72eeeb23d 100644 --- a/openvmm/openvmm_entry/src/microvm/filesystem.rs +++ b/openvmm/openvmm_entry/src/microvm/filesystem.rs @@ -953,7 +953,9 @@ pub(crate) fn microvm_filesystem_slot_from_snapshot( /// Returns the effective filesystem that `request` attaches. A restore must /// supply exactly the snapshot's filesystem, unless the snapshot's slot is -/// dormant, in which case one `--mount` may bind a new filesystem to it. +/// dormant, in which case the restore may bind a new filesystem to it, which +/// it validates as a cold boot does: one `--mount`, or, at the restore of a +/// platform snapshot, which `prepare_restore` checks, an aggregate. pub(super) fn effective_microvm_filesystem( request: Option>, policy: MicrovmFilesystemPolicyPaths<'_>, @@ -987,11 +989,9 @@ pub(super) fn effective_microvm_filesystem( == openvmm_helpers::snapshot::microvm::MICROVM_FILESYSTEM_SLOT_VERSION, "snapshot does not support restore-time microVM filesystem attachment" ); - // Only the dormant slot exists in the snapshot's machine. - anyhow::ensure!( - matches!(request, MicrovmFilesystemRequest::Single(_)), - "a dormant-slot snapshot can attach only a --mount on restore" - ); + // The snapshot's machine has only the dormant slot, whose device and + // command line stay as captured whatever the restore binds to it, so + // the restore builds the filesystem as a cold boot does. return microvm_filesystem_from_request(request, policy, owner).map(Some); }; let request = @@ -2006,32 +2006,225 @@ mod tests { } #[test] - fn restore_attaches_an_aggregate_only_where_the_snapshot_did() { + fn restore_attaches_an_aggregate_only_to_a_dormant_slot() { let workspace = tempfile::tempdir().unwrap(); let toolcache = tempfile::tempdir().unwrap(); let children = workspace_and_toolcache(workspace.path(), toolcache.path(), "ro"); - let restore = ["--restore-snapshot", "snapshot"]; + let options = aggregate_options(&children, &["--restore-snapshot", "snapshot"]); - // A dormant slot binds only a single directory on restore. - let error = options_filesystem( - &aggregate_options(&children, &restore), - Some(&dormant_filesystem_contract()), - ) - .unwrap_err(); - assert!(error.to_string().contains("only a --mount"), "{error:#}"); + // A dormant slot binds the aggregate. + let filesystem = + options_filesystem(&options, Some(&dormant_filesystem_contract())).unwrap(); + assert!(filesystem.config.is_aggregate()); + + // A snapshot that predates the dormant slot binds no filesystem. + let error = options_filesystem(&options, Some(&network_contract())).unwrap_err(); + assert!( + error + .to_string() + .contains("does not support restore-time microVM filesystem attachment"), + "{error:#}" + ); // A snapshot of a single directory requires that directory. - let error = options_filesystem( - &aggregate_options(&children, &restore), - Some(&filesystem_contract(workspace.path())), - ) - .unwrap_err(); + let error = + options_filesystem(&options, Some(&filesystem_contract(workspace.path()))).unwrap_err(); assert!( error.to_string().contains("requires the --mount"), "{error:#}" ); } + /// Resolves the aggregate that `children` and the access-policy options + /// `policy` request with `owner`, at a cold boot and at a restore that + /// binds it to a dormant slot. + fn cold_and_dormant_aggregates( + children: &[String], + policy: &[&str], + owner: openvmm_defs::microvm::MicrovmFilesystemOwner, + ) -> [anyhow::Result; 2] { + let mut restore = policy.to_vec(); + restore.extend(["--restore-snapshot", "snapshot"]); + let dormant = dormant_filesystem_contract(); + [ + (aggregate_options(children, policy), None), + (aggregate_options(children, &restore), Some(&dormant)), + ] + .map(|(options, contract)| { + effective_microvm_filesystem( + MicrovmFilesystemRequest::from_options(&options), + MicrovmFilesystemPolicyPaths { + denied: &options.microvm.microvm_mount_deny, + allowed: &options.microvm.microvm_mount_allow, + writable: &options.microvm.microvm_mount_write, + }, + owner, + contract, + ) + .and_then(|filesystem| filesystem.context("the options request no filesystem")) + }) + } + + #[test] + fn dormant_slot_restore_attaches_an_aggregate_as_a_cold_boot_does() { + use openvmm_defs::microvm::MicrovmFilesystemAccess::ReadOnly; + use openvmm_defs::microvm::MicrovmFilesystemAccess::ReadWrite; + use openvmm_defs::microvm::MicrovmFilesystemChildKind::Directory; + use openvmm_defs::microvm::MicrovmFilesystemChildKind::File; + + let workspace = policy_share(); + let toolcache = tempfile::tempdir().unwrap(); + let files = tempfile::tempdir().unwrap(); + let settings = files.path().join("settings.json"); + let output = files.path().join("output.txt"); + fs_err::write(&settings, b"{}").unwrap(); + fs_err::write(&output, b"").unwrap(); + let children = [ + format!("work,{},rw", workspace.path().display()), + format!("tools,{},ro", toolcache.path().display()), + format!("settings,{},ro,file", settings.display()), + format!("output,{},rw,file", output.display()), + ]; + let path = |relative: &str| workspace.path().join(relative).display().to_string(); + let (logs, payloads, out, log) = ( + path("logs"), + path("logs/payloads"), + path("out"), + path("build.log"), + ); + let policy = [ + "--mount-deny", + &logs, + "--mount-allow", + &payloads, + "--mount-write", + &out, + "--mount-write", + &log, + ]; + let [cold, restored] = cold_and_dormant_aggregates(&children, &policy, VMM); + let (cold, restored) = (cold.unwrap(), restored.unwrap()); + assert_eq!( + restored + .config + .children + .iter() + .map(|child| (child.name.as_str(), child.kind, child.access)) + .collect::>(), + [ + ("work", Directory, ReadWrite), + ("tools", Directory, ReadOnly), + ("settings", File, ReadOnly), + ("output", File, ReadWrite), + ] + ); + let work = &restored.config.children[0]; + assert_eq!(work.denied_paths, ["logs"]); + assert_eq!(work.allowed_paths, ["logs/payloads"]); + assert_eq!(work.writable_paths, ["build.log", "out"]); + // The restore resolves the roots, identities, and policy of the cold + // boot. + assert_eq!(restored.config, cold.config); + assert_eq!(restored.attachment, cold.attachment); + assert_eq!(restored.children, cold.children); + } + + #[test] + fn dormant_slot_restore_refuses_the_aggregates_that_a_cold_boot_refuses() { + let parent = tempfile::tempdir().unwrap(); + let work = parent.path().join("work"); + let nested = work.join("nested"); + fs_err::create_dir_all(&nested).unwrap(); + let settings = parent.path().join("settings.json"); + fs_err::write(&settings, b"{}").unwrap(); + let other = tempfile::tempdir().unwrap(); + let outside = tempfile::tempdir().unwrap(); + let directory = |name: &str, path: &Path| format!("{name},{},rw", path.display()); + let file = |name: &str, path: &Path| format!("{name},{},ro,file", path.display()); + let settings_path = settings.display().to_string(); + let outside_path = outside.path().display().to_string(); + for (children, policy, expected) in [ + // Host paths that overlap. + ( + vec![directory("work", &work), directory("nested", &nested)], + vec![], + "must not overlap", + ), + ( + vec![ + directory("parent", parent.path()), + file("settings", &settings), + ], + vec![], + "must not overlap", + ), + // Host objects of another kind than the child's. + ( + vec![format!("work,{},rw,file", work.display())], + vec![], + "not a regular file", + ), + ( + vec![format!("settings,{},ro", settings.display())], + vec![], + "not a plain directory", + ), + // A repeated name. + ( + vec![directory("same", &work), directory("same", other.path())], + vec![], + "not unique", + ), + // Policy paths that no child takes. + ( + vec![file("settings", &settings)], + vec!["--mount-deny", settings_path.as_str()], + "exposes the file", + ), + ( + vec![directory("work", &work)], + vec!["--mount-deny", outside_path.as_str()], + "outside the filesystem export roots", + ), + ( + vec![directory("work", &work)], + vec!["--mount-deny", "nested"], + "absolute host path", + ), + ] { + let [cold, restored] = cold_and_dormant_aggregates(&children, &policy, VMM); + let (cold, restored) = (cold.unwrap_err(), restored.unwrap_err()); + assert!(format!("{cold:#}").contains(expected), "{cold:#}"); + assert_eq!(format!("{restored:#}"), format!("{cold:#}")); + } + } + + #[test] + fn dormant_slot_restore_applies_the_owner_of_a_cold_boot() { + let work = tempfile::tempdir().unwrap(); + let children = [format!("work,{},rw", work.path().display())]; + let [cold, restored] = cold_and_dormant_aggregates( + &children, + &[], + openvmm_defs::microvm::MicrovmFilesystemOwner::Caller, + ); + // Caller ownership needs a Linux host and roots that root does not + // own, so a cold boot may refuse it here; the restore must then refuse + // it too. + match (cold, restored) { + (Ok(cold), Ok(restored)) => { + assert!(restored.config.owner.is_caller()); + assert_eq!(restored.config, cold.config); + } + (Err(cold), Err(restored)) => { + assert_eq!(format!("{restored:#}"), format!("{cold:#}")); + } + (cold, restored) => { + panic!("the cold boot and the restore disagree: {cold:?} and {restored:?}") + } + } + } + #[test] fn aggregate_restore_requires_the_same_children() { let workspace = policy_share(); diff --git a/openvmm/openvmm_entry/src/microvm/restore.rs b/openvmm/openvmm_entry/src/microvm/restore.rs index d02bdc4a699..7388ab8b0c2 100644 --- a/openvmm/openvmm_entry/src/microvm/restore.rs +++ b/openvmm/openvmm_entry/src/microvm/restore.rs @@ -201,7 +201,8 @@ pub(crate) fn restore_packet_base( /// platform snapshot's `distro` block keeps its captured capacity, whatever /// the length of the image that the restore attaches to it, and /// `--restore-absent-scratch` attaches its fresh scratch with no file. Only a -/// platform snapshot admits `--restore-network-policy`. +/// platform snapshot admits `--restore-network-policy`, and `--mount-aggregate` +/// on the filesystem slot that it left dormant. pub(crate) fn prepare_restore( opt: &mut Options, restore_snapshot: Option<&OpenedSnapshot>, @@ -245,6 +246,11 @@ pub(crate) fn prepare_restore( &manifest.snapshot_tier, opt.microvm.restore_network_policy, )?; + validate_filesystem_aggregate_restore( + &manifest.snapshot_tier, + contract.microvm_filesystem.is_some(), + opt.microvm.microvm_mount_aggregate.is_some(), + )?; if let Some(restore_processors) = opt.microvm.restore_processors { openvmm_helpers::snapshot::microvm::validate_restore_online_vp_count( manifest, @@ -415,6 +421,29 @@ fn validate_network_policy_restore( Ok(()) } +/// Rejects `--mount-aggregate`, which `aggregate` reports, on the filesystem +/// slot that a snapshot of a tier other than the platform tier left dormant, +/// which `captured_filesystem` reports when false. +/// +/// A platform snapshot is captured before any tenant input, so its restore +/// may attach the filesystem that a cold boot would. Later tiers, and +/// untiered snapshots, keep what their dormant slot admitted before, one +/// `--mount` directory, and a snapshot that captured a filesystem requires it +/// again, whatever its tier. +fn validate_filesystem_aggregate_restore( + snapshot_tier: &str, + captured_filesystem: bool, + aggregate: bool, +) -> anyhow::Result<()> { + anyhow::ensure!( + captured_filesystem + || !aggregate + || snapshot_tier == openvmm_helpers::snapshot::format::SNAPSHOT_TIER_PLATFORM, + "--mount-aggregate on a dormant filesystem slot requires a platform snapshot" + ); + Ok(()) +} + /// How a restore attaches the sandbox blocks of a snapshot. #[derive(Debug, Default, PartialEq, Eq)] struct SandboxBlockRestore<'a> { @@ -905,6 +934,37 @@ pub(crate) mod tests { } } + #[test] + fn only_a_platform_restore_attaches_an_aggregate_to_a_dormant_slot() { + use openvmm_helpers::snapshot::format::SNAPSHOT_TIER_INSTANCE_CHECKPOINT; + use openvmm_helpers::snapshot::format::SNAPSHOT_TIER_PLATFORM; + use openvmm_helpers::snapshot::format::SNAPSHOT_TIER_WORKLOAD_START; + + for tier in [ + SNAPSHOT_TIER_PLATFORM, + SNAPSHOT_TIER_WORKLOAD_START, + SNAPSHOT_TIER_INSTANCE_CHECKPOINT, + "", + ] { + // A dormant slot may stay dormant or bind a --mount, and a slot + // that the snapshot attached requires what it captured, at every + // tier. + for (captured_filesystem, aggregate) in [(false, false), (true, false), (true, true)] { + validate_filesystem_aggregate_restore(tier, captured_filesystem, aggregate) + .unwrap(); + } + let attached = validate_filesystem_aggregate_restore(tier, false, true); + if tier == SNAPSHOT_TIER_PLATFORM { + attached.unwrap(); + } else { + assert_eq!( + attached.unwrap_err().to_string(), + "--mount-aggregate on a dormant filesystem slot requires a platform snapshot" + ); + } + } + } + fn time_abi_restore(capture_generation: u32) -> MicrovmRestore { let mut contract: SnapshotMachineContract = mesh::payload::decode(&[]).unwrap(); contract.time = Some(SnapshotTimeContract { diff --git a/vm/devices/virtio/virtiofs/src/microvm/device.rs b/vm/devices/virtio/virtiofs/src/microvm/device.rs index 06f5ae33ae2..0352c0960b1 100644 --- a/vm/devices/virtio/virtiofs/src/microvm/device.rs +++ b/vm/devices/virtio/virtiofs/src/microvm/device.rs @@ -437,6 +437,78 @@ mod tests { destination.restore_device(Some(state)).unwrap(); } + #[async_test] + async fn microvm_dormant_state_restores_into_aggregate_attachment(driver: DefaultDriver) { + use crate::microvm::saved_state::SAVED_DIRECTORY_CHILD; + use crate::microvm::saved_state::SAVED_FILE_CHILD; + use crate::profile::microvm_file_identity; + + let driver_source = VmTaskDriverSource::new(SingleDriverBackend::new(driver)); + let mut source = VirtioFsDevice::new_microvm_dormant( + &driver_source, + MICROVM_ATTACHMENT_ID.to_owned(), + None, + ) + .unwrap(); + source.quiesce_input().await.unwrap(); + let state = source.save_device().unwrap().unwrap(); + + // The slot that a capture left dormant takes an aggregate of a + // read-write directory and a read-only file, as a cold boot's would. + let work = tempfile::tempdir().unwrap(); + let files = tempfile::tempdir().unwrap(); + let settings = files.path().join("settings.json"); + std::fs::write(&settings, b"{}").unwrap(); + let mut destination = VirtioFsDevice::new_microvm_aggregate( + &driver_source, + MICROVM_ATTACHMENT_ID.to_owned(), + vec![ + MicroVmAggregateChild::new( + "work".to_owned(), + microvm_root_identity(work.path()).unwrap(), + false, + Vec::new(), + Vec::new(), + Vec::new(), + ) + .unwrap(), + MicroVmAggregateChild::new_file( + "settings".to_owned(), + microvm_file_identity(&settings).unwrap(), + true, + ) + .unwrap(), + ], + MicroVmOwnerMode::Vmm, + &[work.path(), settings.as_path()], + None, + ) + .unwrap(); + destination.restore_device(Some(state)).unwrap(); + + // From then on, the device saves the aggregate rather than the + // dormant slot. + destination.quiesce_input().await.unwrap(); + let saved = destination + .save_device() + .unwrap() + .unwrap() + .parse::() + .unwrap(); + assert!(!saved.dormant); + assert_eq!( + saved + .aggregate_children + .iter() + .map(|child| (child.name.as_str(), child.kind, child.access_mode)) + .collect::>(), + [ + ("work", SAVED_DIRECTORY_CHILD, 2), + ("settings", SAVED_FILE_CHILD, 1), + ] + ); + } + #[async_test] async fn microvm_active_state_rejects_dormant_destination(driver: DefaultDriver) { let driver_source = VmTaskDriverSource::new(SingleDriverBackend::new(driver));