Skip to content

openvmm: virtiofs: microVM directory snapshot aggregate bounds are enforced only at save, allowing guests to break VM snapshot capture #1613

Open

Description

@tangruize

Summary

In microVM virtio-fs, each open directory handle caches its enumerated directory entries on first read_dir / read_dir_plus subject only to per-handle limits (MAX_DIRECTORY_ENTRIES_PER_HANDLE = 4096, MAX_DIRECTORY_SNAPSHOT_BYTES = 64 KiB), across up to MAX_HANDLES = 4096 open handles. However, VirtioFs::save_microvm_state rejects VM snapshot capture if the sum across all open handles exceeds MAX_DIRECTORY_ENTRIES = 16384 or MAX_DIRECTORY_BYTES = 4 MiB, so a guest that simply opens and reads 5 directories of 4,096 entries (or >64 directory handles with 64 KiB of entry names) succeeds at runtime and causes host microVM save/snapshot capture to fail. Found by static code inspection.

Root Cause

saved_state.rs defines per-handle directory snapshot caps whose worst-case product with MAX_HANDLES (4096 * 4096 = 16,777,216 entries and 4096 * 64 KiB = 256 MiB of entry names) is orders of magnitude larger than the aggregate saved-state caps:

pub(crate) const MAX_HANDLES: usize = 4096;
...
pub(crate) const MAX_DIRECTORY_ENTRIES_PER_HANDLE: usize = 4096;
pub(crate) const MAX_DIRECTORY_ENTRY_NAME_BYTES: usize = 255;
pub(crate) const MAX_DIRECTORY_SNAPSHOT_BYTES: usize = 64 * 1024;
pub(crate) const MAX_DIRECTORY_ENTRIES: usize = 16 * 1024;
pub(crate) const MAX_DIRECTORY_BYTES: usize = 4 * 1024 * 1024;

At runtime, VirtioFsFile::read_dir_microvm populates self.directory_snapshot via VirtioFsFile::build_microvm_directory_snapshot, which checks only the single handle's entry count and byte total without tracking or bounding aggregate usage across VirtioFs::inner.files:

        self.file.write().read_dir(0, |entry| {
            if entries.len() == MAX_DIRECTORY_ENTRIES_PER_HANDLE {
                return Err(lx::Error::E2BIG);
            }
            ...
            total_bytes = total_bytes
                .checked_add(name.len())
                .ok_or(lx::Error::E2BIG)?;
            if total_bytes > MAX_DIRECTORY_SNAPSHOT_BYTES {
                return Err(lx::Error::E2BIG);
            }

The aggregate caps MAX_DIRECTORY_ENTRIES (16384) and MAX_DIRECTORY_BYTES (4 MiB) are checked only when the host captures device state in VirtioFs::save_microvm_state (invoked from microvm::device::save_device):

                directory_entry_count = directory_entry_count
                    .checked_add(snapshot_entries.len())
                    .context("saved directory entry count overflow")?;
                anyhow::ensure!(
                    directory_entry_count <= MAX_DIRECTORY_ENTRIES,
                    "saved directory entries exceed the aggregate bound"
                );
                for entry in &snapshot_entries {
                    directory_bytes = directory_bytes
                        .checked_add(entry.name.len())
                        .context("saved directory bytes overflow")?;
                }
                anyhow::ensure!(
                    directory_bytes <= MAX_DIRECTORY_BYTES,
                    "saved directory entries exceed the aggregate byte bound"
                );

Consequence

Both benign guest workloads and untrusted guest processes can cause host-initiated microVM save/snapshot operations (save_device -> save_microvm_state) to fail deterministically with SaveError::Other("saved directory entries exceed the aggregate bound") or SaveError::Other("saved directory entries exceed the aggregate byte bound"):

  • Holding open 5 directory handles that each enumerate 4,096 entries (including . and .., so 4,094 files — or even opening 5 concurrent directory handles on the same read-only host directory containing >=3,277 entries and reading offset 0 on each) caches 5 * 4096 = 20,480 > 16,384 entries, succeeding at runtime and breaking snapshot capture even on a read-only share.
  • Holding open 65 directory handles whose cached entry names total 64 KiB each caches 65 * 64 KiB = 4,160 KiB > 4 MiB, likewise succeeding at runtime and failing save_microvm_state.
  • Without releasing handles, an untrusted guest can also accumulate up to 4,096 * 64 KiB = 256 MiB of cached SavedDirectoryEntry name buffers (plus Vec and struct overhead across up to 16,777,216 entries) in VMM heap memory, bypassing the intended 4 MiB / 16,384-entry microVM resource budget at runtime.

Reproduction

  1. Create a host directory containing 4,094 files (yielding 4,096 directory entries including . and ..) and attach it to VirtioFs::new_microvm (even with read_only = true).
  2. Issue FUSE_OPENDIR 5 times on FUSE_ROOT_ID (obtaining 5 distinct open directory handles) and issue FUSE_READDIR with offset = 0 once on each handle: all 5 FUSE_OPENDIR and FUSE_READDIR requests succeed, caching 5 * 4,096 = 20,480 entries across self.inner.files.
  3. Call fs.save_microvm_state(&profile, SessionState::default()): snapshot capture fails with "saved directory entries exceed the aggregate bound" (20480 > MAX_DIRECTORY_ENTRIES).

Suggested Fix

Maintain running aggregate counters (or an atomic/mutex-guarded reservation tracker on VirtioFsInner) for total cached directory entries and total cached directory name bytes across all live handles in self.inner.files. Reserve against MAX_DIRECTORY_ENTRIES and MAX_DIRECTORY_BYTES when read_dir_microvm builds and commits a handle's DirectorySnapshot (returning lx::Error::E2BIG or lx::Error::ENOSPC before storing snapshot.entries if committing would exceed either aggregate bound), seed those counters when restore_microvm_state populates restored_files, and decrement them when a VirtioFsFile with a built snapshot is removed on release / release_dir or dropped on destroy.

Related: #1611, #1626.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: openvmmInitial title-keyword triage suggestionarea: snapshotInitial title-keyword triage suggestion

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions