Repository navigation
Description
Summary
In microVM virtio-fs, each open directory handle caches its enumerated directory entries on first read_dir / read_dir_plus subject only to per-handle limits (MAX_DIRECTORY_ENTRIES_PER_HANDLE = 4096, MAX_DIRECTORY_SNAPSHOT_BYTES = 64 KiB), across up to MAX_HANDLES = 4096 open handles. However, VirtioFs::save_microvm_state rejects VM snapshot capture if the sum across all open handles exceeds MAX_DIRECTORY_ENTRIES = 16384 or MAX_DIRECTORY_BYTES = 4 MiB, so a guest that simply opens and reads 5 directories of 4,096 entries (or >64 directory handles with 64 KiB of entry names) succeeds at runtime and causes host microVM save/snapshot capture to fail. Found by static code inspection.
Root Cause
saved_state.rs defines per-handle directory snapshot caps whose worst-case product with MAX_HANDLES (4096 * 4096 = 16,777,216 entries and 4096 * 64 KiB = 256 MiB of entry names) is orders of magnitude larger than the aggregate saved-state caps:
pub(crate) const MAX_HANDLES: usize = 4096;
...
pub(crate) const MAX_DIRECTORY_ENTRIES_PER_HANDLE: usize = 4096;
pub(crate) const MAX_DIRECTORY_ENTRY_NAME_BYTES: usize = 255;
pub(crate) const MAX_DIRECTORY_SNAPSHOT_BYTES: usize = 64 * 1024;
pub(crate) const MAX_DIRECTORY_ENTRIES: usize = 16 * 1024;
pub(crate) const MAX_DIRECTORY_BYTES: usize = 4 * 1024 * 1024;At runtime, VirtioFsFile::read_dir_microvm populates self.directory_snapshot via VirtioFsFile::build_microvm_directory_snapshot, which checks only the single handle's entry count and byte total without tracking or bounding aggregate usage across VirtioFs::inner.files:
self.file.write().read_dir(0, |entry| {
if entries.len() == MAX_DIRECTORY_ENTRIES_PER_HANDLE {
return Err(lx::Error::E2BIG);
}
...
total_bytes = total_bytes
.checked_add(name.len())
.ok_or(lx::Error::E2BIG)?;
if total_bytes > MAX_DIRECTORY_SNAPSHOT_BYTES {
return Err(lx::Error::E2BIG);
}The aggregate caps MAX_DIRECTORY_ENTRIES (16384) and MAX_DIRECTORY_BYTES (4 MiB) are checked only when the host captures device state in VirtioFs::save_microvm_state (invoked from microvm::device::save_device):
directory_entry_count = directory_entry_count
.checked_add(snapshot_entries.len())
.context("saved directory entry count overflow")?;
anyhow::ensure!(
directory_entry_count <= MAX_DIRECTORY_ENTRIES,
"saved directory entries exceed the aggregate bound"
);
for entry in &snapshot_entries {
directory_bytes = directory_bytes
.checked_add(entry.name.len())
.context("saved directory bytes overflow")?;
}
anyhow::ensure!(
directory_bytes <= MAX_DIRECTORY_BYTES,
"saved directory entries exceed the aggregate byte bound"
);Consequence
Both benign guest workloads and untrusted guest processes can cause host-initiated microVM save/snapshot operations (save_device -> save_microvm_state) to fail deterministically with SaveError::Other("saved directory entries exceed the aggregate bound") or SaveError::Other("saved directory entries exceed the aggregate byte bound"):
- Holding open 5 directory handles that each enumerate 4,096 entries (including
.and.., so 4,094 files — or even opening 5 concurrent directory handles on the same read-only host directory containing >=3,277 entries and reading offset0on each) caches5 * 4096 = 20,480 > 16,384entries, succeeding at runtime and breaking snapshot capture even on a read-only share. - Holding open 65 directory handles whose cached entry names total 64 KiB each caches
65 * 64 KiB = 4,160 KiB > 4 MiB, likewise succeeding at runtime and failingsave_microvm_state. - Without releasing handles, an untrusted guest can also accumulate up to
4,096 * 64 KiB = 256 MiBof cachedSavedDirectoryEntryname buffers (plusVecand struct overhead across up to16,777,216entries) in VMM heap memory, bypassing the intended 4 MiB / 16,384-entry microVM resource budget at runtime.
Reproduction
- Create a host directory containing 4,094 files (yielding 4,096 directory entries including
.and..) and attach it toVirtioFs::new_microvm(even withread_only = true). - Issue
FUSE_OPENDIR5 times onFUSE_ROOT_ID(obtaining 5 distinct open directory handles) and issueFUSE_READDIRwithoffset = 0once on each handle: all 5FUSE_OPENDIRandFUSE_READDIRrequests succeed, caching5 * 4,096 = 20,480entries acrossself.inner.files. - Call
fs.save_microvm_state(&profile, SessionState::default()): snapshot capture fails with"saved directory entries exceed the aggregate bound"(20480 > MAX_DIRECTORY_ENTRIES).
Suggested Fix
Maintain running aggregate counters (or an atomic/mutex-guarded reservation tracker on VirtioFsInner) for total cached directory entries and total cached directory name bytes across all live handles in self.inner.files. Reserve against MAX_DIRECTORY_ENTRIES and MAX_DIRECTORY_BYTES when read_dir_microvm builds and commits a handle's DirectorySnapshot (returning lx::Error::E2BIG or lx::Error::ENOSPC before storing snapshot.entries if committing would exceed either aggregate bound), seed those counters when restore_microvm_state populates restored_files, and decrement them when a VirtioFsFile with a built snapshot is removed on release / release_dir or dropped on destroy.
Activity