Skip to content

openvmm: virtiofs: inode deduplication key omits host device number (st_dev) #1604

Open

Description

@tangruize

Summary

On volumes that report stable file IDs, VirtioFs deduplicates inodes in InodeMap by (volume_id, inode_nr) instead of (volume_id, device_nr, inode_nr). When a single shared directory tree spans multiple host st_dev values — such as nested bind mounts or btrfs subvolumes, where every subvolume root has st_ino == 256 on its own anonymous st_dev — two distinct host objects with the same st_ino collide onto one FUSE node ID and overwrite that node's stored host path. Found by testing (also blocks formal verification).

Root Cause

In vm/devices/virtio/virtiofs/src/inode.rs, DedupKey::Ino omits stat.device_nr, and VirtioFsInode::dedup_key keys stable-ID volumes solely by (self.volume_id(), self.inode_nr()):

#[derive(Clone, PartialEq, Eq, Hash)]
pub(crate) enum DedupKey {
    /// `(volume_id, host_inode_number)`, for stable-id volumes.
    Ino(u32, lx::ino_t),
    /// `(volume_id, host_path)`, for volumes that recycle inode numbers.
    Path(u32, PathBuf),
}
    pub(crate) fn dedup_key(&self) -> DedupKey {
        if self.volume.supports_stable_file_id() {
            return DedupKey::Ino(self.volume_id(), self.inode_nr());
        }
        DedupKey::Path(self.volume_id(), self.get_path().clone())
    }

When a lookup resolves a second host object with the same st_ino and file type on a different st_dev, InodeMap::insert takes the Entry::Occupied branch and calls VirtioFsInode::lookup, replacing existing.path with new_path and returning existing's node_id.

Consequence

In POSIX, file identity is (st_dev, st_ino), not st_ino alone. On btrfs, unprivileged users can create subvolumes whose root directory always has st_ino == 256 with a distinct st_dev; container rootfs layers, bind mounts, and nested mounts inside a shared directory produce the same condition on ext4/xfs/overlayfs. If the share root is a btrfs subvolume (FUSE_ROOT_ID == 1, st_ino == 256, path == "") and contains a nested subvolume sub (st_ino == 256, path == "sub"), looking up sub returns nodeid == 1 (producing a directory cycle in the guest VFS) and overwrites the root inode's stored path from "" to "sub", causing subsequent lookups at the share root to resolve inside "sub" and making entries outside "sub" disappear with ENOENT. Colliding regular files across subvolumes or nested mounts likewise alias onto one FUSE node ID, misdirecting path-based operations (getattr, open, setattr, unlink, and microVM subtree policy checks) to whichever path was looked up most recently, and causing microVM save_microvm_state to fail when validate_saved_identity detects that a single inode's recorded aliases have different device_nr values.

Reproduction

  1. On a Linux host with btrfs (for example by running cargo nextest with TMPDIR on a btrfs filesystem), create a btrfs subvolume root, create a nested btrfs subvolume root/sub, and create a regular file root/only_in_root.
  2. Instantiate VirtioFs::new(&root_path, None) so FUSE_ROOT_ID (1) refers to root (st_ino == 256).
  3. Call fs.lookup(&request(FUSE_ROOT_ID), lx::LxStr::from_bytes(b"sub")). It erroneously returns sub.nodeid == FUSE_ROOT_ID (1) and rewrites node 1's path from "" to "sub".
  4. Call fs.lookup(&request(FUSE_ROOT_ID), lx::LxStr::from_bytes(b"only_in_root")). It fails with ENOENT because VirtioFs resolves "sub/only_in_root" instead of "only_in_root".
  5. Without btrfs, constructing two VirtioFsInode values via VirtioFsInode::with_attr with (device_nr: 1, inode_nr: 256) at "a" and (device_nr: 2, inode_nr: 256) at "b" and inserting both into InodeMap reproduces the same node ID collision deterministically on any host filesystem.

Related: #1605, #1619, #1602.

Activity

  1. tangruize commented on Nov 15, 2026

    @tangruize
    ContributorAuthor

    Suggested Fix

    Store device_nr: u64 (initialized from stat.device_nr in VirtioFsInode::with_attr, which is 0 on Windows where st_dev is unused) on VirtioFsInode, change DedupKey::Ino(u32, lx::ino_t) to DedupKey::Ino(u32, u64, lx::ino_t) representing (volume_id, device_nr, inode_nr), and update VirtioFsInode::dedup_key to include self.device_nr(). Any other lookup into inodes_by_key that constructs DedupKey::Ino directly from an lx::Stat should likewise pass stat.device_nr.

    Patch against OpenVMM 9d64daad3cbe14fb0591081b73c6c9648e18d0fb. Save the contents below as fix.patch and run git apply fix.patch.

    fix.patch
    diff --git a/vm/devices/virtio/virtiofs/src/inode.rs b/vm/devices/virtio/virtiofs/src/inode.rs
    index 6ec061ce3..cc542363d 100644
    --- a/vm/devices/virtio/virtiofs/src/inode.rs
    +++ b/vm/devices/virtio/virtiofs/src/inode.rs
    @@ -113,15 +113,17 @@ fn deref(&self) -> &Self::Target {
     /// Key used to deduplicate inodes in the `InodeMap` so that repeated lookups of
     /// the same host file return a single, stable FUSE node id.
     ///
    -/// Volumes that report stable inode numbers key by `(volume_id, inode_nr)`.
    -/// Volumes that recycle inode numbers (FAT/exFAT) cannot trust the inode
    -/// number as an identity, so they key by `(volume_id, path)` instead — a given
    -/// path then maps to one node id across lookups while its file type is unchanged,
    -/// even though the host inode number is not reliable.
    +/// Volumes that report stable inode numbers key by `(volume_id, device_nr,
    +/// inode_nr)`, matching POSIX `(st_dev, st_ino)` identity across nested mounts
    +/// and btrfs subvolumes within a volume. Volumes that recycle inode numbers
    +/// (FAT/exFAT) cannot trust the inode number as an identity, so they key by
    +/// `(volume_id, path)` instead — a given path then maps to one node id across
    +/// lookups while its file type is unchanged, even though the host inode number
    +/// is not reliable.
     #[derive(Clone, PartialEq, Eq, Hash)]
     pub(crate) enum DedupKey {
    -    /// `(volume_id, host_inode_number)`, for stable-id volumes.
    -    Ino(u32, lx::ino_t),
    +    /// `(volume_id, host_device_number, host_inode_number)`, for stable-id volumes.
    +    Ino(u32, u64, lx::ino_t),
         /// `(volume_id, host_path)`, for volumes that recycle inode numbers.
         Path(u32, PathBuf),
     }
    @@ -132,6 +134,7 @@ pub struct VirtioFsInode {
         pub(crate) path: RwLock<PathBuf>,
         pub(crate) aliases: RwLock<BTreeSet<PathBuf>>,
         pub(crate) lookup_count: AtomicU64,
    +    device_nr: u64,
         inode_nr: lx::ino_t,
         file_type: lx::mode_t,
         /// This inode's number as reported to the guest: its namespaced inode
    @@ -158,12 +161,18 @@ pub fn with_attr(volume: Arc<VirtioFsVolume>, path: PathBuf, stat: &lx::Stat) ->
                 path: RwLock::new(path),
                 aliases: RwLock::new(aliases),
                 lookup_count: AtomicU64::new(1),
    +            device_nr: stat.device_nr,
                 inode_nr: stat.inode_nr,
                 file_type: stat.mode & lx::S_IFMT,
                 guest_inode_nr,
             }
         }
     
    +    /// Return the host device number reported by `lstat` (`0` on Windows).
    +    pub(crate) fn device_nr(&self) -> u64 {
    +        self.device_nr
    +    }
    +
         /// Return the files inode number as reported by the underlying file system.
         ///
         /// N.B. This may be different from its FUSE node ID.
    @@ -557,7 +566,7 @@ pub(crate) fn exchange_alias_prefixes(&self, first: &Path, second: &Path) {
         /// The key used to deduplicate this inode in the `InodeMap`, so that
         /// repeated lookups of the same host file return one stable FUSE node id.
         ///
    -    /// Stable-id volumes key by `(volume_id, inode_nr)`. Volumes that recycle
    +    /// Stable-id volumes key by `(volume_id, device_nr, inode_nr)`. Volumes that recycle
         /// inode numbers (FAT/exFAT) key by `(volume_id, path)` instead, so a given
         /// path maps to a stable FUSE node id while its file type is unchanged. This includes
         /// the empty path of a volume root, so an aggregate child root keeps one
    @@ -565,7 +574,7 @@ pub(crate) fn exchange_alias_prefixes(&self, first: &Path, second: &Path) {
         /// each time.
         pub(crate) fn dedup_key(&self) -> DedupKey {
             if self.volume.supports_stable_file_id() {
    -            return DedupKey::Ino(self.volume_id(), self.inode_nr());
    +            return DedupKey::Ino(self.volume_id(), self.device_nr(), self.inode_nr());
             }
             DedupKey::Path(self.volume_id(), self.get_path().clone())
         }
    @@ -945,4 +954,42 @@ fn path_keyed_reserved_insert_file_to_directory() {
                 assert_eq!(inodes.insert(same).unwrap().1, current_id);
             }
         }
    +
    +    /// Two host objects in one volume can share `st_ino` when a nested mount
    +    /// or btrfs subvolume gives each its own `st_dev`. Each `(st_dev, st_ino)`
    +    /// pair must receive its own FUSE node ID.
    +    #[test]
    +    fn one_inode_number_on_two_devices_is_two_nodes() {
    +        let tmp = tempfile::tempdir().unwrap();
    +        std::fs::write(tmp.path().join("a"), b"from-device-1").unwrap();
    +        std::fs::write(tmp.path().join("b"), b"from-device-2").unwrap();
    +
    +        let volume = Arc::new(VirtioFsVolume::new(
    +            LxVolume::new(tmp.path()).unwrap(),
    +            0,
    +            false,
    +        ));
    +        assert!(volume.supports_stable_file_id());
    +
    +        let mut stat_a = volume.lstat(Path::new("a")).unwrap();
    +        stat_a.device_nr = 1;
    +        stat_a.inode_nr = 256;
    +        let mut stat_b = volume.lstat(Path::new("b")).unwrap();
    +        stat_b.device_nr = 2;
    +        stat_b.inode_nr = 256;
    +
    +        let mut inodes = InodeMap::new(false);
    +        let (root, _) = VirtioFsInode::new(Arc::clone(&volume), PathBuf::new()).unwrap();
    +        let _ = inodes.insert(root).unwrap();
    +
    +        let a = VirtioFsInode::with_attr(Arc::clone(&volume), PathBuf::from("a"), &stat_a);
    +        let (a_inode, a_id) = inodes.insert(a).unwrap();
    +
    +        let b = VirtioFsInode::with_attr(Arc::clone(&volume), PathBuf::from("b"), &stat_b);
    +        let (b_inode, b_id) = inodes.insert(b).unwrap();
    +
    +        assert_ne!(a_id, b_id, "distinct (st_dev, st_ino) objects collided");
    +        assert_eq!(&*a_inode.get_path(), Path::new("a"));
    +        assert_eq!(&*b_inode.get_path(), Path::new("b"));
    +    }
     }
    diff --git a/vm/devices/virtio/virtiofs/src/lib.rs b/vm/devices/virtio/virtiofs/src/lib.rs
    index 71780f585..d940f3013 100644
    --- a/vm/devices/virtio/virtiofs/src/lib.rs
    +++ b/vm/devices/virtio/virtiofs/src/lib.rs
    @@ -1990,4 +1990,46 @@ fn handle_map_reuses_latest_released_reservation() {
             handles.release_reservation(&reservation);
             assert_eq!(handles.insert("last"), Some(u64::MAX));
         }
    +
    +    /// Every btrfs subvolume root has `st_ino == 256` with its own `st_dev`.
    +    /// Sharing a subvolume that contains a nested subvolume must keep the two
    +    /// directories distinct instead of overwriting the root node's path.
    +    #[cfg(target_os = "linux")]
    +    #[test]
    +    fn nested_subvolume_is_not_the_root() {
    +        let outer = tempfile::tempdir().unwrap();
    +        let root_path = outer.path().join("root");
    +        let sub_path = root_path.join("sub");
    +        let create_subvolume = |path: &Path| {
    +            std::process::Command::new("btrfs")
    +                .args(["subvolume", "create"])
    +                .arg(path)
    +                .output()
    +                .is_ok_and(|out| out.status.success())
    +        };
    +        if !create_subvolume(&root_path) || !create_subvolume(&sub_path) {
    +            eprintln!("skipping: unprivileged `btrfs subvolume create` is unavailable");
    +            return;
    +        }
    +        std::fs::write(root_path.join("only_in_root"), b"root-content").unwrap();
    +
    +        let fs = VirtioFs::new(&root_path, None).unwrap();
    +        let sub = fs
    +            .lookup(&request(FUSE_ROOT_ID), lx::LxStr::from_bytes(b"sub"))
    +            .unwrap();
    +        assert_ne!(sub.nodeid, FUSE_ROOT_ID, "nested subvolume aliased root");
    +
    +        let found = fs
    +            .lookup(
    +                &request(FUSE_ROOT_ID),
    +                lx::LxStr::from_bytes(b"only_in_root"),
    +            )
    +            .expect("root entry disappeared after looking up nested subvolume");
    +        assert_ne!(found.nodeid, FUSE_ROOT_ID);
    +
    +        // Clean up subvolumes inside `outer` before `TempDir` removes it.
    +        let _ = std::fs::remove_file(root_path.join("only_in_root"));
    +        let _ = std::fs::remove_dir(&sub_path);
    +        let _ = std::fs::remove_dir(&root_path);
    +    }
     }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: openvmmInitial title-keyword triage suggestion

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions