From bac1cdd7434c5de07377936b400775c2896c2e9e Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Sun, 5 Jul 2026 10:58:41 -0700 Subject: [PATCH 01/39] docs(uffs-mft): NTFS full-volume capture plan Design for capturing a complete NTFS metafile set (MFT + $Secure/$UsnJrnl/ $Bitmap/$LogFile/$Boot/$UpCase/...) into a verifiable, transferable bundle: two-phase live+VSS-frozen model (no read-only remount), a `capture` subcommand as the home, a rust-script VSS/zip wrapper, and the `sysinfo` host-probe that gates best-effort by OS class (client/server). Co-Authored-By: Claude Opus 4.8 --- docs/architecture/mft-full-capture.md | 252 ++++++++++++++++++++++++++ 1 file changed, 252 insertions(+) create mode 100644 docs/architecture/mft-full-capture.md diff --git a/docs/architecture/mft-full-capture.md b/docs/architecture/mft-full-capture.md new file mode 100644 index 000000000..e44c495ae --- /dev/null +++ b/docs/architecture/mft-full-capture.md @@ -0,0 +1,252 @@ + + + +# NTFS Full-Volume Capture (`uffs-mft capture`) + +> **Status:** Draft / plan (pre-implementation) +> **Owner:** UFFS core +> **Goal:** Capture *everything* needed to reconstitute a live Windows NTFS volume +> offline "as accurately as possible" — not just the namespace, but ACLs, the +> change journal, free-space map, and transaction log — into one verifiable, +> transferable bundle per drive. Future-proof: capture now, use later. + +--- + +## 1. Intent & scope + +Today UFFS captures the MFT (`.iocp`, `.raw`, `.bin`, `.compressed.bin`) plus +derived listings (`cpp_*.txt`, `rust_*.txt`). That is an *Everything-class* +name/size/timestamp snapshot. This feature extends capture to a **complete +NTFS metafile set** so an offline machine can reconstitute the volume's +namespace **and** its security, temporal, and allocation state. + +Non-goals: file *contents* (data clusters) and non-resident directory index +buffers — the namespace is rebuilt from each record's `$FILE_NAME` parent ref, +so index buffers and file data are never needed. + +## 2. Capture model — two phases, no read-only mode + +A VSS shadow copy is **read-only**, and write-protected volumes reject IOCP I/O +(see `reader/dataframe_read.rs`, `reader/index_read.rs`). Therefore the +authentic **live IOCP ingestion flow** and a **frozen consistent snapshot** are +mutually exclusive *sources* and must be captured in two phases. VSS **replaces** +the old "set drives read-only" step entirely. + +| Phase | Source | Artifacts | Property | +|---|---|---|---| +| **Live** | live volume `\\.\X:` | `.iocp`, `cpp_x.txt`, `rust_x.txt` | authentic ingestion order (regression/debug) | +| **Frozen** | VSS shadow of `X:` | `$MFT` raw+compressed, `$Secure:$SDS`, `$UsnJrnl:$J`, `$Bitmap`, `$LogFile`, `$Boot`, `$UpCase`, `$AttrDef`, `$Volume`, `$MFTMirr`, `$BadClus`, `$Extend\*` | one crash-consistent instant | + +Applies to **every** drive (each: live `.iocp` + its own shadow for the frozen +set). No drive is ever remounted read-only. + +### 2.1 Best-effort by host capability + +Capture **probes the host first** (`uffs-mft sysinfo`, §5) and does *what the +machine allows*, then **records exactly what it did**. The authoritative +client/server discriminator is the registry value +`HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\InstallationType` +(`"Client"` | `"Server"`), corroborated by `GetProductInfo` / `EditionID`. + +| Host | Shadow (frozen metafiles, Rust) | C++ golden on shadow | Live `.iocp` | Net | +|---|---|---|---|---| +| **Server** | ✅ Rust reads shadow device path | ✅ `diskshadow expose %s% Z:` → `uffs.com --drives=Z` (skew-free) | ✅ live volume | Full: consistent frozen set **and** skew-free parity | +| **Client** | ✅ Rust reads shadow device path | ❌ no easy raw-volume exposure → keep C++ **live** | ✅ live volume | Best-effort: frozen metafiles from shadow; parity baselines live (skew reconciled) | +| **No VSS / not elevated** | ❌ | ❌ | ✅ (if elevated) | Degraded: live `.iocp` + listings only; frozen set skipped, **and the skip is recorded** | + +The manifest records, **per drive**, what was captured vs. skipped and *why*, so +an offline consumer never mistakes a capability gap for missing data. + +## 3. Artifact catalog + +| FRS | Metafile / stream | File | ~Size (1 TB) | Irreducible because | +|---|---|---|---|---| +| 0 | `$MFT` | `x_mft.iocp` (live) + `x_mft.compressed.bin` (frozen) | 464 M | namespace + tree metrics; `.iocp` also carries ingestion order + `reserved_allocated_bytes` | +| 9 | `$Secure:$SDS` | `x_secure.sds` | few–tens M | ACLs/owner/DACL+SACL; MFT holds only a `SecurityId` index. `$SDH`/`$SII` rebuildable from `$SDS` | +| 11 | `$Extend\$UsnJrnl:$J` (+`$Max`) | `x_usn.jrnl` | 32 M–1 G | change flow / temporal dimension | +| 6 | `$Bitmap` | `x_bitmap.bin` | ~32 M | authoritative free-space (approx. from dataruns only) | +| 2 | `$LogFile` | `x_logfile.bin` | 64–256 M | crash-consistent metadata replay | +| 7 | `$Boot` | `x_boot.bin` | 8 K | geometry + volume serial (identity, offsets) | +| 10 | `$UpCase` | `x_upcase.bin` | 128 K | exact case-folding for offline name collation (`--upcase` exists) | +| 4 | `$AttrDef` | `x_attrdef.bin` | 2.5 K | generic attribute-type interpretation | +| 3 | `$Volume` | `x_volume.bin` | tiny | label, NTFS version, dirty flag | +| 1 | `$MFTMirr` | `x_mftmirr.bin` | 4 K | corruption cross-check | +| 8 | `$BadClus` | `x_badclus.bin` | ~0 (sparse) | bad-cluster map | +| 11 | `$Extend\$ObjId`,`$Quota`,`$Reparse` | `x_extend_*.bin` | KB–MB | link tracking, quotas, reparse index | + +Frozen additions total ≈ 0.2–1.5 GB (dominated by `$UsnJrnl` + `$LogFile`). + +## 4. VSS orchestration (rust-script wrapper) + +`scripts/capture.rs` (rust-script, deps via the `//! ```cargo` block like +`verify_parity.rs`). **No PowerShell.** + +1. **Create shadow** via WMI `Win32_ShadowCopy::Create(Volume, "ClientAccessible")` + using the **`wmi` crate** (encapsulates COM; works on Win client *and* server). + Note: `vssadmin create` is Server-only; `diskshadow` may be absent on client — + WMI is the portable path. +2. Read back `DeviceObject` → `\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopyN`. +3. **Frozen phase:** `uffs-mft capture --volume-path --frozen --out `. +4. **Live phase:** `uffs-mft capture --drive X --live --out ` (`.iocp` + + optional cpp/rust baselines). +5. **Delete shadow** (WMI `Win32_ShadowCopy.Delete` / `Win32_ShadowCopy` instance). +6. **Package** (§7). + +Requires elevation (shadow create + raw volume read). The Access Broker path +still applies for the non-elevated daemon, but capture is an admin operation. + +## 5. CLI surface + +Home = the crate (tested, lint-covered, broker-aware, reuses datarun/fixup/format +code). New per-metafile `save` targets + one orchestrating `capture` subcommand: + +``` +# host + drive environment probe — runs FIRST, gates best-effort, records the fact +uffs-mft sysinfo --out capture_host.txt [--json] # also embedded in manifest.json + +# individual targets (compose the frozen set; each writes a self-describing header) +uffs-mft save --secure --volume-path -o x_secure.sds +uffs-mft save --usn --drive X -o x_usn.jrnl +uffs-mft save --bitmap --volume-path -o x_bitmap.bin +uffs-mft save --logfile --volume-path -o x_logfile.bin +uffs-mft save --boot | --attrdef | --volume | --mftmirr | --badclus | --upcase + +# orchestrator (the entry point the rust-script calls) +uffs-mft capture --drive X --out ~/uffs_data \ + [--all] [--live] [--frozen] [--volume-path ] \ + [--iocp] [--forensic] [--zip] [--split-size 1GiB] +``` + +`VolumeHandle::open` gains a `--volume-path ` mode so the frozen phase +reads the shadow device directly (offset math from the shadow's own `$Boot`). + +## 6. On-disk layout + manifest + +``` +~/uffs_data/drive_x/ + x_mft.iocp x_secure.sds x_boot.bin + x_mft.compressed.bin x_usn.jrnl x_upcase.bin + x_bitmap.bin x_logfile.bin x_attrdef.bin … + cpp_x.txt rust_x.txt *.log + capture_host.txt manifest.json SHA256SUMS +``` + +`capture_host.txt` — human-readable descriptor of the machine the capture ran on: + +```text +UFFS Capture Host Report + Captured: 2026-07-05T18:22:04Z (America/Los_Angeles, UTC-07:00) + Tool: uffs-mft 0.5.x + Host: MACHINE-NAME (workgroup/domain) + OS: Windows 11 Pro 24H2 build 10.0.26100 x64 + InstallType: Client <-- gates VSS/C++-on-shadow strategy + Elevated: yes + VSS service: running (shadow create: permitted) + CPU / RAM: 24 cores / 64 GB + Capture mode: best-effort(client) = live .iocp + shadow(frozen, Rust) ; C++ golden live + + Drives (NTFS): + C: NVMe 931 GB (312 GB free) serial 0x…. MFT 4547 MB / 5.03M recs / 28 extents + iocp:✅ frozen(shadow):✅ cpp/rust:live usn:✅ secure:✅ bitmap:✅ logfile:✅ + D: SSD 465 GB (…) … + iocp:✅ frozen(shadow):✅ … + S: HDD 11.0 TB (…) … + iocp:✅ frozen(shadow):✅ … +``` + +`manifest.json` (self-describing, drives offline `load` + verification): + +```json +{ + "schema": 1, + "drive": "C", + "captured_at": "2026-07-05T18:22:04Z", + "tool_version": "uffs-mft 0.5.x", + "host": { + "machine": "MACHINE-NAME", "os": "Windows 11 Pro 24H2", + "build": "10.0.26100", "arch": "x64", + "installation_type": "Client", "edition_id": "Professional", + "elevated": true, "vss": { "service": "running", "create_permitted": true }, + "cpu_cores": 24, "ram_bytes": 68719476736, + "timezone": "America/Los_Angeles", "utc_offset_minutes": -420, + "capture_mode": "best-effort(client)" + }, + "volume": { "serial": "0x….", "ntfs_version": "3.1", "media_type": "NVMe", + "bytes_per_cluster": 4096, "mft_record_size": 1024 }, + "vss": { "used": true, "device": "\\\\?\\GLOBALROOT\\Device\\…ShadowCopy7", + "created_at": "…", "snapshot_id": "{GUID}" }, + "phases": { "live": ["x_mft.iocp","cpp_x.txt","rust_x.txt"], + "frozen": ["x_mft.compressed.bin","x_secure.sds","x_usn.jrnl","…"] }, + "artifacts": [ { "file":"x_mft.iocp","frs":0,"stream":"$MFT", + "bytes":486539264,"sha256":"…","compression":"zstd" }, … ] +} +``` + +Each binary artifact keeps a small typed header (magic / version / volume-serial / +timestamp / drive), consistent with the existing `.iocp` and `$UpCase` headers. + +## 7. Packaging: `--zip` + 1 GiB split (best practice) + +- **No double-compression:** ZIP entry method = `store` for already-compressed + (`.iocp`, `.compressed.bin`), `deflate` for text baselines + `$Bitmap` + + `$LogFile`. +- **Single archive, then raw-split** into fixed parts — *not* spanned ZIP + (Explorer can't extract `.z01`). Output: `drive_x.zip.001 … NNN`. +- **`--split-size` default 1 GiB.** Reassemble: `copy /b drive_x.zip.* drive_x.zip` + (Win) or `cat drive_x.zip.* > drive_x.zip` (Mac/Linux). +- **`SHA256SUMS`** covers the whole zip **and** every part → verify each chunk + post-transfer, then re-verify after reassembly. +- Implemented in the rust-script via the `zip` + `sha2` crates. + +## 8. Offline symmetry + +Every capture target gains a matching `uffs-mft load` path (as `save`↔`load`, +`save_iocp`↔`load_iocp_capture` already do), so the offline machine reconstitutes +from the same crate that created the artifacts. `verify_parity.rs --regenerate` +continues to regenerate the Rust listing from `x_mft.iocp`/`.compressed.bin` and +compare against the `cpp_x.txt` golden. `$Secure`/`$UsnJrnl`/`$Bitmap`/`$LogFile` +loaders are additive (namespace path unchanged). + +## 9. Crate placement + +- Metafile readers → `uffs-mft::ntfs::metafiles` (new module) reusing + `VolumeHandle`, datarun resolution, and fixup code. No new `unsafe`. +- Capture file formats → `uffs-mft::raw` (alongside `raw_iocp.rs`). +- `capture` subcommand → `uffs-mft::commands::windows::capture`. +- Orchestration + VSS + packaging → `scripts/capture.rs` (rust-script). + +## 10. Testing + +- Fixture/golden round-trip per artifact (`save` → `load` → structural equal). +- Header (de)serialization unit tests (as `raw_iocp` has). +- Manifest schema + SHA256SUMS verification test. +- Split/reassemble round-trip test (byte-identical). +- Live-only paths `#[ignore]` (elevated Windows), fixtures elsewhere. + +## 11. Phasing + +0. **P0 — `uffs-mft sysinfo`** (host probe): OS + `InstallationType` (client/server), + elevation, VSS availability, per-drive media type (HDD/SSD/NVMe) + geometry. + Emits `capture_host.txt` + the manifest `host` block; returns a capability set + the orchestrator uses to pick the best-effort path. Cross-platform-buildable + (facts are host-gathered; non-Windows returns a stub). **Ships first** — it + gates everything and is independently useful. +1. **P1 — frozen metafile readers + `save` targets** (`$Secure`, `$Bitmap`, + `$Boot`, `$AttrDef`, `$Volume`, `$MFTMirr`, `$UpCase`, `$BadClus`). +2. **P2 — `$UsnJrnl:$J` + `$LogFile` + `$Extend\*` dump.** +3. **P3 — `--volume-path` (shadow device) support in `VolumeHandle`.** +4. **P4 — `capture` orchestrator subcommand + `manifest.json`.** +5. **P5 — `scripts/capture.rs`: WMI VSS create/delete + `--zip`/split + hashes.** +6. **P6 — matching `load` + `verify_parity` wiring.** + +## 12. Open questions + +- `$UsnJrnl`: dump the **raw `$J` stream** (lossless, replayable) vs. parsed USN + records (smaller, queryable). Proposal: raw `$J` now, parser later. +- Cross-volume **atomic** shadow *set* (VSS COM) vs. per-volume shadows. + Proposal: per-volume (independent drives don't need cross-vol atomicity). +- `$Secure`: capture `$SDS` only vs. `$SDS`+`$SDH`+`$SII`. Proposal: `$SDS` only + (indexes rebuildable); revisit if offline rebuild proves costly. +- Subcommand naming: `uffs-mft sysinfo` (host+drive capture-time environment) is + distinct from the daemon's runtime `uffs_status`/`status` (health/uptime). Keep + the names separate to avoid confusion. Proposal: `sysinfo`. From 56b09cec026cebeabe905fd60404d6f285d4d03b Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Sun, 5 Jul 2026 10:59:04 -0700 Subject: [PATCH 02/39] feat(uffs-mft): sysinfo capture-host probe Add `uffs-mft sysinfo`: a cross-platform host + storage provenance probe that runs before a capture and records the machine it ran on. - Host: OS name/build/arch, client/server InstallationType (the VSS/shadow discriminator), elevation, VSS availability, CPU/RAM, timezone, tool version. - NTFS drives (Windows): media type, geometry, $MFT stats, and per-drive capture-capability flags (iocp_capturable, vss_eligible). - Volumes (all OS): every mounted filesystem with type / format / size / used%, collapsing shared-container volumes (e.g. synthetic APFS mounts). - Derives the best-effort capture mode (full-server / best-effort-client / live-only / unsupported); also useful as a benchmark provenance stamp. Text report + `--json` (Windows). Windows OS identity via a small read-only RegGetValueW reader (new Win32_System_Registry feature); other probes reuse existing safe wrappers. Split into capture_mode/report/windows/unix submodules. Verified: host + cargo-xwin Windows clippy clean, 8 unit tests, macOS run. Co-Authored-By: Claude Opus 4.8 --- Cargo.toml | 1 + crates/uffs-mft/src/cli.rs | 14 + crates/uffs-mft/src/commands/mod.rs | 3 + .../src/commands/sysinfo/capture_mode.rs | 173 ++++++++++ crates/uffs-mft/src/commands/sysinfo/mod.rs | 83 +++++ .../uffs-mft/src/commands/sysinfo/report.rs | 292 +++++++++++++++++ crates/uffs-mft/src/commands/sysinfo/unix.rs | 270 +++++++++++++++ .../uffs-mft/src/commands/sysinfo/windows.rs | 309 ++++++++++++++++++ 8 files changed, 1145 insertions(+) create mode 100644 crates/uffs-mft/src/commands/sysinfo/capture_mode.rs create mode 100644 crates/uffs-mft/src/commands/sysinfo/mod.rs create mode 100644 crates/uffs-mft/src/commands/sysinfo/report.rs create mode 100644 crates/uffs-mft/src/commands/sysinfo/unix.rs create mode 100644 crates/uffs-mft/src/commands/sysinfo/windows.rs diff --git a/Cargo.toml b/Cargo.toml index 6910de54e..120252e4c 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -202,6 +202,7 @@ windows = { version = "0.62.2", features = [ "Win32_System_Memory", "Win32_System_Pipes", "Win32_System_ProcessStatus", + "Win32_System_Registry", "Win32_System_Services", "Win32_System_SystemInformation", "Win32_System_Threading", diff --git a/crates/uffs-mft/src/cli.rs b/crates/uffs-mft/src/cli.rs index fc9248e61..83f24e99b 100644 --- a/crates/uffs-mft/src/cli.rs +++ b/crates/uffs-mft/src/cli.rs @@ -113,6 +113,20 @@ pub(crate) enum Commands { format: OutputFormat, }, + /// Probe the capture host: OS class (client/server), elevation, VSS + /// availability, and per-drive media/geometry. Runs before a capture and + /// records the machine the capture was taken on. + Sysinfo { + /// Write the report to this file (also printed to stdout). + #[arg(short, long)] + out: Option, + + /// Emit machine-readable JSON instead of the text report (Windows + /// only). + #[arg(long)] + json: bool, + }, + /// Benchmark MFT reading with detailed phase timing Bench { /// Drive letter (e.g., C, D, E) diff --git a/crates/uffs-mft/src/commands/mod.rs b/crates/uffs-mft/src/commands/mod.rs index 442c309f8..f16ac6886 100644 --- a/crates/uffs-mft/src/commands/mod.rs +++ b/crates/uffs-mft/src/commands/mod.rs @@ -8,6 +8,7 @@ use anyhow::Result; use crate::cli::Commands; mod load; +mod sysinfo; #[cfg(windows)] mod windows; @@ -35,6 +36,7 @@ pub(crate) async fn dispatch_command(command: Commands) -> Result<()> { format, } => windows::cmd_info(drive, deep, no_bitmap, unique, format).await, Commands::Drives { format } => windows::cmd_drives(format).await, + Commands::Sysinfo { out, json } => sysinfo::run(out.as_deref(), json), Commands::Bench { drive, json, @@ -189,6 +191,7 @@ pub(crate) async fn dispatch_command(command: Commands) -> Result<()> { drive, forensic, ), + Commands::Sysinfo { out, json } => sysinfo::run(out.as_deref(), json), Commands::Read { .. } | Commands::Info { .. } | Commands::Drives { .. } diff --git a/crates/uffs-mft/src/commands/sysinfo/capture_mode.rs b/crates/uffs-mft/src/commands/sysinfo/capture_mode.rs new file mode 100644 index 000000000..8d346335e --- /dev/null +++ b/crates/uffs-mft/src/commands/sysinfo/capture_mode.rs @@ -0,0 +1,173 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) 2025-2026 SKY, LLC. + +//! Capture-strategy classification: OS install class and the best-effort mode +//! derived from host capabilities. Pure logic — unit-tested here. + +/// OS installation class — the capture-strategy discriminator. +/// +/// Sourced from `HKLM\...\CurrentVersion\InstallationType` on Windows. The +/// `Client`/`Server` variants are constructed only by the Windows collector and +/// the tests, so non-Windows non-test builds legitimately never see them. +#[cfg_attr( + all(not(windows), not(test)), + expect( + dead_code, + reason = "Client/Server are constructed only by the Windows collector and the tests" + ) +)] +#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize)] +#[serde(rename_all = "lowercase")] +pub(super) enum InstallationType { + /// Windows client (10/11) — VSS shadow create needs WMI; C++-on-shadow is + /// awkward. + Client, + /// Windows Server — `diskshadow expose` enables skew-free C++-on-shadow. + Server, + /// Could not be determined (registry read failed or non-Windows host). + Unknown, +} + +impl InstallationType { + /// Short label for the human report. + pub(super) const fn label(self) -> &'static str { + match self { + Self::Client => "Client", + Self::Server => "Server", + Self::Unknown => "Unknown", + } + } +} + +/// Parse the registry `InstallationType` string into the typed enum. +/// +/// Compiled on Windows (used by the Windows collector) and under `test`. +#[cfg(any(windows, test))] +pub(super) fn parse_installation_type(raw: &str) -> InstallationType { + match raw.trim().to_ascii_lowercase().as_str() { + "client" => InstallationType::Client, + "server" | "server core" => InstallationType::Server, + _ => InstallationType::Unknown, + } +} + +/// Best-effort capture strategy chosen from host capabilities (§2.1). +#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize)] +#[serde(rename_all = "kebab-case")] +pub(super) enum CaptureMode { + /// Server + VSS: frozen shadow set + skew-free C++-on-shadow + live + /// `.iocp`. + FullServer, + /// Client + VSS: frozen shadow set (Rust) + C++ golden live + live `.iocp`. + BestEffortClient, + /// No VSS or not elevated: live `.iocp` + listings only; frozen set + /// skipped. + LiveOnly, + /// Non-Windows host: probe only, no capture possible. + Unsupported, +} + +impl CaptureMode { + /// One-line description for the human report. + pub(super) const fn describe(self) -> &'static str { + match self { + Self::FullServer => { + "full(server) = live .iocp + shadow(frozen) + C++ golden on shadow (skew-free)" + } + Self::BestEffortClient => { + "best-effort(client) = live .iocp + shadow(frozen, Rust) ; C++ golden live" + } + Self::LiveOnly => { + "live-only = .iocp + listings ; frozen set SKIPPED (no VSS/elevation)" + } + Self::Unsupported => "unsupported = non-Windows host, probe only", + } + } +} + +/// Choose the capture strategy from host capabilities. Pure — unit-tested. +pub(super) const fn decide_capture_mode( + os: InstallationType, + elevated: bool, + vss_available: bool, + is_windows: bool, +) -> CaptureMode { + if !is_windows { + return CaptureMode::Unsupported; + } + // Shadow create + raw MFT read both require elevation; without it the only + // thing achievable is a degraded live capture (or nothing). + if !elevated || !vss_available { + return CaptureMode::LiveOnly; + } + match os { + InstallationType::Server => CaptureMode::FullServer, + InstallationType::Client | InstallationType::Unknown => CaptureMode::BestEffortClient, + } +} + +#[cfg(test)] +mod tests { + use super::{CaptureMode, InstallationType, decide_capture_mode, parse_installation_type}; + + #[test] + fn server_with_vss_is_full() { + assert_eq!( + decide_capture_mode(InstallationType::Server, true, true, true), + CaptureMode::FullServer + ); + } + + #[test] + fn client_with_vss_is_best_effort() { + assert_eq!( + decide_capture_mode(InstallationType::Client, true, true, true), + CaptureMode::BestEffortClient + ); + } + + #[test] + fn unknown_os_defaults_to_client_effort() { + assert_eq!( + decide_capture_mode(InstallationType::Unknown, true, true, true), + CaptureMode::BestEffortClient + ); + } + + #[test] + fn no_vss_falls_back_to_live_only() { + assert_eq!( + decide_capture_mode(InstallationType::Server, true, false, true), + CaptureMode::LiveOnly + ); + } + + #[test] + fn not_elevated_falls_back_to_live_only() { + assert_eq!( + decide_capture_mode(InstallationType::Server, false, true, true), + CaptureMode::LiveOnly + ); + } + + #[test] + fn non_windows_is_unsupported() { + assert_eq!( + decide_capture_mode(InstallationType::Server, true, true, false), + CaptureMode::Unsupported + ); + } + + #[test] + fn parses_installation_type() { + assert_eq!(parse_installation_type("Client"), InstallationType::Client); + assert_eq!( + parse_installation_type(" server "), + InstallationType::Server + ); + assert_eq!( + parse_installation_type("nonsense"), + InstallationType::Unknown + ); + } +} diff --git a/crates/uffs-mft/src/commands/sysinfo/mod.rs b/crates/uffs-mft/src/commands/sysinfo/mod.rs new file mode 100644 index 000000000..fda0a288a --- /dev/null +++ b/crates/uffs-mft/src/commands/sysinfo/mod.rs @@ -0,0 +1,83 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) 2025-2026 SKY, LLC. + +//! `sysinfo` command — capture-host environment probe. +//! +//! Runs **before** a capture to record the machine the capture is taken on and +//! to pick the best-effort strategy (§2.1 of +//! `docs/architecture/mft-full-capture.md`). It reports the OS class +//! (`InstallationType` = client/server — the VSS/shadow discriminator), +//! elevation, VSS availability, host resources, NTFS capture targets, and every +//! mounted volume (type / format / size / used%), then emits a human-readable +//! `capture_host.txt` (and, on Windows, `--json`). +//! +//! This is deliberately distinct from the daemon's runtime +//! `status`/`uffs_status` (health/uptime). Layout: +//! - [`capture_mode`]: pure OS-class → best-effort strategy logic +//! (unit-tested). +//! - [`report`]: the report data model, rendering, and the shared host builder. +//! - `windows` / `unix`: the platform-specific collectors. +#![expect( + clippy::print_stdout, + reason = "intentional user-facing CLI capture-host report output" +)] + +mod capture_mode; +mod report; +#[cfg(not(windows))] +mod unix; +#[cfg(windows)] +mod windows; + +use std::path::Path; + +use anyhow::{Context as _, Result}; + +use self::report::Report; + +/// Run the `sysinfo` probe: print the report and optionally write it to `out`. +/// +/// # Errors +/// +/// Returns an error if the report cannot be serialised (`--json`) or the output +/// file cannot be written. +pub(crate) fn run(out: Option<&Path>, json: bool) -> Result<()> { + let report = collect(); + let rendered = render(&report, json)?; + if let Some(path) = out { + std::fs::write(path, rendered.as_bytes()) + .with_context(|| format!("writing capture-host report to {}", path.display()))?; + } + println!("{rendered}"); + Ok(()) +} + +/// Build the report using the platform-specific collector. +#[cfg(windows)] +fn collect() -> Report { + windows::collect() +} + +/// Build the report using the platform-specific collector. +#[cfg(not(windows))] +fn collect() -> Report { + unix::collect() +} + +/// Render the report as text, or JSON on Windows. +#[cfg(windows)] +fn render(report: &Report, json: bool) -> Result { + if json { + return serde_json::to_string_pretty(report).context("serialising sysinfo report to JSON"); + } + Ok(report.to_string()) +} + +/// Render the report as text (JSON is Windows-only in this crate). +#[cfg(not(windows))] +fn render(report: &Report, json: bool) -> Result { + if json { + anyhow::bail!("--json output is only available on the Windows build"); + } + Ok(report.to_string()) +} diff --git a/crates/uffs-mft/src/commands/sysinfo/report.rs b/crates/uffs-mft/src/commands/sysinfo/report.rs new file mode 100644 index 000000000..d25bbca59 --- /dev/null +++ b/crates/uffs-mft/src/commands/sysinfo/report.rs @@ -0,0 +1,292 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) 2025-2026 SKY, LLC. + +//! The capture-host report data model and its human-readable rendering, plus +//! the cross-platform host-facts builder shared by both platform collectors. +#![expect( + clippy::default_numeric_fallback, + reason = "unsuffixed byte/time unit divisors (1024, 60) in human-readable report formatting" +)] + +use core::fmt; + +use super::capture_mode::{CaptureMode, InstallationType, decide_capture_mode}; + +/// Host facts describing the machine a capture ran on. +#[derive(Debug, Clone, serde::Serialize)] +pub(super) struct HostInfo { + /// Computer name. + machine: String, + /// OS product name + display version (e.g. `Windows 11 Pro 24H2`). + os_name: String, + /// OS build number (e.g. `26100`). + os_build: String, + /// Target architecture (e.g. `x86_64`). + arch: String, + /// Client/Server discriminator. + installation_type: InstallationType, + /// Edition id (e.g. `Professional`, `ServerStandard`). + edition: String, + /// Whether the process is elevated (admin/root). + elevated: bool, + /// Whether the Volume Shadow Copy service is installed and not disabled. + vss_available: bool, + /// Logical CPU count. + cpu_cores: usize, + /// Total physical RAM in bytes. + ram_bytes: u64, + /// Local UTC offset in minutes (relevant to timestamp collation). + utc_offset_minutes: i32, + /// Capture timestamp (RFC 3339, UTC). + captured_at: String, + /// `uffs-mft` version string. + tool_version: String, + /// Selected best-effort capture strategy. + capture_mode: CaptureMode, +} + +/// Per-drive media, geometry, and capture capability. +#[expect( + clippy::struct_excessive_bools, + reason = "four independent per-drive capability flags; a bitfield would obscure the JSON schema and report" +)] +#[derive(Debug, Clone, serde::Serialize)] +pub(super) struct DriveCapture { + /// Drive letter (e.g. `C`). + pub(super) drive: String, + /// `true` when this drive hosts the running OS. + pub(super) boot: bool, + /// Storage kind: `NVMe`, `SSD`, `HDD`, `Removable`, `Virtual`, `Unknown`. + pub(super) media_type: String, + /// Volume serial number, hex. + pub(super) volume_serial: String, + /// Total capacity in bytes (0 when the volume handle could not be opened). + pub(super) total_bytes: u64, + /// Free capacity in bytes. + pub(super) free_bytes: u64, + /// Cluster size in bytes. + pub(super) bytes_per_cluster: u32, + /// `$MFT` size in bytes. + pub(super) mft_size_bytes: u64, + /// Allocated MFT record count. + pub(super) mft_records: u64, + /// NTFS version (e.g. `3.1`). + pub(super) ntfs_version: String, + /// Whether the volume is mounted read-only. + pub(super) read_only: bool, + /// Whether a live `.iocp` capture is possible (writable volume). + pub(super) iocp_capturable: bool, + /// Whether the drive is VSS-eligible (fixed, non-removable). + pub(super) vss_eligible: bool, +} + +/// A mounted volume on the host (any filesystem) — provenance, not a capture +/// target. +#[derive(Debug, Clone, serde::Serialize)] +pub(super) struct Volume { + /// Mount point (Unix) or drive root (Windows, e.g. `C:\`). + pub(super) mount: String, + /// Filesystem type (e.g. `apfs`, `ext4`, `NTFS`, `exFAT`). + pub(super) filesystem: String, + /// Storage media kind where detectable, else `n/a`. + pub(super) media_type: String, + /// Total capacity in bytes. + pub(super) total_bytes: u64, + /// Free capacity in bytes. + pub(super) free_bytes: u64, +} + +/// Full capture-host report: host facts + per-drive capabilities. +#[derive(Debug, Clone, serde::Serialize)] +pub(super) struct Report { + /// Host-level facts. + pub(super) host: HostInfo, + /// One entry per detected NTFS drive (Windows capture targets). + pub(super) drives: Vec, + /// All mounted volumes (any filesystem) — host provenance. + pub(super) volumes: Vec, +} + +impl fmt::Display for Report { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let host = &self.host; + writeln!(f, "UFFS Capture Host Report")?; + writeln!( + f, + " Captured: {} (UTC{:+03}:{:02})", + host.captured_at, + host.utc_offset_minutes / 60, + host.utc_offset_minutes.rem_euclid(60), + )?; + writeln!(f, " Tool: uffs-mft {}", host.tool_version)?; + writeln!(f, " Host: {}", host.machine)?; + writeln!( + f, + " OS: {} build {} {}", + host.os_name, host.os_build, host.arch + )?; + writeln!( + f, + " InstallType: {:<8} <-- gates VSS / C++-on-shadow strategy", + host.installation_type.label() + )?; + writeln!(f, " Edition: {}", host.edition)?; + writeln!(f, " Elevated: {}", yes_no(host.elevated))?; + writeln!(f, " VSS service: {}", available(host.vss_available))?; + writeln!( + f, + " CPU / RAM: {} cores / {} GB", + host.cpu_cores, + host.ram_bytes / (1024 * 1024 * 1024) + )?; + writeln!(f, " Capture mode: {}", host.capture_mode.describe())?; + writeln!(f)?; + writeln!(f, " Drives (NTFS):")?; + if self.drives.is_empty() { + writeln!(f, " (none detected / not a Windows host)")?; + } + for drive in &self.drives { + let used = drive.total_bytes.saturating_sub(drive.free_bytes); + let used_pct = (used * 100).checked_div(drive.total_bytes).unwrap_or(0); + writeln!( + f, + " {}: {:<9} {} GB ({}% used, serial {}) MFT {} MB / {} recs", + drive.drive, + drive.media_type, + drive.total_bytes / (1024 * 1024 * 1024), + used_pct, + drive.volume_serial, + drive.mft_size_bytes / (1024 * 1024), + drive.mft_records, + )?; + writeln!( + f, + " ntfs:{} read_only:{} iocp:{} vss_eligible:{}", + drive.ntfs_version, + yes_no(drive.read_only), + yes_no(drive.iocp_capturable), + yes_no(drive.vss_eligible), + )?; + } + writeln!(f)?; + writeln!(f, " Volumes (all filesystems):")?; + if self.volumes.is_empty() { + writeln!(f, " (none detected)")?; + } + for vol in &self.volumes { + let used = vol.total_bytes.saturating_sub(vol.free_bytes); + let used_pct = (used * 100).checked_div(vol.total_bytes).unwrap_or(0); + writeln!( + f, + " {:<26} {:<8} {:<9} {} GB / {} GB used ({}%)", + vol.mount, + vol.filesystem, + vol.media_type, + vol.total_bytes / (1024 * 1024 * 1024), + used / (1024 * 1024 * 1024), + used_pct, + )?; + } + Ok(()) + } +} + +/// Render a bool as `yes`/`no`. +const fn yes_no(value: bool) -> &'static str { + if value { "yes" } else { "no" } +} + +/// Render a service-availability bool for the report. +const fn available(value: bool) -> &'static str { + if value { + "installed (shadow create: probe on capture)" + } else { + "not available" + } +} + +/// Cross-platform host facts shared by both collectors. +/// +/// `installation_type`, `os_name`, `os_build`, `edition`, and `vss_available` +/// are filled by the platform-specific caller; everything else is portable. +pub(super) fn base_host( + os_name: String, + os_build: String, + edition: String, + installation_type: InstallationType, + vss_available: bool, + is_windows: bool, +) -> HostInfo { + let now_local = chrono::Local::now(); + let utc_offset_minutes = { + use chrono::Offset as _; + now_local.offset().fix().local_minus_utc() / 60 + }; + let elevated = uffs_mft::is_elevated(); + let cpu_cores = std::thread::available_parallelism().map_or(0, core::num::NonZeroUsize::get); + HostInfo { + machine: machine_name(), + os_name, + os_build, + arch: std::env::consts::ARCH.to_owned(), + installation_type, + edition, + elevated, + vss_available, + cpu_cores, + ram_bytes: uffs_mft::query_system_memory().total_bytes, + utc_offset_minutes, + captured_at: chrono::Utc::now().to_rfc3339_opts(chrono::SecondsFormat::Secs, true), + tool_version: env!("CARGO_PKG_VERSION").to_owned(), + capture_mode: decide_capture_mode(installation_type, elevated, vss_available, is_windows), + } +} + +/// Real computer name via the OS (falls back to `unknown`). +fn machine_name() -> String { + hostname::get() + .ok() + .and_then(|name| name.into_string().ok()) + .filter(|name| !name.is_empty()) + .unwrap_or_else(|| "unknown".to_owned()) +} + +#[cfg(test)] +mod tests { + use super::super::capture_mode::{CaptureMode, InstallationType}; + use super::{HostInfo, Report}; + + /// Build a deterministic host record for rendering tests. + fn sample_host() -> HostInfo { + HostInfo { + machine: "TEST-PC".to_owned(), + os_name: "Windows 11 Pro 24H2".to_owned(), + os_build: "26100".to_owned(), + arch: "x86_64".to_owned(), + installation_type: InstallationType::Client, + edition: "Professional".to_owned(), + elevated: true, + vss_available: true, + cpu_cores: 24, + ram_bytes: 64 * 1024 * 1024 * 1024, + utc_offset_minutes: -420, + captured_at: "2026-07-05T18:22:04Z".to_owned(), + tool_version: "0.0.0".to_owned(), + capture_mode: CaptureMode::BestEffortClient, + } + } + + #[test] + fn report_renders_key_fields() { + let report = Report { + host: sample_host(), + drives: Vec::new(), + volumes: Vec::new(), + }; + let text = report.to_string(); + assert!(text.contains("UFFS Capture Host Report")); + assert!(text.contains("InstallType: Client")); + assert!(text.contains("Capture mode:")); + assert!(text.contains("(none detected")); + } +} diff --git a/crates/uffs-mft/src/commands/sysinfo/unix.rs b/crates/uffs-mft/src/commands/sysinfo/unix.rs new file mode 100644 index 000000000..d9959dd22 --- /dev/null +++ b/crates/uffs-mft/src/commands/sysinfo/unix.rs @@ -0,0 +1,270 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) 2025-2026 SKY, LLC. + +//! Non-Windows capture-host collectors: real OS identity + all-volume +//! enumeration with per-device filesystem and media-type detection. This whole +//! module is compiled only on non-Windows hosts (see the `mod unix;` gate in +//! `mod.rs`); MFT capture itself is Windows-only, so `drives` is always empty. + +use super::capture_mode::InstallationType; +use super::report::{Report, Volume, base_host}; + +/// Collect the capture-host report on non-Windows hosts. +/// +/// Host identity (OS, CPU, RAM, hostname) is real — useful as a benchmark +/// provenance stamp — but the capture mode resolves to `Unsupported`. +pub(super) fn collect() -> Report { + let (os_name, os_build) = os_release(); + let host = base_host( + os_name, + os_build, + "n/a".to_owned(), + InstallationType::Unknown, + false, + false, + ); + Report { + host, + drives: Vec::new(), + volumes: collect_volumes(), + } +} + +/// Friendly OS name + build for macOS via `sw_vers`. +#[cfg(target_os = "macos")] +fn os_release() -> (String, String) { + let sw_vers = |arg: &str| -> Option { + let output = std::process::Command::new("sw_vers") + .arg(arg) + .output() + .ok()?; + if !output.status.success() { + return None; + } + // AUDIT-OK(bytes): decodes `sw_vers` stdout into a display string; the + // emptiness filter below fails closed on any non-UTF-8 garbage. + let text = String::from_utf8_lossy(&output.stdout).trim().to_owned(); + (!text.is_empty()).then_some(text) + }; + let name = sw_vers("-productName").unwrap_or_else(|| "macOS".to_owned()); + let version = sw_vers("-productVersion").unwrap_or_default(); + let build = sw_vers("-buildVersion").unwrap_or_else(|| "?".to_owned()); + let os_name = if version.is_empty() { + name + } else { + format!("{name} {version}") + }; + (os_name, build) +} + +/// Friendly OS name + kernel for Linux via `/etc/os-release` + `uname -r`. +#[cfg(target_os = "linux")] +fn os_release() -> (String, String) { + let pretty = std::fs::read_to_string("/etc/os-release") + .ok() + .and_then(|content| { + content.lines().find_map(|line| { + line.strip_prefix("PRETTY_NAME=") + .map(|value| value.trim_matches('"').to_owned()) + }) + }) + .unwrap_or_else(|| "Linux".to_owned()); + let kernel = std::process::Command::new("uname") + .arg("-r") + .output() + .ok() + .filter(|output| output.status.success()) + // AUDIT-OK(bytes): decodes `uname -r` stdout into a display string. + .map(|output| String::from_utf8_lossy(&output.stdout).trim().to_owned()) + .filter(|kernel| !kernel.is_empty()) + .unwrap_or_else(|| "?".to_owned()); + (pretty, kernel) +} + +/// Fallback OS identity for other Unix targets. +#[cfg(not(any(target_os = "macos", target_os = "linux")))] +fn os_release() -> (String, String) { + (std::env::consts::OS.to_owned(), "?".to_owned()) +} + +/// List mounted volumes (any filesystem) via `df` + a mount→fstype map. +/// +/// Only real block devices (`/dev/...`) are listed; synthetic mounts are +/// skipped, and volumes sharing one physical container collapse to a single +/// row (e.g. the many synthetic APFS `/System/Volumes/*` mounts). +fn collect_volumes() -> Vec { + let output = match std::process::Command::new("df").args(["-k", "-P"]).output() { + Ok(out) if out.status.success() => out, + _ => return Vec::new(), + }; + // AUDIT-OK(bytes): decodes `df` stdout for a display; rows that fail to + // parse are skipped (fail-closed). + let text = String::from_utf8_lossy(&output.stdout); + let fstypes = fstype_map(); + let mut seen = std::collections::HashSet::new(); + text.lines() + .skip(1) + .filter_map(|line| { + // POSIX `df -P`: Filesystem 1024-blocks Used Available Capacity Mounted-on + let cols: Vec<&str> = line.split_whitespace().collect(); + let device = *cols.first()?; + if !device.starts_with("/dev/") { + return None; + } + let total_kib: u64 = cols.get(1)?.parse().ok()?; + let avail_kib: u64 = cols.get(3)?.parse().ok()?; + if !seen.insert((base_device(device), total_kib, avail_kib)) { + return None; + } + let mount = cols.get(5..)?.join(" "); + let filesystem = fstypes + .get(&mount) + .cloned() + .unwrap_or_else(|| device.to_owned()); + Some(Volume { + mount, + filesystem, + media_type: disk_media_type(device), + total_bytes: total_kib * 1024, + free_bytes: avail_kib * 1024, + }) + }) + .collect() +} + +/// Map of mount point → filesystem type on Linux (`/proc/mounts`). +#[cfg(target_os = "linux")] +fn fstype_map() -> std::collections::HashMap { + let mut map = std::collections::HashMap::new(); + if let Ok(content) = std::fs::read_to_string("/proc/mounts") { + for line in content.lines() { + let cols: Vec<&str> = line.split_whitespace().collect(); + if let (Some(mount), Some(fstype)) = (cols.get(1), cols.get(2)) { + map.insert((*mount).to_owned(), (*fstype).to_owned()); + } + } + } + map +} + +/// Map of mount point → filesystem type on macOS (`mount` output). +#[cfg(target_os = "macos")] +fn fstype_map() -> std::collections::HashMap { + let mut map = std::collections::HashMap::new(); + let Ok(output) = std::process::Command::new("mount").output() else { + return map; + }; + if !output.status.success() { + return map; + } + // AUDIT-OK(bytes): decodes `mount` stdout into a display map. + let text = String::from_utf8_lossy(&output.stdout); + for line in text.lines() { + // "/dev/disk3s1s1 on / (apfs, sealed, local, ...)" + if let Some((_, rest)) = line.split_once(" on ") + && let Some((mount, paren)) = rest.split_once(" (") + { + let fstype = paren.split([',', ')']).next().unwrap_or("").trim(); + if !fstype.is_empty() { + map.insert(mount.to_owned(), fstype.to_owned()); + } + } + } + map +} + +/// Empty mount→fstype map on other Unix targets. +#[cfg(not(any(target_os = "linux", target_os = "macos")))] +fn fstype_map() -> std::collections::HashMap { + std::collections::HashMap::new() +} + +/// Detect SSD/HDD/NVMe/Removable for a device via `diskutil info` on macOS. +#[cfg(target_os = "macos")] +fn disk_media_type(device: &str) -> String { + let Ok(output) = std::process::Command::new("diskutil") + .args(["info", device]) + .output() + else { + return "Unknown".to_owned(); + }; + if !output.status.success() { + return "Unknown".to_owned(); + } + // AUDIT-OK(bytes): decodes `diskutil info` stdout for a display string. + let text = String::from_utf8_lossy(&output.stdout); + let field = |name: &str| -> String { + text.lines() + .find_map(|line| { + let (key, value) = line.split_once(':')?; + (key.trim() == name).then(|| value.trim().to_owned()) + }) + .unwrap_or_default() + }; + let solid_state = field("Solid State"); + let protocol = field("Protocol"); + if protocol.eq_ignore_ascii_case("USB") || protocol.contains("External") { + "Removable".to_owned() + } else if solid_state.eq_ignore_ascii_case("Yes") { + if protocol.contains("PCI") { + "NVMe".to_owned() + } else { + "SSD".to_owned() + } + } else if solid_state.eq_ignore_ascii_case("No") { + "HDD".to_owned() + } else { + "Unknown".to_owned() + } +} + +/// Detect SSD/HDD/NVMe for a device via `/sys/block//queue/rotational`. +#[cfg(target_os = "linux")] +fn disk_media_type(device: &str) -> String { + let base = base_device(device); + let rotational = std::fs::read_to_string(format!("/sys/block/{base}/queue/rotational")) + .ok() + .map(|text| text.trim().to_owned()); + match rotational.as_deref() { + Some("0") if base.starts_with("nvme") => "NVMe".to_owned(), + Some("0") => "SSD".to_owned(), + Some("1") => "HDD".to_owned(), + _ => "Unknown".to_owned(), + } +} + +/// Media detection is unavailable on other Unix targets. +#[cfg(not(any(target_os = "linux", target_os = "macos")))] +fn disk_media_type(_device: &str) -> String { + "Unknown".to_owned() +} + +/// Reduce a device node to its parent physical device, so volumes sharing one +/// container collapse to a single row (`/dev/disk3s1s1` → `disk3`, +/// `/dev/sda1` → `sda`, `/dev/nvme0n1p2` → `nvme0n1`, `/dev/mmcblk0p1` → +/// `mmcblk0`). +fn base_device(device: &str) -> String { + let name = device.strip_prefix("/dev/").unwrap_or(device); + // macOS: `diskNsM...` → `diskN`. + if let Some(rest) = name.strip_prefix("disk") { + let digits: String = rest.chars().take_while(char::is_ascii_digit).collect(); + if !digits.is_empty() { + return format!("disk{digits}"); + } + } + // Linux `nvme`/`mmcblk`: strip the `pN` partition suffix. + if name.starts_with("nvme") || name.starts_with("mmcblk") { + if let Some(idx) = name.rfind('p') { + let is_partition = name.get(idx + 1..).is_some_and(|rest| { + !rest.is_empty() && rest.bytes().all(|byte| byte.is_ascii_digit()) + }); + if is_partition { + return name.get(..idx).unwrap_or(name).to_owned(); + } + } + return name.to_owned(); + } + // Linux `sdXN`/`vdXN`: strip trailing partition digits. + name.trim_end_matches(|character: char| character.is_ascii_digit()) + .to_owned() +} diff --git a/crates/uffs-mft/src/commands/sysinfo/windows.rs b/crates/uffs-mft/src/commands/sysinfo/windows.rs new file mode 100644 index 000000000..4d82d9661 --- /dev/null +++ b/crates/uffs-mft/src/commands/sysinfo/windows.rs @@ -0,0 +1,309 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) 2025-2026 SKY, LLC. + +//! Windows capture-host collectors: OS identity (registry), VSS availability, +//! per-drive NTFS geometry, and all-volume enumeration. This whole module is +//! compiled only on Windows (see the `mod windows;` gate in `mod.rs`). + +use super::capture_mode::{InstallationType, parse_installation_type}; +use super::report::{DriveCapture, Report, Volume, base_host}; + +/// Collect the capture-host report on Windows. +pub(super) fn collect() -> Report { + let (os_name, os_build, edition, installation_type) = os_identity(); + let host = base_host( + os_name, + os_build, + edition, + installation_type, + vss_available(), + true, + ); + Report { + host, + drives: collect_drives(), + volumes: collect_volumes(), + } +} + +/// Human label for a [`uffs_mft::DriveType`]. +const fn media_label(kind: uffs_mft::DriveType) -> &'static str { + use uffs_mft::DriveType; + match kind { + DriveType::Nvme => "NVMe", + DriveType::Ssd => "SSD", + DriveType::Hdd => "HDD", + DriveType::Removable => "Removable", + DriveType::Virtual => "Virtual", + DriveType::Unknown => "Unknown", + } +} + +/// Read the OS identity strings from the registry. +fn os_identity() -> (String, String, String, InstallationType) { + const CV: &str = r"SOFTWARE\Microsoft\Windows NT\CurrentVersion"; + let product = reg_string(CV, "ProductName").unwrap_or_else(|| "Windows".to_owned()); + let display = reg_string(CV, "DisplayVersion").or_else(|| reg_string(CV, "ReleaseId")); + let os_build = reg_string(CV, "CurrentBuildNumber").unwrap_or_else(|| "?".to_owned()); + let edition = reg_string(CV, "EditionID").unwrap_or_else(|| "?".to_owned()); + let installation_type = reg_string(CV, "InstallationType") + .map_or(InstallationType::Unknown, |raw| { + parse_installation_type(&raw) + }); + let os_name = display.map_or_else(|| product.clone(), |ver| format!("{product} {ver}")); + (os_name, os_build, edition, installation_type) +} + +/// Whether the Volume Shadow Copy service is installed and not disabled. +fn vss_available() -> bool { + // `Start == 4` means "Disabled"; anything else (or presence) means usable. + reg_u32(r"SYSTEM\CurrentControlSet\Services\VSS", "Start").is_some_and(|start| start != 4) +} + +/// Volume-derived geometry, or `unavailable` placeholders when the handle +/// cannot be opened (e.g. not elevated). +struct VolFields { + /// Total capacity in bytes. + total_bytes: u64, + /// Free capacity in bytes. + free_bytes: u64, + /// Cluster size in bytes. + bytes_per_cluster: u32, + /// `$MFT` size in bytes. + mft_size_bytes: u64, + /// Allocated MFT record count. + mft_records: u64, + /// Volume serial (hex), or a placeholder when unavailable. + volume_serial: String, + /// NTFS version, or `?` when unavailable. + ntfs_version: String, +} + +impl VolFields { + /// Placeholders for a volume whose handle could not be opened. + fn unavailable() -> Self { + Self { + total_bytes: 0, + free_bytes: 0, + bytes_per_cluster: 0, + mft_size_bytes: 0, + mft_records: 0, + volume_serial: String::from("(needs elevation)"), + ntfs_version: String::from("?"), + } + } + + /// Extract geometry from an open volume handle. + fn from_handle(handle: &uffs_mft::platform::VolumeHandle) -> Self { + let vol = handle.volume_data(); + let bytes_per_cluster = vol.bytes_per_cluster; + let mft_size_bytes = vol.mft_valid_data_length; + let mft_records = if vol.bytes_per_file_record_segment > 0 { + mft_size_bytes / u64::from(vol.bytes_per_file_record_segment) + } else { + 0 + }; + Self { + total_bytes: vol.total_clusters * u64::from(bytes_per_cluster), + free_bytes: vol.free_clusters * u64::from(bytes_per_cluster), + bytes_per_cluster, + mft_size_bytes, + mft_records, + volume_serial: format!("0x{:016X}", vol.volume_serial_number), + ntfs_version: format!("{}.{}", vol.ntfs_major_version, vol.ntfs_minor_version), + } + } +} + +/// Enumerate NTFS drives with media type, geometry, and capture capability. +fn collect_drives() -> Vec { + use uffs_mft::DriveType; + use uffs_mft::platform::{ + VolumeHandle, detect_drive_type, detect_ntfs_drives, is_boot_drive, is_volume_read_only, + }; + + detect_ntfs_drives() + .into_iter() + .map(|drive| { + let media = detect_drive_type(drive); + let read_only = is_volume_read_only(drive); + let fields = VolumeHandle::open(drive) + .ok() + .map_or_else(VolFields::unavailable, |handle| { + VolFields::from_handle(&handle) + }); + DriveCapture { + drive: drive.to_string(), + boot: is_boot_drive(drive), + media_type: media_label(media).to_owned(), + volume_serial: fields.volume_serial, + total_bytes: fields.total_bytes, + free_bytes: fields.free_bytes, + bytes_per_cluster: fields.bytes_per_cluster, + mft_size_bytes: fields.mft_size_bytes, + mft_records: fields.mft_records, + ntfs_version: fields.ntfs_version, + read_only, + iocp_capturable: !read_only, + vss_eligible: !matches!(media, DriveType::Removable), + } + }) + .collect() +} + +/// List all mounted logical volumes (any filesystem) for host provenance. +#[expect( + unsafe_code, + reason = "FFI: GetLogicalDrives / GetVolumeInformationW / GetDiskFreeSpaceExW to enumerate all mounted volumes" +)] +fn collect_volumes() -> Vec { + use uffs_mft::platform::{DriveLetter, detect_drive_type}; + use windows::Win32::Storage::FileSystem::{ + GetDiskFreeSpaceExW, GetLogicalDrives, GetVolumeInformationW, + }; + use windows::core::PCWSTR; + + // SAFETY: `GetLogicalDrives` takes no pointers and returns a bitmask by value. + let mask = unsafe { GetLogicalDrives() }; + let mut volumes = Vec::new(); + for index in 0_u8..26 { + if (mask & (1_u32 << index)) == 0 { + continue; + } + let letter = char::from(b'A' + index); + let root: Vec = format!("{letter}:\\") + .encode_utf16() + .chain(core::iter::once(0)) + .collect(); + + let mut fs_buf = [0_u16; 32]; + // SAFETY: `root` is NUL-terminated UTF-16 valid for the call; `fs_buf` is + // a writable 32-element buffer for the filesystem name; the other + // out-parameters are documented as accepting `None`. + let info = unsafe { + GetVolumeInformationW( + PCWSTR(root.as_ptr()), + None, + None, + None, + None, + Some(&mut fs_buf), + ) + }; + if info.is_err() { + // Empty removable slot / disconnected network drive — skip. + continue; + } + + let mut free_to_caller = 0_u64; + let mut total = 0_u64; + let mut total_free = 0_u64; + // SAFETY: `root` is NUL-terminated UTF-16 valid for the call; the three + // out-parameters point to writable `u64` locals. Sizes are best-effort; + // on failure the locals stay zero. + let _sizes_ok = unsafe { + GetDiskFreeSpaceExW( + PCWSTR(root.as_ptr()), + Some(&raw mut free_to_caller), + Some(&raw mut total), + Some(&raw mut total_free), + ) + } + .is_ok(); + + let media_type = DriveLetter::parse(letter) + .ok() + .map(detect_drive_type) + .map_or("Unknown", media_label); + + volumes.push(Volume { + mount: format!("{letter}:\\"), + filesystem: u16_to_string(&fs_buf), + media_type: media_type.to_owned(), + total_bytes: total, + free_bytes: free_to_caller, + }); + } + volumes +} + +/// Decode a NUL-terminated UTF-16 buffer up to its first NUL. +fn u16_to_string(buf: &[u16]) -> String { + let len = buf.iter().position(|&code| code == 0).unwrap_or(buf.len()); + String::from_utf16_lossy(buf.get(..len).unwrap_or(&[])) +} + +/// Read a read-only `HKLM` string value via `RegGetValueW`. +/// +/// Returns `None` when the key/value is absent or not a string. +#[expect( + unsafe_code, + reason = "FFI: Win32 RegGetValueW to read read-only HKLM OS-identity strings" +)] +fn reg_string(subkey: &str, value: &str) -> Option { + use windows::Win32::Foundation::ERROR_SUCCESS; + use windows::Win32::System::Registry::{HKEY_LOCAL_MACHINE, RRF_RT_REG_SZ, RegGetValueW}; + use windows::core::PCWSTR; + + let subkey_w: Vec = subkey.encode_utf16().chain(core::iter::once(0)).collect(); + let value_w: Vec = value.encode_utf16().chain(core::iter::once(0)).collect(); + let mut buf = [0_u16; 512]; + // Bytes available in `buf` (512 * size_of::()); updated in place. + let mut cb: u32 = 1024; + + // SAFETY: `subkey_w`/`value_w` are NUL-terminated UTF-16 buffers valid for + // the call; `buf` is writable for `cb` bytes; `RRF_RT_REG_SZ` restricts the + // result to a string value; `cb` receives the byte length written. + let status = unsafe { + RegGetValueW( + HKEY_LOCAL_MACHINE, + PCWSTR(subkey_w.as_ptr()), + PCWSTR(value_w.as_ptr()), + RRF_RT_REG_SZ, + None, + Some(buf.as_mut_ptr().cast::()), + Some(&raw mut cb), + ) + }; + if status != ERROR_SUCCESS { + return None; + } + // `cb` counts bytes including the trailing NUL; convert to a u16 length. + let units = usize::try_from(cb).unwrap_or(0) / size_of::(); + let len = units.saturating_sub(1).min(buf.len()); + let text = String::from_utf16_lossy(buf.get(..len)?); + let trimmed = text.trim().to_owned(); + (!trimmed.is_empty()).then_some(trimmed) +} + +/// Read a read-only `HKLM` `DWORD` value via `RegGetValueW`. +#[expect( + unsafe_code, + reason = "FFI: Win32 RegGetValueW to read a read-only service-config DWORD" +)] +fn reg_u32(subkey: &str, value: &str) -> Option { + use windows::Win32::Foundation::ERROR_SUCCESS; + use windows::Win32::System::Registry::{HKEY_LOCAL_MACHINE, RRF_RT_REG_DWORD, RegGetValueW}; + use windows::core::PCWSTR; + + let subkey_w: Vec = subkey.encode_utf16().chain(core::iter::once(0)).collect(); + let value_w: Vec = value.encode_utf16().chain(core::iter::once(0)).collect(); + let mut data: u32 = 0; + let mut cb: u32 = 4; + + // SAFETY: NUL-terminated UTF-16 key/value valid for the call; `data` is a + // writable `u32` of `cb` bytes; `RRF_RT_REG_DWORD` restricts the result to a + // DWORD. + let status = unsafe { + RegGetValueW( + HKEY_LOCAL_MACHINE, + PCWSTR(subkey_w.as_ptr()), + PCWSTR(value_w.as_ptr()), + RRF_RT_REG_DWORD, + None, + Some((&raw mut data).cast::()), + Some(&raw mut cb), + ) + }; + (status == ERROR_SUCCESS).then_some(data) +} From 5789997eaa5eec5fdad668e1ed9cd6e444847581 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Sun, 5 Jul 2026 11:41:45 -0700 Subject: [PATCH 03/39] =?UTF-8?q?feat(uffs-mft):=20NTFS=20metafile=20captu?= =?UTF-8?q?re=20=E2=80=94=20$Boot=20(P1)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First frozen-metafile capture target. Adds a self-describing capture format (`MetafileHeader`: magic/version/kind/drive/serial/timestamp/size) and a `uffs-mft metafile --drive C --kind boot -o C_boot.bin` command that reads $Boot (the 8 KiB volume boot region at offset 0) via the broker-safe read_handle_at primitive and writes header + payload. - New `platform::metafile` module (mirrors `platform::upcase`): MetafileKind, MetafileHeader (to_bytes/from_bytes), read_metafile (Windows) + non-Windows stub, save/load_metafile_to/from_file. - `metafile` subcommand + MetafileKind CLI value-enum; dispatch on Windows, "requires Windows" elsewhere. - 6 cross-platform header round-trip/validation tests. Foundation for the remaining targets ($Bitmap, $Secure:$SDS, $AttrDef, ...), which reuse the same header/format via data-run reads. Verified: host + cargo-xwin Windows clippy clean, tests green. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/cli.rs | 22 ++ crates/uffs-mft/src/commands/mod.rs | 8 +- crates/uffs-mft/src/commands/windows/mod.rs | 2 +- crates/uffs-mft/src/commands/windows/save.rs | 50 +++ crates/uffs-mft/src/platform.rs | 2 + crates/uffs-mft/src/platform/metafile.rs | 303 +++++++++++++++++++ 6 files changed, 385 insertions(+), 2 deletions(-) create mode 100644 crates/uffs-mft/src/platform/metafile.rs diff --git a/crates/uffs-mft/src/cli.rs b/crates/uffs-mft/src/cli.rs index 83f24e99b..269134b5e 100644 --- a/crates/uffs-mft/src/cli.rs +++ b/crates/uffs-mft/src/cli.rs @@ -23,6 +23,13 @@ pub(crate) enum OutputFormat { Json, } +/// NTFS metafile selectable via `uffs-mft metafile --kind`. +#[derive(Copy, Clone, Debug, PartialEq, Eq, ValueEnum)] +pub(crate) enum MetafileKind { + /// `$Boot` — volume boot record + BPB (geometry, volume serial). + Boot, +} + /// `uffs-mft`: Low-level NTFS MFT reading tool. #[derive(Parser)] #[command(name = "uffs-mft")] @@ -127,6 +134,21 @@ pub(crate) enum Commands { json: bool, }, + /// Save an NTFS metafile ($Boot, ...) for offline analysis + Metafile { + /// Drive letter (e.g., C, D, E) + #[arg(short, long)] + drive: uffs_mft::platform::DriveLetter, + + /// Which metafile to capture + #[arg(short, long, value_enum)] + kind: MetafileKind, + + /// Output file path + #[arg(short, long)] + output: PathBuf, + }, + /// Benchmark MFT reading with detailed phase timing Bench { /// Drive letter (e.g., C, D, E) diff --git a/crates/uffs-mft/src/commands/mod.rs b/crates/uffs-mft/src/commands/mod.rs index f16ac6886..1f7f2a4dc 100644 --- a/crates/uffs-mft/src/commands/mod.rs +++ b/crates/uffs-mft/src/commands/mod.rs @@ -37,6 +37,11 @@ pub(crate) async fn dispatch_command(command: Commands) -> Result<()> { } => windows::cmd_info(drive, deep, no_bitmap, unique, format).await, Commands::Drives { format } => windows::cmd_drives(format).await, Commands::Sysinfo { out, json } => sysinfo::run(out.as_deref(), json), + Commands::Metafile { + drive, + kind, + output, + } => windows::cmd_metafile(drive, kind, &output).await, Commands::Bench { drive, json, @@ -192,7 +197,8 @@ pub(crate) async fn dispatch_command(command: Commands) -> Result<()> { forensic, ), Commands::Sysinfo { out, json } => sysinfo::run(out.as_deref(), json), - Commands::Read { .. } + Commands::Metafile { .. } + | Commands::Read { .. } | Commands::Info { .. } | Commands::Drives { .. } | Commands::Bench { .. } diff --git a/crates/uffs-mft/src/commands/windows/mod.rs b/crates/uffs-mft/src/commands/windows/mod.rs index ad1ec9c0d..421214a2c 100644 --- a/crates/uffs-mft/src/commands/windows/mod.rs +++ b/crates/uffs-mft/src/commands/windows/mod.rs @@ -29,5 +29,5 @@ pub(crate) use self::incremental::{ }; pub(crate) use self::info::cmd_info; pub(crate) use self::read::cmd_read; -pub(crate) use self::save::cmd_save; +pub(crate) use self::save::{cmd_metafile, cmd_save}; pub(crate) use self::usn::{cmd_usn_info, cmd_usn_read}; diff --git a/crates/uffs-mft/src/commands/windows/save.rs b/crates/uffs-mft/src/commands/windows/save.rs index 988c587f9..ff2d0ce62 100644 --- a/crates/uffs-mft/src/commands/windows/save.rs +++ b/crates/uffs-mft/src/commands/windows/save.rs @@ -383,3 +383,53 @@ async fn cmd_save_upcase(drive: uffs_mft::platform::DriveLetter, output: &Path) Ok(()) } + +// ============================================================================ +// NTFS Metafile Capture ($Boot, ...) +// ============================================================================ + +/// Save an NTFS metafile ($Boot, ...) to a file for offline analysis. +#[cfg(windows)] +pub(crate) async fn cmd_metafile( + drive: uffs_mft::platform::DriveLetter, + kind: crate::cli::MetafileKind, + output: &Path, +) -> Result<()> { + use uffs_mft::platform::VolumeHandle; + use uffs_mft::platform::metafile::{self, MetafileHeader, MetafileKind}; + + let lib_kind = match kind { + crate::cli::MetafileKind::Boot => MetafileKind::Boot, + }; + + // Volume serial for the header. + let handle = VolumeHandle::open(drive).with_context(|| format!("Failed to open {drive}:"))?; + let vol = handle.volume_data(); + + let data = metafile::read_metafile(drive, lib_kind) + .with_context(|| format!("Failed to read {} from {drive}:", lib_kind.name()))?; + + let timestamp = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |elapsed| elapsed.as_secs()); + let header = MetafileHeader { + kind: lib_kind, + drive, + volume_serial: vol.volume_serial_number, + timestamp, + data_size: usize_to_u64(data.len()), + }; + metafile::save_metafile_to_file(output, &header, &data) + .with_context(|| format!("Failed to save {} to {}", lib_kind.name(), output.display()))?; + + let abs_path = std::fs::canonicalize(output).unwrap_or_else(|_| output.to_path_buf()); + let abs_path = clean_path_for_display(&abs_path); + + println!("💾 {} saved", lib_kind.name()); + println!(" Drive: {drive}:"); + println!(" Size: {}", format_bytes(usize_to_u64(data.len()))); + println!(" Serial: 0x{:016X}", vol.volume_serial_number); + println!(" Path: {}", abs_path.display()); + + Ok(()) +} diff --git a/crates/uffs-mft/src/platform.rs b/crates/uffs-mft/src/platform.rs index 9a3f897d1..8ca1fe69b 100644 --- a/crates/uffs-mft/src/platform.rs +++ b/crates/uffs-mft/src/platform.rs @@ -37,6 +37,8 @@ mod extents; /// detection through [`Lcn::is_hole`] / [`Lcn::is_zero`] instead of /// open-coded `< 0` / `== 0` checks at every call site. pub mod lcn; +/// NTFS metafile capture ($Boot, ...) from a live volume. +pub mod metafile; /// Native Windows process introspection for the self-update detector. /// /// Image path + pid enumeration — keeps the `unsafe` FFI out of `uffs-cli`. diff --git a/crates/uffs-mft/src/platform/metafile.rs b/crates/uffs-mft/src/platform/metafile.rs new file mode 100644 index 000000000..3081a95d7 --- /dev/null +++ b/crates/uffs-mft/src/platform/metafile.rs @@ -0,0 +1,303 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) 2025-2026 SKY, LLC. + +//! Capture NTFS metafiles (the reserved `$`-files) from a live Windows volume +//! and persist them with a small self-describing header. +//! +//! These artifacts extend a capture beyond the `$MFT` namespace toward a +//! complete offline representation of the volume (see +//! `docs/architecture/mft-full-capture.md`). Each is read straight off the +//! volume via the same broker-safe primitive as `$UpCase` +//! ([`super::volume::read_handle_at`]). +//! +//! # File format +//! +//! 1. [`MetafileHeader`] (64 bytes) — magic, version, kind, drive, serial, +//! timestamp, payload size. +//! 2. Raw metafile payload. +//! +//! # Usage +//! +//! ```text +//! uffs-mft metafile --drive C --kind boot --output C_boot.bin +//! ``` + +use std::path::Path; + +use crate::error::{MftError, Result}; +use crate::platform::DriveLetter; + +/// Magic bytes identifying a UFFS metafile capture. +const METAFILE_MAGIC: &[u8; 8] = b"UFFSMETA"; + +/// Current metafile capture format version. +const METAFILE_VERSION: u32 = 1; + +/// Fixed header size in bytes (payload starts at this offset). +const HEADER_SIZE: usize = 64; + +/// `$Boot` payload size: the boot region is 16 sectors × 512 bytes. +#[cfg(windows)] +const BOOT_BYTES: usize = 8192; + +/// An NTFS metafile that can be captured. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum MetafileKind { + /// `$Boot` (FRS 7) — the volume boot record + BPB (geometry, serial). + Boot, +} + +impl MetafileKind { + /// The NTFS metafile name (e.g. `$Boot`). + #[must_use] + pub const fn name(self) -> &'static str { + match self { + Self::Boot => "$Boot", + } + } + + /// The MFT file-record-segment (FRS) number of this metafile. Stored in the + /// header as a stable, self-documenting kind code. + #[must_use] + pub const fn frs(self) -> u8 { + match self { + Self::Boot => 7, + } + } + + /// Reconstruct a kind from its FRS code (header round-trip). + const fn from_frs(frs: u8) -> Option { + match frs { + 7 => Some(Self::Boot), + _ => None, + } + } +} + +/// Self-describing header prefixed to a captured NTFS metafile. +/// +/// ```text +/// Offset Size Field +/// 0 8 Magic b"UFFSMETA" +/// 8 4 Format version (u32 LE) +/// 12 1 Metafile FRS code (u8) +/// 13 1 Drive letter (ASCII uppercase) +/// 14 2 Reserved +/// 16 8 Volume serial number (u64 LE) +/// 24 8 Timestamp — Unix epoch seconds (u64 LE) +/// 32 8 Payload size in bytes (u64 LE) +/// 40 24 Reserved +/// ─────────── +/// 64 Raw metafile payload +/// ``` +#[derive(Debug, Clone)] +pub struct MetafileHeader { + /// Which metafile this capture holds. + pub kind: MetafileKind, + /// Source drive letter. + pub drive: DriveLetter, + /// Source volume serial number. + pub volume_serial: u64, + /// Capture timestamp (Unix epoch seconds). + pub timestamp: u64, + /// Payload size in bytes. + pub data_size: u64, +} + +impl MetafileHeader { + /// Serialize the header to its fixed 64-byte on-disk form. + #[must_use] + pub fn to_bytes(&self) -> [u8; HEADER_SIZE] { + let mut buf = [0_u8; HEADER_SIZE]; + buf[0..8].copy_from_slice(METAFILE_MAGIC); + buf[8..12].copy_from_slice(&METAFILE_VERSION.to_le_bytes()); + buf[12] = self.kind.frs(); + buf[13] = self.drive.as_byte(); + // 14..16 reserved (already zeroed) + buf[16..24].copy_from_slice(&self.volume_serial.to_le_bytes()); + buf[24..32].copy_from_slice(&self.timestamp.to_le_bytes()); + buf[32..40].copy_from_slice(&self.data_size.to_le_bytes()); + // 40..64 reserved (already zeroed) + buf + } + + /// Parse a header from the first 64 bytes of a captured file. + /// + /// # Errors + /// + /// Returns [`MftError::InvalidData`] if the buffer is too short, has the + /// wrong magic, an unsupported version, an unknown kind, or a bad drive + /// letter. + #[expect( + clippy::indexing_slicing, + clippy::missing_asserts_for_indexing, + reason = "length validated at the top; every index below is < HEADER_SIZE (64)" + )] + pub fn from_bytes(data: &[u8]) -> Result { + if data.len() < HEADER_SIZE { + return Err(MftError::InvalidData(format!( + "metafile header too short: {} < {HEADER_SIZE}", + data.len() + ))); + } + if &data[0..8] != METAFILE_MAGIC { + return Err(MftError::InvalidData("invalid metafile magic".to_owned())); + } + let version = u32::from_le_bytes([data[8], data[9], data[10], data[11]]); + if version > METAFILE_VERSION { + return Err(MftError::InvalidData(format!( + "unsupported metafile version: {version}" + ))); + } + let kind = MetafileKind::from_frs(data[12]).ok_or_else(|| { + MftError::InvalidData(format!("unknown metafile FRS code: {}", data[12])) + })?; + let drive = DriveLetter::parse(char::from(data[13])).map_err(|err| { + MftError::InvalidData(format!("invalid drive letter in metafile header: {err}")) + })?; + let volume_serial = u64::from_le_bytes([ + data[16], data[17], data[18], data[19], data[20], data[21], data[22], data[23], + ]); + let timestamp = u64::from_le_bytes([ + data[24], data[25], data[26], data[27], data[28], data[29], data[30], data[31], + ]); + let data_size = u64::from_le_bytes([ + data[32], data[33], data[34], data[35], data[36], data[37], data[38], data[39], + ]); + Ok(Self { + kind, + drive, + volume_serial, + timestamp, + data_size, + }) + } +} + +/// Read a metafile's raw bytes from a live NTFS volume. +/// +/// # Errors +/// +/// Returns [`MftError::Io`] / [`MftError::Windows`] if opening the volume or +/// reading fails. +#[cfg(windows)] +pub fn read_metafile(drive: DriveLetter, kind: MetafileKind) -> Result> { + let handle = crate::platform::VolumeHandle::open(drive)?; + match kind { + MetafileKind::Boot => read_boot(&handle), + } +} + +/// Read a metafile's raw bytes (non-Windows stub). +/// +/// # Errors +/// +/// Always returns [`MftError::PlatformNotSupported`]. +#[cfg(not(windows))] +pub const fn read_metafile(_drive: DriveLetter, _kind: MetafileKind) -> Result> { + Err(MftError::PlatformNotSupported) +} + +/// `$Boot` is the volume boot region: 8 KiB starting at LCN 0 (byte offset 0). +#[cfg(windows)] +fn read_boot(handle: &crate::platform::VolumeHandle) -> Result> { + let mut buf = vec![0_u8; BOOT_BYTES]; + super::volume::read_handle_at(handle.raw_handle(), 0, &mut buf)?; + Ok(buf) +} + +/// Write a captured metafile (header + payload) to `path` atomically. +/// +/// # Errors +/// +/// Returns [`MftError::InvalidData`] if the write fails. +pub fn save_metafile_to_file(path: &Path, header: &MetafileHeader, data: &[u8]) -> Result<()> { + let mut out = Vec::with_capacity(HEADER_SIZE + data.len()); + out.extend_from_slice(&header.to_bytes()); + out.extend_from_slice(data); + crate::cache::atomic_write(path, &out) + .map_err(|err| MftError::InvalidData(format!("Failed to write metafile: {err}")))?; + tracing::info!( + path = %path.display(), + kind = header.kind.name(), + bytes = out.len(), + "Saved NTFS metafile" + ); + Ok(()) +} + +/// Load a captured metafile, returning its header and raw payload. +/// +/// # Errors +/// +/// Returns [`MftError::InvalidData`] if the file is unreadable, has a bad +/// header, or is truncated. +pub fn load_metafile_from_file(path: &Path) -> Result<(MetafileHeader, Vec)> { + let data = std::fs::read(path).map_err(|err| { + MftError::InvalidData(format!("Failed to read {}: {err}", path.display())) + })?; + let header = MetafileHeader::from_bytes(&data)?; + let payload = data + .get(HEADER_SIZE..) + .ok_or_else(|| MftError::InvalidData("metafile payload missing".to_owned()))? + .to_vec(); + Ok((header, payload)) +} + +#[cfg(test)] +mod tests { + use super::{HEADER_SIZE, MetafileHeader, MetafileKind}; + use crate::platform::DriveLetter; + + fn sample() -> MetafileHeader { + MetafileHeader { + kind: MetafileKind::Boot, + drive: DriveLetter::parse('C').expect("valid drive letter"), + volume_serial: 0xDEAD_BEEF_1234_5678, + timestamp: 1_700_000_000, + data_size: 8192, + } + } + + #[test] + fn header_round_trips() { + let header = sample(); + let bytes = header.to_bytes(); + let back = MetafileHeader::from_bytes(&bytes).expect("round-trip"); + assert_eq!(back.kind, MetafileKind::Boot); + assert_eq!(back.drive, header.drive); + assert_eq!(back.volume_serial, header.volume_serial); + assert_eq!(back.timestamp, header.timestamp); + assert_eq!(back.data_size, header.data_size); + } + + #[test] + fn kind_frs_is_stable() { + assert_eq!(MetafileKind::Boot.frs(), 7); + assert_eq!(MetafileKind::Boot.name(), "$Boot"); + } + + #[test] + fn from_bytes_rejects_bad_magic() { + let mut bytes = sample().to_bytes(); + bytes[0] = b'X'; + MetafileHeader::from_bytes(&bytes).unwrap_err(); + } + + #[test] + fn from_bytes_rejects_short_buffer() { + MetafileHeader::from_bytes(&[0_u8; 10]).unwrap_err(); + } + + #[test] + fn from_bytes_rejects_unknown_kind() { + let mut bytes = sample().to_bytes(); + bytes[12] = 200; // no metafile has FRS 200 + MetafileHeader::from_bytes(&bytes).unwrap_err(); + } + + #[test] + fn payload_offset_is_header_size() { + assert_eq!(HEADER_SIZE, 64); + } +} From cc1328c2c10577089e045c718146812a295e4b0c Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Sun, 5 Jul 2026 11:52:06 -0700 Subject: [PATCH 04/39] =?UTF-8?q?feat(uffs-mft):=20metafile=20capture=20?= =?UTF-8?q?=E2=80=94=20$Bitmap=20+=20generic=20stream=20reader=20(P1.2)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the keystone data-run stream primitive and the first target that uses it. - read_frs_record: read a raw MFT record (any FRS) with USA fixup. - read_data_stream: resolve a metafile's non-resident $DATA attribute (unnamed, or a named stream like $SDS) and assemble its data runs. - read_runs: read data runs into a data_size-truncated buffer; sparse runs stay zeroed. Reuses the broker-safe read_handle_at. - $Bitmap (FRS 6): the volume cluster-allocation bitmap, captured via the unnamed non-resident $DATA stream. `metafile --kind bitmap`. The named-stream support lands $Secure:$SDS next with no new I/O code. Verified: host + cargo-xwin Windows clippy (pedantic/nursery) clean, tests green. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/cli.rs | 2 + crates/uffs-mft/src/commands/windows/save.rs | 1 + crates/uffs-mft/src/platform/metafile.rs | 114 +++++++++++++++++++ 3 files changed, 117 insertions(+) diff --git a/crates/uffs-mft/src/cli.rs b/crates/uffs-mft/src/cli.rs index 269134b5e..6f21e4a55 100644 --- a/crates/uffs-mft/src/cli.rs +++ b/crates/uffs-mft/src/cli.rs @@ -28,6 +28,8 @@ pub(crate) enum OutputFormat { pub(crate) enum MetafileKind { /// `$Boot` — volume boot record + BPB (geometry, volume serial). Boot, + /// `$Bitmap` — volume cluster-allocation bitmap (free space). + Bitmap, } /// `uffs-mft`: Low-level NTFS MFT reading tool. diff --git a/crates/uffs-mft/src/commands/windows/save.rs b/crates/uffs-mft/src/commands/windows/save.rs index ff2d0ce62..867e2b800 100644 --- a/crates/uffs-mft/src/commands/windows/save.rs +++ b/crates/uffs-mft/src/commands/windows/save.rs @@ -400,6 +400,7 @@ pub(crate) async fn cmd_metafile( let lib_kind = match kind { crate::cli::MetafileKind::Boot => MetafileKind::Boot, + crate::cli::MetafileKind::Bitmap => MetafileKind::Bitmap, }; // Volume serial for the header. diff --git a/crates/uffs-mft/src/platform/metafile.rs b/crates/uffs-mft/src/platform/metafile.rs index 3081a95d7..82292b3c5 100644 --- a/crates/uffs-mft/src/platform/metafile.rs +++ b/crates/uffs-mft/src/platform/metafile.rs @@ -45,6 +45,8 @@ const BOOT_BYTES: usize = 8192; pub enum MetafileKind { /// `$Boot` (FRS 7) — the volume boot record + BPB (geometry, serial). Boot, + /// `$Bitmap` (FRS 6) — the volume cluster-allocation bitmap (free space). + Bitmap, } impl MetafileKind { @@ -53,6 +55,7 @@ impl MetafileKind { pub const fn name(self) -> &'static str { match self { Self::Boot => "$Boot", + Self::Bitmap => "$Bitmap", } } @@ -62,6 +65,7 @@ impl MetafileKind { pub const fn frs(self) -> u8 { match self { Self::Boot => 7, + Self::Bitmap => 6, } } @@ -69,6 +73,7 @@ impl MetafileKind { const fn from_frs(frs: u8) -> Option { match frs { 7 => Some(Self::Boot), + 6 => Some(Self::Bitmap), _ => None, } } @@ -183,8 +188,12 @@ impl MetafileHeader { #[cfg(windows)] pub fn read_metafile(drive: DriveLetter, kind: MetafileKind) -> Result> { let handle = crate::platform::VolumeHandle::open(drive)?; + let vol = handle.volume_data(); match kind { + // `$Boot` is fixed at LCN 0; read it directly (no data-run parse). MetafileKind::Boot => read_boot(&handle), + // `$Bitmap` is a non-resident unnamed `$DATA` stream. + MetafileKind::Bitmap => read_data_stream(&handle, vol, 6, None), } } @@ -206,6 +215,105 @@ fn read_boot(handle: &crate::platform::VolumeHandle) -> Result> { Ok(buf) } +/// Read a raw MFT file-record segment (FRS) with USA fixup applied. +#[cfg(windows)] +fn read_frs_record( + handle: &crate::platform::VolumeHandle, + vol: &crate::platform::NtfsVolumeData, + frs: u64, +) -> Result> { + let record_size = vol.bytes_per_file_record_segment as usize; + let offset = handle.mft_byte_offset() + frs * u64::from(vol.bytes_per_file_record_segment); + let mut record = vec![0_u8; record_size]; + super::volume::read_handle_at(handle.raw_handle(), offset, &mut record)?; + crate::parse::apply_fixup(&mut record); + Ok(record) +} + +/// Read a metafile's non-resident `$DATA` stream — the unnamed stream when +/// `stream_name` is `None`, or the named stream (e.g. `$SDS`) otherwise — by +/// resolving its data runs and reading the referenced clusters. +#[cfg(windows)] +fn read_data_stream( + handle: &crate::platform::VolumeHandle, + vol: &crate::platform::NtfsVolumeData, + frs: u64, + stream_name: Option<&str>, +) -> Result> { + use crate::ntfs::{AttributeIterator, AttributeType}; + + let record = read_frs_record(handle, vol, frs)?; + let want_name: Option> = stream_name.map(|name| name.encode_utf16().collect()); + + let mut attrs = AttributeIterator::new(&record) + .ok_or_else(|| MftError::InvalidData(format!("FRS {frs}: invalid record header")))?; + let data_attr = attrs + .find(|attr| { + attr.attribute_type() == Some(AttributeType::Data) + && attr.is_non_resident() + && want_name + .as_deref() + .map_or(attr.header.name_length == 0, |want| { + attr.name() == Some(want) + }) + }) + .ok_or_else(|| { + MftError::InvalidData(format!( + "FRS {frs}: no matching non-resident DATA stream (name={stream_name:?})" + )) + })?; + + let non_resident = data_attr.non_resident_data().ok_or_else(|| { + MftError::InvalidData(format!("FRS {frs}: cannot decode non-resident DATA header")) + })?; + let data_size = non_resident.data_size.cast_unsigned(); + let runs = data_attr.data_runs(); + read_runs(handle.raw_handle(), &runs, vol.bytes_per_cluster, data_size) +} + +/// Assemble a stream's data runs into a `data_size`-byte buffer. +/// +/// Sparse runs leave their window zeroed; the buffer is truncated to the +/// attribute's real `data_size` (runs are cluster-rounded). +#[cfg(windows)] +fn read_runs( + handle: windows::Win32::Foundation::HANDLE, + runs: &[crate::ntfs::DataRun], + bytes_per_cluster: u32, + data_size: u64, +) -> Result> { + let bpc = u64::from(bytes_per_cluster); + let total = usize::try_from(data_size).map_err(|_err| { + MftError::InvalidData("metafile data_size exceeds usize::MAX".to_owned()) + })?; + let mut buf = vec![0_u8; total]; + let mut offset: usize = 0; + + for run in runs { + if offset >= total { + break; + } + let run_bytes = usize::try_from(run.cluster_count * bpc).map_err(|_err| { + MftError::InvalidData("metafile run byte count exceeds usize::MAX".to_owned()) + })?; + let read_len = run_bytes.min(total - offset); + if run.is_sparse() { + // Sparse run — leave the buffer window zeroed. + offset += read_len; + continue; + } + let disk_offset = crate::index::nonneg_to_u64(run.lcn.raw() * bpc.cast_signed()); + let Some(window) = buf.get_mut(offset..offset + read_len) else { + return Err(MftError::InvalidData(format!( + "metafile run at offset {offset} len {read_len} exceeds buffer {total}" + ))); + }; + super::volume::read_handle_at(handle, disk_offset, window)?; + offset += read_len; + } + Ok(buf) +} + /// Write a captured metafile (header + payload) to `path` atomically. /// /// # Errors @@ -275,6 +383,12 @@ mod tests { fn kind_frs_is_stable() { assert_eq!(MetafileKind::Boot.frs(), 7); assert_eq!(MetafileKind::Boot.name(), "$Boot"); + assert_eq!(MetafileKind::Bitmap.frs(), 6); + assert_eq!(MetafileKind::Bitmap.name(), "$Bitmap"); + // FRS code round-trips through the header field. + assert_eq!(MetafileKind::from_frs(6), Some(MetafileKind::Bitmap)); + assert_eq!(MetafileKind::from_frs(7), Some(MetafileKind::Boot)); + assert_eq!(MetafileKind::from_frs(200), None); } #[test] From 193807d19f497432e28947049734c6284bdae557 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Sun, 5 Jul 2026 11:54:03 -0700 Subject: [PATCH 05/39] =?UTF-8?q?feat(uffs-mft):=20metafile=20capture=20?= =?UTF-8?q?=E2=80=94=20$Secure:$SDS=20ACLs=20(P1.3)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Capture the NTFS security-descriptor store via the named-stream path added in P1.2 (no new I/O code): $Secure (FRS 9), named $DATA stream $SDS. This is the ACL / owner / DACL-SACL store that the $MFT namespace only references by SecurityId. `metafile --kind secure`. Verified: host + cargo-xwin Windows clippy (pedantic/nursery) clean, tests green. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/cli.rs | 2 ++ crates/uffs-mft/src/commands/windows/save.rs | 1 + crates/uffs-mft/src/platform/metafile.rs | 10 ++++++++++ 3 files changed, 13 insertions(+) diff --git a/crates/uffs-mft/src/cli.rs b/crates/uffs-mft/src/cli.rs index 6f21e4a55..c98df5dde 100644 --- a/crates/uffs-mft/src/cli.rs +++ b/crates/uffs-mft/src/cli.rs @@ -30,6 +30,8 @@ pub(crate) enum MetafileKind { Boot, /// `$Bitmap` — volume cluster-allocation bitmap (free space). Bitmap, + /// `$Secure:$SDS` — security-descriptor store (ACLs / owner). + Secure, } /// `uffs-mft`: Low-level NTFS MFT reading tool. diff --git a/crates/uffs-mft/src/commands/windows/save.rs b/crates/uffs-mft/src/commands/windows/save.rs index 867e2b800..48a50ce38 100644 --- a/crates/uffs-mft/src/commands/windows/save.rs +++ b/crates/uffs-mft/src/commands/windows/save.rs @@ -401,6 +401,7 @@ pub(crate) async fn cmd_metafile( let lib_kind = match kind { crate::cli::MetafileKind::Boot => MetafileKind::Boot, crate::cli::MetafileKind::Bitmap => MetafileKind::Bitmap, + crate::cli::MetafileKind::Secure => MetafileKind::Secure, }; // Volume serial for the header. diff --git a/crates/uffs-mft/src/platform/metafile.rs b/crates/uffs-mft/src/platform/metafile.rs index 82292b3c5..159fbe654 100644 --- a/crates/uffs-mft/src/platform/metafile.rs +++ b/crates/uffs-mft/src/platform/metafile.rs @@ -47,6 +47,8 @@ pub enum MetafileKind { Boot, /// `$Bitmap` (FRS 6) — the volume cluster-allocation bitmap (free space). Bitmap, + /// `$Secure:$SDS` (FRS 9) — the security-descriptor store (ACLs / owner). + Secure, } impl MetafileKind { @@ -56,6 +58,7 @@ impl MetafileKind { match self { Self::Boot => "$Boot", Self::Bitmap => "$Bitmap", + Self::Secure => "$Secure", } } @@ -66,6 +69,7 @@ impl MetafileKind { match self { Self::Boot => 7, Self::Bitmap => 6, + Self::Secure => 9, } } @@ -74,6 +78,7 @@ impl MetafileKind { match frs { 7 => Some(Self::Boot), 6 => Some(Self::Bitmap), + 9 => Some(Self::Secure), _ => None, } } @@ -194,6 +199,8 @@ pub fn read_metafile(drive: DriveLetter, kind: MetafileKind) -> Result> MetafileKind::Boot => read_boot(&handle), // `$Bitmap` is a non-resident unnamed `$DATA` stream. MetafileKind::Bitmap => read_data_stream(&handle, vol, 6, None), + // `$Secure:$SDS` holds deduplicated security descriptors (ACLs). + MetafileKind::Secure => read_data_stream(&handle, vol, 9, Some("$SDS")), } } @@ -385,9 +392,12 @@ mod tests { assert_eq!(MetafileKind::Boot.name(), "$Boot"); assert_eq!(MetafileKind::Bitmap.frs(), 6); assert_eq!(MetafileKind::Bitmap.name(), "$Bitmap"); + assert_eq!(MetafileKind::Secure.frs(), 9); + assert_eq!(MetafileKind::Secure.name(), "$Secure"); // FRS code round-trips through the header field. assert_eq!(MetafileKind::from_frs(6), Some(MetafileKind::Bitmap)); assert_eq!(MetafileKind::from_frs(7), Some(MetafileKind::Boot)); + assert_eq!(MetafileKind::from_frs(9), Some(MetafileKind::Secure)); assert_eq!(MetafileKind::from_frs(200), None); } From 2cad9376329f2672da6ccd180611e4b16bf8449f Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Sun, 5 Jul 2026 11:57:25 -0700 Subject: [PATCH 06/39] =?UTF-8?q?feat(uffs-mft):=20metafile=20capture=20?= =?UTF-8?q?=E2=80=94=20$AttrDef/$MFTMirr/$Volume/$BadClus=20(P1.4)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Complete the frozen-metafile target set: - $AttrDef (FRS 4) and $MFTMirr (FRS 1): unnamed non-resident $DATA streams, via the P1.2 stream reader. - $Volume (FRS 3) and $BadClus (FRS 8): their useful data lives in the MFT record itself ($VOLUME_NAME/$VOLUME_INFORMATION; the $Bad run list), so the fixed-up record is captured via read_frs_record. `metafile --kind attr-def|mft-mirr|volume|bad-clus`. With $Boot/$Bitmap/$Secure and the existing `save --upcase`, the complete metafile set is now capturable. Verified: host + cargo-xwin Windows clippy (pedantic/nursery) clean, tests green. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/cli.rs | 8 +++++ crates/uffs-mft/src/commands/windows/save.rs | 4 +++ crates/uffs-mft/src/platform/metafile.rs | 37 ++++++++++++++++++++ 3 files changed, 49 insertions(+) diff --git a/crates/uffs-mft/src/cli.rs b/crates/uffs-mft/src/cli.rs index c98df5dde..9c703eeee 100644 --- a/crates/uffs-mft/src/cli.rs +++ b/crates/uffs-mft/src/cli.rs @@ -32,6 +32,14 @@ pub(crate) enum MetafileKind { Bitmap, /// `$Secure:$SDS` — security-descriptor store (ACLs / owner). Secure, + /// `$AttrDef` — NTFS attribute-type definitions. + AttrDef, + /// `$MFTMirr` — backup of the first four `$MFT` records. + MftMirr, + /// `$Volume` — the MFT record (volume name / version / flags). + Volume, + /// `$BadClus` — the MFT record (bad-cluster run list). + BadClus, } /// `uffs-mft`: Low-level NTFS MFT reading tool. diff --git a/crates/uffs-mft/src/commands/windows/save.rs b/crates/uffs-mft/src/commands/windows/save.rs index 48a50ce38..987fd294e 100644 --- a/crates/uffs-mft/src/commands/windows/save.rs +++ b/crates/uffs-mft/src/commands/windows/save.rs @@ -402,6 +402,10 @@ pub(crate) async fn cmd_metafile( crate::cli::MetafileKind::Boot => MetafileKind::Boot, crate::cli::MetafileKind::Bitmap => MetafileKind::Bitmap, crate::cli::MetafileKind::Secure => MetafileKind::Secure, + crate::cli::MetafileKind::AttrDef => MetafileKind::AttrDef, + crate::cli::MetafileKind::MftMirr => MetafileKind::MftMirr, + crate::cli::MetafileKind::Volume => MetafileKind::Volume, + crate::cli::MetafileKind::BadClus => MetafileKind::BadClus, }; // Volume serial for the header. diff --git a/crates/uffs-mft/src/platform/metafile.rs b/crates/uffs-mft/src/platform/metafile.rs index 159fbe654..28c549ec1 100644 --- a/crates/uffs-mft/src/platform/metafile.rs +++ b/crates/uffs-mft/src/platform/metafile.rs @@ -49,6 +49,15 @@ pub enum MetafileKind { Bitmap, /// `$Secure:$SDS` (FRS 9) — the security-descriptor store (ACLs / owner). Secure, + /// `$AttrDef` (FRS 4) — NTFS attribute-type definitions. + AttrDef, + /// `$MFTMirr` (FRS 1) — backup of the first four `$MFT` records. + MftMirr, + /// `$Volume` (FRS 3) — the MFT record (`$VOLUME_NAME` / + /// `$VOLUME_INFORMATION`). + Volume, + /// `$BadClus` (FRS 8) — the MFT record (the `$Bad` bad-cluster run list). + BadClus, } impl MetafileKind { @@ -59,6 +68,10 @@ impl MetafileKind { Self::Boot => "$Boot", Self::Bitmap => "$Bitmap", Self::Secure => "$Secure", + Self::AttrDef => "$AttrDef", + Self::MftMirr => "$MFTMirr", + Self::Volume => "$Volume", + Self::BadClus => "$BadClus", } } @@ -70,6 +83,10 @@ impl MetafileKind { Self::Boot => 7, Self::Bitmap => 6, Self::Secure => 9, + Self::AttrDef => 4, + Self::MftMirr => 1, + Self::Volume => 3, + Self::BadClus => 8, } } @@ -79,6 +96,10 @@ impl MetafileKind { 7 => Some(Self::Boot), 6 => Some(Self::Bitmap), 9 => Some(Self::Secure), + 4 => Some(Self::AttrDef), + 1 => Some(Self::MftMirr), + 3 => Some(Self::Volume), + 8 => Some(Self::BadClus), _ => None, } } @@ -201,6 +222,14 @@ pub fn read_metafile(drive: DriveLetter, kind: MetafileKind) -> Result> MetafileKind::Bitmap => read_data_stream(&handle, vol, 6, None), // `$Secure:$SDS` holds deduplicated security descriptors (ACLs). MetafileKind::Secure => read_data_stream(&handle, vol, 9, Some("$SDS")), + // `$AttrDef` / `$MFTMirr` are unnamed non-resident `$DATA` streams. + MetafileKind::AttrDef => read_data_stream(&handle, vol, 4, None), + MetafileKind::MftMirr => read_data_stream(&handle, vol, 1, None), + // `$Volume` / `$BadClus` keep their useful data in the MFT record itself + // (the $VOLUME_* attributes; the $Bad run list), so capture the fixed-up + // record. + MetafileKind::Volume => read_frs_record(&handle, vol, 3), + MetafileKind::BadClus => read_frs_record(&handle, vol, 8), } } @@ -394,10 +423,18 @@ mod tests { assert_eq!(MetafileKind::Bitmap.name(), "$Bitmap"); assert_eq!(MetafileKind::Secure.frs(), 9); assert_eq!(MetafileKind::Secure.name(), "$Secure"); + assert_eq!(MetafileKind::AttrDef.frs(), 4); + assert_eq!(MetafileKind::MftMirr.frs(), 1); + assert_eq!(MetafileKind::Volume.frs(), 3); + assert_eq!(MetafileKind::BadClus.frs(), 8); // FRS code round-trips through the header field. assert_eq!(MetafileKind::from_frs(6), Some(MetafileKind::Bitmap)); assert_eq!(MetafileKind::from_frs(7), Some(MetafileKind::Boot)); assert_eq!(MetafileKind::from_frs(9), Some(MetafileKind::Secure)); + assert_eq!(MetafileKind::from_frs(4), Some(MetafileKind::AttrDef)); + assert_eq!(MetafileKind::from_frs(1), Some(MetafileKind::MftMirr)); + assert_eq!(MetafileKind::from_frs(3), Some(MetafileKind::Volume)); + assert_eq!(MetafileKind::from_frs(8), Some(MetafileKind::BadClus)); assert_eq!(MetafileKind::from_frs(200), None); } From 40c38c706c18d717e9163e15978b04f107fbf8d9 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Sun, 5 Jul 2026 12:25:50 -0700 Subject: [PATCH 07/39] feat(uffs-mft): capture orchestrator + manifest bundle (P1.5) Add `uffs-mft capture --drive C --out `: capture every NTFS metafile for a drive into one hashed, manifested bundle under /drive_/. - Writes each metafile ($Boot/$Bitmap/$Secure/$AttrDef/$MFTMirr/$Volume/ $BadClus) via the P1.1-P1.4 metafile module, best-effort (a metafile that can't be read is skipped and noted, not fatal). - manifest.json: schema, drive, captured_at, tool version, volume facts (serial/ntfs-version/cluster+record size), and per-artifact {file, kind, frs, bytes, sha256}. - SHA256SUMS for transfer verification. sha2 + hex added to the Windows dep set for the manifest hashing. Verified: cargo-xwin Windows clippy + host pedantic clean, tests green. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/Cargo.toml | 5 + crates/uffs-mft/src/cli.rs | 11 + crates/uffs-mft/src/commands/mod.rs | 3 + .../uffs-mft/src/commands/windows/capture.rs | 204 ++++++++++++++++++ crates/uffs-mft/src/commands/windows/mod.rs | 2 + crates/uffs-mft/src/lib.rs | 6 +- 6 files changed, 230 insertions(+), 1 deletion(-) create mode 100644 crates/uffs-mft/src/commands/windows/capture.rs diff --git a/crates/uffs-mft/Cargo.toml b/crates/uffs-mft/Cargo.toml index b74684ccb..c48c808ac 100644 --- a/crates/uffs-mft/Cargo.toml +++ b/crates/uffs-mft/Cargo.toml @@ -115,6 +115,11 @@ windows.workspace = true # benchmark report) — only the Windows-gated commands use it. serde_json.workspace = true +# SHA-256 + hex for the `capture` bundle manifest / SHA256SUMS integrity file +# (Windows-only capture path). +sha2.workspace = true +hex.workspace = true + # Unix privilege check (`is_elevated` → geteuid() == 0) [target.'cfg(unix)'.dependencies] libc.workspace = true diff --git a/crates/uffs-mft/src/cli.rs b/crates/uffs-mft/src/cli.rs index 9c703eeee..436b0af4c 100644 --- a/crates/uffs-mft/src/cli.rs +++ b/crates/uffs-mft/src/cli.rs @@ -161,6 +161,17 @@ pub(crate) enum Commands { output: PathBuf, }, + /// Capture all NTFS metafiles for a drive into a hashed, manifested bundle + Capture { + /// Drive letter (e.g., C, D, E) + #[arg(short, long)] + drive: uffs_mft::platform::DriveLetter, + + /// Output directory (bundle written to `/drive_/`) + #[arg(short, long)] + out: PathBuf, + }, + /// Benchmark MFT reading with detailed phase timing Bench { /// Drive letter (e.g., C, D, E) diff --git a/crates/uffs-mft/src/commands/mod.rs b/crates/uffs-mft/src/commands/mod.rs index 1f7f2a4dc..509173c7d 100644 --- a/crates/uffs-mft/src/commands/mod.rs +++ b/crates/uffs-mft/src/commands/mod.rs @@ -15,6 +15,7 @@ mod windows; /// Dispatches parsed CLI commands to their handlers. #[expect( clippy::too_many_lines, + clippy::cognitive_complexity, reason = "single match arm per CLI subcommand keeps the dispatch table flat and easy to audit; splitting fragments the routing surface" )] #[cfg(windows)] @@ -42,6 +43,7 @@ pub(crate) async fn dispatch_command(command: Commands) -> Result<()> { kind, output, } => windows::cmd_metafile(drive, kind, &output).await, + Commands::Capture { drive, out } => windows::cmd_capture(drive, &out).await, Commands::Bench { drive, json, @@ -198,6 +200,7 @@ pub(crate) async fn dispatch_command(command: Commands) -> Result<()> { ), Commands::Sysinfo { out, json } => sysinfo::run(out.as_deref(), json), Commands::Metafile { .. } + | Commands::Capture { .. } | Commands::Read { .. } | Commands::Info { .. } | Commands::Drives { .. } diff --git a/crates/uffs-mft/src/commands/windows/capture.rs b/crates/uffs-mft/src/commands/windows/capture.rs new file mode 100644 index 000000000..38b7bcabf --- /dev/null +++ b/crates/uffs-mft/src/commands/windows/capture.rs @@ -0,0 +1,204 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) 2025-2026 SKY, LLC. + +//! `capture` command — bundle a drive's NTFS metafiles into one hashed, +//! manifested directory (`docs/architecture/mft-full-capture.md` §5–6). +//! +//! Writes each metafile (via [`uffs_mft::platform::metafile`]) into +//! `out/drive_/`, plus a `manifest.json` (volume facts + per-artifact +//! SHA-256) and a `SHA256SUMS` file for transfer verification. Best-effort: a +//! metafile that cannot be read is skipped and noted, not fatal. +#![expect( + clippy::print_stdout, + reason = "intentional user-facing CLI capture progress output" +)] + +use std::path::Path; + +use anyhow::{Context as _, Result}; +use sha2::{Digest as _, Sha256}; +use uffs_mft::platform::metafile::{self, MetafileHeader, MetafileKind}; +use uffs_mft::platform::{DriveLetter, VolumeHandle}; +use uffs_mft::usize_to_u64; + +/// One captured artifact, recorded in the manifest. +#[derive(serde::Serialize)] +struct ArtifactRecord { + /// File name within the capture directory. + file: String, + /// NTFS metafile name (e.g. `$Boot`). + kind: String, + /// Source MFT FRS number. + frs: u8, + /// File size in bytes (header + payload). + bytes: u64, + /// SHA-256 of the file, lowercase hex. + sha256: String, +} + +/// Source-volume facts recorded in the manifest. +#[derive(serde::Serialize)] +struct VolumeInfo { + /// Volume serial number, hex. + serial: String, + /// NTFS version (e.g. `3.1`). + ntfs_version: String, + /// Cluster size in bytes. + bytes_per_cluster: u32, + /// MFT record size in bytes. + mft_record_size: u32, +} + +/// The capture bundle manifest (`manifest.json`). +#[derive(serde::Serialize)] +struct Manifest { + /// Manifest schema version. + schema: u32, + /// Captured drive letter. + drive: String, + /// Capture timestamp (RFC 3339, UTC). + captured_at: String, + /// `uffs-mft` version. + tool_version: String, + /// Source-volume facts. + volume: VolumeInfo, + /// Captured artifacts. + artifacts: Vec, +} + +/// SHA-256 of a byte slice, lowercase hex. +fn sha256_hex(bytes: &[u8]) -> String { + let mut hasher = Sha256::new(); + hasher.update(bytes); + hex::encode(hasher.finalize()) +} + +/// Capture one metafile: read → save (with header) → hash. Returns its record. +fn capture_metafile( + drive: DriveLetter, + kind: MetafileKind, + dir: &Path, + drive_lower: &str, + serial: u64, + timestamp: u64, +) -> Result { + let stem = kind.name().trim_start_matches('$').to_lowercase(); + let file = format!("{drive_lower}_{stem}.bin"); + let path = dir.join(&file); + + let data = + metafile::read_metafile(drive, kind).with_context(|| format!("reading {}", kind.name()))?; + let header = MetafileHeader { + kind, + drive, + volume_serial: serial, + timestamp, + data_size: usize_to_u64(data.len()), + }; + metafile::save_metafile_to_file(&path, &header, &data) + .with_context(|| format!("saving {}", kind.name()))?; + let bytes = std::fs::read(&path).with_context(|| format!("re-reading {}", path.display()))?; + Ok(ArtifactRecord { + file, + kind: kind.name().to_owned(), + frs: kind.frs(), + bytes: usize_to_u64(bytes.len()), + sha256: sha256_hex(&bytes), + }) +} + +/// Assemble the capture manifest from volume facts and collected artifacts. +fn build_manifest( + drive: DriveLetter, + vol: &uffs_mft::platform::NtfsVolumeData, + artifacts: Vec, +) -> Manifest { + Manifest { + schema: 1, + drive: drive.to_string(), + captured_at: chrono::Utc::now().to_rfc3339_opts(chrono::SecondsFormat::Secs, true), + tool_version: env!("CARGO_PKG_VERSION").to_owned(), + volume: VolumeInfo { + serial: format!("0x{:016X}", vol.volume_serial_number), + ntfs_version: format!("{}.{}", vol.ntfs_major_version, vol.ntfs_minor_version), + bytes_per_cluster: vol.bytes_per_cluster, + mft_record_size: vol.bytes_per_file_record_segment, + }, + artifacts, + } +} + +/// Write `manifest.json` + `SHA256SUMS` into the capture directory. +fn write_bundle(dir: &Path, manifest: &Manifest) -> Result<()> { + let manifest_json = + serde_json::to_string_pretty(manifest).context("serialising manifest.json")?; + std::fs::write(dir.join("manifest.json"), &manifest_json).context("writing manifest.json")?; + + let mut sums = String::new(); + for artifact in &manifest.artifacts { + sums.push_str(&artifact.sha256); + sums.push_str(" "); + sums.push_str(&artifact.file); + sums.push('\n'); + } + std::fs::write(dir.join("SHA256SUMS"), sums).context("writing SHA256SUMS")?; + Ok(()) +} + +/// `capture` command — write all metafiles + `manifest.json` + `SHA256SUMS` +/// for a drive into `out/drive_/`. +pub(crate) async fn cmd_capture(drive: DriveLetter, out: &Path) -> Result<()> { + let drive_lower = drive.to_string().to_lowercase(); + let dir = out.join(format!("drive_{drive_lower}")); + std::fs::create_dir_all(&dir) + .with_context(|| format!("creating capture dir {}", dir.display()))?; + + let handle = VolumeHandle::open(drive).with_context(|| format!("Failed to open {drive}:"))?; + let vol = handle.volume_data(); + let serial = vol.volume_serial_number; + let timestamp = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |elapsed| elapsed.as_secs()); + + println!("═══════════════════════════════════════════════════════════════"); + println!( + " UFFS metafile capture — drive {drive}: → {}", + dir.display() + ); + println!("═══════════════════════════════════════════════════════════════"); + + let kinds = [ + MetafileKind::Boot, + MetafileKind::Bitmap, + MetafileKind::Secure, + MetafileKind::AttrDef, + MetafileKind::MftMirr, + MetafileKind::Volume, + MetafileKind::BadClus, + ]; + + let mut artifacts: Vec = Vec::new(); + for kind in kinds { + match capture_metafile(drive, kind, &dir, &drive_lower, serial, timestamp) { + Ok(record) => { + println!( + " ✅ {:<9} {:>12} bytes {}", + kind.name(), + record.bytes, + record.file + ); + artifacts.push(record); + } + Err(err) => println!(" ⚠️ {:<9} skipped — {err:#}", kind.name()), + } + } + + let manifest = build_manifest(drive, vol, artifacts); + write_bundle(&dir, &manifest)?; + + println!(); + println!(" Manifest: {}", dir.join("manifest.json").display()); + println!(" Hashes: {}", dir.join("SHA256SUMS").display()); + println!(" {} artifact(s) captured.", manifest.artifacts.len()); + Ok(()) +} diff --git a/crates/uffs-mft/src/commands/windows/mod.rs b/crates/uffs-mft/src/commands/windows/mod.rs index 421214a2c..aa0a2e333 100644 --- a/crates/uffs-mft/src/commands/windows/mod.rs +++ b/crates/uffs-mft/src/commands/windows/mod.rs @@ -7,6 +7,7 @@ mod bench; mod benchmark_index; mod benchmark_mft; mod bitmap_diag; +mod capture; mod drives; mod incremental; mod info; @@ -22,6 +23,7 @@ pub(crate) use self::benchmark_index::{ }; pub(crate) use self::benchmark_mft::cmd_benchmark_mft; pub(crate) use self::bitmap_diag::cmd_bitmap_diag; +pub(crate) use self::capture::cmd_capture; pub(crate) use self::drives::cmd_drives; pub(crate) use self::incremental::{ cmd_cache_clear, cmd_cache_get, cmd_cache_status, cmd_index_all, cmd_index_load, diff --git a/crates/uffs-mft/src/lib.rs b/crates/uffs-mft/src/lib.rs index f9a9b3b68..541abdc4c 100644 --- a/crates/uffs-mft/src/lib.rs +++ b/crates/uffs-mft/src/lib.rs @@ -176,6 +176,8 @@ use criterion as _; use dirs_next as _; #[cfg(test)] use hex as _; +#[cfg(windows)] +use hex as _; use hostname as _; use indicatif as _; #[cfg(test)] @@ -194,7 +196,9 @@ use rustc_hash as _; // commands (src/commands/windows/info.rs); silence the library's view of it. #[cfg(windows)] use serde_json as _; -#[cfg(test)] +// `sha2` + `hex` power the Windows-only `capture` command's manifest / SHA256SUMS +// (src/commands/windows/capture.rs) and the chaos tests; silence the library's view. +#[cfg(any(test, windows))] use sha2 as _; use smallvec as _; #[cfg(test)] From 15d5b8232b3ee0a262650902ff98f90f1ff42ce2 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Sun, 5 Jul 2026 12:28:21 -0700 Subject: [PATCH 08/39] =?UTF-8?q?feat(uffs-mft):=20metafile=20capture=20?= =?UTF-8?q?=E2=80=94=20$LogFile=20(P2)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Capture the NTFS metadata transaction log: $LogFile (FRS 2), an unnamed non-resident $DATA stream, via the P1.2 stream reader (no new I/O code). Wired into `metafile --kind log-file` and the `capture` bundle. This is the crash-consistency / replay artifact. The remaining temporal artifact, $UsnJrnl:$J, needs $Extend directory-index traversal to resolve its FRS and lands separately. Verified: host + cargo-xwin Windows clippy (pedantic/nursery) clean, tests green. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/cli.rs | 2 ++ crates/uffs-mft/src/commands/windows/capture.rs | 1 + crates/uffs-mft/src/commands/windows/save.rs | 1 + crates/uffs-mft/src/platform/metafile.rs | 12 +++++++++++- 4 files changed, 15 insertions(+), 1 deletion(-) diff --git a/crates/uffs-mft/src/cli.rs b/crates/uffs-mft/src/cli.rs index 436b0af4c..a29fbd423 100644 --- a/crates/uffs-mft/src/cli.rs +++ b/crates/uffs-mft/src/cli.rs @@ -40,6 +40,8 @@ pub(crate) enum MetafileKind { Volume, /// `$BadClus` — the MFT record (bad-cluster run list). BadClus, + /// `$LogFile` — the NTFS metadata transaction log. + LogFile, } /// `uffs-mft`: Low-level NTFS MFT reading tool. diff --git a/crates/uffs-mft/src/commands/windows/capture.rs b/crates/uffs-mft/src/commands/windows/capture.rs index 38b7bcabf..bc709a37d 100644 --- a/crates/uffs-mft/src/commands/windows/capture.rs +++ b/crates/uffs-mft/src/commands/windows/capture.rs @@ -175,6 +175,7 @@ pub(crate) async fn cmd_capture(drive: DriveLetter, out: &Path) -> Result<()> { MetafileKind::MftMirr, MetafileKind::Volume, MetafileKind::BadClus, + MetafileKind::LogFile, ]; let mut artifacts: Vec = Vec::new(); diff --git a/crates/uffs-mft/src/commands/windows/save.rs b/crates/uffs-mft/src/commands/windows/save.rs index 987fd294e..93c11eb47 100644 --- a/crates/uffs-mft/src/commands/windows/save.rs +++ b/crates/uffs-mft/src/commands/windows/save.rs @@ -406,6 +406,7 @@ pub(crate) async fn cmd_metafile( crate::cli::MetafileKind::MftMirr => MetafileKind::MftMirr, crate::cli::MetafileKind::Volume => MetafileKind::Volume, crate::cli::MetafileKind::BadClus => MetafileKind::BadClus, + crate::cli::MetafileKind::LogFile => MetafileKind::LogFile, }; // Volume serial for the header. diff --git a/crates/uffs-mft/src/platform/metafile.rs b/crates/uffs-mft/src/platform/metafile.rs index 28c549ec1..f23d818b6 100644 --- a/crates/uffs-mft/src/platform/metafile.rs +++ b/crates/uffs-mft/src/platform/metafile.rs @@ -58,6 +58,8 @@ pub enum MetafileKind { Volume, /// `$BadClus` (FRS 8) — the MFT record (the `$Bad` bad-cluster run list). BadClus, + /// `$LogFile` (FRS 2) — the NTFS metadata transaction log. + LogFile, } impl MetafileKind { @@ -72,6 +74,7 @@ impl MetafileKind { Self::MftMirr => "$MFTMirr", Self::Volume => "$Volume", Self::BadClus => "$BadClus", + Self::LogFile => "$LogFile", } } @@ -87,6 +90,7 @@ impl MetafileKind { Self::MftMirr => 1, Self::Volume => 3, Self::BadClus => 8, + Self::LogFile => 2, } } @@ -100,6 +104,7 @@ impl MetafileKind { 1 => Some(Self::MftMirr), 3 => Some(Self::Volume), 8 => Some(Self::BadClus), + 2 => Some(Self::LogFile), _ => None, } } @@ -222,9 +227,11 @@ pub fn read_metafile(drive: DriveLetter, kind: MetafileKind) -> Result> MetafileKind::Bitmap => read_data_stream(&handle, vol, 6, None), // `$Secure:$SDS` holds deduplicated security descriptors (ACLs). MetafileKind::Secure => read_data_stream(&handle, vol, 9, Some("$SDS")), - // `$AttrDef` / `$MFTMirr` are unnamed non-resident `$DATA` streams. + // `$AttrDef` / `$MFTMirr` / `$LogFile` are unnamed non-resident `$DATA` + // streams. MetafileKind::AttrDef => read_data_stream(&handle, vol, 4, None), MetafileKind::MftMirr => read_data_stream(&handle, vol, 1, None), + MetafileKind::LogFile => read_data_stream(&handle, vol, 2, None), // `$Volume` / `$BadClus` keep their useful data in the MFT record itself // (the $VOLUME_* attributes; the $Bad run list), so capture the fixed-up // record. @@ -427,6 +434,8 @@ mod tests { assert_eq!(MetafileKind::MftMirr.frs(), 1); assert_eq!(MetafileKind::Volume.frs(), 3); assert_eq!(MetafileKind::BadClus.frs(), 8); + assert_eq!(MetafileKind::LogFile.frs(), 2); + assert_eq!(MetafileKind::LogFile.name(), "$LogFile"); // FRS code round-trips through the header field. assert_eq!(MetafileKind::from_frs(6), Some(MetafileKind::Bitmap)); assert_eq!(MetafileKind::from_frs(7), Some(MetafileKind::Boot)); @@ -435,6 +444,7 @@ mod tests { assert_eq!(MetafileKind::from_frs(1), Some(MetafileKind::MftMirr)); assert_eq!(MetafileKind::from_frs(3), Some(MetafileKind::Volume)); assert_eq!(MetafileKind::from_frs(8), Some(MetafileKind::BadClus)); + assert_eq!(MetafileKind::from_frs(2), Some(MetafileKind::LogFile)); assert_eq!(MetafileKind::from_frs(200), None); } From 75571a82132df6c57f46dd48287a8f65ac2a9eac Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Sun, 5 Jul 2026 12:40:01 -0700 Subject: [PATCH 09/39] =?UTF-8?q?feat(uffs-mft):=20metafile=20capture=20?= =?UTF-8?q?=E2=80=94=20$UsnJrnl:$J=20via=20$Extend=20traversal=20(P2)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Capture the NTFS change journal, the one metafile not at a fixed FRS: - resolve_extend_child: walk the $Extend (FRS 11) directory index to resolve a child's FRS by name — parses $INDEX_ROOT entries, and falls back to the $INDEX_ALLOCATION INDX blocks (with USA fixup via apply_usa_fixup) when the index has spilled. - read_usn_journal: resolve $UsnJrnl's FRS, then read its named non-resident $DATA stream $J (sparse; the active tail) via the P1.2 stream reader. - MetafileKind::UsnJrnl uses a reserved sentinel kind code (0xF5) since its FRS is dynamic; wired into `metafile --kind usn-jrnl` and the `capture` bundle. - scan_index_entries is a pure parser with a cross-platform unit test (synthetic $INDEX_ROOT buffer → resolves a child FRS). Completes the NTFS metafile capture set: $MFT + $Boot/$Bitmap/$Secure/$AttrDef/ $MFTMirr/$Volume/$BadClus/$LogFile/$UsnJrnl (+ existing $UpCase). Verified: host pedantic + cargo-xwin Windows clippy clean, tests green. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/cli.rs | 2 + .../uffs-mft/src/commands/windows/capture.rs | 1 + crates/uffs-mft/src/commands/windows/save.rs | 1 + crates/uffs-mft/src/platform/metafile.rs | 191 +++++++++++++++++- 4 files changed, 193 insertions(+), 2 deletions(-) diff --git a/crates/uffs-mft/src/cli.rs b/crates/uffs-mft/src/cli.rs index a29fbd423..0030fee45 100644 --- a/crates/uffs-mft/src/cli.rs +++ b/crates/uffs-mft/src/cli.rs @@ -42,6 +42,8 @@ pub(crate) enum MetafileKind { BadClus, /// `$LogFile` — the NTFS metadata transaction log. LogFile, + /// `$UsnJrnl:$J` — the change journal (resolved via `$Extend`). + UsnJrnl, } /// `uffs-mft`: Low-level NTFS MFT reading tool. diff --git a/crates/uffs-mft/src/commands/windows/capture.rs b/crates/uffs-mft/src/commands/windows/capture.rs index bc709a37d..909925155 100644 --- a/crates/uffs-mft/src/commands/windows/capture.rs +++ b/crates/uffs-mft/src/commands/windows/capture.rs @@ -176,6 +176,7 @@ pub(crate) async fn cmd_capture(drive: DriveLetter, out: &Path) -> Result<()> { MetafileKind::Volume, MetafileKind::BadClus, MetafileKind::LogFile, + MetafileKind::UsnJrnl, ]; let mut artifacts: Vec = Vec::new(); diff --git a/crates/uffs-mft/src/commands/windows/save.rs b/crates/uffs-mft/src/commands/windows/save.rs index 93c11eb47..365b8fe2b 100644 --- a/crates/uffs-mft/src/commands/windows/save.rs +++ b/crates/uffs-mft/src/commands/windows/save.rs @@ -407,6 +407,7 @@ pub(crate) async fn cmd_metafile( crate::cli::MetafileKind::Volume => MetafileKind::Volume, crate::cli::MetafileKind::BadClus => MetafileKind::BadClus, crate::cli::MetafileKind::LogFile => MetafileKind::LogFile, + crate::cli::MetafileKind::UsnJrnl => MetafileKind::UsnJrnl, }; // Volume serial for the header. diff --git a/crates/uffs-mft/src/platform/metafile.rs b/crates/uffs-mft/src/platform/metafile.rs index f23d818b6..6ad864c0e 100644 --- a/crates/uffs-mft/src/platform/metafile.rs +++ b/crates/uffs-mft/src/platform/metafile.rs @@ -60,6 +60,10 @@ pub enum MetafileKind { BadClus, /// `$LogFile` (FRS 2) — the NTFS metadata transaction log. LogFile, + /// `$UsnJrnl:$J` — the change journal. Its FRS is resolved at runtime by + /// walking the `$Extend` directory index, so the header stores a reserved + /// sentinel kind code rather than a fixed FRS. + UsnJrnl, } impl MetafileKind { @@ -75,11 +79,15 @@ impl MetafileKind { Self::Volume => "$Volume", Self::BadClus => "$BadClus", Self::LogFile => "$LogFile", + Self::UsnJrnl => "$UsnJrnl", } } - /// The MFT file-record-segment (FRS) number of this metafile. Stored in the + /// The MFT file-record-segment (FRS) number of this metafile, stored in the /// header as a stable, self-documenting kind code. + /// + /// `$UsnJrnl` uses the reserved sentinel `0xF5` because its real FRS is + /// resolved at runtime via the `$Extend` directory index. #[must_use] pub const fn frs(self) -> u8 { match self { @@ -91,6 +99,7 @@ impl MetafileKind { Self::Volume => 3, Self::BadClus => 8, Self::LogFile => 2, + Self::UsnJrnl => 0xF5, } } @@ -105,6 +114,7 @@ impl MetafileKind { 3 => Some(Self::Volume), 8 => Some(Self::BadClus), 2 => Some(Self::LogFile), + 0xF5 => Some(Self::UsnJrnl), _ => None, } } @@ -237,6 +247,8 @@ pub fn read_metafile(drive: DriveLetter, kind: MetafileKind) -> Result> // record. MetafileKind::Volume => read_frs_record(&handle, vol, 3), MetafileKind::BadClus => read_frs_record(&handle, vol, 8), + // `$UsnJrnl:$J` lives under `$Extend`; resolve its FRS then read `$J`. + MetafileKind::UsnJrnl => read_usn_journal(&handle, vol), } } @@ -357,6 +369,149 @@ fn read_runs( Ok(buf) } +/// FRS of the `$Extend` metadata directory. +#[cfg(windows)] +const EXTEND_FRS: u64 = 11; + +/// Read the `$UsnJrnl:$J` change journal via `$Extend` directory traversal. +#[cfg(windows)] +fn read_usn_journal( + handle: &crate::platform::VolumeHandle, + vol: &crate::platform::NtfsVolumeData, +) -> Result> { + let usn_frs = resolve_extend_child(handle, vol, "$UsnJrnl")?; + read_data_stream(handle, vol, usn_frs, Some("$J")) +} + +/// Resolve the FRS of a `$Extend` (FRS 11) child by name, walking its directory +/// index — `$INDEX_ROOT` first, then `$INDEX_ALLOCATION` if the index is large. +#[cfg(windows)] +fn resolve_extend_child( + handle: &crate::platform::VolumeHandle, + vol: &crate::platform::NtfsVolumeData, + child: &str, +) -> Result { + use crate::ntfs::{AttributeIterator, AttributeType, DataRun}; + + let record = read_frs_record(handle, vol, EXTEND_FRS)?; + let target: Vec = child.encode_utf16().collect(); + + let mut root_value: Option> = None; + let mut block_size: usize = 0; + let mut alloc: Option<(Vec, u64)> = None; + + let attrs = AttributeIterator::new(&record) + .ok_or_else(|| MftError::InvalidData("$Extend: invalid record header".to_owned()))?; + for attr in attrs { + match attr.attribute_type() { + Some(AttributeType::IndexRoot) => { + if let Some(value) = attr.resident_value() { + // IndexRoot.bytes_per_index_block is at offset 8. + block_size = value + .get(8..12) + .and_then(|slice| slice.try_into().ok()) + .map_or(0, |bytes| u32::from_le_bytes(bytes) as usize); + root_value = Some(value.to_vec()); + } + } + Some(AttributeType::IndexAllocation) if attr.is_non_resident() => { + if let Some(nr) = attr.non_resident_data() { + alloc = Some((attr.data_runs(), nr.data_size.cast_unsigned())); + } + } + _ => {} + } + } + + // 1. Search the resident $INDEX_ROOT (its INDEX_HEADER begins at offset 16). + if let Some(frs) = root_value + .as_deref() + .and_then(|root| scan_index_entries(root, 16, &target)) + { + return Ok(frs); + } + + // 2. Search the $INDEX_ALLOCATION INDX blocks, if the index spilled. + if let Some((runs, size)) = alloc + && block_size > 0 + { + let buf = read_runs(handle.raw_handle(), &runs, vol.bytes_per_cluster, size)?; + if let Some(frs) = scan_index_blocks(&buf, block_size, &target) { + return Ok(frs); + } + } + + Err(MftError::InvalidData(format!( + "$Extend index: {child} not found (no active USN journal?)" + ))) +} + +/// Scan NTFS directory-index entries for the entry whose `$FILE_NAME` matches +/// `target` (UTF-16), returning its FRS. `header_start` is the byte offset of +/// the `INDEX_HEADER` within `buf` (whose `first_entry_offset` is relative to +/// it). +#[cfg(any(windows, test))] +fn scan_index_entries(buf: &[u8], header_start: usize, target: &[u16]) -> Option { + let first_entry_offset = + u32::from_le_bytes(buf.get(header_start..header_start + 4)?.try_into().ok()?) as usize; + let mut pos = header_start.checked_add(first_entry_offset)?; + loop { + let entry = buf.get(pos..)?; + let flags = u16::from_le_bytes(entry.get(12..14)?.try_into().ok()?); + // Last-entry flag (0x02): end of this node. + if (flags & 0x02) != 0 { + return None; + } + let file_reference = u64::from_le_bytes(entry.get(0..8)?.try_into().ok()?); + // The $FILE_NAME key starts at entry offset 16: name_length @0x40, + // UTF-16 name @0x42. + let name_length = usize::from(*entry.get(16 + 0x40)?); + let name_bytes = entry.get(16 + 0x42..16 + 0x42 + name_length * 2)?; + let name: Vec = name_bytes + .chunks_exact(2) + .filter_map(|pair| pair.try_into().ok().map(u16::from_le_bytes)) + .collect(); + if name == target { + return Some(crate::ntfs::file_reference_to_frs(file_reference)); + } + let entry_length = usize::from(u16::from_le_bytes(entry.get(8..10)?.try_into().ok()?)); + if entry_length == 0 { + return None; + } + pos = pos.checked_add(entry_length)?; + } +} + +/// Scan the `INDX` blocks in an `$INDEX_ALLOCATION` buffer for `target`. +#[cfg(windows)] +fn scan_index_blocks(buf: &[u8], block_size: usize, target: &[u16]) -> Option { + if block_size == 0 { + return None; + } + for start in (0..buf.len()).step_by(block_size) { + let Some(end) = start.checked_add(block_size) else { + break; + }; + let Some(block) = buf.get(start..end) else { + break; + }; + if !block.starts_with(b"INDX") { + continue; // sparse / unused block + } + let mut owned = block.to_vec(); + let usa_offset = u16::from_le_bytes(owned.get(4..6)?.try_into().ok()?); + let usa_count = u16::from_le_bytes(owned.get(6..8)?.try_into().ok()?); + if !crate::ntfs::apply_usa_fixup(&mut owned, usa_offset, usa_count) { + continue; + } + // The INDEX_HEADER begins at offset 0x18 within an INDX block. + if let Some(frs) = scan_index_entries(&owned, 0x18, target) { + return Some(frs); + } + } + None +} + /// Write a captured metafile (header + payload) to `path` atomically. /// /// # Errors @@ -397,7 +552,7 @@ pub fn load_metafile_from_file(path: &Path) -> Result<(MetafileHeader, Vec)> #[cfg(test)] mod tests { - use super::{HEADER_SIZE, MetafileHeader, MetafileKind}; + use super::{HEADER_SIZE, MetafileHeader, MetafileKind, scan_index_entries}; use crate::platform::DriveLetter; fn sample() -> MetafileHeader { @@ -436,6 +591,8 @@ mod tests { assert_eq!(MetafileKind::BadClus.frs(), 8); assert_eq!(MetafileKind::LogFile.frs(), 2); assert_eq!(MetafileKind::LogFile.name(), "$LogFile"); + assert_eq!(MetafileKind::UsnJrnl.frs(), 0xF5); + assert_eq!(MetafileKind::UsnJrnl.name(), "$UsnJrnl"); // FRS code round-trips through the header field. assert_eq!(MetafileKind::from_frs(6), Some(MetafileKind::Bitmap)); assert_eq!(MetafileKind::from_frs(7), Some(MetafileKind::Boot)); @@ -445,6 +602,7 @@ mod tests { assert_eq!(MetafileKind::from_frs(3), Some(MetafileKind::Volume)); assert_eq!(MetafileKind::from_frs(8), Some(MetafileKind::BadClus)); assert_eq!(MetafileKind::from_frs(2), Some(MetafileKind::LogFile)); + assert_eq!(MetafileKind::from_frs(0xF5), Some(MetafileKind::UsnJrnl)); assert_eq!(MetafileKind::from_frs(200), None); } @@ -471,4 +629,33 @@ mod tests { fn payload_offset_is_header_size() { assert_eq!(HEADER_SIZE, 64); } + + #[test] + #[expect( + clippy::indexing_slicing, + reason = "test builds a fixed 256-byte index buffer with known in-bounds offsets" + )] + fn scan_index_entries_resolves_child_frs() { + // Minimal $INDEX_ROOT-style buffer: INDEX_HEADER at offset 0 + // (first_entry_offset = 16), one entry for "$UsnJrnl" → FRS 42. + let mut buf = vec![0_u8; 256]; + buf[0..4].copy_from_slice(&16_u32.to_le_bytes()); // first_entry_offset + let entry = 16_usize; + buf[entry..entry + 8].copy_from_slice(&42_u64.to_le_bytes()); // file_reference + // flags @ entry+12 stay 0 (not the last entry). + let name: Vec = "$UsnJrnl".encode_utf16().collect(); + let key = entry + 16; + buf[key + 0x40] = 8; // name_length (UTF-16 code units) + for (i, unit) in name.iter().enumerate() { + let off = key + 0x42 + i * 2; + buf[off..off + 2].copy_from_slice(&unit.to_le_bytes()); + } + + let target: Vec = "$UsnJrnl".encode_utf16().collect(); + assert_eq!(scan_index_entries(&buf, 0, &target), Some(42)); + + // A miss: entry_length is 0 after the single entry, so the scan stops. + let miss: Vec = "$Nope".encode_utf16().collect(); + assert_eq!(scan_index_entries(&buf, 0, &miss), None); + } } From 1186186d81e313d23275bbe01823f80509aeda82 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Sun, 5 Jul 2026 12:43:24 -0700 Subject: [PATCH 10/39] feat(uffs-mft): include compressed $MFT in the capture bundle MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `uffs-mft capture` now writes the primary artifact — the LZ4-compressed $MFT (_mft.compressed.bin, via MftReader::save_raw_to_file) — alongside the metafiles, so the bundle is self-sufficient (MFT + all metafiles + manifest + SHA256SUMS). Refactored the per-artifact loop into collect_artifacts/capture_mft to keep cmd_capture simple. Verified: host pedantic + cargo-xwin Windows clippy clean, tests green. Co-Authored-By: Claude Opus 4.8 --- .../uffs-mft/src/commands/windows/capture.rs | 130 ++++++++++++------ 1 file changed, 90 insertions(+), 40 deletions(-) diff --git a/crates/uffs-mft/src/commands/windows/capture.rs b/crates/uffs-mft/src/commands/windows/capture.rs index 909925155..4fb062986 100644 --- a/crates/uffs-mft/src/commands/windows/capture.rs +++ b/crates/uffs-mft/src/commands/windows/capture.rs @@ -1,13 +1,14 @@ // SPDX-License-Identifier: MPL-2.0 // Copyright (c) 2025-2026 SKY, LLC. -//! `capture` command — bundle a drive's NTFS metafiles into one hashed, -//! manifested directory (`docs/architecture/mft-full-capture.md` §5–6). +//! `capture` command — bundle a drive's `$MFT` + NTFS metafiles into one +//! hashed, manifested directory (`docs/architecture/mft-full-capture.md` §5–6). //! -//! Writes each metafile (via [`uffs_mft::platform::metafile`]) into -//! `out/drive_/`, plus a `manifest.json` (volume facts + per-artifact -//! SHA-256) and a `SHA256SUMS` file for transfer verification. Best-effort: a -//! metafile that cannot be read is skipped and noted, not fatal. +//! Writes the compressed `$MFT` and each metafile (via +//! [`uffs_mft::platform::metafile`]) into `out/drive_/`, plus a +//! `manifest.json` (volume facts + per-artifact SHA-256) and a `SHA256SUMS` +//! file for transfer verification. Best-effort: an artifact that cannot be read +//! is skipped and noted, not fatal. #![expect( clippy::print_stdout, reason = "intentional user-facing CLI capture progress output" @@ -145,43 +146,69 @@ fn write_bundle(dir: &Path, manifest: &Manifest) -> Result<()> { Ok(()) } -/// `capture` command — write all metafiles + `manifest.json` + `SHA256SUMS` -/// for a drive into `out/drive_/`. -pub(crate) async fn cmd_capture(drive: DriveLetter, out: &Path) -> Result<()> { - let drive_lower = drive.to_string().to_lowercase(); - let dir = out.join(format!("drive_{drive_lower}")); - std::fs::create_dir_all(&dir) - .with_context(|| format!("creating capture dir {}", dir.display()))?; +/// The NTFS metafiles the `capture` command bundles alongside the `$MFT`. +const METAFILE_KINDS: [MetafileKind; 9] = [ + MetafileKind::Boot, + MetafileKind::Bitmap, + MetafileKind::Secure, + MetafileKind::AttrDef, + MetafileKind::MftMirr, + MetafileKind::Volume, + MetafileKind::BadClus, + MetafileKind::LogFile, + MetafileKind::UsnJrnl, +]; - let handle = VolumeHandle::open(drive).with_context(|| format!("Failed to open {drive}:"))?; - let vol = handle.volume_data(); - let serial = vol.volume_serial_number; - let timestamp = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .map_or(0, |elapsed| elapsed.as_secs()); +/// Capture the compressed `$MFT` (LZ4) into the bundle. +fn capture_mft(drive: DriveLetter, dir: &Path, drive_lower: &str) -> Result { + use uffs_mft::{MftReader, SaveRawOptions}; - println!("═══════════════════════════════════════════════════════════════"); - println!( - " UFFS metafile capture — drive {drive}: → {}", - dir.display() - ); - println!("═══════════════════════════════════════════════════════════════"); + let file = format!("{drive_lower}_mft.compressed.bin"); + let path = dir.join(&file); + let reader = MftReader::open(drive).with_context(|| format!("opening $MFT on {drive}:"))?; + let options = SaveRawOptions { + compress: true, + compression_level: 3, + volume_letter: drive, + raw_compat: false, + }; + reader + .save_raw_to_file(&path, &options) + .with_context(|| format!("saving $MFT to {}", path.display()))?; + let bytes = std::fs::read(&path).with_context(|| format!("re-reading {}", path.display()))?; + Ok(ArtifactRecord { + file, + kind: "$MFT".to_owned(), + frs: 0, + bytes: usize_to_u64(bytes.len()), + sha256: sha256_hex(&bytes), + }) +} - let kinds = [ - MetafileKind::Boot, - MetafileKind::Bitmap, - MetafileKind::Secure, - MetafileKind::AttrDef, - MetafileKind::MftMirr, - MetafileKind::Volume, - MetafileKind::BadClus, - MetafileKind::LogFile, - MetafileKind::UsnJrnl, - ]; - - let mut artifacts: Vec = Vec::new(); - for kind in kinds { - match capture_metafile(drive, kind, &dir, &drive_lower, serial, timestamp) { +/// Capture the `$MFT` + every metafile into `dir`, printing progress and +/// returning their manifest records (best-effort: failures are skipped/noted). +fn collect_artifacts( + drive: DriveLetter, + dir: &Path, + drive_lower: &str, + serial: u64, + timestamp: u64, +) -> Vec { + let mut artifacts = Vec::new(); + + match capture_mft(drive, dir, drive_lower) { + Ok(record) => { + println!( + " ✅ {:<9} {:>12} bytes {}", + "$MFT", record.bytes, record.file + ); + artifacts.push(record); + } + Err(err) => println!(" ⚠️ {:<9} skipped — {err:#}", "$MFT"), + } + + for kind in METAFILE_KINDS { + match capture_metafile(drive, kind, dir, drive_lower, serial, timestamp) { Ok(record) => { println!( " ✅ {:<9} {:>12} bytes {}", @@ -195,6 +222,29 @@ pub(crate) async fn cmd_capture(drive: DriveLetter, out: &Path) -> Result<()> { } } + artifacts +} + +/// `capture` command — write the `$MFT` + all metafiles + `manifest.json` + +/// `SHA256SUMS` for a drive into `out/drive_/`. +pub(crate) async fn cmd_capture(drive: DriveLetter, out: &Path) -> Result<()> { + let drive_lower = drive.to_string().to_lowercase(); + let dir = out.join(format!("drive_{drive_lower}")); + std::fs::create_dir_all(&dir) + .with_context(|| format!("creating capture dir {}", dir.display()))?; + + let handle = VolumeHandle::open(drive).with_context(|| format!("Failed to open {drive}:"))?; + let vol = handle.volume_data(); + let serial = vol.volume_serial_number; + let timestamp = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |elapsed| elapsed.as_secs()); + + println!("═══════════════════════════════════════════════════════════════"); + println!(" UFFS capture — drive {drive}: → {}", dir.display()); + println!("═══════════════════════════════════════════════════════════════"); + + let artifacts = collect_artifacts(drive, &dir, &drive_lower, serial, timestamp); let manifest = build_manifest(drive, vol, artifacts); write_bundle(&dir, &manifest)?; From 51ec34b2a75579af5e0ec5cdf89bf9d774c4a9ef Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Sun, 5 Jul 2026 12:44:44 -0700 Subject: [PATCH 11/39] docs(uffs-mft): mark capture plan P0-P2/P4 shipped Update the full-capture plan status/phasing to reflect what landed: sysinfo, all metafile save targets ($Boot..$UsnJrnl via $Extend traversal), and the capture orchestrator/manifest incl. the compressed $MFT. Remaining: P3 (VSS --volume-path), P5 (rust-script VSS/zip wrapper), P6 (full offline load). Co-Authored-By: Claude Opus 4.8 --- docs/architecture/mft-full-capture.md | 38 +++++++++++++++++---------- 1 file changed, 24 insertions(+), 14 deletions(-) diff --git a/docs/architecture/mft-full-capture.md b/docs/architecture/mft-full-capture.md index e44c495ae..51b3e4a72 100644 --- a/docs/architecture/mft-full-capture.md +++ b/docs/architecture/mft-full-capture.md @@ -3,7 +3,10 @@ # NTFS Full-Volume Capture (`uffs-mft capture`) -> **Status:** Draft / plan (pre-implementation) +> **Status:** In progress — P0–P2 + P4 shipped (`sysinfo`, all metafile `save` +> targets incl. `$UsnJrnl` via `$Extend` traversal, and the `capture` +> orchestrator/manifest incl. the compressed `$MFT`). Remaining: P3 (VSS +> `--volume-path`) and P5 (`scripts/capture.rs` VSS/zip wrapper). > **Owner:** UFFS core > **Goal:** Capture *everything* needed to reconstitute a live Windows NTFS volume > offline "as accurately as possible" — not just the namespace, but ACLs, the @@ -225,19 +228,26 @@ loaders are additive (namespace path unchanged). ## 11. Phasing -0. **P0 — `uffs-mft sysinfo`** (host probe): OS + `InstallationType` (client/server), - elevation, VSS availability, per-drive media type (HDD/SSD/NVMe) + geometry. - Emits `capture_host.txt` + the manifest `host` block; returns a capability set - the orchestrator uses to pick the best-effort path. Cross-platform-buildable - (facts are host-gathered; non-Windows returns a stub). **Ships first** — it - gates everything and is independently useful. -1. **P1 — frozen metafile readers + `save` targets** (`$Secure`, `$Bitmap`, - `$Boot`, `$AttrDef`, `$Volume`, `$MFTMirr`, `$UpCase`, `$BadClus`). -2. **P2 — `$UsnJrnl:$J` + `$LogFile` + `$Extend\*` dump.** -3. **P3 — `--volume-path` (shadow device) support in `VolumeHandle`.** -4. **P4 — `capture` orchestrator subcommand + `manifest.json`.** -5. **P5 — `scripts/capture.rs`: WMI VSS create/delete + `--zip`/split + hashes.** -6. **P6 — matching `load` + `verify_parity` wiring.** +0. ✅ **P0 — `uffs-mft sysinfo`** (host probe): OS + `InstallationType` + (client/server), elevation, VSS availability, per-drive media type + (HDD/SSD/NVMe) + geometry, and every mounted volume (type/format/size/used%). + Emits `capture_host.txt` (+ `--json`); cross-platform (non-Windows = real host + facts, no capture). *(commit 56b09cec0)* +1. ✅ **P1 — metafile `save` targets** (`$Boot`, `$Bitmap`, `$Secure:$SDS`, + `$AttrDef`, `$MFTMirr`, `$Volume`, `$BadClus`; `$UpCase` pre-existing) on a + generic data-run stream reader. `uffs-mft metafile --kind `. +2. ✅ **P2 — `$LogFile`** (FRS 2) **+ `$UsnJrnl:$J` via `$Extend` traversal** + (`$INDEX_ROOT` + `$INDEX_ALLOCATION` INDX-block parse to resolve the FRS). +4. ✅ **P4 — `capture` orchestrator**: `uffs-mft capture --drive C --out ` + writes the compressed `$MFT` + all metafiles + `manifest.json` (per-artifact + SHA-256) + `SHA256SUMS` into `drive_/`. `MetafileHeader` + + `load_metafile_from_file` provide the offline read side (P6 partial). +3. ⏳ **P3 — `--volume-path` (shadow device) support in `VolumeHandle`** (read a + VSS snapshot device directly) — not yet. +5. ⏳ **P5 — `scripts/capture.rs`: WMI VSS create/delete + `--zip`/split + hashes** + — not yet. +6. ⏳ **P6 — full `load` + `verify_parity` wiring** for the new artifacts — partial + (metafile header round-trips; per-metafile parsers TBD). ## 12. Open questions From 725a812167c106726d7f5d94b61bb22451f31ae4 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Sun, 5 Jul 2026 15:47:16 -0700 Subject: [PATCH 12/39] =?UTF-8?q?feat(uffs-mft):=20offline=20metafile=20in?= =?UTF-8?q?spection=20=E2=80=94=20metafile-info=20+=20$Boot=20decode=20(P6?= =?UTF-8?q?)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The offline reconstitute/validate side of the capture flow — cross-platform, so it runs on the transfer target (macOS/Linux) as well as Windows. - platform::metafile::parse_boot: decode volume geometry (bytes/sector, sectors/cluster, cluster + MFT record size, total sectors, MFT LCN, serial) from a captured $Boot payload via NtfsBootSector. - platform::metafile::summarize: human summary of a captured metafile (header + kind-specific detail). - `uffs-mft metafile-info --input `: load a captured metafile and print the summary. Dispatched on all platforms. - Real cross-platform unit test: synthetic 512-byte boot sector → decoded geometry (runs on macOS). Verified end-to-end on macOS against a synthetic $Boot metafile. This closes the loop: capture on Windows → transfer → `metafile-info` on the Mac reproduces the same decode. Verified: host pedantic + cargo-xwin Windows clippy clean, tests green, macOS e2e run. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/cli.rs | 7 ++ crates/uffs-mft/src/commands/metafile_info.rs | 32 ++++++ crates/uffs-mft/src/commands/mod.rs | 3 + crates/uffs-mft/src/platform/metafile.rs | 107 +++++++++++++++++- 4 files changed, 148 insertions(+), 1 deletion(-) create mode 100644 crates/uffs-mft/src/commands/metafile_info.rs diff --git a/crates/uffs-mft/src/cli.rs b/crates/uffs-mft/src/cli.rs index 0030fee45..1db7a7c83 100644 --- a/crates/uffs-mft/src/cli.rs +++ b/crates/uffs-mft/src/cli.rs @@ -176,6 +176,13 @@ pub(crate) enum Commands { out: PathBuf, }, + /// Inspect a captured NTFS metafile offline (header + $Boot geometry, ...) + MetafileInfo { + /// Path to a captured metafile (from `metafile` or `capture`) + #[arg(short, long)] + input: PathBuf, + }, + /// Benchmark MFT reading with detailed phase timing Bench { /// Drive letter (e.g., C, D, E) diff --git a/crates/uffs-mft/src/commands/metafile_info.rs b/crates/uffs-mft/src/commands/metafile_info.rs new file mode 100644 index 000000000..f3a9f94ca --- /dev/null +++ b/crates/uffs-mft/src/commands/metafile_info.rs @@ -0,0 +1,32 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) 2025-2026 SKY, LLC. + +//! `metafile-info` command — offline inspection of a captured NTFS metafile. +//! +//! Cross-platform: reads a file written by `metafile` / `capture`, validates +//! its header, and prints a kind-specific summary (e.g. `$Boot` geometry). Runs +//! on the offline machine (macOS/Linux) as well as Windows — the reconstitute / +//! validate side of the capture flow. +#![expect( + clippy::print_stdout, + reason = "intentional user-facing CLI metafile summary output" +)] + +use std::path::Path; + +use anyhow::{Context as _, Result}; + +/// Inspect a captured metafile: load it and print its header + summary. +/// +/// # Errors +/// +/// Returns an error if the file cannot be read or lacks a valid metafile +/// header. +pub(crate) fn cmd_metafile_info(input: &Path) -> Result<()> { + use uffs_mft::platform::metafile; + + let (header, payload) = metafile::load_metafile_from_file(input) + .with_context(|| format!("loading metafile {}", input.display()))?; + print!("{}", metafile::summarize(&header, &payload)); + Ok(()) +} diff --git a/crates/uffs-mft/src/commands/mod.rs b/crates/uffs-mft/src/commands/mod.rs index 509173c7d..cd603ecdc 100644 --- a/crates/uffs-mft/src/commands/mod.rs +++ b/crates/uffs-mft/src/commands/mod.rs @@ -8,6 +8,7 @@ use anyhow::Result; use crate::cli::Commands; mod load; +mod metafile_info; mod sysinfo; #[cfg(windows)] mod windows; @@ -44,6 +45,7 @@ pub(crate) async fn dispatch_command(command: Commands) -> Result<()> { output, } => windows::cmd_metafile(drive, kind, &output).await, Commands::Capture { drive, out } => windows::cmd_capture(drive, &out).await, + Commands::MetafileInfo { input } => metafile_info::cmd_metafile_info(&input), Commands::Bench { drive, json, @@ -199,6 +201,7 @@ pub(crate) async fn dispatch_command(command: Commands) -> Result<()> { forensic, ), Commands::Sysinfo { out, json } => sysinfo::run(out.as_deref(), json), + Commands::MetafileInfo { input } => metafile_info::cmd_metafile_info(&input), Commands::Metafile { .. } | Commands::Capture { .. } | Commands::Read { .. } diff --git a/crates/uffs-mft/src/platform/metafile.rs b/crates/uffs-mft/src/platform/metafile.rs index 6ad864c0e..5a2a95cf8 100644 --- a/crates/uffs-mft/src/platform/metafile.rs +++ b/crates/uffs-mft/src/platform/metafile.rs @@ -550,9 +550,86 @@ pub fn load_metafile_from_file(path: &Path) -> Result<(MetafileHeader, Vec)> Ok((header, payload)) } +/// Volume geometry decoded from a captured `$Boot` payload (offline read side). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct BootGeometry { + /// Bytes per sector. + pub bytes_per_sector: u16, + /// Sectors per cluster. + pub sectors_per_cluster: u8, + /// Cluster size in bytes. + pub bytes_per_cluster: u32, + /// MFT file-record size in bytes. + pub mft_record_size: u32, + /// Total sectors on the volume. + pub total_sectors: u64, + /// Logical cluster number of `$MFT`. + pub mft_start_lcn: u64, + /// Volume serial number. + pub volume_serial: u64, +} + +/// Decode the NTFS volume geometry from a captured `$Boot` payload. +/// +/// # Errors +/// +/// Returns [`MftError::InvalidData`] if the payload is too small or is not a +/// valid NTFS boot sector. +pub fn parse_boot(payload: &[u8]) -> Result { + use zerocopy::FromBytes as _; + + let (boot, _) = crate::ntfs::NtfsBootSector::read_from_prefix(payload) + .map_err(|_err| MftError::InvalidData("$Boot payload too small".to_owned()))?; + if !boot.is_valid() { + return Err(MftError::InvalidData( + "payload is not a valid NTFS boot sector".to_owned(), + )); + } + Ok(BootGeometry { + bytes_per_sector: boot.bytes_per_sector, + sectors_per_cluster: boot.sectors_per_cluster, + bytes_per_cluster: boot.cluster_size(), + mft_record_size: boot.file_record_size(), + total_sectors: boot.total_sectors.cast_unsigned(), + mft_start_lcn: boot.mft_start_lcn.cast_unsigned(), + volume_serial: boot.volume_serial_number.cast_unsigned(), + }) +} + +/// A human-readable summary of a captured metafile (its header, plus +/// kind-specific detail such as `$Boot` geometry). Offline / cross-platform. +#[must_use] +pub fn summarize(header: &MetafileHeader, payload: &[u8]) -> String { + let base = format!( + "Metafile: {}\n Drive: {}:\n Serial: 0x{:016X}\n Captured (epoch s): {}\n Payload: {} bytes\n", + header.kind.name(), + header.drive, + header.volume_serial, + header.timestamp, + payload.len(), + ); + let detail = if header.kind == MetafileKind::Boot { + match parse_boot(payload) { + Ok(geo) => format!( + " $Boot: {} B/sector x {} sec/clu = {} B/cluster; MFT rec {} B; MFT LCN {}; total sectors {}\n", + geo.bytes_per_sector, + geo.sectors_per_cluster, + geo.bytes_per_cluster, + geo.mft_record_size, + geo.mft_start_lcn, + geo.total_sectors, + ), + Err(err) => format!(" $Boot parse failed: {err}\n"), + } + } else { + String::new() + }; + format!("{base}{detail}") +} + #[cfg(test)] mod tests { - use super::{HEADER_SIZE, MetafileHeader, MetafileKind, scan_index_entries}; + use super::{HEADER_SIZE, MetafileHeader, MetafileKind, parse_boot, scan_index_entries}; use crate::platform::DriveLetter; fn sample() -> MetafileHeader { @@ -658,4 +735,32 @@ mod tests { let miss: Vec = "$Nope".encode_utf16().collect(); assert_eq!(scan_index_entries(&buf, 0, &miss), None); } + + #[test] + #[expect( + clippy::indexing_slicing, + reason = "test builds a fixed 512-byte boot sector with known in-bounds offsets" + )] + fn parse_boot_decodes_geometry() { + let mut boot = vec![0_u8; 512]; + boot[3..7].copy_from_slice(b"NTFS"); // oem_id + boot[11..13].copy_from_slice(&512_u16.to_le_bytes()); // bytes_per_sector + boot[13] = 8; // sectors_per_cluster + boot[40..48].copy_from_slice(&1_000_000_i64.to_le_bytes()); // total_sectors + boot[48..56].copy_from_slice(&786_432_i64.to_le_bytes()); // mft_start_lcn + boot[64] = (-10_i8).cast_unsigned(); // clusters_per_file_record → 2^10 = 1024 + boot[72..80].copy_from_slice(&0x1122_3344_5566_7788_i64.to_le_bytes()); // serial + + let geo = parse_boot(&boot).expect("valid boot sector"); + assert_eq!(geo.bytes_per_sector, 512); + assert_eq!(geo.sectors_per_cluster, 8); + assert_eq!(geo.bytes_per_cluster, 4096); + assert_eq!(geo.mft_record_size, 1024); + assert_eq!(geo.total_sectors, 1_000_000); + assert_eq!(geo.mft_start_lcn, 786_432); + assert_eq!(geo.volume_serial, 0x1122_3344_5566_7788); + + // Not a boot sector → error. + parse_boot(&[0_u8; 512]).unwrap_err(); + } } From 7d4403333413e2b6b1d2f35c07e35e70d7055bc6 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Sun, 5 Jul 2026 15:59:06 -0700 Subject: [PATCH 13/39] feat(uffs-mft): offline $Bitmap decode + split metafile_decode module (P6) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - parse_bitmap: decode cluster-allocation stats (total/used/free clusters) from a captured $Bitmap payload by popcounting the allocation bitmap; folded into summarize / `metafile-info` for offline free-space validation (vs fsutil). - Extract the pure offline decoders (BootGeometry/parse_boot, BitmapStats/ parse_bitmap, summarize) into a new cross-platform `platform::metafile_decode` module — the $Bitmap addition pushed metafile.rs over the file-size limit, so this is the real fix (metafile.rs 826 → 661 LOC), no exception. Cross-platform + unit-tested; verified end-to-end on macOS. host pedantic + cargo-xwin Windows clippy clean. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/commands/metafile_info.rs | 4 +- crates/uffs-mft/src/platform.rs | 3 + crates/uffs-mft/src/platform/metafile.rs | 107 +---------- .../uffs-mft/src/platform/metafile_decode.rs | 179 ++++++++++++++++++ 4 files changed, 185 insertions(+), 108 deletions(-) create mode 100644 crates/uffs-mft/src/platform/metafile_decode.rs diff --git a/crates/uffs-mft/src/commands/metafile_info.rs b/crates/uffs-mft/src/commands/metafile_info.rs index f3a9f94ca..459635ffe 100644 --- a/crates/uffs-mft/src/commands/metafile_info.rs +++ b/crates/uffs-mft/src/commands/metafile_info.rs @@ -23,10 +23,10 @@ use anyhow::{Context as _, Result}; /// Returns an error if the file cannot be read or lacks a valid metafile /// header. pub(crate) fn cmd_metafile_info(input: &Path) -> Result<()> { - use uffs_mft::platform::metafile; + use uffs_mft::platform::{metafile, metafile_decode}; let (header, payload) = metafile::load_metafile_from_file(input) .with_context(|| format!("loading metafile {}", input.display()))?; - print!("{}", metafile::summarize(&header, &payload)); + print!("{}", metafile_decode::summarize(&header, &payload)); Ok(()) } diff --git a/crates/uffs-mft/src/platform.rs b/crates/uffs-mft/src/platform.rs index 8ca1fe69b..af76ab6bd 100644 --- a/crates/uffs-mft/src/platform.rs +++ b/crates/uffs-mft/src/platform.rs @@ -39,6 +39,9 @@ mod extents; pub mod lcn; /// NTFS metafile capture ($Boot, ...) from a live volume. pub mod metafile; +/// Offline decoders for captured metafiles ($Boot geometry, $Bitmap free +/// space). +pub mod metafile_decode; /// Native Windows process introspection for the self-update detector. /// /// Image path + pid enumeration — keeps the `unsafe` FFI out of `uffs-cli`. diff --git a/crates/uffs-mft/src/platform/metafile.rs b/crates/uffs-mft/src/platform/metafile.rs index 5a2a95cf8..6ad864c0e 100644 --- a/crates/uffs-mft/src/platform/metafile.rs +++ b/crates/uffs-mft/src/platform/metafile.rs @@ -550,86 +550,9 @@ pub fn load_metafile_from_file(path: &Path) -> Result<(MetafileHeader, Vec)> Ok((header, payload)) } -/// Volume geometry decoded from a captured `$Boot` payload (offline read side). -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub struct BootGeometry { - /// Bytes per sector. - pub bytes_per_sector: u16, - /// Sectors per cluster. - pub sectors_per_cluster: u8, - /// Cluster size in bytes. - pub bytes_per_cluster: u32, - /// MFT file-record size in bytes. - pub mft_record_size: u32, - /// Total sectors on the volume. - pub total_sectors: u64, - /// Logical cluster number of `$MFT`. - pub mft_start_lcn: u64, - /// Volume serial number. - pub volume_serial: u64, -} - -/// Decode the NTFS volume geometry from a captured `$Boot` payload. -/// -/// # Errors -/// -/// Returns [`MftError::InvalidData`] if the payload is too small or is not a -/// valid NTFS boot sector. -pub fn parse_boot(payload: &[u8]) -> Result { - use zerocopy::FromBytes as _; - - let (boot, _) = crate::ntfs::NtfsBootSector::read_from_prefix(payload) - .map_err(|_err| MftError::InvalidData("$Boot payload too small".to_owned()))?; - if !boot.is_valid() { - return Err(MftError::InvalidData( - "payload is not a valid NTFS boot sector".to_owned(), - )); - } - Ok(BootGeometry { - bytes_per_sector: boot.bytes_per_sector, - sectors_per_cluster: boot.sectors_per_cluster, - bytes_per_cluster: boot.cluster_size(), - mft_record_size: boot.file_record_size(), - total_sectors: boot.total_sectors.cast_unsigned(), - mft_start_lcn: boot.mft_start_lcn.cast_unsigned(), - volume_serial: boot.volume_serial_number.cast_unsigned(), - }) -} - -/// A human-readable summary of a captured metafile (its header, plus -/// kind-specific detail such as `$Boot` geometry). Offline / cross-platform. -#[must_use] -pub fn summarize(header: &MetafileHeader, payload: &[u8]) -> String { - let base = format!( - "Metafile: {}\n Drive: {}:\n Serial: 0x{:016X}\n Captured (epoch s): {}\n Payload: {} bytes\n", - header.kind.name(), - header.drive, - header.volume_serial, - header.timestamp, - payload.len(), - ); - let detail = if header.kind == MetafileKind::Boot { - match parse_boot(payload) { - Ok(geo) => format!( - " $Boot: {} B/sector x {} sec/clu = {} B/cluster; MFT rec {} B; MFT LCN {}; total sectors {}\n", - geo.bytes_per_sector, - geo.sectors_per_cluster, - geo.bytes_per_cluster, - geo.mft_record_size, - geo.mft_start_lcn, - geo.total_sectors, - ), - Err(err) => format!(" $Boot parse failed: {err}\n"), - } - } else { - String::new() - }; - format!("{base}{detail}") -} - #[cfg(test)] mod tests { - use super::{HEADER_SIZE, MetafileHeader, MetafileKind, parse_boot, scan_index_entries}; + use super::{HEADER_SIZE, MetafileHeader, MetafileKind, scan_index_entries}; use crate::platform::DriveLetter; fn sample() -> MetafileHeader { @@ -735,32 +658,4 @@ mod tests { let miss: Vec = "$Nope".encode_utf16().collect(); assert_eq!(scan_index_entries(&buf, 0, &miss), None); } - - #[test] - #[expect( - clippy::indexing_slicing, - reason = "test builds a fixed 512-byte boot sector with known in-bounds offsets" - )] - fn parse_boot_decodes_geometry() { - let mut boot = vec![0_u8; 512]; - boot[3..7].copy_from_slice(b"NTFS"); // oem_id - boot[11..13].copy_from_slice(&512_u16.to_le_bytes()); // bytes_per_sector - boot[13] = 8; // sectors_per_cluster - boot[40..48].copy_from_slice(&1_000_000_i64.to_le_bytes()); // total_sectors - boot[48..56].copy_from_slice(&786_432_i64.to_le_bytes()); // mft_start_lcn - boot[64] = (-10_i8).cast_unsigned(); // clusters_per_file_record → 2^10 = 1024 - boot[72..80].copy_from_slice(&0x1122_3344_5566_7788_i64.to_le_bytes()); // serial - - let geo = parse_boot(&boot).expect("valid boot sector"); - assert_eq!(geo.bytes_per_sector, 512); - assert_eq!(geo.sectors_per_cluster, 8); - assert_eq!(geo.bytes_per_cluster, 4096); - assert_eq!(geo.mft_record_size, 1024); - assert_eq!(geo.total_sectors, 1_000_000); - assert_eq!(geo.mft_start_lcn, 786_432); - assert_eq!(geo.volume_serial, 0x1122_3344_5566_7788); - - // Not a boot sector → error. - parse_boot(&[0_u8; 512]).unwrap_err(); - } } diff --git a/crates/uffs-mft/src/platform/metafile_decode.rs b/crates/uffs-mft/src/platform/metafile_decode.rs new file mode 100644 index 000000000..8a1acdc64 --- /dev/null +++ b/crates/uffs-mft/src/platform/metafile_decode.rs @@ -0,0 +1,179 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) 2025-2026 SKY, LLC. + +//! Offline decoders for captured NTFS metafiles. +//! +//! The reconstitute / validate side of the capture flow — pure, cross-platform +//! byte parsing (no live volume or Windows I/O), so it runs on the transfer +//! target (macOS/Linux) too. + +use super::metafile::{MetafileHeader, MetafileKind}; +use crate::error::{MftError, Result}; + +/// Volume geometry decoded from a captured `$Boot` payload. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct BootGeometry { + /// Bytes per sector. + pub bytes_per_sector: u16, + /// Sectors per cluster. + pub sectors_per_cluster: u8, + /// Cluster size in bytes. + pub bytes_per_cluster: u32, + /// MFT file-record size in bytes. + pub mft_record_size: u32, + /// Total sectors on the volume. + pub total_sectors: u64, + /// Logical cluster number of `$MFT`. + pub mft_start_lcn: u64, + /// Volume serial number. + pub volume_serial: u64, +} + +/// Decode the NTFS volume geometry from a captured `$Boot` payload. +/// +/// # Errors +/// +/// Returns [`MftError::InvalidData`] if the payload is too small or is not a +/// valid NTFS boot sector. +pub fn parse_boot(payload: &[u8]) -> Result { + use zerocopy::FromBytes as _; + + let (boot, _) = crate::ntfs::NtfsBootSector::read_from_prefix(payload) + .map_err(|_err| MftError::InvalidData("$Boot payload too small".to_owned()))?; + if !boot.is_valid() { + return Err(MftError::InvalidData( + "payload is not a valid NTFS boot sector".to_owned(), + )); + } + Ok(BootGeometry { + bytes_per_sector: boot.bytes_per_sector, + sectors_per_cluster: boot.sectors_per_cluster, + bytes_per_cluster: boot.cluster_size(), + mft_record_size: boot.file_record_size(), + total_sectors: boot.total_sectors.cast_unsigned(), + mft_start_lcn: boot.mft_start_lcn.cast_unsigned(), + volume_serial: boot.volume_serial_number.cast_unsigned(), + }) +} + +/// Cluster-allocation stats decoded from a captured `$Bitmap` payload. +#[expect( + clippy::struct_field_names, + reason = "the `_clusters` suffix documents the unit in this public stats struct" +)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct BitmapStats { + /// Total clusters covered by the bitmap (one bit each). + pub total_clusters: u64, + /// Allocated (in-use) clusters — the set bits. + pub used_clusters: u64, + /// Free clusters — the clear bits. + pub free_clusters: u64, +} + +/// Decode cluster-allocation stats from a captured `$Bitmap` payload. +/// +/// Each bit maps one cluster (1 = allocated). Trailing padding bits in the last +/// byte read as free, matching how NTFS rounds the bitmap up to a byte. +#[must_use] +pub fn parse_bitmap(payload: &[u8]) -> BitmapStats { + let used: u64 = payload + .iter() + .map(|byte| u64::from(byte.count_ones())) + .sum(); + let total = u64::try_from(payload.len()).unwrap_or(0).saturating_mul(8); + BitmapStats { + total_clusters: total, + used_clusters: used, + free_clusters: total.saturating_sub(used), + } +} + +/// A human-readable summary of a captured metafile (its header, plus +/// kind-specific detail such as `$Boot` geometry). +#[must_use] +pub fn summarize(header: &MetafileHeader, payload: &[u8]) -> String { + let base = format!( + "Metafile: {}\n Drive: {}:\n Serial: 0x{:016X}\n Captured (epoch s): {}\n Payload: {} bytes\n", + header.kind.name(), + header.drive, + header.volume_serial, + header.timestamp, + payload.len(), + ); + let detail = match header.kind { + MetafileKind::Boot => match parse_boot(payload) { + Ok(geo) => format!( + " $Boot: {} B/sector x {} sec/clu = {} B/cluster; MFT rec {} B; MFT LCN {}; total sectors {}\n", + geo.bytes_per_sector, + geo.sectors_per_cluster, + geo.bytes_per_cluster, + geo.mft_record_size, + geo.mft_start_lcn, + geo.total_sectors, + ), + Err(err) => format!(" $Boot parse failed: {err}\n"), + }, + MetafileKind::Bitmap => { + let stats = parse_bitmap(payload); + format!( + " $Bitmap: {} clusters total, {} used, {} free\n", + stats.total_clusters, stats.used_clusters, stats.free_clusters, + ) + } + MetafileKind::Secure + | MetafileKind::AttrDef + | MetafileKind::MftMirr + | MetafileKind::Volume + | MetafileKind::BadClus + | MetafileKind::LogFile + | MetafileKind::UsnJrnl => String::new(), + }; + format!("{base}{detail}") +} + +#[cfg(test)] +mod tests { + use super::{parse_bitmap, parse_boot}; + + #[test] + #[expect( + clippy::indexing_slicing, + reason = "test builds a fixed 512-byte boot sector with known in-bounds offsets" + )] + fn parse_boot_decodes_geometry() { + let mut boot = vec![0_u8; 512]; + boot[3..7].copy_from_slice(b"NTFS"); // oem_id + boot[11..13].copy_from_slice(&512_u16.to_le_bytes()); // bytes_per_sector + boot[13] = 8; // sectors_per_cluster + boot[40..48].copy_from_slice(&1_000_000_i64.to_le_bytes()); // total_sectors + boot[48..56].copy_from_slice(&786_432_i64.to_le_bytes()); // mft_start_lcn + boot[64] = (-10_i8).cast_unsigned(); // clusters_per_file_record → 2^10 = 1024 + boot[72..80].copy_from_slice(&0x1122_3344_5566_7788_i64.to_le_bytes()); // serial + + let geo = parse_boot(&boot).expect("valid boot sector"); + assert_eq!(geo.bytes_per_sector, 512); + assert_eq!(geo.sectors_per_cluster, 8); + assert_eq!(geo.bytes_per_cluster, 4096); + assert_eq!(geo.mft_record_size, 1024); + assert_eq!(geo.total_sectors, 1_000_000); + assert_eq!(geo.mft_start_lcn, 786_432); + assert_eq!(geo.volume_serial, 0x1122_3344_5566_7788); + + // Not a boot sector → error. + parse_boot(&[0_u8; 512]).unwrap_err(); + } + + #[test] + fn parse_bitmap_counts_clusters() { + // 0xFF = 8 set, 0x00 = 0 set, 0x0F = 4 set → 12 used / 12 free of 24. + let stats = parse_bitmap(&[0xFF, 0x00, 0x0F]); + assert_eq!(stats.total_clusters, 24); + assert_eq!(stats.used_clusters, 12); + assert_eq!(stats.free_clusters, 12); + + let empty = parse_bitmap(&[]); + assert_eq!(empty.total_clusters, 0); + assert_eq!(empty.free_clusters, 0); + } +} From cf34bf9b4af2013ac7eba3333e658dd4d3091b5f Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Sun, 5 Jul 2026 16:03:28 -0700 Subject: [PATCH 14/39] feat(uffs-mft): offline $UsnJrnl:$J change-journal decode (P6) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit parse_usn decodes a captured $UsnJrnl:$J payload — skips the leading sparse region, walks USN_RECORD_V2/V3 records (8-byte-aligned), and returns a record count, USN range, and a small sample (usn / reason / filename). Folded into summarize / `metafile-info` so the offline machine can inspect the change flow (the temporal dimension) without the live volume. Cross-platform + unit-tested (synthetic USN_RECORD_V2); verified end-to-end on macOS. host pedantic + cargo-xwin Windows clippy clean. Co-Authored-By: Claude Opus 4.8 --- .../uffs-mft/src/platform/metafile_decode.rs | 174 +++++++++++++++++- 1 file changed, 171 insertions(+), 3 deletions(-) diff --git a/crates/uffs-mft/src/platform/metafile_decode.rs b/crates/uffs-mft/src/platform/metafile_decode.rs index 8a1acdc64..8868888d6 100644 --- a/crates/uffs-mft/src/platform/metafile_decode.rs +++ b/crates/uffs-mft/src/platform/metafile_decode.rs @@ -89,6 +89,117 @@ pub fn parse_bitmap(payload: &[u8]) -> BitmapStats { } } +/// Read a little-endian `u16` at `off`, or `None` if out of bounds. +fn rd_u16(buf: &[u8], off: usize) -> Option { + buf.get(off..off + 2) + .and_then(|slice| slice.try_into().ok()) + .map(u16::from_le_bytes) +} + +/// Read a little-endian `u32` at `off`, or `None` if out of bounds. +fn rd_u32(buf: &[u8], off: usize) -> Option { + buf.get(off..off + 4) + .and_then(|slice| slice.try_into().ok()) + .map(u32::from_le_bytes) +} + +/// Read a little-endian `i64` at `off`, or `None` if out of bounds. +fn rd_i64(buf: &[u8], off: usize) -> Option { + buf.get(off..off + 8) + .and_then(|slice| slice.try_into().ok()) + .map(i64::from_le_bytes) +} + +/// Decode UTF-16LE bytes to a lossy UTF-8 string. +fn decode_utf16_lossy(bytes: &[u8]) -> String { + let units: Vec = bytes + .chunks_exact(2) + .filter_map(|pair| pair.try_into().ok().map(u16::from_le_bytes)) + .collect(); + String::from_utf16_lossy(&units) +} + +/// One decoded USN change-journal record (the surfaced fields). +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct UsnEntry { + /// USN of this record. + pub usn: i64, + /// Change-reason bitmask (`USN_REASON_*`). + pub reason: u32, + /// Affected file/dir name. + pub name: String, +} + +/// Summary of a captured `$UsnJrnl:$J` change-journal payload. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct UsnSummary { + /// Number of USN records parsed. + pub record_count: u64, + /// First (oldest) USN seen, or 0 when empty. + pub first_usn: i64, + /// Last (newest) USN seen, or 0 when empty. + pub last_usn: i64, + /// Up to [`USN_SAMPLE_MAX`] leading records, for a quick look. + pub sample: Vec, +} + +/// Maximum sample records surfaced from a `$UsnJrnl:$J` payload. +pub const USN_SAMPLE_MAX: usize = 8; + +/// Decode a `$UsnJrnl:$J` payload into a record count + a small sample. +/// +/// Skips the leading sparse region, then walks `USN_RECORD_V2`/`V3` records +/// (8-byte-aligned; a zero / sub-header `RecordLength` marks a gap). +#[must_use] +pub fn parse_usn(payload: &[u8]) -> UsnSummary { + // Skip the leading sparse (all-zero) region to the first record, aligned + // down to an 8-byte boundary. + let mut pos = payload + .iter() + .position(|&byte| byte != 0) + .map_or(payload.len(), |idx| idx & !0b111); + let mut summary = UsnSummary { + record_count: 0, + first_usn: 0, + last_usn: 0, + sample: Vec::new(), + }; + + while pos + 0x3C <= payload.len() { + let Some(record) = payload.get(pos..) else { + break; + }; + let rec_len = rd_u32(record, 0).unwrap_or(0) as usize; + let major = rd_u16(record, 4).unwrap_or(0); + if rec_len < 0x3C || (major != 2 && major != 3) { + pos += 8; // sparse gap / alignment padding + continue; + } + if pos + rec_len > payload.len() { + break; + } + let usn = rd_i64(record, 0x18).unwrap_or(0); + let reason = rd_u32(record, 0x28).unwrap_or(0); + let name_len = usize::from(rd_u16(record, 0x38).unwrap_or(0)); + let name_off = usize::from(rd_u16(record, 0x3A).unwrap_or(0)); + + if summary.record_count == 0 { + summary.first_usn = usn; + } + summary.last_usn = usn; + if summary.sample.len() < USN_SAMPLE_MAX { + let name = record + .get(name_off..name_off + name_len) + .map(decode_utf16_lossy) + .unwrap_or_default(); + summary.sample.push(UsnEntry { usn, reason, name }); + } + summary.record_count += 1; + pos += (rec_len + 7) & !0b111; // advance to the next 8-aligned record + } + summary +} + /// A human-readable summary of a captured metafile (its header, plus /// kind-specific detail such as `$Boot` geometry). #[must_use] @@ -121,20 +232,41 @@ pub fn summarize(header: &MetafileHeader, payload: &[u8]) -> String { stats.total_clusters, stats.used_clusters, stats.free_clusters, ) } + MetafileKind::UsnJrnl => { + let usn = parse_usn(payload); + let lines: Vec = usn + .sample + .iter() + .map(|entry| { + format!( + " usn {} reason 0x{:08X} {}", + entry.usn, entry.reason, entry.name + ) + }) + .collect(); + let body = if lines.is_empty() { + String::new() + } else { + format!("{}\n", lines.join("\n")) + }; + format!( + " $UsnJrnl: {} records; USN {}..{}\n{body}", + usn.record_count, usn.first_usn, usn.last_usn, + ) + } MetafileKind::Secure | MetafileKind::AttrDef | MetafileKind::MftMirr | MetafileKind::Volume | MetafileKind::BadClus - | MetafileKind::LogFile - | MetafileKind::UsnJrnl => String::new(), + | MetafileKind::LogFile => String::new(), }; format!("{base}{detail}") } #[cfg(test)] mod tests { - use super::{parse_bitmap, parse_boot}; + use super::{parse_bitmap, parse_boot, parse_usn}; #[test] #[expect( @@ -176,4 +308,40 @@ mod tests { assert_eq!(empty.total_clusters, 0); assert_eq!(empty.free_clusters, 0); } + + #[test] + #[expect( + clippy::indexing_slicing, + reason = "test builds a fixed USN_RECORD_V2 buffer with known in-bounds offsets" + )] + fn parse_usn_reads_records() { + let name: Vec = "a.txt".encode_utf16().collect(); // 5 units → 10 bytes + let name_bytes = u16::try_from(name.len() * 2).unwrap_or(0); + let rec_len = 0x3C + name.len() * 2; // header + name = 70 + let padded = (rec_len + 7) & !7; // 72 + let base = 16_usize; // 16 leading sparse zero bytes + let mut buf = vec![0_u8; base + padded]; + + buf[base..base + 4].copy_from_slice(&u32::try_from(rec_len).unwrap_or(0).to_le_bytes()); + buf[base + 4..base + 6].copy_from_slice(&2_u16.to_le_bytes()); // major version + buf[base + 0x18..base + 0x20].copy_from_slice(&1000_i64.to_le_bytes()); // usn + buf[base + 0x28..base + 0x2C].copy_from_slice(&1_u32.to_le_bytes()); // reason + buf[base + 0x38..base + 0x3A].copy_from_slice(&name_bytes.to_le_bytes()); // name_len + buf[base + 0x3A..base + 0x3C].copy_from_slice(&0x3C_u16.to_le_bytes()); // name_off + for (i, unit) in name.iter().enumerate() { + let off = base + 0x3C + i * 2; + buf[off..off + 2].copy_from_slice(&unit.to_le_bytes()); + } + + let summary = parse_usn(&buf); + assert_eq!(summary.record_count, 1); + assert_eq!(summary.first_usn, 1000); + assert_eq!(summary.last_usn, 1000); + assert_eq!(summary.sample.len(), 1); + assert_eq!(summary.sample[0].name, "a.txt"); + assert_eq!(summary.sample[0].reason, 1); + + // Empty / all-sparse payload → no records. + assert_eq!(parse_usn(&[0_u8; 64]).record_count, 0); + } } From 910a4b4f05225e7f2e4320d12728581c4e01c8d8 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Sun, 5 Jul 2026 16:04:35 -0700 Subject: [PATCH 15/39] docs(uffs-mft): mark P6 offline decoders shipped Update the capture plan: metafile-info + parse_boot/parse_bitmap/parse_usn offline decoders (cross-platform, Mac-verified) are in. Remaining: $Secure/ $Volume/$AttrDef parsers, verify_parity wiring, and the Windows-only P3 (VSS --volume-path) / P5 (rust-script VSS+zip wrapper). Co-Authored-By: Claude Opus 4.8 --- docs/architecture/mft-full-capture.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/docs/architecture/mft-full-capture.md b/docs/architecture/mft-full-capture.md index 51b3e4a72..9071b886a 100644 --- a/docs/architecture/mft-full-capture.md +++ b/docs/architecture/mft-full-capture.md @@ -3,10 +3,12 @@ # NTFS Full-Volume Capture (`uffs-mft capture`) -> **Status:** In progress — P0–P2 + P4 shipped (`sysinfo`, all metafile `save` -> targets incl. `$UsnJrnl` via `$Extend` traversal, and the `capture` -> orchestrator/manifest incl. the compressed `$MFT`). Remaining: P3 (VSS -> `--volume-path`) and P5 (`scripts/capture.rs` VSS/zip wrapper). +> **Status:** In progress — P0–P2, P4, and P6 (offline) shipped: `sysinfo`; all +> metafile `save` targets incl. `$UsnJrnl` via `$Extend` traversal; the +> `capture` orchestrator/manifest incl. the compressed `$MFT`; and offline +> `metafile-info` with `$Boot`/`$Bitmap`/`$UsnJrnl` decoders (cross-platform, +> Mac-verified). Remaining: P3 (VSS `--volume-path`), P5 (`scripts/capture.rs` +> VSS/zip wrapper), and more P6 decoders (`$Secure`/`$Volume`/`$AttrDef`). > **Owner:** UFFS core > **Goal:** Capture *everything* needed to reconstitute a live Windows NTFS volume > offline "as accurately as possible" — not just the namespace, but ACLs, the @@ -246,8 +248,11 @@ loaders are additive (namespace path unchanged). VSS snapshot device directly) — not yet. 5. ⏳ **P5 — `scripts/capture.rs`: WMI VSS create/delete + `--zip`/split + hashes** — not yet. -6. ⏳ **P6 — full `load` + `verify_parity` wiring** for the new artifacts — partial - (metafile header round-trips; per-metafile parsers TBD). +6. ◑ **P6 — offline read side** — shipped: `uffs-mft metafile-info ` + + `load_metafile_from_file`, and cross-platform decoders `parse_boot` + (geometry), `parse_bitmap` (free space), `parse_usn` (change journal), all + unit-tested + Mac-verified. Remaining: `$Secure`/`$Volume`/`$AttrDef` parsers + and `verify_parity` wiring. ## 12. Open questions From 5c724f2676c2f99ca13b360e53dc18f13712c237 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Sun, 5 Jul 2026 16:50:07 -0700 Subject: [PATCH 16/39] docs(uffs-mft): fix broken intra-doc links in metafile module doc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `[super::volume::read_handle_at]` (volume is #[cfg(windows)], absent in the macOS doc build) and `[MetafileHeader]` failed `cargo doc -D warnings` at pre-push. Both were illustrative — demote to plain code spans. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/platform/metafile.rs | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/crates/uffs-mft/src/platform/metafile.rs b/crates/uffs-mft/src/platform/metafile.rs index 6ad864c0e..57e66637f 100644 --- a/crates/uffs-mft/src/platform/metafile.rs +++ b/crates/uffs-mft/src/platform/metafile.rs @@ -7,12 +7,11 @@ //! These artifacts extend a capture beyond the `$MFT` namespace toward a //! complete offline representation of the volume (see //! `docs/architecture/mft-full-capture.md`). Each is read straight off the -//! volume via the same broker-safe primitive as `$UpCase` -//! ([`super::volume::read_handle_at`]). +//! volume via the same broker-safe `read_handle_at` primitive as `$UpCase`. //! //! # File format //! -//! 1. [`MetafileHeader`] (64 bytes) — magic, version, kind, drive, serial, +//! 1. `MetafileHeader` (64 bytes) — magic, version, kind, drive, serial, //! timestamp, payload size. //! 2. Raw metafile payload. //! From 657a17f75ecb412a11c3925c8f6b6310560e7a5f Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Sun, 5 Jul 2026 21:58:21 -0700 Subject: [PATCH 17/39] fix(uffs-mft): sector-align metafile data-run reads ($Bitmap capture) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Raw volume ReadFile requires sector-aligned length, but read_runs clamped the final read to the attribute's exact data_size — which for $Bitmap on a 64 KB- cluster volume is not sector-aligned (bitmap size = ceil(clusters/8)), so the read failed with ERROR_INVALID_PARAMETER (os error 87). Now allocate the whole cluster-aligned span the runs cover, read each run in full (aligned), and truncate to data_size afterwards. $LogFile/$AttrDef worked before only because their sizes happened to be aligned. Found by the first real Windows e2e run (C: 64 KB clusters). Needs a re-run to confirm $Bitmap now captures. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/platform/metafile.rs | 43 ++++++++++++------------ 1 file changed, 22 insertions(+), 21 deletions(-) diff --git a/crates/uffs-mft/src/platform/metafile.rs b/crates/uffs-mft/src/platform/metafile.rs index 57e66637f..7aacc0ecf 100644 --- a/crates/uffs-mft/src/platform/metafile.rs +++ b/crates/uffs-mft/src/platform/metafile.rs @@ -327,8 +327,11 @@ fn read_data_stream( /// Assemble a stream's data runs into a `data_size`-byte buffer. /// -/// Sparse runs leave their window zeroed; the buffer is truncated to the -/// attribute's real `data_size` (runs are cluster-rounded). +/// Raw volume reads must be sector-aligned in length, so we allocate the whole +/// cluster-aligned span the runs cover (≥ `data_size`), read each run in full, +/// then truncate to the attribute's real `data_size`. Sparse runs leave their +/// window zeroed. (A naïve last-read clamped to a non-cluster-aligned +/// `data_size` fails raw I/O with `ERROR_INVALID_PARAMETER`.) #[cfg(windows)] fn read_runs( handle: windows::Win32::Foundation::HANDLE, @@ -337,34 +340,32 @@ fn read_runs( data_size: u64, ) -> Result> { let bpc = u64::from(bytes_per_cluster); - let total = usize::try_from(data_size).map_err(|_err| { - MftError::InvalidData("metafile data_size exceeds usize::MAX".to_owned()) + let allocated: u64 = runs.iter().map(|run| run.cluster_count * bpc).sum(); + let capacity = usize::try_from(allocated.max(data_size)).map_err(|_err| { + MftError::InvalidData("metafile stream size exceeds usize::MAX".to_owned()) })?; - let mut buf = vec![0_u8; total]; + let mut buf = vec![0_u8; capacity]; let mut offset: usize = 0; for run in runs { - if offset >= total { - break; - } let run_bytes = usize::try_from(run.cluster_count * bpc).map_err(|_err| { MftError::InvalidData("metafile run byte count exceeds usize::MAX".to_owned()) })?; - let read_len = run_bytes.min(total - offset); - if run.is_sparse() { - // Sparse run — leave the buffer window zeroed. - offset += read_len; - continue; + if !run.is_sparse() { + // Cluster-aligned offset + length → sector-aligned raw read. + let disk_offset = crate::index::nonneg_to_u64(run.lcn.raw() * bpc.cast_signed()); + let Some(window) = buf.get_mut(offset..offset + run_bytes) else { + return Err(MftError::InvalidData(format!( + "metafile run at offset {offset} len {run_bytes} exceeds buffer {capacity}" + ))); + }; + super::volume::read_handle_at(handle, disk_offset, window)?; } - let disk_offset = crate::index::nonneg_to_u64(run.lcn.raw() * bpc.cast_signed()); - let Some(window) = buf.get_mut(offset..offset + read_len) else { - return Err(MftError::InvalidData(format!( - "metafile run at offset {offset} len {read_len} exceeds buffer {total}" - ))); - }; - super::volume::read_handle_at(handle, disk_offset, window)?; - offset += read_len; + offset = offset.saturating_add(run_bytes); } + + let final_len = usize::try_from(data_size).unwrap_or(capacity).min(capacity); + buf.truncate(final_len); Ok(buf) } From c97226a8b5899932a439e6f526c3df8de0ef304f Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Sun, 5 Jul 2026 22:06:04 -0700 Subject: [PATCH 18/39] fix(uffs-mft): follow $ATTRIBUTE_LIST to capture named $DATA streams $Secure:$SDS and $UsnJrnl:$J live in extension records referenced by $ATTRIBUTE_LIST when the base MFT record overflows, so the base-only reader missed them. Add a pure, testable attribute_list_data_frs() parser and fall back to reading the extension record(s) for the named non-resident $DATA stream, merging runs in VCN order. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/platform/metafile.rs | 119 ++++++++++++++---- .../uffs-mft/src/platform/metafile_decode.rs | 75 ++++++++++- 2 files changed, 168 insertions(+), 26 deletions(-) diff --git a/crates/uffs-mft/src/platform/metafile.rs b/crates/uffs-mft/src/platform/metafile.rs index 7aacc0ecf..5dc18b52d 100644 --- a/crates/uffs-mft/src/platform/metafile.rs +++ b/crates/uffs-mft/src/platform/metafile.rs @@ -284,6 +284,86 @@ fn read_frs_record( Ok(record) } +/// Locate a non-resident `$DATA` attribute in one MFT record — the unnamed +/// stream when `want_name` is `None`, or the named stream otherwise — returning +/// its data runs and logical size. +#[cfg(windows)] +fn find_data_attr( + record: &[u8], + want_name: Option<&[u16]>, +) -> Option<(Vec, u64)> { + use crate::ntfs::{AttributeIterator, AttributeType}; + + let mut attrs = AttributeIterator::new(record)?; + let attr = attrs.find(|attr| { + attr.attribute_type() == Some(AttributeType::Data) + && attr.is_non_resident() + && want_name.map_or(attr.header.name_length == 0, |want| { + attr.name() == Some(want) + }) + })?; + let non_resident = attr.non_resident_data()?; + Some((attr.data_runs(), non_resident.data_size.cast_unsigned())) +} + +/// Follow `$ATTRIBUTE_LIST` from a base record to the extension record(s) that +/// hold the named `$DATA` stream, returning its merged runs + logical size. +/// +/// NTFS relocates attributes to extension records when a base record overflows, +/// which is how `$Secure:$SDS` and `$UsnJrnl:$J` are stored. Runs are +/// concatenated in `$ATTRIBUTE_LIST` (VCN) order; the real `data_size` lives in +/// the first (VCN-0) instance and later instances report `0`. +#[cfg(windows)] +fn find_data_in_extensions( + handle: &crate::platform::VolumeHandle, + vol: &crate::platform::NtfsVolumeData, + base_record: &[u8], + stream_name: &str, +) -> Result, u64)>> { + use crate::ntfs::{AttributeIterator, AttributeType}; + + // Materialize the $ATTRIBUTE_LIST payload (resident or non-resident). + let list = { + let mut attrs = AttributeIterator::new(base_record) + .ok_or_else(|| MftError::InvalidData("record: invalid header".to_owned()))?; + let Some(list_attr) = + attrs.find(|attr| attr.attribute_type() == Some(AttributeType::AttributeList)) + else { + return Ok(None); + }; + if let Some(resident) = list_attr.resident_value() { + resident.to_vec() + } else if let Some(non_resident) = list_attr.non_resident_data() { + read_runs( + handle.raw_handle(), + &list_attr.data_runs(), + vol.bytes_per_cluster, + non_resident.data_size.cast_unsigned(), + )? + } else { + return Ok(None); + } + }; + + let want: Vec = stream_name.encode_utf16().collect(); + let mut runs: Vec = Vec::new(); + let mut data_size = 0_u64; + for ext_frs in crate::platform::metafile_decode::attribute_list_data_frs(&list, stream_name) { + let ext = read_frs_record(handle, vol, ext_frs)?; + if let Some((ext_runs, ext_size)) = find_data_attr(&ext, Some(&want)) { + if data_size == 0 { + data_size = ext_size; // VCN-0 instance carries the real size + } + runs.extend(ext_runs); + } + } + if runs.is_empty() { + Ok(None) + } else { + Ok(Some((runs, data_size))) + } +} + /// Read a metafile's non-resident `$DATA` stream — the unnamed stream when /// `stream_name` is `None`, or the named stream (e.g. `$SDS`) otherwise — by /// resolving its data runs and reading the referenced clusters. @@ -294,35 +374,24 @@ fn read_data_stream( frs: u64, stream_name: Option<&str>, ) -> Result> { - use crate::ntfs::{AttributeIterator, AttributeType}; - let record = read_frs_record(handle, vol, frs)?; let want_name: Option> = stream_name.map(|name| name.encode_utf16().collect()); - let mut attrs = AttributeIterator::new(&record) - .ok_or_else(|| MftError::InvalidData(format!("FRS {frs}: invalid record header")))?; - let data_attr = attrs - .find(|attr| { - attr.attribute_type() == Some(AttributeType::Data) - && attr.is_non_resident() - && want_name - .as_deref() - .map_or(attr.header.name_length == 0, |want| { - attr.name() == Some(want) - }) - }) - .ok_or_else(|| { - MftError::InvalidData(format!( - "FRS {frs}: no matching non-resident DATA stream (name={stream_name:?})" - )) - })?; + // The attribute usually lives in the base record. + if let Some((runs, size)) = find_data_attr(&record, want_name.as_deref()) { + return read_runs(handle.raw_handle(), &runs, vol.bytes_per_cluster, size); + } - let non_resident = data_attr.non_resident_data().ok_or_else(|| { - MftError::InvalidData(format!("FRS {frs}: cannot decode non-resident DATA header")) - })?; - let data_size = non_resident.data_size.cast_unsigned(); - let runs = data_attr.data_runs(); - read_runs(handle.raw_handle(), &runs, vol.bytes_per_cluster, data_size) + // Named streams ($SDS, $J) can overflow into an extension record. + if let Some(name) = stream_name + && let Some((runs, size)) = find_data_in_extensions(handle, vol, &record, name)? + { + return read_runs(handle.raw_handle(), &runs, vol.bytes_per_cluster, size); + } + + Err(MftError::InvalidData(format!( + "FRS {frs}: no matching non-resident DATA stream (name={stream_name:?})" + ))) } /// Assemble a stream's data runs into a `data_size`-byte buffer. diff --git a/crates/uffs-mft/src/platform/metafile_decode.rs b/crates/uffs-mft/src/platform/metafile_decode.rs index 8868888d6..9072a58ab 100644 --- a/crates/uffs-mft/src/platform/metafile_decode.rs +++ b/crates/uffs-mft/src/platform/metafile_decode.rs @@ -110,6 +110,54 @@ fn rd_i64(buf: &[u8], off: usize) -> Option { .map(i64::from_le_bytes) } +/// Read a little-endian `u64` at `off`, or `None` if out of bounds. +fn rd_u64(buf: &[u8], off: usize) -> Option { + buf.get(off..off + 8) + .and_then(|slice| slice.try_into().ok()) + .map(u64::from_le_bytes) +} + +/// Parse an `$ATTRIBUTE_LIST` payload and return the distinct MFT FRS numbers +/// of the records holding the `$DATA` attribute named `stream_name`. +/// +/// NTFS moves attributes into extension records (referenced by +/// `$ATTRIBUTE_LIST`) when a file's base record overflows — as happens for the +/// named `$DATA` streams `$Secure:$SDS` and `$UsnJrnl:$J`. Pure/testable. +#[must_use] +pub fn attribute_list_data_frs(list: &[u8], stream_name: &str) -> Vec { + /// NTFS `$DATA` attribute type code. + const DATA_TYPE: u32 = 0x80; + /// Minimum `$ATTRIBUTE_LIST` entry size (fixed header before the name). + const MIN_ENTRY: usize = 0x1A; + + let mut out: Vec = Vec::new(); + let mut pos = 0_usize; + while pos + MIN_ENTRY <= list.len() { + let entry_len = usize::from(rd_u16(list, pos + 4).unwrap_or(0)); + if entry_len < MIN_ENTRY { + break; + } + if rd_u32(list, pos).unwrap_or(0) == DATA_TYPE { + let name_len = usize::from(*list.get(pos + 6).unwrap_or(&0)); // UTF-16 units + let name_off = usize::from(*list.get(pos + 7).unwrap_or(&0)); + let name = list + .get(pos + name_off..pos + name_off + name_len * 2) + .map(decode_utf16_lossy) + .unwrap_or_default(); + if name == stream_name + && let Some(base_ref) = rd_u64(list, pos + 0x10) + { + let frs = crate::ntfs::file_reference_to_frs(base_ref); + if !out.contains(&frs) { + out.push(frs); + } + } + } + pos = pos.saturating_add(entry_len); + } + out +} + /// Decode UTF-16LE bytes to a lossy UTF-8 string. fn decode_utf16_lossy(bytes: &[u8]) -> String { let units: Vec = bytes @@ -266,7 +314,7 @@ pub fn summarize(header: &MetafileHeader, payload: &[u8]) -> String { #[cfg(test)] mod tests { - use super::{parse_bitmap, parse_boot, parse_usn}; + use super::{attribute_list_data_frs, parse_bitmap, parse_boot, parse_usn}; #[test] #[expect( @@ -344,4 +392,29 @@ mod tests { // Empty / all-sparse payload → no records. assert_eq!(parse_usn(&[0_u8; 64]).record_count, 0); } + + #[test] + #[expect( + clippy::indexing_slicing, + reason = "test builds a fixed $ATTRIBUTE_LIST entry with known in-bounds offsets" + )] + fn attribute_list_finds_data_extension_frs() { + // One entry: $DATA (0x80), name "$SDS", base ref → FRS 100. + let name: Vec = "$SDS".encode_utf16().collect(); // 4 units → 8 bytes + let name_off = 0x1A_usize; + let entry_len = (name_off + name.len() * 2 + 7) & !7; // 8-aligned + let mut list = vec![0_u8; entry_len]; + list[0..4].copy_from_slice(&0x80_u32.to_le_bytes()); // type = $DATA + list[4..6].copy_from_slice(&u16::try_from(entry_len).unwrap_or(0).to_le_bytes()); + list[6] = 4; // name length (units) + list[7] = u8::try_from(name_off).unwrap_or(0); // name offset + list[0x10..0x18].copy_from_slice(&100_u64.to_le_bytes()); // base file reference → FRS 100 + for (i, unit) in name.iter().enumerate() { + let off = name_off + i * 2; + list[off..off + 2].copy_from_slice(&unit.to_le_bytes()); + } + + assert_eq!(attribute_list_data_frs(&list, "$SDS"), vec![100]); + assert_eq!(attribute_list_data_frs(&list, "$J"), Vec::::new()); + } } From 4e058018f94cc886859b94a5da642435e9448a07 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Sun, 5 Jul 2026 22:09:19 -0700 Subject: [PATCH 19/39] fix(uffs-mft): populate NTFS major/minor version from extended volume data FSCTL_GET_NTFS_VOLUME_DATA only returns the NTFS version in the NTFS_EXTENDED_VOLUME_DATA block that trails NTFS_VOLUME_DATA_BUFFER, so ntfs_major_version/ntfs_minor_version were hardcoded to 0 and every capture reported "ntfs:0.0". Query with a combined base+extended buffer and read MajorVersion/MinorVersion when NTFS filled the extended block (graceful 0.0 fallback on systems that don't). Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/platform/volume.rs | 66 +++++++++++++++++++------- 1 file changed, 49 insertions(+), 17 deletions(-) diff --git a/crates/uffs-mft/src/platform/volume.rs b/crates/uffs-mft/src/platform/volume.rs index 5c8986cf2..85a60adca 100644 --- a/crates/uffs-mft/src/platform/volume.rs +++ b/crates/uffs-mft/src/platform/volume.rs @@ -16,7 +16,9 @@ use windows::Win32::Storage::FileSystem::{ FILE_READ_ATTRIBUTES, FILE_SHARE_DELETE, FILE_SHARE_READ, FILE_SHARE_WRITE, OPEN_EXISTING, SYNCHRONIZE, }; -use windows::Win32::System::Ioctl::{FSCTL_GET_NTFS_VOLUME_DATA, NTFS_VOLUME_DATA_BUFFER}; +use windows::Win32::System::Ioctl::{ + FSCTL_GET_NTFS_VOLUME_DATA, NTFS_EXTENDED_VOLUME_DATA, NTFS_VOLUME_DATA_BUFFER, +}; use windows::core::PCWSTR; use zerocopy::FromBytes as _; @@ -462,6 +464,22 @@ unsafe impl Send for VolumeHandle {} // threads. unsafe impl Sync for VolumeHandle {} +/// Combined output buffer for `FSCTL_GET_NTFS_VOLUME_DATA`. +/// +/// NTFS writes an [`NTFS_EXTENDED_VOLUME_DATA`] block (which carries the NTFS +/// major/minor version) immediately after the base +/// [`NTFS_VOLUME_DATA_BUFFER`] when the supplied buffer is large enough. The +/// two C structs are laid out contiguously here so a single ioctl fills both; +/// `#[repr(C)]` reproduces that on-disk adjacency. +#[repr(C)] +#[derive(Clone, Copy)] +struct NtfsVolumeDataCombined { + /// Base volume data (serial, cluster counts, `$MFT` geometry). + base: NTFS_VOLUME_DATA_BUFFER, + /// Extended data (NTFS version, physical sector size, TRIM limits). + extended: NTFS_EXTENDED_VOLUME_DATA, +} + /// NTFS volume data retrieved from `FSCTL_GET_NTFS_VOLUME_DATA`. #[derive(Debug, Clone, Copy)] pub struct NtfsVolumeData { @@ -640,25 +658,27 @@ impl VolumeHandle { fn get_ntfs_volume_data(handle: HANDLE, volume: super::DriveLetter) -> Result { use windows::Win32::System::IO::DeviceIoControl; - let mut buffer = NTFS_VOLUME_DATA_BUFFER::default(); + let mut combined = NtfsVolumeDataCombined { + base: NTFS_VOLUME_DATA_BUFFER::default(), + extended: NTFS_EXTENDED_VOLUME_DATA::default(), + }; let mut bytes_returned: u32 = 0; - // `size_of::()` is ~96 bytes — always fits u32. - let ntfs_volume_data_buffer_size = - u32::try_from(size_of::()).unwrap_or(u32::MAX); + // The combined buffer (base + extended) is ~140 bytes — always fits u32. + let combined_size = u32::try_from(size_of::()).unwrap_or(u32::MAX); - // SAFETY: `handle` is an open volume handle, `buffer` points to valid - // writable storage for `NTFS_VOLUME_DATA_BUFFER`, and - // `bytes_returned` is a valid out-parameter for the duration of the - // call. + // SAFETY: `handle` is an open volume handle, `combined` points to valid + // writable storage of `combined_size` bytes (base + extended blocks + // laid out as NTFS expects them), and `bytes_returned` is a valid + // out-parameter for the duration of the call. let result = unsafe { DeviceIoControl( handle, FSCTL_GET_NTFS_VOLUME_DATA, None, 0, - Some(core::ptr::from_mut(&mut buffer).cast()), - ntfs_volume_data_buffer_size, + Some(core::ptr::from_mut(&mut combined).cast()), + combined_size, Some(&raw mut bytes_returned), None, ) @@ -668,10 +688,22 @@ impl VolumeHandle { return Err(MftError::NotNtfs(volume)); } - // Note: NTFS major/minor version requires NTFS_EXTENDED_VOLUME_DATA - // (not available in NTFS_VOLUME_DATA_BUFFER). Default to 0; callers - // should use `query_ntfs_version()` if they need the actual version. - // + let buffer = combined.base; + + // The NTFS version lives in the extended block, which NTFS only fills + // when it wrote past the base buffer. When it didn't (older systems), + // the default-zeroed extended fields degrade gracefully to `0.0`. + let extended_filled = + usize::try_from(bytes_returned).unwrap_or(0) > size_of::(); + let (ntfs_major_version, ntfs_minor_version) = if extended_filled { + ( + combined.extended.MajorVersion, + combined.extended.MinorVersion, + ) + } else { + (0, 0) + }; + // Every `i64 -> u64` reinterpret below comes from an on-disk // NTFS count (sector / cluster / LCN / length) that the NTFS // on-disk format and Microsoft's `NTFS_VOLUME_DATA_BUFFER` MSDN @@ -681,8 +713,8 @@ impl VolumeHandle { // `cast_sign_loss` expect. let volume_data = NtfsVolumeData { volume_serial_number: buffer.VolumeSerialNumber.cast_unsigned(), - ntfs_major_version: 0, - ntfs_minor_version: 0, + ntfs_major_version, + ntfs_minor_version, number_of_sectors: buffer.NumberSectors.cast_unsigned(), total_clusters: buffer.TotalClusters.cast_unsigned(), free_clusters: buffer.FreeClusters.cast_unsigned(), From 8c2e76edfc4c44f485a8262abfa4ec93a0bdc798 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Mon, 6 Jul 2026 07:57:22 -0700 Subject: [PATCH 20/39] fix(uffs-mft): decode attribute names so named $DATA streams match MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AttributeRef::name() was a stub that decoded the UTF-16 name and then unconditionally returned None (it could not safely return a borrowed &[u16] from unaligned record bytes). Every named-stream match therefore failed, so $Secure:$SDS and $UsnJrnl:$J were never captured — the real root cause behind the earlier $ATTRIBUTE_LIST work, which relied on this broken primitive. Return an owned Vec instead and lock it with a decode test. Also make read_runs' buffer allocation fallible: a sparse stream like $UsnJrnl:$J can span a huge logical size, so surface a clean error rather than aborting on an infallible vec![0; capacity]. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/ntfs/records.rs | 25 ++++++++--------- crates/uffs-mft/src/ntfs/tests.rs | 35 ++++++++++++++++++++++++ crates/uffs-mft/src/platform/metafile.rs | 13 +++++++-- 3 files changed, 58 insertions(+), 15 deletions(-) diff --git a/crates/uffs-mft/src/ntfs/records.rs b/crates/uffs-mft/src/ntfs/records.rs index 60eb0cbfc..156eb8c0e 100644 --- a/crates/uffs-mft/src/ntfs/records.rs +++ b/crates/uffs-mft/src/ntfs/records.rs @@ -522,9 +522,13 @@ impl<'a> AttributeRef<'a> { extract_data_runs_from_attribute(self.data) } - /// Returns the attribute name, if present. + /// Returns the decoded attribute name (UTF-16 code units), if present. + /// + /// The name is stored as UTF-16LE bytes at an offset that carries no + /// alignment guarantee, so the units are decoded into an owned `Vec` + /// rather than reinterpreted in place. #[must_use] - pub fn name(&self) -> Option<&'a [u16]> { + pub fn name(&self) -> Option> { if self.header.name_length == 0 { return None; } @@ -533,18 +537,13 @@ impl<'a> AttributeRef<'a> { let name_length = self.header.name_length as usize; let name_byte_len = name_length * 2; let name_bytes = self.data.get(name_offset..name_offset + name_byte_len)?; - if name_bytes.len() % 2 != 0 { - return None; - } - - for chunk in name_bytes.chunks_exact(2) { - let Ok(arr): Result<[u8; 2], _> = chunk.try_into() else { - return None; - }; - let _char = u16::from_le_bytes(arr); - } - None + Some( + name_bytes + .chunks_exact(2) + .filter_map(|chunk| chunk.try_into().ok().map(u16::from_le_bytes)) + .collect(), + ) } } diff --git a/crates/uffs-mft/src/ntfs/tests.rs b/crates/uffs-mft/src/ntfs/tests.rs index f70fb265c..abca38d81 100644 --- a/crates/uffs-mft/src/ntfs/tests.rs +++ b/crates/uffs-mft/src/ntfs/tests.rs @@ -211,3 +211,38 @@ fn non_resident_attribute_helpers_decode_mapping_pairs() { lcn: crate::platform::Lcn::new(10), }]); } + +#[test] +fn attribute_name_decodes_utf16() { + let name_offset = 24_usize; + let name: Vec = "$SDS".encode_utf16().collect(); + let mut data = vec![0_u8; name_offset + name.len() * 2]; + for (i, unit) in name.iter().enumerate() { + let off = name_offset + i * 2; + data[off..off + 2].copy_from_slice(&unit.to_le_bytes()); + } + + let named = AttributeRef { + data: &data, + header: AttributeRecordHeader { + type_code: AttributeType::DATA_TYPE, + length: crate::len_to_u32(data.len()), + is_non_resident: 1, + name_length: 4, + name_offset: crate::len_to_u16(name_offset), + flags: 0, + instance: 0, + }, + }; + assert_eq!(named.name().as_deref(), Some(name.as_slice())); + + // An unnamed attribute yields no name. + let unnamed = AttributeRef { + data: &data, + header: AttributeRecordHeader { + name_length: 0, + ..named.header + }, + }; + assert_eq!(unnamed.name(), None); +} diff --git a/crates/uffs-mft/src/platform/metafile.rs b/crates/uffs-mft/src/platform/metafile.rs index 5dc18b52d..498ac9491 100644 --- a/crates/uffs-mft/src/platform/metafile.rs +++ b/crates/uffs-mft/src/platform/metafile.rs @@ -299,7 +299,7 @@ fn find_data_attr( attr.attribute_type() == Some(AttributeType::Data) && attr.is_non_resident() && want_name.map_or(attr.header.name_length == 0, |want| { - attr.name() == Some(want) + attr.name().as_deref() == Some(want) }) })?; let non_resident = attr.non_resident_data()?; @@ -413,7 +413,16 @@ fn read_runs( let capacity = usize::try_from(allocated.max(data_size)).map_err(|_err| { MftError::InvalidData("metafile stream size exceeds usize::MAX".to_owned()) })?; - let mut buf = vec![0_u8; capacity]; + // Fallible allocation: a sparse stream (e.g. `$UsnJrnl:$J`) can span a huge + // logical size, so reserve up front and surface a clean error instead of + // aborting the process on an infallible `vec![0; capacity]`. + let mut buf: Vec = Vec::new(); + buf.try_reserve_exact(capacity).map_err(|_err| { + MftError::InvalidData(format!( + "metafile stream too large to materialize: {capacity} bytes" + )) + })?; + buf.resize(capacity, 0); let mut offset: usize = 0; for run in runs { From c1d107fdc8618c2062b21368bf5406e617d8aa19 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Mon, 6 Jul 2026 08:33:35 -0700 Subject: [PATCH 21/39] feat(uffs-mft): capture $UsnJrnl:$J via sparse-compact run reader MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit $UsnJrnl:$J is a sparse file whose logical size can reach ~150 GB while the live journal is a tiny allocated tail (the purged prefix is a deallocated hole). The full-span reader hit the "too large to materialize" guard and skipped it. Add read_runs_sparse_compact, which reads only the non-sparse runs concatenated — each USN record is self-describing, so dropping the leading hole loses no data. Share run-resolution (base record + $ATTRIBUTE_LIST extensions) via resolve_data_runs and the fallible allocation via alloc_zeroed. Split the live-volume readers out of metafile.rs into a new platform/metafile_read.rs (metafile.rs kept the on-disk format + save/ load); the growing reader set pushed metafile.rs to the 800-line gate. read_metafile is re-exported so callers are unaffected. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/platform.rs | 2 + crates/uffs-mft/src/platform/metafile.rs | 408 +-------------- crates/uffs-mft/src/platform/metafile_read.rs | 486 ++++++++++++++++++ 3 files changed, 492 insertions(+), 404 deletions(-) create mode 100644 crates/uffs-mft/src/platform/metafile_read.rs diff --git a/crates/uffs-mft/src/platform.rs b/crates/uffs-mft/src/platform.rs index af76ab6bd..e35f6d437 100644 --- a/crates/uffs-mft/src/platform.rs +++ b/crates/uffs-mft/src/platform.rs @@ -42,6 +42,8 @@ pub mod metafile; /// Offline decoders for captured metafiles ($Boot geometry, $Bitmap free /// space). pub mod metafile_decode; +/// Live-volume NTFS metafile readers ($Boot, $DATA streams, $UsnJrnl). +pub mod metafile_read; /// Native Windows process introspection for the self-update detector. /// /// Image path + pid enumeration — keeps the `unsafe` FFI out of `uffs-cli`. diff --git a/crates/uffs-mft/src/platform/metafile.rs b/crates/uffs-mft/src/platform/metafile.rs index 498ac9491..415d4fc3e 100644 --- a/crates/uffs-mft/src/platform/metafile.rs +++ b/crates/uffs-mft/src/platform/metafile.rs @@ -35,10 +35,6 @@ const METAFILE_VERSION: u32 = 1; /// Fixed header size in bytes (payload starts at this offset). const HEADER_SIZE: usize = 64; -/// `$Boot` payload size: the boot region is 16 sectors × 512 bytes. -#[cfg(windows)] -const BOOT_BYTES: usize = 8192; - /// An NTFS metafile that can be captured. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum MetafileKind { @@ -219,376 +215,9 @@ impl MetafileHeader { } } -/// Read a metafile's raw bytes from a live NTFS volume. -/// -/// # Errors -/// -/// Returns [`MftError::Io`] / [`MftError::Windows`] if opening the volume or -/// reading fails. -#[cfg(windows)] -pub fn read_metafile(drive: DriveLetter, kind: MetafileKind) -> Result> { - let handle = crate::platform::VolumeHandle::open(drive)?; - let vol = handle.volume_data(); - match kind { - // `$Boot` is fixed at LCN 0; read it directly (no data-run parse). - MetafileKind::Boot => read_boot(&handle), - // `$Bitmap` is a non-resident unnamed `$DATA` stream. - MetafileKind::Bitmap => read_data_stream(&handle, vol, 6, None), - // `$Secure:$SDS` holds deduplicated security descriptors (ACLs). - MetafileKind::Secure => read_data_stream(&handle, vol, 9, Some("$SDS")), - // `$AttrDef` / `$MFTMirr` / `$LogFile` are unnamed non-resident `$DATA` - // streams. - MetafileKind::AttrDef => read_data_stream(&handle, vol, 4, None), - MetafileKind::MftMirr => read_data_stream(&handle, vol, 1, None), - MetafileKind::LogFile => read_data_stream(&handle, vol, 2, None), - // `$Volume` / `$BadClus` keep their useful data in the MFT record itself - // (the $VOLUME_* attributes; the $Bad run list), so capture the fixed-up - // record. - MetafileKind::Volume => read_frs_record(&handle, vol, 3), - MetafileKind::BadClus => read_frs_record(&handle, vol, 8), - // `$UsnJrnl:$J` lives under `$Extend`; resolve its FRS then read `$J`. - MetafileKind::UsnJrnl => read_usn_journal(&handle, vol), - } -} - -/// Read a metafile's raw bytes (non-Windows stub). -/// -/// # Errors -/// -/// Always returns [`MftError::PlatformNotSupported`]. -#[cfg(not(windows))] -pub const fn read_metafile(_drive: DriveLetter, _kind: MetafileKind) -> Result> { - Err(MftError::PlatformNotSupported) -} - -/// `$Boot` is the volume boot region: 8 KiB starting at LCN 0 (byte offset 0). -#[cfg(windows)] -fn read_boot(handle: &crate::platform::VolumeHandle) -> Result> { - let mut buf = vec![0_u8; BOOT_BYTES]; - super::volume::read_handle_at(handle.raw_handle(), 0, &mut buf)?; - Ok(buf) -} - -/// Read a raw MFT file-record segment (FRS) with USA fixup applied. -#[cfg(windows)] -fn read_frs_record( - handle: &crate::platform::VolumeHandle, - vol: &crate::platform::NtfsVolumeData, - frs: u64, -) -> Result> { - let record_size = vol.bytes_per_file_record_segment as usize; - let offset = handle.mft_byte_offset() + frs * u64::from(vol.bytes_per_file_record_segment); - let mut record = vec![0_u8; record_size]; - super::volume::read_handle_at(handle.raw_handle(), offset, &mut record)?; - crate::parse::apply_fixup(&mut record); - Ok(record) -} - -/// Locate a non-resident `$DATA` attribute in one MFT record — the unnamed -/// stream when `want_name` is `None`, or the named stream otherwise — returning -/// its data runs and logical size. -#[cfg(windows)] -fn find_data_attr( - record: &[u8], - want_name: Option<&[u16]>, -) -> Option<(Vec, u64)> { - use crate::ntfs::{AttributeIterator, AttributeType}; - - let mut attrs = AttributeIterator::new(record)?; - let attr = attrs.find(|attr| { - attr.attribute_type() == Some(AttributeType::Data) - && attr.is_non_resident() - && want_name.map_or(attr.header.name_length == 0, |want| { - attr.name().as_deref() == Some(want) - }) - })?; - let non_resident = attr.non_resident_data()?; - Some((attr.data_runs(), non_resident.data_size.cast_unsigned())) -} - -/// Follow `$ATTRIBUTE_LIST` from a base record to the extension record(s) that -/// hold the named `$DATA` stream, returning its merged runs + logical size. -/// -/// NTFS relocates attributes to extension records when a base record overflows, -/// which is how `$Secure:$SDS` and `$UsnJrnl:$J` are stored. Runs are -/// concatenated in `$ATTRIBUTE_LIST` (VCN) order; the real `data_size` lives in -/// the first (VCN-0) instance and later instances report `0`. -#[cfg(windows)] -fn find_data_in_extensions( - handle: &crate::platform::VolumeHandle, - vol: &crate::platform::NtfsVolumeData, - base_record: &[u8], - stream_name: &str, -) -> Result, u64)>> { - use crate::ntfs::{AttributeIterator, AttributeType}; - - // Materialize the $ATTRIBUTE_LIST payload (resident or non-resident). - let list = { - let mut attrs = AttributeIterator::new(base_record) - .ok_or_else(|| MftError::InvalidData("record: invalid header".to_owned()))?; - let Some(list_attr) = - attrs.find(|attr| attr.attribute_type() == Some(AttributeType::AttributeList)) - else { - return Ok(None); - }; - if let Some(resident) = list_attr.resident_value() { - resident.to_vec() - } else if let Some(non_resident) = list_attr.non_resident_data() { - read_runs( - handle.raw_handle(), - &list_attr.data_runs(), - vol.bytes_per_cluster, - non_resident.data_size.cast_unsigned(), - )? - } else { - return Ok(None); - } - }; - - let want: Vec = stream_name.encode_utf16().collect(); - let mut runs: Vec = Vec::new(); - let mut data_size = 0_u64; - for ext_frs in crate::platform::metafile_decode::attribute_list_data_frs(&list, stream_name) { - let ext = read_frs_record(handle, vol, ext_frs)?; - if let Some((ext_runs, ext_size)) = find_data_attr(&ext, Some(&want)) { - if data_size == 0 { - data_size = ext_size; // VCN-0 instance carries the real size - } - runs.extend(ext_runs); - } - } - if runs.is_empty() { - Ok(None) - } else { - Ok(Some((runs, data_size))) - } -} - -/// Read a metafile's non-resident `$DATA` stream — the unnamed stream when -/// `stream_name` is `None`, or the named stream (e.g. `$SDS`) otherwise — by -/// resolving its data runs and reading the referenced clusters. -#[cfg(windows)] -fn read_data_stream( - handle: &crate::platform::VolumeHandle, - vol: &crate::platform::NtfsVolumeData, - frs: u64, - stream_name: Option<&str>, -) -> Result> { - let record = read_frs_record(handle, vol, frs)?; - let want_name: Option> = stream_name.map(|name| name.encode_utf16().collect()); - - // The attribute usually lives in the base record. - if let Some((runs, size)) = find_data_attr(&record, want_name.as_deref()) { - return read_runs(handle.raw_handle(), &runs, vol.bytes_per_cluster, size); - } - - // Named streams ($SDS, $J) can overflow into an extension record. - if let Some(name) = stream_name - && let Some((runs, size)) = find_data_in_extensions(handle, vol, &record, name)? - { - return read_runs(handle.raw_handle(), &runs, vol.bytes_per_cluster, size); - } - - Err(MftError::InvalidData(format!( - "FRS {frs}: no matching non-resident DATA stream (name={stream_name:?})" - ))) -} - -/// Assemble a stream's data runs into a `data_size`-byte buffer. -/// -/// Raw volume reads must be sector-aligned in length, so we allocate the whole -/// cluster-aligned span the runs cover (≥ `data_size`), read each run in full, -/// then truncate to the attribute's real `data_size`. Sparse runs leave their -/// window zeroed. (A naïve last-read clamped to a non-cluster-aligned -/// `data_size` fails raw I/O with `ERROR_INVALID_PARAMETER`.) -#[cfg(windows)] -fn read_runs( - handle: windows::Win32::Foundation::HANDLE, - runs: &[crate::ntfs::DataRun], - bytes_per_cluster: u32, - data_size: u64, -) -> Result> { - let bpc = u64::from(bytes_per_cluster); - let allocated: u64 = runs.iter().map(|run| run.cluster_count * bpc).sum(); - let capacity = usize::try_from(allocated.max(data_size)).map_err(|_err| { - MftError::InvalidData("metafile stream size exceeds usize::MAX".to_owned()) - })?; - // Fallible allocation: a sparse stream (e.g. `$UsnJrnl:$J`) can span a huge - // logical size, so reserve up front and surface a clean error instead of - // aborting the process on an infallible `vec![0; capacity]`. - let mut buf: Vec = Vec::new(); - buf.try_reserve_exact(capacity).map_err(|_err| { - MftError::InvalidData(format!( - "metafile stream too large to materialize: {capacity} bytes" - )) - })?; - buf.resize(capacity, 0); - let mut offset: usize = 0; - - for run in runs { - let run_bytes = usize::try_from(run.cluster_count * bpc).map_err(|_err| { - MftError::InvalidData("metafile run byte count exceeds usize::MAX".to_owned()) - })?; - if !run.is_sparse() { - // Cluster-aligned offset + length → sector-aligned raw read. - let disk_offset = crate::index::nonneg_to_u64(run.lcn.raw() * bpc.cast_signed()); - let Some(window) = buf.get_mut(offset..offset + run_bytes) else { - return Err(MftError::InvalidData(format!( - "metafile run at offset {offset} len {run_bytes} exceeds buffer {capacity}" - ))); - }; - super::volume::read_handle_at(handle, disk_offset, window)?; - } - offset = offset.saturating_add(run_bytes); - } - - let final_len = usize::try_from(data_size).unwrap_or(capacity).min(capacity); - buf.truncate(final_len); - Ok(buf) -} - -/// FRS of the `$Extend` metadata directory. -#[cfg(windows)] -const EXTEND_FRS: u64 = 11; - -/// Read the `$UsnJrnl:$J` change journal via `$Extend` directory traversal. -#[cfg(windows)] -fn read_usn_journal( - handle: &crate::platform::VolumeHandle, - vol: &crate::platform::NtfsVolumeData, -) -> Result> { - let usn_frs = resolve_extend_child(handle, vol, "$UsnJrnl")?; - read_data_stream(handle, vol, usn_frs, Some("$J")) -} - -/// Resolve the FRS of a `$Extend` (FRS 11) child by name, walking its directory -/// index — `$INDEX_ROOT` first, then `$INDEX_ALLOCATION` if the index is large. -#[cfg(windows)] -fn resolve_extend_child( - handle: &crate::platform::VolumeHandle, - vol: &crate::platform::NtfsVolumeData, - child: &str, -) -> Result { - use crate::ntfs::{AttributeIterator, AttributeType, DataRun}; - - let record = read_frs_record(handle, vol, EXTEND_FRS)?; - let target: Vec = child.encode_utf16().collect(); - - let mut root_value: Option> = None; - let mut block_size: usize = 0; - let mut alloc: Option<(Vec, u64)> = None; - - let attrs = AttributeIterator::new(&record) - .ok_or_else(|| MftError::InvalidData("$Extend: invalid record header".to_owned()))?; - for attr in attrs { - match attr.attribute_type() { - Some(AttributeType::IndexRoot) => { - if let Some(value) = attr.resident_value() { - // IndexRoot.bytes_per_index_block is at offset 8. - block_size = value - .get(8..12) - .and_then(|slice| slice.try_into().ok()) - .map_or(0, |bytes| u32::from_le_bytes(bytes) as usize); - root_value = Some(value.to_vec()); - } - } - Some(AttributeType::IndexAllocation) if attr.is_non_resident() => { - if let Some(nr) = attr.non_resident_data() { - alloc = Some((attr.data_runs(), nr.data_size.cast_unsigned())); - } - } - _ => {} - } - } - - // 1. Search the resident $INDEX_ROOT (its INDEX_HEADER begins at offset 16). - if let Some(frs) = root_value - .as_deref() - .and_then(|root| scan_index_entries(root, 16, &target)) - { - return Ok(frs); - } - - // 2. Search the $INDEX_ALLOCATION INDX blocks, if the index spilled. - if let Some((runs, size)) = alloc - && block_size > 0 - { - let buf = read_runs(handle.raw_handle(), &runs, vol.bytes_per_cluster, size)?; - if let Some(frs) = scan_index_blocks(&buf, block_size, &target) { - return Ok(frs); - } - } - - Err(MftError::InvalidData(format!( - "$Extend index: {child} not found (no active USN journal?)" - ))) -} - -/// Scan NTFS directory-index entries for the entry whose `$FILE_NAME` matches -/// `target` (UTF-16), returning its FRS. `header_start` is the byte offset of -/// the `INDEX_HEADER` within `buf` (whose `first_entry_offset` is relative to -/// it). -#[cfg(any(windows, test))] -fn scan_index_entries(buf: &[u8], header_start: usize, target: &[u16]) -> Option { - let first_entry_offset = - u32::from_le_bytes(buf.get(header_start..header_start + 4)?.try_into().ok()?) as usize; - let mut pos = header_start.checked_add(first_entry_offset)?; - loop { - let entry = buf.get(pos..)?; - let flags = u16::from_le_bytes(entry.get(12..14)?.try_into().ok()?); - // Last-entry flag (0x02): end of this node. - if (flags & 0x02) != 0 { - return None; - } - let file_reference = u64::from_le_bytes(entry.get(0..8)?.try_into().ok()?); - // The $FILE_NAME key starts at entry offset 16: name_length @0x40, - // UTF-16 name @0x42. - let name_length = usize::from(*entry.get(16 + 0x40)?); - let name_bytes = entry.get(16 + 0x42..16 + 0x42 + name_length * 2)?; - let name: Vec = name_bytes - .chunks_exact(2) - .filter_map(|pair| pair.try_into().ok().map(u16::from_le_bytes)) - .collect(); - if name == target { - return Some(crate::ntfs::file_reference_to_frs(file_reference)); - } - let entry_length = usize::from(u16::from_le_bytes(entry.get(8..10)?.try_into().ok()?)); - if entry_length == 0 { - return None; - } - pos = pos.checked_add(entry_length)?; - } -} - -/// Scan the `INDX` blocks in an `$INDEX_ALLOCATION` buffer for `target`. -#[cfg(windows)] -fn scan_index_blocks(buf: &[u8], block_size: usize, target: &[u16]) -> Option { - if block_size == 0 { - return None; - } - for start in (0..buf.len()).step_by(block_size) { - let Some(end) = start.checked_add(block_size) else { - break; - }; - let Some(block) = buf.get(start..end) else { - break; - }; - if !block.starts_with(b"INDX") { - continue; // sparse / unused block - } - let mut owned = block.to_vec(); - let usa_offset = u16::from_le_bytes(owned.get(4..6)?.try_into().ok()?); - let usa_count = u16::from_le_bytes(owned.get(6..8)?.try_into().ok()?); - if !crate::ntfs::apply_usa_fixup(&mut owned, usa_offset, usa_count) { - continue; - } - // The INDEX_HEADER begins at offset 0x18 within an INDX block. - if let Some(frs) = scan_index_entries(&owned, 0x18, target) { - return Some(frs); - } - } - None -} +/// Live-volume metafile reader. Defined in [`crate::platform::metafile_read`] +/// and re-exported here so callers keep using `metafile::read_metafile`. +pub use super::metafile_read::read_metafile; /// Write a captured metafile (header + payload) to `path` atomically. /// @@ -630,7 +259,7 @@ pub fn load_metafile_from_file(path: &Path) -> Result<(MetafileHeader, Vec)> #[cfg(test)] mod tests { - use super::{HEADER_SIZE, MetafileHeader, MetafileKind, scan_index_entries}; + use super::{HEADER_SIZE, MetafileHeader, MetafileKind}; use crate::platform::DriveLetter; fn sample() -> MetafileHeader { @@ -707,33 +336,4 @@ mod tests { fn payload_offset_is_header_size() { assert_eq!(HEADER_SIZE, 64); } - - #[test] - #[expect( - clippy::indexing_slicing, - reason = "test builds a fixed 256-byte index buffer with known in-bounds offsets" - )] - fn scan_index_entries_resolves_child_frs() { - // Minimal $INDEX_ROOT-style buffer: INDEX_HEADER at offset 0 - // (first_entry_offset = 16), one entry for "$UsnJrnl" → FRS 42. - let mut buf = vec![0_u8; 256]; - buf[0..4].copy_from_slice(&16_u32.to_le_bytes()); // first_entry_offset - let entry = 16_usize; - buf[entry..entry + 8].copy_from_slice(&42_u64.to_le_bytes()); // file_reference - // flags @ entry+12 stay 0 (not the last entry). - let name: Vec = "$UsnJrnl".encode_utf16().collect(); - let key = entry + 16; - buf[key + 0x40] = 8; // name_length (UTF-16 code units) - for (i, unit) in name.iter().enumerate() { - let off = key + 0x42 + i * 2; - buf[off..off + 2].copy_from_slice(&unit.to_le_bytes()); - } - - let target: Vec = "$UsnJrnl".encode_utf16().collect(); - assert_eq!(scan_index_entries(&buf, 0, &target), Some(42)); - - // A miss: entry_length is 0 after the single entry, so the scan stops. - let miss: Vec = "$Nope".encode_utf16().collect(); - assert_eq!(scan_index_entries(&buf, 0, &miss), None); - } } diff --git a/crates/uffs-mft/src/platform/metafile_read.rs b/crates/uffs-mft/src/platform/metafile_read.rs new file mode 100644 index 000000000..5cc53a860 --- /dev/null +++ b/crates/uffs-mft/src/platform/metafile_read.rs @@ -0,0 +1,486 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) 2025-2026 SKY, LLC. + +//! Live-volume NTFS metafile readers (Windows). +//! +//! Reads the reserved `$`-files straight off a live volume via the broker-safe +//! `read_handle_at` primitive: `$Boot`, non-resident `$DATA` streams (including +//! the named `$SDS` / `$J` streams that overflow into `$ATTRIBUTE_LIST` +//! extension records), fixed-up MFT records, and `$Extend` directory traversal +//! for `$UsnJrnl`. The persisted on-disk format and the offline decoders live +//! in [`crate::platform::metafile`] and [`crate::platform::metafile_decode`]. + +use super::metafile::MetafileKind; +use crate::error::{MftError, Result}; +use crate::platform::DriveLetter; + +/// `$Boot` payload size: the boot region is 16 sectors × 512 bytes. +#[cfg(windows)] +const BOOT_BYTES: usize = 8192; + +/// Read a metafile's raw bytes from a live NTFS volume. +/// +/// # Errors +/// +/// Returns [`MftError::Io`] / [`MftError::Windows`] if opening the volume or +/// reading fails. +#[cfg(windows)] +pub fn read_metafile(drive: DriveLetter, kind: MetafileKind) -> Result> { + let handle = crate::platform::VolumeHandle::open(drive)?; + let vol = handle.volume_data(); + match kind { + // `$Boot` is fixed at LCN 0; read it directly (no data-run parse). + MetafileKind::Boot => read_boot(&handle), + // `$Bitmap` is a non-resident unnamed `$DATA` stream. + MetafileKind::Bitmap => read_data_stream(&handle, vol, 6, None), + // `$Secure:$SDS` holds deduplicated security descriptors (ACLs). + MetafileKind::Secure => read_data_stream(&handle, vol, 9, Some("$SDS")), + // `$AttrDef` / `$MFTMirr` / `$LogFile` are unnamed non-resident `$DATA` + // streams. + MetafileKind::AttrDef => read_data_stream(&handle, vol, 4, None), + MetafileKind::MftMirr => read_data_stream(&handle, vol, 1, None), + MetafileKind::LogFile => read_data_stream(&handle, vol, 2, None), + // `$Volume` / `$BadClus` keep their useful data in the MFT record itself + // (the $VOLUME_* attributes; the $Bad run list), so capture the fixed-up + // record. + MetafileKind::Volume => read_frs_record(&handle, vol, 3), + MetafileKind::BadClus => read_frs_record(&handle, vol, 8), + // `$UsnJrnl:$J` lives under `$Extend`; resolve its FRS then read `$J`. + MetafileKind::UsnJrnl => read_usn_journal(&handle, vol), + } +} + +/// Read a metafile's raw bytes (non-Windows stub). +/// +/// # Errors +/// +/// Always returns [`MftError::PlatformNotSupported`]. +#[cfg(not(windows))] +pub const fn read_metafile(_drive: DriveLetter, _kind: MetafileKind) -> Result> { + Err(MftError::PlatformNotSupported) +} + +/// `$Boot` is the volume boot region: 8 KiB starting at LCN 0 (byte offset 0). +#[cfg(windows)] +fn read_boot(handle: &crate::platform::VolumeHandle) -> Result> { + let mut buf = vec![0_u8; BOOT_BYTES]; + super::volume::read_handle_at(handle.raw_handle(), 0, &mut buf)?; + Ok(buf) +} + +/// Read a raw MFT file-record segment (FRS) with USA fixup applied. +#[cfg(windows)] +fn read_frs_record( + handle: &crate::platform::VolumeHandle, + vol: &crate::platform::NtfsVolumeData, + frs: u64, +) -> Result> { + let record_size = vol.bytes_per_file_record_segment as usize; + let offset = handle.mft_byte_offset() + frs * u64::from(vol.bytes_per_file_record_segment); + let mut record = vec![0_u8; record_size]; + super::volume::read_handle_at(handle.raw_handle(), offset, &mut record)?; + crate::parse::apply_fixup(&mut record); + Ok(record) +} + +/// Locate a non-resident `$DATA` attribute in one MFT record — the unnamed +/// stream when `want_name` is `None`, or the named stream otherwise — returning +/// its data runs and logical size. +#[cfg(windows)] +fn find_data_attr( + record: &[u8], + want_name: Option<&[u16]>, +) -> Option<(Vec, u64)> { + use crate::ntfs::{AttributeIterator, AttributeType}; + + let mut attrs = AttributeIterator::new(record)?; + let attr = attrs.find(|attr| { + attr.attribute_type() == Some(AttributeType::Data) + && attr.is_non_resident() + && want_name.map_or(attr.header.name_length == 0, |want| { + attr.name().as_deref() == Some(want) + }) + })?; + let non_resident = attr.non_resident_data()?; + Some((attr.data_runs(), non_resident.data_size.cast_unsigned())) +} + +/// Follow `$ATTRIBUTE_LIST` from a base record to the extension record(s) that +/// hold the named `$DATA` stream, returning its merged runs + logical size. +/// +/// NTFS relocates attributes to extension records when a base record overflows, +/// which is how `$Secure:$SDS` and `$UsnJrnl:$J` are stored. Runs are +/// concatenated in `$ATTRIBUTE_LIST` (VCN) order; the real `data_size` lives in +/// the first (VCN-0) instance and later instances report `0`. +#[cfg(windows)] +fn find_data_in_extensions( + handle: &crate::platform::VolumeHandle, + vol: &crate::platform::NtfsVolumeData, + base_record: &[u8], + stream_name: &str, +) -> Result, u64)>> { + use crate::ntfs::{AttributeIterator, AttributeType}; + + // Materialize the $ATTRIBUTE_LIST payload (resident or non-resident). + let list = { + let mut attrs = AttributeIterator::new(base_record) + .ok_or_else(|| MftError::InvalidData("record: invalid header".to_owned()))?; + let Some(list_attr) = + attrs.find(|attr| attr.attribute_type() == Some(AttributeType::AttributeList)) + else { + return Ok(None); + }; + if let Some(resident) = list_attr.resident_value() { + resident.to_vec() + } else if let Some(non_resident) = list_attr.non_resident_data() { + read_runs( + handle.raw_handle(), + &list_attr.data_runs(), + vol.bytes_per_cluster, + non_resident.data_size.cast_unsigned(), + )? + } else { + return Ok(None); + } + }; + + let want: Vec = stream_name.encode_utf16().collect(); + let mut runs: Vec = Vec::new(); + let mut data_size = 0_u64; + for ext_frs in crate::platform::metafile_decode::attribute_list_data_frs(&list, stream_name) { + let ext = read_frs_record(handle, vol, ext_frs)?; + if let Some((ext_runs, ext_size)) = find_data_attr(&ext, Some(&want)) { + if data_size == 0 { + data_size = ext_size; // VCN-0 instance carries the real size + } + runs.extend(ext_runs); + } + } + if runs.is_empty() { + Ok(None) + } else { + Ok(Some((runs, data_size))) + } +} + +/// Resolve a stream's data runs + logical size — the unnamed `$DATA` when +/// `stream_name` is `None`, or the named stream (e.g. `$SDS` / `$J`) otherwise +/// — looking in the base record first, then any `$ATTRIBUTE_LIST` extensions. +#[cfg(windows)] +fn resolve_data_runs( + handle: &crate::platform::VolumeHandle, + vol: &crate::platform::NtfsVolumeData, + frs: u64, + stream_name: Option<&str>, +) -> Result<(Vec, u64)> { + let record = read_frs_record(handle, vol, frs)?; + let want_name: Option> = stream_name.map(|name| name.encode_utf16().collect()); + + // The attribute usually lives in the base record. + if let Some(found) = find_data_attr(&record, want_name.as_deref()) { + return Ok(found); + } + + // Named streams ($SDS, $J) can overflow into an extension record. + if let Some(name) = stream_name + && let Some(found) = find_data_in_extensions(handle, vol, &record, name)? + { + return Ok(found); + } + + Err(MftError::InvalidData(format!( + "FRS {frs}: no matching non-resident DATA stream (name={stream_name:?})" + ))) +} + +/// Read a metafile's non-resident `$DATA` stream — the unnamed stream when +/// `stream_name` is `None`, or the named stream (e.g. `$SDS`) otherwise — by +/// resolving its data runs and reading the referenced clusters. +#[cfg(windows)] +fn read_data_stream( + handle: &crate::platform::VolumeHandle, + vol: &crate::platform::NtfsVolumeData, + frs: u64, + stream_name: Option<&str>, +) -> Result> { + let (runs, size) = resolve_data_runs(handle, vol, frs, stream_name)?; + read_runs(handle.raw_handle(), &runs, vol.bytes_per_cluster, size) +} + +/// Assemble a stream's data runs into a `data_size`-byte buffer. Raw volume +/// reads must be sector-aligned in length, so allocate the whole +/// cluster-aligned span the runs cover (≥ `data_size`), read each run in full, +/// then truncate to the real `data_size` (a last read clamped to a +/// non-cluster-aligned `data_size` fails with `ERROR_INVALID_PARAMETER`). +/// Sparse runs stay zeroed. +#[cfg(windows)] +fn read_runs( + handle: windows::Win32::Foundation::HANDLE, + runs: &[crate::ntfs::DataRun], + bytes_per_cluster: u32, + data_size: u64, +) -> Result> { + let bpc = u64::from(bytes_per_cluster); + let allocated: u64 = runs.iter().map(|run| run.cluster_count * bpc).sum(); + let capacity = usize::try_from(allocated.max(data_size)).map_err(|_err| { + MftError::InvalidData("metafile stream size exceeds usize::MAX".to_owned()) + })?; + let mut buf = alloc_zeroed(capacity)?; + let mut offset: usize = 0; + + for run in runs { + let run_bytes = usize::try_from(run.cluster_count * bpc).map_err(|_err| { + MftError::InvalidData("metafile run byte count exceeds usize::MAX".to_owned()) + })?; + if !run.is_sparse() { + // Cluster-aligned offset + length → sector-aligned raw read. + let disk_offset = crate::index::nonneg_to_u64(run.lcn.raw() * bpc.cast_signed()); + let Some(window) = buf.get_mut(offset..offset + run_bytes) else { + return Err(MftError::InvalidData(format!( + "metafile run at offset {offset} len {run_bytes} exceeds buffer {capacity}" + ))); + }; + super::volume::read_handle_at(handle, disk_offset, window)?; + } + offset = offset.saturating_add(run_bytes); + } + + let final_len = usize::try_from(data_size).unwrap_or(capacity).min(capacity); + buf.truncate(final_len); + Ok(buf) +} + +/// Allocate a zeroed buffer, failing cleanly (not aborting) when a sparse +/// stream's span is too large to materialize. +#[cfg(windows)] +fn alloc_zeroed(capacity: usize) -> Result> { + let mut buf: Vec = Vec::new(); + buf.try_reserve_exact(capacity).map_err(|_err| { + MftError::InvalidData(format!( + "metafile stream too large to materialize: {capacity} bytes" + )) + })?; + buf.resize(capacity, 0); + Ok(buf) +} + +/// Read only the non-sparse (allocated) runs of a stream, concatenated. +/// +/// `$UsnJrnl:$J` is a sparse file whose logical size can reach hundreds of GB +/// while the live journal is a tiny allocated tail (the purged prefix is a hole +/// carrying no data). So this captures just the allocated bytes — each USN +/// record is self-describing, so dropping the leading hole loses no data. +#[cfg(windows)] +fn read_runs_sparse_compact( + handle: windows::Win32::Foundation::HANDLE, + runs: &[crate::ntfs::DataRun], + bytes_per_cluster: u32, +) -> Result> { + let bpc = u64::from(bytes_per_cluster); + let allocated: u64 = runs + .iter() + .filter(|run| !run.is_sparse()) + .map(|run| run.cluster_count * bpc) + .sum(); + let capacity = usize::try_from(allocated).map_err(|_err| { + MftError::InvalidData("metafile stream size exceeds usize::MAX".to_owned()) + })?; + let mut buf = alloc_zeroed(capacity)?; + let mut offset: usize = 0; + for run in runs.iter().filter(|run| !run.is_sparse()) { + let run_bytes = usize::try_from(run.cluster_count * bpc).map_err(|_err| { + MftError::InvalidData("metafile run byte count exceeds usize::MAX".to_owned()) + })?; + // Cluster-aligned offset + length → sector-aligned raw read. + let disk_offset = crate::index::nonneg_to_u64(run.lcn.raw() * bpc.cast_signed()); + let Some(window) = buf.get_mut(offset..offset + run_bytes) else { + return Err(MftError::InvalidData(format!( + "metafile run at offset {offset} len {run_bytes} exceeds buffer {capacity}" + ))); + }; + super::volume::read_handle_at(handle, disk_offset, window)?; + offset = offset.saturating_add(run_bytes); + } + Ok(buf) +} + +/// FRS of the `$Extend` metadata directory. +#[cfg(windows)] +const EXTEND_FRS: u64 = 11; + +/// Read the `$UsnJrnl:$J` change journal via `$Extend` directory traversal. +/// +/// Captures only the allocated journal data (see [`read_runs_sparse_compact`]); +/// the huge sparse prefix of `$J` is a purged hole with nothing to store. +#[cfg(windows)] +fn read_usn_journal( + handle: &crate::platform::VolumeHandle, + vol: &crate::platform::NtfsVolumeData, +) -> Result> { + let usn_frs = resolve_extend_child(handle, vol, "$UsnJrnl")?; + let (runs, _size) = resolve_data_runs(handle, vol, usn_frs, Some("$J"))?; + read_runs_sparse_compact(handle.raw_handle(), &runs, vol.bytes_per_cluster) +} + +/// Resolve the FRS of a `$Extend` (FRS 11) child by name, walking its directory +/// index — `$INDEX_ROOT` first, then `$INDEX_ALLOCATION` if the index is large. +#[cfg(windows)] +fn resolve_extend_child( + handle: &crate::platform::VolumeHandle, + vol: &crate::platform::NtfsVolumeData, + child: &str, +) -> Result { + use crate::ntfs::{AttributeIterator, AttributeType, DataRun}; + + let record = read_frs_record(handle, vol, EXTEND_FRS)?; + let target: Vec = child.encode_utf16().collect(); + + let mut root_value: Option> = None; + let mut block_size: usize = 0; + let mut alloc: Option<(Vec, u64)> = None; + + let attrs = AttributeIterator::new(&record) + .ok_or_else(|| MftError::InvalidData("$Extend: invalid record header".to_owned()))?; + for attr in attrs { + match attr.attribute_type() { + Some(AttributeType::IndexRoot) => { + if let Some(value) = attr.resident_value() { + // IndexRoot.bytes_per_index_block is at offset 8. + block_size = value + .get(8..12) + .and_then(|slice| slice.try_into().ok()) + .map_or(0, |bytes| u32::from_le_bytes(bytes) as usize); + root_value = Some(value.to_vec()); + } + } + Some(AttributeType::IndexAllocation) if attr.is_non_resident() => { + if let Some(nr) = attr.non_resident_data() { + alloc = Some((attr.data_runs(), nr.data_size.cast_unsigned())); + } + } + _ => {} + } + } + + // 1. Search the resident $INDEX_ROOT (its INDEX_HEADER begins at offset 16). + if let Some(frs) = root_value + .as_deref() + .and_then(|root| scan_index_entries(root, 16, &target)) + { + return Ok(frs); + } + + // 2. Search the $INDEX_ALLOCATION INDX blocks, if the index spilled. + if let Some((runs, size)) = alloc + && block_size > 0 + { + let buf = read_runs(handle.raw_handle(), &runs, vol.bytes_per_cluster, size)?; + if let Some(frs) = scan_index_blocks(&buf, block_size, &target) { + return Ok(frs); + } + } + + Err(MftError::InvalidData(format!( + "$Extend index: {child} not found (no active USN journal?)" + ))) +} + +/// Scan NTFS directory-index entries for the entry whose `$FILE_NAME` matches +/// `target` (UTF-16), returning its FRS. `header_start` is the byte offset of +/// the `INDEX_HEADER` within `buf` (whose `first_entry_offset` is relative to +/// it). +#[cfg(any(windows, test))] +fn scan_index_entries(buf: &[u8], header_start: usize, target: &[u16]) -> Option { + let first_entry_offset = + u32::from_le_bytes(buf.get(header_start..header_start + 4)?.try_into().ok()?) as usize; + let mut pos = header_start.checked_add(first_entry_offset)?; + loop { + let entry = buf.get(pos..)?; + let flags = u16::from_le_bytes(entry.get(12..14)?.try_into().ok()?); + // Last-entry flag (0x02): end of this node. + if (flags & 0x02) != 0 { + return None; + } + let file_reference = u64::from_le_bytes(entry.get(0..8)?.try_into().ok()?); + // The $FILE_NAME key starts at entry offset 16: name_length @0x40, + // UTF-16 name @0x42. + let name_length = usize::from(*entry.get(16 + 0x40)?); + let name_bytes = entry.get(16 + 0x42..16 + 0x42 + name_length * 2)?; + let name: Vec = name_bytes + .chunks_exact(2) + .filter_map(|pair| pair.try_into().ok().map(u16::from_le_bytes)) + .collect(); + if name == target { + return Some(crate::ntfs::file_reference_to_frs(file_reference)); + } + let entry_length = usize::from(u16::from_le_bytes(entry.get(8..10)?.try_into().ok()?)); + if entry_length == 0 { + return None; + } + pos = pos.checked_add(entry_length)?; + } +} + +/// Scan the `INDX` blocks in an `$INDEX_ALLOCATION` buffer for `target`. +#[cfg(windows)] +fn scan_index_blocks(buf: &[u8], block_size: usize, target: &[u16]) -> Option { + if block_size == 0 { + return None; + } + for start in (0..buf.len()).step_by(block_size) { + let Some(end) = start.checked_add(block_size) else { + break; + }; + let Some(block) = buf.get(start..end) else { + break; + }; + if !block.starts_with(b"INDX") { + continue; // sparse / unused block + } + let mut owned = block.to_vec(); + let usa_offset = u16::from_le_bytes(owned.get(4..6)?.try_into().ok()?); + let usa_count = u16::from_le_bytes(owned.get(6..8)?.try_into().ok()?); + if !crate::ntfs::apply_usa_fixup(&mut owned, usa_offset, usa_count) { + continue; + } + // The INDEX_HEADER begins at offset 0x18 within an INDX block. + if let Some(frs) = scan_index_entries(&owned, 0x18, target) { + return Some(frs); + } + } + None +} + +#[cfg(test)] +mod tests { + use super::scan_index_entries; + + #[test] + #[expect( + clippy::indexing_slicing, + reason = "test builds a fixed 256-byte index buffer with known in-bounds offsets" + )] + fn scan_index_entries_resolves_child_frs() { + // Minimal $INDEX_ROOT-style buffer: INDEX_HEADER at offset 0 + // (first_entry_offset = 16), one entry for "$UsnJrnl" → FRS 42. + let mut buf = vec![0_u8; 256]; + buf[0..4].copy_from_slice(&16_u32.to_le_bytes()); // first_entry_offset + let entry = 16_usize; + buf[entry..entry + 8].copy_from_slice(&42_u64.to_le_bytes()); // file_reference + // flags @ entry+12 stay 0 (not the last entry). + let name: Vec = "$UsnJrnl".encode_utf16().collect(); + let key = entry + 16; + buf[key + 0x40] = 8; // name_length (UTF-16 code units) + for (i, unit) in name.iter().enumerate() { + let off = key + 0x42 + i * 2; + buf[off..off + 2].copy_from_slice(&unit.to_le_bytes()); + } + + let target: Vec = "$UsnJrnl".encode_utf16().collect(); + assert_eq!(scan_index_entries(&buf, 0, &target), Some(42)); + + // A miss: entry_length is 0 after the single entry, so the scan stops. + let miss: Vec = "$Nope".encode_utf16().collect(); + assert_eq!(scan_index_entries(&buf, 0, &miss), None); + } +} From cf1fc816f803f523d37b278bc2a441a789f33a34 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Mon, 6 Jul 2026 10:28:01 -0700 Subject: [PATCH 22/39] fix(uffs-mft): reject garbage USN records in $UsnJrnl decoder A USN is the record's byte offset in the journal, so it is always positive and 8-byte aligned. The sparse-compact $J capture can leave stale/padding bytes after the live tail that pass the RecordLength + MajorVersion checks, producing a bogus record (observed last_usn 281479271678612, misaligned). Reject records whose USN is non-positive or not 8-aligned, and bound RecordLength / require MinorVersion==0. Locked with a misaligned-tail test. Co-Authored-By: Claude Opus 4.8 --- .../uffs-mft/src/platform/metafile_decode.rs | 39 ++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/crates/uffs-mft/src/platform/metafile_decode.rs b/crates/uffs-mft/src/platform/metafile_decode.rs index 9072a58ab..f0edd2003 100644 --- a/crates/uffs-mft/src/platform/metafile_decode.rs +++ b/crates/uffs-mft/src/platform/metafile_decode.rs @@ -219,7 +219,8 @@ pub fn parse_usn(payload: &[u8]) -> UsnSummary { }; let rec_len = rd_u32(record, 0).unwrap_or(0) as usize; let major = rd_u16(record, 4).unwrap_or(0); - if rec_len < 0x3C || (major != 2 && major != 3) { + let minor = rd_u16(record, 6).unwrap_or(0xFFFF); + if !(0x3C..=0x400).contains(&rec_len) || minor != 0 || (major != 2 && major != 3) { pos += 8; // sparse gap / alignment padding continue; } @@ -227,6 +228,13 @@ pub fn parse_usn(payload: &[u8]) -> UsnSummary { break; } let usn = rd_i64(record, 0x18).unwrap_or(0); + // A USN is the record's byte offset in the journal, so it is always + // positive and 8-byte aligned. A misaligned/negative value means we + // walked into stale or padding bytes rather than a real record. + if usn <= 0 || (usn & 0b111) != 0 { + pos += 8; + continue; + } let reason = rd_u32(record, 0x28).unwrap_or(0); let name_len = usize::from(rd_u16(record, 0x38).unwrap_or(0)); let name_off = usize::from(rd_u16(record, 0x3A).unwrap_or(0)); @@ -393,6 +401,35 @@ mod tests { assert_eq!(parse_usn(&[0_u8; 64]).record_count, 0); } + #[test] + #[expect( + clippy::indexing_slicing, + reason = "test builds fixed USN records with known in-bounds offsets" + )] + fn parse_usn_rejects_misaligned_usn() { + // One valid V2 record (usn 800, 8-aligned) followed by a well-formed + // header whose USN is misaligned (804) — stale bytes, not a real record. + let rec_len = 0x3C_usize; // header-only record + let mut buf = vec![0_u8; rec_len * 2]; + + // Valid record at offset 0. + buf[0..4].copy_from_slice(&u32::try_from(rec_len).unwrap_or(0).to_le_bytes()); + buf[4..6].copy_from_slice(&2_u16.to_le_bytes()); // major + buf[0x18..0x20].copy_from_slice(&800_i64.to_le_bytes()); // usn (8-aligned) + + // Garbage "record" at offset rec_len: plausible header, misaligned usn. + let garbage = rec_len; + buf[garbage..garbage + 4] + .copy_from_slice(&u32::try_from(rec_len).unwrap_or(0).to_le_bytes()); + buf[garbage + 4..garbage + 6].copy_from_slice(&2_u16.to_le_bytes()); + buf[garbage + 0x18..garbage + 0x20].copy_from_slice(&804_i64.to_le_bytes()); // usn & 7 == 4 + + let summary = parse_usn(&buf); + assert_eq!(summary.record_count, 1); + assert_eq!(summary.first_usn, 800); + assert_eq!(summary.last_usn, 800); + } + #[test] #[expect( clippy::indexing_slicing, From f9f8651988bfd541c42edc4d9155378d84138485 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Mon, 6 Jul 2026 10:45:19 -0700 Subject: [PATCH 23/39] feat(uffs-mft): add capture --all-drives Capture every eligible NTFS volume in one run, each into its own out/drive_/ subfolder. --drive is now optional (required only for the single-drive path); per-drive failures are reported and skipped so the run continues, and the command errors at the end if any drive failed. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/cli.rs | 9 +++- crates/uffs-mft/src/commands/mod.rs | 6 ++- .../uffs-mft/src/commands/windows/capture.rs | 46 +++++++++++++++++-- 3 files changed, 55 insertions(+), 6 deletions(-) diff --git a/crates/uffs-mft/src/cli.rs b/crates/uffs-mft/src/cli.rs index 1db7a7c83..4bbf3efcf 100644 --- a/crates/uffs-mft/src/cli.rs +++ b/crates/uffs-mft/src/cli.rs @@ -167,13 +167,18 @@ pub(crate) enum Commands { /// Capture all NTFS metafiles for a drive into a hashed, manifested bundle Capture { - /// Drive letter (e.g., C, D, E) + /// Drive letter (e.g., C, D, E). Required unless `--all-drives` is set. #[arg(short, long)] - drive: uffs_mft::platform::DriveLetter, + drive: Option, /// Output directory (bundle written to `/drive_/`) #[arg(short, long)] out: PathBuf, + + /// Capture every eligible NTFS volume, each into its own + /// `/drive_/` subfolder. + #[arg(long)] + all_drives: bool, }, /// Inspect a captured NTFS metafile offline (header + $Boot geometry, ...) diff --git a/crates/uffs-mft/src/commands/mod.rs b/crates/uffs-mft/src/commands/mod.rs index cd603ecdc..b4a6428fb 100644 --- a/crates/uffs-mft/src/commands/mod.rs +++ b/crates/uffs-mft/src/commands/mod.rs @@ -44,7 +44,11 @@ pub(crate) async fn dispatch_command(command: Commands) -> Result<()> { kind, output, } => windows::cmd_metafile(drive, kind, &output).await, - Commands::Capture { drive, out } => windows::cmd_capture(drive, &out).await, + Commands::Capture { + drive, + out, + all_drives, + } => windows::cmd_capture(drive, &out, all_drives).await, Commands::MetafileInfo { input } => metafile_info::cmd_metafile_info(&input), Commands::Bench { drive, diff --git a/crates/uffs-mft/src/commands/windows/capture.rs b/crates/uffs-mft/src/commands/windows/capture.rs index 4fb062986..faf6383b2 100644 --- a/crates/uffs-mft/src/commands/windows/capture.rs +++ b/crates/uffs-mft/src/commands/windows/capture.rs @@ -225,9 +225,9 @@ fn collect_artifacts( artifacts } -/// `capture` command — write the `$MFT` + all metafiles + `manifest.json` + -/// `SHA256SUMS` for a drive into `out/drive_/`. -pub(crate) async fn cmd_capture(drive: DriveLetter, out: &Path) -> Result<()> { +/// Capture one drive's `$MFT` + all metafiles + `manifest.json` + `SHA256SUMS` +/// into `out/drive_/`. +fn capture_one_drive(drive: DriveLetter, out: &Path) -> Result<()> { let drive_lower = drive.to_string().to_lowercase(); let dir = out.join(format!("drive_{drive_lower}")); std::fs::create_dir_all(&dir) @@ -254,3 +254,43 @@ pub(crate) async fn cmd_capture(drive: DriveLetter, out: &Path) -> Result<()> { println!(" {} artifact(s) captured.", manifest.artifacts.len()); Ok(()) } + +/// `capture` command — bundle one drive (`--drive C`) or every eligible NTFS +/// volume (`--all-drives`) into `out/drive_/`. +/// +/// With `--all-drives`, a per-drive failure is reported and skipped so the run +/// continues; the command still errors at the end if any drive failed. +pub(crate) async fn cmd_capture( + drive: Option, + out: &Path, + all_drives: bool, +) -> Result<()> { + if !all_drives { + let only = + drive.context("`--drive ` is required unless `--all-drives` is given")?; + return capture_one_drive(only, out); + } + + let drives = uffs_mft::platform::detect_ntfs_drives(); + if drives.is_empty() { + anyhow::bail!("no NTFS drives detected to capture"); + } + println!("Capturing {} NTFS drive(s)…", drives.len()); + + let mut failures: Vec = Vec::new(); + for letter in drives { + if let Err(err) = capture_one_drive(letter, out) { + println!(" ⚠️ drive {letter}: capture failed — {err:#}"); + failures.push(letter); + } + } + if !failures.is_empty() { + let list = failures + .iter() + .map(ToString::to_string) + .collect::>() + .join(", "); + anyhow::bail!("{} drive(s) failed to capture: {list}", failures.len()); + } + Ok(()) +} From ce39012dccfb5cd4793ae9daf679d0082bb387c3 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Mon, 6 Jul 2026 10:52:05 -0700 Subject: [PATCH 24/39] feat(uffs-mft): add capture --zip / --split-gib bundling Pack each captured drive directory into a single .tar.zst (extractable with `tar --zstd -xf`), optionally split into --split-gib GiB parts (.tar.zst.NNN) for transfer. No new dependency: add a small, unit-tested ustar writer + byte splitter (src/archive.rs) and compress with the existing zstd. Works per-drive, so it composes with --all-drives (each volume archived independently, bounding memory). Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/archive.rs | 164 ++++++++++++++++++ crates/uffs-mft/src/cli.rs | 10 ++ crates/uffs-mft/src/commands/mod.rs | 4 +- .../uffs-mft/src/commands/windows/capture.rs | 76 +++++++- crates/uffs-mft/src/lib.rs | 3 + 5 files changed, 248 insertions(+), 9 deletions(-) create mode 100644 crates/uffs-mft/src/archive.rs diff --git a/crates/uffs-mft/src/archive.rs b/crates/uffs-mft/src/archive.rs new file mode 100644 index 000000000..08056a3f7 --- /dev/null +++ b/crates/uffs-mft/src/archive.rs @@ -0,0 +1,164 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) 2025-2026 SKY, LLC. + +//! Minimal `ustar` (tar) archive writer and byte splitter for capture bundles. +//! +//! No third-party archive dependency: a capture bundle is packed as a plain +//! `ustar` archive (readable by any `tar`), which the caller compresses with +//! the existing `zstd` dependency into a `.tar.zst` and optionally splits into +//! fixed-size parts for transfer. Pure and cross-platform, so it is unit-tested +//! on the build host and the offline (macOS/Linux) side can reason about it +//! too. + +use crate::error::{MftError, Result}; +use crate::usize_to_u64; + +/// tar block size in bytes. +const BLOCK: usize = 512; + +/// Maximum file name length in the `ustar` name field. +const NAME_MAX: usize = 100; + +/// Offset of the 8-byte checksum field within a `ustar` header. +const CHKSUM_OFFSET: usize = 148; + +/// Push `content` into `buf`, truncated or zero-padded to exactly `width` +/// bytes. +fn push_field(buf: &mut Vec, content: &[u8], width: usize) { + let take = content.len().min(width); + if let Some(slice) = content.get(..take) { + buf.extend_from_slice(slice); + } + buf.resize(buf.len() + (width - take), 0); +} + +/// Append one regular-file entry (`name` → `data`) to a `ustar` archive `buf`. +/// +/// # Errors +/// +/// Returns [`MftError::InvalidData`] if `name` exceeds the 100-byte `ustar` +/// name field. +pub fn push_entry(buf: &mut Vec, name: &str, data: &[u8]) -> Result<()> { + if name.len() > NAME_MAX { + return Err(MftError::InvalidData(format!( + "tar entry name too long ({} > {NAME_MAX}): {name}", + name.len() + ))); + } + let header_start = buf.len(); + push_field(buf, name.as_bytes(), NAME_MAX); // name + push_field(buf, b"0000644\0", 8); // mode + push_field(buf, b"0000000\0", 8); // uid + push_field(buf, b"0000000\0", 8); // gid + push_field( + buf, + format!("{:011o}\0", usize_to_u64(data.len())).as_bytes(), + 12, + ); // size + push_field(buf, b"00000000000\0", 12); // mtime (0) + push_field(buf, b" ", 8); // chksum placeholder (spaces) + push_field(buf, b"0", 1); // typeflag: regular file + push_field(buf, b"", NAME_MAX); // linkname + push_field(buf, b"ustar\0", 6); // magic + push_field(buf, b"00", 2); // version + push_field(buf, b"", 32); // uname + push_field(buf, b"", 32); // gname + push_field(buf, b"", 8); // devmajor + push_field(buf, b"", 8); // devminor + push_field(buf, b"", 155); // prefix + push_field(buf, b"", 12); // pad to 512 + + // Header checksum: sum of all header bytes with the checksum field as + // spaces (the placeholder above), written back as 6 octal digits + NUL + ' '. + let sum: u32 = buf + .get(header_start..header_start + BLOCK) + .map_or(0, |header| header.iter().map(|&byte| u32::from(byte)).sum()); + if let Some(field) = buf.get_mut(header_start + CHKSUM_OFFSET..header_start + CHKSUM_OFFSET + 8) + { + for (slot, byte) in field.iter_mut().zip(format!("{sum:06o}\0 ").bytes()) { + *slot = byte; + } + } + + // File data, zero-padded up to a block boundary. + buf.extend_from_slice(data); + let rem = data.len() % BLOCK; + if rem != 0 { + buf.resize(buf.len() + (BLOCK - rem), 0); + } + Ok(()) +} + +/// Append the two zero blocks that mark end-of-archive. +pub fn finish(buf: &mut Vec) { + buf.resize(buf.len() + BLOCK * 2, 0); +} + +/// Split `data` into consecutive parts of at most `part_size` bytes (the last +/// part may be smaller). A `part_size` of `0` returns the data as one part. +#[must_use] +pub fn split(data: &[u8], part_size: usize) -> Vec<&[u8]> { + if part_size == 0 || data.is_empty() { + return vec![data]; + } + data.chunks(part_size).collect() +} + +#[cfg(test)] +mod tests { + use super::{BLOCK, CHKSUM_OFFSET, finish, push_entry, split}; + + #[test] + #[expect( + clippy::indexing_slicing, + reason = "test inspects fixed ustar header offsets known to be in bounds" + )] + fn push_entry_writes_valid_ustar_header() { + let mut buf = Vec::new(); + push_entry(&mut buf, "c_boot.bin", b"hello").expect("valid name"); + finish(&mut buf); + + // name, magic, size (octal of 5), and data land at their ustar offsets. + assert_eq!(&buf[0..10], b"c_boot.bin"); + assert_eq!(&buf[257..263], b"ustar\0"); + assert_eq!(&buf[124..135], b"00000000005"); + assert_eq!(&buf[512..517], b"hello"); + + // 512 header + 512 data block + 2×512 end blocks. + assert_eq!(buf.len(), BLOCK + BLOCK + BLOCK * 2); + + // The stored checksum equals the header sum computed with the checksum + // field read as spaces. + let stored = u32::from_str_radix( + core::str::from_utf8(&buf[CHKSUM_OFFSET..CHKSUM_OFFSET + 6]) + .expect("ascii") + .trim(), + 8, + ) + .expect("octal checksum"); + let mut header = buf[0..BLOCK].to_vec(); + for byte in &mut header[CHKSUM_OFFSET..CHKSUM_OFFSET + 8] { + *byte = b' '; + } + let recomputed: u32 = header.iter().map(|&byte| u32::from(byte)).sum(); + assert_eq!(stored, recomputed); + } + + #[test] + fn push_entry_rejects_overlong_name() { + let mut buf = Vec::new(); + push_entry(&mut buf, &"a".repeat(101), b"x").unwrap_err(); + } + + #[test] + fn split_chunks_data() { + let data = [1_u8, 2, 3, 4, 5]; + let parts = split(&data, 2); + assert_eq!(parts.len(), 3); + assert_eq!(parts.first().copied(), Some(&[1, 2][..])); + assert_eq!(parts.get(2).copied(), Some(&[5][..])); + + // Zero part size → one part covering all the data. + assert_eq!(split(&data, 0), vec![&data[..]]); + } +} diff --git a/crates/uffs-mft/src/cli.rs b/crates/uffs-mft/src/cli.rs index 4bbf3efcf..0cb97d783 100644 --- a/crates/uffs-mft/src/cli.rs +++ b/crates/uffs-mft/src/cli.rs @@ -179,6 +179,16 @@ pub(crate) enum Commands { /// `/drive_/` subfolder. #[arg(long)] all_drives: bool, + + /// Also pack each drive bundle into a `.tar.zst` + /// (extract with `tar --zstd -xf`). + #[arg(long)] + zip: bool, + + /// With `--zip`, split the archive into parts of this many GiB + /// (`.tar.zst.NNN`). 0 = no split. + #[arg(long, default_value_t = 0, value_name = "GIB")] + split_gib: u64, }, /// Inspect a captured NTFS metafile offline (header + $Boot geometry, ...) diff --git a/crates/uffs-mft/src/commands/mod.rs b/crates/uffs-mft/src/commands/mod.rs index b4a6428fb..a66366900 100644 --- a/crates/uffs-mft/src/commands/mod.rs +++ b/crates/uffs-mft/src/commands/mod.rs @@ -48,7 +48,9 @@ pub(crate) async fn dispatch_command(command: Commands) -> Result<()> { drive, out, all_drives, - } => windows::cmd_capture(drive, &out, all_drives).await, + zip, + split_gib, + } => windows::cmd_capture(drive, &out, all_drives, zip, split_gib).await, Commands::MetafileInfo { input } => metafile_info::cmd_metafile_info(&input), Commands::Bench { drive, diff --git a/crates/uffs-mft/src/commands/windows/capture.rs b/crates/uffs-mft/src/commands/windows/capture.rs index faf6383b2..97b8ee02a 100644 --- a/crates/uffs-mft/src/commands/windows/capture.rs +++ b/crates/uffs-mft/src/commands/windows/capture.rs @@ -226,8 +226,8 @@ fn collect_artifacts( } /// Capture one drive's `$MFT` + all metafiles + `manifest.json` + `SHA256SUMS` -/// into `out/drive_/`. -fn capture_one_drive(drive: DriveLetter, out: &Path) -> Result<()> { +/// into `out/drive_/`, returning that directory. +fn capture_one_drive(drive: DriveLetter, out: &Path) -> Result { let drive_lower = drive.to_string().to_lowercase(); let dir = out.join(format!("drive_{drive_lower}")); std::fs::create_dir_all(&dir) @@ -252,11 +252,57 @@ fn capture_one_drive(drive: DriveLetter, out: &Path) -> Result<()> { println!(" Manifest: {}", dir.join("manifest.json").display()); println!(" Hashes: {}", dir.join("SHA256SUMS").display()); println!(" {} artifact(s) captured.", manifest.artifacts.len()); + Ok(dir) +} + +/// Pack a captured drive directory into a single `.tar.zst` (extractable +/// with `tar --zstd -xf`), optionally split into `split_gib`-GiB parts named +/// `.tar.zst.NNN` for transfer. +fn archive_dir(dir: &Path, split_gib: u64) -> Result<()> { + let mut files: Vec = std::fs::read_dir(dir) + .with_context(|| format!("listing {}", dir.display()))? + .filter_map(core::result::Result::ok) + .map(|entry| entry.path()) + .filter(|path| path.is_file()) + .collect(); + files.sort(); + + let mut tar = Vec::new(); + for path in &files { + let name = path + .file_name() + .map(|os| os.to_string_lossy().into_owned()) + .unwrap_or_default(); + let data = std::fs::read(path).with_context(|| format!("reading {}", path.display()))?; + uffs_mft::archive::push_entry(&mut tar, &name, &data) + .with_context(|| format!("archiving {name}"))?; + } + uffs_mft::archive::finish(&mut tar); + + let compressed = + zstd::encode_all(tar.as_slice(), 3).context("zstd-compressing capture archive")?; + let base = format!("{}.tar.zst", dir.display()); + + if split_gib == 0 { + std::fs::write(&base, &compressed).with_context(|| format!("writing {base}"))?; + println!(" 📦 archive: {base} ({} bytes)", compressed.len()); + } else { + let part_size = usize::try_from(split_gib.saturating_mul(1 << 30)).unwrap_or(usize::MAX); + for (idx, part) in uffs_mft::archive::split(&compressed, part_size) + .iter() + .enumerate() + { + let name = format!("{base}.{idx:03}"); + std::fs::write(&name, part).with_context(|| format!("writing {name}"))?; + println!(" 📦 part {idx:03}: {name} ({} bytes)", part.len()); + } + } Ok(()) } /// `capture` command — bundle one drive (`--drive C`) or every eligible NTFS -/// volume (`--all-drives`) into `out/drive_/`. +/// volume (`--all-drives`) into `out/drive_/`, optionally packing each into +/// a `.tar.zst` (`--zip`, split with `--split-gib`). /// /// With `--all-drives`, a per-drive failure is reported and skipped so the run /// continues; the command still errors at the end if any drive failed. @@ -264,11 +310,17 @@ pub(crate) async fn cmd_capture( drive: Option, out: &Path, all_drives: bool, + zip: bool, + split_gib: u64, ) -> Result<()> { if !all_drives { let only = drive.context("`--drive ` is required unless `--all-drives` is given")?; - return capture_one_drive(only, out); + let dir = capture_one_drive(only, out)?; + if zip { + archive_dir(&dir, split_gib)?; + } + return Ok(()); } let drives = uffs_mft::platform::detect_ntfs_drives(); @@ -279,9 +331,17 @@ pub(crate) async fn cmd_capture( let mut failures: Vec = Vec::new(); for letter in drives { - if let Err(err) = capture_one_drive(letter, out) { - println!(" ⚠️ drive {letter}: capture failed — {err:#}"); - failures.push(letter); + match capture_one_drive(letter, out) { + Ok(dir) => { + if zip && let Err(err) = archive_dir(&dir, split_gib) { + println!(" ⚠️ drive {letter}: archive failed — {err:#}"); + failures.push(letter); + } + } + Err(err) => { + println!(" ⚠️ drive {letter}: capture failed — {err:#}"); + failures.push(letter); + } } } if !failures.is_empty() { @@ -290,7 +350,7 @@ pub(crate) async fn cmd_capture( .map(ToString::to_string) .collect::>() .join(", "); - anyhow::bail!("{} drive(s) failed to capture: {list}", failures.len()); + anyhow::bail!("{} drive(s) failed: {list}", failures.len()); } Ok(()) } diff --git a/crates/uffs-mft/src/lib.rs b/crates/uffs-mft/src/lib.rs index 541abdc4c..0a8c649e8 100644 --- a/crates/uffs-mft/src/lib.rs +++ b/crates/uffs-mft/src/lib.rs @@ -251,6 +251,9 @@ pub mod frs; pub mod cache; +/// Minimal `ustar` archive writer + byte splitter for capture bundles. +pub mod archive; + mod reader; // WI-7.1 — pathological-name parity corpus (Tier 1 decoder pins + Tier 2 From 68ad3d72a701e4ea325d1e5a9fb498013f601035 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Mon, 6 Jul 2026 11:01:34 -0700 Subject: [PATCH 25/39] feat(uffs-mft): add verify command for MFT CSV parity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Compare two MFT CSV exports (from `load`) — Rust-on-Windows vs Rust-on-macOS, or Rust vs a C++ golden — to tie down the capture flow. Rows are projected onto key columns matched by header name (so column order may differ), canonicalized as a multiset, and diffed; the command exits non-zero on any divergence so it drops into scripts. Adds a pure, unit-tested parity module (RFC 4180 field splitter, name-based column projection, multiset diff) and the cross-platform `verify --left A.csv --right B.csv [--columns ...]` command. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/cli.rs | 17 ++ crates/uffs-mft/src/commands/mod.rs | 11 ++ crates/uffs-mft/src/commands/verify.rs | 102 ++++++++++++ crates/uffs-mft/src/lib.rs | 3 + crates/uffs-mft/src/parity.rs | 210 +++++++++++++++++++++++++ 5 files changed, 343 insertions(+) create mode 100644 crates/uffs-mft/src/commands/verify.rs create mode 100644 crates/uffs-mft/src/parity.rs diff --git a/crates/uffs-mft/src/cli.rs b/crates/uffs-mft/src/cli.rs index 0cb97d783..1744ffb51 100644 --- a/crates/uffs-mft/src/cli.rs +++ b/crates/uffs-mft/src/cli.rs @@ -198,6 +198,23 @@ pub(crate) enum Commands { input: PathBuf, }, + /// Compare two MFT CSV exports (from `load`) for parity — e.g. Rust on + /// Windows vs macOS, or Rust vs a C++ golden. Exits non-zero on mismatch. + Verify { + /// First CSV export. + #[arg(short, long)] + left: PathBuf, + + /// Second CSV export. + #[arg(short, long)] + right: PathBuf, + + /// Columns to compare (matched by header name; default: all columns of + /// `--left`). Comma-separated, e.g. `frs,parent_frs,name,size`. + #[arg(long, value_delimiter = ',')] + columns: Vec, + }, + /// Benchmark MFT reading with detailed phase timing Bench { /// Drive letter (e.g., C, D, E) diff --git a/crates/uffs-mft/src/commands/mod.rs b/crates/uffs-mft/src/commands/mod.rs index a66366900..2ebb44fb9 100644 --- a/crates/uffs-mft/src/commands/mod.rs +++ b/crates/uffs-mft/src/commands/mod.rs @@ -10,6 +10,7 @@ use crate::cli::Commands; mod load; mod metafile_info; mod sysinfo; +mod verify; #[cfg(windows)] mod windows; @@ -52,6 +53,11 @@ pub(crate) async fn dispatch_command(command: Commands) -> Result<()> { split_gib, } => windows::cmd_capture(drive, &out, all_drives, zip, split_gib).await, Commands::MetafileInfo { input } => metafile_info::cmd_metafile_info(&input), + Commands::Verify { + left, + right, + columns, + } => verify::cmd_verify(&left, &right, &columns), Commands::Bench { drive, json, @@ -208,6 +214,11 @@ pub(crate) async fn dispatch_command(command: Commands) -> Result<()> { ), Commands::Sysinfo { out, json } => sysinfo::run(out.as_deref(), json), Commands::MetafileInfo { input } => metafile_info::cmd_metafile_info(&input), + Commands::Verify { + left, + right, + columns, + } => verify::cmd_verify(&left, &right, &columns), Commands::Metafile { .. } | Commands::Capture { .. } | Commands::Read { .. } diff --git a/crates/uffs-mft/src/commands/verify.rs b/crates/uffs-mft/src/commands/verify.rs new file mode 100644 index 000000000..d57ae2a26 --- /dev/null +++ b/crates/uffs-mft/src/commands/verify.rs @@ -0,0 +1,102 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) 2025-2026 SKY, LLC. + +//! `verify` command — canonical parity comparison of two MFT CSV exports. +//! +//! Cross-platform: compares two CSVs written by `uffs-mft load` (e.g. +//! Rust-on-Windows vs Rust-on-macOS, or Rust vs a C++ golden). Rows are +//! projected onto key columns matched by header *name*, so column order may +//! differ; duplicates are compared as a multiset. Exits non-zero on any +//! divergence, so it drops into scripts. +#![expect( + clippy::print_stdout, + reason = "intentional user-facing CLI parity report" +)] + +use std::io::BufRead as _; +use std::path::Path; + +use anyhow::{Context as _, Result}; +use uffs_mft::parity::{self, ParityReport}; + +/// Stream one CSV into canonical row keys. When `columns` is empty the file's +/// own header is used as the comparison column set (returned so the second file +/// can be projected onto the same columns). +fn stream_keys(path: &Path, columns: &[String]) -> Result<(Vec, Vec)> { + let file = std::fs::File::open(path).with_context(|| format!("opening {}", path.display()))?; + let mut lines = std::io::BufReader::new(file).lines(); + + let header_line = lines + .next() + .with_context(|| format!("{} is empty (no CSV header)", path.display()))? + .with_context(|| format!("reading header of {}", path.display()))?; + let header = parity::split_csv_line(&header_line); + let wanted: Vec = if columns.is_empty() { + header.clone() + } else { + columns.to_vec() + }; + let indices = parity::header_indices(&header, &wanted)?; + + let mut keys: Vec = Vec::new(); + for line_result in lines { + let line = line_result.with_context(|| format!("reading {}", path.display()))?; + if line.is_empty() { + continue; + } + let fields = parity::split_csv_line(&line); + keys.push(parity::row_key(&fields, &indices)); + } + Ok((wanted, keys)) +} + +/// Render a key (control-separated components) for display. +fn show_key(key: &str) -> String { + key.replace('\u{1}', " | ") +} + +/// Print the parity report; returns whether the sides matched. +fn print_report(left: &Path, right: &Path, wanted: &[String], report: &ParityReport) { + println!("═══ UFFS verify — MFT parity ═══"); + println!(" A: {} ({} rows)", left.display(), report.a_rows); + println!(" B: {} ({} rows)", right.display(), report.b_rows); + println!(" Columns compared: {}", wanted.join(", ")); + println!(" Common rows: {}", report.common); + println!(" Only in A: {}", report.only_a_total); + println!(" Only in B: {}", report.only_b_total); + + for (label, sample) in [("A", &report.only_a), ("B", &report.only_b)] { + for key in sample { + println!(" only {label}: {}", show_key(key)); + } + } + + if report.matched() { + println!(" ✅ MATCH — the two exports are identical over the compared columns."); + } else { + println!(" ❌ MISMATCH — see divergent rows above."); + } +} + +/// Compare two MFT CSV exports and report parity. +/// +/// # Errors +/// +/// Returns an error if either file cannot be read, a requested column is +/// missing, or the two exports diverge (so the process exits non-zero). +pub(crate) fn cmd_verify(left: &Path, right: &Path, columns: &[String]) -> Result<()> { + // The first file establishes the comparison columns when none were given. + let (wanted, left_keys) = stream_keys(left, columns)?; + let (_, right_keys) = stream_keys(right, &wanted)?; + let report = parity::compare_keys(left_keys, right_keys); + print_report(left, right, &wanted, &report); + if report.matched() { + Ok(()) + } else { + anyhow::bail!( + "parity mismatch: {} row(s) only in A, {} only in B", + report.only_a_total, + report.only_b_total + ) + } +} diff --git a/crates/uffs-mft/src/lib.rs b/crates/uffs-mft/src/lib.rs index 0a8c649e8..8c55720fa 100644 --- a/crates/uffs-mft/src/lib.rs +++ b/crates/uffs-mft/src/lib.rs @@ -254,6 +254,9 @@ pub mod cache; /// Minimal `ustar` archive writer + byte splitter for capture bundles. pub mod archive; +/// Canonical CSV parity comparison for the capture verification flow. +pub mod parity; + mod reader; // WI-7.1 — pathological-name parity corpus (Tier 1 decoder pins + Tier 2 diff --git a/crates/uffs-mft/src/parity.rs b/crates/uffs-mft/src/parity.rs new file mode 100644 index 000000000..ceba4f7a7 --- /dev/null +++ b/crates/uffs-mft/src/parity.rs @@ -0,0 +1,210 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) 2025-2026 SKY, LLC. + +//! Canonical CSV parity comparison for the capture verification flow. +//! +//! Compares two MFT CSV exports (from `uffs-mft load`) — e.g. Rust-on-Windows +//! vs Rust-on-macOS, or Rust vs a C++ golden — by projecting each row to a set +//! of key columns matched by header *name* (so column order may differ), +//! canonically sorting, and diffing. Pure and cross-platform; the command layer +//! streams the files into `compare_keys`. + +use alloc::collections::BTreeMap; + +use crate::error::{MftError, Result}; + +/// Field separator for a composed row key — a control byte that cannot appear +/// unquoted in CSV data. +const KEY_SEP: char = '\u{1}'; + +/// Maximum number of divergent rows retained as samples per side. +pub const SAMPLE_MAX: usize = 20; + +/// Outcome of comparing two canonicalized CSV row sets. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ParityReport { + /// Data-row count of the `a` side. + pub a_rows: usize, + /// Data-row count of the `b` side. + pub b_rows: usize, + /// Rows present (matched) on both sides. + pub common: usize, + /// Sample of keys present only on the `a` side (capped at [`SAMPLE_MAX`]). + pub only_a: Vec, + /// Sample of keys present only on the `b` side (capped at [`SAMPLE_MAX`]). + pub only_b: Vec, + /// Total keys present only on the `a` side. + pub only_a_total: usize, + /// Total keys present only on the `b` side. + pub only_b_total: usize, +} + +impl ParityReport { + /// True when the two sides are identical (no divergent rows). + #[must_use] + pub const fn matched(&self) -> bool { + self.only_a_total == 0 && self.only_b_total == 0 + } +} + +/// Split one CSV line into fields per RFC 4180. +/// +/// Double-quote quoting with `""` escaping a literal quote. Embedded newlines +/// are not supported (each record is assumed to be one physical line, as +/// `uffs-mft load` emits). +#[must_use] +pub fn split_csv_line(line: &str) -> Vec { + let mut fields: Vec = Vec::new(); + let mut field = String::new(); + let mut in_quotes = false; + let mut chars = line.chars().peekable(); + while let Some(ch) = chars.next() { + match ch { + '"' if in_quotes => { + if chars.peek() == Some(&'"') { + field.push('"'); + chars.next(); + } else { + in_quotes = false; + } + } + '"' => in_quotes = true, + ',' if !in_quotes => fields.push(core::mem::take(&mut field)), + other => field.push(other), + } + } + fields.push(field); + fields +} + +/// Resolve the column indices of `wanted` names within `header`. An empty +/// `wanted` selects every header column, in order. +/// +/// # Errors +/// +/// Returns [`MftError::InvalidData`] if a wanted column is absent from +/// `header`. +pub fn header_indices(header: &[String], wanted: &[String]) -> Result> { + if wanted.is_empty() { + return Ok((0..header.len()).collect()); + } + wanted + .iter() + .map(|name| { + header.iter().position(|col| col == name).ok_or_else(|| { + MftError::InvalidData(format!("column {name:?} not found in CSV header")) + }) + }) + .collect() +} + +/// Compose a canonical key from the selected field indices (a missing field +/// contributes an empty component). +#[must_use] +pub fn row_key(fields: &[String], indices: &[usize]) -> String { + let mut key = String::new(); + for (position, &idx) in indices.iter().enumerate() { + if position > 0 { + key.push(KEY_SEP); + } + key.push_str(fields.get(idx).map_or("", String::as_str)); + } + key +} + +/// Compare two collections of row keys, returning a parity report. Duplicate +/// keys are treated as a multiset, so cardinality differences are reported too. +#[must_use] +pub fn compare_keys(left: Vec, right: Vec) -> ParityReport { + let (a_rows, b_rows) = (left.len(), right.len()); + let mut counts: BTreeMap = BTreeMap::new(); + for key in left { + counts.entry(key).or_default().0 += 1; + } + for key in right { + counts.entry(key).or_default().1 += 1; + } + + let mut report = ParityReport { + a_rows, + b_rows, + common: 0, + only_a: Vec::new(), + only_b: Vec::new(), + only_a_total: 0, + only_b_total: 0, + }; + for (key, (count_a, count_b)) in counts { + report.common += usize::try_from(count_a.min(count_b)).unwrap_or(0); + if count_a > count_b { + report.only_a_total += usize::try_from(count_a - count_b).unwrap_or(0); + if report.only_a.len() < SAMPLE_MAX { + report.only_a.push(key); + } + } else if count_b > count_a { + report.only_b_total += usize::try_from(count_b - count_a).unwrap_or(0); + if report.only_b.len() < SAMPLE_MAX { + report.only_b.push(key); + } + } + } + report +} + +#[cfg(test)] +mod tests { + use super::{compare_keys, header_indices, row_key, split_csv_line}; + + #[test] + fn split_handles_quoted_commas_and_escapes() { + assert_eq!(split_csv_line("1,a,10"), vec!["1", "a", "10"]); + // Quoted field containing a comma stays one field. + assert_eq!(split_csv_line("1,\"a,b\",10"), vec!["1", "a,b", "10"]); + // Doubled quote → one literal quote inside a quoted field. + assert_eq!(split_csv_line("\"a\"\"b\""), vec!["a\"b"]); + } + + #[test] + fn header_indices_match_by_name() { + let header: Vec = ["frs", "name", "size"] + .iter() + .map(|item| (*item).to_owned()) + .collect(); + let wanted: Vec = ["size", "frs"] + .iter() + .map(|item| (*item).to_owned()) + .collect(); + assert_eq!(header_indices(&header, &wanted).expect("found"), vec![2, 0]); + // Empty selection → all columns in order. + assert_eq!(header_indices(&header, &[]).expect("all"), vec![0, 1, 2]); + // Missing column → error. + let missing: Vec = vec!["nope".to_owned()]; + header_indices(&header, &missing).unwrap_err(); + } + + #[test] + fn row_key_projects_selected_columns() { + let fields: Vec = ["7", "a.txt", "10"] + .iter() + .map(|item| (*item).to_owned()) + .collect(); + // Same key regardless of source column order (indices carry the order). + assert_eq!(row_key(&fields, &[0, 2]), row_key(&fields, &[0, 2])); + assert_ne!(row_key(&fields, &[0, 2]), row_key(&fields, &[0, 1])); + } + + #[test] + fn compare_keys_reports_divergence() { + let left = vec!["x".to_owned(), "y".to_owned(), "y".to_owned()]; + let right = vec!["x".to_owned(), "z".to_owned()]; + let report = compare_keys(left, right); + assert_eq!(report.common, 1); // one "x" + assert_eq!(report.only_a_total, 2); // two "y" + assert_eq!(report.only_b_total, 1); // one "z" + assert!(!report.matched()); + + // Identical multisets match. + let same = vec!["a".to_owned(), "a".to_owned()]; + assert!(compare_keys(same.clone(), same).matched()); + } +} From 70d043a0e8abb30130dcdc42011d03c9ffe2e434 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Mon, 6 Jul 2026 11:03:19 -0700 Subject: [PATCH 26/39] =?UTF-8?q?docs(uffs-mft):=20add=20MFT=20capture=20?= =?UTF-8?q?=E2=86=92=20transfer=20=E2=86=92=20verify=20runbook?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit End-to-end runbook for the capture flow: capture (single / --all-drives / --zip --split-gib), transfer + SHA256SUMS verification, offline metafile-info inspection, and the three-way parity check (CPP golden vs Rust-on-Windows vs Rust-on-Mac) via load → verify. Plus a CLI cheat-sheet and the sparse-$UsnJrnl / CSV-quoting caveats. Co-Authored-By: Claude Opus 4.8 --- .../user-manual/mft-capture-verify-runbook.md | 144 ++++++++++++++++++ 1 file changed, 144 insertions(+) create mode 100644 docs/user-manual/mft-capture-verify-runbook.md diff --git a/docs/user-manual/mft-capture-verify-runbook.md b/docs/user-manual/mft-capture-verify-runbook.md new file mode 100644 index 000000000..9bfaa459d --- /dev/null +++ b/docs/user-manual/mft-capture-verify-runbook.md @@ -0,0 +1,144 @@ + + + +# MFT Capture → Transfer → Verify Runbook + +End-to-end flow for capturing a live Windows NTFS volume, moving the bundle to a +Mac, and proving the offline reconstruction is faithful — including a three-way +parity check between the C++ golden tool, `uffs-mft` on Windows, and `uffs-mft` +on the Mac. + +> Design/internals: `docs/architecture/mft-full-capture.md`. +> All `uffs-mft` MFT reads require **Windows, elevated (Administrator)**. The +> offline steps (`metafile-info`, `load`, `verify`) run anywhere. + +--- + +## What "tying down the flow" means + +Three outputs of the same volume's `$MFT`, compared pairwise: + +| Output | Where it comes from | +|--------|---------------------| +| **CPP golden** | the C++ tool, run live on Windows | +| **Rust / Windows** | `uffs-mft load` of the captured `$MFT`, on Windows | +| **Rust / Mac** | `uffs-mft load` of the *same* captured `$MFT`, on the Mac | + +- **Rust-Windows == Rust-Mac** proves the capture + offline reconstruction is + byte-faithful and the parser is platform-independent. +- **Rust ≈ CPP golden** proves the Rust parse matches the reference. + +The `verify` command performs each comparison and exits non-zero on any +divergence. + +--- + +## Step 1 — Capture (Windows, elevated) + +One drive (full `$MFT` + all 10 metafiles + `manifest.json` + `SHA256SUMS`): + +```powershell +uffs-mft capture --drive C --out cap +``` + +Every eligible NTFS volume, each into its own `cap\drive_\`: + +```powershell +uffs-mft capture --all-drives --out cap +``` + +Package each drive bundle into a single transfer artifact (extract on the Mac +with `tar --zstd -xf`), optionally split into ≤N-GiB parts: + +```powershell +uffs-mft capture --all-drives --out cap --zip # cap\drive_c.tar.zst, ... +uffs-mft capture --drive C --out cap --zip --split-gib 1 # cap\drive_c.tar.zst.000, .001, ... +``` + +Each `cap\drive_\` contains: + +| File | Contents | +|------|----------| +| `c_mft.compressed.bin` | full `$MFT` (zstd) | +| `c_boot.bin` … `c_usnjrnl.bin` | the 10 NTFS metafiles | +| `manifest.json` | volume facts + per-artifact SHA-256 | +| `SHA256SUMS` | transfer-verification hashes | + +## Step 2 — Transfer to the Mac + +USB is blocked by DLP on the managed Mac — use Google Drive / SMB. Copy the +whole `drive_\` folder (or the `.tar.zst[.NNN]`). + +On the Mac, reassemble (if split) and verify integrity **before** trusting the +data: + +```bash +cat drive_c.tar.zst.* > drive_c.tar.zst # only if --split-gib was used +tar --zstd -xf drive_c.tar.zst # only if --zip was used +cd drive_c +shasum -c SHA256SUMS # every line must say "OK" +``` + +## Step 3 — Inspect the metafiles offline (Mac) + +```bash +uffs-mft metafile-info --input c_boot.bin # geometry (cluster size, MFT LCN, ...) +uffs-mft metafile-info --input c_bitmap.bin # total/used/free clusters +uffs-mft metafile-info --input c_secure.bin # $Secure:$SDS payload present +uffs-mft metafile-info --input c_usnjrnl.bin # USN record count + sample entries +``` + +## Step 4 — Three-way parity + +Export each source to CSV, then `verify`. The Rust CSV schema is identical on +both platforms, so a full-column compare is exact. + +```powershell +# Windows: parse the captured $MFT → CSV (Rust / Windows) +uffs-mft load cap\drive_c\c_mft.compressed.bin -o rust_win_c.csv +``` + +```bash +# Mac: parse the SAME captured file → CSV (Rust / Mac) +uffs-mft load c_mft.compressed.bin -o rust_mac_c.csv + +# Rust-Windows vs Rust-Mac — expect ✅ MATCH (exit 0) +uffs-mft verify --left rust_win_c.csv --right rust_mac_c.csv +``` + +Compare against the **CPP golden**. Because the C++ tool may emit different +column names/order/formatting, restrict the compare to the shared identity +columns (matched by header name, so order does not matter): + +```bash +uffs-mft verify --left rust_mac_c.csv --right cpp_c.csv \ + --columns frs,parent_frs,name,size +``` + +`verify` reports per-side row counts, common rows, rows only on each side (with +a sample), and a final ✅/❌; it exits non-zero on mismatch so you can gate a +script on it. + +--- + +## CLI cheat-sheet + +| Command | Purpose | Platform | +|---------|---------|----------| +| `capture --drive C --out DIR` | bundle one drive | Windows (elevated) | +| `capture --all-drives --out DIR` | bundle every NTFS volume | Windows (elevated) | +| `capture … --zip [--split-gib N]` | pack `.tar.zst` (+split) | Windows (elevated) | +| `metafile-info --input FILE` | decode one metafile | any | +| `load FILE -o out.csv` | parse `$MFT` → CSV | any | +| `verify --left A --right B [--columns …]` | CSV parity, exits non-zero on mismatch | any | + +## Notes & limits + +- `$UsnJrnl:$J` is captured **sparse-compacted** — only the live (allocated) + journal is stored, not the multi-GB purged hole. Records are self-describing + by USN, so nothing is lost for change-journal analysis. +- `verify`'s CSV parser handles RFC 4180 quoting (commas/quotes in file names) + but assumes one record per physical line. +- For a CPP-vs-Rust compare, pick columns both tools emit with the same + semantics; timestamp/format differences on non-identity columns will show as + false mismatches otherwise. From 942078a59b9ab5cc0361a7e1e1ca3d3a7362bb3e Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Mon, 6 Jul 2026 12:40:19 -0700 Subject: [PATCH 27/39] fix(uffs-mft): name captured $MFT _mft.bin for verify-flow compat MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit capture wrote c_mft.compressed.bin, which verify_parity.rs --regenerate (and test_runs.ps1) don't recognize — they look for _mft.bin / .iocp. Rename the captured MFT to _mft.bin so new captures drop straight into the existing offline parity harness alongside old ones. Still zstd-compressed on disk; load_raw_mft auto-detects the flag and decompresses, so the .bin name is correct. Runbook updated to match. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/commands/windows/capture.rs | 14 ++++++++++---- docs/user-manual/mft-capture-verify-runbook.md | 6 +++--- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/crates/uffs-mft/src/commands/windows/capture.rs b/crates/uffs-mft/src/commands/windows/capture.rs index 97b8ee02a..0b1ce56c9 100644 --- a/crates/uffs-mft/src/commands/windows/capture.rs +++ b/crates/uffs-mft/src/commands/windows/capture.rs @@ -159,11 +159,17 @@ const METAFILE_KINDS: [MetafileKind; 9] = [ MetafileKind::UsnJrnl, ]; -/// Capture the compressed `$MFT` (LZ4) into the bundle. -fn capture_mft(drive: DriveLetter, dir: &Path, drive_lower: &str) -> Result { +/// Capture the compressed `$MFT` into the bundle. +/// +/// Named `_mft.bin` to match the verify flow (`test_runs.ps1` / +/// `verify_parity.rs --regenerate`), so old and new captures are +/// interchangeable. It is zstd-compressed on disk; `load_raw_mft` auto-detects +/// the compression flag and decompresses on read, so the `.bin` name is correct +/// regardless. +fn capture_mft(drive: DriveLetter, dir: &Path) -> Result { use uffs_mft::{MftReader, SaveRawOptions}; - let file = format!("{drive_lower}_mft.compressed.bin"); + let file = format!("{drive}_mft.bin"); let path = dir.join(&file); let reader = MftReader::open(drive).with_context(|| format!("opening $MFT on {drive}:"))?; let options = SaveRawOptions { @@ -196,7 +202,7 @@ fn collect_artifacts( ) -> Vec { let mut artifacts = Vec::new(); - match capture_mft(drive, dir, drive_lower) { + match capture_mft(drive, dir) { Ok(record) => { println!( " ✅ {:<9} {:>12} bytes {}", diff --git a/docs/user-manual/mft-capture-verify-runbook.md b/docs/user-manual/mft-capture-verify-runbook.md index 9bfaa459d..532d9ac93 100644 --- a/docs/user-manual/mft-capture-verify-runbook.md +++ b/docs/user-manual/mft-capture-verify-runbook.md @@ -59,7 +59,7 @@ Each `cap\drive_\` contains: | File | Contents | |------|----------| -| `c_mft.compressed.bin` | full `$MFT` (zstd) | +| `C_mft.bin` | full `$MFT` (zstd) | | `c_boot.bin` … `c_usnjrnl.bin` | the 10 NTFS metafiles | | `manifest.json` | volume facts + per-artifact SHA-256 | | `SHA256SUMS` | transfer-verification hashes | @@ -95,12 +95,12 @@ both platforms, so a full-column compare is exact. ```powershell # Windows: parse the captured $MFT → CSV (Rust / Windows) -uffs-mft load cap\drive_c\c_mft.compressed.bin -o rust_win_c.csv +uffs-mft load cap\drive_c\C_mft.bin -o rust_win_c.csv ``` ```bash # Mac: parse the SAME captured file → CSV (Rust / Mac) -uffs-mft load c_mft.compressed.bin -o rust_mac_c.csv +uffs-mft load C_mft.bin -o rust_mac_c.csv # Rust-Windows vs Rust-Mac — expect ✅ MATCH (exit 0) uffs-mft verify --left rust_win_c.csv --right rust_mac_c.csv From 1e045d2caf96f224c940d9ddf70422ed5da7f4f4 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Mon, 6 Jul 2026 12:51:41 -0700 Subject: [PATCH 28/39] feat(scripts): add verify_parity --live --bundle to capture + zip transport After each live per-drive compare, --bundle captures the MFT + metafiles (uffs-mft capture) into the same drive dir as the cpp/rust outputs, then zips it (PowerShell Compress-Archive) into drive_.zip. The zip carries the MFT, all metafiles, the manifest, and both tools' outputs, ready for offline --regenerate on the Mac. uffs-mft is auto-detected (~/bin or target/release) or set via --mft-bin. Bundling implies --keep. Co-Authored-By: Claude Opus 4.8 --- scripts/verify_parity.rs | 120 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 119 insertions(+), 1 deletion(-) diff --git a/scripts/verify_parity.rs b/scripts/verify_parity.rs index be22e58a0..7439e5749 100644 --- a/scripts/verify_parity.rs +++ b/scripts/verify_parity.rs @@ -21,6 +21,11 @@ //! rust-script scripts/verify_parity.rs --live //! rust-script scripts/verify_parity.rs --live --drive C --keep //! rust-script scripts/verify_parity.rs --live --drive C,D,F --out-dir D:\parity +//! +//! # Live + transport bundle: also capture the MFT + metafiles and zip each +//! # drive dir (MFT + metafiles + manifest + cpp/rust outputs) → drive_.zip, +//! # for offline `--regenerate` on the Mac. Needs uffs-mft (--mft-bin or auto). +//! rust-script scripts/verify_parity.rs --live --drive C --bundle //! ``` //! //! # Parity contract @@ -467,6 +472,20 @@ fn run_live_mode(args: &[String]) { let keep_files = args.iter().any(|a| a == "--keep"); let name_only = args.iter().any(|a| a == "--name-only"); + // --bundle: after each live compare, capture the MFT + metafiles and zip + // the drive dir (MFT + metafiles + manifest + cpp/rust outputs) for offline + // transport. Needs the uffs-mft binary (--mft-bin or auto-detected). + let bundle = args.iter().any(|a| a == "--bundle"); + let mft_bin: Option = parse_live_arg(args, "--mft-bin") + .map(PathBuf::from) + .or_else(find_uffs_mft_bin); + if bundle { + match mft_bin { + Some(ref p) => println!(" Bundle: ON — uffs-mft: {}", p.display()), + None => println!(" Bundle: ON — ⚠️ uffs-mft not found (use --mft-bin )"), + } + } + // Determine drives let drives: Vec = if let Some(d) = parse_drive_filter(args) { vec![d.to_uppercase()] @@ -516,6 +535,8 @@ fn run_live_mode(args: &[String]) { es_bin.as_deref(), &out_dir, keep_files, + bundle, + mft_bin.as_deref(), pipeline.as_deref(), i + 1, drives.len(), @@ -592,6 +613,8 @@ fn run_live_drive_parity( es_bin: Option<&Path>, out_dir: &Path, keep_files: bool, + bundle: bool, + mft_bin: Option<&Path>, pipeline: Option<&str>, drive_index: usize, total_drives: usize, @@ -884,7 +907,13 @@ fn run_live_drive_parity( // compare_with_everything(&es_raw_path, &rust_raw, &drive_upper); // } - cleanup_live_files(keep_files, &[&cpp_raw, &rust_raw, &es_raw_path]); + // Build the offline transport bundle (capture MFT + metafiles into the same + // drive dir, then zip it with the cpp/rust outputs). Bundling implies keep. + if bundle { + create_transport_bundle(&drive_upper, &drive_lower, out_dir, &drive_dir, mft_bin); + } + + cleanup_live_files(keep_files || bundle, &[&cpp_raw, &rust_raw, &es_raw_path]); LiveDriveResult { drive_letter: drive_upper, @@ -897,6 +926,95 @@ fn run_live_drive_parity( } } +/// Auto-detect the `uffs-mft` binary (MFT capture tool). +/// +/// Looks in `$USERPROFILE\bin` / `$HOME/bin` first (the standard deploy +/// location), then the workspace `target/release`. +fn find_uffs_mft_bin() -> Option { + let name = if cfg!(windows) { "uffs-mft.exe" } else { "uffs-mft" }; + for var in ["USERPROFILE", "HOME"] { + if let Ok(home) = env::var(var) { + let candidate = PathBuf::from(home).join("bin").join(name); + if candidate.exists() { + return Some(candidate); + } + } + } + let workspace = find_workspace_root().join("target").join("release").join(name); + workspace.exists().then_some(workspace) +} + +/// Capture the MFT + metafiles into the drive dir and zip it for transport. +/// +/// `uffs-mft capture --out ` writes into `/drive_/` — the +/// same dir the live cpp/rust outputs already live in — so the resulting +/// `drive_.zip` carries the MFT, all metafiles, the manifest, and both +/// tools' outputs. On the Mac: unzip, then `verify_parity.rs --regenerate`. +fn create_transport_bundle( + drive_upper: &str, + drive_lower: &str, + out_dir: &Path, + drive_dir: &Path, + mft_bin: Option<&Path>, +) { + println!(); + println!(" [bundle] Building transport bundle for drive {drive_upper}..."); + + let Some(mft_bin) = mft_bin else { + println!(" [bundle] ⚠️ skipped: uffs-mft binary not found (pass --mft-bin )"); + return; + }; + + // 1. Capture MFT + metafiles alongside the cpp/rust outputs. + print!(" [bundle] uffs-mft capture --drive {drive_upper}..."); + io::stdout().flush().ok(); + let capture = Command::new(mft_bin) + .args([ + "capture", + "--drive", + drive_upper, + "--out", + &out_dir.to_string_lossy(), + ]) + .stdout(std::process::Stdio::null()) + .stderr(std::process::Stdio::piped()) + .output(); + match capture { + Ok(out) if out.status.success() => println!(" ✅"), + Ok(out) => { + println!( + " ⚠️ exited {} — bundle may be incomplete: {}", + out.status.code().unwrap_or(-1), + String::from_utf8_lossy(&out.stderr).trim() + ); + } + Err(e) => { + println!(" ❌ failed to run: {e}"); + return; + } + } + + // 2. Zip the drive dir with PowerShell's built-in Compress-Archive. + let zip_path = out_dir.join(format!("drive_{drive_lower}.zip")); + print!(" [bundle] Compress-Archive → {}...", zip_path.display()); + io::stdout().flush().ok(); + let ps_command = format!( + "Compress-Archive -Path '{}\\*' -DestinationPath '{}' -Force", + drive_dir.display(), + zip_path.display() + ); + let zip = Command::new("powershell") + .args(["-NoProfile", "-NonInteractive", "-Command", &ps_command]) + .stdout(std::process::Stdio::null()) + .stderr(std::process::Stdio::piped()) + .status(); + match zip { + Ok(status) if status.success() => println!(" ✅"), + Ok(status) => println!(" ⚠️ Compress-Archive exited {:?}", status.code()), + Err(e) => println!(" ❌ failed to run: {e}"), + } +} + /// Auto-detect es.exe (Everything CLI) on the system. fn find_es_exe() -> Option { // Check PATH first From d1834da78577f3da5593201415ddbad917813819 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Mon, 6 Jul 2026 15:08:27 -0700 Subject: [PATCH 29/39] fix(scripts): verify_parity --bundle uses tar.zst, not Compress-Archive PowerShell Compress-Archive silently fails past ~2 GB, so on large drives (e.g. S:, whose _mft.bin exceeds 2 GB) the bundle reported success but wrote no zip. Replace the capture + Compress-Archive two-step with a single `uffs-mft capture --zip` (ustar + zstd, no size limit), which packs the whole drive dir into drive_.tar.zst, and verify the archive actually exists (never trust the exit code alone). Co-Authored-By: Claude Opus 4.8 --- scripts/verify_parity.rs | 69 +++++++++++++++++++--------------------- 1 file changed, 33 insertions(+), 36 deletions(-) diff --git a/scripts/verify_parity.rs b/scripts/verify_parity.rs index 7439e5749..2c4a6c942 100644 --- a/scripts/verify_parity.rs +++ b/scripts/verify_parity.rs @@ -22,9 +22,10 @@ //! rust-script scripts/verify_parity.rs --live --drive C --keep //! rust-script scripts/verify_parity.rs --live --drive C,D,F --out-dir D:\parity //! -//! # Live + transport bundle: also capture the MFT + metafiles and zip each -//! # drive dir (MFT + metafiles + manifest + cpp/rust outputs) → drive_.zip, +//! # Live + transport bundle: also capture the MFT + metafiles and pack each +//! # drive dir (MFT + metafiles + manifest + cpp/rust outputs) → drive_.tar.zst //! # for offline `--regenerate` on the Mac. Needs uffs-mft (--mft-bin or auto). +//! # (ustar+zstd, no PowerShell 2 GB Compress-Archive limit.) //! rust-script scripts/verify_parity.rs --live --drive C --bundle //! ``` //! @@ -910,7 +911,7 @@ fn run_live_drive_parity( // Build the offline transport bundle (capture MFT + metafiles into the same // drive dir, then zip it with the cpp/rust outputs). Bundling implies keep. if bundle { - create_transport_bundle(&drive_upper, &drive_lower, out_dir, &drive_dir, mft_bin); + create_transport_bundle(&drive_upper, &drive_lower, out_dir, mft_bin); } cleanup_live_files(keep_files || bundle, &[&cpp_raw, &rust_raw, &es_raw_path]); @@ -944,17 +945,19 @@ fn find_uffs_mft_bin() -> Option { workspace.exists().then_some(workspace) } -/// Capture the MFT + metafiles into the drive dir and zip it for transport. +/// Capture the MFT + metafiles into the drive dir and pack it for transport. /// -/// `uffs-mft capture --out ` writes into `/drive_/` — the -/// same dir the live cpp/rust outputs already live in — so the resulting -/// `drive_.zip` carries the MFT, all metafiles, the manifest, and both -/// tools' outputs. On the Mac: unzip, then `verify_parity.rs --regenerate`. +/// A single `uffs-mft capture --out --zip` writes into +/// `/drive_/` — the same dir the live cpp/rust outputs already live +/// in — then packs the whole dir into `/drive_.tar.zst`. That +/// bundler is ustar + zstd, so unlike PowerShell `Compress-Archive` (which +/// silently fails past ~2 GB — e.g. a large drive's `_mft.bin`) it has no +/// size limit. On the Mac: `tar --zstd -xf drive_.tar.zst`, then +/// `verify_parity.rs --regenerate`. fn create_transport_bundle( drive_upper: &str, drive_lower: &str, out_dir: &Path, - drive_dir: &Path, mft_bin: Option<&Path>, ) { println!(); @@ -965,8 +968,9 @@ fn create_transport_bundle( return; }; - // 1. Capture MFT + metafiles alongside the cpp/rust outputs. - print!(" [bundle] uffs-mft capture --drive {drive_upper}..."); + // Capture MFT + metafiles alongside the cpp/rust outputs, then --zip packs + // the whole drive dir into drive_.tar.zst. + print!(" [bundle] uffs-mft capture --drive {drive_upper} --zip..."); io::stdout().flush().ok(); let capture = Command::new(mft_bin) .args([ @@ -975,43 +979,36 @@ fn create_transport_bundle( drive_upper, "--out", &out_dir.to_string_lossy(), + "--zip", ]) .stdout(std::process::Stdio::null()) .stderr(std::process::Stdio::piped()) .output(); match capture { Ok(out) if out.status.success() => println!(" ✅"), - Ok(out) => { - println!( - " ⚠️ exited {} — bundle may be incomplete: {}", - out.status.code().unwrap_or(-1), - String::from_utf8_lossy(&out.stderr).trim() - ); - } + Ok(out) => println!( + " ⚠️ exited {}: {}", + out.status.code().unwrap_or(-1), + String::from_utf8_lossy(&out.stderr).trim() + ), Err(e) => { println!(" ❌ failed to run: {e}"); return; } } - // 2. Zip the drive dir with PowerShell's built-in Compress-Archive. - let zip_path = out_dir.join(format!("drive_{drive_lower}.zip")); - print!(" [bundle] Compress-Archive → {}...", zip_path.display()); - io::stdout().flush().ok(); - let ps_command = format!( - "Compress-Archive -Path '{}\\*' -DestinationPath '{}' -Force", - drive_dir.display(), - zip_path.display() - ); - let zip = Command::new("powershell") - .args(["-NoProfile", "-NonInteractive", "-Command", &ps_command]) - .stdout(std::process::Stdio::null()) - .stderr(std::process::Stdio::piped()) - .status(); - match zip { - Ok(status) if status.success() => println!(" ✅"), - Ok(status) => println!(" ⚠️ Compress-Archive exited {:?}", status.code()), - Err(e) => println!(" ❌ failed to run: {e}"), + // Verify the archive actually landed (never trust exit code alone). + let archive = out_dir.join(format!("drive_{drive_lower}.tar.zst")); + match fs::metadata(&archive) { + Ok(meta) if meta.len() > 0 => { + #[allow(clippy::cast_precision_loss)] // display only + let mb = meta.len() as f64 / (1024.0 * 1024.0); + println!(" [bundle] ✅ {} ({mb:.1} MB)", archive.display()); + } + _ => println!( + " [bundle] ❌ archive not produced: {}", + archive.display() + ), } } From 151bd8077d3a706f597204fbf8d6175d2f8ce121 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Mon, 6 Jul 2026 15:21:22 -0700 Subject: [PATCH 30/39] fix(uffs-mft): store real mtimes in capture --zip archives MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ustar writer hardcoded mtime=0, so extracting a bundle produced 1970-dated files — and the offline parity flow derives its --tz-offset from the baseline file's mtime, which that broke. push_entry now takes an mtime; capture reads each file's modified time so extraction restores it. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/archive.rs | 19 +++++++++++++------ .../uffs-mft/src/commands/windows/capture.rs | 7 ++++++- 2 files changed, 19 insertions(+), 7 deletions(-) diff --git a/crates/uffs-mft/src/archive.rs b/crates/uffs-mft/src/archive.rs index 08056a3f7..d70bd9094 100644 --- a/crates/uffs-mft/src/archive.rs +++ b/crates/uffs-mft/src/archive.rs @@ -32,13 +32,18 @@ fn push_field(buf: &mut Vec, content: &[u8], width: usize) { buf.resize(buf.len() + (width - take), 0); } -/// Append one regular-file entry (`name` → `data`) to a `ustar` archive `buf`. +/// Append one regular-file entry (`name` → `data`, modified at `mtime` Unix +/// seconds) to a `ustar` archive `buf`. +/// +/// A real `mtime` is stored so extraction restores it: the offline parity flow +/// derives its timezone from the baseline file's mtime, which a zeroed mtime +/// would break. /// /// # Errors /// /// Returns [`MftError::InvalidData`] if `name` exceeds the 100-byte `ustar` /// name field. -pub fn push_entry(buf: &mut Vec, name: &str, data: &[u8]) -> Result<()> { +pub fn push_entry(buf: &mut Vec, name: &str, data: &[u8], mtime: u64) -> Result<()> { if name.len() > NAME_MAX { return Err(MftError::InvalidData(format!( "tar entry name too long ({} > {NAME_MAX}): {name}", @@ -55,7 +60,7 @@ pub fn push_entry(buf: &mut Vec, name: &str, data: &[u8]) -> Result<()> { format!("{:011o}\0", usize_to_u64(data.len())).as_bytes(), 12, ); // size - push_field(buf, b"00000000000\0", 12); // mtime (0) + push_field(buf, format!("{mtime:011o}\0").as_bytes(), 12); // mtime push_field(buf, b" ", 8); // chksum placeholder (spaces) push_field(buf, b"0", 1); // typeflag: regular file push_field(buf, b"", NAME_MAX); // linkname @@ -115,13 +120,15 @@ mod tests { )] fn push_entry_writes_valid_ustar_header() { let mut buf = Vec::new(); - push_entry(&mut buf, "c_boot.bin", b"hello").expect("valid name"); + push_entry(&mut buf, "c_boot.bin", b"hello", 0o1234).expect("valid name"); finish(&mut buf); - // name, magic, size (octal of 5), and data land at their ustar offsets. + // name, magic, size (octal of 5), mtime (octal), and data land at their + // ustar offsets. assert_eq!(&buf[0..10], b"c_boot.bin"); assert_eq!(&buf[257..263], b"ustar\0"); assert_eq!(&buf[124..135], b"00000000005"); + assert_eq!(&buf[136..147], b"00000001234"); assert_eq!(&buf[512..517], b"hello"); // 512 header + 512 data block + 2×512 end blocks. @@ -147,7 +154,7 @@ mod tests { #[test] fn push_entry_rejects_overlong_name() { let mut buf = Vec::new(); - push_entry(&mut buf, &"a".repeat(101), b"x").unwrap_err(); + push_entry(&mut buf, &"a".repeat(101), b"x", 0).unwrap_err(); } #[test] diff --git a/crates/uffs-mft/src/commands/windows/capture.rs b/crates/uffs-mft/src/commands/windows/capture.rs index 0b1ce56c9..6a856c396 100644 --- a/crates/uffs-mft/src/commands/windows/capture.rs +++ b/crates/uffs-mft/src/commands/windows/capture.rs @@ -280,7 +280,12 @@ fn archive_dir(dir: &Path, split_gib: u64) -> Result<()> { .map(|os| os.to_string_lossy().into_owned()) .unwrap_or_default(); let data = std::fs::read(path).with_context(|| format!("reading {}", path.display()))?; - uffs_mft::archive::push_entry(&mut tar, &name, &data) + let mtime = std::fs::metadata(path) + .and_then(|meta| meta.modified()) + .ok() + .and_then(|time| time.duration_since(std::time::UNIX_EPOCH).ok()) + .map_or(0, |elapsed| elapsed.as_secs()); + uffs_mft::archive::push_entry(&mut tar, &name, &data, mtime) .with_context(|| format!("archiving {name}"))?; } uffs_mft::archive::finish(&mut tar); From 134a41d7807bc3c13558c035c26ac9780c231373 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Mon, 6 Jul 2026 15:27:02 -0700 Subject: [PATCH 31/39] feat(scripts): verify_parity expands drive_.tar.zst bundles offline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Offline mode now discovers drive_.tar.zst transport bundles in the base dir, temp-expands each (tar --zstd -xf, or plain tar -xf which libarchive auto-detects) into a drive_/ dir, validates it via the normal --regenerate path, and removes the expansion afterward — so a dropped-in bundle is validated without keeping an uncompressed copy. Skips expansion when a drive_/ dir already exists. Co-Authored-By: Claude Opus 4.8 --- scripts/verify_parity.rs | 97 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) diff --git a/scripts/verify_parity.rs b/scripts/verify_parity.rs index 2c4a6c942..d0263109b 100644 --- a/scripts/verify_parity.rs +++ b/scripts/verify_parity.rs @@ -17,6 +17,10 @@ //! rust-script scripts/verify_parity.rs ~/uffs_data --regenerate //! rust-script scripts/verify_parity.rs ~/uffs_data --drive G --regenerate //! +//! # Offline from bundles: drop drive_.tar.zst files in a dir; each is +//! # temp-expanded, validated, then removed (no uncompressed copies kept). +//! rust-script scripts/verify_parity.rs ~/bundles --regenerate +//! //! # Live: run both tools on Windows (elevated), auto-detect NTFS drives //! rust-script scripts/verify_parity.rs --live //! rust-script scripts/verify_parity.rs --live --drive C --keep @@ -293,12 +297,38 @@ fn run_multi_drive_mode(args: &[String], base_dir: &Path) { std::process::exit(1); } + // Expand any drive_.tar.zst transport bundles into temp drive dirs so a + // dropped-in archive can be validated without keeping the uncompressed copy + // around. Extracted dirs are removed again once verification finishes. + let mut temp_dirs: Vec = Vec::new(); + for (drive_lower, archive) in discover_archives(base_dir, specific_drive.as_deref()) { + let target = base_dir.join(format!("drive_{drive_lower}")); + if target.exists() { + println!( + "ℹ️ {} already exists — using it (not expanding {})", + target.display(), + archive.display() + ); + continue; + } + print!("Expanding {} → {} ...", archive.display(), target.display()); + io::stdout().flush().ok(); + match extract_archive(&archive, &target) { + Ok(()) => { + println!(" ✅"); + temp_dirs.push(target); + } + Err(e) => println!(" ❌ {e}"), + } + } + // Discover all drive directories let drives = discover_drives(base_dir, specific_drive.as_deref()); if drives.is_empty() { eprintln!("ERROR: No drive directories found in {}", base_dir.display()); eprintln!(" Expected directories like: drive_d, drive_e, drive_f, ..."); + eprintln!(" (or drive_.tar.zst bundles to expand)"); std::process::exit(1); } @@ -386,6 +416,13 @@ fn run_multi_drive_mode(args: &[String], base_dir: &Path) { // Print summary print_summary(&results); + // Dump the temp dirs we expanded from .tar.zst bundles. + for temp in &temp_dirs { + if fs::remove_dir_all(temp).is_ok() { + println!("🧹 removed expanded {}", temp.display()); + } + } + // Exit with failure if any drive mismatched let any_mismatch = results.iter().any(|r| r.result == VerifyResult::Mismatch); std::process::exit(i32::from(any_mismatch)); @@ -2023,6 +2060,66 @@ fn discover_drives(base_dir: &Path, filter: Option<&str>) -> Vec { drives } +/// Discover `drive_.tar.zst` transport bundles in `base_dir`, honoring the +/// optional single-drive filter. Returns `(drive_lower, archive_path)` pairs. +fn discover_archives(base_dir: &Path, filter: Option<&str>) -> Vec<(String, PathBuf)> { + let mut archives = Vec::new(); + + let Ok(entries) = fs::read_dir(base_dir) else { + return archives; + }; + + for entry in entries.flatten() { + let path = entry.path(); + let Some(name) = path.file_name().and_then(|n| n.to_str()) else { + continue; + }; + let Some(letter) = name + .strip_prefix("drive_") + .and_then(|rest| rest.strip_suffix(".tar.zst")) + else { + continue; + }; + if letter.len() != 1 || !letter.chars().all(|c| c.is_ascii_alphabetic()) { + continue; + } + if let Some(f) = filter { + if !letter.eq_ignore_ascii_case(f) { + continue; + } + } + archives.push((letter.to_lowercase(), path)); + } + + archives.sort(); + archives +} + +/// Extract a `.tar.zst` bundle into `target` (created if absent). +/// +/// Tries `tar --zstd -xf` first, then plain `tar -xf` (libarchive on macOS / +/// bsdtar auto-detects zstd on extraction). +fn extract_archive(archive: &Path, target: &Path) -> Result<(), String> { + fs::create_dir_all(target).map_err(|e| format!("mkdir {}: {e}", target.display()))?; + let archive_str = archive.to_string_lossy().to_string(); + let target_str = target.to_string_lossy().to_string(); + + let attempts: [Vec<&str>; 2] = [ + vec!["--zstd", "-xf", &archive_str, "-C", &target_str], + vec!["-xf", &archive_str, "-C", &target_str], + ]; + let mut last_err = String::from("tar not found"); + for args in &attempts { + match Command::new("tar").args(args).output() { + Ok(out) if out.status.success() => return Ok(()), + Ok(out) => last_err = String::from_utf8_lossy(&out.stderr).trim().to_string(), + Err(e) => last_err = e.to_string(), + } + } + let _ = fs::remove_dir_all(target); + Err(format!("tar extract failed: {last_err}")) +} + /// Verify a single drive and return the result fn verify_single_drive( base_dir: &Path, From 496fc71c8997350822806cbb83e07e1c6931c230 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Mon, 6 Jul 2026 15:50:35 -0700 Subject: [PATCH 32/39] fix(scripts): reset daemon before offline --regenerate (clean single-drive load) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Offline --regenerate ran `uffs --mft-file ` via autospawn, but a daemon left resident from an earlier drive/session served its stale in-memory index instead — so a --drive M run hit a daemon still holding C: (327ms "scan", no verify_rust_m.txt, then FATAL not-found). --no-cache only skips the cache file, not the hot daemon. Kill the daemon + purge caches (cross-platform) before invoking uffs, mirroring the live mode's cold-start, so only the requested drive's MFT is loaded. Co-Authored-By: Claude Opus 4.8 --- scripts/verify_parity.rs | 46 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/scripts/verify_parity.rs b/scripts/verify_parity.rs index d0263109b..4ef207f40 100644 --- a/scripts/verify_parity.rs +++ b/scripts/verify_parity.rs @@ -3023,6 +3023,44 @@ fn parse_bin_path(args: &[String]) -> Option { None } +/// Kill any running daemon and purge on-disk caches so the next `uffs` run +/// starts from a clean slate, loading only the drive we ask for. +/// +/// Offline `--regenerate` uses `uffs --mft-file` via autospawn; if a daemon +/// from a previous drive/session is still resident it serves that stale +/// in-memory index instead (`--no-cache` only skips the cache *file*, not the +/// hot daemon). We don't know which drive/version a running daemon holds, so we +/// reset it. +fn purge_daemon_and_cache(uffs_bin: &Path) { + let _ = Command::new(uffs_bin) + .args(["--daemon", "kill"]) + .stdout(std::process::Stdio::null()) + .stderr(std::process::Stdio::null()) + .output(); + std::thread::sleep(Duration::from_secs(1)); + + // Best-effort cache removal across platforms. + let mut cache_dirs: Vec = Vec::new(); + if let Ok(local) = env::var("LOCALAPPDATA") { + cache_dirs.push(PathBuf::from(local).join("uffs").join("cache")); // Windows + } + if let Ok(tmp) = env::var("TEMP") { + cache_dirs.push(PathBuf::from(tmp).join("uffs_index_cache")); // Windows legacy + } + if let Ok(home) = env::var("HOME") { + cache_dirs.push(PathBuf::from(&home).join("Library/Caches/uffs")); // macOS + cache_dirs.push(PathBuf::from(&home).join(".cache/uffs")); // Linux + } + if let Ok(xdg) = env::var("XDG_CACHE_HOME") { + cache_dirs.push(PathBuf::from(xdg).join("uffs")); + } + for dir in cache_dirs { + if dir.exists() { + let _ = fs::remove_dir_all(&dir); + } + } +} + fn regenerate_rust_output( data_dir: &Path, drive_letter: &str, @@ -3158,6 +3196,14 @@ fn regenerate_rust_output( // NOTE: --pipeline flag removed from uffs binary (Step 4). let _ = pipeline; println!("Pipeline: unified (only pipeline)"); + + // Clean slate: reset any daemon (it may hold a different drive/version from + // an earlier run) so uffs loads only this drive's MFT via autospawn. + print!("Resetting daemon + cache for a clean single-drive load..."); + io::stdout().flush().ok(); + purge_daemon_and_cache(&binary_path); + println!(" ✅"); + let status = Command::new(&binary_path) .args(&args) .status(); From d33155b069e593fc07dbf2d5c2840300691ee90a Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Mon, 6 Jul 2026 15:59:42 -0700 Subject: [PATCH 33/39] fix(scripts): tolerate dir size-on-disk skew in live parity reconciler classify_live_skew tolerated a directory's Size and Descendant-count aggregates as live-read skew but not Size-on-Disk (col 4), which is equally a subtree aggregate. So when Norton populated the S:\$AV_NLL quarantine vault between the cold Rust scan and the warm C++ scan, the vault dir's size/size-on-disk/descendants all moved; Size + Descendants were tolerated but Size-on-Disk tripped a false "real diff", leaving 2 entries that flipped S: from SUPERSET MATCH to MISMATCH. Tolerate Size-on-Disk for directories too. Co-Authored-By: Claude Opus 4.8 --- scripts/verify_parity.rs | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/scripts/verify_parity.rs b/scripts/verify_parity.rs index 4ef207f40..2b974c1be 100644 --- a/scripts/verify_parity.rs +++ b/scripts/verify_parity.rs @@ -3896,6 +3896,7 @@ fn verify_hardlinks_inline(golden_baseline_file: &Path, only_in_rust_data: &[Str /// 0=path 1=name 2=parent 3=size 4=allocated 5=created 6=modified /// 7=accessed 8=descendant-count … const COL_SIZE: usize = 3; +const COL_ALLOCATED: usize = 4; const COL_ACCESSED: usize = 7; const COL_COUNT: usize = 8; @@ -3967,8 +3968,10 @@ fn classify_live_skew(baseline: &str, rust: &str) -> Option<&'static str> { if i == COL_ACCESSED { continue; } - if is_dir && (i == COL_SIZE || i == COL_COUNT) { - accessed_only = false; // a subtree aggregate also moved + if is_dir && (i == COL_SIZE || i == COL_ALLOCATED || i == COL_COUNT) { + // Size, size-on-disk, and descendant count are all subtree + // aggregates that move when a child changes between the two reads. + accessed_only = false; continue; } return None; // a structural field differs — real diff From b12b4be9f52a775b615d9f59e649b2609ae4fe2c Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Mon, 6 Jul 2026 16:31:35 -0700 Subject: [PATCH 34/39] fix(uffs-mft): write reserved_allocated.txt sidecar in capture bundle MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Offline --regenerate under-counted the root's size-on-disk by the reserved-cluster bytes (total_reserved × cluster_size) because the sequential .bin MFT format doesn't persist reserved_allocated_bytes (only the IOCP format does), so it loads as 0. On drive M this was the single 52 MB (13,327-cluster) root diff. Capture now writes the value from volume data into reserved_allocated.txt, which verify_parity.rs --regenerate already consumes via --reserved-allocated so the root matches the live/C++ baseline. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/commands/windows/capture.rs | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/crates/uffs-mft/src/commands/windows/capture.rs b/crates/uffs-mft/src/commands/windows/capture.rs index 6a856c396..e9ee30375 100644 --- a/crates/uffs-mft/src/commands/windows/capture.rs +++ b/crates/uffs-mft/src/commands/windows/capture.rs @@ -254,9 +254,19 @@ fn capture_one_drive(drive: DriveLetter, out: &Path) -> Result Date: Mon, 6 Jul 2026 17:16:24 -0700 Subject: [PATCH 35/39] fix(uffs-cli): bound ps calls in --status so it can't hang MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --status shelled out to `ps` twice (process list + parent-name lookup) via Command::output() with no timeout, so a process with a huge argument vector (or one stuck in uninterruptible I/O) made `ps` block and hung the whole command. Route both through a shared capture_with_timeout helper (2s watchdog on a helper thread; kills the child and reports "process scan timed out — skipped" on timeout). Require the real `--mcp` flag when matching sessions so build tooling (cargo/sccache compiling uffs-mcp) is not listed. Split the MCP-stdio scan into a process_scan submodule to stay under the file-size gate. Co-Authored-By: Claude Opus 4.8 --- .../mod.rs} | 176 +------------- .../commands/system_status/process_scan.rs | 230 ++++++++++++++++++ 2 files changed, 232 insertions(+), 174 deletions(-) rename crates/uffs-cli/src/commands/{system_status.rs => system_status/mod.rs} (77%) create mode 100644 crates/uffs-cli/src/commands/system_status/process_scan.rs diff --git a/crates/uffs-cli/src/commands/system_status.rs b/crates/uffs-cli/src/commands/system_status/mod.rs similarity index 77% rename from crates/uffs-cli/src/commands/system_status.rs rename to crates/uffs-cli/src/commands/system_status/mod.rs index f84fabfc9..540af8f50 100644 --- a/crates/uffs-cli/src/commands/system_status.rs +++ b/crates/uffs-cli/src/commands/system_status/mod.rs @@ -541,180 +541,8 @@ fn mcp_http_json() -> serde_json::Value { }) } -// ── MCP Stdio Sessions ─────────────────────────────────────────────────────── - -/// Print the MCP stdio session list (running `uffs --mcp run` processes). -#[expect(clippy::print_stdout, reason = "CLI user-facing output")] -fn print_mcp_stdio_section(palette: Palette) { - println!("{}", section(palette, "MCP Stdio Sessions")); - let sessions = find_mcp_stdio_processes(); - if sessions.is_empty() { - println!(" {}", palette.dim("(none)")); - return; - } - let mut any_stale = false; - for (idx, session) in sessions.iter().enumerate() { - let parent = session - .parent_name - .as_deref() - .map_or(String::new(), |name| format!(" (parent: {name})")); - let glyph = if session.is_stale { - any_stale = true; - Glyph::Warn - } else { - Glyph::Up - }; - let name = format!("{}. PID {}", idx + 1, session.pid); - let detail = format!( - "uptime {}{parent}", - uffs_client::format::format_duration(session.uptime) - ); - println!("{}", status_row(palette, glyph, &name, &detail)); - } - if any_stale { - println!( - " {}", - palette - .dim("Older binary; the AI host that launched them refreshes on its next start.") - ); - } -} - -/// JSON for the MCP stdio session list. -fn mcp_stdio_json() -> serde_json::Value { - let sessions: Vec = find_mcp_stdio_processes() - .iter() - .map(|session| { - serde_json::json!({ - "pid": session.pid, - "uptime_secs": session.uptime.as_secs(), - "parent": session.parent_name, - "stale": session.is_stale, - }) - }) - .collect(); - serde_json::Value::Array(sessions) -} - -/// Information about a running MCP stdio process. -struct StdioSession { - /// Process ID. - pid: u32, - /// How long the process has been running. - uptime: core::time::Duration, - /// Name of the parent process (the AI host), if available. - parent_name: Option, - /// True if the process's binary is older than the current binary. - is_stale: bool, -} - -/// Find running `uffs --mcp run` processes via `ps`, flagging stale binaries. -fn find_mcp_stdio_processes() -> Vec { - let Ok(raw_output) = std::process::Command::new("ps") - .args(["-eo", "pid,ppid,etime,args"]) - .stdout(std::process::Stdio::piped()) - .stderr(std::process::Stdio::null()) - .output() - else { - return Vec::new(); - }; - // AUDIT-OK(bytes): per-line PID scan of process-list output; each line's - // PID parses-or-skips (fail-safe). Whole-buffer strict decode would drop - // the whole list on one bad byte. (WI-4.3 follow-up) - let text = String::from_utf8_lossy(&raw_output.stdout); - let my_pid = std::process::id(); - let current_mtime = std::env::current_exe() - .ok() - .and_then(|path| std::fs::metadata(path).ok()) - .and_then(|meta| meta.modified().ok()); - - let mut sessions = Vec::new(); - for line in text.lines().skip(1) { - if let Some(session) = parse_stdio_line(line, my_pid, current_mtime) { - sessions.push(session); - } - } - sessions -} - -/// Parse one `ps` line into a [`StdioSession`] if it is an MCP stdio process. -fn parse_stdio_line( - line: &str, - my_pid: u32, - current_mtime: Option, -) -> Option { - let mut fields = line.split_whitespace(); - let proc_pid = fields.next()?.parse::().ok()?; - if proc_pid == my_pid { - return None; - } - let parent_pid: u32 = fields.next().and_then(|val| val.parse().ok()).unwrap_or(0); - let elapsed_time = fields.next()?; - let cmdline: String = fields.collect::>().join(" "); - - if !cmdline.contains("mcp") || !cmdline.contains("run") { - return None; - } - if cmdline.contains("serve") || cmdline.contains("start") || cmdline.contains("kill") { - return None; - } - - let uptime = parse_ps_etime(elapsed_time); - let is_stale = current_mtime.is_some_and(|bin_mtime| { - let proc_started = std::time::SystemTime::now() - uptime; - proc_started < bin_mtime - }); - Some(StdioSession { - pid: proc_pid, - uptime, - parent_name: resolve_parent_name(parent_pid), - is_stale, - }) -} - -/// Parse `ps` elapsed time format: `[[dd-]hh:]mm:ss`. -fn parse_ps_etime(etime: &str) -> core::time::Duration { - let mut total_secs: u64 = 0; - let (days_part, time_part) = if let Some((days, rest)) = etime.split_once('-') { - (days.parse::().unwrap_or(0), rest) - } else { - (0, etime) - }; - total_secs += days_part * 86400; - let mut parts = time_part.rsplit(':'); - if let Some(sec) = parts.next() { - total_secs += sec.parse::().unwrap_or(0); - } - if let Some(min) = parts.next() { - total_secs += min.parse::().unwrap_or(0) * 60; - } - if let Some(hour) = parts.next() { - total_secs += hour.parse::().unwrap_or(0) * 3600; - } - core::time::Duration::from_secs(total_secs) -} - -/// Resolve the name of a parent process by PID. -fn resolve_parent_name(ppid: u32) -> Option { - if ppid == 0 { - return None; - } - let output = std::process::Command::new("ps") - .args(["-p", &ppid.to_string(), "-o", "comm="]) - .stdout(std::process::Stdio::piped()) - .stderr(std::process::Stdio::null()) - .output() - .ok()?; - // Strict decode: this process name is returned and used for a - // comparison/targeting decision, so invalid UTF-8 fails closed (None) - // rather than yielding a U+FFFD-mangled name. (WI-4.3 follow-up) - let name = core::str::from_utf8(&output.stdout).ok()?.trim().to_owned(); - if name.is_empty() { - return None; - } - let short = name.rsplit('/').next().unwrap_or(&name).to_owned(); - Some(short) -} +mod process_scan; +use process_scan::{mcp_stdio_json, print_mcp_stdio_section}; // ── small shared helpers ───────────────────────────────────────────────────── diff --git a/crates/uffs-cli/src/commands/system_status/process_scan.rs b/crates/uffs-cli/src/commands/system_status/process_scan.rs new file mode 100644 index 000000000..dd82b4c4e --- /dev/null +++ b/crates/uffs-cli/src/commands/system_status/process_scan.rs @@ -0,0 +1,230 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) 2025-2026 SKY, LLC. + +//! MCP stdio session discovery for `uffs --status`. +//! +//! Enumerates running `uffs --mcp run` processes via `ps`, under a watchdog +//! timeout so a process with a huge argument vector can never hang `--status`. + +use super::{Glyph, Palette, section, status_row}; + +// ── MCP Stdio Sessions ─────────────────────────────────────────────────────── + +/// Print the MCP stdio session list (running `uffs --mcp run` processes). +#[expect(clippy::print_stdout, reason = "CLI user-facing output")] +pub(super) fn print_mcp_stdio_section(palette: Palette) { + println!("{}", section(palette, "MCP Stdio Sessions")); + let Some(sessions) = find_mcp_stdio_processes() else { + println!(" {}", palette.dim("(process scan timed out — skipped)")); + return; + }; + if sessions.is_empty() { + println!(" {}", palette.dim("(none)")); + return; + } + let mut any_stale = false; + for (idx, session) in sessions.iter().enumerate() { + let parent = session + .parent_name + .as_deref() + .map_or(String::new(), |name| format!(" (parent: {name})")); + let glyph = if session.is_stale { + any_stale = true; + Glyph::Warn + } else { + Glyph::Up + }; + let name = format!("{}. PID {}", idx + 1, session.pid); + let detail = format!( + "uptime {}{parent}", + uffs_client::format::format_duration(session.uptime) + ); + println!("{}", status_row(palette, glyph, &name, &detail)); + } + if any_stale { + println!( + " {}", + palette + .dim("Older binary; the AI host that launched them refreshes on its next start.") + ); + } +} + +/// JSON for the MCP stdio session list. +pub(super) fn mcp_stdio_json() -> serde_json::Value { + let sessions: Vec = find_mcp_stdio_processes() + .unwrap_or_default() + .iter() + .map(|session| { + serde_json::json!({ + "pid": session.pid, + "uptime_secs": session.uptime.as_secs(), + "parent": session.parent_name, + "stale": session.is_stale, + }) + }) + .collect(); + serde_json::Value::Array(sessions) +} + +/// Information about a running MCP stdio process. +struct StdioSession { + /// Process ID. + pid: u32, + /// How long the process has been running. + uptime: core::time::Duration, + /// Name of the parent process (the AI host), if available. + parent_name: Option, + /// True if the process's binary is older than the current binary. + is_stale: bool, +} + +/// Max time to wait for a `ps` invocation before abandoning it. +const PS_TIMEOUT: core::time::Duration = core::time::Duration::from_secs(2); + +/// Run `program args…`, capturing stdout, but abandon it — killing the child — +/// if it exceeds `timeout`. A `ps` reading a process with a huge argument +/// vector (or one stuck in uninterruptible I/O) can block for a long time, and +/// `--status` must never hang on it. Returns `None` on spawn failure or +/// timeout. +fn capture_with_timeout( + program: &str, + args: &[&str], + timeout: core::time::Duration, +) -> Option> { + let child = std::process::Command::new(program) + .args(args) + .stdout(std::process::Stdio::piped()) + .stderr(std::process::Stdio::null()) + .spawn() + .ok()?; + let pid = child.id(); + + // Read the child on a helper thread so the main thread can enforce a + // deadline via `recv_timeout`. + let (tx, rx) = std::sync::mpsc::channel(); + std::thread::spawn(move || { + drop(tx.send(child.wait_with_output())); + }); + + if let Ok(Ok(output)) = rx.recv_timeout(timeout) { + Some(output.stdout) + } else { + // Timed out (or the child errored): terminate the process so it can't + // linger, and report the scan as unavailable rather than blocking. + drop( + std::process::Command::new("kill") + .arg("-TERM") + .arg(pid.to_string()) + .stdout(std::process::Stdio::null()) + .stderr(std::process::Stdio::null()) + .status(), + ); + None + } +} + +/// Find running `uffs --mcp run` processes via `ps`, flagging stale binaries. +/// +/// The `ps` calls run under [`PS_TIMEOUT`] (see [`capture_with_timeout`]), so a +/// process with a huge argument vector can never hang `--status`. Returns +/// `None` when the scan could not complete (spawn failed or timed out). +fn find_mcp_stdio_processes() -> Option> { + let stdout = capture_with_timeout("ps", &["-eo", "pid,ppid,etime,args"], PS_TIMEOUT)?; + + // AUDIT-OK(bytes): per-line PID scan of process-list output; each line's + // PID parses-or-skips (fail-safe). Whole-buffer strict decode would drop + // the whole list on one bad byte. (WI-4.3 follow-up) + let text = String::from_utf8_lossy(&stdout); + let my_pid = std::process::id(); + let current_mtime = std::env::current_exe() + .ok() + .and_then(|path| std::fs::metadata(path).ok()) + .and_then(|meta| meta.modified().ok()); + + let mut sessions = Vec::new(); + for line in text.lines().skip(1) { + if let Some(session) = parse_stdio_line(line, my_pid, current_mtime) { + sessions.push(session); + } + } + Some(sessions) +} + +/// Parse one `ps` line into a [`StdioSession`] if it is an MCP stdio process. +fn parse_stdio_line( + line: &str, + my_pid: u32, + current_mtime: Option, +) -> Option { + let mut fields = line.split_whitespace(); + let proc_pid = fields.next()?.parse::().ok()?; + if proc_pid == my_pid { + return None; + } + let parent_pid: u32 = fields.next().and_then(|val| val.parse().ok()).unwrap_or(0); + let elapsed_time = fields.next()?; + let cmdline: String = fields.collect::>().join(" "); + + // Match the actual `uffs --mcp run` invocation. Requiring the `--mcp` flag + // (not a bare "mcp" substring) excludes build tooling — cargo / sccache / + // rustc compiling the `uffs-mcp` crate carry "uffs-mcp" but never "--mcp". + if !cmdline.contains("--mcp") || !cmdline.contains("run") { + return None; + } + if cmdline.contains("serve") || cmdline.contains("start") || cmdline.contains("kill") { + return None; + } + + let uptime = parse_ps_etime(elapsed_time); + let is_stale = current_mtime.is_some_and(|bin_mtime| { + let proc_started = std::time::SystemTime::now() - uptime; + proc_started < bin_mtime + }); + Some(StdioSession { + pid: proc_pid, + uptime, + parent_name: resolve_parent_name(parent_pid), + is_stale, + }) +} + +/// Parse `ps` elapsed time format: `[[dd-]hh:]mm:ss`. +fn parse_ps_etime(etime: &str) -> core::time::Duration { + let mut total_secs: u64 = 0; + let (days_part, time_part) = if let Some((days, rest)) = etime.split_once('-') { + (days.parse::().unwrap_or(0), rest) + } else { + (0, etime) + }; + total_secs += days_part * 86400; + let mut parts = time_part.rsplit(':'); + if let Some(sec) = parts.next() { + total_secs += sec.parse::().unwrap_or(0); + } + if let Some(min) = parts.next() { + total_secs += min.parse::().unwrap_or(0) * 60; + } + if let Some(hour) = parts.next() { + total_secs += hour.parse::().unwrap_or(0) * 3600; + } + core::time::Duration::from_secs(total_secs) +} + +/// Resolve the name of a parent process by PID. +fn resolve_parent_name(ppid: u32) -> Option { + if ppid == 0 { + return None; + } + let ppid_str = ppid.to_string(); + let stdout = capture_with_timeout("ps", &["-p", &ppid_str, "-o", "comm="], PS_TIMEOUT)?; + // Strict decode: this process name is returned and used for a + // comparison/targeting decision, so invalid UTF-8 fails closed (None) + // rather than yielding a U+FFFD-mangled name. (WI-4.3 follow-up) + let name = core::str::from_utf8(&stdout).ok()?.trim().to_owned(); + if name.is_empty() { + return None; + } + let short = name.rsplit('/').next().unwrap_or(&name).to_owned(); + Some(short) +} From 320e27e3dfc5d373e7ec93b8ef5df4069cd3780c Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Mon, 6 Jul 2026 17:17:12 -0700 Subject: [PATCH 36/39] feat(just): use-local stops daemon + MCP before rebuilding MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `just use-local` rebuilt the workspace and installed to ~/bin but left a running daemon/MCP holding the old binaries — a stale in-memory index could shadow the fresh build. Add a step 0 that runs `uffs --daemon kill` + `pkill -x uffsd uffsmcp` before building. Co-Authored-By: Claude Opus 4.8 --- just/build.just | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/just/build.just b/just/build.just index 402317ed3..7d558fb3b 100644 --- a/just/build.just +++ b/just/build.just @@ -139,6 +139,13 @@ use-local: printf "\033[1;34m📦 Build + install ALL UFFS binaries to ~/bin\033[0m\n" echo "========================================================" + # 0. Stop the running daemon + MCP so the old binaries release their locks + # and no stale in-memory index shadows the freshly installed build. + printf "\n\033[1;33m🔪 Stopping daemon + MCP...\033[0m\n" + timeout 10 uffs --daemon kill >/dev/null 2>&1 || true + pkill -x uffsd 2>/dev/null || true + pkill -x uffsmcp 2>/dev/null || true + # 1. Single workspace-wide release build printf "\n\033[0;34m🔨 Building release workspace...\033[0m\n" START=$(date +%s) From fb5a18eddaa34a198cc38ed2f7bfecdf53ea95f8 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Mon, 6 Jul 2026 21:01:45 -0700 Subject: [PATCH 37/39] fix(uffs-mft): store reserved_allocated_bytes in .bin header (v3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The offline root Size-on-Disk was short by the reserved-cluster bytes (total_reserved × cluster_size) because the sequential .bin format never carried the value (only IOCP did) and the --reserved-allocated flag is a no-op stub. Bump RawMftHeader to v3 with a reserved_allocated_bytes field, thread it through SaveRawOptions + the streaming writer, populate it in capture from volume data, and restore it on load — including load_raw_to_index_direct, the loader the daemon actually uses for --mft-file. Validated on the Mac: G: offline flips MISMATCH → SUPERSET MATCH (root only-in-baseline 1 → 0). Co-Authored-By: Claude Opus 4.8 --- .../uffs-mft/src/commands/windows/capture.rs | 20 +++++++------ crates/uffs-mft/src/commands/windows/save.rs | 3 ++ crates/uffs-mft/src/raw/mod.rs | 30 +++++++++++++++++-- crates/uffs-mft/src/raw/streaming_writer.rs | 8 +++++ crates/uffs-mft/src/raw/tests.rs | 10 +++++++ crates/uffs-mft/src/reader/persistence.rs | 18 +++++++++-- 6 files changed, 75 insertions(+), 14 deletions(-) diff --git a/crates/uffs-mft/src/commands/windows/capture.rs b/crates/uffs-mft/src/commands/windows/capture.rs index e9ee30375..626d5fb65 100644 --- a/crates/uffs-mft/src/commands/windows/capture.rs +++ b/crates/uffs-mft/src/commands/windows/capture.rs @@ -166,7 +166,7 @@ const METAFILE_KINDS: [MetafileKind; 9] = [ /// interchangeable. It is zstd-compressed on disk; `load_raw_mft` auto-detects /// the compression flag and decompresses on read, so the `.bin` name is correct /// regardless. -fn capture_mft(drive: DriveLetter, dir: &Path) -> Result { +fn capture_mft(drive: DriveLetter, dir: &Path, reserved_allocated: u64) -> Result { use uffs_mft::{MftReader, SaveRawOptions}; let file = format!("{drive}_mft.bin"); @@ -177,6 +177,9 @@ fn capture_mft(drive: DriveLetter, dir: &Path) -> Result { compression_level: 3, volume_letter: drive, raw_compat: false, + // v3 header carries this so an offline `.bin` load reproduces the live + // root size-on-disk (tree_allocated root adjustment). + reserved_allocated_bytes: reserved_allocated, }; reader .save_raw_to_file(&path, &options) @@ -199,10 +202,11 @@ fn collect_artifacts( drive_lower: &str, serial: u64, timestamp: u64, + reserved_allocated: u64, ) -> Vec { let mut artifacts = Vec::new(); - match capture_mft(drive, dir) { + match capture_mft(drive, dir, reserved_allocated) { Ok(record) => { println!( " ✅ {:<9} {:>12} bytes {}", @@ -242,6 +246,10 @@ fn capture_one_drive(drive: DriveLetter, out: &Path) -> Result Result= 3 { + u64::from_le_bytes([ + buf[45], buf[46], buf[47], buf[48], buf[49], buf[50], buf[51], buf[52], + ]) + } else { + 0 + }; + Ok(Self { version, flags, @@ -162,6 +179,7 @@ impl RawMftHeader { original_size, compressed_size, volume_letter, + reserved_allocated_bytes, }) } } @@ -180,6 +198,10 @@ pub struct SaveRawOptions { /// If true, save in raw compatibility mode (no header, just raw MFT bytes). /// This format is compatible with other MFT tools like analyzeMFT, MFT2CSV. pub raw_compat: bool, + /// NTFS reserved-cluster bytes (`total_reserved × bytes_per_cluster`) to + /// record in the header for the root `tree_allocated` adjustment. `0` when + /// unknown (e.g. re-saving a raw dump with no volume data). + pub reserved_allocated_bytes: u64, } impl Default for SaveRawOptions { @@ -189,6 +211,7 @@ impl Default for SaveRawOptions { compression_level: 3, volume_letter: crate::platform::DriveLetter::X, raw_compat: false, + reserved_allocated_bytes: 0, } } } @@ -318,6 +341,7 @@ pub fn save_raw_mft>( original_size, compressed_size, volume_letter: options.volume_letter, + reserved_allocated_bytes: options.reserved_allocated_bytes, }; // Write file @@ -463,6 +487,7 @@ pub fn load_raw_mft>(path: P, options: &LoadRawOptions) -> Result volume_letter: options .volume_letter .unwrap_or(crate::platform::DriveLetter::X), + reserved_allocated_bytes: 0, // raw NTFS dump has no volume data }; // If header-only mode, return empty data @@ -510,6 +535,7 @@ fn load_iocp_as_raw_mft>(path: P, options: &LoadRawOptions) -> Re original_size: total_records * u64::from(record_size), compressed_size: 0, volume_letter, + reserved_allocated_bytes: 0, // IOCP reassembly path (carries via IOCP header elsewhere) }; // If header-only mode, return early diff --git a/crates/uffs-mft/src/raw/streaming_writer.rs b/crates/uffs-mft/src/raw/streaming_writer.rs index b0baaa7ca..cca3d6ef3 100644 --- a/crates/uffs-mft/src/raw/streaming_writer.rs +++ b/crates/uffs-mft/src/raw/streaming_writer.rs @@ -42,6 +42,8 @@ pub struct StreamingRawMftWriter { compress: bool, /// Volume letter (e.g., 'C', 'D'). volume_letter: crate::platform::DriveLetter, + /// NTFS reserved-cluster bytes for the header's root-allocated adjustment. + reserved_allocated_bytes: u64, /// Whether raw compatibility mode is enabled (no header). raw_compat: bool, /// Zstd encoder (if compressing). @@ -78,6 +80,7 @@ impl StreamingRawMftWriter { bytes_written: 0, compress: false, // Raw compat mode doesn't support compression volume_letter: options.volume_letter, + reserved_allocated_bytes: options.reserved_allocated_bytes, raw_compat: true, encoder: None, }); @@ -92,6 +95,7 @@ impl StreamingRawMftWriter { original_size: 0, compressed_size: 0, volume_letter: options.volume_letter, + reserved_allocated_bytes: options.reserved_allocated_bytes, }; writer.write_all(&placeholder_header.to_bytes())?; @@ -120,6 +124,7 @@ impl StreamingRawMftWriter { bytes_written: 0, compress: true, volume_letter: options.volume_letter, + reserved_allocated_bytes: options.reserved_allocated_bytes, raw_compat: false, encoder: Some(encoder), }); @@ -132,6 +137,7 @@ impl StreamingRawMftWriter { bytes_written: 0, compress: options.compress, volume_letter: options.volume_letter, + reserved_allocated_bytes: options.reserved_allocated_bytes, raw_compat: false, encoder: None, }) @@ -186,6 +192,7 @@ impl StreamingRawMftWriter { original_size, compressed_size: 0, volume_letter: self.volume_letter, + reserved_allocated_bytes: self.reserved_allocated_bytes, }); } @@ -213,6 +220,7 @@ impl StreamingRawMftWriter { original_size, compressed_size, volume_letter: self.volume_letter, + reserved_allocated_bytes: self.reserved_allocated_bytes, }; // For uncompressed, update the header at the beginning of the file diff --git a/crates/uffs-mft/src/raw/tests.rs b/crates/uffs-mft/src/raw/tests.rs index e7bb1ac06..c8e0208b9 100644 --- a/crates/uffs-mft/src/raw/tests.rs +++ b/crates/uffs-mft/src/raw/tests.rs @@ -17,6 +17,7 @@ fn header_roundtrip() -> TestResult { original_size: 1024 * 1000, compressed_size: 500_000, volume_letter: crate::platform::DriveLetter::G, + reserved_allocated_bytes: 54_587_392, }; let bytes = header.to_bytes(); @@ -29,6 +30,10 @@ fn header_roundtrip() -> TestResult { assert_eq!(parsed.original_size, header.original_size); assert_eq!(parsed.compressed_size, header.compressed_size); assert_eq!(parsed.volume_letter, header.volume_letter); + assert_eq!( + parsed.reserved_allocated_bytes, + header.reserved_allocated_bytes + ); assert!(parsed.is_compressed()); Ok(()) @@ -63,6 +68,7 @@ fn save_load_uncompressed() -> TestResult { compression_level: 3, volume_letter: crate::platform::DriveLetter::C, raw_compat: false, + reserved_allocated_bytes: 0, }; let header = save_raw_mft(&path, &data, record_size, &options)?; @@ -129,6 +135,7 @@ fn load_header_only() -> TestResult { compression_level: 3, volume_letter: crate::platform::DriveLetter::D, raw_compat: false, + reserved_allocated_bytes: 0, }; save_raw_mft(&path, &data, record_size, &options)?; @@ -155,6 +162,7 @@ fn iter_records() { original_size: 12, compressed_size: 0, volume_letter: crate::platform::DriveLetter::X, + reserved_allocated_bytes: 0, }; let data = vec![1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]; @@ -180,6 +188,7 @@ fn volume_letter_preserved() -> TestResult { compression_level: 3, volume_letter: crate::platform::DriveLetter::G, raw_compat: false, + reserved_allocated_bytes: 0, }; let header = save_raw_mft(&path, &data, record_size, &options)?; assert_eq!(header.volume_letter, crate::platform::DriveLetter::G); @@ -212,6 +221,7 @@ fn raw_compat_mode() -> TestResult { compression_level: 3, volume_letter: crate::platform::DriveLetter::G, raw_compat: true, + reserved_allocated_bytes: 0, }; let mut writer = StreamingRawMftWriter::new(&path, record_size, &options)?; diff --git a/crates/uffs-mft/src/reader/persistence.rs b/crates/uffs-mft/src/reader/persistence.rs index cb79ce789..819afe39a 100644 --- a/crates/uffs-mft/src/reader/persistence.rs +++ b/crates/uffs-mft/src/reader/persistence.rs @@ -336,7 +336,11 @@ impl MftReader { let record_count = parsed_records.len(); let t_build = Instant::now(); - let index = MftIndex::from_parsed_records(raw.header.volume_letter, parsed_records); + let mut index = MftIndex::from_parsed_records(raw.header.volume_letter, parsed_records); + // v3+ .bin files carry the reserved-cluster bytes for the root + // `tree_allocated` adjustment; restore it so an offline load + // reproduces the live root size-on-disk (0 for older files). + index.reserved_allocated_bytes = raw.header.reserved_allocated_bytes; if profile { let build_ms = t_build.elapsed().as_millis(); tracing::debug!( @@ -394,7 +398,9 @@ impl MftReader { let parse_ms = parse_start.elapsed().as_millis(); let record_count = parsed_records.len(); let t_build = Instant::now(); - let index = MftIndex::from_parsed_records(raw.header.volume_letter, parsed_records); + let mut index = + MftIndex::from_parsed_records(raw.header.volume_letter, parsed_records); + index.reserved_allocated_bytes = raw.header.reserved_allocated_bytes; if profile { let build_ms = t_build.elapsed().as_millis(); tracing::debug!( @@ -492,7 +498,9 @@ impl MftReader { let parse_ms = parse_start.elapsed().as_millis(); let record_count = parsed_records.len(); let t_build = Instant::now(); - let index = MftIndex::from_parsed_records(raw.header.volume_letter, parsed_records); + let mut index = + MftIndex::from_parsed_records(raw.header.volume_letter, parsed_records); + index.reserved_allocated_bytes = raw.header.reserved_allocated_bytes; if profile { let build_ms = t_build.elapsed().as_millis(); tracing::debug!( @@ -700,6 +708,10 @@ impl MftReader { // Create index with pre-allocated capacity let mut index = MftIndex::with_capacity(raw.header.volume_letter, capacity); + // v3+ .bin files carry the reserved-cluster bytes for the root + // `tree_allocated` adjustment; restore it so an offline load (this is + // the loader the daemon uses) reproduces the live root size-on-disk. + index.reserved_allocated_bytes = raw.header.reserved_allocated_bytes; // Parse records directly into index let mut fixup_success: u64 = 0; From d1aad61c02c9a772c1dd46ae303b878fca0a6e34 Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Mon, 6 Jul 2026 21:12:34 -0700 Subject: [PATCH 38/39] feat(uffs-mft): add extract-mft to pull the raw $MFT from a .bin MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `extract-mft --input C_mft.bin --output C.mft` decompresses a captured UFFS-MFT .bin and writes the byte-exact $MFT records — the format analyzeMFT / MFT2CSV / ntfstool consume. Cross-platform, so a .bin moved to a Mac yields a tool-ready $MFT with no re-capture. Validated: G_mft.bin → 20224 records × 1024 B, every record carrying the FILE magic. Co-Authored-By: Claude Opus 4.8 --- crates/uffs-mft/src/cli.rs | 13 ++++++ crates/uffs-mft/src/commands/extract_mft.rs | 51 +++++++++++++++++++++ crates/uffs-mft/src/commands/mod.rs | 3 ++ 3 files changed, 67 insertions(+) create mode 100644 crates/uffs-mft/src/commands/extract_mft.rs diff --git a/crates/uffs-mft/src/cli.rs b/crates/uffs-mft/src/cli.rs index 1744ffb51..ed7aacd4a 100644 --- a/crates/uffs-mft/src/cli.rs +++ b/crates/uffs-mft/src/cli.rs @@ -198,6 +198,19 @@ pub(crate) enum Commands { input: PathBuf, }, + /// Extract the raw `$MFT` from a captured `.bin` (decompress + strip + /// header) into a byte-exact `$MFT` other tools (analyzeMFT, MFT2CSV, + /// ntfstool) can read. Cross-platform. + ExtractMft { + /// Captured `.bin` (e.g. `C_mft.bin` from `capture`). + #[arg(short, long)] + input: PathBuf, + + /// Output path for the raw `$MFT`. + #[arg(short, long)] + output: PathBuf, + }, + /// Compare two MFT CSV exports (from `load`) for parity — e.g. Rust on /// Windows vs macOS, or Rust vs a C++ golden. Exits non-zero on mismatch. Verify { diff --git a/crates/uffs-mft/src/commands/extract_mft.rs b/crates/uffs-mft/src/commands/extract_mft.rs new file mode 100644 index 000000000..033467fc2 --- /dev/null +++ b/crates/uffs-mft/src/commands/extract_mft.rs @@ -0,0 +1,51 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) 2025-2026 SKY, LLC. + +//! `extract-mft` command — extract the raw `$MFT` from a captured `.bin`. +//! +//! Cross-platform: decompresses a UFFS-MFT `.bin` (or passes a headerless raw +//! dump straight through) and writes the byte-exact `$MFT` records — the format +//! other MFT tools (analyzeMFT, MFT2CSV, ntfstool) consume. Runs on the +//! transfer target too, so a `.bin` moved to a Mac yields a tool-ready `$MFT` +//! with no re-capture. +#![expect( + clippy::print_stdout, + reason = "intentional user-facing CLI extract summary" +)] + +use std::path::Path; + +use anyhow::{Context as _, Result}; + +/// Extract the raw `$MFT` bytes from a captured `.bin` to `output`. +/// +/// # Errors +/// +/// Returns an error if the input cannot be loaded/decompressed or the output +/// cannot be written. +pub(crate) fn cmd_extract_mft(input: &Path, output: &Path) -> Result<()> { + use uffs_mft::raw::{LoadRawOptions, load_raw_mft}; + + let raw = load_raw_mft(input, &LoadRawOptions::default()) + .with_context(|| format!("loading {}", input.display()))?; + std::fs::write(output, &raw.data).with_context(|| format!("writing {}", output.display()))?; + + println!("✅ Extracted raw $MFT (tool-ready: analyzeMFT / MFT2CSV / ntfstool)"); + println!( + " Input: {} ({})", + input.display(), + if raw.header.is_compressed() { + "compressed UFFS-MFT" + } else { + "raw / uncompressed" + } + ); + println!( + " Output: {} ({} bytes, {} records × {} B)", + output.display(), + raw.data.len(), + raw.header.record_count, + raw.header.record_size, + ); + Ok(()) +} diff --git a/crates/uffs-mft/src/commands/mod.rs b/crates/uffs-mft/src/commands/mod.rs index 2ebb44fb9..674a7bfeb 100644 --- a/crates/uffs-mft/src/commands/mod.rs +++ b/crates/uffs-mft/src/commands/mod.rs @@ -7,6 +7,7 @@ use anyhow::Result; use crate::cli::Commands; +mod extract_mft; mod load; mod metafile_info; mod sysinfo; @@ -53,6 +54,7 @@ pub(crate) async fn dispatch_command(command: Commands) -> Result<()> { split_gib, } => windows::cmd_capture(drive, &out, all_drives, zip, split_gib).await, Commands::MetafileInfo { input } => metafile_info::cmd_metafile_info(&input), + Commands::ExtractMft { input, output } => extract_mft::cmd_extract_mft(&input, &output), Commands::Verify { left, right, @@ -214,6 +216,7 @@ pub(crate) async fn dispatch_command(command: Commands) -> Result<()> { ), Commands::Sysinfo { out, json } => sysinfo::run(out.as_deref(), json), Commands::MetafileInfo { input } => metafile_info::cmd_metafile_info(&input), + Commands::ExtractMft { input, output } => extract_mft::cmd_extract_mft(&input, &output), Commands::Verify { left, right, From 12fe3cbf78b0bf65ac7e15c36dea7de45a8d28ac Mon Sep 17 00:00:00 2001 From: Robert M1 <50460704+githubrobbi@users.noreply.github.com> Date: Tue, 7 Jul 2026 07:31:30 -0700 Subject: [PATCH 39/39] chore(deps): bump crossbeam-epoch 0.9.18->0.9.20 (RUSTSEC-2026-0204) New advisory RUSTSEC-2026-0204: invalid-pointer dereference in crossbeam-epoch's fmt::Pointer impl for Atomic/Shared (transitive via crossbeam-deque). Bump to the patched 0.9.20. Delta audited safe-to-deploy: a soundness fix that removes the unsafe deref (prints the raw address instead), a benign sanitizer-cfg build.rs, no new runtime deps or ambient capabilities, publisher taiki-e (crossbeam maintainer). Vet-Reviewed-Diff: crossbeam-epoch@0.9.18->0.9.20 Co-Authored-By: Claude Opus 4.8 --- Cargo.lock | 4 ++-- supply-chain/audits.toml | 6 ++++++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 9c9371df2..49825729f 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -831,9 +831,9 @@ dependencies = [ [[package]] name = "crossbeam-epoch" -version = "0.9.18" +version = "0.9.20" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e" +checksum = "2d6914041f254d6e9176c01941b21115dcfb7089e55135a35411081bd106ef3f" dependencies = [ "crossbeam-utils", ] diff --git a/supply-chain/audits.toml b/supply-chain/audits.toml index 4ab46cae8..e02f71366 100644 --- a/supply-chain/audits.toml +++ b/supply-chain/audits.toml @@ -73,6 +73,12 @@ criteria = "safe-to-deploy" delta = "0.16.3 -> 0.16.4" notes = "Delta audit (cargo vet diff 0.16.3 -> 0.16.4, 435-line diff). (1) term.rs: TermInner construction deduplicated into new()/new_buffered()/with_buffer() helpers - pure refactor; new pub fn is_dumb() reads TERM env var (read-only env access, same capability class the crate already had via is_a_color_terminal). (2) utils.rs: Style::from_dotted_str gains is_ascii() guards on '#RRGGBB'/'on_#RRGGBB' parsing - fixes a panic on non-ASCII input that slices mid-UTF-8 char (robustness improvement for untrusted style strings), plus regression tests. (3) windows_term: as_handle() helper removed in favor of out.as_raw_handle() at existing call sites - unchanged unsafe surface (same SetConsoleCursorPosition/SetConsoleCursorInfo calls as before); read_secure backspace arm converted to a match guard, logic identical. No new unsafe blocks, no new fs/net/process capability. Publisher djc (Dirkjan Ochtman, console-rs maintainer, same publisher as 0.16.3)." +[[audits.crossbeam-epoch]] +who = "Robert M1 <50460704+githubrobbi@users.noreply.github.com>" +criteria = "safe-to-deploy" +delta = "0.9.18 -> 0.9.20" +notes = "Delta audit (cargo vet diff 0.9.18 -> 0.9.20), anchored on the existing 0.9.18 exemption. Fixes RUSTSEC-2026-0204: the fmt::Pointer impl for Atomic/Shared no longer dereferences the underlying raw pointer (UB when the pointer is invalid or null) — it prints the address directly (raw as *const ()), covering the invalid-ptr (#1276) and null-ptr (#1273) cases. Net -2 unsafe over the delta (3 removed, 1 added); the one added unsafe is Guard::defer_destroy re-emitted with a broadened bound ( -> ) with an unchanged body (self.defer_unchecked(move || ptr.into_owned())), so no new unsafe operation. The new build.rs is benign: it emits rustc-check-cfg plus a crossbeam_sanitize_thread cfg derived from CARGO_CFG_SANITIZE (ThreadSanitizer detection migrated out of lib-level cfg); it reads one env var and does no fs/net/process/exec. No new runtime dependencies (crossbeam-utils pre-existing; rand is dev-only). No new ambient-capability surface. Publisher taiki-e (crossbeam maintainer, same publisher as 0.9.18). The change is strictly a soundness fix plus a benign cfg migration, so trust extends from the audited 0.9.18 base." + [[audits.crypto-common]] who = "Robert Nio " criteria = "safe-to-deploy"