v0.0.9

What's new

• --max-time — stop deduplication gracefully after a duration (e.g. --max-time 23h)
• --scrub — run btrfs scrub after dedup completes (requires root, btrfs only)
• --defrag — run btrfs defragment after dedup/scrub (requires root, btrfs only)
• Wave-based memory processing — default mode now uses iterative waves with eviction for better memory management on large directories
• Cron job enhancements — scrub, defrag, and max-time options available in /etc/default/fastdedup

Bug fixes

• Fix integer overflow on 32-bit architectures in progress bar calculations
• Fix btrfs filesystem detection on 32-bit systems
• Fix scrub status polling infinite loop when btrfs command fails
• Fix defrag ETA always showing ~0s
• Fix cache not saving on batch mode early return
• Fix wave deferred count showing inflated numbers
• Remove vendor/ from git tracking (regenerated during release)

v0.0.8

v0.0.8

• Per-root lock file prevents concurrent runs on the same directory
• Size-based progress bar on btrfs (where inode counts are unavailable)
• ETA estimates on all progress bars (~30s, ~1.5m, ~2.0h)
• Time-based progress updates every 200ms instead of every 10,000 files
• Cron script uses flock to prevent overlapping scheduled runs
• Slack/Mattermost webhooks for run summaries and critical alerts
• Healthchecks.io dead man's switch URL
• Anonymized error reporting with deduplication
• Logrotate integration for cron log output
• Release script runs unit tests first, builds all packages before uploading
• README updated with all new features and environment variables

v0.0.7

Changes

• Cached sizes no longer count against -top limit — Subsequent runs now process the full requested number of active size groups, regardless of how many are cached
• Cache pruning — Stale entries for file sizes no longer present on disk are automatically removed
• Better messaging — Shows cached group count when no new duplicates are found

v0.0.6

fastdedup v0.0.6

UX improvements

• Scan rate — pass 1 shows files/s throughput
• Total row in top sizes table — see aggregate file count and potential savings at a glance
• Collection progress bar — pass 2 file collection now shows a progress bar instead of raw counter
• Cleaner output — "no duplicates" groups are suppressed; count shown in final summary
• Elapsed time — final summary shows how long the run took
• Quiet mode path — -q output now includes the directory path, useful for cron logs
• Exit code 1 on errors — scripts and cron can detect failures

Bug fix

• --fix-perms handles read-only files — in-place reflink fallback now temporarily chmods 0444 files writable, then restores original permissions

v0.0.5

fastdedup v0.0.5

New features

• fastdedup-daily / fastdedup-weekly packages — cron-based scheduled deduplication for all mounted btrfs, XFS, and ZFS filesystems, configurable via /etc/default/fastdedup
• Pass 2 progress summary — shows total groups, files, and potential savings before deduplication starts; per-group progress bar includes overall completion %
• PPA support for Ubuntu 24.10 (oracular), 25.04 (plucky), 26.04 (resolute)

Bug fixes

• Path resolution — fastdedup . now correctly resolves to the canonical absolute path for display and cache keying
• Robust dedup error handling — on failure, tries remaining reference files instead of giving up; promotes the file as an alternative ref so future files in the same content group can still be deduplicated

Install scheduled dedup

# Via PPA
sudo add-apt-repository ppa:phntm/ppa
sudo apt install fastdedup-daily   # or fastdedup-weekly

# From GitHub release
sudo dpkg -i fastdedup_0.0.5_amd64.deb fastdedup-daily_0.0.5_all.deb

Configure via /etc/default/fastdedup — by default runs in quiet mode on all detected reflink-capable filesystems.

v0.0.4

fastdedup v0.0.4

New features

• Memory-efficient path storage — directory string interning reduces RAM usage for large trees
• --mem-budget flag — configurable memory budget (default 256 MiB) for the path cache
• --fix-perms flag — temporarily adds write permission to read-only directories during dedup, then restores
• In-place reflink fallback — deduplicates files in write-protected directories without needing directory entry changes
• Incremental cache saving — cache survives Ctrl+C mid-run; progress saved after each completed size group
• Atomic cache writes — write-to-tmp + rename prevents corruption on interruption

PPA

Available for Ubuntu: jammy (22.04), noble (24.04), oracular (24.10), plucky (25.04), questing (25.10), resolute (26.04)

sudo add-apt-repository ppa:phntm/ppa
sudo apt update
sudo apt install fastdedup

v0.0.3

What's new in v0.0.3

Multi-filesystem support

• ZFS (OpenZFS 2.2+ with block_cloning enabled) and XFS (reflink=1) now work alongside btrfs
• FIEMAP is no longer required — falls back to byte-by-byte content comparison on filesystems without extent mapping
• Post-dedup verification adapts to filesystem capabilities

Hard link mode (--hardlink)

• New --hardlink flag for filesystems without reflink support
• Works on any Linux filesystem by replacing duplicates with hard links
• Prints a warning about shared-inode implications (editing one file changes all copies)

Test suite (cmd/testdedup)

• Synthetic test generator that creates file groups with known duplicate/unique patterns
• Validates dedup count, content integrity, and cache behavior
• Supports -dir flag to test on specific filesystems (e.g., -dir /mnt/btrfs)
• Gracefully handles non-reflink filesystems

Bug fixes

• Cache no longer saves state when dedup errors occurred, preventing false skips on subsequent runs
• Improved --no-cache flag description for clarity

v0.0.2

What's new

• -q (quiet mode) — single summary line, ideal for cronjobs
• -v (verbose) — shows file paths of deduped files and diagnostics
• --batch — collect all target files in one pass (old default; faster, more memory)
• --snapshots — include .snapshots directories (skipped by default)
• Lower memory default — pass 2 now scans per file size instead of collecting all paths at once
• Rank by savings — table and sort order use size * (count - 1) instead of size * count
• Permission errors are now silent unless -v

v0.0.1

fastdedup v0.0.1

Fast file deduplication for btrfs filesystems using reflinks.

Features

• Two-pass deduplication: surveys file sizes first, then deduplicates the most impactful duplicates
• Uses btrfs reflinks for instant, copy-on-write deduplication (no extra disk space)
• Visual progress bars and human-readable output
• Dry-run mode to preview changes before applying
• Configurable limits for tracked sizes and dedup targets

Installation

Debian/Ubuntu

sudo dpkg -i fastdedup_0.0.1_amd64.deb

RHEL/Fedora/CentOS

sudo rpm -i fastdedup-0.0.1-1.x86_64.rpm

Alpine Linux

sudo apk add --allow-untrusted fastdedup_0.0.1_x86_64.apk

Self-extracting archive

chmod +x fastdedup-0.0.1-linux-amd64.sh
sudo ./fastdedup-0.0.1-linux-amd64.sh

Binary

chmod +x fastdedup-linux-amd64
sudo mv fastdedup-linux-amd64 /usr/local/bin/fastdedup

Usage

fastdedup [flags] [directory]

Flag	Default	Description
--max-sizes	1,000,000	Maximum unique file sizes to track in pass 1
--top	10,000	Number of most impactful file sizes to dedup in pass 2
--dry-run	false	Report what would be deduped without making changes
-v	false	Verbose output
--raw-sizes	false	Show raw byte counts instead of human-readable
-C		Change to directory before running

Supported architectures

amd64, arm64, i386, armhf, riscv64, ppc64le, s390x, mips64le

Package formats

• .deb - Debian, Ubuntu, and derivatives
• .rpm - RHEL, Fedora, CentOS, openSUSE
• .apk - Alpine Linux
• .sh - Self-extracting shell archive (any Linux)
• Raw binaries for manual installation