bit-backup is a command-line utility designed for long-term data integrity. It prevents "bit rot" (silent data corruption) by storing and verifying SHA-512 checksums of your files in a local SQLite database.
- Integrity Checks: Detects changes in files by comparing current SHA-512 hashes with stored ones.
- Ignore Patterns: Exclude specific files or directories using
.bitbackupignore(wildcards supported). - SQLite Backend: Efficiently stores file metadata and hashes.
- Self-Integrity: Verifies the integrity of the backup database itself using a separate hash sum.
- Reporting: Generates reports for detected bit rot and full file indexes in CSV format.
- C++23 compliant compiler (GCC 12+, Clang 16+, or MSVC 2022)
- CMake 3.25+
- OpenSSL library (for cryptographic hashing)
- SQLite3 (bundled as a submodule)
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config ReleaseThe primary command is check, which scans the specified directory and updates the integrity database.
# Basic check (current directory)
./build/bit_backup
# Explicit check on a specific directory
./build/bit_backup check dir=/path/to/my/data
# Generate a bit rot report
./build/bit_backup check report=true
# Enable verbose logging
./build/bit_backup check verbose=truecheck: (Default) Scans files and verifies hashes.help: Shows help information.version: Shows the tool version.
| Argument | Description | Default |
|---|---|---|
dir |
Path to the directory to be checked for bit rot. | . (Current) |
report |
Set to true to generate a report file (.bitbackupreport.csv). |
false |
verbose |
Set to true to show detailed scan information. |
false |
bitbackupindex |
Set to true to generate a full file index (.bitbackupindex.csv). |
false |
threads |
Explicit number of hashing workers (1–16). Without this option, storage-aware automatic selection is used. |
HDD/unknown: 1; SSD: up to 4; NVMe: up to 16 |
quick |
true skips re-hashing files whose modification time is unchanged. Fast, but does not detect silent bit rot. |
false |
scrub |
Re-hash only the oldest N% of unchanged-modtime files this run (rotating coverage, like a scrub). 100 = full check, 0 = same as quick. |
100 |
Note: options must follow the explicit
checkcommand, e.g.bit_backup check quick=true threads=8.
On Linux, bit-backup detects whether the filesystem containing dir is backed
by rotational, SATA/general solid-state, or NVMe storage. Detection follows
underlying devices through dm-crypt, LVM, md, and similar block-device layers.
A rotational drive (or storage whose type cannot be detected safely) uses one
sequential hashing stream, an SSD uses up to four workers, and NVMe uses up to
16, all bounded by the available CPU count. threads=N overrides this automatic
choice and is capped at 16. The selected storage type, worker count, and whether
it was automatic or manual are printed at startup.
Files are processed in path order for better HDD locality, reads use 1 MiB
chunks with a sequential-access hint on Linux, and all database
inserts/updates/deletes are batched into single transactions. For routine runs
over very large trees, quick=true (skip unchanged files) or scrub=N (verify
a rotating slice each run) keep wall-clock bounded while scrub still
eventually re-verifies everything. quick=true does not detect silent bit rot
in files whose modification time is unchanged.
Create a file named .bitbackupignore in the root of your scanned directory.
Patterns use shell-style wildcards with a few gitignore-like conventions:
*matches any run of characters,?matches a single character.- A leading
/anchors the pattern to the scanned root (/logs/*matcheslogs/...but notsrc/logs/...). - A trailing
/marks a directory:build/ignores everything underbuild. - A leading
!re-includes a previously ignored path (!keep.log). - Lines starting with
#are comments; blank lines are ignored.
# Ignore whole directories (their contents are skipped without descending)
/logs/
/node_modules/
build/
# Ignore specific file types (at any depth)
*.tmp
*.log
# ...but keep one important log
!important.logbit-backup's own metadata files (.bitbackup.sqlite3, its .sha512,
.bitbackupignore, .bitbackupindex.csv, *.bitbackupreport.csv,
.bitbackuplock) are always excluded automatically.
Drop an (empty) .bitbackuplock file into a directory to freeze that
directory and everything below it. First index the files normally, then add the
marker - the currently stored state becomes the source of truth.
For a locked subtree bit-backup will:
- never overwrite the stored modification time or hash of its files, and
- report a violation (red text on a terminal, non-zero exit) for any change: a modified file, a new file, or a deleted file.
This is the opposite of the normal behavior, where a changed modification time is treated as a legitimate edit and absorbed into the database. Remove the marker to unlock the directory and resume normal updates.
check returns a non-zero exit code when it finds bit rot or a lock
violation, so it can be used directly in scripts and cron jobs.
The tool creates several hidden files in the target directory to manage its state:
.bitbackup.sqlite3: The database containing file metadata and hashes..bitbackup.sqlite3.sha512: A hash of the database to ensure its own integrity..bitbackupreport.csv: Generated whenreport=true, listing files with detected corruption..bitbackupindex.csv: Generated whenbitbackupindex=true, containing a list of all scanned files.
- C++23: Utilizing modern features like
std::filesystem. - SQLiteCpp: A clean C++ wrapper for SQLite3.
- OpenSSL: For high-performance SHA-512 hashing.
include/BitBackup/: Public headers organized by module (Core, Commands, Entity, Files, Persistence).src/BitBackup/: Private implementation files.tests/: GTest-based unit tests for core functionality.thirdparty/: External dependencies (SQLiteCpp).
(Note: Tests are optional and disabled by default)
# Enable tests during configuration
cmake -B build -DENABLE_TESTS=ON
cmake --build build
# Run tests
cd build
ctest- Table FILE – add new columns – linux_rights, owner, group.
- Improve performance for large datasets (>1M files).
License: MIT