diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 2aebcb2..e592f4e 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -45,18 +45,18 @@ uv run python -m py_compile bbackup.py bbman.py bbackup/*.py bbackup/data/*.py b ## Commit messages -This project uses [conventional commits](https://www.conventionalcommits.org/). The prefix determines how the version is bumped on the next release: +This project uses [conventional commits](https://www.conventionalcommits.org/) for readable history. Normal development batches release as one patch increment unless a commit body includes an explicit `Release-Type: major|minor|patch|none` trailer. | Prefix | Bump | When to use | |---|---|---| -| `feat:` | minor | New user-visible feature | +| `feat:` | patch by default | New user-visible feature | | `fix:` | patch | Bug fix | | `docs:` | patch | Documentation only | | `refactor:` | patch | Code restructure, no behavior change | | `perf:` | patch | Performance improvement | | `test:` | patch | Test additions or changes | | `chore:` | patch | Build, tooling, dependency updates | -| `feat!:` or `BREAKING CHANGE:` in body | major | Incompatible change | +| `feat!:` or `BREAKING CHANGE:` in body | requires explicit `Release-Type:` | Incompatible change | One subject line, imperative mood, no trailing period. Example: diff --git a/AGENTS.md b/AGENTS.md index ab3a290..5d6003a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -76,3 +76,9 @@ git diff --check - Prefer updating existing docs over creating new documents. - Keep command examples copy-ready and avoid documenting machine-specific secrets or private paths unless they are explicitly local runtime examples. + +## Agent Client Compatibility + +- Codex CLI and OpenCode are both supported agent clients for this repo. +- Root `AGENTS.md` is the shared instruction surface for both tools. +- Keep Codex-only behavior in Codex-native surfaces and OpenCode-only behavior in OpenCode-native surfaces. diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a6ed2d..71414cc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,32 @@ current uv-based install, development, and release instructions. --- +## [1.8.5] - 2026-07-09 + +### Added + +- Native restic snapshot profiles with commands for `snapshot init`, `snapshot run`, `snapshot check`, `snapshot restore`, `snapshot retire`, `snapshot purge-plan`, and `snapshot schedule`. +- Local restic integration coverage that initializes a temporary repository, snapshots a Git repo, checks the repository, restores content, and verifies the state ledger snapshot ID. +- Profile schedule rendering for daily backup, weekly non-destructive maintenance checks, and monthly verification checks with configurable `verification_read_data_subset`. +- Generated CLI skills metadata for the native snapshot command surface. + +### Changed + +- Snapshot schedule units now render matching service/timer pairs, use `/usr/bin/env bbackup`, and escape systemd percent specifiers. +- Snapshot commands now accept required options from `--input-json` consistently for agent-driven workflows. +- Retired-repo destructive cleanup remains dry-run-first/manual through `snapshot purge-plan`. +- The pytest `integration` marker now covers external-tool integration tests such as Docker and restic. + +### Fixed + +- Google Drive rclone snapshot profiles without a dedicated `client_id` remain rejected by default, while `allow_default_rclone_drive_client: true` provides a conservative profile-local opt-in. +- Non-dry snapshot operations now enforce profile safety preflight checks before running restic. +- Git repo identity no longer changes on every commit; legacy head-derived repo IDs migrate safely, and unknown active same-path identity conflicts block history merging. +- `list-backups` and restore dry-run planning no longer initialize Docker for file-only planning paths. +- Restore dry-runs now validate filesystem destination and multi-filesystem constraints before returning a success plan. + +--- + ## [1.8.4] - 2026-06-22 ### Security @@ -270,7 +296,8 @@ current uv-based install, development, and release instructions. --- -[Unreleased]: https://github.com/CruxExperts/best-backup/compare/v1.8.4...HEAD +[Unreleased]: https://github.com/CruxExperts/best-backup/compare/v1.8.5...HEAD +[1.8.5]: https://github.com/CruxExperts/best-backup/compare/v1.8.4...v1.8.5 [1.8.4]: https://github.com/CruxExperts/best-backup/compare/v1.8.3...v1.8.4 [1.8.3]: https://github.com/CruxExperts/best-backup/compare/v1.8.2...v1.8.3 [1.8.2]: https://github.com/CruxExperts/best-backup/compare/v1.8.1...v1.8.2 diff --git a/README.md b/README.md index b9010ae..2bab865 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ [![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-3776ab?style=flat-square&logo=python&logoColor=white)](https://www.python.org/downloads/) [![License: MIT](https://img.shields.io/badge/license-MIT-22c55e?style=flat-square)](LICENSE) -[![Version](https://img.shields.io/badge/version-1.8.4-6366f1?style=flat-square)](CHANGELOG.md) +[![Version](https://img.shields.io/badge/version-1.8.5-6366f1?style=flat-square)](CHANGELOG.md) [Quick start](#quick-start) · [Filesystem backup](#filesystem-backup) · [Agent integration](#agent-integration) · [CLI reference](#cli-reference) · [Docs](#documentation) @@ -206,6 +206,63 @@ Or pass paths directly, no config needed: bbackup backup --paths /home/user/docs /srv/data --exclude "*.tmp" ``` +### Native restic snapshots + +Add a top-level `snapshot_profiles:` section for encrypted deduplicating restic +snapshots. bbackup owns discovery, safety checks, state, and scheduling while +restic owns the snapshot repository. + +```yaml +snapshot_profiles: + essentials-daily: + engine: restic + host_id: WORKSTATION01 + repository: rclone:my-drive:backups/WORKSTATION01/restic/essentials-daily + # Prefer a dedicated Google Drive rclone OAuth client. Set this only as an + # explicit YAML boolean true when accepting rclone's shared default client. + allow_default_rclone_drive_client: false + cache_dir: ~/.cache/bbackup/restic/WORKSTATION01/essentials-daily + state_dir: ~/.local/state/bbackup/WORKSTATION01/essentials-daily + password_file: ~/.local/share/bbackup-credentials/restic/WORKSTATION01/essentials-daily.password + repo_homes: + - ~/Projects + - ~/Work + explicit_repos: + - /home/user/my-control-repo + include_paths: + - ~/.config/bbackup + - ~/Documents +``` + +The password file must stay outside selected backup paths and should be `0600`. +Escrow the password outside the backup. + +Google Drive rclone remotes should use a dedicated OAuth `client_id`. A profile +may opt into rclone's shared default Drive client with +`allow_default_rclone_drive_client: true`; quoted strings do not enable it. + +Plan, initialize, run, check, restore, retire, purge-plan, and render schedule +units with: + +```bash +bbackup snapshot plan --profile essentials-daily --output json +``` + +```bash +bbackup snapshot init --profile essentials-daily --dry-run --output json +``` + +```bash +bbackup snapshot run --profile essentials-daily +``` + +Deleted Git repositories are marked retired only after at least one successful +snapshot. Automated retention must target active repo IDs or configured path +scopes; use `snapshot purge-plan` for dry-run-first retired-repo purges. +Rendered schedule units include a daily `snapshot run`, weekly non-destructive +`snapshot check`, and monthly `snapshot check --read-data-subset`, defaulting +to `5%` unless `schedule.verification_read_data_subset` is set. + --- ## CLI reference @@ -320,7 +377,7 @@ Level-0 JSON output: ```json { "cli": "bbackup", - "version": "1.8.4", + "version": "1.8.5", "agent_hint": "Set BBACKUP_OUTPUT=json and BBACKUP_NO_INTERACTIVE=1 for fully non-interactive use.", "skills": [ {"id": "docker-backup", "summary": "Back up Docker containers, volumes, networks, and configs.", "common": true}, diff --git a/VERSION b/VERSION index bfa363e..8decb92 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.8.4 +1.8.5 diff --git a/agent-repo-shape.json b/agent-repo-shape.json new file mode 100644 index 0000000..d3060cf --- /dev/null +++ b/agent-repo-shape.json @@ -0,0 +1,23 @@ +{ + "agent_clients": [ + "codex", + "opencode" + ], + "external_skills": [], + "platform_overrides": { + "codex": { + "enabled": true + }, + "opencode": { + "enabled": true + } + }, + "profile": "lean-universal", + "repo": { + "name": "bbackup" + }, + "repo_profile": "universal-agent-repo", + "repo_skills": [], + "schema_version": "1", + "shared_skill_surface": ".agents/skills" +} diff --git a/bbackup/__init__.py b/bbackup/__init__.py index bfcd4bb..db6dd8b 100644 --- a/bbackup/__init__.py +++ b/bbackup/__init__.py @@ -3,5 +3,5 @@ A comprehensive backup solution for Docker containers, volumes, and configurations. """ -__version__ = "1.8.4" +__version__ = "1.8.5" __author__ = "Slavic Kozyuk / Crux Experts LLC" diff --git a/bbackup/cli.py b/bbackup/cli.py index 28caccd..d18f494 100644 --- a/bbackup/cli.py +++ b/bbackup/cli.py @@ -26,9 +26,21 @@ from .remote import RemoteStorageManager from .archive import create_solid_archive, is_solid_archive_name, unpack_solid_archive from .backup_runner import BackupRunner -from .restore import DockerRestore, list_volume_backup_names +from .restore import DockerRestore, list_available_backups, list_volume_backup_names from .logging import setup_logging from .encryption import EncryptionManager +from .snapshot import ( + SnapshotError, + expand_path, + purge_plan, + retire_repo, + schedule_units, + snapshot_check, + snapshot_init, + snapshot_plan, + snapshot_restore, + snapshot_run, +) from .resources import read_text_resource, resource_exists from .cli_utils import ( output_option, @@ -109,6 +121,28 @@ def _backup_encryption_result(status: BackupStatus, backup_path: Path) -> str: return "disabled" +def _snapshot_profile_or_exit(config: Config, profile: str, output: str): + snapshot_profile = config.get_snapshot_profile(profile) + if snapshot_profile is None: + json_error("snapshot", f"Snapshot profile not found: {profile}", EXIT_USER_ERROR, output) + return snapshot_profile + + +def _snapshot_required_param(ctx: click.Context, name: str, command: str, output: str): + value = ctx.params.get(name) + if value in (None, ""): + json_error(command, f"Missing required option: --{name.replace('_', '-')}", EXIT_USER_ERROR, output) + return value + + +def _snapshot_result(command: str, result: dict, output: str, console: Console) -> None: + success = bool(result.get("success", result.get("ok", True))) + render_output(result, output, command, success=success) + if output != "json": + console.print_json(data=result) + sys.exit(EXIT_SUCCESS if success else EXIT_SYSTEM_ERROR) + + @click.group() @click.version_option(version=__import__("bbackup").__version__) @click.option( @@ -447,6 +481,202 @@ def backup_operation(): sys.exit(EXIT_PARTIAL if run_results else EXIT_SYSTEM_ERROR) +# --------------------------------------------------------------------------- +# snapshot +# --------------------------------------------------------------------------- + +@cli.group() +def snapshot(): + """Native deduplicating snapshot backups.""" + + +@snapshot.command("plan") +@click.option("--profile", "-p", required=False, help="Snapshot profile name") +@output_option +@input_json_option +@click.pass_context +def snapshot_plan_cmd(ctx, profile, output, input_json): + """Resolve a snapshot profile without executing restic.""" + merge_json_input(ctx, input_json) + output = ctx.params.get("output", output) + profile = _snapshot_required_param(ctx, "profile", "snapshot plan", output) + config: Config = ctx.obj["config"] + console: Console = ctx.obj["console"] + snapshot_profile = _snapshot_profile_or_exit(config, profile, output) + try: + result = snapshot_plan(snapshot_profile) + render_output(result, output, "snapshot plan", success=not bool(result.get("alerts"))) + if output != "json": + console.print_json(data=result) + sys.exit(EXIT_SYSTEM_ERROR if result.get("alerts") else EXIT_SUCCESS) + except SnapshotError as exc: + json_error("snapshot plan", str(exc), EXIT_SYSTEM_ERROR, output) + + +@snapshot.command("init") +@click.option("--profile", "-p", required=False, help="Snapshot profile name") +@output_option +@input_json_option +@dry_run_option +@click.pass_context +def snapshot_init_cmd(ctx, profile, output, input_json, dry_run): + """Initialize the restic repository for a snapshot profile.""" + merge_json_input(ctx, input_json) + output = ctx.params.get("output", output) + dry_run = ctx.params.get("dry_run", dry_run) + profile = _snapshot_required_param(ctx, "profile", "snapshot init", output) + config: Config = ctx.obj["config"] + console: Console = ctx.obj["console"] + snapshot_profile = _snapshot_profile_or_exit(config, profile, output) + try: + _snapshot_result("snapshot init", snapshot_init(snapshot_profile, dry_run=dry_run), output, console) + except SnapshotError as exc: + json_error("snapshot init", str(exc), EXIT_SYSTEM_ERROR, output) + + +@snapshot.command("run") +@click.option("--profile", "-p", required=False, help="Snapshot profile name") +@output_option +@input_json_option +@dry_run_option +@click.pass_context +def snapshot_run_cmd(ctx, profile, output, input_json, dry_run): + """Run a native snapshot profile.""" + merge_json_input(ctx, input_json) + output = ctx.params.get("output", output) + dry_run = ctx.params.get("dry_run", dry_run) + profile = _snapshot_required_param(ctx, "profile", "snapshot run", output) + config: Config = ctx.obj["config"] + console: Console = ctx.obj["console"] + snapshot_profile = _snapshot_profile_or_exit(config, profile, output) + try: + _snapshot_result("snapshot run", snapshot_run(snapshot_profile, dry_run=dry_run), output, console) + except SnapshotError as exc: + json_error("snapshot run", str(exc), EXIT_SYSTEM_ERROR, output) + + +@snapshot.command("check") +@click.option("--profile", "-p", required=False, help="Snapshot profile name") +@click.option("--read-data-subset", default=None, help="Pass through to restic check") +@output_option +@input_json_option +@dry_run_option +@click.pass_context +def snapshot_check_cmd(ctx, profile, read_data_subset, output, input_json, dry_run): + """Check a native snapshot repository.""" + merge_json_input(ctx, input_json) + output = ctx.params.get("output", output) + dry_run = ctx.params.get("dry_run", dry_run) + read_data_subset = ctx.params.get("read_data_subset", read_data_subset) + profile = _snapshot_required_param(ctx, "profile", "snapshot check", output) + config: Config = ctx.obj["config"] + console: Console = ctx.obj["console"] + snapshot_profile = _snapshot_profile_or_exit(config, profile, output) + try: + result = snapshot_check(snapshot_profile, read_data_subset=read_data_subset, dry_run=dry_run) + _snapshot_result("snapshot check", result, output, console) + except SnapshotError as exc: + json_error("snapshot check", str(exc), EXIT_SYSTEM_ERROR, output) + + +@snapshot.command("restore") +@click.option("--profile", "-p", required=False, help="Snapshot profile name") +@click.option("--snapshot-id", required=False, help="Restic snapshot ID to restore") +@click.option("--target", required=False, type=click.Path(), help="Empty restore target directory") +@click.option("--include", "include_path", default=None, help="Optional restic include filter") +@output_option +@input_json_option +@dry_run_option +@click.pass_context +def snapshot_restore_cmd(ctx, profile, snapshot_id, target, include_path, output, input_json, dry_run): + """Restore a restic snapshot into a target directory.""" + merge_json_input(ctx, input_json) + output = ctx.params.get("output", output) + dry_run = ctx.params.get("dry_run", dry_run) + include_path = ctx.params.get("include_path", include_path) + profile = _snapshot_required_param(ctx, "profile", "snapshot restore", output) + snapshot_id = _snapshot_required_param(ctx, "snapshot_id", "snapshot restore", output) + target = _snapshot_required_param(ctx, "target", "snapshot restore", output) + config: Config = ctx.obj["config"] + console: Console = ctx.obj["console"] + snapshot_profile = _snapshot_profile_or_exit(config, profile, output) + target_path = expand_path(target) + if not dry_run and target_path.exists() and any(target_path.iterdir()): + json_error("snapshot restore", f"Restore target is not empty: {target}", EXIT_USER_ERROR, output) + try: + result = snapshot_restore( + snapshot_profile, + snapshot_id=snapshot_id, + target=target, + include=include_path, + dry_run=dry_run, + ) + _snapshot_result("snapshot restore", result, output, console) + except SnapshotError as exc: + json_error("snapshot restore", str(exc), EXIT_SYSTEM_ERROR, output) + + +@snapshot.command("retire") +@click.option("--profile", "-p", required=False, help="Snapshot profile name") +@click.option("--repo-id", required=False, help="Repo ID to mark retired") +@output_option +@input_json_option +@click.pass_context +def snapshot_retire_cmd(ctx, profile, repo_id, output, input_json): + """Manually mark a repo as retired after it has a successful snapshot.""" + merge_json_input(ctx, input_json) + output = ctx.params.get("output", output) + profile = _snapshot_required_param(ctx, "profile", "snapshot retire", output) + repo_id = _snapshot_required_param(ctx, "repo_id", "snapshot retire", output) + config: Config = ctx.obj["config"] + console: Console = ctx.obj["console"] + snapshot_profile = _snapshot_profile_or_exit(config, profile, output) + try: + result = retire_repo(snapshot_profile, repo_id) + _snapshot_result("snapshot retire", result, output, console) + except SnapshotError as exc: + json_error("snapshot retire", str(exc), EXIT_USER_ERROR, output) + + +@snapshot.command("purge-plan") +@click.option("--profile", "-p", required=False, help="Snapshot profile name") +@click.option("--repo-id", required=False, help="Retired repo ID to plan for purge") +@output_option +@input_json_option +@click.pass_context +def snapshot_purge_plan_cmd(ctx, profile, repo_id, output, input_json): + """Build a dry-run-first purge plan for a retired repo.""" + merge_json_input(ctx, input_json) + output = ctx.params.get("output", output) + profile = _snapshot_required_param(ctx, "profile", "snapshot purge-plan", output) + repo_id = _snapshot_required_param(ctx, "repo_id", "snapshot purge-plan", output) + config: Config = ctx.obj["config"] + console: Console = ctx.obj["console"] + snapshot_profile = _snapshot_profile_or_exit(config, profile, output) + try: + result = purge_plan(snapshot_profile, repo_id) + _snapshot_result("snapshot purge-plan", result, output, console) + except SnapshotError as exc: + json_error("snapshot purge-plan", str(exc), EXIT_USER_ERROR, output) + + +@snapshot.command("schedule") +@click.option("--profile", "-p", required=False, help="Snapshot profile name") +@output_option +@input_json_option +@click.pass_context +def snapshot_schedule_cmd(ctx, profile, output, input_json): + """Render user systemd units/timers for a snapshot profile.""" + merge_json_input(ctx, input_json) + output = ctx.params.get("output", output) + profile = _snapshot_required_param(ctx, "profile", "snapshot schedule", output) + config: Config = ctx.obj["config"] + console: Console = ctx.obj["console"] + snapshot_profile = _snapshot_profile_or_exit(config, profile, output) + result = {"profile": profile, "units": schedule_units(snapshot_profile, config_path=config.config_path)} + _snapshot_result("snapshot schedule", result, output, console) + + # --------------------------------------------------------------------------- # restore # --------------------------------------------------------------------------- @@ -498,8 +728,6 @@ def restore( if not backup_path.exists(): json_error("restore", f"Backup path does not exist: {backup_path}", EXIT_USER_ERROR, output) - restore_mgr = DockerRestore(config) - rename_map = {} if rename: for mapping in rename: @@ -541,6 +769,21 @@ def restore( output, ) + if filesystems_to_restore and fs_destination is None: + json_error( + "restore", + "Filesystem restore requires --filesystem-destination", + EXIT_USER_ERROR, + output, + ) + if filesystems_to_restore and len(filesystems_to_restore) > 1: + json_error( + "restore", + "Restore one filesystem target at a time; multiple filesystem targets cannot share one destination", + EXIT_USER_ERROR, + output, + ) + # Gap 9: dry-run if dry_run: plan = { @@ -558,24 +801,10 @@ def restore( console.print("[cyan]Dry-run: no restore executed.[/cyan]") sys.exit(EXIT_SUCCESS) - if filesystems_to_restore and fs_destination is None: - json_error( - "restore", - "Filesystem restore requires --filesystem-destination", - EXIT_USER_ERROR, - output, - ) - if filesystems_to_restore and len(filesystems_to_restore) > 1: - json_error( - "restore", - "Restore one filesystem target at a time; multiple filesystem targets cannot share one destination", - EXIT_USER_ERROR, - output, - ) - if output != "json": console.print(f"[bold]Restoring from backup: {backup_path}[/bold]\n") + restore_mgr = DockerRestore(config) results = restore_mgr.restore_backup( backup_path=backup_path, containers=containers_to_restore, @@ -747,8 +976,7 @@ def list_backups(ctx, backup_dir, skills, output, input_json): console: Console = ctx.obj["console"] backup_path = Path(backup_dir) if backup_dir else Path(config.get_staging_dir()) - restore_mgr = DockerRestore(config) - backups = restore_mgr.list_backups(backup_path) + backups = list_available_backups(backup_path) backups_data = [ { diff --git a/bbackup/cli_metadata.py b/bbackup/cli_metadata.py index 2552be3..0c3a03f 100644 --- a/bbackup/cli_metadata.py +++ b/bbackup/cli_metadata.py @@ -83,6 +83,107 @@ def _register_bbackup(cmd: CliCommand) -> None: BBACKUP_COMMANDS[cmd.id] = cmd +def _snapshot_common_parameters( + *, + profile_required: bool = True, + dry_run: bool = False, + repo_id: bool = False, + snapshot_id: bool = False, + target: bool = False, + include_path: bool = False, + read_data_subset: bool = False, +) -> List[Parameter]: + params = [ + Parameter( + name="profile", + kind="flag", + type="string", + description="Snapshot profile name from config.yaml.", + cli_flag="--profile", + json_key="profile", + required=profile_required, + ), + ] + if repo_id: + params.append(Parameter( + name="repo_id", + kind="flag", + type="string", + description="Tracked Git repository ID from the snapshot state ledger.", + cli_flag="--repo-id", + json_key="repo_id", + required=True, + )) + if snapshot_id: + params.append(Parameter( + name="snapshot_id", + kind="flag", + type="string", + description="Restic snapshot ID or snapshot:path selector to restore.", + cli_flag="--snapshot-id", + json_key="snapshot_id", + required=True, + )) + if target: + params.append(Parameter( + name="target", + kind="flag", + type="path", + description="Empty restore target directory.", + cli_flag="--target", + json_key="target", + required=True, + )) + if include_path: + params.append(Parameter( + name="include_path", + kind="flag", + type="string", + description="Optional restic include filter for targeted restore.", + cli_flag="--include", + json_key="include_path", + )) + if read_data_subset: + params.append(Parameter( + name="read_data_subset", + kind="flag", + type="string", + description="Optional restic check --read-data-subset value, such as 5%.", + cli_flag="--read-data-subset", + json_key="read_data_subset", + )) + if dry_run: + params.append(Parameter( + name="dry_run", + kind="flag", + type="bool", + description="Return the resolved restic plan without executing it.", + cli_flag="--dry-run", + json_key="dry_run", + default=False, + )) + params.extend([ + Parameter( + name="output", + kind="flag", + type="string", + description="Output format: text or json.", + cli_flag="--output", + json_key="output", + allowed_values=["text", "json"], + shape="enum", + ), + Parameter( + name="input_json", + kind="json_field", + type="object", + description="Flat JSON object providing all parameters.", + shape="object", + ), + ]) + return params + + _register_bbackup( CliCommand( cli="bbackup", @@ -259,6 +360,199 @@ def _register_bbackup(cmd: CliCommand) -> None: ) +_register_bbackup( + CliCommand( + cli="bbackup", + name="snapshot plan", + summary="Resolve a native snapshot profile without executing restic.", + description=( + "Discover Git repositories and explicit paths for a snapshot profile, " + "then return the restic commands and safety alerts without persisting state." + ), + category="snapshot", + parameters=_snapshot_common_parameters(), + examples=[ + Example( + description="Plan a native snapshot profile with JSON output.", + cli="bbackup snapshot plan --profile essentials-daily --output json", + input_json={"profile": "essentials-daily", "output": "json"}, + ), + ], + ) +) + + +_register_bbackup( + CliCommand( + cli="bbackup", + name="snapshot init", + summary="Initialize the restic repository for a snapshot profile.", + description=( + "Initialize the configured restic repository after profile safety preflight. " + "Use dry-run mode to inspect the exact restic init command first." + ), + category="snapshot", + parameters=_snapshot_common_parameters(dry_run=True), + examples=[ + Example( + description="Inspect the restic init command without executing it.", + cli="bbackup snapshot init --profile essentials-daily --dry-run --output json", + input_json={"profile": "essentials-daily", "dry_run": True, "output": "json"}, + ), + ], + ) +) + + +_register_bbackup( + CliCommand( + cli="bbackup", + name="snapshot run", + summary="Run a native restic snapshot profile.", + description=( + "Discover active Git repositories and configured paths, enforce safety " + "preflight for non-dry runs, run restic backup, and record successful " + "repository snapshot IDs in the state ledger." + ), + category="snapshot", + parameters=_snapshot_common_parameters(dry_run=True), + examples=[ + Example( + description="Dry-run a profile before taking a snapshot.", + cli="bbackup snapshot run --profile essentials-daily --dry-run --output json", + input_json={"profile": "essentials-daily", "dry_run": True, "output": "json"}, + ), + ], + ) +) + + +_register_bbackup( + CliCommand( + cli="bbackup", + name="snapshot check", + summary="Run a non-destructive restic repository check.", + description=( + "Check a native snapshot repository, optionally with restic " + "--read-data-subset for verification schedules." + ), + category="snapshot", + parameters=_snapshot_common_parameters(dry_run=True, read_data_subset=True), + examples=[ + Example( + description="Run a lightweight repository check.", + cli="bbackup snapshot check --profile essentials-daily --output json", + input_json={"profile": "essentials-daily", "output": "json"}, + ), + Example( + description="Run a verification check over a subset of pack data.", + cli="bbackup snapshot check --profile essentials-daily --read-data-subset 5% --output json", + input_json={"profile": "essentials-daily", "read_data_subset": "5%", "output": "json"}, + ), + ], + ) +) + + +_register_bbackup( + CliCommand( + cli="bbackup", + name="snapshot restore", + summary="Restore a restic snapshot into an empty target directory.", + description=( + "Restore a snapshot or snapshot:path selector into an empty target. " + "Dry-run mode returns the restic restore command without contacting Docker." + ), + category="snapshot", + parameters=_snapshot_common_parameters( + dry_run=True, + snapshot_id=True, + target=True, + include_path=True, + ), + examples=[ + Example( + description="Plan a targeted restore from a snapshot.", + cli="bbackup snapshot restore --profile essentials-daily --snapshot-id latest --target /tmp/restore --dry-run --output json", + input_json={ + "profile": "essentials-daily", + "snapshot_id": "latest", + "target": "/tmp/restore", + "dry_run": True, + "output": "json", + }, + ), + ], + ) +) + + +_register_bbackup( + CliCommand( + cli="bbackup", + name="snapshot retire", + summary="Mark a tracked repository retired after it has a successful snapshot.", + description=( + "Retire a repository in the snapshot state ledger so future purge planning " + "can remain explicit and dry-run-first." + ), + category="snapshot", + parameters=_snapshot_common_parameters(repo_id=True), + examples=[ + Example( + description="Retire a repository ID after verifying it has a successful snapshot.", + cli="bbackup snapshot retire --profile essentials-daily --repo-id abc123 --output json", + input_json={"profile": "essentials-daily", "repo_id": "abc123", "output": "json"}, + ), + ], + ) +) + + +_register_bbackup( + CliCommand( + cli="bbackup", + name="snapshot purge-plan", + summary="Build a dry-run-first purge plan for a retired repository.", + description=( + "Return restic forget arguments for a retired repo ID. The command keeps " + "destructive cleanup manual by exposing dry-run and post-confirmation arguments separately." + ), + category="snapshot", + parameters=_snapshot_common_parameters(repo_id=True), + examples=[ + Example( + description="Prepare a manual purge plan for a retired repository.", + cli="bbackup snapshot purge-plan --profile essentials-daily --repo-id abc123 --output json", + input_json={"profile": "essentials-daily", "repo_id": "abc123", "output": "json"}, + ), + ], + ) +) + + +_register_bbackup( + CliCommand( + cli="bbackup", + name="snapshot schedule", + summary="Render user systemd units and timers for a snapshot profile.", + description=( + "Render matching service/timer units for daily snapshot runs, weekly " + "non-destructive checks, and monthly verification checks." + ), + category="snapshot", + parameters=_snapshot_common_parameters(), + examples=[ + Example( + description="Render schedule units for manual installation.", + cli="bbackup snapshot schedule --profile essentials-daily --output json", + input_json={"profile": "essentials-daily", "output": "json"}, + ), + ], + ) +) + + _register_bbackup( CliCommand( cli="bbackup", diff --git a/bbackup/config.py b/bbackup/config.py index a7f485c..74baad9 100644 --- a/bbackup/config.py +++ b/bbackup/config.py @@ -45,6 +45,27 @@ class FilesystemBackupSet: targets: List[FilesystemTarget] = field(default_factory=list) +@dataclass +class SnapshotProfile: + """Native snapshot backup profile.""" + name: str + engine: str = "restic" + repository: str = "" + password_file: str = "" + cache_dir: str = "" + state_dir: str = "" + host_id: str = "" + repo_homes: List[str] = field(default_factory=list) + explicit_repos: List[str] = field(default_factory=list) + include_paths: List[str] = field(default_factory=list) + exclude_paths: List[str] = field(default_factory=list) + retention: Dict[str, Any] = field(default_factory=dict) + schedule: Dict[str, Any] = field(default_factory=dict) + retry_lock: str = "5m" + tags: List[str] = field(default_factory=list) + allow_default_rclone_drive_client: bool = False + + RCLONE_OPTIONS_CAP = 32 RCLONE_DEFAULT_TRANSFERS = 8 RCLONE_DEFAULT_CHECKERS = 8 @@ -133,6 +154,7 @@ def __init__(self, config_path: Optional[str] = None): self.data: Dict[str, Any] = {} self.backup_sets: Dict[str, BackupSet] = {} self.filesystem_sets: Dict[str, FilesystemBackupSet] = {} + self.snapshot_profiles: Dict[str, SnapshotProfile] = {} self.remotes: Dict[str, RemoteStorage] = {} self.retention = RetentionPolicy() self.incremental = IncrementalSettings() @@ -249,6 +271,33 @@ def _parse_config(self): targets=targets, ) + # Parse native snapshot profiles + for profile_name, profile_data in self.data.get("snapshot_profiles", {}).items(): + if profile_data is None: + profile_data = {} + if not isinstance(profile_data, dict): + raise ValueError(f"Snapshot profile '{profile_name}' must be a mapping") + self.snapshot_profiles[profile_name] = SnapshotProfile( + name=profile_name, + engine=profile_data.get("engine", "restic"), + repository=profile_data.get("repository", ""), + password_file=profile_data.get("password_file", ""), + cache_dir=profile_data.get("cache_dir", ""), + state_dir=profile_data.get("state_dir", ""), + host_id=profile_data.get("host_id", ""), + repo_homes=list(profile_data.get("repo_homes", []) or []), + explicit_repos=list(profile_data.get("explicit_repos", []) or []), + include_paths=list(profile_data.get("include_paths", []) or []), + exclude_paths=list(profile_data.get("exclude_paths", []) or []), + retention=dict(profile_data.get("retention", {}) or {}), + schedule=dict(profile_data.get("schedule", {}) or {}), + retry_lock=str(profile_data.get("retry_lock", "5m")), + tags=list(profile_data.get("tags", []) or []), + allow_default_rclone_drive_client=( + profile_data.get("allow_default_rclone_drive_client") is True + ), + ) + # Parse top-level rclone default options if "rclone" in self.data: rclone_data = self.data["rclone"] @@ -355,6 +404,10 @@ def get_backup_compression(self) -> Dict[str, Any]: def get_backup_set(self, name: str) -> Optional[BackupSet]: """Get backup set by name.""" return self.backup_sets.get(name) + + def get_snapshot_profile(self, name: str) -> Optional[SnapshotProfile]: + """Get native snapshot profile by name.""" + return self.snapshot_profiles.get(name) def get_enabled_remotes(self) -> List[RemoteStorage]: """Get list of enabled remote storage destinations.""" diff --git a/bbackup/data/cli-skills-index.json b/bbackup/data/cli-skills-index.json index fb222d6..f815a1a 100644 --- a/bbackup/data/cli-skills-index.json +++ b/bbackup/data/cli-skills-index.json @@ -36,55 +36,87 @@ "start": 296 }, "bbackup:skills": { - "end": 380, + "end": 378, "start": 351 }, + "bbackup:snapshot check": { + "end": 421, + "start": 379 + }, + "bbackup:snapshot init": { + "end": 453, + "start": 422 + }, + "bbackup:snapshot plan": { + "end": 484, + "start": 454 + }, + "bbackup:snapshot purge-plan": { + "end": 516, + "start": 485 + }, + "bbackup:snapshot restore": { + "end": 551, + "start": 517 + }, + "bbackup:snapshot retire": { + "end": 583, + "start": 552 + }, + "bbackup:snapshot run": { + "end": 615, + "start": 584 + }, + "bbackup:snapshot schedule": { + "end": 648, + "start": 616 + }, "bbman:check-deps": { - "end": 422, - "start": 381 + "end": 690, + "start": 649 }, "bbman:check-updates": { - "end": 454, - "start": 423 + "end": 722, + "start": 691 }, "bbman:cleanup": { - "end": 490, - "start": 455 + "end": 758, + "start": 723 }, "bbman:diagnostics": { - "end": 522, - "start": 491 + "end": 790, + "start": 759 }, "bbman:health": { - "end": 553, - "start": 523 + "end": 821, + "start": 791 }, "bbman:repo-url": { - "end": 585, - "start": 554 + "end": 853, + "start": 822 }, "bbman:run": { - "end": 606, - "start": 586 + "end": 874, + "start": 854 }, "bbman:setup": { - "end": 637, - "start": 607 + "end": 905, + "start": 875 }, "bbman:skills": { - "end": 665, - "start": 638 + "end": 933, + "start": 906 }, "bbman:status": { - "end": 696, - "start": 666 + "end": 964, + "start": 934 }, "bbman:update": { - "end": 730, - "start": 697 + "end": 998, + "start": 965 }, "bbman:validate-config": { - "end": 761, - "start": 731 + "end": 1029, + "start": 999 } } diff --git a/bbackup/data/cli-skills.md b/bbackup/data/cli-skills.md index 16f37dc..e96750b 100644 --- a/bbackup/data/cli-skills.md +++ b/bbackup/data/cli-skills.md @@ -1,6 +1,6 @@ # CLI skills catalog -> Generated from the bbackup/bbman CLI metadata. Version: 1.8.4. This catalog is authoritative for this version. +> Generated from the bbackup/bbman CLI metadata. Version: 1.8.5. This catalog is authoritative for this version. ## bbackup @@ -376,6 +376,274 @@ List or inspect bbackup skills in JSON or Markdown formats. bbackup skills --format markdown ``` +### bbackup snapshot check + +**Summary**: Run a non-destructive restic repository check. + +Check a native snapshot repository, optionally with restic --read-data-subset for verification schedules. + +#### CLI parameters + +| Name | Type | Required | Default | Description | +|---|---|:---:|---|---| +| `--profile` | `string` | yes | `` | Snapshot profile name from config.yaml. | +| `--read-data-subset` | `string` | no | `` | Optional restic check --read-data-subset value, such as 5%. | +| `--dry-run` | `bool` | no | `False` | Return the resolved restic plan without executing it. | +| `--output` | `string` | no | `` | Output format: text or json. | + +#### JSON / environment parameters + +| Name | Kind | Type | Required | Default | Description | +|---|---|---|:---:|---|---| +| `input_json` | json | `object` | no | `` | Flat JSON object providing all parameters. | + +#### Examples + +- Run a lightweight repository check. + + ```bash + bbackup snapshot check --profile essentials-daily --output json + ``` + + ```bash + bbackup snapshot check --input-json '{"profile":"essentials-daily","output":"json"}' --output json + ``` + +- Run a verification check over a subset of pack data. + + ```bash + bbackup snapshot check --profile essentials-daily --read-data-subset 5% --output json + ``` + + ```bash + bbackup snapshot check --input-json '{"profile":"essentials-daily","read_data_subset":"5%","output":"json"}' --output json + ``` + +### bbackup snapshot init + +**Summary**: Initialize the restic repository for a snapshot profile. + +Initialize the configured restic repository after profile safety preflight. Use dry-run mode to inspect the exact restic init command first. + +#### CLI parameters + +| Name | Type | Required | Default | Description | +|---|---|:---:|---|---| +| `--profile` | `string` | yes | `` | Snapshot profile name from config.yaml. | +| `--dry-run` | `bool` | no | `False` | Return the resolved restic plan without executing it. | +| `--output` | `string` | no | `` | Output format: text or json. | + +#### JSON / environment parameters + +| Name | Kind | Type | Required | Default | Description | +|---|---|---|:---:|---|---| +| `input_json` | json | `object` | no | `` | Flat JSON object providing all parameters. | + +#### Examples + +- Inspect the restic init command without executing it. + + ```bash + bbackup snapshot init --profile essentials-daily --dry-run --output json + ``` + + ```bash + bbackup snapshot init --input-json '{"profile":"essentials-daily","dry_run":true,"output":"json"}' --output json + ``` + +### bbackup snapshot plan + +**Summary**: Resolve a native snapshot profile without executing restic. + +Discover Git repositories and explicit paths for a snapshot profile, then return the restic commands and safety alerts without persisting state. + +#### CLI parameters + +| Name | Type | Required | Default | Description | +|---|---|:---:|---|---| +| `--profile` | `string` | yes | `` | Snapshot profile name from config.yaml. | +| `--output` | `string` | no | `` | Output format: text or json. | + +#### JSON / environment parameters + +| Name | Kind | Type | Required | Default | Description | +|---|---|---|:---:|---|---| +| `input_json` | json | `object` | no | `` | Flat JSON object providing all parameters. | + +#### Examples + +- Plan a native snapshot profile with JSON output. + + ```bash + bbackup snapshot plan --profile essentials-daily --output json + ``` + + ```bash + bbackup snapshot plan --input-json '{"profile":"essentials-daily","output":"json"}' --output json + ``` + +### bbackup snapshot purge-plan + +**Summary**: Build a dry-run-first purge plan for a retired repository. + +Return restic forget arguments for a retired repo ID. The command keeps destructive cleanup manual by exposing dry-run and post-confirmation arguments separately. + +#### CLI parameters + +| Name | Type | Required | Default | Description | +|---|---|:---:|---|---| +| `--profile` | `string` | yes | `` | Snapshot profile name from config.yaml. | +| `--repo-id` | `string` | yes | `` | Tracked Git repository ID from the snapshot state ledger. | +| `--output` | `string` | no | `` | Output format: text or json. | + +#### JSON / environment parameters + +| Name | Kind | Type | Required | Default | Description | +|---|---|---|:---:|---|---| +| `input_json` | json | `object` | no | `` | Flat JSON object providing all parameters. | + +#### Examples + +- Prepare a manual purge plan for a retired repository. + + ```bash + bbackup snapshot purge-plan --profile essentials-daily --repo-id abc123 --output json + ``` + + ```bash + bbackup snapshot purge-plan --input-json '{"profile":"essentials-daily","repo_id":"abc123","output":"json"}' --output json + ``` + +### bbackup snapshot restore + +**Summary**: Restore a restic snapshot into an empty target directory. + +Restore a snapshot or snapshot:path selector into an empty target. Dry-run mode returns the restic restore command without contacting Docker. + +#### CLI parameters + +| Name | Type | Required | Default | Description | +|---|---|:---:|---|---| +| `--profile` | `string` | yes | `` | Snapshot profile name from config.yaml. | +| `--snapshot-id` | `string` | yes | `` | Restic snapshot ID or snapshot:path selector to restore. | +| `--target` | `path` | yes | `` | Empty restore target directory. | +| `--include` | `string` | no | `` | Optional restic include filter for targeted restore. | +| `--dry-run` | `bool` | no | `False` | Return the resolved restic plan without executing it. | +| `--output` | `string` | no | `` | Output format: text or json. | + +#### JSON / environment parameters + +| Name | Kind | Type | Required | Default | Description | +|---|---|---|:---:|---|---| +| `input_json` | json | `object` | no | `` | Flat JSON object providing all parameters. | + +#### Examples + +- Plan a targeted restore from a snapshot. + + ```bash + bbackup snapshot restore --profile essentials-daily --snapshot-id latest --target /tmp/restore --dry-run --output json + ``` + + ```bash + bbackup snapshot restore --input-json '{"profile":"essentials-daily","snapshot_id":"latest","target":"/tmp/restore","dry_run":true,"output":"json"}' --output json + ``` + +### bbackup snapshot retire + +**Summary**: Mark a tracked repository retired after it has a successful snapshot. + +Retire a repository in the snapshot state ledger so future purge planning can remain explicit and dry-run-first. + +#### CLI parameters + +| Name | Type | Required | Default | Description | +|---|---|:---:|---|---| +| `--profile` | `string` | yes | `` | Snapshot profile name from config.yaml. | +| `--repo-id` | `string` | yes | `` | Tracked Git repository ID from the snapshot state ledger. | +| `--output` | `string` | no | `` | Output format: text or json. | + +#### JSON / environment parameters + +| Name | Kind | Type | Required | Default | Description | +|---|---|---|:---:|---|---| +| `input_json` | json | `object` | no | `` | Flat JSON object providing all parameters. | + +#### Examples + +- Retire a repository ID after verifying it has a successful snapshot. + + ```bash + bbackup snapshot retire --profile essentials-daily --repo-id abc123 --output json + ``` + + ```bash + bbackup snapshot retire --input-json '{"profile":"essentials-daily","repo_id":"abc123","output":"json"}' --output json + ``` + +### bbackup snapshot run + +**Summary**: Run a native restic snapshot profile. + +Discover active Git repositories and configured paths, enforce safety preflight for non-dry runs, run restic backup, and record successful repository snapshot IDs in the state ledger. + +#### CLI parameters + +| Name | Type | Required | Default | Description | +|---|---|:---:|---|---| +| `--profile` | `string` | yes | `` | Snapshot profile name from config.yaml. | +| `--dry-run` | `bool` | no | `False` | Return the resolved restic plan without executing it. | +| `--output` | `string` | no | `` | Output format: text or json. | + +#### JSON / environment parameters + +| Name | Kind | Type | Required | Default | Description | +|---|---|---|:---:|---|---| +| `input_json` | json | `object` | no | `` | Flat JSON object providing all parameters. | + +#### Examples + +- Dry-run a profile before taking a snapshot. + + ```bash + bbackup snapshot run --profile essentials-daily --dry-run --output json + ``` + + ```bash + bbackup snapshot run --input-json '{"profile":"essentials-daily","dry_run":true,"output":"json"}' --output json + ``` + +### bbackup snapshot schedule + +**Summary**: Render user systemd units and timers for a snapshot profile. + +Render matching service/timer units for daily snapshot runs, weekly non-destructive checks, and monthly verification checks. + +#### CLI parameters + +| Name | Type | Required | Default | Description | +|---|---|:---:|---|---| +| `--profile` | `string` | yes | `` | Snapshot profile name from config.yaml. | +| `--output` | `string` | no | `` | Output format: text or json. | + +#### JSON / environment parameters + +| Name | Kind | Type | Required | Default | Description | +|---|---|---|:---:|---|---| +| `input_json` | json | `object` | no | `` | Flat JSON object providing all parameters. | + +#### Examples + +- Render schedule units for manual installation. + + ```bash + bbackup snapshot schedule --profile essentials-daily --output json + ``` + + ```bash + bbackup snapshot schedule --input-json '{"profile":"essentials-daily","output":"json"}' --output json + ``` + ## bbman ### bbman check-deps diff --git a/bbackup/data/config.yaml.example b/bbackup/data/config.yaml.example index 3658ed9..1b5aa4c 100644 --- a/bbackup/data/config.yaml.example +++ b/bbackup/data/config.yaml.example @@ -186,6 +186,62 @@ filesystem: excludes: - "*.log" +# Native deduplicating snapshot profiles. +# Restic encrypts content and metadata. Keep password files outside selected +# backup paths, chmod 0600, and escrow the password out-of-band. +snapshot_profiles: + essentials-daily: + engine: restic + host_id: WORKSTATION01 + repository: rclone:my-drive:backups/WORKSTATION01/restic/essentials-daily + # Prefer a dedicated Google Drive rclone OAuth client. Set this only as an + # explicit YAML boolean true when accepting rclone's shared default client. + allow_default_rclone_drive_client: false + cache_dir: ~/.cache/bbackup/restic/WORKSTATION01/essentials-daily + state_dir: ~/.local/state/bbackup/WORKSTATION01/essentials-daily + password_file: ~/.local/share/bbackup-credentials/restic/WORKSTATION01/essentials-daily.password + retry_lock: 5m + repo_homes: + - ~/Projects + - ~/Work + explicit_repos: + - ~/workstation-control + include_paths: + - ~/.codex + - ~/.config/my-agent + - ~/.local/share/my-agent + - ~/.local/state/my-agent + - ~/.config/bbackup + - ~/.config/rclone + - ~/.config/systemd/user + - ~/.ssh + - ~/.pki + - ~/.gitconfig + - ~/.bashrc + - ~/.profile + - ~/.tmux.conf + - ~/.config/gh + - ~/Documents + - ~/Pictures + exclude_paths: + - ~/Projects/archives + - ~/Projects/starred-repos + - ~/Projects/migration-staging + - ~/CloudDrive + - ~/.config/bbackup/restic + - ~/.cache + - "**/node_modules" + - "**/.venv" + retention: + active_repo_daily: 14 + active_repo_weekly: 8 + active_repo_monthly: 12 + schedule: + daily_time: "03:30" + maintenance_time: "Sun 04:30" + verification_time: "monthly" + verification_read_data_subset: "5%" + # Docker Settings docker: socket: /var/run/docker.sock diff --git a/bbackup/management/health.py b/bbackup/management/health.py index 4c69b76..07e63ea 100644 --- a/bbackup/management/health.py +++ b/bbackup/management/health.py @@ -12,6 +12,7 @@ from rich.table import Table from ..config import Config +from ..snapshot import check_all_snapshot_profiles console = Console() @@ -143,6 +144,18 @@ def run_health_check() -> Dict: Returns: Dict with health check results """ + config_error = "" + try: + config = Config() + snapshot_profiles = check_all_snapshot_profiles(config) + except Exception as exc: + config = None + config_error = str(exc) + snapshot_profiles = { + "ok": False, + "profiles": {}, + "error": f"Could not load config for snapshot health checks: {exc}", + } results = { "docker": check_docker(), "docker_socket": check_docker_socket(), @@ -152,7 +165,10 @@ def run_health_check() -> Dict: "python_packages": check_python_packages(), "config": check_config_file(), "directories": check_directories(), + "snapshot_profiles": snapshot_profiles, } + if config_error and results["config"][0]: + results["config"] = (False, f"Config file invalid: {config_error}") # Calculate overall health critical_checks = [ @@ -163,7 +179,8 @@ def run_health_check() -> Dict: results["python_packages"][0], ] - all_critical_ok = all(critical_checks) + snapshot_ok = results["snapshot_profiles"]["ok"] + all_critical_ok = all(critical_checks) and snapshot_ok overall_health = "healthy" if all_critical_ok else "unhealthy" results["overall"] = overall_health @@ -213,6 +230,17 @@ def display_health_report(results: Dict): else: for issue in dir_issues: table.add_row("Directories", "[yellow]⚠[/yellow]", issue) + + # Snapshot profile checks + snapshot_profiles = results.get("snapshot_profiles", {}).get("profiles", {}) + for profile_name, profile_result in snapshot_profiles.items(): + status = "[green]✓[/green]" if profile_result.get("ok") else "[red]✗[/red]" + failed = [ + name for name, check in profile_result.get("checks", {}).items() + if not check.get("ok") + ] + detail = "ready" if not failed else f"failed: {', '.join(failed)}" + table.add_row(f"Snapshot profile: {profile_name}", status, detail) console.print(table) @@ -246,6 +274,11 @@ def generate_health_report(results: Dict) -> str: msg = result[1] status = "✓" if ok else "✗" report.append(f"{status} {check_name}: {msg}") + + snapshot_profiles = results.get("snapshot_profiles", {}).get("profiles", {}) + for profile_name, profile_result in snapshot_profiles.items(): + status = "✓" if profile_result.get("ok") else "✗" + report.append(f"{status} Snapshot profile {profile_name}: {'ready' if profile_result.get('ok') else 'not ready'}") report.append("") report.append(f"Overall Status: {results['overall'].upper()}") diff --git a/bbackup/restore.py b/bbackup/restore.py index 4b80a9c..21a5a12 100644 --- a/bbackup/restore.py +++ b/bbackup/restore.py @@ -52,6 +52,41 @@ def list_volume_backup_names(backup_path: Path) -> List[str]: return sorted(names) +def list_available_backups(backup_dir: Path) -> List[Dict]: + """List available local backups in a directory without connecting to Docker.""" + backups = [] + if not backup_dir.exists(): + return backups + items = list(backup_dir.iterdir()) + for backup_path in sorted(items, key=lambda x: x.stat().st_mtime if x.exists() else 0, reverse=True): + if backup_path.is_dir() and backup_path.name.startswith("backup_"): + metadata_file = backup_path / "backup_metadata.json" + timestamp = None + if metadata_file.exists(): + try: + with open(metadata_file, 'r') as f: + metadata = json.load(f) + timestamp = metadata.get("timestamp") + except Exception: + pass + if not timestamp: + try: + name_part = backup_path.name.replace("backup_", "") + timestamp = datetime.strptime(name_part, "%Y%m%d_%H%M%S").isoformat() + except Exception: + timestamp = backup_path.name + backups.append({"name": backup_path.name, "path": backup_path, "timestamp": timestamp}) + elif backup_path.is_file() and is_solid_archive_name(backup_path.name): + try: + base = strip_solid_archive_suffix(backup_path.name) + name_part = base.replace("backup_", "") + timestamp = datetime.strptime(name_part, "%Y%m%d_%H%M%S").isoformat() + except Exception: + timestamp = backup_path.name + backups.append({"name": backup_path.name, "path": backup_path, "timestamp": timestamp}) + return backups + + class DockerRestore: """Docker restore manager.""" @@ -66,37 +101,7 @@ def __init__(self, config: Config): def list_backups(self, backup_dir: Path) -> List[Dict]: """List available backups in backup directory (dirs and solid archive files).""" - backups = [] - if not backup_dir.exists(): - return backups - items = list(backup_dir.iterdir()) - for backup_path in sorted(items, key=lambda x: x.stat().st_mtime if x.exists() else 0, reverse=True): - if backup_path.is_dir() and backup_path.name.startswith("backup_"): - metadata_file = backup_path / "backup_metadata.json" - timestamp = None - if metadata_file.exists(): - try: - with open(metadata_file, 'r') as f: - metadata = json.load(f) - timestamp = metadata.get("timestamp") - except Exception: - pass - if not timestamp: - try: - name_part = backup_path.name.replace("backup_", "") - timestamp = datetime.strptime(name_part, "%Y%m%d_%H%M%S").isoformat() - except Exception: - timestamp = backup_path.name - backups.append({"name": backup_path.name, "path": backup_path, "timestamp": timestamp}) - elif backup_path.is_file() and is_solid_archive_name(backup_path.name): - try: - base = strip_solid_archive_suffix(backup_path.name) - name_part = base.replace("backup_", "") - timestamp = datetime.strptime(name_part, "%Y%m%d_%H%M%S").isoformat() - except Exception: - timestamp = backup_path.name - backups.append({"name": backup_path.name, "path": backup_path, "timestamp": timestamp}) - return backups + return list_available_backups(backup_dir) def restore_container_config(self, container_name: str, backup_path: Path, new_name: Optional[str] = None) -> bool: """Restore container from backup configuration.""" diff --git a/bbackup/skills.py b/bbackup/skills.py index dd79f1c..4992f7c 100644 --- a/bbackup/skills.py +++ b/bbackup/skills.py @@ -374,6 +374,209 @@ "exit_codes": {0: "success", 1: "user error", 3: "system error"}, }, + "snapshot-workflow": { + "id": "snapshot-workflow", + "summary": "Plan, initialize, run, verify, restore, and retire native restic snapshot backups.", + "common": True, + "workflow": [ + "snapshot plan", + "snapshot init", + "snapshot run", + "snapshot check", + "snapshot restore", + "snapshot retire", + "snapshot purge-plan", + "snapshot schedule", + ], + "steps": [ + { + "command": "bbackup snapshot plan --profile --output json", + "description": "Resolve a configured snapshot profile without executing restic.", + "required_flags": ["--profile"], + "optional_flags": { + "--profile": "snapshot profile name from config", + "--output": "text or json", + "--input-json": "all params as flat JSON object", + }, + "valid_values": {"--output": ["text", "json"]}, + "input_json_schema": { + "type": "object", + "properties": { + "profile": {"type": "string"}, + "output": {"type": "string", "enum": ["text", "json"]}, + }, + "required": ["profile"], + }, + }, + { + "command": "bbackup snapshot init --profile --dry-run --output json", + "description": "Initialize the restic repository for a snapshot profile; dry-run previews the operation.", + "required_flags": ["--profile"], + "optional_flags": { + "--profile": "snapshot profile name from config", + "--dry-run": "return the planned operation without executing", + "--output": "text or json", + "--input-json": "all params as flat JSON object", + }, + "valid_values": {"--output": ["text", "json"]}, + "input_json_schema": { + "type": "object", + "properties": { + "profile": {"type": "string"}, + "dry_run": {"type": "boolean", "default": False}, + "output": {"type": "string", "enum": ["text", "json"]}, + }, + "required": ["profile"], + }, + }, + { + "command": "bbackup snapshot run --profile --output json", + "description": "Run the native deduplicating snapshot backup for a configured profile.", + "required_flags": ["--profile"], + "optional_flags": { + "--profile": "snapshot profile name from config", + "--dry-run": "return the planned operation without executing", + "--output": "text or json", + "--input-json": "all params as flat JSON object", + }, + "valid_values": {"--output": ["text", "json"]}, + "input_json_schema": { + "type": "object", + "properties": { + "profile": {"type": "string"}, + "dry_run": {"type": "boolean", "default": False}, + "output": {"type": "string", "enum": ["text", "json"]}, + }, + "required": ["profile"], + }, + }, + { + "command": "bbackup snapshot check --profile --read-data-subset 5% --output json", + "description": "Check a snapshot repository, optionally checking a subset of stored data.", + "required_flags": ["--profile"], + "optional_flags": { + "--profile": "snapshot profile name from config", + "--read-data-subset": "pass through to restic check", + "--dry-run": "return the planned operation without executing", + "--output": "text or json", + "--input-json": "all params as flat JSON object", + }, + "valid_values": {"--output": ["text", "json"]}, + "input_json_schema": { + "type": "object", + "properties": { + "profile": {"type": "string"}, + "read_data_subset": {"type": "string"}, + "dry_run": {"type": "boolean", "default": False}, + "output": {"type": "string", "enum": ["text", "json"]}, + }, + "required": ["profile"], + }, + }, + { + "command": "bbackup snapshot restore --profile --snapshot-id --target --output json", + "description": "Restore a restic snapshot into an empty target directory.", + "required_flags": ["--profile", "--snapshot-id", "--target"], + "optional_flags": { + "--profile": "snapshot profile name from config", + "--snapshot-id": "restic snapshot ID to restore", + "--target": "empty restore target directory", + "--include": "optional restic include filter", + "--dry-run": "return the planned operation without executing", + "--output": "text or json", + "--input-json": "all params as flat JSON object", + }, + "valid_values": {"--output": ["text", "json"]}, + "input_json_schema": { + "type": "object", + "properties": { + "profile": {"type": "string"}, + "snapshot_id": {"type": "string"}, + "target": {"type": "string"}, + "include_path": {"type": "string"}, + "dry_run": {"type": "boolean", "default": False}, + "output": {"type": "string", "enum": ["text", "json"]}, + }, + "required": ["profile", "snapshot_id", "target"], + }, + }, + { + "command": "bbackup snapshot retire --profile --repo-id --output json", + "description": "Mark a repo as retired after it has at least one successful snapshot.", + "required_flags": ["--profile", "--repo-id"], + "optional_flags": { + "--profile": "snapshot profile name from config", + "--repo-id": "repo ID to mark retired", + "--output": "text or json", + "--input-json": "all params as flat JSON object", + }, + "valid_values": {"--output": ["text", "json"]}, + "input_json_schema": { + "type": "object", + "properties": { + "profile": {"type": "string"}, + "repo_id": {"type": "string"}, + "output": {"type": "string", "enum": ["text", "json"]}, + }, + "required": ["profile", "repo_id"], + }, + }, + { + "command": "bbackup snapshot purge-plan --profile --repo-id --output json", + "description": "Build a dry-run-first purge plan for retired snapshot storage.", + "required_flags": ["--profile", "--repo-id"], + "optional_flags": { + "--profile": "snapshot profile name from config", + "--repo-id": "retired repo ID to plan for purge", + "--output": "text or json", + "--input-json": "all params as flat JSON object", + }, + "valid_values": {"--output": ["text", "json"]}, + "input_json_schema": { + "type": "object", + "properties": { + "profile": {"type": "string"}, + "repo_id": {"type": "string"}, + "output": {"type": "string", "enum": ["text", "json"]}, + }, + "required": ["profile", "repo_id"], + }, + }, + { + "command": "bbackup snapshot schedule --profile --output json", + "description": "Render user systemd service and timer units for a snapshot profile.", + "required_flags": ["--profile"], + "optional_flags": { + "--profile": "snapshot profile name from config", + "--output": "text or json", + "--input-json": "all params as flat JSON object", + }, + "valid_values": {"--output": ["text", "json"]}, + "input_json_schema": { + "type": "object", + "properties": { + "profile": {"type": "string"}, + "output": {"type": "string", "enum": ["text", "json"]}, + }, + "required": ["profile"], + }, + }, + ], + "examples": [ + "bbackup snapshot plan --profile essentials-daily --output json", + "bbackup snapshot init --profile essentials-daily --dry-run --output json", + "bbackup snapshot run --profile essentials-daily --output json", + "bbackup snapshot check --profile essentials-daily --read-data-subset 5% --output json", + "bbackup snapshot restore --profile essentials-daily --snapshot-id abc123 --target /tmp/restore --dry-run --output json", + "bbackup snapshot schedule --profile essentials-daily --output json", + ], + "output_format": ( + '{"profile": "...", "repository": "...", "actions": [...], ' + '"alerts": [], "results": [...], "units": {...}}' + ), + "exit_codes": {0: "success", 1: "user error", 3: "system error"}, + }, + "remote-storage": { "id": "remote-storage", "summary": "List backups stored on remote storage destinations.", diff --git a/bbackup/snapshot.py b/bbackup/snapshot.py new file mode 100644 index 0000000..2961a2b --- /dev/null +++ b/bbackup/snapshot.py @@ -0,0 +1,787 @@ +"""Native restic snapshot support for bbackup.""" + +from __future__ import annotations + +import hashlib +import json +import os +import shlex +import shutil +import socket +import stat +import subprocess +from fnmatch import fnmatch +from dataclasses import dataclass +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Dict, Iterable, List, Optional + +from .config import Config, SnapshotProfile + + +STATE_FILE_NAME = "snapshot-state.json" +STATE_SCHEMA_VERSION = 1 + + +class SnapshotError(RuntimeError): + """Raised when a snapshot profile cannot be executed safely.""" + + +@dataclass +class DiscoveredRepo: + path: str + fingerprint: str + repo_id: str + origin: str = "" + head: str = "" + legacy_fingerprint: str = "" + legacy_repo_id: str = "" + + +def expand_path(path: str) -> Path: + return Path(os.path.expandvars(os.path.expanduser(path))).resolve() + + +def utc_now() -> str: + return datetime.now(timezone.utc).replace(microsecond=0).isoformat() + + +def load_state(profile: SnapshotProfile) -> Dict[str, Any]: + state_path = state_file(profile) + if not state_path.exists(): + return { + "schema_version": STATE_SCHEMA_VERSION, + "profile": profile.name, + "repos": {}, + "alerts": [], + } + with state_path.open("r", encoding="utf-8") as f: + state = json.load(f) or {} + state.setdefault("schema_version", STATE_SCHEMA_VERSION) + state.setdefault("profile", profile.name) + state.setdefault("repos", {}) + state.setdefault("alerts", []) + return state + + +def save_state(profile: SnapshotProfile, state: Dict[str, Any]) -> None: + state_path = state_file(profile) + state_path.parent.mkdir(parents=True, exist_ok=True) + tmp_path = state_path.with_suffix(".tmp") + with tmp_path.open("w", encoding="utf-8") as f: + json.dump(state, f, indent=2, sort_keys=True) + f.write("\n") + tmp_path.replace(state_path) + + +def state_file(profile: SnapshotProfile) -> Path: + state_dir = profile.state_dir or f"~/.local/state/bbackup/{profile.host_id or socket.gethostname()}/{profile.name}" + return expand_path(state_dir) / STATE_FILE_NAME + + +def _git(path: Path, args: List[str]) -> Optional[str]: + try: + result = subprocess.run( + ["git", "-C", str(path), *args], + capture_output=True, + text=True, + timeout=10, + check=False, + ) + except (OSError, subprocess.TimeoutExpired): + return None + if result.returncode != 0: + return None + return result.stdout.strip() + + +def _repo_from_path(path: Path) -> Optional[DiscoveredRepo]: + toplevel = _git(path, ["rev-parse", "--show-toplevel"]) + if not toplevel: + return None + root = Path(toplevel).resolve() + if root != path.resolve(): + return None + origin = _git(root, ["config", "--get", "remote.origin.url"]) or "" + head = _git(root, ["rev-parse", "HEAD"]) or "" + git_dir = _git(root, ["rev-parse", "--git-dir"]) or "" + root_commit = _git(root, ["rev-list", "--max-parents=0", "HEAD"]) or "" + fingerprint_source = "\n".join([str(root), origin, git_dir, root_commit]) + fingerprint = hashlib.sha256(fingerprint_source.encode("utf-8")).hexdigest() + repo_id = hashlib.sha256(f"{origin or root}:{fingerprint}".encode("utf-8")).hexdigest()[:16] + legacy_fingerprint_source = "\n".join([str(root), origin, head, git_dir]) + legacy_fingerprint = hashlib.sha256(legacy_fingerprint_source.encode("utf-8")).hexdigest() + legacy_repo_id = hashlib.sha256(f"{origin or root}:{legacy_fingerprint}".encode("utf-8")).hexdigest()[:16] + return DiscoveredRepo( + path=str(root), + fingerprint=fingerprint, + repo_id=repo_id, + origin=origin, + head=head, + legacy_fingerprint=legacy_fingerprint, + legacy_repo_id=legacy_repo_id, + ) + + +def discover_git_repos(profile: SnapshotProfile) -> List[DiscoveredRepo]: + """Discover immediate child Git roots under repo homes plus explicit roots.""" + repos: Dict[str, DiscoveredRepo] = {} + for home in profile.repo_homes: + home_path = expand_path(home) + if not home_path.is_dir(): + continue + for child in sorted(home_path.iterdir(), key=lambda p: p.name): + if not child.is_dir(): + continue + if _is_excluded_path(child, profile.exclude_paths): + continue + repo = _repo_from_path(child) + if repo: + repos[repo.path] = repo + for explicit in profile.explicit_repos: + path = expand_path(explicit) + if _is_excluded_path(path, profile.exclude_paths): + continue + repo = _repo_from_path(path) + if repo: + repos[repo.path] = repo + return list(repos.values()) + + +def _is_excluded_path(path: Path, exclude_paths: Iterable[str]) -> bool: + resolved = str(path.resolve()) + for raw_pattern in exclude_paths: + pattern = str(expand_path(raw_pattern)) if raw_pattern.startswith(("~", "/", "$")) else raw_pattern + if resolved == pattern or resolved.startswith(pattern.rstrip("/") + "/"): + return True + if fnmatch(resolved, pattern) or fnmatch(path.name, pattern): + return True + return False + + +def reconcile_repos(profile: SnapshotProfile, discovered: List[DiscoveredRepo], save: bool = True) -> Dict[str, Any]: + state = load_state(profile) + repos = state.setdefault("repos", {}) + alerts: List[Dict[str, str]] = [] + now = utc_now() + discovered_by_path = {repo.path: repo for repo in discovered} + + for repo in discovered: + reused = [ + data for data in repos.values() + if data.get("path") == repo.path + and data.get("fingerprint") != repo.fingerprint + and data.get("status") == "retired" + ] + if reused: + alerts.append({ + "severity": "critical", + "code": "path_reuse_conflict", + "path": repo.path, + "message": f"Retired repo path reused with a different Git fingerprint: {repo.path}", + }) + continue + + unexpected_active_reuse = [ + data for existing_id, data in repos.items() + if data.get("path") == repo.path + and data.get("status") == "active" + and existing_id not in {repo.repo_id, repo.legacy_repo_id} + and data.get("fingerprint") not in {repo.fingerprint, repo.legacy_fingerprint} + ] + if unexpected_active_reuse: + alerts.append({ + "severity": "critical", + "code": "active_path_identity_conflict", + "path": repo.path, + "message": f"Active repo path has conflicting identity history: {repo.path}", + }) + continue + + active_ids_for_path = [ + existing_id + for existing_id, data in repos.items() + if data.get("path") == repo.path and data.get("status") == "active" + and ( + existing_id == repo.repo_id + or existing_id == repo.legacy_repo_id + or data.get("fingerprint") in {repo.fingerprint, repo.legacy_fingerprint} + ) + ] + if active_ids_for_path: + merged_entry: Dict[str, Any] = {} + merged_success_ids: List[str] = [] + for existing_id in active_ids_for_path: + existing_entry = repos.pop(existing_id) + merged_entry.update(existing_entry) + for snapshot_id in existing_entry.get("successful_snapshot_ids", []) or []: + if snapshot_id not in merged_success_ids: + merged_success_ids.append(snapshot_id) + if repo.repo_id in repos: + current_entry = repos.pop(repo.repo_id) + merged_entry.update(current_entry) + for snapshot_id in current_entry.get("successful_snapshot_ids", []) or []: + if snapshot_id not in merged_success_ids: + merged_success_ids.append(snapshot_id) + if merged_success_ids: + merged_entry["successful_snapshot_ids"] = merged_success_ids + repos[repo.repo_id] = merged_entry + repo_id = repo.repo_id + entry = repos.setdefault(repo_id, {}) + entry.update({ + "repo_id": repo_id, + "path": repo.path, + "fingerprint": repo.fingerprint, + "origin": repo.origin, + "head": repo.head, + "status": "active", + "last_seen_at": now, + }) + entry.setdefault("first_seen_at", now) + entry.setdefault("successful_snapshot_ids", []) + + for repo_id, entry in repos.items(): + if entry.get("status") != "active": + continue + if entry.get("path") in discovered_by_path: + continue + if entry.get("successful_snapshot_ids"): + entry["status"] = "retired" + entry["retired_at"] = now + else: + alerts.append({ + "severity": "critical", + "code": "missing_without_snapshot", + "path": entry.get("path", ""), + "repo_id": repo_id, + "message": f"Repo disappeared before any successful snapshot: {entry.get('path', repo_id)}", + }) + + state["alerts"] = alerts + if save: + save_state(profile, state) + return { + "profile": profile.name, + "state_file": str(state_file(profile)), + "discovered_repos": [repo.__dict__ for repo in discovered], + "active_repo_ids": sorted( + repo_id for repo_id, entry in repos.items() if entry.get("status") == "active" + ), + "retired_repo_ids": sorted( + repo_id for repo_id, entry in repos.items() if entry.get("status") == "retired" + ), + "alerts": alerts, + } + + +class ResticRunner: + def __init__(self, profile: SnapshotProfile): + if profile.engine != "restic": + raise SnapshotError(f"Unsupported snapshot engine: {profile.engine}") + self.profile = profile + + def env(self) -> Dict[str, str]: + env = dict(os.environ) + if self.profile.cache_dir: + env["RESTIC_CACHE_DIR"] = str(expand_path(self.profile.cache_dir)) + return env + + def base_args(self) -> List[str]: + args = ["restic", "-r", self.profile.repository] + if self.profile.password_file: + args.extend(["--password-file", str(expand_path(self.profile.password_file))]) + if self.profile.cache_dir: + args.extend(["--cache-dir", str(expand_path(self.profile.cache_dir))]) + args.extend(["--json", "--retry-lock", self.profile.retry_lock]) + return args + + def init_args(self) -> List[str]: + return [*self.base_args(), "init"] + + def snapshots_args(self, tags: Optional[Iterable[str]] = None) -> List[str]: + args = [*self.base_args(), "snapshots"] + for tag in tags or []: + args.extend(["--tag", tag]) + return args + + def check_args(self, read_data_subset: Optional[str] = None) -> List[str]: + args = [*self.base_args(), "check"] + if read_data_subset: + args.extend(["--read-data-subset", read_data_subset]) + return args + + def backup_args(self, path: str, tags: Iterable[str]) -> List[str]: + args = [*self.base_args(), "backup", str(expand_path(path))] + if self.profile.host_id: + args.extend(["--host", self.profile.host_id]) + for tag in tags: + args.extend(["--tag", tag]) + for exclude in self.profile.exclude_paths: + args.extend(["--exclude", _restic_exclude_arg(exclude)]) + return args + + def restore_args(self, snapshot_id: str, target: str, include: Optional[str] = None) -> List[str]: + args = [*self.base_args(), "restore", snapshot_id, "--target", str(expand_path(target))] + if include: + args.extend(["--include", include]) + return args + + def forget_args(self, tags: Iterable[str], dry_run: bool = True) -> List[str]: + args = [*self.base_args(), "forget"] + if dry_run: + args.append("--dry-run") + for tag in tags: + args.extend(["--tag", tag]) + return args + + def run(self, args: List[str]) -> Dict[str, Any]: + result = subprocess.run( + args, + capture_output=True, + text=True, + env=self.env(), + check=False, + ) + return { + "args": args, + "returncode": result.returncode, + "stdout": result.stdout, + "stderr": result.stderr, + "ok": result.returncode == 0, + } + + +def common_tags(profile: SnapshotProfile, scope: str, extra: Iterable[str] = ()) -> List[str]: + return [ + "bbackup", + f"profile={profile.name}", + f"scope={scope}", + *profile.tags, + *list(extra), + ] + + +def _restic_exclude_arg(pattern: str) -> str: + if pattern.startswith(("~", "/", "$")): + return str(expand_path(pattern)) + return pattern + + +def _snapshot_plan(profile: SnapshotProfile, save: bool) -> Dict[str, Any]: + discovered = discover_git_repos(profile) + reconcile = reconcile_repos(profile, discovered, save=save) + runner = ResticRunner(profile) + repo_commands = [ + { + "repo_id": repo.repo_id, + "path": repo.path, + "args": runner.backup_args( + repo.path, + common_tags(profile, "repo", [f"repo_id={repo.repo_id}"]), + ), + } + for repo in discovered + if repo.repo_id in set(reconcile["active_repo_ids"]) + ] + include_commands = [ + { + "path": str(expand_path(path)), + "args": runner.backup_args(path, common_tags(profile, "path", [f"path_id={path_id(path)}"])), + } + for path in profile.include_paths + ] + return { + "profile": profile.name, + "repository": profile.repository, + "host_id": profile.host_id, + "state_file": reconcile["state_file"], + "repos": repo_commands, + "paths": include_commands, + "alerts": reconcile["alerts"], + } + + +def snapshot_plan(profile: SnapshotProfile) -> Dict[str, Any]: + return _snapshot_plan(profile, save=False) + + +def path_id(path: str) -> str: + return hashlib.sha256(str(expand_path(path)).encode("utf-8")).hexdigest()[:16] + + +def mark_snapshot_success(profile: SnapshotProfile, repo_id: str, snapshot_id: str) -> None: + state = load_state(profile) + entry = state.setdefault("repos", {}).setdefault(repo_id, {}) + snapshots = entry.setdefault("successful_snapshot_ids", []) + if snapshot_id and snapshot_id not in snapshots: + snapshots.append(snapshot_id) + entry["last_successful_snapshot_id"] = snapshot_id + entry["last_successful_snapshot_at"] = utc_now() + save_state(profile, state) + + +def snapshot_init(profile: SnapshotProfile, dry_run: bool = False) -> Dict[str, Any]: + runner = ResticRunner(profile) + args = runner.init_args() + if dry_run: + return {"dry_run": True, "args": args} + if profile.cache_dir: + expand_path(profile.cache_dir).mkdir(parents=True, exist_ok=True) + expand_path(profile.state_dir).mkdir(parents=True, exist_ok=True) + _require_snapshot_operation_preflight(profile, require_initialized=False) + return runner.run(args) + + +def snapshot_run(profile: SnapshotProfile, dry_run: bool = False) -> Dict[str, Any]: + if not dry_run: + _require_snapshot_operation_preflight(profile) + plan = _snapshot_plan(profile, save=not dry_run) + has_targets = bool(plan["repos"] or plan["paths"]) + if dry_run: + plan["success"] = not bool(plan["alerts"]) and has_targets + plan["dry_run"] = True + return plan + if plan["alerts"]: + raise SnapshotError("Refusing to run snapshot profile with critical alerts") + if not has_targets: + raise SnapshotError("Refusing to run snapshot profile with no repositories or paths") + runner = ResticRunner(profile) + results = [] + for item in [*plan["repos"], *plan["paths"]]: + result = runner.run(item["args"]) + result["target"] = item.get("path") + result["repo_id"] = item.get("repo_id") + results.append(result) + if item.get("repo_id") and result["ok"]: + snapshot_id = _extract_snapshot_id(result["stdout"]) + if snapshot_id: + mark_snapshot_success(profile, item["repo_id"], snapshot_id) + return {"profile": profile.name, "results": results, "success": all(r["ok"] for r in results)} + + +def _extract_snapshot_id(stdout: str) -> str: + for line in stdout.splitlines(): + try: + obj = json.loads(line) + except json.JSONDecodeError: + continue + if isinstance(obj, dict) and obj.get("message_type") == "summary": + return obj.get("snapshot_id", "") + return "" + + +def snapshot_check(profile: SnapshotProfile, read_data_subset: Optional[str] = None, dry_run: bool = False) -> Dict[str, Any]: + runner = ResticRunner(profile) + args = runner.check_args(read_data_subset) + if dry_run: + return {"dry_run": True, "args": args} + _require_snapshot_operation_preflight(profile) + return runner.run(args) + + +def snapshot_restore( + profile: SnapshotProfile, + snapshot_id: str, + target: str, + include: Optional[str] = None, + dry_run: bool = False, +) -> Dict[str, Any]: + runner = ResticRunner(profile) + args = runner.restore_args(snapshot_id, target, include) + if dry_run: + return {"dry_run": True, "args": args} + _require_snapshot_operation_preflight(profile) + return runner.run(args) + + +def retire_repo(profile: SnapshotProfile, repo_id: str) -> Dict[str, Any]: + state = load_state(profile) + entry = state.get("repos", {}).get(repo_id) + if not entry: + raise SnapshotError(f"Unknown repo_id: {repo_id}") + if not entry.get("successful_snapshot_ids"): + raise SnapshotError(f"Cannot retire repo without a successful snapshot: {repo_id}") + entry["status"] = "retired" + entry["retired_at"] = utc_now() + save_state(profile, state) + return {"repo_id": repo_id, "status": "retired", "state_file": str(state_file(profile))} + + +def purge_plan(profile: SnapshotProfile, repo_id: str) -> Dict[str, Any]: + state = load_state(profile) + entry = state.get("repos", {}).get(repo_id) + if not entry: + raise SnapshotError(f"Unknown repo_id: {repo_id}") + if entry.get("status") == "active": + raise SnapshotError(f"Refusing to purge active repo_id: {repo_id}") + runner = ResticRunner(profile) + tags = common_tags(profile, "repo", [f"repo_id={repo_id}"]) + return { + "repo_id": repo_id, + "status": entry.get("status"), + "path": entry.get("path"), + "dry_run_required": True, + "forget_args": runner.forget_args(tags, dry_run=True), + "destructive_args_after_confirmation": runner.forget_args(tags, dry_run=False), + } + + +def schedule_units(profile: SnapshotProfile, config_path: Optional[str] = None) -> Dict[str, str]: + service_name = f"bbackup-{profile.host_id.lower()}-{profile.name}" + schedule = profile.schedule or {} + daily_time = schedule.get("daily_time", "03:30") + maintenance_time = schedule.get("maintenance_time", "Sun 04:30") + verification_time = schedule.get("verification_time", "monthly") + verification_read_data_subset = _systemd_exec_arg(str(schedule.get("verification_read_data_subset", "5%"))) + config_args = f" --config {shlex.quote(config_path)}" if config_path else "" + unit = f"""[Unit] +Description=bbackup {profile.host_id} {profile.name} snapshot + +[Service] +Type=oneshot +ExecStart=/usr/bin/env bbackup{config_args} snapshot run --profile {shlex.quote(profile.name)} +""" + maintenance_service = f"""[Unit] +Description=bbackup {profile.host_id} {profile.name} weekly maintenance + +[Service] +Type=oneshot +ExecStart=/usr/bin/env bbackup{config_args} snapshot check --profile {shlex.quote(profile.name)} +""" + verification_service = f"""[Unit] +Description=bbackup {profile.host_id} {profile.name} monthly verification + +[Service] +Type=oneshot +ExecStart=/usr/bin/env bbackup{config_args} snapshot check --profile {shlex.quote(profile.name)} --read-data-subset {shlex.quote(verification_read_data_subset)} +""" + timer = f"""[Unit] +Description=Run bbackup {profile.host_id} {profile.name} snapshot daily + +[Timer] +OnCalendar=*-*-* {daily_time}:00 +Persistent=true +RandomizedDelaySec=30m + +[Install] +WantedBy=timers.target +""" + maintenance = f"""[Unit] +Description=Run bbackup {profile.host_id} {profile.name} weekly maintenance + +[Timer] +OnCalendar={maintenance_time} +Persistent=true +RandomizedDelaySec=1h +Unit={service_name}-maintenance.service + +[Install] +WantedBy=timers.target +""" + verification = f"""[Unit] +Description=Run bbackup {profile.host_id} {profile.name} monthly verification + +[Timer] +OnCalendar={verification_time} +Persistent=true +RandomizedDelaySec=2h +Unit={service_name}-verification.service + +[Install] +WantedBy=timers.target +""" + return { + f"{service_name}.service": unit, + f"{service_name}.timer": timer, + f"{service_name}-maintenance.service": maintenance_service, + f"{service_name}-maintenance.timer": maintenance, + f"{service_name}-verification.service": verification_service, + f"{service_name}-verification.timer": verification, + } + + +def _systemd_exec_arg(value: str) -> str: + return value.replace("%", "%%") + + +def check_snapshot_profile(profile: SnapshotProfile) -> Dict[str, Any]: + checks: Dict[str, Dict[str, Any]] = {} + checks["engine"] = {"ok": profile.engine == "restic", "message": profile.engine} + checks["restic"] = _tool_check("restic") + checks["rclone"] = _tool_check("rclone") if profile.repository.startswith("rclone:") else { + "ok": True, + "message": "not required for non-rclone repository", + } + checks["rclone_drive_client_id"] = _rclone_drive_client_id_check(profile) + checks["repository"] = {"ok": bool(profile.repository), "message": profile.repository or "missing"} + checks["hostname"] = _hostname_check(profile) + checks["password_file"] = _password_file_check(profile) + checks["cache_dir"] = _dir_check(profile.cache_dir, create=False) + checks["state_dir"] = _dir_check(profile.state_dir, create=False) + if checks["engine"]["ok"]: + try: + checks["repository_initialized"] = _repository_initialized_check(profile) + except SnapshotError as exc: + checks["repository_initialized"] = {"ok": False, "message": str(exc)} + else: + checks["repository_initialized"] = { + "ok": False, + "message": "unsupported snapshot engine", + } + ok = all(check["ok"] for check in checks.values()) + return {"ok": ok, "profile": profile.name, "checks": checks} + + +def _require_rclone_drive_client(profile: SnapshotProfile) -> None: + check = _rclone_drive_client_id_check(profile) + if not check["ok"]: + raise SnapshotError(check["message"]) + + +def _require_snapshot_operation_preflight(profile: SnapshotProfile, require_initialized: bool = True) -> None: + checks: Dict[str, Dict[str, Any]] = { + "engine": {"ok": profile.engine == "restic", "message": profile.engine}, + "restic": _tool_check("restic"), + "rclone": _tool_check("rclone") if profile.repository.startswith("rclone:") else { + "ok": True, + "message": "not required for non-rclone repository", + }, + "rclone_drive_client_id": _rclone_drive_client_id_check(profile), + "repository": {"ok": bool(profile.repository), "message": profile.repository or "missing"}, + "hostname": _hostname_check(profile), + "password_file": _password_file_check(profile), + "cache_dir": _dir_check(profile.cache_dir, create=False), + "state_dir": _dir_check(profile.state_dir, create=False), + } + if require_initialized: + checks["repository_initialized"] = _repository_initialized_check(profile) + failures = [ + f"{name}: {check['message']}" + for name, check in checks.items() + if not check["ok"] + ] + if failures: + raise SnapshotError("Snapshot profile preflight failed: " + "; ".join(failures)) + + +def check_all_snapshot_profiles(config: Config) -> Dict[str, Any]: + profiles = { + name: check_snapshot_profile(profile) + for name, profile in config.snapshot_profiles.items() + } + return { + "ok": all(item["ok"] for item in profiles.values()) if profiles else True, + "profiles": profiles, + } + + +def _tool_check(name: str) -> Dict[str, Any]: + found = shutil.which(name) + if not found: + return {"ok": False, "message": f"{name} not found in PATH"} + try: + result = subprocess.run([name, "--version"], capture_output=True, text=True, timeout=5) + version = result.stdout.splitlines()[0] if result.stdout else found + except (OSError, subprocess.TimeoutExpired): + version = found + return {"ok": True, "message": version} + + +def _rclone_remote_name(repository: str) -> Optional[str]: + if not repository.startswith("rclone:"): + return None + parts = repository.split(":", 2) + if len(parts) < 3 or not parts[1]: + return None + return parts[1] + + +def _rclone_drive_client_id_check(profile: SnapshotProfile) -> Dict[str, Any]: + remote_name = _rclone_remote_name(profile.repository) + if not remote_name: + return {"ok": True, "message": "not required for non-rclone repository"} + if not shutil.which("rclone"): + return {"ok": False, "message": "rclone not found in PATH"} + try: + result = subprocess.run( + ["rclone", "config", "show", remote_name], + capture_output=True, + text=True, + timeout=5, + check=False, + ) + except (OSError, subprocess.TimeoutExpired) as exc: + return {"ok": False, "message": f"could not inspect rclone remote {remote_name}: {exc}"} + if result.returncode != 0: + detail = result.stderr.strip() or result.stdout.strip() or f"exit {result.returncode}" + return {"ok": False, "message": f"could not inspect rclone remote {remote_name}: {detail}"} + + fields: Dict[str, str] = {} + for line in result.stdout.splitlines(): + if "=" not in line: + continue + key, value = line.split("=", 1) + fields[key.strip()] = value.strip() + if fields.get("type") != "drive": + return {"ok": True, "message": f"rclone remote {remote_name} type {fields.get('type', 'unknown')}"} + if fields.get("client_id"): + return {"ok": True, "message": f"Google Drive remote {remote_name} has client_id configured"} + if profile.allow_default_rclone_drive_client: + return { + "ok": True, + "message": ( + f"Google Drive remote {remote_name} uses the default rclone Drive client by " + "explicit profile configuration" + ), + } + return { + "ok": False, + "message": ( + f"Google Drive remote {remote_name} has no client_id; configure a dedicated " + "OAuth client before treating this snapshot profile as reliable" + ), + } + + +def _hostname_check(profile: SnapshotProfile) -> Dict[str, Any]: + if not profile.host_id: + return {"ok": False, "message": "host_id missing"} + actual = socket.gethostname() + if profile.host_id.lower() == actual.lower(): + return {"ok": True, "message": actual} + return {"ok": False, "message": f"profile host_id {profile.host_id} does not match hostname {actual}"} + + +def _password_file_check(profile: SnapshotProfile) -> Dict[str, Any]: + if not profile.password_file: + return {"ok": False, "message": "password_file missing"} + path = expand_path(profile.password_file) + if not path.exists(): + return {"ok": False, "message": f"password file not found: {path}"} + mode = stat.S_IMODE(path.stat().st_mode) + if mode & 0o077: + return {"ok": False, "message": f"password file must not be group/world accessible: {oct(mode)}"} + return {"ok": True, "message": str(path)} + + +def _dir_check(path: str, create: bool = False) -> Dict[str, Any]: + if not path: + return {"ok": False, "message": "missing"} + resolved = expand_path(path) + if create: + resolved.mkdir(parents=True, exist_ok=True) + if not resolved.exists(): + return {"ok": False, "message": f"not found: {resolved}"} + if not os.access(resolved, os.W_OK): + return {"ok": False, "message": f"not writable: {resolved}"} + return {"ok": True, "message": str(resolved)} + + +def _repository_initialized_check(profile: SnapshotProfile) -> Dict[str, Any]: + if not shutil.which("restic") or not profile.repository: + return {"ok": False, "message": "restic or repository missing"} + runner = ResticRunner(profile) + result = runner.run(runner.snapshots_args(["bbackup"])) + if result["ok"]: + return {"ok": True, "message": "repository accessible"} + return {"ok": False, "message": result["stderr"] or result["stdout"] or "repository inaccessible"} diff --git a/config.yaml.example b/config.yaml.example index 3658ed9..1b5aa4c 100644 --- a/config.yaml.example +++ b/config.yaml.example @@ -186,6 +186,62 @@ filesystem: excludes: - "*.log" +# Native deduplicating snapshot profiles. +# Restic encrypts content and metadata. Keep password files outside selected +# backup paths, chmod 0600, and escrow the password out-of-band. +snapshot_profiles: + essentials-daily: + engine: restic + host_id: WORKSTATION01 + repository: rclone:my-drive:backups/WORKSTATION01/restic/essentials-daily + # Prefer a dedicated Google Drive rclone OAuth client. Set this only as an + # explicit YAML boolean true when accepting rclone's shared default client. + allow_default_rclone_drive_client: false + cache_dir: ~/.cache/bbackup/restic/WORKSTATION01/essentials-daily + state_dir: ~/.local/state/bbackup/WORKSTATION01/essentials-daily + password_file: ~/.local/share/bbackup-credentials/restic/WORKSTATION01/essentials-daily.password + retry_lock: 5m + repo_homes: + - ~/Projects + - ~/Work + explicit_repos: + - ~/workstation-control + include_paths: + - ~/.codex + - ~/.config/my-agent + - ~/.local/share/my-agent + - ~/.local/state/my-agent + - ~/.config/bbackup + - ~/.config/rclone + - ~/.config/systemd/user + - ~/.ssh + - ~/.pki + - ~/.gitconfig + - ~/.bashrc + - ~/.profile + - ~/.tmux.conf + - ~/.config/gh + - ~/Documents + - ~/Pictures + exclude_paths: + - ~/Projects/archives + - ~/Projects/starred-repos + - ~/Projects/migration-staging + - ~/CloudDrive + - ~/.config/bbackup/restic + - ~/.cache + - "**/node_modules" + - "**/.venv" + retention: + active_repo_daily: 14 + active_repo_weekly: 8 + active_repo_monthly: 12 + schedule: + daily_time: "03:30" + maintenance_time: "Sun 04:30" + verification_time: "monthly" + verification_read_data_subset: "5%" + # Docker Settings docker: socket: /var/run/docker.sock diff --git a/docs/VERSIONING.md b/docs/VERSIONING.md index 5b9a828..c2caebe 100644 --- a/docs/VERSIONING.md +++ b/docs/VERSIONING.md @@ -10,6 +10,11 @@ references must match it before publishing: - `CHANGELOG.md` current release header and `[Unreleased]` compare target - generated CLI skills docs +Normal release batches use a patch bump by default, including commits with +`feat:` subjects. Use a `Release-Type: major|minor|patch|none` trailer in the +commit body when a release needs a non-default impact or an explicit no-bump +decision. + Run the version sync check: ```bash diff --git a/docs/cli-skills-index.json b/docs/cli-skills-index.json index fb222d6..f815a1a 100644 --- a/docs/cli-skills-index.json +++ b/docs/cli-skills-index.json @@ -36,55 +36,87 @@ "start": 296 }, "bbackup:skills": { - "end": 380, + "end": 378, "start": 351 }, + "bbackup:snapshot check": { + "end": 421, + "start": 379 + }, + "bbackup:snapshot init": { + "end": 453, + "start": 422 + }, + "bbackup:snapshot plan": { + "end": 484, + "start": 454 + }, + "bbackup:snapshot purge-plan": { + "end": 516, + "start": 485 + }, + "bbackup:snapshot restore": { + "end": 551, + "start": 517 + }, + "bbackup:snapshot retire": { + "end": 583, + "start": 552 + }, + "bbackup:snapshot run": { + "end": 615, + "start": 584 + }, + "bbackup:snapshot schedule": { + "end": 648, + "start": 616 + }, "bbman:check-deps": { - "end": 422, - "start": 381 + "end": 690, + "start": 649 }, "bbman:check-updates": { - "end": 454, - "start": 423 + "end": 722, + "start": 691 }, "bbman:cleanup": { - "end": 490, - "start": 455 + "end": 758, + "start": 723 }, "bbman:diagnostics": { - "end": 522, - "start": 491 + "end": 790, + "start": 759 }, "bbman:health": { - "end": 553, - "start": 523 + "end": 821, + "start": 791 }, "bbman:repo-url": { - "end": 585, - "start": 554 + "end": 853, + "start": 822 }, "bbman:run": { - "end": 606, - "start": 586 + "end": 874, + "start": 854 }, "bbman:setup": { - "end": 637, - "start": 607 + "end": 905, + "start": 875 }, "bbman:skills": { - "end": 665, - "start": 638 + "end": 933, + "start": 906 }, "bbman:status": { - "end": 696, - "start": 666 + "end": 964, + "start": 934 }, "bbman:update": { - "end": 730, - "start": 697 + "end": 998, + "start": 965 }, "bbman:validate-config": { - "end": 761, - "start": 731 + "end": 1029, + "start": 999 } } diff --git a/docs/cli-skills.md b/docs/cli-skills.md index 16f37dc..e96750b 100644 --- a/docs/cli-skills.md +++ b/docs/cli-skills.md @@ -1,6 +1,6 @@ # CLI skills catalog -> Generated from the bbackup/bbman CLI metadata. Version: 1.8.4. This catalog is authoritative for this version. +> Generated from the bbackup/bbman CLI metadata. Version: 1.8.5. This catalog is authoritative for this version. ## bbackup @@ -376,6 +376,274 @@ List or inspect bbackup skills in JSON or Markdown formats. bbackup skills --format markdown ``` +### bbackup snapshot check + +**Summary**: Run a non-destructive restic repository check. + +Check a native snapshot repository, optionally with restic --read-data-subset for verification schedules. + +#### CLI parameters + +| Name | Type | Required | Default | Description | +|---|---|:---:|---|---| +| `--profile` | `string` | yes | `` | Snapshot profile name from config.yaml. | +| `--read-data-subset` | `string` | no | `` | Optional restic check --read-data-subset value, such as 5%. | +| `--dry-run` | `bool` | no | `False` | Return the resolved restic plan without executing it. | +| `--output` | `string` | no | `` | Output format: text or json. | + +#### JSON / environment parameters + +| Name | Kind | Type | Required | Default | Description | +|---|---|---|:---:|---|---| +| `input_json` | json | `object` | no | `` | Flat JSON object providing all parameters. | + +#### Examples + +- Run a lightweight repository check. + + ```bash + bbackup snapshot check --profile essentials-daily --output json + ``` + + ```bash + bbackup snapshot check --input-json '{"profile":"essentials-daily","output":"json"}' --output json + ``` + +- Run a verification check over a subset of pack data. + + ```bash + bbackup snapshot check --profile essentials-daily --read-data-subset 5% --output json + ``` + + ```bash + bbackup snapshot check --input-json '{"profile":"essentials-daily","read_data_subset":"5%","output":"json"}' --output json + ``` + +### bbackup snapshot init + +**Summary**: Initialize the restic repository for a snapshot profile. + +Initialize the configured restic repository after profile safety preflight. Use dry-run mode to inspect the exact restic init command first. + +#### CLI parameters + +| Name | Type | Required | Default | Description | +|---|---|:---:|---|---| +| `--profile` | `string` | yes | `` | Snapshot profile name from config.yaml. | +| `--dry-run` | `bool` | no | `False` | Return the resolved restic plan without executing it. | +| `--output` | `string` | no | `` | Output format: text or json. | + +#### JSON / environment parameters + +| Name | Kind | Type | Required | Default | Description | +|---|---|---|:---:|---|---| +| `input_json` | json | `object` | no | `` | Flat JSON object providing all parameters. | + +#### Examples + +- Inspect the restic init command without executing it. + + ```bash + bbackup snapshot init --profile essentials-daily --dry-run --output json + ``` + + ```bash + bbackup snapshot init --input-json '{"profile":"essentials-daily","dry_run":true,"output":"json"}' --output json + ``` + +### bbackup snapshot plan + +**Summary**: Resolve a native snapshot profile without executing restic. + +Discover Git repositories and explicit paths for a snapshot profile, then return the restic commands and safety alerts without persisting state. + +#### CLI parameters + +| Name | Type | Required | Default | Description | +|---|---|:---:|---|---| +| `--profile` | `string` | yes | `` | Snapshot profile name from config.yaml. | +| `--output` | `string` | no | `` | Output format: text or json. | + +#### JSON / environment parameters + +| Name | Kind | Type | Required | Default | Description | +|---|---|---|:---:|---|---| +| `input_json` | json | `object` | no | `` | Flat JSON object providing all parameters. | + +#### Examples + +- Plan a native snapshot profile with JSON output. + + ```bash + bbackup snapshot plan --profile essentials-daily --output json + ``` + + ```bash + bbackup snapshot plan --input-json '{"profile":"essentials-daily","output":"json"}' --output json + ``` + +### bbackup snapshot purge-plan + +**Summary**: Build a dry-run-first purge plan for a retired repository. + +Return restic forget arguments for a retired repo ID. The command keeps destructive cleanup manual by exposing dry-run and post-confirmation arguments separately. + +#### CLI parameters + +| Name | Type | Required | Default | Description | +|---|---|:---:|---|---| +| `--profile` | `string` | yes | `` | Snapshot profile name from config.yaml. | +| `--repo-id` | `string` | yes | `` | Tracked Git repository ID from the snapshot state ledger. | +| `--output` | `string` | no | `` | Output format: text or json. | + +#### JSON / environment parameters + +| Name | Kind | Type | Required | Default | Description | +|---|---|---|:---:|---|---| +| `input_json` | json | `object` | no | `` | Flat JSON object providing all parameters. | + +#### Examples + +- Prepare a manual purge plan for a retired repository. + + ```bash + bbackup snapshot purge-plan --profile essentials-daily --repo-id abc123 --output json + ``` + + ```bash + bbackup snapshot purge-plan --input-json '{"profile":"essentials-daily","repo_id":"abc123","output":"json"}' --output json + ``` + +### bbackup snapshot restore + +**Summary**: Restore a restic snapshot into an empty target directory. + +Restore a snapshot or snapshot:path selector into an empty target. Dry-run mode returns the restic restore command without contacting Docker. + +#### CLI parameters + +| Name | Type | Required | Default | Description | +|---|---|:---:|---|---| +| `--profile` | `string` | yes | `` | Snapshot profile name from config.yaml. | +| `--snapshot-id` | `string` | yes | `` | Restic snapshot ID or snapshot:path selector to restore. | +| `--target` | `path` | yes | `` | Empty restore target directory. | +| `--include` | `string` | no | `` | Optional restic include filter for targeted restore. | +| `--dry-run` | `bool` | no | `False` | Return the resolved restic plan without executing it. | +| `--output` | `string` | no | `` | Output format: text or json. | + +#### JSON / environment parameters + +| Name | Kind | Type | Required | Default | Description | +|---|---|---|:---:|---|---| +| `input_json` | json | `object` | no | `` | Flat JSON object providing all parameters. | + +#### Examples + +- Plan a targeted restore from a snapshot. + + ```bash + bbackup snapshot restore --profile essentials-daily --snapshot-id latest --target /tmp/restore --dry-run --output json + ``` + + ```bash + bbackup snapshot restore --input-json '{"profile":"essentials-daily","snapshot_id":"latest","target":"/tmp/restore","dry_run":true,"output":"json"}' --output json + ``` + +### bbackup snapshot retire + +**Summary**: Mark a tracked repository retired after it has a successful snapshot. + +Retire a repository in the snapshot state ledger so future purge planning can remain explicit and dry-run-first. + +#### CLI parameters + +| Name | Type | Required | Default | Description | +|---|---|:---:|---|---| +| `--profile` | `string` | yes | `` | Snapshot profile name from config.yaml. | +| `--repo-id` | `string` | yes | `` | Tracked Git repository ID from the snapshot state ledger. | +| `--output` | `string` | no | `` | Output format: text or json. | + +#### JSON / environment parameters + +| Name | Kind | Type | Required | Default | Description | +|---|---|---|:---:|---|---| +| `input_json` | json | `object` | no | `` | Flat JSON object providing all parameters. | + +#### Examples + +- Retire a repository ID after verifying it has a successful snapshot. + + ```bash + bbackup snapshot retire --profile essentials-daily --repo-id abc123 --output json + ``` + + ```bash + bbackup snapshot retire --input-json '{"profile":"essentials-daily","repo_id":"abc123","output":"json"}' --output json + ``` + +### bbackup snapshot run + +**Summary**: Run a native restic snapshot profile. + +Discover active Git repositories and configured paths, enforce safety preflight for non-dry runs, run restic backup, and record successful repository snapshot IDs in the state ledger. + +#### CLI parameters + +| Name | Type | Required | Default | Description | +|---|---|:---:|---|---| +| `--profile` | `string` | yes | `` | Snapshot profile name from config.yaml. | +| `--dry-run` | `bool` | no | `False` | Return the resolved restic plan without executing it. | +| `--output` | `string` | no | `` | Output format: text or json. | + +#### JSON / environment parameters + +| Name | Kind | Type | Required | Default | Description | +|---|---|---|:---:|---|---| +| `input_json` | json | `object` | no | `` | Flat JSON object providing all parameters. | + +#### Examples + +- Dry-run a profile before taking a snapshot. + + ```bash + bbackup snapshot run --profile essentials-daily --dry-run --output json + ``` + + ```bash + bbackup snapshot run --input-json '{"profile":"essentials-daily","dry_run":true,"output":"json"}' --output json + ``` + +### bbackup snapshot schedule + +**Summary**: Render user systemd units and timers for a snapshot profile. + +Render matching service/timer units for daily snapshot runs, weekly non-destructive checks, and monthly verification checks. + +#### CLI parameters + +| Name | Type | Required | Default | Description | +|---|---|:---:|---|---| +| `--profile` | `string` | yes | `` | Snapshot profile name from config.yaml. | +| `--output` | `string` | no | `` | Output format: text or json. | + +#### JSON / environment parameters + +| Name | Kind | Type | Required | Default | Description | +|---|---|---|:---:|---|---| +| `input_json` | json | `object` | no | `` | Flat JSON object providing all parameters. | + +#### Examples + +- Render schedule units for manual installation. + + ```bash + bbackup snapshot schedule --profile essentials-daily --output json + ``` + + ```bash + bbackup snapshot schedule --input-json '{"profile":"essentials-daily","output":"json"}' --output json + ``` + ## bbman ### bbman check-deps diff --git a/docs/prompts/bootstrap-planning-agent.md b/docs/prompts/bootstrap-planning-agent.md index 5dd70c7..8c24411 100644 --- a/docs/prompts/bootstrap-planning-agent.md +++ b/docs/prompts/bootstrap-planning-agent.md @@ -1,9 +1,9 @@ # Bootstrap Planning Agent Prompt -Use this prompt to hand `/mnt/data/devzone/bbackup` to a planning agent. +Use this prompt to hand the active `bbackup` checkout to a planning agent. ```text -You are the planning agent for /mnt/data/devzone/bbackup. +You are the planning agent for the active `bbackup` checkout. Objective: Turn this checkout into the working maintenance repo for bbackup and Codex-related @@ -11,7 +11,7 @@ development work without implementing domain changes until the user accepts a plan. Repository context: -- Repo path: /mnt/data/devzone/bbackup +- Repo path: `` - GitHub remote: https://github.com/CruxExperts/best-backup.git - Primary package: bbackup/ - CLI entry points: bbackup.py, bbman.py, bbackup/cli.py, bbackup/bbman_entry.py @@ -21,7 +21,7 @@ Repository context: Planning task: 1. Read AGENTS.md, README.md, INSTALL.md, QUICKSTART.md, and docs/README.md. 2. Inspect Localsetup adapter status with: - localsetup adapters --target-directory /mnt/data/devzone/bbackup --platforms codex + localsetup adapters --target-directory --platforms codex 3. Inspect git status. 4. Produce a concise implementation plan for the requested bbackup maintenance work. @@ -32,8 +32,8 @@ Validation commands to include in the plan: - git status --short --ignored - uv run python -m py_compile bbackup.py bbman.py bbackup/*.py bbackup/data/*.py bbackup/management/*.py scripts/*.py - uv run pytest -- localsetup adapters --target-directory /mnt/data/devzone/bbackup --platforms codex -- localsetup doctor --target-directory /mnt/data/devzone/bbackup --global-preset core --repo-preset core --platforms codex --dependency-mode uv-sync --json +- localsetup adapters --target-directory --platforms codex +- localsetup doctor --target-directory --global-preset core --repo-preset core --platforms codex --dependency-mode uv-sync --json Output format: - Start with assumptions. diff --git a/external_skills.lock.json b/external_skills.lock.json new file mode 100644 index 0000000..40c47ff --- /dev/null +++ b/external_skills.lock.json @@ -0,0 +1,6 @@ +{ + "generated_at": "2026-06-27T20:49:30+00:00", + "schema_version": "1", + "skills": [], + "trust_boundary": "External skills are trusted only after vet/import/normalize/sandbox-test and roster update." +} diff --git a/pyproject.toml b/pyproject.toml index ab85ff7..21783a5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "bbackup" -version = "1.8.4" +version = "1.8.5" description = "Docker Backup Tool with Rich TUI" readme = "README.md" requires-python = ">=3.12" @@ -73,7 +73,7 @@ include = ["bbackup*"] testpaths = ["tests"] addopts = "--tb=short -q" markers = [ - "integration: marks tests requiring a live Docker daemon (deselect with -m 'not integration')", + "integration: marks tests requiring live external tools such as Docker or restic (deselect with -m 'not integration')", ] [tool.ruff] diff --git a/tests/integration/test_snapshot_restic.py b/tests/integration/test_snapshot_restic.py new file mode 100644 index 0000000..3298219 --- /dev/null +++ b/tests/integration/test_snapshot_restic.py @@ -0,0 +1,91 @@ +import json +import shutil +import socket +import subprocess +import textwrap + +import pytest + +from bbackup.config import Config +from bbackup.snapshot import ( + load_state, + snapshot_check, + snapshot_init, + snapshot_restore, + snapshot_run, +) + + +pytestmark = pytest.mark.integration + + +def _init_git_repo(path): + path.mkdir(parents=True) + (path / "README.md").write_text("restic integration\n", encoding="utf-8") + subprocess.run(["git", "-C", str(path), "init"], check=True, capture_output=True) + subprocess.run(["git", "-C", str(path), "config", "user.email", "test@example.com"], check=True) + subprocess.run(["git", "-C", str(path), "config", "user.name", "Test"], check=True) + subprocess.run(["git", "-C", str(path), "add", "README.md"], check=True) + subprocess.run(["git", "-C", str(path), "commit", "-m", "init"], check=True, capture_output=True) + + +def _write_local_restic_config(tmp_path): + cfg_file = tmp_path / "config.yaml" + password_file = tmp_path / "password" + password_file.write_text("secret\n", encoding="utf-8") + password_file.chmod(0o600) + cfg_file.write_text(textwrap.dedent(f""" + snapshot_profiles: + local-test: + engine: restic + host_id: {socket.gethostname()} + repository: {tmp_path / "restic-repo"} + cache_dir: {tmp_path / "cache"} + state_dir: {tmp_path / "state"} + password_file: {password_file} + retry_lock: 1m + repo_homes: + - {tmp_path / "repos"} + """), encoding="utf-8") + return cfg_file + + +def test_local_restic_snapshot_check_restore_records_state(tmp_path): + if not shutil.which("restic"): + pytest.skip("restic executable is unavailable") + if subprocess.run(["restic", "version"], capture_output=True, text=True, check=False).returncode != 0: + pytest.skip("restic executable is unavailable") + + repo_path = tmp_path / "repos" / "repo-a" + _init_git_repo(repo_path) + cfg = Config(config_path=str(_write_local_restic_config(tmp_path))) + profile = cfg.snapshot_profiles["local-test"] + + init_result = snapshot_init(profile) + assert init_result["ok"] is True + + run_result = snapshot_run(profile) + assert run_result["success"] is True + snapshot_id = "" + for line in run_result["results"][0]["stdout"].splitlines(): + payload = json.loads(line) + if payload.get("message_type") == "summary": + snapshot_id = payload["snapshot_id"] + break + assert snapshot_id + + check_result = snapshot_check(profile) + assert check_result["ok"] is True + + restore_target = tmp_path / "restore" + restore_target.mkdir() + restore_result = snapshot_restore(profile, snapshot_id=f"{snapshot_id}:{repo_path}", target=str(restore_target)) + assert restore_result["ok"] is True, restore_result + restored_readmes = list(restore_target.rglob("README.md")) + assert restored_readmes + assert any(path.read_text(encoding="utf-8") == "restic integration\n" for path in restored_readmes) + + state = load_state(profile) + repo_id = run_result["results"][0]["repo_id"] + assert snapshot_id in state["repos"][repo_id]["successful_snapshot_ids"] + assert state["repos"][repo_id]["last_successful_snapshot_id"] == snapshot_id diff --git a/tests/test_cli.py b/tests/test_cli.py index b01b1ed..cdf33ea 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -131,9 +131,10 @@ def test_nonexistent_dir_default(self, tmp_path): result = CliRunner().invoke(cli, ["list-backups"]) assert result.exit_code == 0 - def test_lists_backup_dirs(self, tmp_path, mock_docker_client): + def test_lists_backup_dirs(self, tmp_path): (tmp_path / "backup_20240101_000000").mkdir() - result = CliRunner().invoke(cli, ["list-backups", "--backup-dir", str(tmp_path)]) + with patch("bbackup.cli.DockerRestore", side_effect=AssertionError("Docker should not be used")): + result = CliRunner().invoke(cli, ["list-backups", "--backup-dir", str(tmp_path)]) assert result.exit_code == 0 assert "backup_20240101_000000" in result.output @@ -921,12 +922,12 @@ def test_backup_dry_run_exits_success(self, mock_docker_client): assert result.exit_code == EXIT_SUCCESS def test_restore_dry_run_does_not_call_restore_backup(self, tmp_path): - with patch("bbackup.cli.DockerRestore") as MockRestore: - CliRunner().invoke( + with patch("bbackup.cli.DockerRestore", side_effect=AssertionError("Docker should not be used")): + result = CliRunner().invoke( cli, ["restore", "--backup-path", str(tmp_path), "--containers", "myapp", "--dry-run"], ) - MockRestore.return_value.restore_backup.assert_not_called() + assert result.exit_code == EXIT_SUCCESS def test_dry_run_output_shape(self, mock_docker_client): mock_docker_client.containers.list.return_value = [] @@ -946,10 +947,11 @@ def test_restore_all_dry_run_includes_archive_backed_volumes(self, tmp_path): volumes_dir.mkdir() (volumes_dir / "myvolume.tar.gz").write_bytes(b"") - result = CliRunner().invoke( - cli, - ["restore", "--backup-path", str(tmp_path), "--all", "--dry-run", "--output", "json"], - ) + with patch("bbackup.cli.DockerRestore", side_effect=AssertionError("Docker should not be used")): + result = CliRunner().invoke( + cli, + ["restore", "--backup-path", str(tmp_path), "--all", "--dry-run", "--output", "json"], + ) assert result.exit_code == EXIT_SUCCESS data = json.loads(result.output) @@ -969,10 +971,21 @@ def test_restore_all_dry_run_discovers_solid_archive_targets(self, tmp_path): with tarfile.open(archive_path, "w:gz") as tar: tar.add(backup_root, arcname=backup_root.name) - result = CliRunner().invoke( - cli, - ["restore", "--backup-path", str(archive_path), "--all", "--dry-run", "--output", "json"], - ) + with patch("bbackup.cli.DockerRestore", side_effect=AssertionError("Docker should not be used")): + result = CliRunner().invoke( + cli, + [ + "restore", + "--backup-path", + str(archive_path), + "--all", + "--filesystem-destination", + str(tmp_path / "restore"), + "--dry-run", + "--output", + "json", + ], + ) assert result.exit_code == EXIT_SUCCESS data = json.loads(result.output) @@ -999,6 +1012,19 @@ def test_restore_all_with_filesystem_requires_destination_before_restore(self, t assert "Filesystem restore requires --filesystem-destination" in data["errors"] MockRestore.return_value.restore_backup.assert_not_called() + def test_restore_all_dry_run_with_filesystem_requires_destination_before_docker(self, tmp_path): + (tmp_path / "filesystems" / "documents").mkdir(parents=True) + + with patch("bbackup.cli.DockerRestore", side_effect=AssertionError("Docker should not be used")): + result = CliRunner().invoke( + cli, + ["restore", "--backup-path", str(tmp_path), "--all", "--dry-run", "--output", "json"], + ) + + assert result.exit_code == EXIT_USER_ERROR + data = json.loads(result.output) + assert "Filesystem restore requires --filesystem-destination" in data["errors"] + def test_restore_rejects_multiple_filesystems_sharing_destination_before_restore(self, tmp_path): destination = tmp_path / "restore" @@ -1028,6 +1054,35 @@ def test_restore_rejects_multiple_filesystems_sharing_destination_before_restore ) MockRestore.return_value.restore_backup.assert_not_called() + def test_restore_dry_run_rejects_multiple_filesystems_before_docker(self, tmp_path): + destination = tmp_path / "restore" + + with patch("bbackup.cli.DockerRestore", side_effect=AssertionError("Docker should not be used")): + result = CliRunner().invoke( + cli, + [ + "restore", + "--backup-path", + str(tmp_path), + "--filesystem", + "documents", + "--filesystem", + "photos", + "--filesystem-destination", + str(destination), + "--dry-run", + "--output", + "json", + ], + ) + + assert result.exit_code == EXIT_USER_ERROR + data = json.loads(result.output) + assert any( + "multiple filesystem targets cannot share one destination" in error + for error in data["errors"] + ) + # --------------------------------------------------------------------------- # TestEnvVars diff --git a/tests/test_management.py b/tests/test_management.py index 60ded68..182175b 100644 --- a/tests/test_management.py +++ b/tests/test_management.py @@ -111,6 +111,16 @@ def test_run_health_check_returns_all_keys(self, mock_docker_client): assert "python_packages" in result assert "overall" in result + def test_run_health_check_reports_config_load_failure(self, mock_docker_client): + with patch("subprocess.run") as mock_run: + mock_run.return_value = MagicMock(returncode=0, stdout="v3\n", stderr="") + with patch("bbackup.management.health.Config", side_effect=ValueError("bad yaml")): + result = health_module.run_health_check() + + assert result["snapshot_profiles"]["ok"] is False + assert "bad yaml" in result["snapshot_profiles"]["error"] + assert result["all_critical_ok"] is False + def _make_full_results(self, healthy=True): return { "docker": (healthy, "Docker 24.0.0 accessible"), diff --git a/tests/test_snapshot.py b/tests/test_snapshot.py new file mode 100644 index 0000000..8b22e52 --- /dev/null +++ b/tests/test_snapshot.py @@ -0,0 +1,708 @@ +import json +import hashlib +import textwrap +from unittest.mock import MagicMock, patch + +from click.testing import CliRunner + +from bbackup.cli import cli +from bbackup.config import Config +from bbackup.snapshot import ( + ResticRunner, + SnapshotError, + check_snapshot_profile, + discover_git_repos, + load_state, + purge_plan, + reconcile_repos, + retire_repo, + save_state, + snapshot_check, + snapshot_plan, + snapshot_run, + state_file, +) + + +def write_snapshot_config(tmp_path, extra=""): + cfg_file = tmp_path / "config.yaml" + cfg_file.write_text(textwrap.dedent(f""" + snapshot_profiles: + essentials-daily: + engine: restic + host_id: test-host + repository: rclone:example-drive:backups/WORKSTATION01/restic/essentials-daily + cache_dir: {tmp_path}/cache + state_dir: {tmp_path}/state + password_file: {tmp_path}/password + retry_lock: 7m + repo_homes: + - {tmp_path}/repos + explicit_repos: + - {tmp_path}/myrig + include_paths: + - {tmp_path}/Documents + exclude_paths: + - {tmp_path}/repos/archives + tags: + - workstation01 + {extra} + """)) + (tmp_path / "password").write_text("secret\n") + (tmp_path / "password").chmod(0o600) + (tmp_path / "cache").mkdir() + (tmp_path / "state").mkdir() + return cfg_file + + +def init_git_repo(path): + path.mkdir(parents=True) + (path / "README.md").write_text("test\n") + import subprocess + + subprocess.run(["git", "-C", str(path), "init"], check=True, capture_output=True) + subprocess.run(["git", "-C", str(path), "config", "user.email", "test@example.com"], check=True) + subprocess.run(["git", "-C", str(path), "config", "user.name", "Test"], check=True) + subprocess.run(["git", "-C", str(path), "add", "README.md"], check=True) + subprocess.run(["git", "-C", str(path), "commit", "-m", "init"], check=True, capture_output=True) + + +def test_snapshot_profile_config_parsed(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + assert profile.engine == "restic" + assert profile.repository.startswith("rclone:") + assert profile.repo_homes == [f"{tmp_path}/repos"] + assert profile.explicit_repos == [f"{tmp_path}/myrig"] + assert profile.exclude_paths == [f"{tmp_path}/repos/archives"] + + +def test_snapshot_profile_default_drive_client_opt_in_requires_boolean_true(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path, 'allow_default_rclone_drive_client: "false"'))) + profile = cfg.snapshot_profiles["essentials-daily"] + assert profile.allow_default_rclone_drive_client is False + + +def test_restic_backup_args_include_required_flags(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + args = ResticRunner(profile).backup_args("/tmp/source", ["bbackup", "profile=essentials-daily"]) + assert args[:3] == ["restic", "-r", profile.repository] + assert "--password-file" in args + assert "--cache-dir" in args + assert "--json" in args + assert "--retry-lock" in args + assert "7m" in args + assert "--host" in args + assert "test-host" in args + assert "--tag" in args + assert "--exclude" in args + + +def test_restic_backup_args_expand_path_based_excludes(tmp_path, monkeypatch): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + monkeypatch.setenv("BBACKUP_EXCLUDE_ROOT", str(tmp_path / "cache-root")) + profile.exclude_paths = ["$BBACKUP_EXCLUDE_ROOT/cache", "~/CloudDrive", "node_modules"] + + args = ResticRunner(profile).backup_args("/tmp/source", ["bbackup"]) + exclude_values = [ + args[index + 1] + for index, value in enumerate(args) + if value == "--exclude" + ] + + assert str((tmp_path / "cache-root" / "cache").resolve()) in exclude_values + assert "node_modules" in exclude_values + assert not any(value.startswith("$BBACKUP_EXCLUDE_ROOT") for value in exclude_values) + + +def test_discover_git_repos_uses_immediate_children_and_explicit_roots(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + init_git_repo(tmp_path / "repos" / "repo-a") + (tmp_path / "repos" / "not-git").mkdir(parents=True) + init_git_repo(tmp_path / "myrig") + + discovered = discover_git_repos(profile) + paths = {repo.path for repo in discovered} + assert str((tmp_path / "repos" / "repo-a").resolve()) in paths + assert str((tmp_path / "myrig").resolve()) in paths + assert str((tmp_path / "repos" / "not-git").resolve()) not in paths + + +def test_discover_git_repos_skips_excluded_repo_home_children(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + profile.exclude_paths.append(f"{tmp_path}/repos/starred-repos") + init_git_repo(tmp_path / "repos" / "repo-a") + init_git_repo(tmp_path / "repos" / "starred-repos") + + discovered = discover_git_repos(profile) + paths = {repo.path for repo in discovered} + assert str((tmp_path / "repos" / "repo-a").resolve()) in paths + assert str((tmp_path / "repos" / "starred-repos").resolve()) not in paths + + +def test_reconcile_marks_deleted_repo_retired_only_after_snapshot(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + init_git_repo(tmp_path / "repos" / "repo-a") + discovered = discover_git_repos(profile) + first = reconcile_repos(profile, discovered) + repo_id = first["active_repo_ids"][0] + state = load_state(profile) + state["repos"][repo_id]["successful_snapshot_ids"] = ["abc123"] + from bbackup.snapshot import save_state + + save_state(profile, state) + + import shutil + + shutil.rmtree(tmp_path / "repos" / "repo-a") + result = reconcile_repos(profile, []) + assert repo_id in result["retired_repo_ids"] + assert result["alerts"] == [] + + +def test_reconcile_alerts_when_deleted_repo_has_no_successful_snapshot(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + init_git_repo(tmp_path / "repos" / "repo-a") + discovered = discover_git_repos(profile) + reconcile_repos(profile, discovered) + + import shutil + + shutil.rmtree(tmp_path / "repos" / "repo-a") + result = reconcile_repos(profile, []) + assert result["alerts"][0]["code"] == "missing_without_snapshot" + + +def test_path_reuse_with_different_fingerprint_alerts(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + init_git_repo(tmp_path / "repos" / "repo-a") + discovered = discover_git_repos(profile) + first = reconcile_repos(profile, discovered) + repo_id = first["active_repo_ids"][0] + state = load_state(profile) + state["repos"][repo_id]["status"] = "retired" + state["repos"][repo_id]["successful_snapshot_ids"] = ["abc123"] + state["repos"][repo_id]["fingerprint"] = "different" + from bbackup.snapshot import save_state + + save_state(profile, state) + + result = reconcile_repos(profile, discovered) + assert result["alerts"][0]["code"] == "path_reuse_conflict" + + +def test_no_origin_retired_path_reuse_alerts_after_reinit(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + repo_path = tmp_path / "repos" / "repo-a" + init_git_repo(repo_path) + first = reconcile_repos(profile, discover_git_repos(profile)) + repo_id = first["active_repo_ids"][0] + state = load_state(profile) + state["repos"][repo_id]["status"] = "retired" + state["repos"][repo_id]["successful_snapshot_ids"] = ["abc123"] + save_state(profile, state) + + import shutil + import subprocess + + shutil.rmtree(repo_path) + repo_path.mkdir(parents=True) + (repo_path / "README.md").write_text("different repo\n") + subprocess.run(["git", "-C", str(repo_path), "init"], check=True, capture_output=True) + subprocess.run(["git", "-C", str(repo_path), "config", "user.email", "test@example.com"], check=True) + subprocess.run(["git", "-C", str(repo_path), "config", "user.name", "Test"], check=True) + subprocess.run(["git", "-C", str(repo_path), "add", "README.md"], check=True) + subprocess.run(["git", "-C", str(repo_path), "commit", "-m", "different"], check=True, capture_output=True) + result = reconcile_repos(profile, discover_git_repos(profile)) + + assert result["alerts"][0]["code"] == "path_reuse_conflict" + + +def test_reconcile_updates_active_repo_after_new_commit(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + repo_path = tmp_path / "repos" / "repo-a" + init_git_repo(repo_path) + first = reconcile_repos(profile, discover_git_repos(profile)) + repo_id = first["active_repo_ids"][0] + + (repo_path / "README.md").write_text("changed\n") + import subprocess + + subprocess.run(["git", "-C", str(repo_path), "add", "README.md"], check=True) + subprocess.run(["git", "-C", str(repo_path), "commit", "-m", "change"], check=True, capture_output=True) + second = reconcile_repos(profile, discover_git_repos(profile)) + + assert second["alerts"] == [] + assert second["active_repo_ids"] == [repo_id] + state = load_state(profile) + assert state["repos"][repo_id]["head"] + + +def test_reconcile_migrates_legacy_head_derived_repo_id(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + repo_path = tmp_path / "repos" / "repo-a" + init_git_repo(repo_path) + discovered = discover_git_repos(profile) + repo = discovered[0] + legacy_fingerprint = repo.legacy_fingerprint + legacy_repo_id = repo.legacy_repo_id + assert legacy_repo_id != repo.repo_id + + save_state(profile, { + "repos": { + legacy_repo_id: { + "repo_id": legacy_repo_id, + "path": repo.path, + "fingerprint": legacy_fingerprint, + "origin": repo.origin, + "head": repo.head, + "status": "active", + "successful_snapshot_ids": ["old-snapshot"], + } + } + }) + + plan = snapshot_plan(profile) + assert plan["repos"][0]["repo_id"] == repo.repo_id + assert f"repo_id={repo.repo_id}" in plan["repos"][0]["args"] + + result = reconcile_repos(profile, discovered) + state = load_state(profile) + assert result["active_repo_ids"] == [repo.repo_id] + assert legacy_repo_id not in state["repos"] + assert state["repos"][repo.repo_id]["successful_snapshot_ids"] == ["old-snapshot"] + + +def test_reconcile_blocks_unknown_active_same_path_identity_change(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + repo_path = tmp_path / "repos" / "repo-a" + init_git_repo(repo_path) + repo = discover_git_repos(profile)[0] + conflicting_fingerprint = hashlib.sha256(b"not-this-repo").hexdigest() + conflicting_repo_id = hashlib.sha256(f"{repo.origin or repo.path}:{conflicting_fingerprint}".encode("utf-8")).hexdigest()[:16] + + save_state(profile, { + "repos": { + conflicting_repo_id: { + "repo_id": conflicting_repo_id, + "path": repo.path, + "fingerprint": conflicting_fingerprint, + "origin": repo.origin, + "head": repo.head, + "status": "active", + "successful_snapshot_ids": ["old-snapshot"], + } + } + }) + + plan = snapshot_plan(profile) + + assert plan["repos"] == [] + assert plan["alerts"][0]["code"] == "active_path_identity_conflict" + + +def test_purge_plan_refuses_active_repo(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + init_git_repo(tmp_path / "repos" / "repo-a") + result = reconcile_repos(profile, discover_git_repos(profile)) + repo_id = result["active_repo_ids"][0] + try: + purge_plan(profile, repo_id) + except SnapshotError as exc: + assert "active" in str(exc) + else: + raise AssertionError("purge_plan should refuse active repos") + + +def test_purge_plan_for_retired_repo_is_dry_run_first(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + init_git_repo(tmp_path / "repos" / "repo-a") + result = reconcile_repos(profile, discover_git_repos(profile)) + repo_id = result["active_repo_ids"][0] + state = load_state(profile) + state["repos"][repo_id]["successful_snapshot_ids"] = ["abc123"] + from bbackup.snapshot import save_state + + save_state(profile, state) + retire_repo(profile, repo_id) + plan = purge_plan(profile, repo_id) + assert plan["dry_run_required"] is True + assert "--dry-run" in plan["forget_args"] + assert "--dry-run" not in plan["destructive_args_after_confirmation"] + assert f"repo_id={repo_id}" in plan["forget_args"] + + +def test_snapshot_plan_cli_json(tmp_path): + cfg_file = write_snapshot_config(tmp_path) + init_git_repo(tmp_path / "repos" / "repo-a") + result = CliRunner().invoke( + cli, + ["--config", str(cfg_file), "snapshot", "plan", "--profile", "essentials-daily", "--output", "json"], + ) + assert result.exit_code == 0 + data = json.loads(result.output) + assert data["success"] is True + assert data["data"]["repos"][0]["repo_id"] + + +def test_snapshot_plan_does_not_persist_state(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + init_git_repo(tmp_path / "repos" / "repo-a") + + plan = snapshot_plan(profile) + + assert plan["repos"] + assert not state_file(profile).exists() + + +def test_snapshot_init_dry_run_cli_json(tmp_path): + cfg_file = write_snapshot_config(tmp_path) + result = CliRunner().invoke( + cli, + [ + "--config", str(cfg_file), + "snapshot", "init", + "--profile", "essentials-daily", + "--dry-run", + "--output", "json", + ], + ) + assert result.exit_code == 0 + data = json.loads(result.output) + assert data["data"]["dry_run"] is True + assert data["data"]["args"][-1] == "init" + + +def test_snapshot_commands_accept_required_options_from_input_json(tmp_path): + cfg_file = write_snapshot_config(tmp_path) + init_git_repo(tmp_path / "repos" / "repo-a") + cfg = Config(config_path=str(cfg_file)) + profile = cfg.snapshot_profiles["essentials-daily"] + repo_id = reconcile_repos(profile, discover_git_repos(profile))["active_repo_ids"][0] + state = load_state(profile) + state["repos"][repo_id]["successful_snapshot_ids"] = ["abc123"] + save_state(profile, state) + + runner = CliRunner() + commands = [ + ("plan", {"profile": "essentials-daily", "output": "json"}), + ("init", {"profile": "essentials-daily", "dry_run": True, "output": "json"}), + ("run", {"profile": "essentials-daily", "dry_run": True, "output": "json"}), + ( + "check", + { + "profile": "essentials-daily", + "read_data_subset": "5%", + "dry_run": True, + "output": "json", + }, + ), + ( + "restore", + { + "profile": "essentials-daily", + "snapshot_id": "latest", + "target": str(tmp_path / "restore-target"), + "dry_run": True, + "output": "json", + }, + ), + ("retire", {"profile": "essentials-daily", "repo_id": repo_id, "output": "json"}), + ("purge-plan", {"profile": "essentials-daily", "repo_id": repo_id, "output": "json"}), + ("schedule", {"profile": "essentials-daily", "output": "json"}), + ] + + for command, payload in commands: + result = runner.invoke( + cli, + [ + "--config", str(cfg_file), + "snapshot", command, + "--input-json", json.dumps(payload), + ], + ) + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["success"] is True + + +def test_snapshot_command_missing_json_required_param_returns_json_error(tmp_path): + cfg_file = write_snapshot_config(tmp_path) + result = CliRunner().invoke( + cli, + [ + "--config", str(cfg_file), + "snapshot", "restore", + "--input-json", json.dumps({"profile": "essentials-daily", "output": "json"}), + ], + ) + + assert result.exit_code == 1 + data = json.loads(result.output) + assert data["success"] is False + assert "Missing required option: --snapshot-id" in data["errors"][0] + + +def test_snapshot_restore_env_target_guard_uses_expanded_path(tmp_path, monkeypatch): + cfg_file = write_snapshot_config(tmp_path) + target = tmp_path / "restore-target" + target.mkdir() + (target / "existing.txt").write_text("existing\n") + monkeypatch.setenv("RESTORE_TARGET", str(target)) + + result = CliRunner().invoke( + cli, + [ + "--config", str(cfg_file), + "snapshot", "restore", + "--profile", "essentials-daily", + "--snapshot-id", "latest", + "--target", "$RESTORE_TARGET", + "--output", "json", + ], + ) + + assert result.exit_code == 1 + data = json.loads(result.output) + assert "Restore target is not empty" in data["errors"][0] + + +def test_snapshot_run_dry_run_reports_alerts_as_unsuccessful(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + init_git_repo(tmp_path / "repos" / "repo-a") + reconcile_repos(profile, discover_git_repos(profile)) + + import shutil + + shutil.rmtree(tmp_path / "repos" / "repo-a") + result = snapshot_run(profile, dry_run=True) + + assert result["success"] is False + assert result["alerts"][0]["code"] == "missing_without_snapshot" + + +def test_snapshot_run_dry_run_reports_no_targets_as_unsuccessful(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + profile.repo_homes = [] + profile.explicit_repos = [] + profile.include_paths = [] + + result = snapshot_run(profile, dry_run=True) + + assert result["success"] is False + assert result["alerts"] == [] + + +def test_snapshot_health_reports_profile_dependencies(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + with patch("bbackup.snapshot.shutil.which", return_value="/usr/bin/tool"), \ + patch("bbackup.snapshot.socket.gethostname", return_value="test-host"), \ + patch("bbackup.snapshot.subprocess.run") as mock_run: + mock_run.return_value = MagicMock(returncode=0, stdout="restic 0.19.0\n", stderr="") + result = check_snapshot_profile(profile) + assert result["ok"] is True + assert result["checks"]["restic"]["ok"] is True + assert result["checks"]["rclone"]["ok"] is True + + +def test_snapshot_health_rejects_google_drive_remote_without_client_id(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + + def run_side_effect(args, **kwargs): + if args[:3] == ["rclone", "config", "show"]: + return MagicMock(returncode=0, stdout="[example-drive]\ntype = drive\nscope = drive\n", stderr="") + return MagicMock(returncode=0, stdout="restic 0.19.0\n", stderr="") + + with patch("bbackup.snapshot.shutil.which", return_value="/usr/bin/tool"), \ + patch("bbackup.snapshot.socket.gethostname", return_value="test-host"), \ + patch("bbackup.snapshot.subprocess.run", side_effect=run_side_effect): + result = check_snapshot_profile(profile) + + assert result["ok"] is False + assert result["checks"]["rclone_drive_client_id"]["ok"] is False + assert "no client_id" in result["checks"]["rclone_drive_client_id"]["message"] + + +def test_snapshot_health_accepts_google_drive_remote_with_client_id(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + + def run_side_effect(args, **kwargs): + if args[:3] == ["rclone", "config", "show"]: + return MagicMock( + returncode=0, + stdout="[example-drive]\ntype = drive\nclient_id = example.apps.googleusercontent.com\n", + stderr="", + ) + return MagicMock(returncode=0, stdout="restic 0.19.0\n", stderr="") + + with patch("bbackup.snapshot.shutil.which", return_value="/usr/bin/tool"), \ + patch("bbackup.snapshot.socket.gethostname", return_value="test-host"), \ + patch("bbackup.snapshot.subprocess.run", side_effect=run_side_effect): + result = check_snapshot_profile(profile) + + assert result["ok"] is True + assert result["checks"]["rclone_drive_client_id"]["ok"] is True + + +def test_snapshot_health_accepts_default_drive_client_when_profile_allows_it(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path, "allow_default_rclone_drive_client: true"))) + profile = cfg.snapshot_profiles["essentials-daily"] + + def run_side_effect(args, **kwargs): + if args[:3] == ["rclone", "config", "show"]: + return MagicMock(returncode=0, stdout="[example-drive]\ntype = drive\nscope = drive\n", stderr="") + return MagicMock(returncode=0, stdout="restic 0.19.0\n", stderr="") + + with patch("bbackup.snapshot.shutil.which", return_value="/usr/bin/tool"), \ + patch("bbackup.snapshot.socket.gethostname", return_value="test-host"), \ + patch("bbackup.snapshot.subprocess.run", side_effect=run_side_effect): + result = check_snapshot_profile(profile) + + assert result["ok"] is True + assert result["checks"]["rclone_drive_client_id"]["ok"] is True + assert "explicit profile configuration" in result["checks"]["rclone_drive_client_id"]["message"] + + +def test_snapshot_health_reports_unsupported_engine_without_crashing(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + profile.engine = "unknown" + + with patch("bbackup.snapshot.shutil.which", return_value="/usr/bin/tool"), \ + patch("bbackup.snapshot.socket.gethostname", return_value="test-host"), \ + patch("bbackup.snapshot.subprocess.run") as mock_run: + mock_run.return_value = MagicMock(returncode=0, stdout="restic 0.19.0\n", stderr="") + result = check_snapshot_profile(profile) + + assert result["ok"] is False + assert result["checks"]["engine"]["ok"] is False + assert result["checks"]["repository_initialized"]["message"] == "unsupported snapshot engine" + + +def test_snapshot_check_enforces_google_drive_client_id_or_opt_in(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + + def run_side_effect(args, **kwargs): + if args[:3] == ["rclone", "config", "show"]: + return MagicMock(returncode=0, stdout="[example-drive]\ntype = drive\nscope = drive\n", stderr="") + return MagicMock(returncode=0, stdout="", stderr="") + + with patch("bbackup.snapshot.shutil.which", return_value="/usr/bin/tool"), \ + patch("bbackup.snapshot.subprocess.run", side_effect=run_side_effect): + try: + snapshot_check(profile) + except SnapshotError as exc: + assert "no client_id" in str(exc) + else: + raise AssertionError("snapshot_check should enforce Google Drive client_id") + + +def test_snapshot_check_preflight_rejects_wrong_hostname(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + + def run_side_effect(args, **kwargs): + if args[:3] == ["rclone", "config", "show"]: + return MagicMock( + returncode=0, + stdout="[example-drive]\ntype = drive\nclient_id = example.apps.googleusercontent.com\n", + stderr="", + ) + return MagicMock(returncode=0, stdout="restic 0.19.0\n", stderr="") + + with patch("bbackup.snapshot.shutil.which", return_value="/usr/bin/tool"), \ + patch("bbackup.snapshot.socket.gethostname", return_value="other-host"), \ + patch("bbackup.snapshot.subprocess.run", side_effect=run_side_effect): + try: + snapshot_check(profile) + except SnapshotError as exc: + assert "hostname" in str(exc) + else: + raise AssertionError("snapshot_check should reject profiles for a different host") + + +def test_snapshot_check_preflight_rejects_permissive_password_file(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + (tmp_path / "password").chmod(0o644) + + def run_side_effect(args, **kwargs): + if args[:3] == ["rclone", "config", "show"]: + return MagicMock( + returncode=0, + stdout="[example-drive]\ntype = drive\nclient_id = example.apps.googleusercontent.com\n", + stderr="", + ) + return MagicMock(returncode=0, stdout="restic 0.19.0\n", stderr="") + + with patch("bbackup.snapshot.shutil.which", return_value="/usr/bin/tool"), \ + patch("bbackup.snapshot.socket.gethostname", return_value="test-host"), \ + patch("bbackup.snapshot.subprocess.run", side_effect=run_side_effect): + try: + snapshot_check(profile) + except SnapshotError as exc: + assert "password file must not be group/world accessible" in str(exc) + else: + raise AssertionError("snapshot_check should reject permissive password files") + + +def test_schedule_units_include_matching_services_for_all_timers(tmp_path): + cfg = Config(config_path=str(write_snapshot_config( + tmp_path, + ( + 'schedule:\n' + ' daily_time: "02:15"\n' + ' maintenance_time: "Sun 04:30"\n' + ' verification_time: "monthly"\n' + ' verification_read_data_subset: "10%"' + ), + ))) + profile = cfg.snapshot_profiles["essentials-daily"] + from bbackup.snapshot import schedule_units + + units = schedule_units(profile, config_path=str(tmp_path / "custom-config.yaml")) + + assert "bbackup-test-host-essentials-daily.service" in units + assert "bbackup-test-host-essentials-daily.timer" in units + assert "bbackup-test-host-essentials-daily-maintenance.service" in units + assert "bbackup-test-host-essentials-daily-maintenance.timer" in units + assert "bbackup-test-host-essentials-daily-verification.service" in units + assert "bbackup-test-host-essentials-daily-verification.timer" in units + assert "--config" in units["bbackup-test-host-essentials-daily.service"] + assert "custom-config.yaml snapshot run --profile essentials-daily" in units[ + "bbackup-test-host-essentials-daily.service" + ] + assert "snapshot check --profile essentials-daily\n" in units[ + "bbackup-test-host-essentials-daily-maintenance.service" + ] + assert "--read-data-subset 10%%" in units["bbackup-test-host-essentials-daily-verification.service"] + + +def test_schedule_units_escape_default_percent_in_verification_subset(tmp_path): + cfg = Config(config_path=str(write_snapshot_config(tmp_path))) + profile = cfg.snapshot_profiles["essentials-daily"] + from bbackup.snapshot import schedule_units + + units = schedule_units(profile) + + assert "--read-data-subset 5%%" in units["bbackup-test-host-essentials-daily-verification.service"] diff --git a/uv.lock b/uv.lock index 469c362..451ac24 100644 --- a/uv.lock +++ b/uv.lock @@ -4,7 +4,7 @@ requires-python = ">=3.12" [[package]] name = "bbackup" -version = "1.8.4" +version = "1.8.5" source = { editable = "." } dependencies = [ { name = "click" },