Add incremental backup and restore by Chyenne8 · Pull Request #1 · Chyenne8/foremanctl

Chyenne8 · 2026-06-30T17:36:12Z

Summary

Implements full incremental backup/restore capability using GNU tar's --listed-incremental mechanism. Reduces backup time and storage for large Pulp deployments by only backing up changed files.

Backup Features

Add --incremental CLI parameter:

foremanctl backup /backups --incremental /backups/foreman-backup-TIMESTAMP

Implementation:

Modified tar commands to use --listed-incremental with .snar snapshot files
.config.snar tracks foremanctl state changes
.pulp.snar tracks Pulp content changes
Replaced community.general.archive with raw tar for --listed-incremental support
Copy .snar files from previous backup before tar runs
Validate previous backup exists and contains .snar files
Enhanced error messages suggest alternative backups when .snar missing
Metadata records: is_incremental, base_backup_dir, base_backup_timestamp

Error handling automation:

Scans parent directory for backups with .snar support
Lists available alternatives with type and timestamp
Provides ready-to-use command with suggested backup

Restore Features

Auto-detect incremental backups from metadata:

Parse metadata.yml to identify backup type
Display full vs incremental in validation output

Backup chain validation:

Verify base backup directory exists
Confirm base backup metadata matches expected timestamp
Check if base backup already restored via .last_restore_timestamp
Fail with clear instructions if chain incomplete

Add --chain for automatic chain restore:

foremanctl restore /backups/foreman-backup-TIMESTAMP --chain

Chain restore automation:

Recursively walks metadata to build full dependency chain
Restores backups in chronological order (full -> inc1 -> inc2)
Databases restored only from full backup
Incrementals apply file changes using --listed-incremental=/dev/null
Eliminates manual multi-step restore process

Handle .snar files during restore:

Detect .snar presence to identify incremental archives
Use tar --listed-incremental=/dev/null for extraction
Fall back to unarchive module for full backups

Usage Examples

Create full backup (generates .snar files):

foremanctl backup /backups

Create incremental backup:

foremanctl backup /backups --incremental /backups/foreman-backup-20260629T120000

Restore with auto-chain (recommended):

foremanctl restore /backups/foreman-backup-20260701T080000 --chain

Manual chain restore:

foremanctl restore /backups/foreman-backup-20260629T120000
foremanctl restore /backups/foreman-backup-20260630T080000 --force
foremanctl restore /backups/foreman-backup-20260701T080000 --force

Validate incremental backup:

foremanctl restore /backups/foreman-backup-TIMESTAMP --validate

Testing

Backup:

✅ Full backup creates .config.snar and .pulp.snar files
✅ Incremental backup significantly smaller than full
✅ Error with suggestions when .snar files missing
✅ Metadata correctly tracks incremental status

Restore:

✅ Incremental detected from metadata
✅ Chain validation prevents incomplete restores
✅ --chain automatically handles dependencies
✅ Clear errors for missing base backups

Files Modified

Backup:

src/playbooks/backup/metadata.obsah.yaml
src/roles/backup/tasks/main.yaml
src/roles/backup/tasks/pulp_content.yaml
src/roles/backup/tasks/metadata.yaml
docs/user/backup.md

Restore:

src/playbooks/restore/metadata.obsah.yaml
src/roles/restore/tasks/validate.yaml
src/roles/restore/tasks/main.yaml
src/roles/restore/tasks/restore_pulp_content.yaml
src/roles/restore/tasks/restore_foremanctl_state.yaml
src/roles/restore/tasks/restore_databases.yaml
src/roles/restore/tasks/restore_chain.yaml (new)
src/roles/restore/tasks/restore_chain_walk.yaml (new)
src/roles/restore/tasks/restore_single.yaml (new)
docs/user/restore.md

Stats

15 files changed
550 insertions
24 deletions

Replace hardcoded quay.io image references with an iop_image fixture that reads container_image and container_tag from each role's defaults, keeping tests in sync with deployment configuration. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Adds an Ansible lint rule to help ensure any future volume mounts that are added follow this rule.

Standardize certificate directory from /root/certificates to /var/lib/foremanctl/certs across IOP roles, variable files, and documentation for consistency with the default certificate source. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

… documentation

Define a canonical `databases` list in database.yml with feature tags, replacing scattered conditional enumeration across consumers. A computed `all_databases` filters by enabled_features, and custom Jinja2 filters derive postgresql_databases/postgresql_users from it — eliminating static list duplication and the merge pre_tasks in deploy.yaml. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The development playbooks load src/vars/database.yml which now uses custom filters (to_postgresql_databases, to_postgresql_users). Without this path, playbooks run from the development directory fail with "No filter named" errors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The version published in the Hashicorp portal is broken, see https://redhat.atlassian.net/browse/CS-3411 for details

The default and custom_server certificate vars files defined identical paths since custom certificates are normalized into the same directory structure during deployment. Remove the vars file indirection and use a single certificates.yml for all certificate sources. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Guard CA generation with a stat check so existing CA artifacts are never overwritten during deploy. Without this, the community.crypto modules detect parameter mismatches (different subject, passphrase) against a migrated CA and silently regenerate it, breaking certificate trust. The CA generation block now only runs when ca.crt does not exist (fresh install) or when certificates_renew_ca is explicitly set to true. Add --certificate-renew-ca as a deploy parameter for explicit CA renewal. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Move foreman-installer certificate normalization into the migrate subcommand so it runs once during migration rather than on every deploy. The migrate_foreman_installer role copies certs from /root/ssl-build/ into /var/lib/foremanctl/certs/, persists the CA passphrase to a dedicated file, and backs up the original directory. Detect custom server certificates by comparing the internal CA with the server CA. When they differ, persist certificates_source: custom_server to prevent subsequent deploys from overwriting the custom server cert. Remove the installer certificate source since migrated certs use the default source paths after normalization. Mark certificate path parameters as IGNORE in the answer file migration since the role handles cert files directly. Separate I/O from the migrate_answers module so it only transforms and returns mapped parameters. The playbook handles writing to stdout, output files, and the parameters file. Migration is preview-by-default and requires --apply to perform changes. Update integration tests to read control-node state files from OBSAH_STATE rather than hardcoding paths or checking the remote server. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Implements comprehensive offline backup functionality for Foreman deployments: - Backs up all databases (foreman, candlepin, pulp, 5 IOP DBs) - Backs up podman secrets, networks, volumes, quadlet files - Backs up systemd units and foremanctl state - Includes metadata with container image digests for restore compatibility - Preflight checks for running tasks and database integrity (amcheck) - Automatic service restoration on failure Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Three columns dropped upstream require view changes: - canonical_facts: read insights_id directly as UUID column (RedHatInsights/insights-host-inventory@7754a5ad) - system_profile_facts: LEFT JOIN to system_profiles_static for operating_system, host_type, owner_id (RedHatInsights/insights-host-inventory@62f4d244) - stale_timestamp: compute from last_check_in using HBI defaults: 29h/7d/30d (RedHatInsights/insights-host-inventory@426d5e0d). Fixes pre-existing bug where stale_warning and culled were chained off stale_timestamp instead of computed independently from last_check_in. WHERE clause updated: canonical_facts->'insights_id' IS NOT NULL replaced with insights_id != '00000000-...' (DEFAULT_INSIGHTS_ID sentinel). Staleness intervals are hardcoded HBI defaults. Per-org overrides from hbi.staleness table are not reflected in this view. Signed-off-by: Roman Blanco <rblanco@redhat.com>

You can now skip them with: pytest -k "not slow"

Features can now declare conflicts in features.yaml to prevent incompatible features from being enabled together. The check_features role validates enabled_features for conflicts during deploy. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Fixes multiple issues preventing deploy-dev from running over existing PG 13 databases: - Add failed_when: false to service stops (handles missing foreman.target) - Change upgrade container volume from :Z to :z (fixes SELinux permission denied) - Replace creates: guard with proper stream check for dnf module switch-to Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

The deploy-dev playbook was missing the systemd_target role, so foreman.target was never created. This meant services couldn't be stopped cleanly before re-deploying, causing handler failures (e.g. pulp-api.socket refusing to start because pulp-api.service was already active). Add the systemd_target role (matching production deploy) and a pre-task to stop foreman.target before re-deploying. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Remove the deploy-dev pre-task that stopped foreman.target before deployment - the roles should handle idempotent re-deploys. Remove the explicit postgresql stop from the upgrade path since postgresql is PartOf=foreman.target and stopping the target cascades to it. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Revert pre_install changes since production always deploys on clean systems. Instead, handle the postgresql module switch as a deploy-dev pre-task for dev re-deploy scenarios. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The SELinux issue was caused by the main postgresql container not being stopped before the upgrade container ran, leading to conflicting MCS categories. With foreman.target properly stopping services first, :Z (private label) is correct. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Move the systemd_target role before postgresql in deploy-dev so foreman.target exists when the upgrade task needs to stop it. This eliminates the need for failed_when: false. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Move the dnf module switch-to task from the deploy-dev pre-tasks into the foreman_development role where it belongs alongside other dev-specific module setup like nodejs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

foreman-installer PR #935 adds foreman-certs but is not yet merged, so the mock-installer role was pinned to a COPR build of that PR. Deploy the binary and Kafo scenario config as role files instead, relying only on the nightly foreman-installer-katello package for puppet, modules, and kafo-export-params. Extract repository setup into a reusable role so mock-installer is self-contained and does not require forge setup-repositories to run first. Co-Authored-By: Claude <noreply@anthropic.com>

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 6 to 7. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v6...v7) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>

Co-Authored-By: Claude <noreply@anthropic.com>

Since theforeman/foreman-documentation#5082 we no longer need to allow this action in foreman-rake.

The parameter was renamed in theforeman#672 but the documentation was not updated to reflect the new name. Co-authored-by: Cursor <cursoragent@cursor.com>

Complete offline backup restore with validation, database/content restoration, and automatic redeployment. Includes safety checks to prevent accidental data loss.

Implements incremental backup/restore using GNU tar's --listed-incremental mechanism. Reduces backup time and storage for large Pulp deployments by only backing up changed files. Backup: - Add --incremental CLI parameter pointing to previous backup directory - Use --listed-incremental with .snar snapshot files (.config.snar, .pulp.snar) - Validate previous backup exists and contains .snar files - Enhanced error messages suggest alternative backups when .snar missing - Metadata tracks: is_incremental, base_backup_dir, base_backup_timestamp Restore: - Auto-detect incremental backups from metadata - Validate backup chain (base backup exists, timestamps match) - Use tar --listed-incremental=/dev/null for incremental extraction - Only delete state/content directories for full backups, not incrementals Usage: foremanctl backup /backups foremanctl backup /backups --incremental /backups/foreman-backup-TIMESTAMP foremanctl restore /backups/foreman-backup-FULL foremanctl restore /backups/foreman-backup-INC1 --force foremanctl restore /backups/foreman-backup-INC2 --force

evgeni and others added 30 commits June 8, 2026 07:42

add a var-defaults[no-empty] check that ensures no defaults are empty

e9a542d

add no-static-secrets ansible-lint rule

20ae1dc

fix static default passwords in iop roles

ec8d6a3

fix static default password in postgresql role

cb30613

allow empty ca paths

512917a

mark ca_key_password as no-qa for static secret -- it's a path

687bb4f

fix new lint rules in foreman_development role

a172c2b

move tests to the normal location, so they get executed

f3e2b69

Rename rules

f84331a

add missing params

bc7caca

Fix broker inventory parsing of ruamel YAML tags

377474a

test on push to stable branches

c8f3a05

Set explicit volume mount permissions

e8e0971

Adds an Ansible lint rule to help ensure any future volume mounts that are added follow this rule.

Set choices for flavor, so users can't select a wrong one

3040e07

Fixes theforeman#525 - Add checks to health subcommand; add developer…

5454b1a

… documentation

Fixes theforeman#562 - Remove dead code from health subcommand

099a9fe

update iop-core-kafka image tag to latest-kafka-4.2.0

c3f96f0

Remove the duplicate HTTPS entry

e384672

Fix small typo

3831701

Download CentOS Stream 10 Vagrant box directly from CentOS mirrors

286895b

The version published in the Hashicorp portal is broken, see https://redhat.atlassian.net/browse/CS-3411 for details

don't load template if feature is disabled

757f936

Fixes theforeman#560 - Add systemctl wants to foreman.target for httpd

7854249

Chyenne8 force-pushed the incremental-backup-restore branch from 37e667e to fd5cad4 Compare July 15, 2026 19:04

ekohl and others added 20 commits July 15, 2026 16:34

Mark slow tests to allow for quicker iteration

15b6880

You can now skip them with: pytest -k "not slow"

Add links to testinfra's documentation

441d4dc

Rename certificate-renew-ca to certificate-ca-renew

4d2d756

Add foreman-rake wrapper

6bb3f24

assert /rhsm on the proxy gives 200

8c3f4b3

Symlink foreman-test script to /usr/local/bin

4767211

Co-Authored-By: Claude <noreply@anthropic.com>

Remove katello:migrate_deb_content_attributes_to_pulp3 from foreman-rake

9fe6c66

Since theforeman/foreman-documentation#5082 we no longer need to allow this action in foreman-rake.

Fix --certificate-renew-ca to --certificate-ca-renew in docs

dfc9b84

The parameter was renamed in theforeman#672 but the documentation was not updated to reflect the new name. Co-authored-by: Cursor <cursoragent@cursor.com>

Rename cname to server_alias (theforeman#677)

8fec558

Add foremanctl restore command

947113b

Complete offline backup restore with validation, database/content restoration, and automatic redeployment. Includes safety checks to prevent accidental data loss.

Chyenne8 force-pushed the incremental-backup-restore branch 5 times, most recently from 9f89545 to 83aee45 Compare July 22, 2026 16:00

Chyenne8 force-pushed the incremental-backup-restore branch from 83aee45 to e37c295 Compare July 29, 2026 17:16

Chyenne8 force-pushed the incremental-backup-restore branch from e37c295 to 81bca87 Compare July 29, 2026 17:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add incremental backup and restore - #1

Add incremental backup and restore#1
Chyenne8 wants to merge 121 commits into
masterfrom
incremental-backup-restore

Chyenne8 commented Jun 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

18 participants

Conversation

Chyenne8 commented Jun 30, 2026

Summary

Backup Features

Restore Features

Usage Examples

Testing

Files Modified

Stats

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

18 participants