Rootless linger - #3
Open
pablomh wants to merge 31 commits into
Open
Conversation
Enables Foreman deployment using rootless Podman containers, improving security by eliminating root privileges for container operations while maintaining full functionality of the Foreman/Katello stack. What this achieves: - Rootless deployment: All containers run without root privileges - Single service user: Entire stack (foreman, candlepin, postgresql, redis, pulp) runs under dedicated foremanctl user - Migration support: Automated playbook converts existing rootful deployments - Namespace isolation: Container volumes properly mapped to user namespaces - Port access: Unprivileged ports configured for HTTP/HTTPS access New capabilities: - Fresh deployments use rootless mode by default - Existing rootful deployments can migrate with automated playbook - All certificates, data, and configuration preserved during migration - User lingering enabled for service persistence across reboots This change modernizes the deployment model to follow container security best practices while maintaining backward compatibility through the migration playbook. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…s/systemd/users/$UID Changes: - Remove XDG_RUNTIME_DIR management (systemd manages it automatically) - Keep linger support (enable-linger and linger checks) - Move quadlet files from ~/.config/containers/systemd to /etc/containers/systemd/users/$UID - Remove XDG_RUNTIME_DIR from all environment blocks and sudo commands - Update playbooks to map only foremanctl_quadlet_dir variable - Simplify rootless_user role to not track or verify XDG_RUNTIME_DIR Benefits: - Simpler deployment without manual XDG_RUNTIME_DIR management - Quadlet files in /etc are easier to manage (no become_user needed) - Systemd automatically manages runtime directories with linger enabled Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
The containers.podman.podman_container module needs to know where to write quadlet files. Without the quadlet_dir parameter, it defaults to writing to ~/.config/containers/systemd/, but we want files in /etc/containers/systemd/users/$UID/. This fixes errors like: src file does not exist: /etc/containers/systemd/users/996/pulp-worker@.container Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Applied architecture: 1. Quadlet management: Root (writes to /etc/containers/systemd/users/$UID/) 2. Image/secret management: User (via become_user) 3. Systemd operations: User (via become_user with scope: user) Changes: - postgresql: Split into user block (images/secrets), root task (quadlet), user block (systemd) - redis: Split into user task (image), root task (quadlet), user block (systemd) - candlepin: Split into user block (secrets/artemis/image), root task (quadlet), user block (systemd) - pulp: Split into user block (images/secrets), root tasks (quadlets), user block (systemd/migration) Note: foreman and foreman_proxy roles still need similar refactoring Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Applied the same architecture to foreman and foreman_proxy roles: 1. Quadlet management: Root (writes to /etc/containers/systemd/users/$UID/) 2. Image/secret management: User (via become_user) 3. Systemd operations: User (via become_user with scope: user) Foreman role changes: - User block: Pull image and create all secrets - Root tasks: Deploy foreman quadlet, dynflow template, create symlinks - Root task (conditional): Deploy recurring task quadlets - User block: Render timers, daemon-reload, migrate DB, start services - No change: Wait for services and configure proxy Foreman_proxy role changes: - User task: Pull image - Includes: configs.yaml and certs.yaml (self-contained with own become_user) - Root task: Deploy quadlet - User block: daemon-reload, start service - No change: Register proxy to Foreman All roles now consistently separate: - Root-managed quadlet files in /etc/containers/systemd/users/$UID/ - User-managed images/secrets/services in user's podman storage Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
When tasks run as root (without become_user), files/directories/symlinks are created with root:root ownership by default. Explicitly setting owner: root and group: root is redundant. Removed from: - foreman: Dynflow symlinks - pulp: Worker symlinks - rootless_user: Quadlet directory Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Changed all documentation references from:
systemctl --user <command>
To:
systemctl --machine={{ foremanctl_user }}@ --user <command>
And for journalctl:
journalctl --user --machine={{ foremanctl_user }}@ <command>
This is necessary because without XDG_RUNTIME_DIR being set, systemctl
cannot connect to the user's systemd instance. Using --machine allows
root to interact with a specific user's systemd instance without
needing to set environment variables or use sudo -u.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Systemd module needs XDG_RUNTIME_DIR to connect to the user's systemd instance, even when using become_user. Without it, we get: "Could not find the requested service postgresql: host" Changes: - Map foremanctl_xdg_runtime_dir in deploy playbooks - Add environment: XDG_RUNTIME_DIR to postgresql systemd operations - Add environment: XDG_RUNTIME_DIR to postgresql handler Testing with postgresql first before applying to all roles. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This reverts commit 7a8b98a.
Added DEBUG tasks to investigate why systemd can't find the service: 1. Check if quadlet .container file was created 2. Verify file permissions and ownership 3. Test if foremanctl user can read the file 4. Check quadlet directory permissions 5. List systemd units after daemon-reload 6. Check for generated .service file location 7. Display journal logs This will help identify if the issue is: - File not created - Permission denied - Generator not running - Service not being generated from quadlet Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Set all quadlet files to mode 0644 to ensure the foremanctl user's systemd instance can read them when they're stored in /etc/containers/systemd/users/$UID/. Modified roles: - postgresql: Make quadlet file readable (0644) - redis: Make quadlet file readable (0644) - candlepin: Make quadlet file readable (0644) - pulp: Make all three quadlet files readable (0644) - foreman: Make foreman and dynflow quadlet files readable (0644) - foreman_proxy: Make quadlet file readable (0644) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
pablomh
force-pushed
the
rootless_linger
branch
6 times, most recently
from
February 23, 2026 17:41
db6ecc0 to
43271a8
Compare
Testing if running daemon-reload as root via systemctl --machine avoids the need to become_user and set XDG_RUNTIME_DIR. Also commented out chmod 0644 on quadlet files to test if it's still needed. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
pablomh
force-pushed
the
rootless_linger
branch
from
February 23, 2026 17:42
43271a8 to
cb72cca
Compare
The systemd module with scope: user doesn't work when running as root without become_user. Use systemctl --machine command instead. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Confirmed that quadlet files must be world-readable even when using systemctl --machine. The systemd-generator runs as the user and needs to read the quadlet files to generate service files. Without 0644 permissions, the generator silently fails and no service file is created. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Changes applied to all service roles:
1. SELinux labels (container_var_lib_t):
- PostgreSQL: data directory
- Redis: data directory
- Pulp: storage directories and secret files
- Candlepin: log directories
2. systemctl --machine pattern:
- Use 'systemctl --machine=${user}@ --user' for all systemd operations
- Run as root instead of become_user
- Eliminates need for XDG_RUNTIME_DIR environment variable
- Applies to daemon-reload, start, restart, enable operations
3. Quadlet files:
- Keep chmod 0644 (confirmed necessary for systemd-generator)
4. Cleanup:
- Remove debug tasks from postgresql
- Remove flush_handlers where not needed
- Simplify pulp service start (no async needed)
Modified roles:
- postgresql: SELinux label, systemctl pattern, cleanup
- redis: SELinux label, systemctl pattern
- candlepin: SELinux label, systemctl pattern
- pulp: SELinux labels, systemctl pattern, refactor service management
- foreman: systemctl pattern, refactor service management
- foreman_proxy: systemctl pattern
All handlers updated to use systemctl --machine pattern.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Remove commented-out task block that was used during testing. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Changes: - Fix role paths in dev playbooks (use 'rootless_user' instead of relative path) - Update variable mapping to use foremanctl_quadlet_dir (was foremanctl_xdg_runtime_dir) - Remove obsolete fapolicyd rule for /var/lib/foremanctl/.ansible/tmp/ Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Changes: - Add UserService class in conftest.py to check user systemd services - Add user_service fixture that uses systemctl --machine to check user services - Add foremanctl_user and foremanctl_uid fixtures - Update all service tests to use user_service instead of server.service(): * foreman_test.py: foreman, dynflow instances, recurring timers * postgresql_test.py: postgresql service * redis_test.py: redis service * candlepin_test.py: candlepin service * pulp_test.py: pulp-api, pulp-content, pulp-worker services and target * foreman_proxy_test.py: foreman-proxy service * foreman_target_test.py: foreman.target - Update quadlet file location tests to check /etc/containers/systemd/users/$UID/ All services now run in the foremanctl user's systemd instance, not the system systemd, so tests must check the user systemd via systemctl --machine. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add mode: '0755' to rootless_user_systemd_user_dir (~/.config/systemd/user) to follow standard systemd conventions. The directory needs to be: - Readable by systemd (755) - Owned by the user - Standard location for user unit files (.timer, .target, etc.) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Removed 5 debug tasks that were leftover from development: - rootless_user_home display - rootless_user_quadlet_dir display - rootless_user_systemd_user_dir display - .config/containers path display - rootless_user_linger_result display These are no longer needed in production. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Moved become: true from play level to task level for 'Install podman'. Why this change: - With play-level become: true, Ansible stages modules as root in /var/tmp/ - When tasks use become_user: foremanctl, this creates cross-user execution - fapolicyd blocks: 'foremanctl trying to execute root-owned files from /var/tmp' After this change: - Ansible stages modules as SSH user (vagrant) in ~/.ansible/tmp/ - 'Install podman' task: becomes root to install package - Image pull tasks: become foremanctl_user to pull images - No cross-user execution from /var/tmp/ -> fapolicyd happy The fapolicyd rule for /home/vagrant/.ansible/tmp/ already exists in CI. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add skip_list for command-instead-of-module rule in .ansible-lint config. Why we skip this rule: We intentionally use 'systemctl --machine=user@ --user' commands instead of the ansible.builtin.systemd module for managing user systemd instances. Reasons: - systemctl --machine is the recommended systemd approach for cross-session management (managing user systemd from root context) - Avoids need for XDG_RUNTIME_DIR environment variable setup - More reliable than systemd module with become_user - Avoids D-Bus security policy issues The ansible.builtin.systemd module with become_user requires environment variable management and can hit D-Bus permission issues, making systemctl --machine the cleaner and more reliable solution for rootless deployments. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
In rootless Podman deployment, secrets are stored per-user. Running 'podman secret inspect' as root cannot access the foremanctl user's secrets. Updated assert_secret_content helper to run podman commands as the foremanctl user using sudo -u.
- Update postgresql tests to use 'sudo -u foremanctl podman exec' for internal database mode (rootless containers) - Update pulp test to access containers in foremanctl user's storage - Fix upgrade workflow to stop foreman.target using systemctl --machine pattern instead of system systemctl In rootless Podman, containers run in the user's storage namespace and cannot be accessed by root's podman commands. Tests now check database mode and use appropriate command prefix.
- Rename migrate.yaml to migrate-to-rootless.yaml to match directory name - Add metadata.obsah.yaml with help text for obsah/foremanctl CLI This fixes test_filename_matches_directory and test_is_documented test failures.
Add -H flag to 'sudo -u foremanctl' commands to set HOME to the target user's home directory. Without this, sudo tries to preserve the current working directory (/home/vagrant) which the foremanctl user cannot access, causing 'cannot chdir to /home/vagrant: Permission denied' errors. The -H flag makes sudo change to the target user's home directory, avoiding permission issues when running podman commands as foremanctl.
The -H flag only sets HOME but doesn't change the working directory. The -i flag simulates an initial login, which: - Sets HOME to the target user's home directory - Changes to that home directory - Processes the user's shell profile This fully avoids the 'cannot chdir to /home/vagrant: Permission denied' error when running podman commands as the foremanctl user.
The foremanctl user has shell=/sbin/nologin and cannot access /home/vagrant. Following the pattern used in playbooks, change to /tmp (which is world-accessible) before running podman commands: cd /tmp && sudo -u foremanctl podman ... Since testinfra's server.run() already wraps commands in 'sudo /bin/sh -c', there's no need for an additional sh -c wrapper. This avoids both: - 'This account is currently not available' (from sudo -i with nologin) - 'cannot chdir to /home/vagrant: Permission denied' (from sudo -u) The /tmp directory is accessible to all users, so foremanctl can use it as a working directory.
The certificates fixture was hardcoding certificates_ca_directory to '/var/lib/foremanctl/certificates' in the Jinja2 context, which overrode the template's own definition during variable substitution. For installer certificates, this caused all paths to use /var/lib/foremanctl/certificates instead of /root/ssl-build, resulting in 'No such file or directory' errors. Use two-pass rendering: 1. First pass: render template to extract certificates_ca_directory value 2. Second pass: render with that value in context for proper Jinja2 variable substitution in path definitions This allows each certificate source template to define its own certificates_ca_directory while still supporting Jinja2 templating.
Unified service checking for: - User services (systemctl --machine=user@ --user) - Podman containers (podman inspect) Replaced separate service checking implementations with a single GenericService class that takes an optional 'user' parameter: - If user is set: checks user systemd service - If user is None: checks podman container This simplifies the code and makes it easier to support both internal (user service) and external (container) database modes.
External database setup (remote-database.yaml) also configures rootless user environment and runs PostgreSQL as a user service, not as a rootful container. The playbook runs: 1. rootless_user role - creates foremanctl user 2. postgresql role - runs PostgreSQL as user service This is identical to internal database setup, just on a separate host. Updated: - database_user_service: Always use user=foremanctl_user for both modes - PostgreSQL tests: Remove database_mode branching, use same pattern for both internal and external (cd /tmp && sudo -u foremanctl podman) This fixes 'no such object: postgresql' errors in external database mode where GenericService was incorrectly checking root's podman instead of foremanctl user's podman.
pablomh
force-pushed
the
rootless_linger
branch
2 times, most recently
from
February 25, 2026 00:33
49c3dab to
d8938ca
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.