Networks rootless - #7
Open
pablomh wants to merge 98 commits into
Open
Conversation
Introduces the deploy_network role, which wraps containers.podman.podman_network to create a named bridge network with configurable properties: internal (no external routing), isolate (no cross-network packet forwarding), IPv6, subnet, gateway, and DNS. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace host networking with four named Podman bridge networks: - foreman-db (internal, isolated): PostgreSQL and its clients - foreman-cache (internal, isolated): Redis and its clients - foreman-app: application layer — Foreman, Candlepin, Pulp, httpd - foreman-proxy-net: Foreman and the smart proxy Services reach each other by container DNS name instead of localhost. Per-service notes: - postgresql: also accessible via a Unix socket bind-mounted at /var/run/postgresql, used by Ansible's community.postgresql modules during deployment so no TCP port needs to be published to the host. - redis: publishes no ports; clients reach it by DNS name on foreman-cache. - candlepin: Tomcat binds to 0.0.0.0 so that other containers on the bridge network can reach it via its DNS name. - migration containers (foreman db:migrate, pulp migrations): attached to foreman-db when the database is internal; switched to foreman-app when the database is external, because foreman-db uses internal: true which removes the default gateway, making it impossible for containers on that network to reach an external database host. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The development setup runs the Foreman Rails process directly on the host rather than in a container, so it cannot resolve container bridge DNS names such as postgresql or candlepin. Keeping host networking for all services in the devel environment means every service remains reachable on localhost as before, without requiring per-service URL overrides for the Rails process. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
foreman-certs generates certificates only for localhost and the server FQDN. Now that Foreman connects to Candlepin via the bridge DNS name "candlepin", TLS hostname validation fails against those certificates. A dedicated certificate with SAN=candlepin, signed by the installer CA, is generated using the openssl CLI (consistent with the rest of the codebase). The certificate validity is set to 7300 days (20 years), matching the default used by puppet-certs (theforeman/puppet-certs manifests/init.pp $expiration parameter). The signing step runs on every deployment to ensure the certificate is always freshly dated, consistent with how foreman-certs handles the other installer certificates. The localhost certificate previously used by Candlepin's Tomcat is now unused and removed: - "localhost" dropped from certificates_hostnames — the certificates role no longer generates the cert. - localhost_key / localhost_certificate removed from default_certificates.yml and installer_certificates.yml. The healthcheck is updated to validate the new certificate instead of skipping verification with --insecure. --resolve candlepin:23443:127.0.0.1 forces the connection to the loopback so it works in both bridge networking (where other containers reach Candlepin by DNS name) and host networking (where "candlepin" would not resolve via container DNS). Tests are updated to route connectivity checks through the foreman container (which shares the same bridge network as candlepin), to verify against the new certificate, and to check candlepin_certificate expiry instead of the now-removed localhost_certificate. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds docs/deployment.md with a description of the four bridge networks introduced by this series, their properties (internal, isolate), which containers are attached to each, and the rationale for the port publishing decisions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pablomh
force-pushed
the
networks_rootless
branch
from
March 9, 2026 14:35
fb87658 to
8ccd08a
Compare
Creates a dedicated service user (foremanctl) for running Podman containers in rootless mode. The role handles all prerequisites: - acl package (required for Ansible become_user on CentOS 10) - User and group creation with a non-login shell - subuid/subgid ranges for user namespace mapping - Unprivileged port binding via sysctl - loginctl enable-linger so the user session survives across boots - XDG_RUNTIME_DIR discovery (exposed as rootless_user_xdg_runtime_dir) - Quadlet and systemd user config directories Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pablomh
force-pushed
the
networks_rootless
branch
3 times, most recently
from
March 9, 2026 15:52
77eb19e to
101ae2a
Compare
…plays
base.yaml gains foremanctl_user/group/home/quadlet_dir/systemd_user_dir
vars. It also introduces foremanctl_become_method and foremanctl_runtime_env
as the single place to control how all rootless become blocks operate:
- foremanctl_become_method: swap between 'sudo' (default) or
'community.general.machinectl' (or any future method) without touching
individual roles
- foremanctl_runtime_env: holds XDG_RUNTIME_DIR for sudo; set to {} when
the become method provides the session environment automatically
deploy.yaml is split into two plays so the rootless_user role can expose
foremanctl_xdg_runtime_dir as a fact before the service roles run.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Move the default certificate directory from /root/certificates to /var/lib/foremanctl/certificates so that the rootless containers running in foremanctl's user namespace can read them without elevated privileges. The path is hardcoded rather than using foremanctl_home because default_certificates.yml is loaded before base.yaml and base.yaml itself references certificate path vars. ca.yml: add base directory creation; set owner/group to foremanctl_user/foremanctl_group on all directories and generated files. issue.yml: set owner/group to foremanctl_user/foremanctl_group on all generated keys, CSRs and certificates. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Directories created owned by foremanctl_user/group so the rootless user can manage them - podman_unshare become method used to chown the data directory to the postgres container UID/GID (26:26) inside the user namespace, avoiding the need for a shell-based 'sudo -u foremanctl podman unshare chown' - Container runs inside a foremanctl_become_method block with foremanctl_runtime_env, so the become method and XDG handling are controlled by a single pair of variables in base.yaml - Socket dir moved from /var/run/postgresql to /var/lib/foremanctl/postgresql-socket (mode 0777) and mounted inside the container at /tmp/socket to avoid the sclorg entrypoint chowning /var/run/postgresql to postgres:postgres with mode 0700 - community.postgresql.* tasks remain socket-based: root bypasses DAC on the 0777 socket dir and the socket file itself is world-accessible - unix_socket_directories configured to include /tmp/socket so the existing socket path is reachable from within other containers - All systemd calls use scope: user (tasks and handler) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Data directory created owned by foremanctl_user/group - podman_unshare become method used to chown /var/lib/redis to the redis container UID/GID (1001:1001) inside the user namespace - Container runs inside a foremanctl_become_method/foremanctl_runtime_env block for consistent become abstraction across all roles - All systemd calls use scope: user - REGISTRY_AUTH_FILE removed from image pull (handled by podman's user credential store in rootless mode) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Log directories created owned by foremanctl_user/group (containers run as uid 0 in user namespace = foremanctl on the host) - All podman secret, image and container tasks moved inside a foremanctl_become_method/foremanctl_runtime_env block - All systemd calls use scope: user (tasks and handler) - REGISTRY_AUTH_FILE removed from image pull Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Storage directories created owned by foremanctl_user/group with appropriate SELinux labels - Secret key and symmetric key files generated as foremanctl_user/group - podman_unshare become method used to chown /var/lib/pulp to pulp container UID/GID (700:700) inside the user namespace - All podman secret, image and container tasks moved inside a foremanctl_become_method/foremanctl_runtime_env block - pulp-worker.target moved from /etc/systemd/system/ to foremanctl_systemd_user_dir (user scope) - Pulp worker symlinks updated to use foremanctl_quadlet_dir instead of /etc/containers/systemd/ - Old pulp-worker cleanup also uses foremanctl_quadlet_dir - All systemd calls use scope: user (tasks and handlers) - REGISTRY_AUTH_FILE removed from image pulls Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- All podman secret, image and container tasks moved inside a foremanctl_become_method/foremanctl_runtime_env block - Dynflow symlinks updated to use foremanctl_quadlet_dir instead of /etc/containers/systemd/ - Recurring task timer files written to foremanctl_systemd_user_dir instead of /etc/systemd/system/ - All systemd calls use scope: user (tasks and handlers) - REGISTRY_AUTH_FILE removed from image pull Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- All podman secret, image and container tasks moved inside a foremanctl_become_method/foremanctl_runtime_env block - All systemd calls use scope: user (tasks and handler) - REGISTRY_AUTH_FILE removed from image pull - smart_proxy registration task kept outside the become block as it communicates with Foreman over HTTPS, not with Podman Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- foreman.target moved from /etc/systemd/system/ to foremanctl_systemd_user_dir so it lives in the user systemd scope - All tasks run inside a foremanctl_become_method/foremanctl_runtime_env block - systemd daemon_reload and service start both use scope: user Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduces three building blocks for testing rootless Podman deployments:
run_as(server, user, cmd)
Runs a command on the remote host as a specific user. The underlying
mechanism is controlled by the FOREMANCTL_RUN_AS_METHOD environment
variable so different approaches can be tried without touching test
files:
- runuser (default): propagates exit codes correctly
- machinectl: full login session, XDG_RUNTIME_DIR auto-set, but
always exits 0 (use stdout/stderr assertions, not result.succeeded)
- systemd-run: runs in the user systemd scope
foremanctl_run(server, cmd)
Convenience shortcut for run_as(server, 'foremanctl', cmd).
UserService
Testinfra-compatible class for user-scope systemd units, exposing
is_running, is_enabled and exists properties backed by
'systemctl --user' commands run via run_as().
user_service fixture
Module-scoped factory: user_service('redis') returns a UserService
for the foremanctl user, so test assertions read identically to the
existing server.service() ones.
Also fixes the certificates fixture to use the rootless certificate
directory (/var/lib/foremanctl/certificates) instead of /root/certificates.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Service checks use user_service() fixture (user-scope systemd) - Socket path updated to /var/lib/foremanctl/postgresql-socket/ - podman exec calls wrapped with foremanctl_run() so they run in the foremanctl user namespace regardless of RUN_AS_METHOD Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Service check uses user_service() fixture (user-scope systemd) - podman exec wrapped with foremanctl_run() Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Service check uses user_service() fixture (user-scope systemd) - podman exec and podman secret calls wrapped with foremanctl_run() - openssl certificate read on the host (no container exec) unchanged Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Service checks use user_service() fixture (user-scope systemd) - pulp-worker list query uses 'systemctl --user' via foremanctl_run() - pulp-worker.target check uses user_service() - podman exec call wrapped with foremanctl_run() and -ti flag dropped (non-interactive in CI) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Service checks use user_service() fixture (user-scope systemd) - Dynflow symlink path updated from /etc/containers/systemd/ to /var/lib/foremanctl/.config/containers/systemd/ - Dynflow service instance checks use user_service() - Recurring timer/service checks use user_service() Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both service checks now use user_service() (user-scope systemd). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When fapolicyd is enabled, Ansible become_user: foremanctl fails because fapolicyd blocks Python module execution from /var/tmp/ (the temp path Ansible uses when becoming an unprivileged user). Add the missing rule. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pablomh
force-pushed
the
networks_rootless
branch
from
March 9, 2026 16:00
101ae2a to
8894813
Compare
When deploying rootlessly, networks must be created in the foremanctl user's Podman namespace. Previously the task ran as root, making networks invisible to rootless containers running as foremanctl. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The remote-database playbook was missing the rootless user setup play and base.yaml vars, causing foremanctl_user/foremanctl_group/foremanctl_runtime_env to be undefined when the postgresql role ran on the database host. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Revert back to the original working implementation that uses runuser -l flag with XDG_RUNTIME_DIR export and single-quote escaping (escaped), as it was in commit a2c086d. XDG_RUNTIME_DIR is required so systemctl --user can locate the D-Bus session bus socket. Also revert postgresql and pulp tests to their original commands from a2c086d, avoiding the multiple-shell-layer escaping issues introduced by subsequent "fix" commits. The original psql -c style commands work correctly through testinfra's sudo wrapper. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pablomh
force-pushed
the
networks_rootless
branch
from
March 14, 2026 00:28
16560be to
5c91c63
Compare
Add AddHost={{ ansible_facts['fqdn'] }}:host-gateway to dynflow-sidekiq
containers so they can resolve the server FQDN to reach the HTTPD
reverse proxy and Pulp API. Without this, Katello background workers
fail with PulpRpmClient::ApiError: Connection failed when processing
repository tasks. The Foreman web container already had this fix
(added in 975e3d8), but the dynflow workers were missed.
Skip test_postgresql_service when database_mode=external: in that
configuration postgresql runs on a separate database VM, not on
quadlet. The companion test_postgresql_missing_with_external already
asserts the correct behaviour for external mode.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
pablomh
force-pushed
the
networks_rootless
branch
from
March 14, 2026 11:11
f1f3926 to
a641bdc
Compare
- Remove || true from upgrade stop step; the stop should succeed - Group installer certificate pre_tasks in a single block with one when condition; drop redundant become: true (inherited from play) - Remove noisy comment from candlepin handler - Fix hardcoded /var/lib/foremanctl in certs_secrets.yml comments - Add blank line before Create Candlepin configuration task - Add foremanctl_registry_auth_file to base.yaml and include REGISTRY_AUTH_FILE in foremanctl_runtime_env so all podman pull tasks in foremanctl blocks get registry authentication automatically Flush handlers after service start is intentional: restarting before start would fail on first deploy. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
runuser requires root and misparses --user as its own flag. Replace with sudo -u foremanctl and set XDG_RUNTIME_DIR explicitly so systemctl --user can reach the D-Bus session bus. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pablomh
force-pushed
the
networks_rootless
branch
6 times, most recently
from
March 14, 2026 22:21
e97e9f7 to
a087be1
Compare
Tests run last (filename sorts after redis_test.py) since stopping the target would break any subsequent service checks. This also validates the stop command used in the upgrade workflow. Use server.run with systemctl --machine=foremanctl@ --user for all systemctl operations, consistent with how non-podman commands are run in other tests. After each lifecycle operation, poll the Foreman HTTPS frontend (https://server_fqdn/api/v2/ping with CA cert) matching the pattern used in test_https_foreman_ping. On timeout, dump user service states for diagnosis. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pablomh
force-pushed
the
networks_rootless
branch
5 times, most recently
from
March 15, 2026 03:52
248bedd to
23920dc
Compare
Tests run last (filename sorts after redis_test.py) since stopping the target would break any subsequent service checks. This also validates the stop command used in the upgrade workflow. Use server.run with systemctl --machine=foremanctl@ --user for all systemctl operations. After each lifecycle operation, poll the Foreman HTTPS frontend (https://server_fqdn/api/v2/ping with CA cert) matching the pattern used in test_https_foreman_ping. On timeout, dump service states, foreman.service status, container states and foreman logs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pablomh
force-pushed
the
networks_rootless
branch
4 times, most recently
from
March 15, 2026 13:02
2102bca to
69f50bc
Compare
Tests run last (filename sorts after redis_test.py) since stopping the target would break any subsequent service checks. This also validates the stop command used in the upgrade workflow. Use server.run with systemctl --machine=foremanctl@ --user for all systemctl operations. After each lifecycle operation, poll the Foreman HTTPS frontend (https://server_fqdn/api/v2/ping with CA cert) matching the pattern used in test_https_foreman_ping. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pablomh
force-pushed
the
networks_rootless
branch
from
March 15, 2026 18:32
69f50bc to
e1eb1a7
Compare
Wrap the systemd restart in block/rescue to collect service status and container logs when the restart fails, then re-raise so CI still marks the run as failed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This reverts commit 31505fa.
Foreman depends on Candlepin being fully initialised before it can
serve requests reliably. Without this ordering, foreman.service starts
simultaneously with candlepin.service, competing for CPU and disk I/O
during the JVM startup. This pushes Rails loading time close to the
default TimeoutStartSec (90s) and causes intermittent failures on
restart, particularly when the system is under load.
Because candlepin.service already declares:
After=redis.service postgresql.service
the transitive ordering becomes:
postgresql --+
+--> candlepin --> foreman
redis -------+
redis.service and postgresql.service are kept as explicit dependencies
to preserve correct ordering in future non-Katello deployments that
may not include candlepin.service.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extends foreman_target_test.py with lifecycle tests intended to run at the end of the test suite. Tests that foreman.target can be cleanly stopped and restarted, and that Foreman becomes available again afterwards via the HTTPS endpoint (same curl pattern as test_https_foreman_ping in httpd_test.py).
Incorporates upstream commits: - Use candlepin.service in foreman startup ordering (f2e162d) - Add foreman.target lifecycle tests (5a2eb8b) Conflicts resolved keeping rootless implementations; remote-database playbook updated to include both SSL cert vars and postgresql_networks. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.